In my recent project there are quite many tasks that run in the background – generate thumbnails, detect colours on the pictures, run DB updates etc. Sure, all that is handled using cron and Yii commands. However there is a little problem. Consider we have DB update routine that should import new stock data from the datafeed. Datafeed is uploaded hourly, but upload can’t be scheduled to minutes – connection speed, different errors may interrupt the upload. On the other hand, sometimes processing takes 5 mins and sometimes – a few hours because in the first variant we just replace records and in the latter – download images for the new products.

The problem

So if we schedule to run every 5 mins (to check if update is ready), then we get the following:

As a result, DB will have wrong data because records were inserted twice. Sure, we can control this with DB constraints, but duplicates is not the only possible error.

Solution overview

The only variant to fix this – go Unix way and create lock file when some process starts. Actually that is a good variant, but I decided to use one file for all processes. Sure, I have to lock it when I read/write from it and other processes have to wait when I’m done with this. It is not brilliant, but I decided to go this way and I’ll show how I did this in this post.

First of all, I didn’t want to put the same code for locking/checking/unlocking in every command. So I created the taskmanager extension. It is not yet finished, so I don’t publish it.

Technical details

This is an application component which is extended from CApplicationComponent class and implements TaskManger interface:

Each command/process has it’s ID. ID is a plain text without spaces. Although spaces are not restricted and it will work with them, I decided to apply the same naming conventions as for class properties in Yii.
Let’s go through the methods:

• isRunning($id). Takes process ID as a parameter and return true if this process is running.
• logStarted($id). This checks if process is not already running (or if duplicates are allowed) and adds records to the pids file. Actually this interface may be implemented using the DB, so no locking or other things will be needed.
• logFinished($id). Removes record from the pids file (or from DB if implemented using it).
• logTerminated($id). Nearly the same as previous, but it doesn’t check if process is running, it tries to remove the record anyway.
• sendSignal($id, $signal). Sends signal to the process. Process should check it’s buffer file when it is running and react to signals. It is not implemented because requires changes in the command code.
• getStatus($id). Returns current status of a given process. It is not implemented either for the same reason.
• getLog($id). Should return log of a process execution. Not implemented either.

And now let’s take a look at the class implementing it.
First of all, it’s initialization:

• $pidFile is where all currently running processes are stored
• $pidLogFile is where we write “DD.MM.YYY HH:MM:SS process XXX started” and “ DD.MM.YYY HH:MM:SS process XXX finished”
• $processDir is where log files and exchange files should reside (methods sendSignal, getStatus, getLog)

We get these variables from the config file and update to fit current application path. Also, since we extend the class from CApplicationComponent, we should perform standard initialization, an we do this by calling parent::init() in the last line.

Since we’re using file, it should be locked from other processes when we’re working with it. So we need a method that will open it and wait if it is busy now:

Idea is pretty simple – we try to lock the file. If we succeed, we return the file pointer and we throw exception otherwise. We try it several times until we reach timeout. Intervals are random to ensure that different processes started simultaneously will not try to lock it at the same time.
And now we can implement our methods:

So now we just need to call these methods from our command. But you know, I’m very lazy, so I created a base class for all my commands:

It’s constructor created the process ID from the class name. It is quite convenient feature. It also handles start and stop so all we need to do is just to implement runCmd() method in the derived classes like this:

Then you just type:

Can’t start process! It is already running!

First command starts routine in background. Second command starts the same command but in the terminal. So you’ll see the output for it. It says that it can’t be started because previous run is not finished.

There is one thing you should do in order to make that work. You should add a component to the application configuration. Open the protected/config/console.php file and add into the components list:

And into the import section:

Other problems

I’ve been using this solution for 5 months. Practice shows that sometimes process may run out of memory or some other error occurs that is not caught by out exception handling and it is terminated without our $tm->logFinished() call. This leads to the situation when process is not actually running, but task manager thinks it is because it’s record is still present in the pids file. In order to prevent such situations, I modified isRunning routine like this:

This performs additional check by trying to find the selected PID in the system process list by listing all processes ps -A and checking if our pid is there (grep [pid]). If result is empty, then process is not running.

There is a potential pitfall in the isRunning method. If you try to do something with pids file before it is closed in line 21, you’ll get into the deadlock because file is locked for modifications. That’s why I call logTerminated after it is closed. Be attentive!

I’d be thankful if someone takes the code and adds signal processing there and maybe implements DBTaskManager in addition to FileTaskManager. Let me know if someone is interested, we can collaborate on this and deliver a nice ready-made extension.

As usual, any comments are welcome!

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

In the previous post I outlined the system’s specs and use cases. We also selected the primary use case we should start from – that is filtering screen.

The complexity with DB design here is that we want maximum flexibility on the one hand and we don’t want to lose much in performance on the other hand.
Let’s recap what data about student we should store:

1. Phone
2. Password
3. Email
4. ICQ
5. Name
6. Surname
7. Patronymic
8. Gender
9. Birthday
10. Faculty
11. Course
12. Group
13. Clubs
14. Tags
15. OK to send SMS (true/false)

Items 1-9 and #15 are scalar values and can be easily stored in User table as fields of the correspondent type. While Faculty, Course, Group and Clubs are either N-M (Clubs) or 1-N relations. So obvious way to handle them is to create tables and cross-table for N-M relation. Tags can be also represented as N-M relation.
It seems OK until we don’t want to extend the system by adding some more fields. But even without extensibility in mind, we should create 3 separate tables (for courses, faculties and clubs). However if we look closer at these, we’ll see that all of them are kind of groups and we can create a kind of hierarchy:

• University
  • Faculty
    • Course
      • Group
  • Club
    • Section

So we can consider everything to be group. With this approach we also store information about relations between these groups.
With such approach we can add more types of groups in the future if needed. Or this solution can be ported to another domain model (e.g. school or business organization) without changes in the structure. We should only define, which groups are required for user. In this case Faculty, course and group are required. However, knowing the group, we can get both course and faculty. But I’m not sure if that is a good idea because this way we’ll have to make several queries to the DB in order to get faculty instead of one.
Here’s the DB scheme we came up with:

Database design

Filtering it

Let’s try to compose some simple queries to test if our DB can cope with them.

1. Let’s get all male users who’re elder than 18:
   1. SELECT * FROM User WHERE gender=‘male’ AND birthday <= ‘1992-03-19 23:59:59′

   Date for the birthday is set to the 18 years back from now. Everyone who was born before or on 19 of March 1992 will fall into the condition. And gender is obvious.

2. Let’s try to get all male that have first names starting with ‘K’ and who study on the first or second course:
   1. SELECT u.* FROM User u INNER JOIN UserGroup ug ON u.userID=ug.userID WHERE u.gender=‘male’ AND u.name LIKE ‘K%’ AND ug.groupID IN (5,6)

   Suppose that in Group table we have groups for the first course and second course and they have Ids 5 and 6 respectively.

3. Let’s try to search for all who is on the 3rd course and is also in Geeks’ Club:
   1. SELECT u.* FROM User u INNER JOIN UserGroup ug ON u.userID=ug.userID WHERE ug.groupID IN (7, 15)

   7 is ID of the group “3rd course” and 15 is ID of the group “Geeks’ Club”.
   We have the problem with the last query It will find all users who either studies on the 3rd course or attends Geeks’ Club. However we need 3rd course and Geeks’ club.
   There are two possible solutions here. One is SQL-based, another is PHP-based.

   1. 1.SQL-based:
      1. SELECT * FROM User WHERE userID IN (SELECT userID FROM UserGroup WHERE userID IN(SELECT userID FROM UserGroup WHERE groupID=15) AND groupID=7)

      Here we select users who have group=15. Then from these users we select the ones who have group=7 as well, and finally we select user records that correspond the userIDs found.
      I think, the variant with INNER JOIN is also possible here (at least, joining results of the groups filtering instead of sub-querying). It just takes some time to figure it out and it has to deal more with optimization than with the principle of filtering.
      Anyway, the more groups we have here, the more complex this query becomes.
      UPDATE: Here I’ve been advised to use the following query which is more simple and efficient:

      1. SELECT
      2.   u.*
      3. FROM
      4.   USER u
      5. INNER JOIN
      6.   UserGroup ug1
      7. ON
      8.   u.id = ug1.userID AND ug1.groupID=7
      9. INNER JOIN
      10.   UserGroup ug2
      11. ON
      12.   u.id = ug2.userID AND ug2.groupID=15

      I think this is the best way of making such queries. Great thanks to ScallioXTX

   2. PHP-based. Basically, the principle is the same:
      1. $group7 = getUsersForGroup(7);
      2. $group15 = getUsersForGroup(15);
      3. $userIDs = array_intersect($group7, $group15);
      4. $users = getUsersByIDs($userIDs);

      getUsersForGroup actually executes the query:

      1. SELECT userID FROM UserGroup WHERE groupID=[GROUP]

      getUsersByID executes the following query:

      1. SELECT * FROM User WHERE userID IN ([ids])

      array_intersect is a PHP function that returns the values present in all given arrays, returns the intersection of sets.
      For example, if we give it 3 arrays:
      (1, 5, 8, 15)
      (5, 8, 15, 19)
      (1, 5, 15, 25)
      it will return (5, 15).
      In this case we execute several little queries instead of one big. In MySQL this often gives benefits in performance.

So that’s our filtering approach.

There is one thing to add here. If we have group #1 and #2 and group#2 is a child of #1. And user selected to filter users which are both in #1 and #2, then we should perform a little optimization and search only for #2 because that will be the correct answer to the query. This situation is stupid, but we should act smart even in a stupid queries

Maybe there are better solutions. We’re open to them! Moreover, we realize that there are more experienced developers that already solved such problem. Help us

Tomorrow we have a meeting where we’ll be discussing the architecture, classes etc. I’ll upload our results on the weekends or early next week. This depends on the amount of work done during the meeting.
Stay tuned! Subscribe to my blog’s RSS, share your opinions/suggestions in comments and don’t miss the next post in this series!

Liked the post? Bookmark it

Hide Sites

Related posts:

1. Requirements and Use Cases. SMS Notification System. Background I’ve graduated in March 2010 and I don’t need...

Related posts brought to you by Yet Another Related Posts Plugin.

I’ve graduated in March 2010 and I don’t need to visit university any more. However, I enjoyed studying there and I know what challenges my fellow students-programmers have. That’s why I continue the job started this autumn – computer club, we call it “Geeks’ Club”. Sure, we’re not true geeks there, but we tend to be So I’m helping students who’re interested in web-technologies and in object-oriented programming. I explain them things that are not covered in the standard university course. Things, that are more practical and they can be paid money for.

Here I’ll be covering both OOP and web-dev, but we’ll start from design hence OOP. The story began when some time passed after we started learning OOP at the club meetings and I decided that there is enough theory and separate examples, it’s time to build something practical that will be challenging for me too.
I thought of different systems, but finally decided to pick up the SMS notification one. It requires both user interface, database, API (my friends want to communicate to it from standalone Java and C++ applications) and overall application design. In addition, I’d like such system to be deployed at our university and it is interesting for me to create one.

Introduction

This is going to be a series of articles, I’ll describe the progress and challenges we come across as we proceed with development. I think this kind of posts will be interesting for those who study OOP and want to apply what they’ve learned to some practical things. I’ll try to guide my students through the process using the iterative development methodology.

We have meetings once a week (on Saturdays), so I’ll post once or twice a week, depending on amount of work done on weekends and my free time during the week.

I think, this series will contain around 10 posts. We’re going to share the work done as an open-source project. There is nothing to share at the moment, so I’ll give the link when we code something

System overview

Vision

Obviously, the main purpose of the system is to send SMS notifications to students or tutors. These notifications may be quite different – from the timetable changes to some announcements or even warnings (e.g. “Due to the weather conditions, today is a day-off for all. Stay at home and have a good time”).

Actually the system isn’t that complex. Basically all it needs to do is to enable administrator pick a group of users, enter the message and send it through the SMS gateway. The tricky things start from “pick a group of users”. We want this to be as flexible as possible so we could use this system not only for university, but any other organization.

University structure

This is obvious to anyone who studied at the university in Ukraine, Russia and former USSR, but I’m not sure the structure is the same abroad. So we have the university. There are faculties (e.g. Computer Science, Philology etc). These faculties have specialties. For example, Computer Science faculty has 2 specialties – “Intellectual Systems of Decision-Making” (mine) and “Computer Engineering”. (For those interested about the difference, first one deals more with AI technologies while the latter – with system programming, networks etc).

Next level is “course”. We call it so, but actually that is the year of study. I know, there are special words for the 1st year students, for the 2nd etc. Here we just say “I’m on the first course”. That means “I’m studying the first year”.

All students that study the same specialty and are on the same course are called “Stream”. This is literal translation and I’m sure there is some more correct term. I’d be thankful if someone posts it in comments.

There are 2-4 groups in the “stream”. In our university they are names as follows: 101, 102, 103 etc. First number is course. In this case it is the first one. Second number is the faculty code. 0 is for computer science. And the third number is the group number. Thus 102 is the second group of first-year students studying Computer Science. I realize that not all universities name groups like this, so we don’t want to stick to such naming in our product.

Now let’s go through the tutors and staff. There is a rector of a university. There are deans for every faculty. And all tutors are grouped by departments. Usually, departments relate to the faculty, but that’s not a rule. For example, on our faculty we have Department of Higher Mathematics, Department of Intellectual systems ans some others. One tutor usually belongs to one department.

In addition, there are different clubs for students like our “Geeks’ Club”, translators’ “Pickwick Club” etc. Some of these clubs (like ours) have departments (Web-development, OOP, Java, C++, Delphi, Administration etc).

We want to be able to filter users using all the above groups plus some more user-specific filters like age or gender.
Let’s go through the planned features now.

User perspective

First of all, user can log in. User should enter his phone number and password. We’ll possibly add additional security for administrators. If administrator logs in, he enters his phone and password, but then a dynamic password is sent to his phone. And he completes login procedure only after he enters this dynamic password. We’re not sure if we’ll implement this. Your comments are welcome!

Next, user can edit his profile. He’ll have the following fields there:

• Phone
• Email
• Password (and confirmation)
• Name, surname, patronymic
• Gender
• Birthday
• ICQ
• Faculty
• Course (1st year, 2nd year etc)
• Group
• Clubs user is member of
• Any tags user assigns to himself (e.g. “musician”)

Plus user should check the option “receive SMS”.

However, user can’t edit phone directly because we’ll have to confirm it. So instead of giving him the field in the profile editing form, we’d rather give him a button or a link which will bring him a from where he puts new phone number and password. After that a dynamic password is sent to his phone via SMS and he enters this dynamic password and phone is updated.

Finally, user may register. When registering, he enters nearly the same data as when editing profile, but there is a dynamic password sent to him in order to confirm the phone number.
All these screens are shown below.

User area screens

Group Administrator perspective

Group administrator role may be assigned to the club’s leader, to the group monitor (or it’s better to say headman? What sound more “English”), to the tutor or headmaster. Basically everyone who needs to send notifications will have this role.

So, this type of users can do the same as the simple user plus searching users, sending email notifications and viewing statistics.
Each group administrator will be limited to the groups he may view/notify. For example, headmaster can notify anyone. Tutor may notify all groups of students he tutors in. Group monitor may notify all students within his/her group.

Listing users and searching through them is done in the same screen. In the main part of the screen there is a list of users that are in the groups available to the group administrator. In the left side there is a “narrow search” panel which lists all filters that can be applied to the current set of users. Clicking the filter option (e.g. “Geeks’ Club”) lists all users that fall into the current filter conditions. And it also refreshes the filtering options. For example, if there is no girls in the “Geeks’ Club” then there is no point to show these option in the filter. This is harder to code, but it will bring excellent user experience.

In the right hand side there will be the “saved searches” panel. When you applied some searching criteria, you may save it with some label so you can quickly access this search in the future. Were going to create standard searches in addition to filtering to simplify the system’s usage.

This works as following. For example, you can view all Computer Faculty. But you need only girls from the Geeks’ Club. So you filter by gender and then by club. And save this search as “Girls-geeks” This immediately appears in the right “saved searches” panel.

These saved searched may be group-specific and user-specific. This basically means that when group admin saves search, he may select if he wants it to be private or he wants to share this search with other group administrators, who have access to the same groups as he does. We think, this will help users creating the searches that they really need. Sure, saved searched may be easily removed.

In the right side there will be also a “Send notification” panel. It will contain only textarea field and a “send” button. When you have your group ready, you type the message an click “send”. It asks for confirmation, displaying how much it would cost and sends it over.

Another page in the Group administrator’s perspective is statistics. Here he’ll see how much messages he sent during the last day/week/month/overall and how much did it cost.

As we progress with development, we may add some settings, that will be the third page then. Here’s the filtering screen:

Filter users screen

Statistics screen is pretty straightforward, so we’ll omit it here.

Administrator’s perspective

Administrator is almost god here He can manage users, assign rights to the group administrators, set limits of SMS count, view logs and overall statistics.
Users management screen will be nearly the same as filtering, with the only difference that when clicking the row with user’s you’ll see the profile screen with additional options:

• ban/unban
• delete
• phone editing does not require confirmation
• administrator of (group selection)
• notifications limit

Logs and statistics screens are quite simple too.

Use cases

After analyzing the above overview, we come up with the following use case diagram. It is pretty self-explanatory and written after the description above, so it is useful only as a summary.

SMS Notification Use Cases

Following the iterative development methodology, we should describe all usecases now, but I find this quite useless. After describing all usecases briefly, we should select 10% critical usecases and describe them in full. Full description includes the sequence of actions taken in the main success scenario and all alternative scenarios that may happen (e.g. DB error, connection failure, no users found etc). This is useful sometimes, but I think that for such simple application as this one, we ma skip this step either.

What is really useful about the iterative development at this stage is that it recommends picking 10% of the most critical usecases and start implementing them in full. By “critical” I usually mean those which may result in essential DB structure or application design changes. In our case there are 14 usecases.10% is 1.4 usecase. Let’s pick only one. The most critical usecase, that affects all the application design and DB structure is the user filtering. The success of an application depend on the stability and usability of this feature. So we should code it efficiently and then move to the other usecases.

I like this approach cause it is much more practical than designing the system from the interface. Changing interfaces and adopting them to core is much more simple and has less risk than doing vice versa.

So we decided to start with user filtering. The first thing we should create is the database structure. This will be covered in the tomorrow’s post.

I’d like to learn your opinion on all that. Maybe you can propose some better interface, or some nice filtering approach or just some idea for this. We’re open to all suggestions and comments! Thank you for reading my blog and following this project

Subscribe to RSS and stay tuned!

Liked the post? Bookmark it

Hide Sites

Related posts:

1. Database structure and filtering approach. SMS Notification system. The database structure In the previous post I outlined the...
2. Low Coupling and High Cohesion – GRASP (Design patterns series) Low coupling When designing some architecture, you face with the...

Related posts brought to you by Yet Another Related Posts Plugin.

1. Web developer toolbar – great thing for the web development. Personally I use resizing features, password revealing and elements highlighting.
2. Firebug – absolute winner. This is actually the main tool for development I use. I’m going to write a more detailed post about it later. For now, you can enjoy this video
3. Cache status shows cached size in the status bar. But the main use is it’s ability to clear cache in a few clicks. This saves lots of time clicking here and there when developing complex JS application.
4. Live HTTP Headers allows to view all HTTP headers that go here and there while you’re requesting the page. It’s a great thing when you need to grab some content from the password-protected area or inspect what’s going on when you request page.
5. YSlow is a firebug plug-in from the Yahoo team which analyzes your content and shows you the ways of the client-size optimization of your page. It’s recommendations are really valuable. Try it and you’ll never uninstall it
6. SitePoint HTML reference is a firebug plug-in too, that adds HTML help to the every HTML tag and CSS property. It also provides examples and best practices. I don’t think this is great value for gurus, but definitely useful for all others
7. Fasterfox is a plug-in that boost Firefox performance. It is in the “experimental” state, but quite stable. I haven’t ever experience any problems with it.
8. Selenium IDE is designed as a helper tool for creating selenium tests. It is really great and handy tool if you use automated testing. I’m going to cover this in my blog later.

And what are your favourite plug-ins? How do you use them?

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Most of the beginners and, sometimes, even more advanced programmers, want to develop everything from scratch because they want have control over the every aspect of their application. I entirely understand this tendency. When you learned how to program and you can code nearly anything you want (this relates more closely to scripting languages like php, perl, js etc), you start to do something global. Something that will change the world. This may be a new CMS, a new framework, new blog of e-commerce engine. And when you start this, you want to code it in the most efficient way, optimizing every little thing: every function, every SQL query. At the same time, you want to reuse the code you’re writing. So you tend to create more general solutions than you really need. And this slows down the performance. And again you go into optimization, limiting the possible uses to some extent.
Finally, if you finish your application, it may be faster or better than similar ones (however, usually it is the same or worse), but remember how much time you spent developing it. Yes, you’re proud of yourself – you’ve created your own DB abstraction layer, that handles SQL injections in a very smart way. You’ve also developed your own ORM engine, that goes nicely with your DB abstraction layer. It is quite likely, that you’ve already created your very own template engine and a nice JS library that makes animation very easy. Great job, my fellow programmer!

…but is it really brilliant?

Now let’s stop for a moment and look back. Why you’ve started developing all that? Say you wanted to create a brand new blogging engine, that is better than Wordpress. You’ve spent several months (I can’t believe you’ve created nice DB abstraction, ORM and template engine in a week) to develop all that stack. And all you got is just another blog engine! Yes, it is cool and is capable of doing hundreds of things. You’ve even thought of the external API, so you can send SMS to be automatically published on your blog. But actually that’s just a blog!
Do you really think that the final result – blog engine was worth 2 months of the hard work and sleepless nights? If you answer “yes” and you haven’t got a feeling that something is wrong, then then you’re either mega-guru and develop only facebook- and google-scale projects or just a beginner who is still very proud of himself that he can code anything and you want to overemphasize this.
I think, every experienced developer was at that stage. I was there too. Moreover, I was there quite long and I regret of this.

So what is the framework?

By this time you may be quite bored and start thinking – what all that story has to deal with the framework. Actually the problem described above when you develop everything from scratch occurs when you don’t use any frameworks.
So what is the framework? Basically it is a set of libraries/classes, tied together that speed up the development of a certain type of applications. For example, MFC (Microsoft Foundation Classes), Qt, wxWidgets are frameworks that simplify the user interface development. It obvious that you should not draw 6 states of the button and change it on different events when it comes to placing the button on some form. You just use the ready-made class. But when it comes to the web-development, some developers think that they don’t need the ready-made (and well-tested!) DB abstraction layer, proven authorization classes. They want to develop it themselves. As a result, we often see low-quality and buggy applications. On the other hand, using frameworks does not necessarily gives you the excellent code, but it definitely helps you to get where you need much faster.

Use frameworks!

When you utilize good framework, you automatically get proven solutions in different spheres – DB connectivity, unit testing (most of frameworks provide tools for this or make it more simple), authentication, users and templates management. In addition, you also get a number of widgets, ready-made components for the problems you even don’t know about at the moment you start coding, but which usually come up during the development – caching, JS and CSS minimizing and publishing etc. Finally, you also get a bunch of extensions for a variety of tasks: RSS creation, CAPTCHA solutions, autocomplete widgets, treeview, drag&drop components to name a few.
The problem with framework usage for scripting languages is that they are quite simple and lots of developers don’t want to learn complex framework in order to achieve some simple task. And they are right… as long as their task is really simple. I think, “simple” is something that soesn’t require heavy DB usage, caching, user management, web-services integration and has very basic MVC implementation. That’s a simple homepage. All other projects should be developed with frameworks because that saves time and allows you to achieve better result.

Choosing the framework

But there is a little problem here What framework is “good”? I wrote about this in one of my previous posts where I explained why I chose Yii as a PHP framework for my project. Selecting wrong framework may result in project failure, so please refer to that post to see how I did that, maybe you’ll find that approach reasonable and join us in Yii community
And the last question I want to discuss here. The quality of code. I believe that using a framework with good design helps you to create better-structured applications that can be easily managed and supported. And the quality of code… I think this is up to programmer. It is definitely easier to create good application with properly designed framework just because you follow it’s patterns. However if you ignore the framework’s ideas and continue to create classes with static methods instead of simple functions, the framework is not very useful for you

When you should not use the frameworks

Frameworks are good, but you should clearly understand what are they doing and use them only when appropriate. As a rule, framework provides a set of generic tools that simplify creation of any project. Key work is any. These techniques give good result for a standard use. However when it comes to some specific things, especially if they are very demanding in terms of performance, it’s time to evaluate your tools from the point of view of this specific project.
Consider you have a framework designed for a web 2.0 applications. It works well for the most of the projects you create. However you’ve got to build an SMS portal. Unlike your usual applications, this doesn’t deal with HTTP a lot, it uses it’s own protocol. So 80-90% of your favourite framework’s tools will be useless here.
If you’re building very large application that will be working under the high load, you should evaluate your framework and either reconfigure it or refuse from using it at all. Most of the great websites like facebook, for example, don’t use standard frameworks, but in order to become “facebook”, you should gain popularity and using the framework for your first version ill bring you to the marketing and launch much faster and you’ll start earning money.
Similarly, if you’re building the business-card-like page, using the framework is a bit overkill

Summary

1. Use frameworks because they simplify your development, help getting better code and simplify documenting your solution (framework documentation is already written!)
2. Choose framework very carefully!
3. When solving non-trivial tasks and develop extremely large applications, re-evaluate your framework’s features and decide if it really worth using it there.

Disclaimer

1. All above is my own opinion, formed during the 5+ years of everyday web-development. Maybe this is not the absolute truth, but I believe it is somewhere near it
2. Example with SMS portal was for illustration only, I’ve never developed ones and don’t know which protocols are used there. I’d be thankful if someone of my readers clarifies this.

And what do you think about it? I’d be thankful if you share your thoughts in comments. Thanks!

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

When designing some architecture, you face with the problem “which object should perform X task?” We discussed this question in my previous post in this series. There we noted, that object should perform the tasks he has enough info for. He should be an “expert”. But because one of the main OOP characteristics is interaction between objects, it’s often hard to answer this question.

Introducing example

Consider you should print the table parameters (see What is OOP post for this example). There are tables of 2 types: square and circle ones. Square has parameter “side”, circe has parameter – “radius”. And we need the table:

Tables list

Identifying the problem

However, this method is strictly linked to using only CircleTable and SquareTable, it will not work without CircleTable and adding other table types is impossible. This can be imagined like this:

Classes coupling

Coming up with a solution

The correct variant of solving the task of table output is the following.

1. Refactor Table classes putting the output logic, that is specific to the table into the class
2. Use these new methods of table classes in the Lister::out method

Enough talk, let’s walk!

And we’ll use these methods in the Lister class:

Running the code with the same data will give us exactly the same thing. But only one difference – this code is easy to maintain, change and expand. We can imagine this structure as following:

Low coupling illustration

And now add this table to the test script:

And we’ll get the following table with no modifications in the Lister class:

Modified list of tables

High cohesion

When you do your best to implement the Lout Coupling principle, you write your code in a small portions, and each of them does some definite task. And since each class is small and does only one or several cohesive jobs, you get High Cohesion in your code.
These two principles go together. When you fulfil one of them, you automatically follow another one.

Summary

Putting it all together, we may say that Low Coupling principle requires writing isolated pieces of code, that form complex logic only by interacting with each other, not by themselves. And While following this principle, our little parts automatically become focused on one task, so they become cohesive. This way we fulfil both principles at once.
And good OO-design follow all GRASP principles at the same time just because they characterise nice application design.

What can you add here? Maybe you’ve got some questions? Feel free to ask here in comments or by email.
Grab the code of examples here.

Liked the post? Bookmark it

Hide Sites

Related posts:

1. Requirements and Use Cases. SMS Notification System. Background I’ve graduated in March 2010 and I don’t need...

Related posts brought to you by Yet Another Related Posts Plugin.

Websites, that aggregate something become more and more popular because we need all information available in one place and accessed fast and easily. From time to time, you come across with tasks, that require retrieving some data from the password-protected area. For example, I use Guru.com for my job search. They post lots of the projects there, but they don’t offer convenient listing and filtering. So I developed my own tool, that grabs everything from there and ranks it all in the way I need. So, we’ll take a look at different types of password-protected areas and see how to deal with any of them

Cookie-based authorization

We’ll start from the most popular type of authorization First, let’s take a look, how data goes between the user and the server:

Cookie-based authentication

So all we have to do is:

1. Send login request
2. Catch cookie
3. Send all data requests we need

The second part is done automatically, so we can just focus on authentication and data retrieval.

Implementation

We’ll basically need 3 functions:

• requestContent – this will make actual HTTP request to the URL we provide with parameters we provide (POST, GET variables, HTTP_REFERRER etc)
• authenticate – this will send authentication info to the server using the requestContent function
• getPage – this will actually get data we need using the same requestContent function. Actually, you may use requestContent instead, but you’ll definitely want some post-processing the result, for example, fetch some info. So it’s better to define a separate function, that does it.

So let’s go! I’ve created 3 files: config.php where we’ll store config data, functions.php where we’ll define out functions and index.php, that actually carries out the job.
config.php follows:

In functions.php I define our 3 functions. requestContent looks as following:

Basically, here we init cURL, and get the handler. It is similar to the fopen(). Then we set all options we need and finally execute the HTTP query and return result.
Authentication is based on the previous function, that carries our all the low-level job. We only have to examine our login form and determine, what variables to send. So, open the login page of the service you want to access, find the <form> tag and copy the login URL. You’ll have to put it into the config.php file. Then see what fields are there. Note, that there my be hidden fields also, note their variables and create an associative array for them (see the following code). If you’re not sure what variables are actually sent, you may use Live Http Headers plugin for the Firefox.
Usage is simple. Open the login page. Fill it in. Then start the plugin (Tools -> Live HTTP headers). Submit the form and see the number of queries are shown. You actually need only the first one – this is your request. See the screenshot:

Http headers

After getting result, you may parse it and see if you got a successful login. After you performed this query and got response, you’ve also got the session ID stored in cookie. You may open the cookie file and take a look yourself. After this you may perform any requests to the servic and it will respond as if you are logged user. The trick is that cUrl will send him cookies from the cookie file. And we have the session ID there or anything else that site uses for user identification.
So my getPage() function is a simple wrapper over the requestContent():

Basically, that’s all you need in order to get into the protected area. Now you’re free to do anything you want. Using regular expressions for parsing is the most common way to get the data from the page, so maybe you’ll need the tool for testing your regexps. I really recommend “The Regext Coach”, it greatly simplifies debugging

HTTP authentication

Another approach to the authentication, that is sometimes used in different control panels is HTTP authentication. You should have seen that – you’re presented with a standard screen, prompting for username and password. This username and password are sent with each request. You are not prompted fro them every time just because browser remembers it. To access such area, you don’t need any authenticate() function. You just pass login and pass with each request.
Here are modifications you should do in the above code:

That’s all I wanted to share with you today. Hope you fins it helpful! If you have any questions/comments, feel free to add them!

Quick Note on OOP

I didn’t create the Curl class or present solution as a class because I don’t think it makes my explanations more clear or the code easier to use. This is not full solution, this is just a technique and I tried to present the code in such a way, that you could use them with minimal changes. You can easily put that functions to the class

Further reading

• cUrl on Wikipedia
• cUrl home page
• PHP documentation on cUrl

Download code for the post

Liked the post? Bookmark it

Hide Sites

Related posts:

1. Concurrent process management in Yii Introduction In my recent project there are quite many tasks...
2. 8 Firefox plugins I use every day Firefox is great for web development mainly because of the...

Related posts brought to you by Yet Another Related Posts Plugin.

Personal productivity depends on the tools greatly, so I pay much attention to the programs I use. My previous post in this field was about MySQL workbench which I consider the best free DB design tool for MySQL.

When I started coding PHP, I used Linux so I used Kate, which only had syntax highlighting and allowed to save multiple files opened as a project Next was Quanta, the Linux IDE for PHP. I spent lots of time developing there until I installed trial version of Zend Development Environment (ZDE) 5.5. That was really cool. I liked it very much and used for quite long time. Around half a year ago I heard about NetBeans and decided to try it out. I was thinking quite long about it, I didn’t have enough time to install and go through all it’s functions. Finally I saw that I will never have this time, so I started using it around 1.5 months ago. While using, I was putting down some notes about the things I liked and the ones I didn’t. So now I just want to present my list

Advantages

What I really liked about NetBeans is that it is “the only IDE you need”. It has excellent Java support, good HTML, JS and PHP support and it even has C++ support, but I didn’t try it out, I use the old good Visual Studio 6
So here are the things I liked:

• HTML support is excellent. It founds tag mismatch, highlights start and end tags (this is extremely helpful when you deal with large portions of bad-formatted code
• When you’re in HTML scope, and typing “<p style=”, it adds double quotes automatically! Little, but very nice feature
• Code formatting is great in everywhere. When working in Zend Studio, I was using 3rd party tools to format HTML, NetBeans does it perfectly. As for other languages, the only problem I have with it is that NetBeans puts opening curly bracket on the same line as loop or function definition, I like putting it on the next line so I see opening and closing brackets one under another. However, it highlights the opposite bracket, so it’s not a big problem
• Greatest thing I got there is normal debugging in PHP, not just “echo”, “print_r()” and “exit”. NetBeans integrates with XDebug and it is VERY convenient. It provides normal info as any other debugger – stack trace, local variables, superglobals, watches etc. It saves great amount of time. The only drawback is that NetBeans starts debugging session slowly, but it is definitely faster, than “echo” and “exit”
• When switching from Zend Studio, I experienced some inconveniences because NetBeans uses other shortcuts. However, it provides more useful shortcuts than ZDE, and I got used to it quickly
• When you code in languages like JS or PHP, where using variables without declaration is not compilation error, it is a common mistake to misspell variable and then get some interesting bugs. That’s where variable highlighting helps greatly. In NetBeans you can put cursor into any variable and in a second you’ll see all occurrences of this variable in the file. That is also very helpful when you are digging through someone’s code
• Search is also much better, than in Zend. When you search in files, search results are grouped by file. When you search in the file, you get all occurrences highlighted. This is also very useful when you’re looking into someone’s code or performing some refactoring
• Commenting support is also great. For example, you write some //comment. If the line is long, you want to break it into several lines and press enter when you’re inside the line. NetBeans inserts line break and next line continues as comment! This is so helpful when you’re commenting some file and your comments are quite long.
• PHPDoc support is also great. The fact, that you get help for function, its arguments and return value if you specify function description in the PHPDoc format wasn’t new to me, but if you start writing PHPDoc comment for the ready function and put “/**“, you immediately get full PHPDoc template – all arguments list with placeholders, @return keyword, so everything remained is only put actual description, everything is automated. That’s great! The only thing I’d like to add here is that when function definition is changed, it should update PHPDoc comment – remove correspondent line or add more variables I guess, that’s a dream
• Very useful feature is auto-reloading files. You don’t need to reopen log every time it is updated. However, there is a slight disadvantage – if file is deleted in the file system, it doesn’t propose to save a copy like Zend does.
• It has built-in diff tool. It is MEGA-useful I have to compare files quite frequently, I was using WinMerge before that, but it was inconvenient. NetBeans integrates everything in one environment and it is very nice!
• Auto-complete is better, than in ZDE, it takes scope into consideration. However, it is sometimes buggy.
• In NetBeans you can view your DB in the same IDE, perform queries, view tables structure and much more. This tool is not as good as phpMyAdmin, so I am using the latter most of the time. Main reason is that NetBeans is slow when switching from code mode to the DB mode. But maybe that’s my PC

Disadvantages

And now some things I didn’t like:

• Code templates are worse, than in Zend Studio. This thing is really simple there and behaved as supposed. In NetBeans I add some template, e.g. “echo ‘<pre>’.print_r($var,true).’</pre>’;” for the “pri” keyword. And sometimes it doesn’t appear, sometimes it is inserted with additional
• There is no PHP or JS help integrated into IDE. In Zend Studio I could select php function and click “F1″ and get full help. In Aptana IDE there is the same for JS, but there is noting like this here! Why? Is it something complex?
• No FTP support without additional plug-ins. I didn’t have time to explore this feature, but I don’t like the fact I have to install something more for such common feature
• There is UML package for Java, why there is nothing like this for JS?

Conclusion

Anyway, despite of the disadvantages, I think, that NetBeans is better, because it offers even more features, than Zend, but for free! And it’s really good with Java I especially liked refactoring there. Just put the name of the class and everything is changed correspondently!

What are your thoughts about NetBeans? Are you using/planning to use it? What are your workarounds for the disadvantages I noted? Can you add more “pros”?

Liked the post? Bookmark it

Hide Sites

Related posts:

1. Concurrent process management in Yii Introduction In my recent project there are quite many tasks...

Related posts brought to you by Yet Another Related Posts Plugin.

In November 2008 I’ve got a nice project, that resulted in a complex search engine, that specializes in clothing. I can’t give the link now, but will do when site officially launches. Key features of such service are:

• Large number of users, which search with various parameters, rate items, leave comments, create articles etc.
• Large database, that is updated frequently

Sure, the site is running on a dedicated server, sure I use caching, but optimal DB is a key to success.

MyISAM or InnoDB?

Since we need reliability, I created different tests, search the internet etc. My main question was – what to use MyISAM or InnoDB? I found the post on the MySQL Perfomance Blog (also this and this) and came to the conclusion, that when SELECT’ing, InnoDB is definitely faster (or, at least, is not slower). In addition, it provides constraints mechanism and allows transactions. The only thing it doesn’t have is FULLTEXT search. One more disadvantage is slower inserts. However, 80% of DB usage in this project are SELECT queries without FULLTEXT. For the fulltext search I created MyISAM table with fields I need and it works fine. The project is finished, but there will be phase 2 and I’m thinking of introducing Sphinx there. Anybody has experience with it?
So I came up with InnoDB storage engine.

Optimizing structure

Next important step is DB structure. I’ll give some background info about the project so you could better understand what I’m talking about. The site collects info from different UK shops. Then users can rate them, add some tags, create articles (called “looks”) and link that articles to the products they’re writing about. And the same (tags, comments, rating) applies to the looks. Here we come across with an interesting question. We can create structure of different types for this:

• Create 2 separate cross-tables for linking Rating to Look or Product
• Create one cross-table and add additional field “type” there to differentiate between Product and Look entities

See both variants in the following images:

Normalized variant

Denormalized version

• If there are 0-2 ratings linked to entity, then normalized variant is better
• If there are 3-5 ratings linked to entity, then denormalized variant is slightly (less than 1%) better
• If there are 6-8 ratings linked to the entity, the normalized variant wins again

I did these tests for different situations similar to this one, in each test I generated random records and randomly linked them with each other. Number of record in Product, Look tables was 10000, in Rating – 20000. Number of records in cross tables varies because the linkings are random.
So I came up with normalizing everything like it should be done according to the DB theory.

The only exclusion of this rule is the following situation. In this project you can add different alternatives to the product. Each alternative is “better” or “cheaper” or “different” etc. And since the only case when these alternatives are pulled from DB is when user views the product, they are pulled altogether, so it is better to keep them in the same table.

Conclusion

• InnoDB turns out to behave better under the high load, it supports transactions and maintains constraints. All that helps to develop better and never leave orphan records.
• Using normalization techniques you usually get better performance
• Create tests and test everything if you are going to do something great

Further reading

• Database Normalization
• MySQL Performance Blog

P.S. I’m looking forward to see your comments on implementation techniques in the high-loaded websites, your thoughts about the InnoDB vs MyISAM question and anything else you can think of

Liked the post? Bookmark it

Hide Sites

Related posts:

1. Low Coupling and High Cohesion – GRASP (Design patterns series) Low coupling When designing some architecture, you face with the...

Related posts brought to you by Yet Another Related Posts Plugin.

• Albania – The girl was nice and the song also seemed good to me
• Island – Nice girl, nice song, but my girl disagrees it was so good to get 2nd place
• Turkey – Nice dance, nice girls, good style
• Greece – VERY nice dance and song
• Armenia – really, really good style and song
• Azerbaijan – I like Arash, I liked the style, the song, the dance and the overall show
• UK – Great voice, nice song, but I didn’t like her dress.

(sorry, didn’t find yesterday’s videos for UK and Armenia )
And I really liked Alexander Rybak. The Norway style, great acrobatics, really good song and an interesting mix of Slavic music with some Norway motives resulted in excellent impressions. When I just saw him on the scene, I thought – what that guy is doing there? But when he started singing, I immediately changed my mind – he was doing really good! I entirely agree, that he got his 1st place because he deserves it. However, he got 387 points and the 2nd place – around 200. I don’t think he was twice as good as Iceland

Anyway, I’m thankful to my girl who made me see this great show, I liked it all.

And what did you like there?

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.