In my recent project there are quite many tasks that run in the background – generate thumbnails, detect colours on the pictures, run DB updates etc. Sure, all that is handled using cron and Yii commands. However there is a little problem. Consider we have DB update routine that should import new stock data from the datafeed. Datafeed is uploaded hourly, but upload can’t be scheduled to minutes – connection speed, different errors may interrupt the upload. On the other hand, sometimes processing takes 5 mins and sometimes – a few hours because in the first variant we just replace records and in the latter – download images for the new products.

The problem

So if we schedule to run every 5 mins (to check if update is ready), then we get the following:

As a result, DB will have wrong data because records were inserted twice. Sure, we can control this with DB constraints, but duplicates is not the only possible error.

Solution overview

The only variant to fix this – go Unix way and create lock file when some process starts. Actually that is a good variant, but I decided to use one file for all processes. Sure, I have to lock it when I read/write from it and other processes have to wait when I’m done with this. It is not brilliant, but I decided to go this way and I’ll show how I did this in this post.

First of all, I didn’t want to put the same code for locking/checking/unlocking in every command. So I created the taskmanager extension. It is not yet finished, so I don’t publish it.

Technical details

This is an application component which is extended from CApplicationComponent class and implements TaskManger interface:

Each command/process has it’s ID. ID is a plain text without spaces. Although spaces are not restricted and it will work with them, I decided to apply the same naming conventions as for class properties in Yii.
Let’s go through the methods:

• isRunning($id). Takes process ID as a parameter and return true if this process is running.
• logStarted($id). This checks if process is not already running (or if duplicates are allowed) and adds records to the pids file. Actually this interface may be implemented using the DB, so no locking or other things will be needed.
• logFinished($id). Removes record from the pids file (or from DB if implemented using it).
• logTerminated($id). Nearly the same as previous, but it doesn’t check if process is running, it tries to remove the record anyway.
• sendSignal($id, $signal). Sends signal to the process. Process should check it’s buffer file when it is running and react to signals. It is not implemented because requires changes in the command code.
• getStatus($id). Returns current status of a given process. It is not implemented either for the same reason.
• getLog($id). Should return log of a process execution. Not implemented either.

And now let’s take a look at the class implementing it.
First of all, it’s initialization:

• $pidFile is where all currently running processes are stored
• $pidLogFile is where we write “DD.MM.YYY HH:MM:SS process XXX started” and “ DD.MM.YYY HH:MM:SS process XXX finished”
• $processDir is where log files and exchange files should reside (methods sendSignal, getStatus, getLog)

We get these variables from the config file and update to fit current application path. Also, since we extend the class from CApplicationComponent, we should perform standard initialization, an we do this by calling parent::init() in the last line.

Since we’re using file, it should be locked from other processes when we’re working with it. So we need a method that will open it and wait if it is busy now:

Idea is pretty simple – we try to lock the file. If we succeed, we return the file pointer and we throw exception otherwise. We try it several times until we reach timeout. Intervals are random to ensure that different processes started simultaneously will not try to lock it at the same time.
And now we can implement our methods:

So now we just need to call these methods from our command. But you know, I’m very lazy, so I created a base class for all my commands:

It’s constructor created the process ID from the class name. It is quite convenient feature. It also handles start and stop so all we need to do is just to implement runCmd() method in the derived classes like this:

Then you just type:

Can’t start process! It is already running!

First command starts routine in background. Second command starts the same command but in the terminal. So you’ll see the output for it. It says that it can’t be started because previous run is not finished.

There is one thing you should do in order to make that work. You should add a component to the application configuration. Open the protected/config/console.php file and add into the components list:

And into the import section:

Other problems

I’ve been using this solution for 5 months. Practice shows that sometimes process may run out of memory or some other error occurs that is not caught by out exception handling and it is terminated without our $tm->logFinished() call. This leads to the situation when process is not actually running, but task manager thinks it is because it’s record is still present in the pids file. In order to prevent such situations, I modified isRunning routine like this:

This performs additional check by trying to find the selected PID in the system process list by listing all processes ps -A and checking if our pid is there (grep [pid]). If result is empty, then process is not running.

There is a potential pitfall in the isRunning method. If you try to do something with pids file before it is closed in line 21, you’ll get into the deadlock because file is locked for modifications. That’s why I call logTerminated after it is closed. Be attentive!

I’d be thankful if someone takes the code and adds signal processing there and maybe implements DBTaskManager in addition to FileTaskManager. Let me know if someone is interested, we can collaborate on this and deliver a nice ready-made extension.

As usual, any comments are welcome!

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

I’ve graduated in March 2010 and I don’t need to visit university any more. However, I enjoyed studying there and I know what challenges my fellow students-programmers have. That’s why I continue the job started this autumn – computer club, we call it “Geeks’ Club”. Sure, we’re not true geeks there, but we tend to be So I’m helping students who’re interested in web-technologies and in object-oriented programming. I explain them things that are not covered in the standard university course. Things, that are more practical and they can be paid money for.

Here I’ll be covering both OOP and web-dev, but we’ll start from design hence OOP. The story began when some time passed after we started learning OOP at the club meetings and I decided that there is enough theory and separate examples, it’s time to build something practical that will be challenging for me too.
I thought of different systems, but finally decided to pick up the SMS notification one. It requires both user interface, database, API (my friends want to communicate to it from standalone Java and C++ applications) and overall application design. In addition, I’d like such system to be deployed at our university and it is interesting for me to create one.

Introduction

This is going to be a series of articles, I’ll describe the progress and challenges we come across as we proceed with development. I think this kind of posts will be interesting for those who study OOP and want to apply what they’ve learned to some practical things. I’ll try to guide my students through the process using the iterative development methodology.

We have meetings once a week (on Saturdays), so I’ll post once or twice a week, depending on amount of work done on weekends and my free time during the week.

I think, this series will contain around 10 posts. We’re going to share the work done as an open-source project. There is nothing to share at the moment, so I’ll give the link when we code something

System overview

Vision

Obviously, the main purpose of the system is to send SMS notifications to students or tutors. These notifications may be quite different – from the timetable changes to some announcements or even warnings (e.g. “Due to the weather conditions, today is a day-off for all. Stay at home and have a good time”).

Actually the system isn’t that complex. Basically all it needs to do is to enable administrator pick a group of users, enter the message and send it through the SMS gateway. The tricky things start from “pick a group of users”. We want this to be as flexible as possible so we could use this system not only for university, but any other organization.

University structure

This is obvious to anyone who studied at the university in Ukraine, Russia and former USSR, but I’m not sure the structure is the same abroad. So we have the university. There are faculties (e.g. Computer Science, Philology etc). These faculties have specialties. For example, Computer Science faculty has 2 specialties – “Intellectual Systems of Decision-Making” (mine) and “Computer Engineering”. (For those interested about the difference, first one deals more with AI technologies while the latter – with system programming, networks etc).

Next level is “course”. We call it so, but actually that is the year of study. I know, there are special words for the 1st year students, for the 2nd etc. Here we just say “I’m on the first course”. That means “I’m studying the first year”.

All students that study the same specialty and are on the same course are called “Stream”. This is literal translation and I’m sure there is some more correct term. I’d be thankful if someone posts it in comments.

There are 2-4 groups in the “stream”. In our university they are names as follows: 101, 102, 103 etc. First number is course. In this case it is the first one. Second number is the faculty code. 0 is for computer science. And the third number is the group number. Thus 102 is the second group of first-year students studying Computer Science. I realize that not all universities name groups like this, so we don’t want to stick to such naming in our product.

Now let’s go through the tutors and staff. There is a rector of a university. There are deans for every faculty. And all tutors are grouped by departments. Usually, departments relate to the faculty, but that’s not a rule. For example, on our faculty we have Department of Higher Mathematics, Department of Intellectual systems ans some others. One tutor usually belongs to one department.

In addition, there are different clubs for students like our “Geeks’ Club”, translators’ “Pickwick Club” etc. Some of these clubs (like ours) have departments (Web-development, OOP, Java, C++, Delphi, Administration etc).

We want to be able to filter users using all the above groups plus some more user-specific filters like age or gender.
Let’s go through the planned features now.

User perspective

First of all, user can log in. User should enter his phone number and password. We’ll possibly add additional security for administrators. If administrator logs in, he enters his phone and password, but then a dynamic password is sent to his phone. And he completes login procedure only after he enters this dynamic password. We’re not sure if we’ll implement this. Your comments are welcome!

Next, user can edit his profile. He’ll have the following fields there:

• Phone
• Email
• Password (and confirmation)
• Name, surname, patronymic
• Gender
• Birthday
• ICQ
• Faculty
• Course (1st year, 2nd year etc)
• Group
• Clubs user is member of
• Any tags user assigns to himself (e.g. “musician”)

Plus user should check the option “receive SMS”.

However, user can’t edit phone directly because we’ll have to confirm it. So instead of giving him the field in the profile editing form, we’d rather give him a button or a link which will bring him a from where he puts new phone number and password. After that a dynamic password is sent to his phone via SMS and he enters this dynamic password and phone is updated.

Finally, user may register. When registering, he enters nearly the same data as when editing profile, but there is a dynamic password sent to him in order to confirm the phone number.
All these screens are shown below.

User area screens

Group Administrator perspective

Group administrator role may be assigned to the club’s leader, to the group monitor (or it’s better to say headman? What sound more “English”), to the tutor or headmaster. Basically everyone who needs to send notifications will have this role.

So, this type of users can do the same as the simple user plus searching users, sending email notifications and viewing statistics.
Each group administrator will be limited to the groups he may view/notify. For example, headmaster can notify anyone. Tutor may notify all groups of students he tutors in. Group monitor may notify all students within his/her group.

Listing users and searching through them is done in the same screen. In the main part of the screen there is a list of users that are in the groups available to the group administrator. In the left side there is a “narrow search” panel which lists all filters that can be applied to the current set of users. Clicking the filter option (e.g. “Geeks’ Club”) lists all users that fall into the current filter conditions. And it also refreshes the filtering options. For example, if there is no girls in the “Geeks’ Club” then there is no point to show these option in the filter. This is harder to code, but it will bring excellent user experience.

In the right hand side there will be the “saved searches” panel. When you applied some searching criteria, you may save it with some label so you can quickly access this search in the future. Were going to create standard searches in addition to filtering to simplify the system’s usage.

This works as following. For example, you can view all Computer Faculty. But you need only girls from the Geeks’ Club. So you filter by gender and then by club. And save this search as “Girls-geeks” This immediately appears in the right “saved searches” panel.

These saved searched may be group-specific and user-specific. This basically means that when group admin saves search, he may select if he wants it to be private or he wants to share this search with other group administrators, who have access to the same groups as he does. We think, this will help users creating the searches that they really need. Sure, saved searched may be easily removed.

In the right side there will be also a “Send notification” panel. It will contain only textarea field and a “send” button. When you have your group ready, you type the message an click “send”. It asks for confirmation, displaying how much it would cost and sends it over.

Another page in the Group administrator’s perspective is statistics. Here he’ll see how much messages he sent during the last day/week/month/overall and how much did it cost.

As we progress with development, we may add some settings, that will be the third page then. Here’s the filtering screen:

Filter users screen

Statistics screen is pretty straightforward, so we’ll omit it here.

Administrator’s perspective

Administrator is almost god here He can manage users, assign rights to the group administrators, set limits of SMS count, view logs and overall statistics.
Users management screen will be nearly the same as filtering, with the only difference that when clicking the row with user’s you’ll see the profile screen with additional options:

• ban/unban
• delete
• phone editing does not require confirmation
• administrator of (group selection)
• notifications limit

Logs and statistics screens are quite simple too.

Use cases

After analyzing the above overview, we come up with the following use case diagram. It is pretty self-explanatory and written after the description above, so it is useful only as a summary.

SMS Notification Use Cases

Following the iterative development methodology, we should describe all usecases now, but I find this quite useless. After describing all usecases briefly, we should select 10% critical usecases and describe them in full. Full description includes the sequence of actions taken in the main success scenario and all alternative scenarios that may happen (e.g. DB error, connection failure, no users found etc). This is useful sometimes, but I think that for such simple application as this one, we ma skip this step either.

What is really useful about the iterative development at this stage is that it recommends picking 10% of the most critical usecases and start implementing them in full. By “critical” I usually mean those which may result in essential DB structure or application design changes. In our case there are 14 usecases.10% is 1.4 usecase. Let’s pick only one. The most critical usecase, that affects all the application design and DB structure is the user filtering. The success of an application depend on the stability and usability of this feature. So we should code it efficiently and then move to the other usecases.

I like this approach cause it is much more practical than designing the system from the interface. Changing interfaces and adopting them to core is much more simple and has less risk than doing vice versa.

So we decided to start with user filtering. The first thing we should create is the database structure. This will be covered in the tomorrow’s post.

I’d like to learn your opinion on all that. Maybe you can propose some better interface, or some nice filtering approach or just some idea for this. We’re open to all suggestions and comments! Thank you for reading my blog and following this project

Subscribe to RSS and stay tuned!

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Websites, that aggregate something become more and more popular because we need all information available in one place and accessed fast and easily. From time to time, you come across with tasks, that require retrieving some data from the password-protected area. For example, I use Guru.com for my job search. They post lots of the projects there, but they don’t offer convenient listing and filtering. So I developed my own tool, that grabs everything from there and ranks it all in the way I need. So, we’ll take a look at different types of password-protected areas and see how to deal with any of them

Cookie-based authorization

We’ll start from the most popular type of authorization First, let’s take a look, how data goes between the user and the server:

Cookie-based authentication

So all we have to do is:

1. Send login request
2. Catch cookie
3. Send all data requests we need

The second part is done automatically, so we can just focus on authentication and data retrieval.

Implementation

We’ll basically need 3 functions:

• requestContent – this will make actual HTTP request to the URL we provide with parameters we provide (POST, GET variables, HTTP_REFERRER etc)
• authenticate – this will send authentication info to the server using the requestContent function
• getPage – this will actually get data we need using the same requestContent function. Actually, you may use requestContent instead, but you’ll definitely want some post-processing the result, for example, fetch some info. So it’s better to define a separate function, that does it.

So let’s go! I’ve created 3 files: config.php where we’ll store config data, functions.php where we’ll define out functions and index.php, that actually carries out the job.
config.php follows:

In functions.php I define our 3 functions. requestContent looks as following:

Basically, here we init cURL, and get the handler. It is similar to the fopen(). Then we set all options we need and finally execute the HTTP query and return result.
Authentication is based on the previous function, that carries our all the low-level job. We only have to examine our login form and determine, what variables to send. So, open the login page of the service you want to access, find the <form> tag and copy the login URL. You’ll have to put it into the config.php file. Then see what fields are there. Note, that there my be hidden fields also, note their variables and create an associative array for them (see the following code). If you’re not sure what variables are actually sent, you may use Live Http Headers plugin for the Firefox.
Usage is simple. Open the login page. Fill it in. Then start the plugin (Tools -> Live HTTP headers). Submit the form and see the number of queries are shown. You actually need only the first one – this is your request. See the screenshot:

Http headers

After getting result, you may parse it and see if you got a successful login. After you performed this query and got response, you’ve also got the session ID stored in cookie. You may open the cookie file and take a look yourself. After this you may perform any requests to the servic and it will respond as if you are logged user. The trick is that cUrl will send him cookies from the cookie file. And we have the session ID there or anything else that site uses for user identification.
So my getPage() function is a simple wrapper over the requestContent():

Basically, that’s all you need in order to get into the protected area. Now you’re free to do anything you want. Using regular expressions for parsing is the most common way to get the data from the page, so maybe you’ll need the tool for testing your regexps. I really recommend “The Regext Coach”, it greatly simplifies debugging

HTTP authentication

Another approach to the authentication, that is sometimes used in different control panels is HTTP authentication. You should have seen that – you’re presented with a standard screen, prompting for username and password. This username and password are sent with each request. You are not prompted fro them every time just because browser remembers it. To access such area, you don’t need any authenticate() function. You just pass login and pass with each request.
Here are modifications you should do in the above code:

That’s all I wanted to share with you today. Hope you fins it helpful! If you have any questions/comments, feel free to add them!

Quick Note on OOP

I didn’t create the Curl class or present solution as a class because I don’t think it makes my explanations more clear or the code easier to use. This is not full solution, this is just a technique and I tried to present the code in such a way, that you could use them with minimal changes. You can easily put that functions to the class

Further reading

• cUrl on Wikipedia
• cUrl home page
• PHP documentation on cUrl

Download code for the post

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Personal productivity depends on the tools greatly, so I pay much attention to the programs I use. My previous post in this field was about MySQL workbench which I consider the best free DB design tool for MySQL.

When I started coding PHP, I used Linux so I used Kate, which only had syntax highlighting and allowed to save multiple files opened as a project Next was Quanta, the Linux IDE for PHP. I spent lots of time developing there until I installed trial version of Zend Development Environment (ZDE) 5.5. That was really cool. I liked it very much and used for quite long time. Around half a year ago I heard about NetBeans and decided to try it out. I was thinking quite long about it, I didn’t have enough time to install and go through all it’s functions. Finally I saw that I will never have this time, so I started using it around 1.5 months ago. While using, I was putting down some notes about the things I liked and the ones I didn’t. So now I just want to present my list

Advantages

What I really liked about NetBeans is that it is “the only IDE you need”. It has excellent Java support, good HTML, JS and PHP support and it even has C++ support, but I didn’t try it out, I use the old good Visual Studio 6
So here are the things I liked:

• HTML support is excellent. It founds tag mismatch, highlights start and end tags (this is extremely helpful when you deal with large portions of bad-formatted code
• When you’re in HTML scope, and typing “<p style=”, it adds double quotes automatically! Little, but very nice feature
• Code formatting is great in everywhere. When working in Zend Studio, I was using 3rd party tools to format HTML, NetBeans does it perfectly. As for other languages, the only problem I have with it is that NetBeans puts opening curly bracket on the same line as loop or function definition, I like putting it on the next line so I see opening and closing brackets one under another. However, it highlights the opposite bracket, so it’s not a big problem
• Greatest thing I got there is normal debugging in PHP, not just “echo”, “print_r()” and “exit”. NetBeans integrates with XDebug and it is VERY convenient. It provides normal info as any other debugger – stack trace, local variables, superglobals, watches etc. It saves great amount of time. The only drawback is that NetBeans starts debugging session slowly, but it is definitely faster, than “echo” and “exit”
• When switching from Zend Studio, I experienced some inconveniences because NetBeans uses other shortcuts. However, it provides more useful shortcuts than ZDE, and I got used to it quickly
• When you code in languages like JS or PHP, where using variables without declaration is not compilation error, it is a common mistake to misspell variable and then get some interesting bugs. That’s where variable highlighting helps greatly. In NetBeans you can put cursor into any variable and in a second you’ll see all occurrences of this variable in the file. That is also very helpful when you are digging through someone’s code
• Search is also much better, than in Zend. When you search in files, search results are grouped by file. When you search in the file, you get all occurrences highlighted. This is also very useful when you’re looking into someone’s code or performing some refactoring
• Commenting support is also great. For example, you write some //comment. If the line is long, you want to break it into several lines and press enter when you’re inside the line. NetBeans inserts line break and next line continues as comment! This is so helpful when you’re commenting some file and your comments are quite long.
• PHPDoc support is also great. The fact, that you get help for function, its arguments and return value if you specify function description in the PHPDoc format wasn’t new to me, but if you start writing PHPDoc comment for the ready function and put “/**“, you immediately get full PHPDoc template – all arguments list with placeholders, @return keyword, so everything remained is only put actual description, everything is automated. That’s great! The only thing I’d like to add here is that when function definition is changed, it should update PHPDoc comment – remove correspondent line or add more variables I guess, that’s a dream
• Very useful feature is auto-reloading files. You don’t need to reopen log every time it is updated. However, there is a slight disadvantage – if file is deleted in the file system, it doesn’t propose to save a copy like Zend does.
• It has built-in diff tool. It is MEGA-useful I have to compare files quite frequently, I was using WinMerge before that, but it was inconvenient. NetBeans integrates everything in one environment and it is very nice!
• Auto-complete is better, than in ZDE, it takes scope into consideration. However, it is sometimes buggy.
• In NetBeans you can view your DB in the same IDE, perform queries, view tables structure and much more. This tool is not as good as phpMyAdmin, so I am using the latter most of the time. Main reason is that NetBeans is slow when switching from code mode to the DB mode. But maybe that’s my PC

Disadvantages

And now some things I didn’t like:

• Code templates are worse, than in Zend Studio. This thing is really simple there and behaved as supposed. In NetBeans I add some template, e.g. “echo ‘<pre>’.print_r($var,true).’</pre>’;” for the “pri” keyword. And sometimes it doesn’t appear, sometimes it is inserted with additional
• There is no PHP or JS help integrated into IDE. In Zend Studio I could select php function and click “F1″ and get full help. In Aptana IDE there is the same for JS, but there is noting like this here! Why? Is it something complex?
• No FTP support without additional plug-ins. I didn’t have time to explore this feature, but I don’t like the fact I have to install something more for such common feature
• There is UML package for Java, why there is nothing like this for JS?

Conclusion

Anyway, despite of the disadvantages, I think, that NetBeans is better, because it offers even more features, than Zend, but for free! And it’s really good with Java I especially liked refactoring there. Just put the name of the class and everything is changed correspondently!

What are your thoughts about NetBeans? Are you using/planning to use it? What are your workarounds for the disadvantages I noted? Can you add more “pros”?

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.