In my recent project there are quite many tasks that run in the background – generate thumbnails, detect colours on the pictures, run DB updates etc. Sure, all that is handled using cron and Yii commands. However there is a little problem. Consider we have DB update routine that should import new stock data from the datafeed. Datafeed is uploaded hourly, but upload can’t be scheduled to minutes – connection speed, different errors may interrupt the upload. On the other hand, sometimes processing takes 5 mins and sometimes – a few hours because in the first variant we just replace records and in the latter – download images for the new products.

The problem

So if we schedule to run every 5 mins (to check if update is ready), then we get the following:

As a result, DB will have wrong data because records were inserted twice. Sure, we can control this with DB constraints, but duplicates is not the only possible error.

Solution overview

The only variant to fix this – go Unix way and create lock file when some process starts. Actually that is a good variant, but I decided to use one file for all processes. Sure, I have to lock it when I read/write from it and other processes have to wait when I’m done with this. It is not brilliant, but I decided to go this way and I’ll show how I did this in this post.

First of all, I didn’t want to put the same code for locking/checking/unlocking in every command. So I created the taskmanager extension. It is not yet finished, so I don’t publish it.

Technical details

This is an application component which is extended from CApplicationComponent class and implements TaskManger interface:

Each command/process has it’s ID. ID is a plain text without spaces. Although spaces are not restricted and it will work with them, I decided to apply the same naming conventions as for class properties in Yii.
Let’s go through the methods:

• isRunning($id). Takes process ID as a parameter and return true if this process is running.
• logStarted($id). This checks if process is not already running (or if duplicates are allowed) and adds records to the pids file. Actually this interface may be implemented using the DB, so no locking or other things will be needed.
• logFinished($id). Removes record from the pids file (or from DB if implemented using it).
• logTerminated($id). Nearly the same as previous, but it doesn’t check if process is running, it tries to remove the record anyway.
• sendSignal($id, $signal). Sends signal to the process. Process should check it’s buffer file when it is running and react to signals. It is not implemented because requires changes in the command code.
• getStatus($id). Returns current status of a given process. It is not implemented either for the same reason.
• getLog($id). Should return log of a process execution. Not implemented either.

And now let’s take a look at the class implementing it.
First of all, it’s initialization:

• $pidFile is where all currently running processes are stored
• $pidLogFile is where we write “DD.MM.YYY HH:MM:SS process XXX started” and “ DD.MM.YYY HH:MM:SS process XXX finished”
• $processDir is where log files and exchange files should reside (methods sendSignal, getStatus, getLog)

We get these variables from the config file and update to fit current application path. Also, since we extend the class from CApplicationComponent, we should perform standard initialization, an we do this by calling parent::init() in the last line.

Since we’re using file, it should be locked from other processes when we’re working with it. So we need a method that will open it and wait if it is busy now:

Idea is pretty simple – we try to lock the file. If we succeed, we return the file pointer and we throw exception otherwise. We try it several times until we reach timeout. Intervals are random to ensure that different processes started simultaneously will not try to lock it at the same time.
And now we can implement our methods:

So now we just need to call these methods from our command. But you know, I’m very lazy, so I created a base class for all my commands:

It’s constructor created the process ID from the class name. It is quite convenient feature. It also handles start and stop so all we need to do is just to implement runCmd() method in the derived classes like this:

Then you just type:

Can’t start process! It is already running!

First command starts routine in background. Second command starts the same command but in the terminal. So you’ll see the output for it. It says that it can’t be started because previous run is not finished.

There is one thing you should do in order to make that work. You should add a component to the application configuration. Open the protected/config/console.php file and add into the components list:

And into the import section:

Other problems

I’ve been using this solution for 5 months. Practice shows that sometimes process may run out of memory or some other error occurs that is not caught by out exception handling and it is terminated without our $tm->logFinished() call. This leads to the situation when process is not actually running, but task manager thinks it is because it’s record is still present in the pids file. In order to prevent such situations, I modified isRunning routine like this:

This performs additional check by trying to find the selected PID in the system process list by listing all processes ps -A and checking if our pid is there (grep [pid]). If result is empty, then process is not running.

There is a potential pitfall in the isRunning method. If you try to do something with pids file before it is closed in line 21, you’ll get into the deadlock because file is locked for modifications. That’s why I call logTerminated after it is closed. Be attentive!

I’d be thankful if someone takes the code and adds signal processing there and maybe implements DBTaskManager in addition to FileTaskManager. Let me know if someone is interested, we can collaborate on this and deliver a nice ready-made extension.

As usual, any comments are welcome!

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Nearly every large application has some configuration parameters – site admin email, cache time for different blocks, number of latest news, number of items per search or catalogue page etc.

Standard Way

Yii comes with the built-in mechanism for this. You can create a file with an associative array of all your variables, like this:

Then put it somewhere, for example in /protected/config/params.php and configure your application to use it:

Usage is simple, just refer to it as to array:

This is quite convenient when you just want to save some constants. End-user will not be able to change this from admin in any way. Sure, you can create the interface for editing this config and update the file, but much more flexible way is to store the configuration in DB.

The Database Way

To make the life easier, I’ve created an extension, that can be used as an application component. Extension can be downloaded from the Yii extension page. Here I’ll give an overview of it’s work.

First thing to note, it implements IApplicationComponent, so it should implement init() method where initialization is performed. In this method we check if there are any parameters to load. If so – we try to load them. If there is an exception, it means, that no table is present, so we create it and initialize the parameters that should be preloaded with null values

If parameter is requested, in the __get() method we reuse the parent’s (CAttributesCollection) contains() method, which checks if we have this parameter in the collection already. If so – we simply return it. If not – we call the loadParam() method, it loads if from DB and adds to the component collection. If there is no attribute, it throws an exception. We catch it in the __get() and call the parent::__get(). Here is the code:

Setter method simply updates the collection. We check if the parameter is already present in DB. If so – we update it. If not – we create it. And, if everything was OK, we add the parameter to the internal collection ($this->add($name,$value)):

When configuring the extension or using load() or purge() method, we can specify parameters either as array or as comma-separated string. This is achieved by the preprocessing method, which also converts parameter names to lowercase and they are case insensitive (if this is required by the caseSensitive property, it must be set to true). Here is it:

And, finally, load and purge methods:

Usage Instructions

Using the extension is simple. Like any extension, you should unzip it to the /protected/extensions/ folder and then set it up as an application component. To do this, you should add it with any name to the components array in the application configuration:

‘par’ is the ID of the component, you will refer to it in any place of the application by it’s ID:

That’s it! Enjoy!

UPDATE: Updated on March, 5 2009 according to the Jonah’s comments an this discussion. Version 3 is planned with great functionality

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.