In my recent project there are quite many tasks that run in the background – generate thumbnails, detect colours on the pictures, run DB updates etc. Sure, all that is handled using cron and Yii commands. However there is a little problem. Consider we have DB update routine that should import new stock data from the datafeed. Datafeed is uploaded hourly, but upload can’t be scheduled to minutes – connection speed, different errors may interrupt the upload. On the other hand, sometimes processing takes 5 mins and sometimes – a few hours because in the first variant we just replace records and in the latter – download images for the new products.

The problem

So if we schedule to run every 5 mins (to check if update is ready), then we get the following:

As a result, DB will have wrong data because records were inserted twice. Sure, we can control this with DB constraints, but duplicates is not the only possible error.

Solution overview

The only variant to fix this – go Unix way and create lock file when some process starts. Actually that is a good variant, but I decided to use one file for all processes. Sure, I have to lock it when I read/write from it and other processes have to wait when I’m done with this. It is not brilliant, but I decided to go this way and I’ll show how I did this in this post.

First of all, I didn’t want to put the same code for locking/checking/unlocking in every command. So I created the taskmanager extension. It is not yet finished, so I don’t publish it.

Technical details

This is an application component which is extended from CApplicationComponent class and implements TaskManger interface:

Each command/process has it’s ID. ID is a plain text without spaces. Although spaces are not restricted and it will work with them, I decided to apply the same naming conventions as for class properties in Yii.
Let’s go through the methods:

• isRunning($id). Takes process ID as a parameter and return true if this process is running.
• logStarted($id). This checks if process is not already running (or if duplicates are allowed) and adds records to the pids file. Actually this interface may be implemented using the DB, so no locking or other things will be needed.
• logFinished($id). Removes record from the pids file (or from DB if implemented using it).
• logTerminated($id). Nearly the same as previous, but it doesn’t check if process is running, it tries to remove the record anyway.
• sendSignal($id, $signal). Sends signal to the process. Process should check it’s buffer file when it is running and react to signals. It is not implemented because requires changes in the command code.
• getStatus($id). Returns current status of a given process. It is not implemented either for the same reason.
• getLog($id). Should return log of a process execution. Not implemented either.

And now let’s take a look at the class implementing it.
First of all, it’s initialization:

• $pidFile is where all currently running processes are stored
• $pidLogFile is where we write “DD.MM.YYY HH:MM:SS process XXX started” and “ DD.MM.YYY HH:MM:SS process XXX finished”
• $processDir is where log files and exchange files should reside (methods sendSignal, getStatus, getLog)

We get these variables from the config file and update to fit current application path. Also, since we extend the class from CApplicationComponent, we should perform standard initialization, an we do this by calling parent::init() in the last line.

Since we’re using file, it should be locked from other processes when we’re working with it. So we need a method that will open it and wait if it is busy now:

Idea is pretty simple – we try to lock the file. If we succeed, we return the file pointer and we throw exception otherwise. We try it several times until we reach timeout. Intervals are random to ensure that different processes started simultaneously will not try to lock it at the same time.
And now we can implement our methods:

So now we just need to call these methods from our command. But you know, I’m very lazy, so I created a base class for all my commands:

It’s constructor created the process ID from the class name. It is quite convenient feature. It also handles start and stop so all we need to do is just to implement runCmd() method in the derived classes like this:

Then you just type:

Can’t start process! It is already running!

First command starts routine in background. Second command starts the same command but in the terminal. So you’ll see the output for it. It says that it can’t be started because previous run is not finished.

There is one thing you should do in order to make that work. You should add a component to the application configuration. Open the protected/config/console.php file and add into the components list:

And into the import section:

Other problems

I’ve been using this solution for 5 months. Practice shows that sometimes process may run out of memory or some other error occurs that is not caught by out exception handling and it is terminated without our $tm->logFinished() call. This leads to the situation when process is not actually running, but task manager thinks it is because it’s record is still present in the pids file. In order to prevent such situations, I modified isRunning routine like this:

This performs additional check by trying to find the selected PID in the system process list by listing all processes ps -A and checking if our pid is there (grep [pid]). If result is empty, then process is not running.

There is a potential pitfall in the isRunning method. If you try to do something with pids file before it is closed in line 21, you’ll get into the deadlock because file is locked for modifications. That’s why I call logTerminated after it is closed. Be attentive!

I’d be thankful if someone takes the code and adds signal processing there and maybe implements DBTaskManager in addition to FileTaskManager. Let me know if someone is interested, we can collaborate on this and deliver a nice ready-made extension.

As usual, any comments are welcome!

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Most of the beginners and, sometimes, even more advanced programmers, want to develop everything from scratch because they want have control over the every aspect of their application. I entirely understand this tendency. When you learned how to program and you can code nearly anything you want (this relates more closely to scripting languages like php, perl, js etc), you start to do something global. Something that will change the world. This may be a new CMS, a new framework, new blog of e-commerce engine. And when you start this, you want to code it in the most efficient way, optimizing every little thing: every function, every SQL query. At the same time, you want to reuse the code you’re writing. So you tend to create more general solutions than you really need. And this slows down the performance. And again you go into optimization, limiting the possible uses to some extent.
Finally, if you finish your application, it may be faster or better than similar ones (however, usually it is the same or worse), but remember how much time you spent developing it. Yes, you’re proud of yourself – you’ve created your own DB abstraction layer, that handles SQL injections in a very smart way. You’ve also developed your own ORM engine, that goes nicely with your DB abstraction layer. It is quite likely, that you’ve already created your very own template engine and a nice JS library that makes animation very easy. Great job, my fellow programmer!

…but is it really brilliant?

Now let’s stop for a moment and look back. Why you’ve started developing all that? Say you wanted to create a brand new blogging engine, that is better than WordPress. You’ve spent several months (I can’t believe you’ve created nice DB abstraction, ORM and template engine in a week) to develop all that stack. And all you got is just another blog engine! Yes, it is cool and is capable of doing hundreds of things. You’ve even thought of the external API, so you can send SMS to be automatically published on your blog. But actually that’s just a blog!
Do you really think that the final result – blog engine was worth 2 months of the hard work and sleepless nights? If you answer “yes” and you haven’t got a feeling that something is wrong, then then you’re either mega-guru and develop only facebook- and google-scale projects or just a beginner who is still very proud of himself that he can code anything and you want to overemphasize this.
I think, every experienced developer was at that stage. I was there too. Moreover, I was there quite long and I regret of this.

So what is the framework?

By this time you may be quite bored and start thinking – what all that story has to deal with the framework. Actually the problem described above when you develop everything from scratch occurs when you don’t use any frameworks.
So what is the framework? Basically it is a set of libraries/classes, tied together that speed up the development of a certain type of applications. For example, MFC (Microsoft Foundation Classes), Qt, wxWidgets are frameworks that simplify the user interface development. It obvious that you should not draw 6 states of the button and change it on different events when it comes to placing the button on some form. You just use the ready-made class. But when it comes to the web-development, some developers think that they don’t need the ready-made (and well-tested!) DB abstraction layer, proven authorization classes. They want to develop it themselves. As a result, we often see low-quality and buggy applications. On the other hand, using frameworks does not necessarily gives you the excellent code, but it definitely helps you to get where you need much faster.

Use frameworks!

When you utilize good framework, you automatically get proven solutions in different spheres – DB connectivity, unit testing (most of frameworks provide tools for this or make it more simple), authentication, users and templates management. In addition, you also get a number of widgets, ready-made components for the problems you even don’t know about at the moment you start coding, but which usually come up during the development – caching, JS and CSS minimizing and publishing etc. Finally, you also get a bunch of extensions for a variety of tasks: RSS creation, CAPTCHA solutions, autocomplete widgets, treeview, drag&drop components to name a few.
The problem with framework usage for scripting languages is that they are quite simple and lots of developers don’t want to learn complex framework in order to achieve some simple task. And they are right… as long as their task is really simple. I think, “simple” is something that soesn’t require heavy DB usage, caching, user management, web-services integration and has very basic MVC implementation. That’s a simple homepage. All other projects should be developed with frameworks because that saves time and allows you to achieve better result.

Choosing the framework

But there is a little problem here What framework is “good”? I wrote about this in one of my previous posts where I explained why I chose Yii as a PHP framework for my project. Selecting wrong framework may result in project failure, so please refer to that post to see how I did that, maybe you’ll find that approach reasonable and join us in Yii community
And the last question I want to discuss here. The quality of code. I believe that using a framework with good design helps you to create better-structured applications that can be easily managed and supported. And the quality of code… I think this is up to programmer. It is definitely easier to create good application with properly designed framework just because you follow it’s patterns. However if you ignore the framework’s ideas and continue to create classes with static methods instead of simple functions, the framework is not very useful for you

When you should not use the frameworks

Frameworks are good, but you should clearly understand what are they doing and use them only when appropriate. As a rule, framework provides a set of generic tools that simplify creation of any project. Key work is any. These techniques give good result for a standard use. However when it comes to some specific things, especially if they are very demanding in terms of performance, it’s time to evaluate your tools from the point of view of this specific project.
Consider you have a framework designed for a web 2.0 applications. It works well for the most of the projects you create. However you’ve got to build an SMS portal. Unlike your usual applications, this doesn’t deal with HTTP a lot, it uses it’s own protocol. So 80-90% of your favourite framework’s tools will be useless here.
If you’re building very large application that will be working under the high load, you should evaluate your framework and either reconfigure it or refuse from using it at all. Most of the great websites like facebook, for example, don’t use standard frameworks, but in order to become “facebook”, you should gain popularity and using the framework for your first version ill bring you to the marketing and launch much faster and you’ll start earning money.
Similarly, if you’re building the business-card-like page, using the framework is a bit overkill

Summary

1. Use frameworks because they simplify your development, help getting better code and simplify documenting your solution (framework documentation is already written!)
2. Choose framework very carefully!
3. When solving non-trivial tasks and develop extremely large applications, re-evaluate your framework’s features and decide if it really worth using it there.

Disclaimer

1. All above is my own opinion, formed during the 5+ years of everyday web-development. Maybe this is not the absolute truth, but I believe it is somewhere near it
2. Example with SMS portal was for illustration only, I’ve never developed ones and don’t know which protocols are used there. I’d be thankful if someone of my readers clarifies this.

And what do you think about it? I’d be thankful if you share your thoughts in comments. Thanks!

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Yes, really it matters. Each call starts after the previous one ends and it takes long to finish them all. But each web application handles the concurrent requests by nature. So I started a forum thread to clarify this. And sebas recommended me to use CDbHttpSession instead of standard one. Yes, I knew, that storing sessions in DB speeds up the application, but I thought that is because of more effective search. And it turns out, that besides more effective search it gives one more DB benefit – concurrent access to the session storage. That’s actually the reason #1 why we usually use databases instead of files
So here is a quick trick, that solves the problem:

It creates a new table YiiSession and uses it for session data. This time the JS test above runs much faster. Try it yourself!

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Nearly every large application has some configuration parameters – site admin email, cache time for different blocks, number of latest news, number of items per search or catalogue page etc.

Standard Way

Yii comes with the built-in mechanism for this. You can create a file with an associative array of all your variables, like this:

Then put it somewhere, for example in /protected/config/params.php and configure your application to use it:

Usage is simple, just refer to it as to array:

This is quite convenient when you just want to save some constants. End-user will not be able to change this from admin in any way. Sure, you can create the interface for editing this config and update the file, but much more flexible way is to store the configuration in DB.

The Database Way

To make the life easier, I’ve created an extension, that can be used as an application component. Extension can be downloaded from the Yii extension page. Here I’ll give an overview of it’s work.

First thing to note, it implements IApplicationComponent, so it should implement init() method where initialization is performed. In this method we check if there are any parameters to load. If so – we try to load them. If there is an exception, it means, that no table is present, so we create it and initialize the parameters that should be preloaded with null values

If parameter is requested, in the __get() method we reuse the parent’s (CAttributesCollection) contains() method, which checks if we have this parameter in the collection already. If so – we simply return it. If not – we call the loadParam() method, it loads if from DB and adds to the component collection. If there is no attribute, it throws an exception. We catch it in the __get() and call the parent::__get(). Here is the code:

Setter method simply updates the collection. We check if the parameter is already present in DB. If so – we update it. If not – we create it. And, if everything was OK, we add the parameter to the internal collection ($this->add($name,$value)):

When configuring the extension or using load() or purge() method, we can specify parameters either as array or as comma-separated string. This is achieved by the preprocessing method, which also converts parameter names to lowercase and they are case insensitive (if this is required by the caseSensitive property, it must be set to true). Here is it:

And, finally, load and purge methods:

Usage Instructions

Using the extension is simple. Like any extension, you should unzip it to the /protected/extensions/ folder and then set it up as an application component. To do this, you should add it with any name to the components array in the application configuration:

‘par’ is the ID of the component, you will refer to it in any place of the application by it’s ID:

That’s it! Enjoy!

UPDATE: Updated on March, 5 2009 according to the Jonah’s comments an this discussion. Version 3 is planned with great functionality

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

In Yii you can have all your JS and CSS files together with your PHP. For example, you have some view in /views/user/create.php and you can have your css & js in /views/user/assets/css/ and /views/user/assets/js/. That’s quite convenient when you reuse your views/controllers/components from application to application. But these directories are not accessible from the web, so there is publishing mechanism there. You should do:

Yes, it’s simple, but why should I do these routines all the time!

…and solution

Luckily, in Yii you can change everything. When we write

we access the clientScript application component. This is default component, it is created from the class CClientScript. So we can extend this CClientScript and replace it in the application configuration file with our class. OK, here is /protected/components/AdvancedClientScript.php file:

I’ve added 4 methods there:

• registerScriptDirectory – registers all files in the specified directory as JS files (no check is performed)
• registerCssDirectory – registers all files in the specified directory as CSS files
• registerScriptFiles – same as first, but deals with an array of files
• registerCssFiles – same as second, but deals with an array of files

And no we’ll replace default component with ours. My config is in the /protected/config/main.php:

In order to make it work, you should enable caching. This is quite simple. I’ll show, how to use caching in DB:

1. You should have your DB connection configured. See details here.
2. Add new component to the application config:
   1. …
   2. ‘cache’ => array(
   3.         ‘class’ => ‘system.caching.CDbCache’,
   4.         ‘connectionID’ => ‘db’//this is id of the DB module. If you didn’t change anything, it should have name "id"
   5. ),
   6. …

We’re done! Now we can publish the files simply calling:

Note, that the code I wrote uses caching and when I call

I cache this for 30 mins (1800 seconds). However, it is much more flexible to have either constant or file with parameters.

UPDATE: Post updated after discussion in the Yii forum according to the Maxximus advice.

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

In December I’ve got a project, that required sophisticated and quite complex DB and that DB was subject to change greatly during first development phase. Before that project I was using Sparx Enterprise Architect both for application design and DB design. But, frankly speaking, it’s not good at databases, especially MySQL. So I decided to pick up a new DB modelling tool.

Requirements

My requirements to it were:

• Convenient entering of the attributes and their types. When I have an idea, I want it to be recorded fast, I don’t want to spend time clicking different “add” and “save” buttons in order to enter table attributes
• Full MySQL features (including all types of indexes) and main storage engines support (MyISAM, InnoDB, MEMORY)
• Visual tables layout like in MS Access
• Maintaining table joins and involved indexes, visual creation of table connections
• SQL export
• Free!

On the way to perfection…

I went through this list and installed and tried around 10 different tools and I didn’t like any of them. One has no visual modelling, other has inconvenient input, third has both, but doesn’t support a number of MySQL features. So finally I ended up with MySQL Workbench. It gives me all I need and even more.

Yes! It’s really perfect

Here you may find a list of features together with versions comparison. You can read it yourself and I want to describe why I liked it:

Nice design

Yes, it matters. When you use Windows95-like program, you can’t feel as good as if you’re using excellent-designed one. Here is a screen:

Model Overview

Creating DB

I liked their “schema approach”. When you create DB, you set the schema default encoding and all the tables in DB are automatically created in this encoding. Sure, you can change it for each table, but this is very helpful. I’ve also set up mydefault storage engine – InnoDB (MyISAM is default). This setting can be found in a seconds; everything is in it’s place (Tools -> Options -> MySQL).
So now I can create my table without remembering that I should change some settings each time I create new table. That’s my first requirement.

Creating tables

I liked their template approach to the generic field names. Near every table has ID field. In my name conventions (since I’m using Yii framework, I follow their), ID field name is [tableName]ID. So I go to the Tools -> Options -> Model and set there:

• Primary key pattern – %table%ID
• Primary key default type – INT(11)
• Foreign key defaults (since I’m using InnoDB, which takes care of data integrity, I set this. I’ll tell more about this in my future posts about MySQL&InnoDB)

Three little settings, but it makes my life much more simple! Actually, you can set up the naming of the auto-generated index fields, but I don’t use it, I create all tables I need myself. However, Workbench can create cross table for Many-to-Many relation for you.
Now I create the table (collation and engine are set automatically), go to the “fields” tab (tabs are in the bottom – I had problems finding them first time) and have my ID field created. Unfortunately, I can’t say that I want first letter in lower-case, so I correct it manually. Then start adding fields. It is very simple. First you double click on the empty row where should one name of the field, enter your name, press TAB, enter the field type (I usually enter 1-3 letters and press “Down” key, it selects correct type), press TAB again and you’re on the next field! So you just keep populating your attributes really fast! Then just tick NN (not null) and AI (auto increment) checkboxes and your table scheme is done. All indexes are set up on the next tab – Indexes except PRIMARY KEY. ID field is PRIMARY KEY by default. To add more fields there, select field and tick PRIMARY KEY checkbox in the right part of the tab.

Visual layout

Consider we created tables we need, now we advance to the scheme and connections. Double click “Add Diagram” in the top of the screen and you’re there. In the centre of the right pane you have tables you created. If you drag table from that list to your diagram, you’ll see big dot near it’s name. That’s also very convenient when you have a number of tables. Sure, here you can snap to grid, show/hide it, but that’s not so impressive.

Visual grouping

And this feature is mega-great! When you have huge DB, you can group tables using coloured layers:

ER Diagram with coloured layers for grouping

Visual connections

Yes, it allows you to connect tables like you want. 1-1,1-N and N-M relations. If you prefer to create foreign keys automatically, yo can use these types of joins. I prefer to create all tables and fields and then connect it. So I click on the last icon in the left toolbar, select foreign key, then click “Pick referenced columns” and finally select the primary key I link to. I’m done. All InnoDB connections are automatically created.

Creating relationship using existing columns

SQL export

Sure, SQL export is not new or impressive feature, but MySQL Workbench allows you not only to create SQL for the scheme, but also to create ALTER DB script for the CREATE DB script already created. For example, you designed a great DB, exported to SQL query, installed on server and then your client wants changes. You open MySQL Workbench project, make necessary changes and then create ALTER script. Very convenient thing

Integration with Yii

One clever guy (unikly) created the Yii shell command, that allows to use mwb file instead of DB connection when creating Yii DB models. This command is available on the Yii extensions page. As for me, it’s a very big plus.

A fly in the ointment

I’ve noticed several bad things in this software:

• When DB becomes huge, you need a good machine to work with it comfortably. It takes too much resources
• Some interface elements are not evident (In the DB overview screen lowest part has tabs both on the top and on the bottom. Top tabs – switch between tables, bottom tabs – switch between table’s sections (properties, fields, indexes)
• When creating joins, it doesn’t maintain the uniqueness of foreign indexes names. Take a close look at the export script (and watch for constraints and auto-created foreign keys) when it fails in phpMyAdmin (or other tool)

Disclaimer: Screenshots are taken from MySQL Workbench site, I was lazy to take my own ones.

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

I’m developing web-applications since 2004, I am crazy about nice architecture and first-class CSS+JS frontend. On the server side, I was using my framework, quite basic one. And all my projects were based on it.

In the October 2008 I finished one of my projects using my own framework. Then looked at it critically, then saw description of the Zend Framework (it had so many features, that mine didn’t!), and decided to move from my own development to the better product.

Main requirements to the framework were:

• 100% OO architecture. I don’t like to work with Drupal or jQuery. I enjoy ExtJS, for example. And I wanted the same nicely designed PHP framework. Nice OOP in PHP began from the PHP5, so all frameworks, that were initially written in PHP4 and not completely rebuilt to PHP 5 are not considered.
• Component/modular architecture
• Event-driven programming. I enjoyed this thing in ExtJS and wanted to write PHP code in the similar way
• Excellent documentation. Again, I compare this to the ExtJS docs, which are VERY nice. So I wanted every little feature explained and documented.
• Active community. It’s never possible to cover everything in the documentation, so active community is a VERY big plus.

Sure, framework must be free.
Initially, I was choosing between ZendFramework, CodeIgniter, Symfony, CakePHP. However, fully PHP5 are only ZF and Symfony. Yes, I know, that both Cake and CI have PHP5 classes, but they are not designed for PHP5. CI, in particular, states:

Note: CodeIgniter will run on PHP 5. It simply does not take advantage of any native features that are only available in that version.

But neither ZF nor Symfony had event-driven programming, so I found one more framework – PRADO. I liked all it’s features, expecially component approach and event-driven programming, but on the prado forums I found an announcement, that Main team members has developed a completely new framework, that took best things from PRADO, but has significantly better performance. Yes, it’s Yii, the framework of my choice.

OK, I’m here, but what’s that?

From the official site:

The name Yii (pronounced as [i:]) stands for easy, efficient and extensible.

I studied the definitive guide. (By the way, it is now available in 9 languages. Translation is done by the community. German and Russian translations are almost finished, others coming soon)
I immediately liked it’s:

• 100% OO architecture. It is really good application design.
• Authentication & roles mechanism
• Caching techniques
• DB access, which is based on PDO
• Active record and relational active record implementation
• Validation – that is really, really nice. To create quite complex register form (check if login is unique, if email is unique, email match with confirmation, passwords match, validate integer/string values, check empty fields and give nice error messages for each field you need only template (view) and model with rules defined. It took me 10-15 mins to do that!)
• Component concept. Just to give an idea, why is it nice – you can define getter and setter methods for properties, you can define read-only properties for components, define and invoke events, attach event handlers and additional features to the class without modifying it, just by attaching additional behaviour to it

And, yes, Yii is quite fast. And in combination with it’s advanced caching it can be used for the high-loaded websites, I’m developing one with it now

By the way, Yii is the only PHP framework selected for the Great Indian Development Award voting. I voted for it, maybe you’ll like it also? Try it and vote for it!

P.S. If you’re Russian, join us in the Russian discssion board, I reply all threads there and clarify all questions.

More info

• Comparison of web application frameworks
• PHP Frameworks

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

So here is a list of things to check when you experience strange things (blank screen without any messages is the worst one that may happen):

• Check PHP version. Yii needs 5.1 at least. When using some specific PDO or other features in your application, ensure they’re available in the server’s PHP installation.
• Check MySQL version. Should be at least 5. If it differs from development one (e.g. server has 5.0.21 and you have 5.0.51), then pay attention to the CDbConnection::emulatePrepare. Actually if you’re using MySQL, you’d better turn it on because MySQL doesn’t cache prepared statements and has buggy support for some types of qeries (e.g. SHOW)
• If you’re using automated timestamps insertion like described in Yii Cookbook, you may encounter problems, when using different PDO or MySQL version. If you see your queries fail, try using
  1. date(‘Y-m-d H:i:s’)

  instead of

  1. new CDbCommand(‘NOW()’)
• When you can’t see any output, configure logging. If log is emply also, try die(‘message’) in all possible places. Starting from the entry script, continuing to your controller’s action. In some cases PHP crash is possible, which leads to immediate application termination, which makes both Yii logging and PHP’s echo ineffective and only die() workds fine
• When using different PHP version, PDO versin may be different also. If you have application crashes when using bindParam(), try using bindValue() instead. This helps sometimes
• Don’t forget to switch off YII_DEBUG in the index.php (entry script). However, if you encounter any problems, switch it back to help debugging

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.