Programmer's Notes » efficiency

8 Firefox plugins I use every day

Konstantin Mirin — Tue, 16 Mar 2010 06:00:39 +0000

Firefox is great for web development mainly because of the large number of plug-ins available. Here’s my list:

Web developer toolbar – great thing for the web development. Personally I use resizing features, password revealing and elements highlighting.
Firebug – absolute winner. This is actually the main tool for development I use. I’m going to write a more detailed post about it later. For now, you can enjoy this video
Cache status shows cached size in the status bar. But the main use is it’s ability to clear cache in a few clicks. This saves lots of time clicking here and there when developing complex JS application.
Live HTTP Headers allows to view all HTTP headers that go here and there while you’re requesting the page. It’s a great thing when you need to grab some content from the password-protected area or inspect what’s going on when you request page.
YSlow is a firebug plug-in from the Yahoo team which analyzes your content and shows you the ways of the client-size optimization of your page. It’s recommendations are really valuable. Try it and you’ll never uninstall it
SitePoint HTML reference is a firebug plug-in too, that adds HTML help to the every HTML tag and CSS property. It also provides examples and best practices. I don’t think this is great value for gurus, but definitely useful for all others
Fasterfox is a plug-in that boost Firefox performance. It is in the “experimental” state, but quite stable. I haven’t ever experience any problems with it.
Selenium IDE is designed as a helper tool for creating selenium tests. It is really great and handy tool if you use automated testing. I’m going to cover this in my blog later.

And what are your favourite plug-ins? How do you use them?

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Building high-loaded portal – InnoDB vs MyISAM

Konstantin Mirin — Wed, 20 May 2009 07:00:55 +0000

Preface

In November 2008 I’ve got a nice project, that resulted in a complex search engine, that specializes in clothing. I can’t give the link now, but will do when site officially launches. Key features of such service are:

Large number of users, which search with various parameters, rate items, leave comments, create articles etc.
Large database, that is updated frequently

Sure, the site is running on a dedicated server, sure I use caching, but optimal DB is a key to success.

MyISAM or InnoDB?

Since we need reliability, I created different tests, search the internet etc. My main question was – what to use MyISAM or InnoDB? I found the post on the MySQL Perfomance Blog (also this and this) and came to the conclusion, that when SELECT’ing, InnoDB is definitely faster (or, at least, is not slower). In addition, it provides constraints mechanism and allows transactions. The only thing it doesn’t have is FULLTEXT search. One more disadvantage is slower inserts. However, 80% of DB usage in this project are SELECT queries without FULLTEXT. For the fulltext search I created MyISAM table with fields I need and it works fine. The project is finished, but there will be phase 2 and I’m thinking of introducing Sphinx there. Anybody has experience with it?
So I came up with InnoDB storage engine.

Optimizing structure

Next important step is DB structure. I’ll give some background info about the project so you could better understand what I’m talking about. The site collects info from different UK shops. Then users can rate them, add some tags, create articles (called “looks”) and link that articles to the products they’re writing about. And the same (tags, comments, rating) applies to the looks. Here we come across with an interesting question. We can create structure of different types for this:

Create 2 separate cross-tables for linking Rating to Look or Product
Create one cross-table and add additional field “type” there to differentiate between Product and Look entities

See both variants in the following images:

Normalized variant

Denormalized version

Initially I thought that the second variant would perform better, but after creating and running tests, I saw, that:

If there are 0-2 ratings linked to entity, then normalized variant is better
If there are 3-5 ratings linked to entity, then denormalized variant is slightly (less than 1%) better
If there are 6-8 ratings linked to the entity, the normalized variant wins again

I did these tests for different situations similar to this one, in each test I generated random records and randomly linked them with each other. Number of record in Product, Look tables was 10000, in Rating – 20000. Number of records in cross tables varies because the linkings are random.
So I came up with normalizing everything like it should be done according to the DB theory.

The only exclusion of this rule is the following situation. In this project you can add different alternatives to the product. Each alternative is “better” or “cheaper” or “different” etc. And since the only case when these alternatives are pulled from DB is when user views the product, they are pulled altogether, so it is better to keep them in the same table.

Conclusion

InnoDB turns out to behave better under the high load, it supports transactions and maintains constraints. All that helps to develop better and never leave orphan records.
Using normalization techniques you usually get better performance
Create tests and test everything if you are going to do something great

Creator and Information Expert principles – GRASP (Design patterns series)

Konstantin Mirin — Sat, 28 Mar 2009 13:37:32 +0000

Preface

This is an introductory post to the series, where I’ll cover all design patterns, their usage, diagrams. Besides patterns, I am going to guide you through the principles of the high-quality application design. We’ll start from these principles (GRASP), and then advance to patterns, see how they use these principles and develop several systems (architecture only). The whole series will take 10-15 posts. However, I don’t have exact plan yet So, let’s go!

So what we’re talking about?

Actually, design pattern is no more, than a proven solution to some common problem. When saying “problem” I mean things you face with when developing the application architecture. These solutions are intended to make you application better – more extensible, flexible, simple.

Common examples of design problems (Let’s agree that in this series “design” means application, not graphics or DB design) are:

Organizing access to the shared resource. This can be application object, DB abstraction object, Session object or other similar things. Main problem is that they should be created only once and accessed in every method easily. Singleton pattern solves this problem.
Creating the easily extensible application. For example, code shopping cart in the way, that allows to add new payment or taxation modules easily. Adapter pattern saves us here.
Providing the simple interface for accessing features of the application or application module, hiding all the underlying classes/functions from the user. Façade pattern does this.
Separating data from data processing and from presentation. You’ve definitely heard about MVC (Model-View-Controller) – that’s also a pattern!

So patterns are everywhere in OOP and knowing them is essential for high-quality and extensible application design.

The basics

However, patterns themselves are implementation of the 9 basic principles (GPASP – General Responsibility Assignment Software Principles).

GRASP and patterns

All OOP design is about creating real-world abstractions (classes), design methods (responsibilities) and their interaction. The key thing to good architecture is assigning responsibilities to the classes correctly. GRASP deals exclusively with this problem. We’ll go through all of them:

Creator
Information Expert
Low Coupling
High Cohesion
Controller
Polymorphysm
Pure Fabrication
Indirection
Protected Variations

Today we’ll start from the first principle.

Creator

The most common problem in OO design is “Who should create object X?” This principle states, that object A should create object B if:

A contains or aggregates B (e.g. Object User should initialize object Address)
A records B (saves to DB or file)
A closely uses B
A has the initialization data that should be passed to B when object is created

If any of these conditions are true, then A should create B. Now let’s demonstrate how these principles work in real examples.

A contains or aggregates B

Consider we have table. Common table consists of desk and legs.

Common table

If we use object decomposition to program the table, we’ll get 3 classes:

Table (blue box on image):
- Desk (light grey)
- Leg (wooden)

Object decomposition basically means breaking some entity into parts and representing each part with a object. These parts can be decomposed further. For example, we can break Table into Desk and Leg. And then break Leg into Screw, Plank. In complex system there is a number of such levels.

If we start programming this, we can go in 2 ways. For example, we don’t want to use the Creator principle and will create Table class as follows:

class Table
{
protected $desk;//desk object
protected $legs;//array of legs
public function __construct(Desk $d, $l)
{
$this->desk = $dd;
$this->legs = $l;
}
}

Then Desk and Leg classes:

class Desk
{
protected $width;
protected $length;
protected $height;
public function __construct($w, $l, $h)
{
$this->width = $w;
$this->length = $l;
$this->height = $h;
}
}
class Leg
{
protected $width;
protected $length;
protected $height;
public function __construct($w, $l, $h)
{
$this->width = $w;
$this->length = $l;
$this->height = $h;
}
}

Yes, Desk and Leg classes are really the same cause they both are parallelepipeds. So we can create Parallelepiped class and extend both of them from it. We’ll do it later, now we’re talking about object creation. The above classes can be used in the following way:

$desk = new Desk(900,900,20);//90 cm width, 90 length (square) and 2cm height. A normal table for kitchen
$legs = array(
new Leg(40,40, 880),//4×4 cm square and 88cm height. 88cm leg + 2 cm desk = 90cm full height
new Leg(40,40, 880),
new Leg(40,40, 880),
new Leg(40,40, 880),
);
$table = new Table($desk, $legs);

Yes, the object is created, but large part of the creation logic is outside of the object. The class user (the programmer who is using this class) should know the internal structure of objects to create the whole object. This approach is inefficient when we deal with large application; on the other side, such solution can’t be reused without changing the code. And it violates the first condition of using the Creator principle. Either Desk and Leg are parts of the Table object, so it should take care of their instantiation. So let’s learn how to solve such task correctly:
We’ll create constructor for Table class with parameters:

Width – width of the table
Length – length of the table
Height – height of the table
DeskHeight – height of the desk
LegSection – (assume we want square legs)

So the end-user of the class should specify only these 5 parameters, we don’t ask him to create parts of our object. OK, let’s code it:

class Table
{
protected $desk;
protected $legs;
public function __construct($width, $length, $height, $deskHeight, $legSection)
{
$this->desk = new Desk($width, $length, $deskHeight);
for ($i = 0; $i < 4; $i++)
{
$this->legs[$i] = new Leg($legSection, $legSection, $height – $deskHeight);
}
}
}

Desk and Leg classes remain the same. So here we followed the first condition (actually, the most important) and got nice architecture.

A records B

After detailed discussion of the first condition, this should be quite evident. Usually if some object saves another one, it is his part. For example, we have some system with users. Each user may have several addresses. So Address will be the part of User (it is quite logical to design so), and User object will save all addresses in one field (Using serialization). And when loading, it will load the address info from DB and create Address objects. This is example of both 2, 3 and 4 item. The User object saves Address. User object has initialization info for Address and user object closely uses Address. By the way, first item is also here, Address is part of User object.
As you see, all these conditions often go together.
I think, that the final 2 items should be evident.

Further development

Tasks for improvement your skills:

Extend both Leg and Desk from the Parallelipiped class
Implement getHeight() method for Table, Desk and Leg. Hint: In Leg and Desk classes it should simply return height. In Table class it should return sum of leg height and Desk height
Implement setHeight method for all classes. Hint: when setting height for table, you should actually increment Leg’s height
Implement Saving and loading from file or DB. Hint: actually you have to save only 5 parameters – those passed to table constructor. So when saving and loading Table class with interact with Desk and Leg classes. And when loading – will create these classes.

Information Expert

The last condition of applying the Creator principle shows, that object A should have all info about B to create it. In other words, A should be Expert in info about B. Information Expert principle helps us assigning all responsibilities in the project. Real-world application usually holds a number of classes, that perform hundreds and thousands of actions, so it is essential to assign responsibilities correctly. The principle itself is fairly simple – you should give the responsibilities to the object it has information for. For example, assigning Table class responsibility to create Desk is correct because it has info about it. Assigning User responsibility to save data to DB is also correct, because User holds data about user and can perform this task without any other objects (assuming it deals with DB without DB abstraction layer). And assigning User to save information about Group is incorrect, because User will have to “ask” Group about it’s properties – ID, name, access rights etc. in order to save it correctly. In this case it’s much better to assign Group responsibility to save itself. And Use can call save() method on Group object.

In the next post I’ll cover Low Coupling and High Cohesion principles and illustrate it with practical examples. Subscribe to RSS not to miss it!
Feel free to ask any questions in comments or by email: konstantin.mirin[at]programmersnotes.info

Working with large files eficiently

Konstantin Mirin — Mon, 16 Jun 2008 16:59:20 +0000

Once I had a task to import a number of data from datafeeds (that are in CSV or XML) into MySQL database. See solution in my another post. Here I want to tell about handling large files.

Datafeeds there are 50-200M each, so reading all file at once was not possible due to memory limitations. So we need to split it into portions. Here is the class, that solves the problem of downloading remote file, saving it locally and then reading by portions. Position is saved in $this->Offset:

class WebServiceSource implements GenericSource
{
private $URI;
private $TempFile;
private $URIHandler;
private $TMPHandler;
private $Data;
private $Offset = 0;
private $EOF;
/**
* Size to read from remote host at once, buffer
*/
const BUFFER_LENGTH = 102400;//1 MB
/**
* Number of records to read at once, from the local file
*/
const RECORDS_NUM = 15;
public function __construct($uri = NULL)
{
$this->SetURI($uri);
$this->EOF = false;
}
private function OpenURI()
{
$this->URIHandler = @fopen($this->URI, ‘r’);
if (!$this->URIHandler) throw new ASException(‘WrongDatafeedURL’, ‘DFImport’);
}
private function OpenTempFile($mode = ‘w+’)
{
$this->TMPHandler = @fopen($this->TempFile, $mode);
if (!$this->TMPHandler) throw new ASException(‘TempFileNotAccessible’, ‘DFImport’);
}
private function CloseURI()
{
fclose($this->URIHandler);
}
private function CloseTempFile()
{
fclose($this->TMPHandler);
}
private function IsDownloaded()
{
if (!file_exists($this->TempFile)) return false;
if ((time() – filectime($this->TempFile)) > TMP_EXPIRE) return false;
return true;
}
private function Download()
{
if ($this->IsDownloaded()) return;
// set_time_limit(0);
$this->OpenURI();
$this->OpenTempFile();
while (!feof($this->URIHandler))
{
$buffer = fread($this->URIHandler, self::BUFFER_LENGTH);
fwrite($this->TMPHandler, $buffer);
}
$this->CloseURI();
$this->CloseTempFile();
}
private function GetFragment()
{
$this->OpenTempFile(‘r’);
$RecordsNum = 1;
$this->Data = ”;
fseek($this->TMPHandler, $this->Offset);
while (($row = fgets($this->TMPHandler)) && $RecordsNum <= self::RECORDS_NUM)
{
$this->Data .= $row;
$RecordsNum++;
}
//if number of records read is less then maximum, then we came to the
// end of file
if ($RecordsNum < self::RECORDS_NUM) $this->EOF = true;
$this->Offset = ftell($this->TMPHandler);
$this->CloseTempFile();
return $this->Data;
}
public function GetData()
{
$this->Download();
if (!$this->EOF)
return $this->GetFragment();
return false;
}
public function SetURI($uri)
{
$this->URI = $uri;
$this->TempFile = SYSTEM_ROOT.TMP_DIR.md5($uri);
}
}

And here is an example of how it works:

$source = new WebServiceSource($url);
while ($res = $this->Source->GetData())
{
$this->Parser->SetRawData($res);
$this->Parser->Parse();
}

Liked the post? Bookmark it

Hide Sites

Low Coupling and High Cohesion – GRASP (Design patterns series) Low coupling When designing some architecture, you face with the...

Related posts brought to you by Yet Another Related Posts Plugin.

Programmer's Notes » efficiency

8 Firefox plugins I use every day

Building high-loaded portal – InnoDB vs MyISAM

Preface

MyISAM or InnoDB?

Optimizing structure

Conclusion

Further reading

Creator and Information Expert principles – GRASP (Design patterns series)

Preface

So what we’re talking about?

The basics

Creator

A contains or aggregates B

A records B

Further development

Information Expert

Further reading

Working with large files eficiently