I’ve graduated in March 2010 and I don’t need to visit university any more. However, I enjoyed studying there and I know what challenges my fellow students-programmers have. That’s why I continue the job started this autumn – computer club, we call it “Geeks’ Club”. Sure, we’re not true geeks there, but we tend to be So I’m helping students who’re interested in web-technologies and in object-oriented programming. I explain them things that are not covered in the standard university course. Things, that are more practical and they can be paid money for.

Here I’ll be covering both OOP and web-dev, but we’ll start from design hence OOP. The story began when some time passed after we started learning OOP at the club meetings and I decided that there is enough theory and separate examples, it’s time to build something practical that will be challenging for me too.
I thought of different systems, but finally decided to pick up the SMS notification one. It requires both user interface, database, API (my friends want to communicate to it from standalone Java and C++ applications) and overall application design. In addition, I’d like such system to be deployed at our university and it is interesting for me to create one.

Introduction

This is going to be a series of articles, I’ll describe the progress and challenges we come across as we proceed with development. I think this kind of posts will be interesting for those who study OOP and want to apply what they’ve learned to some practical things. I’ll try to guide my students through the process using the iterative development methodology.

We have meetings once a week (on Saturdays), so I’ll post once or twice a week, depending on amount of work done on weekends and my free time during the week.

I think, this series will contain around 10 posts. We’re going to share the work done as an open-source project. There is nothing to share at the moment, so I’ll give the link when we code something

System overview

Vision

Obviously, the main purpose of the system is to send SMS notifications to students or tutors. These notifications may be quite different – from the timetable changes to some announcements or even warnings (e.g. “Due to the weather conditions, today is a day-off for all. Stay at home and have a good time”).

Actually the system isn’t that complex. Basically all it needs to do is to enable administrator pick a group of users, enter the message and send it through the SMS gateway. The tricky things start from “pick a group of users”. We want this to be as flexible as possible so we could use this system not only for university, but any other organization.

University structure

This is obvious to anyone who studied at the university in Ukraine, Russia and former USSR, but I’m not sure the structure is the same abroad. So we have the university. There are faculties (e.g. Computer Science, Philology etc). These faculties have specialties. For example, Computer Science faculty has 2 specialties – “Intellectual Systems of Decision-Making” (mine) and “Computer Engineering”. (For those interested about the difference, first one deals more with AI technologies while the latter – with system programming, networks etc).

Next level is “course”. We call it so, but actually that is the year of study. I know, there are special words for the 1st year students, for the 2nd etc. Here we just say “I’m on the first course”. That means “I’m studying the first year”.

All students that study the same specialty and are on the same course are called “Stream”. This is literal translation and I’m sure there is some more correct term. I’d be thankful if someone posts it in comments.

There are 2-4 groups in the “stream”. In our university they are names as follows: 101, 102, 103 etc. First number is course. In this case it is the first one. Second number is the faculty code. 0 is for computer science. And the third number is the group number. Thus 102 is the second group of first-year students studying Computer Science. I realize that not all universities name groups like this, so we don’t want to stick to such naming in our product.

Now let’s go through the tutors and staff. There is a rector of a university. There are deans for every faculty. And all tutors are grouped by departments. Usually, departments relate to the faculty, but that’s not a rule. For example, on our faculty we have Department of Higher Mathematics, Department of Intellectual systems ans some others. One tutor usually belongs to one department.

In addition, there are different clubs for students like our “Geeks’ Club”, translators’ “Pickwick Club” etc. Some of these clubs (like ours) have departments (Web-development, OOP, Java, C++, Delphi, Administration etc).

We want to be able to filter users using all the above groups plus some more user-specific filters like age or gender.
Let’s go through the planned features now.

User perspective

First of all, user can log in. User should enter his phone number and password. We’ll possibly add additional security for administrators. If administrator logs in, he enters his phone and password, but then a dynamic password is sent to his phone. And he completes login procedure only after he enters this dynamic password. We’re not sure if we’ll implement this. Your comments are welcome!

Next, user can edit his profile. He’ll have the following fields there:

• Phone
• Email
• Password (and confirmation)
• Name, surname, patronymic
• Gender
• Birthday
• ICQ
• Faculty
• Course (1st year, 2nd year etc)
• Group
• Clubs user is member of
• Any tags user assigns to himself (e.g. “musician”)

Plus user should check the option “receive SMS”.

However, user can’t edit phone directly because we’ll have to confirm it. So instead of giving him the field in the profile editing form, we’d rather give him a button or a link which will bring him a from where he puts new phone number and password. After that a dynamic password is sent to his phone via SMS and he enters this dynamic password and phone is updated.

Finally, user may register. When registering, he enters nearly the same data as when editing profile, but there is a dynamic password sent to him in order to confirm the phone number.
All these screens are shown below.

User area screens

Group Administrator perspective

Group administrator role may be assigned to the club’s leader, to the group monitor (or it’s better to say headman? What sound more “English”), to the tutor or headmaster. Basically everyone who needs to send notifications will have this role.

So, this type of users can do the same as the simple user plus searching users, sending email notifications and viewing statistics.
Each group administrator will be limited to the groups he may view/notify. For example, headmaster can notify anyone. Tutor may notify all groups of students he tutors in. Group monitor may notify all students within his/her group.

Listing users and searching through them is done in the same screen. In the main part of the screen there is a list of users that are in the groups available to the group administrator. In the left side there is a “narrow search” panel which lists all filters that can be applied to the current set of users. Clicking the filter option (e.g. “Geeks’ Club”) lists all users that fall into the current filter conditions. And it also refreshes the filtering options. For example, if there is no girls in the “Geeks’ Club” then there is no point to show these option in the filter. This is harder to code, but it will bring excellent user experience.

In the right hand side there will be the “saved searches” panel. When you applied some searching criteria, you may save it with some label so you can quickly access this search in the future. Were going to create standard searches in addition to filtering to simplify the system’s usage.

This works as following. For example, you can view all Computer Faculty. But you need only girls from the Geeks’ Club. So you filter by gender and then by club. And save this search as “Girls-geeks” This immediately appears in the right “saved searches” panel.

These saved searched may be group-specific and user-specific. This basically means that when group admin saves search, he may select if he wants it to be private or he wants to share this search with other group administrators, who have access to the same groups as he does. We think, this will help users creating the searches that they really need. Sure, saved searched may be easily removed.

In the right side there will be also a “Send notification” panel. It will contain only textarea field and a “send” button. When you have your group ready, you type the message an click “send”. It asks for confirmation, displaying how much it would cost and sends it over.

Another page in the Group administrator’s perspective is statistics. Here he’ll see how much messages he sent during the last day/week/month/overall and how much did it cost.

As we progress with development, we may add some settings, that will be the third page then. Here’s the filtering screen:

Filter users screen

Statistics screen is pretty straightforward, so we’ll omit it here.

Administrator’s perspective

Administrator is almost god here He can manage users, assign rights to the group administrators, set limits of SMS count, view logs and overall statistics.
Users management screen will be nearly the same as filtering, with the only difference that when clicking the row with user’s you’ll see the profile screen with additional options:

• ban/unban
• delete
• phone editing does not require confirmation
• administrator of (group selection)
• notifications limit

Logs and statistics screens are quite simple too.

Use cases

After analyzing the above overview, we come up with the following use case diagram. It is pretty self-explanatory and written after the description above, so it is useful only as a summary.

SMS Notification Use Cases

Following the iterative development methodology, we should describe all usecases now, but I find this quite useless. After describing all usecases briefly, we should select 10% critical usecases and describe them in full. Full description includes the sequence of actions taken in the main success scenario and all alternative scenarios that may happen (e.g. DB error, connection failure, no users found etc). This is useful sometimes, but I think that for such simple application as this one, we ma skip this step either.

What is really useful about the iterative development at this stage is that it recommends picking 10% of the most critical usecases and start implementing them in full. By “critical” I usually mean those which may result in essential DB structure or application design changes. In our case there are 14 usecases.10% is 1.4 usecase. Let’s pick only one. The most critical usecase, that affects all the application design and DB structure is the user filtering. The success of an application depend on the stability and usability of this feature. So we should code it efficiently and then move to the other usecases.

I like this approach cause it is much more practical than designing the system from the interface. Changing interfaces and adopting them to core is much more simple and has less risk than doing vice versa.

So we decided to start with user filtering. The first thing we should create is the database structure. This will be covered in the tomorrow’s post.

I’d like to learn your opinion on all that. Maybe you can propose some better interface, or some nice filtering approach or just some idea for this. We’re open to all suggestions and comments! Thank you for reading my blog and following this project

Subscribe to RSS and stay tuned!

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

In November 2008 I’ve got a nice project, that resulted in a complex search engine, that specializes in clothing. I can’t give the link now, but will do when site officially launches. Key features of such service are:

• Large number of users, which search with various parameters, rate items, leave comments, create articles etc.
• Large database, that is updated frequently

Sure, the site is running on a dedicated server, sure I use caching, but optimal DB is a key to success.

MyISAM or InnoDB?

Since we need reliability, I created different tests, search the internet etc. My main question was – what to use MyISAM or InnoDB? I found the post on the MySQL Perfomance Blog (also this and this) and came to the conclusion, that when SELECT’ing, InnoDB is definitely faster (or, at least, is not slower). In addition, it provides constraints mechanism and allows transactions. The only thing it doesn’t have is FULLTEXT search. One more disadvantage is slower inserts. However, 80% of DB usage in this project are SELECT queries without FULLTEXT. For the fulltext search I created MyISAM table with fields I need and it works fine. The project is finished, but there will be phase 2 and I’m thinking of introducing Sphinx there. Anybody has experience with it?
So I came up with InnoDB storage engine.

Optimizing structure

Next important step is DB structure. I’ll give some background info about the project so you could better understand what I’m talking about. The site collects info from different UK shops. Then users can rate them, add some tags, create articles (called “looks”) and link that articles to the products they’re writing about. And the same (tags, comments, rating) applies to the looks. Here we come across with an interesting question. We can create structure of different types for this:

• Create 2 separate cross-tables for linking Rating to Look or Product
• Create one cross-table and add additional field “type” there to differentiate between Product and Look entities

See both variants in the following images:

Normalized variant

Denormalized version

• If there are 0-2 ratings linked to entity, then normalized variant is better
• If there are 3-5 ratings linked to entity, then denormalized variant is slightly (less than 1%) better
• If there are 6-8 ratings linked to the entity, the normalized variant wins again

I did these tests for different situations similar to this one, in each test I generated random records and randomly linked them with each other. Number of record in Product, Look tables was 10000, in Rating – 20000. Number of records in cross tables varies because the linkings are random.
So I came up with normalizing everything like it should be done according to the DB theory.

The only exclusion of this rule is the following situation. In this project you can add different alternatives to the product. Each alternative is “better” or “cheaper” or “different” etc. And since the only case when these alternatives are pulled from DB is when user views the product, they are pulled altogether, so it is better to keep them in the same table.

Conclusion

• InnoDB turns out to behave better under the high load, it supports transactions and maintains constraints. All that helps to develop better and never leave orphan records.
• Using normalization techniques you usually get better performance
• Create tests and test everything if you are going to do something great

Further reading

• Database Normalization
• MySQL Performance Blog

P.S. I’m looking forward to see your comments on implementation techniques in the high-loaded websites, your thoughts about the InnoDB vs MyISAM question and anything else you can think of

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Yes, really it matters. Each call starts after the previous one ends and it takes long to finish them all. But each web application handles the concurrent requests by nature. So I started a forum thread to clarify this. And sebas recommended me to use CDbHttpSession instead of standard one. Yes, I knew, that storing sessions in DB speeds up the application, but I thought that is because of more effective search. And it turns out, that besides more effective search it gives one more DB benefit – concurrent access to the session storage. That’s actually the reason #1 why we usually use databases instead of files
So here is a quick trick, that solves the problem:

It creates a new table YiiSession and uses it for session data. This time the JS test above runs much faster. Try it yourself!

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Nearly every large application has some configuration parameters – site admin email, cache time for different blocks, number of latest news, number of items per search or catalogue page etc.

Standard Way

Yii comes with the built-in mechanism for this. You can create a file with an associative array of all your variables, like this:

Then put it somewhere, for example in /protected/config/params.php and configure your application to use it:

Usage is simple, just refer to it as to array:

This is quite convenient when you just want to save some constants. End-user will not be able to change this from admin in any way. Sure, you can create the interface for editing this config and update the file, but much more flexible way is to store the configuration in DB.

The Database Way

To make the life easier, I’ve created an extension, that can be used as an application component. Extension can be downloaded from the Yii extension page. Here I’ll give an overview of it’s work.

First thing to note, it implements IApplicationComponent, so it should implement init() method where initialization is performed. In this method we check if there are any parameters to load. If so – we try to load them. If there is an exception, it means, that no table is present, so we create it and initialize the parameters that should be preloaded with null values

If parameter is requested, in the __get() method we reuse the parent’s (CAttributesCollection) contains() method, which checks if we have this parameter in the collection already. If so – we simply return it. If not – we call the loadParam() method, it loads if from DB and adds to the component collection. If there is no attribute, it throws an exception. We catch it in the __get() and call the parent::__get(). Here is the code:

Setter method simply updates the collection. We check if the parameter is already present in DB. If so – we update it. If not – we create it. And, if everything was OK, we add the parameter to the internal collection ($this->add($name,$value)):

When configuring the extension or using load() or purge() method, we can specify parameters either as array or as comma-separated string. This is achieved by the preprocessing method, which also converts parameter names to lowercase and they are case insensitive (if this is required by the caseSensitive property, it must be set to true). Here is it:

And, finally, load and purge methods:

Usage Instructions

Using the extension is simple. Like any extension, you should unzip it to the /protected/extensions/ folder and then set it up as an application component. To do this, you should add it with any name to the components array in the application configuration:

‘par’ is the ID of the component, you will refer to it in any place of the application by it’s ID:

That’s it! Enjoy!

UPDATE: Updated on March, 5 2009 according to the Jonah’s comments an this discussion. Version 3 is planned with great functionality

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

In December I’ve got a project, that required sophisticated and quite complex DB and that DB was subject to change greatly during first development phase. Before that project I was using Sparx Enterprise Architect both for application design and DB design. But, frankly speaking, it’s not good at databases, especially MySQL. So I decided to pick up a new DB modelling tool.

Requirements

My requirements to it were:

• Convenient entering of the attributes and their types. When I have an idea, I want it to be recorded fast, I don’t want to spend time clicking different “add” and “save” buttons in order to enter table attributes
• Full MySQL features (including all types of indexes) and main storage engines support (MyISAM, InnoDB, MEMORY)
• Visual tables layout like in MS Access
• Maintaining table joins and involved indexes, visual creation of table connections
• SQL export
• Free!

On the way to perfection…

I went through this list and installed and tried around 10 different tools and I didn’t like any of them. One has no visual modelling, other has inconvenient input, third has both, but doesn’t support a number of MySQL features. So finally I ended up with MySQL Workbench. It gives me all I need and even more.

Yes! It’s really perfect

Here you may find a list of features together with versions comparison. You can read it yourself and I want to describe why I liked it:

Nice design

Yes, it matters. When you use Windows95-like program, you can’t feel as good as if you’re using excellent-designed one. Here is a screen:

Model Overview

Creating DB

I liked their “schema approach”. When you create DB, you set the schema default encoding and all the tables in DB are automatically created in this encoding. Sure, you can change it for each table, but this is very helpful. I’ve also set up mydefault storage engine – InnoDB (MyISAM is default). This setting can be found in a seconds; everything is in it’s place (Tools -> Options -> MySQL).
So now I can create my table without remembering that I should change some settings each time I create new table. That’s my first requirement.

Creating tables

I liked their template approach to the generic field names. Near every table has ID field. In my name conventions (since I’m using Yii framework, I follow their), ID field name is [tableName]ID. So I go to the Tools -> Options -> Model and set there:

• Primary key pattern – %table%ID
• Primary key default type – INT(11)
• Foreign key defaults (since I’m using InnoDB, which takes care of data integrity, I set this. I’ll tell more about this in my future posts about MySQL&InnoDB)

Three little settings, but it makes my life much more simple! Actually, you can set up the naming of the auto-generated index fields, but I don’t use it, I create all tables I need myself. However, Workbench can create cross table for Many-to-Many relation for you.
Now I create the table (collation and engine are set automatically), go to the “fields” tab (tabs are in the bottom – I had problems finding them first time) and have my ID field created. Unfortunately, I can’t say that I want first letter in lower-case, so I correct it manually. Then start adding fields. It is very simple. First you double click on the empty row where should one name of the field, enter your name, press TAB, enter the field type (I usually enter 1-3 letters and press “Down” key, it selects correct type), press TAB again and you’re on the next field! So you just keep populating your attributes really fast! Then just tick NN (not null) and AI (auto increment) checkboxes and your table scheme is done. All indexes are set up on the next tab – Indexes except PRIMARY KEY. ID field is PRIMARY KEY by default. To add more fields there, select field and tick PRIMARY KEY checkbox in the right part of the tab.

Visual layout

Consider we created tables we need, now we advance to the scheme and connections. Double click “Add Diagram” in the top of the screen and you’re there. In the centre of the right pane you have tables you created. If you drag table from that list to your diagram, you’ll see big dot near it’s name. That’s also very convenient when you have a number of tables. Sure, here you can snap to grid, show/hide it, but that’s not so impressive.

Visual grouping

And this feature is mega-great! When you have huge DB, you can group tables using coloured layers:

ER Diagram with coloured layers for grouping

Visual connections

Yes, it allows you to connect tables like you want. 1-1,1-N and N-M relations. If you prefer to create foreign keys automatically, yo can use these types of joins. I prefer to create all tables and fields and then connect it. So I click on the last icon in the left toolbar, select foreign key, then click “Pick referenced columns” and finally select the primary key I link to. I’m done. All InnoDB connections are automatically created.

Creating relationship using existing columns

SQL export

Sure, SQL export is not new or impressive feature, but MySQL Workbench allows you not only to create SQL for the scheme, but also to create ALTER DB script for the CREATE DB script already created. For example, you designed a great DB, exported to SQL query, installed on server and then your client wants changes. You open MySQL Workbench project, make necessary changes and then create ALTER script. Very convenient thing

Integration with Yii

One clever guy (unikly) created the Yii shell command, that allows to use mwb file instead of DB connection when creating Yii DB models. This command is available on the Yii extensions page. As for me, it’s a very big plus.

A fly in the ointment

I’ve noticed several bad things in this software:

• When DB becomes huge, you need a good machine to work with it comfortably. It takes too much resources
• Some interface elements are not evident (In the DB overview screen lowest part has tabs both on the top and on the bottom. Top tabs – switch between tables, bottom tabs – switch between table’s sections (properties, fields, indexes)
• When creating joins, it doesn’t maintain the uniqueness of foreign indexes names. Take a close look at the export script (and watch for constraints and auto-created foreign keys) when it fails in phpMyAdmin (or other tool)

Disclaimer: Screenshots are taken from MySQL Workbench site, I was lazy to take my own ones.

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.