Database structure and filtering approach. SMS Notification system.

Konstantin Mirin — Fri, 19 Mar 2010 06:00:38 +0000

The database structure

In the previous post I outlined the system’s specs and use cases. We also selected the primary use case we should start from – that is filtering screen.

The complexity with DB design here is that we want maximum flexibility on the one hand and we don’t want to lose much in performance on the other hand.
Let’s recap what data about student we should store:

Phone
Password
Email
ICQ
Name
Surname
Patronymic
Gender
Birthday
Faculty
Course
Group
Clubs
Tags
OK to send SMS (true/false)

Items 1-9 and #15 are scalar values and can be easily stored in User table as fields of the correspondent type. While Faculty, Course, Group and Clubs are either N-M (Clubs) or 1-N relations. So obvious way to handle them is to create tables and cross-table for N-M relation. Tags can be also represented as N-M relation.
It seems OK until we don’t want to extend the system by adding some more fields. But even without extensibility in mind, we should create 3 separate tables (for courses, faculties and clubs). However if we look closer at these, we’ll see that all of them are kind of groups and we can create a kind of hierarchy:

University
- Faculty
  - Course
    - Group
- Club
  - Section

So we can consider everything to be group. With this approach we also store information about relations between these groups.
With such approach we can add more types of groups in the future if needed. Or this solution can be ported to another domain model (e.g. school or business organization) without changes in the structure. We should only define, which groups are required for user. In this case Faculty, course and group are required. However, knowing the group, we can get both course and faculty. But I’m not sure if that is a good idea because this way we’ll have to make several queries to the DB in order to get faculty instead of one.
Here’s the DB scheme we came up with:

Database design

Filtering it

Let’s try to compose some simple queries to test if our DB can cope with them.

Let’s get all male users who’re elder than 18:
1. SELECT * FROM User WHERE gender=‘male’ AND birthday <= ’1992-03-19 23:59:59′
Date for the birthday is set to the 18 years back from now. Everyone who was born before or on 19 of March 1992 will fall into the condition. And gender is obvious.
Let’s try to get all male that have first names starting with ‘K’ and who study on the first or second course:
1. SELECT u.* FROM User u INNER JOIN UserGroup ug ON u.userID=ug.userID WHERE u.gender=‘male’ AND u.name LIKE ‘K%’ AND ug.groupID IN (5,6)
Suppose that in Group table we have groups for the first course and second course and they have Ids 5 and 6 respectively.
Let’s try to search for all who is on the 3rd course and is also in Geeks’ Club:
1. SELECT u.* FROM User u INNER JOIN UserGroup ug ON u.userID=ug.userID WHERE ug.groupID IN (7, 15)
7 is ID of the group “3rd course” and 15 is ID of the group “Geeks’ Club”.
We have the problem with the last query It will find all users who either studies on the 3rd course or attends Geeks’ Club. However we need 3rd course and Geeks’ club.
There are two possible solutions here. One is SQL-based, another is PHP-based.
1. 1.SQL-based:
  1. SELECT * FROM User WHERE userID IN (SELECT userID FROM UserGroup WHERE userID IN(SELECT userID FROM UserGroup WHERE groupID=15) AND groupID=7)
  Here we select users who have group=15. Then from these users we select the ones who have group=7 as well, and finally we select user records that correspond the userIDs found.
  I think, the variant with INNER JOIN is also possible here (at least, joining results of the groups filtering instead of sub-querying). It just takes some time to figure it out and it has to deal more with optimization than with the principle of filtering.
  Anyway, the more groups we have here, the more complex this query becomes.
  UPDATE: Here I’ve been advised to use the following query which is more simple and efficient:
  1. SELECT
  2. u.*
  3. FROM
  4. USER u
  5. INNER JOIN
  6. UserGroup ug1
  7. ON
  8. u.id = ug1.userID AND ug1.groupID=7
  9. INNER JOIN
  10. UserGroup ug2
  11. ON
  12. u.id = ug2.userID AND ug2.groupID=15
  I think this is the best way of making such queries. Great thanks to ScallioXTX
2. PHP-based. Basically, the principle is the same:
  1. $group7 = getUsersForGroup(7);
  2. $group15 = getUsersForGroup(15);
  3. $userIDs = array_intersect($group7, $group15);
  4. $users = getUsersByIDs($userIDs);
  getUsersForGroup actually executes the query:
  1. SELECT userID FROM UserGroup WHERE groupID=[GROUP]
  getUsersByID executes the following query:
  1. SELECT * FROM User WHERE userID IN ([ids])
  array_intersect is a PHP function that returns the values present in all given arrays, returns the intersection of sets.
  For example, if we give it 3 arrays:
  (1, 5, 8, 15)
  (5, 8, 15, 19)
  (1, 5, 15, 25)
  it will return (5, 15).
  In this case we execute several little queries instead of one big. In MySQL this often gives benefits in performance.

So that’s our filtering approach.

There is one thing to add here. If we have group #1 and #2 and group#2 is a child of #1. And user selected to filter users which are both in #1 and #2, then we should perform a little optimization and search only for #2 because that will be the correct answer to the query. This situation is stupid, but we should act smart even in a stupid queries

Maybe there are better solutions. We’re open to them! Moreover, we realize that there are more experienced developers that already solved such problem. Help us

Tomorrow we have a meeting where we’ll be discussing the architecture, classes etc. I’ll upload our results on the weekends or early next week. This depends on the amount of work done during the meeting.
Stay tuned! Subscribe to my blog’s RSS, share your opinions/suggestions in comments and don’t miss the next post in this series!

Liked the post? Bookmark it

Hide Sites

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Building high-loaded portal – InnoDB vs MyISAM

Konstantin Mirin — Wed, 20 May 2009 07:00:55 +0000

Preface

In November 2008 I’ve got a nice project, that resulted in a complex search engine, that specializes in clothing. I can’t give the link now, but will do when site officially launches. Key features of such service are:

Large number of users, which search with various parameters, rate items, leave comments, create articles etc.
Large database, that is updated frequently

Sure, the site is running on a dedicated server, sure I use caching, but optimal DB is a key to success.

MyISAM or InnoDB?

Since we need reliability, I created different tests, search the internet etc. My main question was – what to use MyISAM or InnoDB? I found the post on the MySQL Perfomance Blog (also this and this) and came to the conclusion, that when SELECT’ing, InnoDB is definitely faster (or, at least, is not slower). In addition, it provides constraints mechanism and allows transactions. The only thing it doesn’t have is FULLTEXT search. One more disadvantage is slower inserts. However, 80% of DB usage in this project are SELECT queries without FULLTEXT. For the fulltext search I created MyISAM table with fields I need and it works fine. The project is finished, but there will be phase 2 and I’m thinking of introducing Sphinx there. Anybody has experience with it?
So I came up with InnoDB storage engine.

Optimizing structure

Next important step is DB structure. I’ll give some background info about the project so you could better understand what I’m talking about. The site collects info from different UK shops. Then users can rate them, add some tags, create articles (called “looks”) and link that articles to the products they’re writing about. And the same (tags, comments, rating) applies to the looks. Here we come across with an interesting question. We can create structure of different types for this:

Create 2 separate cross-tables for linking Rating to Look or Product
Create one cross-table and add additional field “type” there to differentiate between Product and Look entities

See both variants in the following images:

Normalized variant

Denormalized version

Initially I thought that the second variant would perform better, but after creating and running tests, I saw, that:

If there are 0-2 ratings linked to entity, then normalized variant is better
If there are 3-5 ratings linked to entity, then denormalized variant is slightly (less than 1%) better
If there are 6-8 ratings linked to the entity, the normalized variant wins again

I did these tests for different situations similar to this one, in each test I generated random records and randomly linked them with each other. Number of record in Product, Look tables was 10000, in Rating – 20000. Number of records in cross tables varies because the linkings are random.
So I came up with normalizing everything like it should be done according to the DB theory.

The only exclusion of this rule is the following situation. In this project you can add different alternatives to the product. Each alternative is “better” or “cheaper” or “different” etc. And since the only case when these alternatives are pulled from DB is when user views the product, they are pulled altogether, so it is better to keep them in the same table.

Conclusion

InnoDB turns out to behave better under the high load, it supports transactions and maintains constraints. All that helps to develop better and never leave orphan records.
Using normalization techniques you usually get better performance
Create tests and test everything if you are going to do something great

Programmer's Notes » DB Design