in-memory database: refactor needed

[mdonadoni]

Job controller is keeping an "in-memory" copy of the Job table of the database.

First of all, we should investigate why this is needed, maybe for performance reason?

If it's not needed, should we drop the in-memory database and only use PostgreSQL?

If the in-memory database needs to be kept around, then:

• use a proper class and not a dict
• all the access to the in-memory database are done through methods of the class, no direct access to the job objects
• use primitives (locks, condition variables, ...) to make sure the database is thread-safe
• the class keeps the in-memory and the PostgreSQL database in sync
• make the Job object a proper class and not a dict
• if possible, avoid keeping the database as a global variable

[mdonadoni]

After the recent performance profiling and improvements, I think this was done for performance reasons to avoid accessing the database to know the current status of jobs. Note that writes are always done also on the database, so there is no performance gain there