[Kolab-devel] Caching in Roundcube's libkolabxml storage layer

[Thomas Brüderli]

Hello devs

Since server side akonadi most likely won't make it into the Kolab 3.0 
release we have to build our own caching engine to improve access speed 
for Kolab objects in Roundcube. The initial plan was to delegate 
reading, caching and synching Kolab objects to a server side akonadi 
instance and keep things it rather simple on the Roundcube side. In 
previous versions we used the entire Horde stack to fetch, cache, read 
and write Kolab groupware objects with all it's known downsides. Now 
that reading and writing Kolab 3 objects using the libkolabxml PHP 
bindings basically works, it's time to get our heads into caching in 
order to speed up the listing of contacts, events and such.

Let me start with a short explanation of the new libkolabxml storage 
layer we've created in Roundcube. This is how one can list all contacts 
from an annotated IMAP folder:

   $folders = kolab_storage::get_folders('contact');
   $folder = $folders[0]; // instance of kolab_storage_folder class;

   foreach ($folder->get_objects() as $contact) {
     // $contact is a hash array containing contact properties
     // which are relevant in Roundcube
     echo $contact['uid'] . "\t" . $contact['name'] . "\n";
   }

Fetching a contact by its UID and updating it is pretty simple, too:

   $contact = $folder->get_object('<some-uid>');
   $contact['name'] = 'John Doe';
   $folder->save($contact);

Without a specific caching in place, the following procedure is executed 
when reading a contact:

kolab_storage::get_folders()
   (1) List all IMAP folders with their metadata and post-filter the list

kolab_storage_folder:get_objects()
   (2a) List all messages of a particular IMAP folder or
   (2b) search messages by HEADER X-Kolab-Type ...

For each of the returned messages or when calling 
kolab_storage_folder:get_object(<uid>) directly, the following happens:

   (3) Fetch bodystructure, parse it and find XML part.
   (4) Fetch the XML part from IMAP and parse it with libkolabxml
   (5) Load object data into a hash array by calling dozens of getter 
methods of the Contact/Event/DistributionList class.
   (6) Compute repeated instances of recurring events

While caching of (1), (2a) and (3) is already provided by the Roundcube 
core IMAP library (if enabled by config), we now have to implement a 
caching layer for (4) and (5) and optionally for (6). Also caching for 
(2x) could be replaced with a more kolab-specific cache which would 
allow one to select ranges of objects (e.g. events from..to) or even 
fulltext searches. I'd strongly recommend to also cache the interpreted 
object data in addition to the raw XML block because reading from 
libkolabxml involves plenty of function calls which are known to be 
rather expensive in PHP.

So here's my proposition for a Kolab object cache data structure:

FOLDER:	<fully qualified IAMAP folder URI>  // example: [1]
MSGUID:	<IMAP message UID>	// used for synchronizing
EXPIRE:	<date-time>		// expiration timestamp
UID:	<object UID>
TYPE:	<object type>		// contact/event/distribution-list/etc.
DATA:	<serialized object data>
XML:	<raw xml block>
DTSTART:	<date-time>	// event start (empty for other types)
DTEND:		<date-time>	// event end
TAGS:	<object specific keywords used for filtering>

[1] imap://bruederli%40kolab.cc@mail.kolab.cc/INBOX/Contacts

This structure can easily be reflected in a SQL database. If it's a 
requirement to have different caching backends such as db, memcache, 
file-based, etc. things become slightly more complicated but still 
doable. Is it?

Using full IMAP URIs would make it possible to re-use the cache of a 
shared folder for multiple users who have access to that folder. 
Together with the MSGUID the cache table can be synchronized with the 
message index consumed from either the IMAP server or from the Roundcube 
internal IMAP index cache.

I intend to implement the caching layer in a transparent way into 
kolab_storage and making it configurable, similar to the current IMAP 
cache implementation in Roundcube.

Any comments or suggestions are welcome!

Regards,
Thomas