[Kolab-devel] redis vs. MongoDB for zpush kolab backend

Fri Jan 7 20:26:18 CET 2011

Zitat von Alain Abbas <alain.abbas at libertech.fr>:

> Gunnar Wrobel a écrit :
>> Hi Bogo,
>>
>> Zitat von Bogomil Shopov <shopov at kolabsys.com>:
>>
>>
>>> Hi all,
>>> I am actively testing MongoDB + PHP driver on my machine right now and I am
>>> getting better results than Redis.
>>>
>>
>> Sounds good.
>>
>>
>>> In fact MongoDB is what we need:
>>>
>>
>> I wonder if this is really so much a question of the backend
>> technology. I admit I don't have much experience with something like
>> MongoDB but my assumption is that there are different technologies
>> around that provide the same feature set. Do we really need to choose
>> a single one?
>>
>> I looked at the code you added in the "backend-mongodb" branch and so
>> far this is a key-value store. That is something you get from many
>> backend technologies and in fact the interface for such services is
>> near to trivial. It boils down to what you established in
>> mongoDBcache.php.
>>
>> "backend-redis" wasn't very different concerning the interface. So I
>> would suggest to complete the abstraction at that point and really
>> rely on a simple interface. In the long term you'll be able to switch
>> to any newer technology that suites your needs better than what might
>> perform best right now.
>>
>> Of course there are already interfaces like that around. Horde_Cache
>> is one of them (it abstracts about 10 key-value storage backend
>> types). And in fact I'd be very happy if we could keep such code
>> segments together.
>>
>> I much rather convert your code into modules to Horde_Cache and add
>> support for both Redis and MongoDB there (both are missing from
>> Horde_Cache at the moment).
>>
>> What is the advantage of keeping such code consolidated: We can use
>> the same caching backend for other applications as well. Did you know
>> that we use the db functions Alain used initially in the Kolab
>> free/busy code as well?
>>
>> You obviously found them to be too slow for our needs. If Redis and/or
>> MongoDB is a better solution I'd like to use it for other systems as
>> well.
>>
>> Does any of this make sense to you and would that be a direction you'd
>> be willing to go into?
>>
>>
> Yes we must have an abstraction to switch the technology . An Abstract
> class at the high level who provide
> method that we need.
>>> - We can make queries (it's usefull for Calendar)
>>> - We can create indexes on whatever we want
>>> - We can use authentication (salt+md5) (with Redis we don't (we should use
>>> "trusted environment"));
>>> - There are more than 10 language drivers
>>> (http://www.mongodb.org/display/DOCS/Drivers) - we can access the
>>> data in more
>>> than 10 languages
>>>
>>
>> I'm aware that many of these are features that are not part of the
>> simple key-value interface I mentioned above. But that doesn't change
>> the fact that the key-value aspect is part of what is required. All
>> other parts can be handled separately from that.
>>
> We have all the same problem with Imap storage, The Performance when
> Calendars (especialy calendar who move often) and
> we must have index not only key values to search the dates . Iterations
> are not good. Try in horde actualy to display 10 shared calendar who have
> each 1000 messages in the folder. This is impossible or take minutes and
> minutes. The deal i think is to have only one process or method to
> populate the cache and each can retrieve what he want quickly.
> a good scheme would be :
>
> Webmail ---------------->
> third party (eg : Caldav) ->    CACHE <-----> Imap
> zpush/backend  --------->
>
> if the webmail ask for an event for exemple why to read again the event
> (with zpush) for example ) if there are already in the cache ?

Of course. That has been one of the reason for having the  
Kolab_Storage module. I never got around to unify the cache for both  
free/busy and Horde but they use exactly the same technology and there  
is no reason not to merge them.

> for me the best solution is to have a GLOBAL namespace for all the datas
> (contacts, calendar ...)
> and a private space for the application ( for example i need to store
> state of the sync for zpush but thos datas are not instesting for
> anothers application.

Of course, I totally agree. I do think we need to discuss exactly this  
point with a lot more detail. But at the moment my comment was only  
oriented towards the key-value based storage. I understand that this  
might not be a sufficient storage model for all our needs. But as long  
as that is all we are doing I would still like to keep it clean.

Cheers,

Gunnar

>
> Regards
> Alain
>> Cheers,
>>
>> Gunnar
>>
>>
>>> I have tried xCache (lighttpd project) under PHP, but it does  
>>> almost the same
>>> like current DB and it's not so good for real chaching, but for opcode
>>> optimization.
>>>
>>>
>>>
>>> --
>>> Bogomil "Bogo" Shopov
>>> Senior Web Engineer
>>>
>>>
>>> Kolab Systems AG
>>> Zürich, Switzerland
>>>
>>> e: shopov at kolabsys.com
>>> t: +41 43 5016691 (ext. 2115 649)
>>> w: http://kolabsys.com
>>> b:http://talkweb.eu
>>>
>>> pgp: E69A226A Bogomil "Bogo" Shopov
>>>
>>> _______________________________________________
>>> Kolab-devel mailing list
>>> Kolab-devel at kolab.org
>>> https://kolab.org/mailman/listinfo/kolab-devel
>>>
>>>
>>
>>
>>
>> --
>> Gunnar Wrobel
>> Developer, Kolab Systems AG
>>
>> e: wrobel at kolabsys.com
>> t: +49 700 6245 0000
>> w: http://www.kolabsys.com
>>
>> pgp: 9703 43BE
>>
>> ----------------------------------------------------------------
>> This message was sent using IMP, the Internet Messaging Program.
>>
>> _______________________________________________
>> Kolab-devel mailing list
>> Kolab-devel at kolab.org
>> https://kolab.org/mailman/listinfo/kolab-devel
>>
>
> _______________________________________________
> Kolab-devel mailing list
> Kolab-devel at kolab.org
> https://kolab.org/mailman/listinfo/kolab-devel
>

--
Gunnar Wrobel
Developer, Kolab Systems AG

e: wrobel at kolabsys.com
t: +49 700 6245 0000
w: http://www.kolabsys.com

pgp: 9703 43BE

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.