[Kolab-devel] Z-push optimization

[Gunnar Wrobel]

Zitat von "Jeroen van Meeuwen (Kolab Systems)" <vanmeeuwen at kolabsys.com>:

> Alain Abbas wrote:
>> Hello
>>
>> i ve got a big problem :
>> Each time that a mobile synchronize i must read all the annotations of
>> all the folders to know if a folder is for sync or not
>> i ve got some customers with big trees ( arround 300 folders in Inbox)
>> become a bottleneck.
>>
>> 1) how to do not read at each time all the annotation folders ?
>> even if i cache some informations that become the hell
>> and i must know if the annotations are changed or not
>> to put or remove the folder to the sync
>>
>> I thought to make that ,to make a special flag in the INBOX annotation a
>> counter :
>> If the user modify the annotation of one folder this number could be
>> incremented
>>
>>
>> STEPS :
>> 1 when the user modify  an annotation  the  "ANNOVERSION"  is
>> incremented in the INBOX folder
>> Zpush would check this ANNOVERSION and compare with the old one
>> if ANNOVERSION is not equal Zpush read again the annotations
>>
>> Stay now the problem of the user space where to put this ANNOVERSION ?
>>
>> I think at this step we see the limitation for this method (annotations
>> per folder) and a simple
>> list in LDAP should be
>> more efficient and simple .
>>
>> I waiting your feedback and ideas to resolve this big performance issue
>> Alain
>>
>
> Hi Alain,
>
> I understand your problem is the iteration of (many) IMAP folders and the
> inherent cost of having to use the IMAP protocol to get to the data that you
> actually require.

Alain, is this indeed your problem? Meaning you are calling  
imap_get_annotation 300 times if there are 300 folders? As far as I  
know this problem is inherent to using the imap_get_annotation call  
from the patched Kolab PHP package.

On the IMAP protocol side there is no limitation to using a wildcard  
to retrieve all annotations of a specific type for 300 folders *at the  
same time within one IMAP call*

See http://tools.ietf.org/html/draft-daboo-imap-annotatemore-07#section-3.4.1

Example:

            C: a GETANNOTATION "INBOX.%" "/comment" "value.priv"
            S: * ANNOTATION "INBOX.1" "/comment" ("value.priv" "My  
comment for 1")
            S: * ANNOTATION "INBOX.2" "/comment" ("value.priv" "My  
comment for 2")
            S: * ANNOTATION "INBOX.3" "/comment" ("value.priv" "My  
comment for 3")
            S: a OK GETANNOTATION complete


I did test this once and if I remember correctly then the Kolab IMAP  
server supports this just fine but imap_get_annotation does not.

This is a significant problem for the Kolab webclient, too. I still  
need to resolve that in Horde4 but this is no big deal as Horde4 does  
not use the c-client based IMAP features of PHP anymore.

For your specific problem it might be an option to look into the  
imap_get_annotation patch, too. Or you'd switch to using another IMAP  
library but that might be a lot of work, too.

>
> I understand this to result in great performance penalties, and hence I was
> thinking the following scenario might make sense to pursue for the future;
>
> If Cyrus were enabled to keep it's mailboxes.db, annotations.db, seen.db, and
> possibly even it's .index, .cache and .header in a true SQL database backend,
> Kolab_Z-Push would be able to use Cyrus' SQL backend as if it were cache.
>
> I've got several threads going on this approach already, so please  
> let me know
> what you think of using Cyrus' "live" as your "cache".

But that would mean that Kolab-Zpush does open a standard IMAP  
connection to Cyrus and for reading annotations in an efficient way it  
takes a short "peek behind the scenes" misusing the fact that is  
installed on the same server as Cyrus imapd.

This sounds completely broken to me. You'd give a webapp access to the  
Cyrus IMAP internal databases. You'd enforce installation of  
Kolab_Zpush on the same server on which Cyrus IMAP is installed  
(unless you want to use networked SQL databases and this is a no-go,  
too). And you'd use two type of communication channels. No way :)

Cheers,

Gunnar

>
> Kind regards,
>
> --
> Jeroen van Meeuwen
> Senior Engineer, Kolab Systems AG
>
> e: vanmeeuwen at kolabsys.com
> t: +316 42 801 403
> w: http://www.kolabsys.com
>
> pgp: 9342 BF08
>
> _______________________________________________
> Kolab-devel mailing list
> Kolab-devel at kolab.org
> https://kolab.org/mailman/listinfo/kolab-devel
>



--
Gunnar Wrobel
Developer, Kolab Systems AG

e: wrobel at kolabsys.com
t: +49 700 6245 0000
w: http://www.kolabsys.com

pgp: 9703 43BE

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.