[Kolab-devel] 10.000 events in a Resource Calendar

Tue May 22 14:26:23 CEST 2012

On 2012-05-22 12:16, Martin Konold wrote:
> Am Dienstag, 22. Mai 2012, 11:55:35 schrieb Jeroen van Meeuwen:
>
> Hi Jeroen,
>

Hi Martin,

>> > - Freebusy only changes when new IMAP objects are created in a
>> > calendar (remember that there is no modify)
>>
>> This assumption is fairly flawed, since removing events from a 
>> calendar
>> does not add new IMAP objects but MUST update free/busy.
>
> No this is not a flaw in any way. A delete operation is handled 
> exactly like
> and together with write operations. (E.g. EVERY modify is actually a
> delete+write operation by nature of how Kolab storage works)
>

Let's take a step back, because we're confusing the issue in OP.

The following *actually* happens when an event is deleted (whether "the 
idea behind 0(1)" design or not);

- Adding or editing an event to a calendar obviously adds a new object 
to IMAP.

- To remove an event from a calendar, the message could be flagged 
\Deleted in IMAP, and (possibly) the folder is expunged (doesn't 
matter),

   - This is *not* a write operation that adds a new object to IMAP. It 
does bump UIDVALIDITY, but... see below.

- The *client* is to trigger the Free/Busy update,

- CONDSTORE (required for UIDVALIDITY) is not enabled on Kolab 2.3 
(Cyrus IMAP 2.3) mailboxes by default,

- The Free/Busy mechanism has little to hold on to, to see what has 
changed, unless it maintains a local cache of at least the UIDs of the 
message it used when it last generated the (partial) Free/Busy,

- Retrieval of relevant events to the relevant period in time could be 
made faster using sorting and retrieving the newest objects first,

- The client triggering Free/Busy does not simply HEAD a URL and 
disconnects, as this would impede the slice of time any web server code 
has available to do what it needs to do. Therefore, a client keeps open 
the connection (and uses GET/POST) until the web server performing the 
Free/Busy updating is done. This is considered a blocking operation for 
clients that cannot do this in the background.

- Obtaining 10.000 events from a single folder can (relatively easily) 
take > 30 seconds.

>> > - Kolab creates partial freebusy information whenever a new object 
>> is
>> > written to a calendar folder in IMAP
>>
>> Euh, as far as I know, it is the client software that triggers an
>> update of the free/busy, and not the Kolab server itself, and unless 
>> the
>> client is multi-threaded like Kontact it is also a blocking 
>> operation.
>
> Sorry this is non-sense.

Thank you for your balanced and well-formulated opinion.

> The Kolab client _triggers_ the creation of of the
> freebusy information. Such a trigger is trivially non-blocking and 
> the time
> required is totally independent of the folders size (perfect O(1).
> There is no need for multi-threading here.
>

As I've illustrated before, it's not like Kolab uses FPM or any other 
FastCGI-like implementation, and it's not like the client can simple 
HEAD a URI and be done with it (close the connection).

> The main point here is that the Client trigger the update of the 
> partial
> freebusy data but they never wait nor block. (A trigger is simply an
> http call which immediately returns and hints the server that the 
> freebusy needs to be
> updated)
>

It only that were true. It sounds very good in theory, but theory is a 
place up north in Narnia. In the real world, there is nothing that 
"hints" the server and nothing to follow up on such "hint". It is the 
client that is actively involved and waiting for the Free/Busy 
information to be updated as part of the trigger URI it is hitting.

>> > - When an invitation e.g. via iTip arives you simply need to check
>> > for an overlap with the freebusy list of the user involved 
>> (freebusy is the
>> > set union of the partial freebusy lists)
>>
>> See the former, ...
>>
>> While Free/Busy and the way that it is going to work is in flux, and
>> while Free/Busy will itself need to do somewhat the same "query" 
>> against
>> raw IMAP, I'm inclined to seek ways the querying itself can be 
>> improved.
>
> Sorry, but you really got things wrong. The basic idea behind Kolab
> is NOT to think in terms of a relational database including terms of 
> doing queries all
> the time.
>
> This is the essential clue behind Kolab that it is so extremly 
> scalable.
>
> Introducing all these "query" concepts will lead to loosing this 
> unique
> property.
>

Well, unique != good and most certainly unique != best. At most, unique 
<> common.

To be honest, the "extremely scalable" argument is starting to get to 
be completely wasted on me.

Every time it is used, it is used as the ultimate argument against 
something, but it misses merit in that the scalability parameter to a 
Kolab deployment is never removed nor reduced by any of the developments 
or ideas to move forward. While you may disagree with that, I have to 
conclude "no-SQL storage" is being confused and arbitrarily substituted 
with "caches, possibly in SQL".

> IMHO the Kolab 3.0 architectures runs in the wrong direction.
>

You are entitled to that opinion, of course, but I also appreciate it 
may be an opinion based on ideas and developments we discuss that need 
to be cleared up in more detail more so than being opposed.

That said, I appreciate your thoughts very much. I have nothing but the 
greatest respect for you, I hope you know that. I applaud your efforts 
holding on to the original Kolab ideas, and I will were I can.

Kind regards,

Jeroen van Meeuwen

-- 
Systems Architect, Kolab Systems AG

e: vanmeeuwen at kolabsys.com
m: +44 74 2516 3817
w: http://www.kolabsys.com

pgp: 9342 BF08