<html><head><meta name="qrichtext" content="1" /></head><body style="font-size:10pt;font-family:Verdana">
<p>Am Mittwoch, 12. Januar 2005 01:17 schrieb Helge Hess:</p>
<p></p>
<p>Hi Helge,</p>
<p></p>
<p></p>
<p>may I suggest to move the discussion about storage formats to </p>
<p></p>
<p>https://kolab.org/mailman/listinfo/kolab-format</p>
<p></p>
<p>> > How do you intend to keep coherency with etags?</p>
<p></p>
<p>> same terminology. What is the situation you want to avoid?</p>
<p>></p>
<p>> Etags (in combination with HTTP if-match and if-none-match headers)</p>
<p>> ensure that an object is only updated in case it wasn't modified in the</p>
<p>> meantime (eg by a different client/user)</p>
<p></p>
<p>How do you know that the object has not been modified in meantime without </p>
<p>locks/being online?</p>
<p></p>
<p>> "Intended" and "cannot" bite anyway, since there are no CalDAV clients</p>
<p>> yet ...</p>
<p></p>
<p>Which leads to the question why you claim that the working Kolab specification </p>
<p>is a NIH phenomena....Actually I claim that the CalDAV people should closely </p>
<p>look at how Kolab solved the issue and maybe adopt some of the semantics, </p>
<p>formats etc.</p>
<p></p>
<p>> > The caching for HTTP is to my knowledge intended for static data.</p>
<p>></p>
<p>> I don't know what you consider static data. HTTP caches are kept</p>
<p>> consistent by querying and comparing just the etag from the server</p>
<p>> prior delivery.</p>
<p></p>
<p>A typical calendar event is a _very_ small amount of data. So the overhead to </p>
<p>query the original server before fetching the actual data from a proxy can </p>
<p>easily lead to a lower speed.</p>
<p></p>
<p>> This allows for excellent scalability and speedup, </p>
<p>> especially when using the cache server in front of a processing intense</p>
<p>> backend implementation (like the regular OGo server, sigh ;-).</p>
<p></p>
<p>Actually this is what I mostly don't like about OGo. It is very fat and needs </p>
<p>enormous processing power compared to Kolab. </p>
<p></p>
<p>The http cache is supposed to hide this problem while imho it cannot really </p>
<p>fix it because you want to keep coherency.</p>
<p></p>
<p>> Of course the "ratio" between read/write is different for CalDAV than</p>
<p>> for some HTML pages, but the mechanism applies.</p>
<p></p>
<p>With calendaring and caching/disconnected Kolab clients the typical R/W ration </p>
<p>on the server is 1:1.</p>
<p></p>
<p>> > Why? Only modifications to individual events need to be regularly</p>
<p>> > fetched.</p>
<p>></p>
<p>> IMAP4 doesn't even allow modifications of events, you should know that</p>
<p></p>
<p>This is a _big_ advantage of IMAP4. "Modification" of events is modelled by a </p>
<p>appending a new object followed by a delete of the old object in the store. </p>
<p></p>
<p>This model allows for _extreme_ lock free performance. </p>
<p></p>
<p>Even with relational databases plain appends are much faster than </p>
<p>modifications of rows.</p>
<p></p>
<p>> a) DASL query on the current state of etags, either all or related to</p>
<p>> some tombstone</p>
<p>> b) retrieval of all new or changed objects</p>
<p></p>
<p>In order to retrieve all new messages I can blindly ask the server to send me </p>
<p>all new stuff with</p>
<p></p>
<p>1 C FETCH <lastUID>:*</p>
<p></p>
<p>So the new messages get immediately delivered to the client. The server can </p>
<p>decide in which order to send messages e.g. allowing the server to first send </p>
<p>small messages and later send the bigger messages.</p>
<p></p>
<p>While the new messages arrive the IMAP4 clients asks in between for an update </p>
<p>of the IMAP FLAGS.</p>
<p></p>
<p>> - IMAP4 is not stateless, high traffic sides need to have the sockets</p>
<p>> open for much</p>
<p>> longer (and you can't have much more than a few thousands open</p>
<p>> sockets on a single</p>
<p>> system)</p>
<p></p>
<p>If the number of sockets is too high the server may simply close some idle </p>
<p>connections. (I think http servers do the same)</p>
<p></p>
<p>> - since you can't keep the connection open at this scale, it requires a</p>
<p>> lot more</p>
<p>> TCP/IP roundtrips - potentially a login, select, retrieve, etc for</p>
<p>> each object</p>
<p></p>
<p>This is the very same with HTTP. For a scalable server doing this for each </p>
<p>object does not solve the issue but is counter productive.</p>
<p></p>
<p>> - IMAP4 is harder to cluster (yes, you can do that, but its way more</p>
<p>> complicated than</p>
<p>> with just putting a Squid in front)</p>
<p></p>
<p>Squid does not help with clustering at all. Actually finally the load remains </p>
<p>on the backend server.</p>
<p></p>
<p>Actually with Kolab 2 we already have full support for multi-location setups. </p>
<p>Nobody is preventing to use the multi-location feature to simply scale the </p>
<p>performance in a single location.</p>
<p></p>
<p>Actually the _big_ advantage of the Kolab multi-location support is that you </p>
<p>don't need that single big backend server.</p>
<p></p>
<p>Last but not least the Kolab architecture actually also allows for IMAP </p>
<p>caches. Technically the disconnected IMAP we use with Kolab is nothing else </p>
<p>than a persistent cache on the client side.</p>
<p></p>
<p>If the need would really arise (I doubt that this happens anytime soon) we can </p>
<p>easily implement an IMAP cache which is then the equivalent to Squid.</p>
<p></p>
<p>> Not that this is a proof at all ;-), but Microsoft is even using HTTP</p>
<p>> for Outlook access to HotMail.</p>
<p></p>
<p>IMHO this is no proof at all.</p>
<p></p>
<p>> Well, thats more an issue of the storage, not of the protocol (which is</p>
<p>> fixed to Cyrus in Kolab, but open in CalDAV). I suppose you can hold</p>
<p>> tens of thousands of items in a single Cyrus which should be enough for</p>
<p>> almost any application you can imagine.</p>
<p></p>
<p>My test server has about 1.000.000 objects in a single partition. I cannote see any degradition sofar.</p>
<p></p>
<p>> The issue is access scalability.</p>
<p></p>
<p>IMAP is scalable. With the Kolab approach (using disconnected IMAP) it is trivial to even add IMAP accelleration caches to the game. (Though I think this will only be useful with much more than 10.000 concurrent users.</p>
<p></p>
<p>> Yet we don't need to finish that discussion since it has little</p>
<p>> practical relevance at all even for the largest agency or enterprise</p>
<p>> setups. And for enduser portals, Yahoo and Web.de are supposed to make</p>
<p>> their minds up on their own ;-)</p>
<p></p>
<p>Well, Kolab could in a future version (maybe Kolab 3) offer many features which are interesting for businesses doing ASP.</p>
<p></p>
<p>> OK, so we are in line here. Thats what pipelining means in the HTTP</p>
<p>> context and is a standard HTTP/1.1 feature.</p>
<p></p>
<p>No. HTTP/1.1 requires replies to be send in order. IMAP on the other hand easily allows for reordering which can help to increased the percieved speed (latency) a lot. E.g when retrieving mails over a small bandwidth link. The server may reorder the sending of messages so that it firstly sends the smaller messages.</p>
<p></p>
<p>> > Kolab the server tells the client that the calendar changed.</p>
<p>></p>
<p>> No, that just means that HTTP doesn't provide the functionality and</p>
<p>> that you need a different protocol to gain that functionality. While</p>
<p>> this isn't included in HTTP, it *is* intended for CalDAV, and thats</p>
<p>> what we are comparing ;-)</p>
<p>> Summary: this means a CalDAV client must *not* poll the server.</p>
<p></p>
<p>Sorry, this ist not about facts but about intentions....</p>
<p></p>
<p>> > Jabber is very inefficient and expensive across slow links e.g. GPRS.</p>
<p>></p>
<p>> Jabber is used in huge setups (much larger than Cyrus deployments) and</p>
<p>> scales extremely well - this is a proven fact you can hardly attack.</p>
<p></p>
<p>You are not replying to my cllaim. I say that Jabber is very inefficient across slow links. I was not saying anything about scalabilty.</p>
<p></p>
<p>> GPRS is ~56K? So you can send out 165 notifications in a second.</p>
<p></p>
<p>GPRS is mainly charged on the traffic not on time. (Typically about 9¢ per 10KB). So a factor of 10 indeed does matter monetary wise!</p>
<p></p>
<p>> Which makes me wonder why you don't support this in practice and force</p>
<p>> Kolab1 users to upgrade to Kolab2.</p>
<p></p>
<p>Technically the Proko2 client is able to handle both formats. But for practical reasons we only support the more recent XML Format with Kolab2.</p>
<p></p>
<p>> >> You need to compare "Kolab/XML" with "iCalendar" and if you do, there</p>
<p>> >> is no advantage here for Kolab/XML.</p>
<p></p>
<p>Look, there are for example many things you can express with valid iCalendar which cannot be reasonably converted to the internal Outlook format. --> this leads to massive real world interoperability issues.</p>
<p></p>
<p>> > It is unfortunately incomplete for our purposes. (Mainly OL stuff</p>
<p>> > missing)</p>
<p>></p>
<p>> Just out of interest, could you give some examples on Outlook features</p>
<p>> which you support in Kolab/XML are missing in xCal?</p>
<p></p>
<p>By definition xCal is an XML representation of iCalendar. To my knowledge xCal has not gained much support and the draft is already expired.</p>
<p></p>
<p>Regards,</p>
<p>-- martin</p>
<p></p>
<p>-- </p>
<p>"I am committed to helping Ohio deliver its electoral votes to the</p>
<p>President next year." -- 2004, Wally O'Dell - CEO of Diebold, Inc. </p>
<p>e r f r a k o n - Stuttgart, Germany</p>
<p>Erlewein, Frank, Konold & Partner - Beratende Ingenieure und Physiker</p>
<p></p>
<p></p>
</body></html>