From vanmeeuwen at kolabsys.com Mon Aug 1 16:38:32 2011 From: vanmeeuwen at kolabsys.com (Jeroen van Meeuwen (Kolab Systems)) Date: Mon, 1 Aug 2011 10:38:32 -0400 Subject: [KEP UPDATE] KEP #9: Storage of configuration and application control information In-Reply-To: <2515704.Z9MGcjsfzr@katana.lair> References: <1727968.ZERSqOCkgA@katana.lair> <201107061240.43506.vanmeeuwen@kolabsys.com> <2515704.Z9MGcjsfzr@katana.lair> Message-ID: <201108011038.34047.vanmeeuwen@kolabsys.com> Georg C. F. Greve wrote: > Hi all, > > Based on the very useful feedback by Jeroen and Alain, I have revised the > draft of KEP 9 to go more into the direction of clarifying the various > mechanisms of configuration. As both Jeroen and Alain pointed out, it is > practically impossible to think of one without thinking of the other. > > So this KEP now gives a set of tools, and the individual configuration > items and use cases will in their description need to define their optimal > mix and usage of these mechanisms. > > The updated draft is now online at > > http://wiki.kolab.org/User:Greve/Drafts/KEP:9 > Hi Georg, overall, looks good. While it says ANNOTATEMORE though, in the abstract, it continues with METADATA (and rightfully so). In the public vs. private annotations[1], I think a value.shared / value.priv both are perfectly feasible for a 'folder-config' type of annotation, for example. [1] http://wiki.kolab.org/User:Greve/Drafts/KEP:9#Public_and_private_annotations I think perhaps a better clause here is to outline that it is forbidden unless specifically stated in the accepted KEP for such annotation...? In the parsing order of folder-based XML, how do we outline whether any annotations found or the XML contents of the folder should prevail? I think it is useful to outline that the settings that are stored in annotations are *exclusive* with what may occur in configuration XML. Then again, I haven't given it a full load of thought yet. Kind regards, Jeroen van Meeuwen -- Senior Engineer, Kolab Systems AG e: vanmeeuwen at kolabsys.com t: +44 144 340 9500 m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08 From florian.samson at bsi.bund.de Thu Aug 11 16:17:54 2011 From: florian.samson at bsi.bund.de (Florian v. Samson) Date: Thu, 11 Aug 2011 16:17:54 +0200 Subject: [KEP UPDATE] KEP #9: Storage of configuration and application control information In-Reply-To: <2515704.Z9MGcjsfzr@katana.lair> References: <1727968.ZERSqOCkgA@katana.lair> <201107061240.43506.vanmeeuwen@kolabsys.com> <2515704.Z9MGcjsfzr@katana.lair> Message-ID: <201108111617.54862.florian.samson@bsi.bund.de> Hi Georg, Am Freitag, 29. Juli 2011 um 15:50:01 schrieb Georg C. F. Greve: > > ... > > The updated draft is now online at > > http://wiki.kolab.org/User:Greve/Drafts/KEP:9 > > and is awaiting further comments. 1. "From its inception in 2003 and up until and including Kolab Format 2.0, the Kolab Groupware Solution has been using the ANNOTATEMORE proposal which has been finalized in February 2009 into ..." 1a. Link is syntactically incorrect (No Wiki-syntax, but HTML-syntax). 1b. Statement is incorrect by content: It was primarily draft-5 which was implemented. 1c. tools.ietf.org provides better views of RFC (-drafts) than the chosen one. Suggestion: Change the Link to and its text to "Annotatemore-05 RFC-draft". > As there is a bit of text that has been > touch, I expect some editorial as well as conceptual thinking to be > required to bring this into final shape. I feel that the next sentence "The Kolab Format that results from this changeset against the Kolab Format 2.0 will consider usage of the ANNOTATEMORE proposal deprecated and specify RFC 4564 only." is phrased a bit too softly, as IIRC we all agreed, that this must result in a normative "Kolab-clients and -servers MUST adhere to RFC4564" in the Kolab Format specification v2.0 and later. "will consider only", rather sounds to me as if this will be just the only solution depicted there. Clarifications: a. General: "Kolab XML version 1.0", "XML version 1.1", "the object version ... at 1.1", "Kolab Format 2.0" all do mean "Kolab Format specification version x.y", right? a.I. If so, please call them consistently "Kolab Format specification version x.y" only, in order to cease the ongoing confusion (for years, already). When talking about Kolab-objects specifically, please call them "a Kolab Object complying with the Kolab Format specification version x.y", or "according to", "adhering", etc. a.II. If not, then the confusion is larger than I can grasp. Consequently this wording-mishmash would be in dire need of more clearing-up and clarification, then. b. "There MUST NEVER be more than one folder per user with the subtype 'default';" is under-determined IMHO. By context one can conclude that shall state (I tried to improve wording as well): "There MUST be at most a single folder of type 'configuration' with the subtype 'default' per user;" c. "The subtype of the configuration folder-type for shared folders MUST NEVER be set to 'default';" -> (Improve wording, pull the most important noun in front) "Shared folders of the type 'configuration' MUST NEVER carry the subtype 'default';" HTH & cheers Florian From vanmeeuwen at kolabsys.com Thu Aug 11 19:58:13 2011 From: vanmeeuwen at kolabsys.com (Jeroen van Meeuwen (Kolab Systems)) Date: Thu, 11 Aug 2011 13:58:13 -0400 Subject: [KEP UPDATE] KEP #9: Storage of configuration and application control information In-Reply-To: <201108111617.54862.florian.samson@bsi.bund.de> References: <1727968.ZERSqOCkgA@katana.lair> <2515704.Z9MGcjsfzr@katana.lair> <201108111617.54862.florian.samson@bsi.bund.de> Message-ID: <201108111358.14991.vanmeeuwen@kolabsys.com> Florian v. Samson wrote: > Hi Georg, > Hi Florian, Georg, > b. "There MUST NEVER be more than one folder per user with the > subtype 'default';" is under-determined IMHO. By context one can conclude > that shall state (I tried to improve wording as well): > "There MUST be at most a single folder of type 'configuration' with the > subtype 'default' per user;" > > > c. "The subtype of the configuration folder-type for shared folders MUST > NEVER be set to 'default';" -> (Improve wording, pull the most important > noun in front) > "Shared folders of the type 'configuration' MUST NEVER carry the > subtype 'default';" > In the realm of nitpicking, I know, but MUST NEVER is not RFC 2119 compliant. MUST NOT and SHALL NOT are to be used to indicate "that the definition is an absolute prohibition of the specification". Kind regards, Jeroen van Meeuwen -- Senior Engineer, Kolab Systems AG e: vanmeeuwen at kolabsys.com t: +44 144 340 9500 m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08 From florian.samson at bsi.bund.de Fri Aug 12 09:04:12 2011 From: florian.samson at bsi.bund.de (Florian v. Samson) Date: Fri, 12 Aug 2011 09:04:12 +0200 Subject: FYI: The Robustness Principle Reconsidered (CACM) Message-ID: <201108120904.12524.florian.samson@bsi.bund.de> Hello all, an interesting, short note on a meta-principle, which has some relevance for design decisions WRT Kolab Format specification v2. The Robustness Principle Reconsidered (CACM) Cheers Florian From greve at kolabsys.com Thu Aug 18 07:41:31 2011 From: greve at kolabsys.com (Georg C. F. Greve) Date: Thu, 18 Aug 2011 07:41:31 +0200 Subject: [GOING TO CLOSE] KEP 8: Priority for events, matching iCalendar priority range In-Reply-To: <201107291542.26124.vanmeeuwen@kolabsys.com> References: <93713080.Stszm7CVXk@katana.lair> <201107291542.26124.vanmeeuwen@kolabsys.com> Message-ID: <2170547.02mu3MvOHe@katana.lair> On Friday 29 July 2011 15.42:25 Jeroen van Meeuwen wrote: > That's about all I have Excellent. Seems noone else had anything else, as well. So I applied your small language fix, latest version now at http://wiki.kolab.org/User:Greve/Drafts/KEP:8 and will ask to issue the closing call. Best regards, Georg -- Georg C. F. Greve Chief Executive Officer Kolab Systems AG Z?rich, Switzerland e: greve at kolabsys.com t: +41 78 904 43 33 w: http://kolabsys.com pgp: 86574ACA Georg C. F. Greve -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 308 bytes Desc: This is a digitally signed message part. URL: From greve at kolabsys.com Thu Aug 18 08:11:41 2011 From: greve at kolabsys.com (Georg C. F. Greve) Date: Thu, 18 Aug 2011 08:11:41 +0200 Subject: [KEP UPDATE] KEP #9: Storage of configuration and application control information In-Reply-To: <201108011038.34047.vanmeeuwen@kolabsys.com> References: <1727968.ZERSqOCkgA@katana.lair> <2515704.Z9MGcjsfzr@katana.lair> <201108011038.34047.vanmeeuwen@kolabsys.com> Message-ID: <2877318.xOX1Zk38Iz@katana.lair> On Monday 01 August 2011 10.38:32 Jeroen van Meeuwen wrote: > overall, looks good. While it says ANNOTATEMORE though, in the abstract, it > continues with METADATA (and rightfully so). Thanks, fixed. > In the public vs. private annotations[1], I think a value.shared / > value.priv both are perfectly feasible for a 'folder-config' type of > annotation, for example. Ok, rephrased. Let me know whether this is better for you. Best regards, Georg -- Georg C. F. Greve Chief Executive Officer Kolab Systems AG Z?rich, Switzerland e: greve at kolabsys.com t: +41 78 904 43 33 w: http://kolabsys.com pgp: 86574ACA Georg C. F. Greve -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 308 bytes Desc: This is a digitally signed message part. URL: From greve at kolabsys.com Thu Aug 18 08:16:03 2011 From: greve at kolabsys.com (Georg C. F. Greve) Date: Thu, 18 Aug 2011 08:16:03 +0200 Subject: [KEP UPDATE] KEP #9: Storage of configuration and application control information In-Reply-To: <201108111617.54862.florian.samson@bsi.bund.de> References: <1727968.ZERSqOCkgA@katana.lair> <2515704.Z9MGcjsfzr@katana.lair> <201108111617.54862.florian.samson@bsi.bund.de> Message-ID: <3478074.M88JnKY8F6@katana.lair> Hi Florian, On Thursday 11 August 2011 16.17:54 Florian v. Samson wrote: > 1c. tools.ietf.org provides better views of RFC (-drafts) than the chosen > one. Very true. This is primarily an issue of the KEP template in the Wiki, though, so for Jeroen to fix. I think I asked him to change the link for the template some time ago, but he probably did not manage to find the time. > Suggestion: > Change the Link to > and > its text to "Annotatemore-05 RFC-draft". Clarified and modified as proposed. > I feel that the next sentence "The Kolab Format that results from this > changeset against the Kolab Format 2.0 will consider usage of the > ANNOTATEMORE proposal deprecated and specify RFC 4564 only." is phrased a > bit too softly, as IIRC we all agreed, that this must result in a > normative "Kolab-clients and -servers MUST adhere to RFC4564" in the Kolab > Format specification v2.0 and later. "will consider only", rather sounds > to me as if this will be just the only solution depicted there. Agreed. Made it stronger in language. > a. General: "Kolab XML version 1.0", "XML version 1.1", "the object > version ... at 1.1", "Kolab Format 2.0" all do mean "Kolab Format > specification version x.y", right? Yes. This is precisely the language that Bernhard and yourself agreed to in the discussions around KEP 2, so I reused the same naming conventions, so Kolab Format for the overall specification, and Kolab XML version for the version number of the XML objects, as the two have unfortunately been allowed to diverge, so now we have a little bit of inherent confusion in there. > b. "There MUST NEVER be more than one folder per user with the > subtype 'default';" is under-determined IMHO. By context one can conclude > that shall state (I tried to improve wording as well): > "There MUST be at most a single folder of type 'configuration' with the > subtype 'default' per user;" Agreed & applied. > c. "The subtype of the configuration folder-type for shared folders MUST > NEVER be set to 'default';" -> (Improve wording, pull the most important > noun in front) > "Shared folders of the type 'configuration' MUST NEVER carry the > subtype 'default';" Agreed & applied with Jeroens improvement. Thanks for the review and input! Best regards, Georg -- Georg C. F. Greve Chief Executive Officer Kolab Systems AG Z?rich, Switzerland e: greve at kolabsys.com t: +41 78 904 43 33 w: http://kolabsys.com pgp: 86574ACA Georg C. F. Greve -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 308 bytes Desc: This is a digitally signed message part. URL: From greve at kolabsys.com Thu Aug 18 08:18:58 2011 From: greve at kolabsys.com (Georg C. F. Greve) Date: Thu, 18 Aug 2011 08:18:58 +0200 Subject: [NEARING CLOSE] KEP #9: Storage of configuration and application control information Message-ID: <2864793.QO0DNb7T7G@katana.lair> Dear all, Thanks to the review and input from both Jeroen and Florian, KEP #9 has now been revised and updated at http://wiki.kolab.org/User:Greve/Drafts/KEP:9 Other than that there has been very little input recently, which means this KEP is also nearing closing time. So if you haven't yet reviewed it and feel that you absolutely must do so, please do so *now* or forever hold your peace. Best regards, Georg -- Georg C. F. Greve Chief Executive Officer Kolab Systems AG Z?rich, Switzerland e: greve at kolabsys.com t: +41 78 904 43 33 w: http://kolabsys.com pgp: 86574ACA Georg C. F. Greve -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 308 bytes Desc: This is a digitally signed message part. URL: From vanmeeuwen at kolabsys.com Wed Aug 24 10:00:24 2011 From: vanmeeuwen at kolabsys.com (Jeroen van Meeuwen (Kolab Systems)) Date: Wed, 24 Aug 2011 09:00:24 +0100 Subject: Closing Call: KEP #9 Message-ID: <201108240900.26528.vanmeeuwen@kolabsys.com> Hello, I would like to announce the closing call stage for KEP #9, a design KEP on storage of configuration and application control information; http://wiki.kolab.org/index.php?title=User:Greve/Drafts/KEP:9&oldid=11706 This KEP was proposed July 5th, 2011, and has gone without any comment or feedback for the last month. With the closing call, we now enter a two week period for final objections. This means that unless serious objections are raised, KEP #9 defaults to being accepted on Tuesday, September 7th, 2011. Thank you for your continued support! Kind regards, Jeroen van Meeuwen -- Senior Engineer, Kolab Systems AG e: vanmeeuwen at kolabsys.com t: +44 144 340 9500 m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: From vanmeeuwen at kolabsys.com Wed Aug 24 10:05:24 2011 From: vanmeeuwen at kolabsys.com (Jeroen van Meeuwen (Kolab Systems)) Date: Wed, 24 Aug 2011 09:05:24 +0100 Subject: Closing Call: KEP #8 Message-ID: <201108240905.25457.vanmeeuwen@kolabsys.com> Hello, I would like to announce the closing call stage for KEP #8, a design KEP on Priority for events, matching iCalendar's priority range; http://wiki.kolab.org/index.php?title=User:Greve/Drafts/KEP:8&oldid=11701 This KEP was proposed June 25th, 2011, and has gone without any comment or feedback for the last month. With the closing call, we now enter a two week period for final objections. This means that unless serious objections are raised, KEP #8 defaults to being accepted on Tuesday, September 7th, 2011. Thank you for your continued support! Kind regards, Jeroen van Meeuwen -- Senior Engineer, Kolab Systems AG e: vanmeeuwen at kolabsys.com t: +44 144 340 9500 m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08 -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanmeeuwen at kolabsys.com Thu Aug 25 12:56:23 2011 From: vanmeeuwen at kolabsys.com (Jeroen van Meeuwen (Kolab Systems)) Date: Thu, 25 Aug 2011 11:56:23 +0100 Subject: Closing Call: KEP #9 In-Reply-To: <201108240900.26528.vanmeeuwen@kolabsys.com> References: <201108240900.26528.vanmeeuwen@kolabsys.com> Message-ID: <201108251156.24883.vanmeeuwen@kolabsys.com> Jeroen van Meeuwen (Kolab Systems) wrote: > Hello, > > I would like to announce the closing call stage for KEP #9, a design KEP on > storage of configuration and application control information; > > http://wiki.kolab.org/index.php?title=User:Greve/Drafts/KEP:9&oldid=11706 > Hi Georg, small correction: ANNOTATE is RFC 5257, not 5267. Kind regards, Jeroen van Meeuwen -- Senior Engineer, Kolab Systems AG e: vanmeeuwen at kolabsys.com t: +44 144 340 9500 m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08 -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanmeeuwen at kolabsys.com Thu Aug 25 13:03:29 2011 From: vanmeeuwen at kolabsys.com (Jeroen van Meeuwen (Kolab Systems)) Date: Thu, 25 Aug 2011 12:03:29 +0100 Subject: Closing Call: KEP #9 In-Reply-To: <201108240900.26528.vanmeeuwen@kolabsys.com> References: <201108240900.26528.vanmeeuwen@kolabsys.com> Message-ID: <201108251203.30792.vanmeeuwen@kolabsys.com> Jeroen van Meeuwen (Kolab Systems) wrote: > Hello, > > I would like to announce the closing call stage for KEP #9, a design KEP on > storage of configuration and application control information; > > http://wiki.kolab.org/index.php?title=User:Greve/Drafts/KEP:9&oldid=11706 > > This KEP was proposed July 5th, 2011, and has gone without any comment or > feedback for the last month. > Language tweak, not sure if it makes the statement more clear but wanted to suggest it anyways: (Both kinds MAY be used (...)) """but each annotations usage in all namespaces""" --> """but the use of an annotation""" (MUST be defined) """for all prefixes""" (in the KEP for said annotation, to avoid confusion and/or conflict) Note how METADATA uses /private and /shared prefixes as opposed to 'value.priv' and 'value.shared' command parameters. Kind regards, Jeroen van Meeuwen -- Senior Engineer, Kolab Systems AG e: vanmeeuwen at kolabsys.com t: +44 144 340 9500 m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08 -------------- next part -------------- An HTML attachment was scrubbed... URL: From greve at kolabsys.com Thu Aug 25 13:26:37 2011 From: greve at kolabsys.com (Georg C. F. Greve) Date: Thu, 25 Aug 2011 13:26:37 +0200 Subject: Closing Call: KEP #9 In-Reply-To: <201108251156.24883.vanmeeuwen@kolabsys.com> References: <201108240900.26528.vanmeeuwen@kolabsys.com> <201108251156.24883.vanmeeuwen@kolabsys.com> Message-ID: <73033785.fkgc5tK3vR@katana.lair> On Thursday 25 August 2011 11.56:23 Jeroen van Meeuwen wrote: > small correction: > ANNOTATE is RFC 5257, not 5267. Good catch! Although actually 5267 existed nowhere in the document. But 5256 did. Damn them RFC numbers. ;) Fixed in latest revision: http://wiki.kolab.org/index.php?title=User:Greve/Drafts/KEP:9&oldid=11727 Best regards, Georg P.S. Cross-posting for such corrections in KEPs is okay to announce minor fixes in the final approval process for me, but the report should just go to the authoritative list, IMHO to avoid cluttering all lists. And I don't think any of this should go to kep-announce, which should be purely for announcements. -- Georg C. F. Greve Chief Executive Officer Kolab Systems AG Z?rich, Switzerland e: greve at kolabsys.com t: +41 78 904 43 33 w: http://kolabsys.com pgp: 86574ACA Georg C. F. Greve -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 308 bytes Desc: This is a digitally signed message part. URL: From greve at kolabsys.com Thu Aug 25 13:27:13 2011 From: greve at kolabsys.com (Georg C. F. Greve) Date: Thu, 25 Aug 2011 13:27:13 +0200 Subject: Closing Call: KEP #9 In-Reply-To: <201108251203.30792.vanmeeuwen@kolabsys.com> References: <201108240900.26528.vanmeeuwen@kolabsys.com> <201108251203.30792.vanmeeuwen@kolabsys.com> Message-ID: <2269162.mbymxj6G0q@katana.lair> On Thursday 25 August 2011 12.03:29 Jeroen van Meeuwen wrote: > Note how METADATA uses /private and /shared prefixes as opposed to > 'value.priv' and 'value.shared' command parameters. ACK. Incorporated in latest revision: http://wiki.kolab.org/index.php?title=User:Greve/Drafts/KEP:9&oldid=11727 Best regards, Georg -- Georg C. F. Greve Chief Executive Officer Kolab Systems AG Z?rich, Switzerland e: greve at kolabsys.com t: +41 78 904 43 33 w: http://kolabsys.com pgp: 86574ACA Georg C. F. Greve -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 308 bytes Desc: This is a digitally signed message part. URL: From vanmeeuwen at kolabsys.com Thu Aug 25 13:32:44 2011 From: vanmeeuwen at kolabsys.com (Jeroen van Meeuwen (Kolab Systems)) Date: Thu, 25 Aug 2011 12:32:44 +0100 Subject: [KEP UPDATE] KEP #9: Storage of configuration and application control information In-Reply-To: <3478074.M88JnKY8F6@katana.lair> References: <1727968.ZERSqOCkgA@katana.lair> <201108111617.54862.florian.samson@bsi.bund.de> <3478074.M88JnKY8F6@katana.lair> Message-ID: <201108251232.45960.vanmeeuwen@kolabsys.com> Georg C. F. Greve wrote: > Hi Florian, > > On Thursday 11 August 2011 16.17:54 Florian v. Samson wrote: > > 1c. tools.ietf.org provides better views of RFC (-drafts) than the chosen > > one. > > Very true. This is primarily an issue of the KEP template in the Wiki, > though, so for Jeroen to fix. I think I asked him to change the link for > the template some time ago, but he probably did not manage to find the > time. > I think I've only templated the reference to RFCs: http://wiki.kolab.org/KEP:4#References_to_RFCs which link to http://tools.ietf.org/html/rfc, list the RFC in the references, for a nice and good printable KEP. Kind regards, Jeroen van Meeuwen -- Senior Engineer, Kolab Systems AG e: vanmeeuwen at kolabsys.com t: +44 144 340 9500 m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08 -------------- next part -------------- An HTML attachment was scrubbed... URL: From greve at kolabsys.com Fri Aug 26 17:28:10 2011 From: greve at kolabsys.com (Georg C. F. Greve) Date: Fri, 26 Aug 2011 17:28:10 +0200 Subject: NEW KEP: Color configuration storage for resources and categories (KEP #12) Message-ID: <3450478.OSfiVuoptM@katana.lair> Dear all, A new KEP has just been released into the public drafting & review process: KEP #12: Color configuration storage for resources and categories http://wiki.kolab.org/User:Greve/Drafts/KEP:12 The purpose of this KEP is to allow clients to consistently use the same colors for resources such as calendars, address books and so on as well as categories. The KEP has been based upon input collected since May onwards and is therefore already fairly substantial. There are however some judgement calls in particular around the category XML object which this KEP defines, and your input would be very much welcome. As this is a design KEP for the format, all comments, feedback, discussion should take place on kolab-format at kolab.org It is strongly recommended that ALL CLIENT AUTHORS have a look at this. With best regards, Georg Greve -- Georg C. F. Greve Chief Executive Officer Kolab Systems AG Z?rich, Switzerland e: greve at kolabsys.com t: +41 78 904 43 33 w: http://kolabsys.com pgp: 86574ACA Georg C. F. Greve -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 308 bytes Desc: This is a digitally signed message part. URL: From greve at kolabsys.com Sat Aug 27 20:39:40 2011 From: greve at kolabsys.com (Georg C. F. Greve) Date: Sat, 27 Aug 2011 20:39:40 +0200 Subject: Request for Input: Storing Searches Message-ID: <2232400.u9sBjAbLUj@katana.lair> Dear all, Some of us have started tossing around thoughts about how to save searches in one Kolab client in a way that they are re-usable in all others, ideally. When giving it some brainspace, it turns out this is not a trivial issue, for a variety of reasons, starting with there being a tradeoff decision between being expensive for the CPU or storage, for instance. But it is a little bit more complex than that, actually. Allow me to higlight a couple of scenarios with advantages and disadvantages: - Scenario 1: Storage with a new KEP 9 based XML object One could attempt to model this as a "search" XML object that would incorporate the fields of the object type searched, plus some special fields, e.g. folders to search, as well as searches across multiple fields and search logic (AND/OR etc), These objects would live in the regular folders for resources, and would potentially even replace the list object in functionality, as they would then model a list of recipients as list of address book entries, which is something that Alain once suggested. [1] Advantages: Fairly close to existing functionality, and likely not too hard to implement for most clients (in comparison, at least), no data duplication anywhere. Disadvantages: Expensive on the CPU, Does not work on all resources because we cannot store these XML elements in email type mailboxes. - Scenario 2: Creation of new folder type w/KEP #9 annotation for metadata, create one folder per saved search In this approach we'd create a new folder of the corresponding resource folder type for each search which would be identified as a stored search folder by existence of the /vendor/kolab/saved-search annotation which carries the metadata for the search in an array, e.g. { 'saved_search': { 'search_locations': 'blabla', 'params': 'blabla', 'filter': 'blabla', 'fuzzyness': 'blabla', 'async': '0' .... } } and the folder would be populated with the results of the search. This DOES mean data duplication on the client, but Cyrus does allow to deduplicate entries on the server side, so it would not affect storage there. I am sure something similar would be possible with Dovecot, so we can for the moment assume data gets duplicated on the client only. Advantages: Allows clients without search functionality to use results, can be automatically regenerated on the server if needs be, least CPU usage, works on email. Disadvantages: Data duplication on the client, possible data set de- synchronization (e.g. contact gets edited in search results, same contact in main box and other search results boxes must be updated, this may be hard to ensure), increases folder clutter, some folder sharing questions. - Scenario 3: Map searches with tags As a Kolab object, each search will carry an ID. If we were to introduce a new email header flag in storage that can carry an arbitrary number of tags, we could tag each object with the ID of every search that it matches. IMAP searching for header fields should make it comparatively easy and fast to find all objects of type X that match a certain tag Y, especially if we ask the server to cache this header field. This would be complemented by a KEP 9 compatible object to describe the search, which could then be automatically applied to new objects on the server, or performed by the client, based on the scenario. Advantages: Low CPU & storage requirements, allows Kolab clients to apply a tag concept over all object types including email with potential server side tagging of incoming email Disadvantages: New concept, some questions around shared folders, e.g. what if a client sees a shared object tagged with an ID for a search it does not know because it does not have access to the folder where that search is defined? There may be other advantages and disadvantages that I did not list. Please help us identify them all, so we can come to a good decision. Likewise, if you can think of a scenario that should be considered in addition to the ones listed here, please let me know. As for the scenarios listed, there are two questions in particular that I wonder about: (a) Compatibility with clients, in particular: How will this integrate (or not) with the new Nepomuk/Akonadi KDE Kontact basis (b) Query language: How do we best formulate/store the query in these scenarios? Anyhow, these are my thoughts on the matter right now. I'd be happy to start drafting on something once we've identified which direction things should go into, but right now I still am not sure which is the path to go. So input is VERY appreciated. Best regards, Georg [1] http://kolab.org/pipermail/kolab-format/2011-July/001415.html -- Georg C. F. Greve Chief Executive Officer Kolab Systems AG Z?rich, Switzerland e: greve at kolabsys.com t: +41 78 904 43 33 w: http://kolabsys.com pgp: 86574ACA Georg C. F. Greve -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 308 bytes Desc: This is a digitally signed message part. URL: From thomas at koch.ro Sun Aug 28 15:08:05 2011 From: thomas at koch.ro (Thomas Koch) Date: Sun, 28 Aug 2011 15:08:05 +0200 Subject: Request for Input: Storing Searches In-Reply-To: <2232400.u9sBjAbLUj@katana.lair> References: <2232400.u9sBjAbLUj@katana.lair> Message-ID: <201108281508.05694.thomas@koch.ro> Georg C. F. Greve: > Dear all, > > Some of us have started tossing around thoughts about how to save searches > in one Kolab client in a way that they are re-usable in all others, > ideally. Hi Georg, this one catched my attention since my last project was about search. I try to understand your mail, but it's hard since you write about different implementation options without describing the underlying use cases/user stories. Who should search for what and why should what be saved? Is it about performance (caching search results) or about sharing the query strings? What about privacy issues with shared query strings? I think it would help if you'd really describe it in terms of "User wants ..." Regards, Thomas Koch, http://www.koch.ro From greve at kolabsys.com Sun Aug 28 18:48:29 2011 From: greve at kolabsys.com (Georg C. F. Greve) Date: Sun, 28 Aug 2011 18:48:29 +0200 Subject: Request for Input: Storing Searches In-Reply-To: <201108281508.05694.thomas@koch.ro> References: <2232400.u9sBjAbLUj@katana.lair> <201108281508.05694.thomas@koch.ro> Message-ID: <4961404.Dgrlh7NzkC@katana.lair> Hi Thomas, On Sunday 28 August 2011 15.08:05 Thomas Koch wrote: > Who should search for what and why should what be saved? Is it about > performance (caching search results) or about sharing the query strings? > What about privacy issues with shared query strings? I think it would help > if you'd really describe it in terms of "User wants ..." My apologies. Sometimes when you've spent a certain amount of time thinking about an issue some things may seem obvious even when they're not. The use case here is that users want to search through their data set for any kind of object (including address books, calendars) and save their searches for later re-use, e.g. have a saved search "Show me all calendar entries that take place in Germany from this set of calendars." So it is about storing what needs to be searched, potentially how it needs to be searched, and to make the subsequent searches as efficient as possible, this should be sharable between different clients, as well as different users if they choose to give each other access to such saved searches. Is it clearler now? Best regards, Georg -- Georg C. F. Greve Chief Executive Officer Kolab Systems AG Z?rich, Switzerland e: greve at kolabsys.com t: +41 78 904 43 33 w: http://kolabsys.com pgp: 86574ACA Georg C. F. Greve -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 308 bytes Desc: This is a digitally signed message part. URL: From greve at kolabsys.com Mon Aug 29 13:43:50 2011 From: greve at kolabsys.com (Georg C. F. Greve) Date: Mon, 29 Aug 2011 13:43:50 +0200 Subject: NEW KEP: Update 'contact' object Message-ID: <1990428.gAzyIjUGzZ@katana.lair> Dear all, A new KEP has just been released into public brainstorming: KEP #13: Update 'contact' object http://wiki.kolab.org/User:Greve/Drafts/KEP:13 This KEP is about updating the 'contact' object which has some well known issues that we should resolve, and as these are going to be breaking changes, of which we've had a couple lined up already, I figured it would make sense to put them all into one changeset so we avoid having too many breaking transitions. The state of the KEP is thus that it outlines several of the issues, and provides some input on how this could be better modeled on the storage level, but it is still far from complete - in particular as it changes the underlying way of modeling some of the aspects. Input and drafting help is very much welcome! All of that should go to kolab-format at kolab.org, please. Best regards, Georg -- Georg C. F. Greve Chief Executive Officer Kolab Systems AG Z?rich, Switzerland e: greve at kolabsys.com t: +41 78 904 43 33 w: http://kolabsys.com pgp: 86574ACA Georg C. F. Greve -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 308 bytes Desc: This is a digitally signed message part. URL: From shopov at kolabsys.com Mon Aug 29 13:51:44 2011 From: shopov at kolabsys.com (Bogomil "Bogo" Shopov) Date: Mon, 29 Aug 2011 14:51:44 +0300 Subject: NEW KEP: Update 'contact' object In-Reply-To: <1990428.gAzyIjUGzZ@katana.lair> References: <1990428.gAzyIjUGzZ@katana.lair> Message-ID: <4E5B7D50.20407@kolabsys.com> On 08/29/2011 02:43 PM, Georg C. F. Greve wrote: > Dear all, > > A new KEP has just been released into public brainstorming: > > KEP #13: Update 'contact' object > http://wiki.kolab.org/User:Greve/Drafts/KEP:13 > > This KEP is about updating the 'contact' object which has some well known > issues that we should resolve, and as these are going to be breaking changes, > of which we've had a couple lined up already, I figured it would make sense to > put them all into one changeset so we avoid having too many breaking > transitions. > > The state of the KEP is thus that it outlines several of the issues, and > provides some input on how this could be better modeled on the storage level, > but it is still far from complete - in particular as it changes the underlying > way of modeling some of the aspects. Maybe we can introduce to x-logo *uri and/or bynari attribute * > Input and drafting help is very much welcome! > > All of that should go to kolab-format at kolab.org, please. > > Best regards, > Georg > > > > > _______________________________________________ > Kolab-format mailing list > Kolab-format at kolab.org > https://kolab.org/mailman/listinfo/kolab-format -- Bogomil "Bogo" Shopov Web and QA Ninja Kolab Systems AG Z?rich, Switzerland e: shopov at kolabsys.com t: +359 888 098 768 w: http://kolabsys.com pgp: 0xE69A226A Bogomil Shopov -------------- next part -------------- An HTML attachment was scrubbed... URL: From alain.abbas at libertech.fr Mon Aug 29 16:27:50 2011 From: alain.abbas at libertech.fr (Alain Abbas) Date: Mon, 29 Aug 2011 16:27:50 +0200 Subject: NEW KEP: Update 'contact' object In-Reply-To: <1990428.gAzyIjUGzZ@katana.lair> References: <1990428.gAzyIjUGzZ@katana.lair> Message-ID: <4E5BA1E6.1000005@libertech.fr> An HTML attachment was scrubbed... URL: From vanmeeuwen at kolabsys.com Tue Aug 30 10:11:09 2011 From: vanmeeuwen at kolabsys.com (Jeroen van Meeuwen (Kolab Systems)) Date: Tue, 30 Aug 2011 09:11:09 +0100 Subject: Request for Input: Storing Searches In-Reply-To: <2232400.u9sBjAbLUj@katana.lair> References: <2232400.u9sBjAbLUj@katana.lair> Message-ID: <201108300911.11175.vanmeeuwen@kolabsys.com> Georg C. F. Greve wrote: > Allow me to higlight a couple of scenarios with advantages and > disadvantages: > > - Scenario 2: Creation of new folder type w/KEP #9 annotation for > metadata, create one folder per saved search > > In this approach we'd create a new folder of the corresponding resource > folder type for each search which would be identified as a stored search > folder by existence of the /vendor/kolab/saved-search annotation Please note a unique annotation is technically not sustainable, because all annotation keys need to be listed in a static list in a configuration file on the server-side. It's better to use /vendor/kolab/folder-type contact, with a /vendor/kolab/folder-config holding the saved search data such as, for example, the following snippet; > which > carries the metadata for the search in an array, e.g. > > { 'saved_search': > { 'search_locations': 'blabla', > 'params': 'blabla', > 'filter': 'blabla', > 'fuzzyness': 'blabla', > 'async': '0' > .... > } > } > > and the folder would be populated with the results of the search. > Please note that the folder *could* be populated with the results of the search. I think this is an option. > This DOES mean data duplication on the client, but Cyrus does allow to > deduplicate entries on the server side, so it would not affect storage > there. I am sure something similar would be possible with Dovecot, so > we can for the moment assume data gets duplicated on the client only. > > Advantages: Allows clients without search functionality to use results, > can be automatically regenerated on the server if needs be, least CPU > usage, works on email. > > Disadvantages: Data duplication on the client, possible data set de- > synchronization (e.g. contact gets edited in search results, same contact > in main box and other search results boxes must be updated, this may be > hard to ensure), increases folder clutter, some folder sharing questions. > Populating the folder with the results would indeed duplicate the contacts/events/mail/whathaveyou. However, it would be a MUA problem to solve editing an object from within the saved search (prevent editting, warn user about editing this copy of the object, or replicate the change back to the original source of the search result -note the object's UID is available in the search results). Perhaps the IMAP folder could be made not-writeable. This could have implications on the ability to edit the saved search though. > - Scenario 3: Map searches with tags > > As a Kolab object, each search will carry an ID. If we were to introduce > a new email header flag in storage that can carry an arbitrary number > of tags, we could tag each object with the ID of every search that it > matches. > There is a concept called IMAP Flags, and a concept of RFC 5257 per-message annotation. Why would we need to insert a header into an email message? > Likewise, if you can think of a scenario that should be considered in > addition to the ones listed here, please let me know. As for the scenarios > listed, there are two questions in particular that I wonder about: > > (a) Compatibility with clients, in particular: How will this integrate (or > not) with the new Nepomuk/Akonadi KDE Kontact basis > > (b) Query language: How do we best formulate/store the query in these > scenarios? > > Anyhow, these are my thoughts on the matter right now. > > I'd be happy to start drafting on something once we've identified which > direction things should go into, but right now I still am not sure which is > the path to go. > These paths do not have to be mutually exclusive. A contact folder-type with folder-config annotations for a saved search does not inherently prevent a Kolab XML object with configuration for a search based on tags/flags/other message metadata (From, To, ...). Kind regards, Jeroen van Meeuwen -- Senior Engineer, Kolab Systems AG e: vanmeeuwen at kolabsys.com t: +44 144 340 9500 m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08 -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanmeeuwen at kolabsys.com Tue Aug 30 10:14:34 2011 From: vanmeeuwen at kolabsys.com (Jeroen van Meeuwen (Kolab Systems)) Date: Tue, 30 Aug 2011 09:14:34 +0100 Subject: NEW KEP: Update 'contact' object In-Reply-To: <1990428.gAzyIjUGzZ@katana.lair> References: <1990428.gAzyIjUGzZ@katana.lair> Message-ID: <201108300914.35708.vanmeeuwen@kolabsys.com> Georg C. F. Greve wrote: > Dear all, > > A new KEP has just been released into public brainstorming: > > KEP #13: Update 'contact' object > http://wiki.kolab.org/User:Greve/Drafts/KEP:13 > > This KEP is about updating the 'contact' object which has some well known > issues that we should resolve, and as these are going to be breaking > changes, of which we've had a couple lined up already, I figured it would > make sense to put them all into one changeset so we avoid having too many > breaking transitions. > Can we include the fact the Subject/To/From headers not represent any of the contact object's metadata? This means searches have to go through a lot more (XML) data then that they would have to if any of such data was stored on any of the email message headers. Kind regards, Jeroen van Meeuwen -- Senior Engineer, Kolab Systems AG e: vanmeeuwen at kolabsys.com t: +44 144 340 9500 m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mollekopf at kolabsys.com Tue Aug 30 11:47:30 2011 From: mollekopf at kolabsys.com (Christian Mollekopf) Date: Tue, 30 Aug 2011 11:47:30 +0200 Subject: Request for Input: Storing Searches In-Reply-To: <2232400.u9sBjAbLUj@katana.lair> References: <2232400.u9sBjAbLUj@katana.lair> Message-ID: <2522707.RBIlZQlMEe@myhost> On Saturday, August 27, 2011 08:39:40 PM Georg C. F. Greve wrote: > Dear all, > > Some of us have started tossing around thoughts about how to save searches > in one Kolab client in a way that they are re-usable in all others, > ideally. > > When giving it some brainspace, it turns out this is not a trivial issue, > for a variety of reasons, starting with there being a tradeoff decision > between being expensive for the CPU or storage, for instance. But it is a > little bit more complex than that, actually. > > Allow me to higlight a couple of scenarios with advantages and > disadvantages: > > - Scenario 1: Storage with a new KEP 9 based XML object > > One could attempt to model this as a "search" XML object that would > incorporate the fields of the object type searched, plus some special > fields, e.g. folders to search, as well as searches across multiple fields > and search logic (AND/OR etc), > > These objects would live in the regular folders for resources, and would > potentially even replace the list object in functionality, as they would > then model a list of recipients as list of address book entries, which is > something that Alain once suggested. [1] > > Advantages: Fairly close to existing functionality, and likely not too > hard to implement for most clients (in comparison, at least), no data > duplication anywhere. > > Disadvantages: Expensive on the CPU, Does not work on all resources > because we cannot store these XML elements in email type mailboxes. > > > - Scenario 2: Creation of new folder type w/KEP #9 annotation for metadata, > create one folder per saved search > > In this approach we'd create a new folder of the corresponding resource > folder type for each search which would be identified as a stored search > folder by existence of the /vendor/kolab/saved-search annotation which > carries the metadata for the search in an array, e.g. > > { 'saved_search': > { 'search_locations': 'blabla', > 'params': 'blabla', > 'filter': 'blabla', > 'fuzzyness': 'blabla', > 'async': '0' > .... > } > } > > and the folder would be populated with the results of the search. > > This DOES mean data duplication on the client, but Cyrus does allow to > deduplicate entries on the server side, so it would not affect storage > there. I am sure something similar would be possible with Dovecot, so > we can for the moment assume data gets duplicated on the client only. > > Advantages: Allows clients without search functionality to use results, > can be automatically regenerated on the server if needs be, least CPU > usage, works on email. > > Disadvantages: Data duplication on the client, possible data set de- > synchronization (e.g. contact gets edited in search results, same contact > in main box and other search results boxes must be updated, this may be > hard to ensure), increases folder clutter, some folder sharing questions. > > > - Scenario 3: Map searches with tags > > As a Kolab object, each search will carry an ID. If we were to introduce > a new email header flag in storage that can carry an arbitrary number > of tags, we could tag each object with the ID of every search that it > matches. > > IMAP searching for header fields should make it comparatively easy > and fast to find all objects of type X that match a certain tag Y, > especially if we ask the server to cache this header field. > > This would be complemented by a KEP 9 compatible object to describe > the search, which could then be automatically applied to new objects on > the server, or performed by the client, based on the scenario. > > Advantages: Low CPU & storage requirements, allows Kolab clients to apply > a tag concept over all object types including email with potential server > side tagging of incoming email > > Disadvantages: New concept, some questions around shared folders, e.g. > what if a client sees a shared object tagged with an ID for a search it > does not know because it does not have access to the folder where that > search is defined? > Something along the lines of tagging the search results via the ANNOTATE Extension and using an XML object will probably gives us the most flexibility. There are however some problems existing: On the Akonadi side searches are afaik implemented as virtual folders which are populated based on a nepomuk query(sparql). Searches don't belong to a single resource however, since you can search accross various resources. This means we would have to implement that functionality from outside of the kolab resource I reckon. What I think is possible is something along the lines of this: - Add an option to search folders "Share this search", where you can choose a resource which supports shared searches (Kolab). - The action would then pick the results from the search which are in this resource, and tag them via ANNOTATE and create an xml object with the search info. >From here were going the same path if we are consumer or creator of the search: - The resource creates a virtual-folder inside the resource i.e. "Shared Searches/Last Search" (The name can be edited and is stored to the xml object), based on the xml object. - The resource populates the virtual folder with virtual items based on the tags. => - No data duplication - items are directly editable - you get only the results for which you have permission. Also we can implement most of the code in the kolab resource, and the only additional thing which we need is the action which creates the saved search. At a later stage we could even rerun the query inside the resource to find additional matches which were not available for the creator of the search, or new matches but that's not even necessarily expected from the user. If we have a revision number in the xml object the search could even be modified and updated. I think this gives us a pretty flexible and powerful way of sharing folders which is at least fully compatible with akonadi. No idea however if something like this can also be implemented in an MS Exchange client. Btw. we could even think of adding a readonly folder with copies for the limited clients (along the lines of option #2). I.e. on akonadi this folder would be hidden so you get the full functionality including editing via virtual folders. On the smartphone (or just any other imap client) you get only the readonly version, but at least you have something for reference. But this, again, can be added at a later stage and is fully optional (as long as we have a field in the xml object to store this folder). I don't see any implications on nepomuk with either of those solutions. With my best regards, Christian > > There may be other advantages and disadvantages that I did not list. > > Please help us identify them all, so we can come to a good decision. > > Likewise, if you can think of a scenario that should be considered in > addition to the ones listed here, please let me know. As for the scenarios > listed, there are two questions in particular that I wonder about: > > (a) Compatibility with clients, in particular: How will this integrate (or > not) with the new Nepomuk/Akonadi KDE Kontact basis > > (b) Query language: How do we best formulate/store the query in these > scenarios? > > Anyhow, these are my thoughts on the matter right now. > > I'd be happy to start drafting on something once we've identified which > direction things should go into, but right now I still am not sure which is > the path to go. So input is VERY appreciated. > > Best regards, > Georg > > > > [1] http://kolab.org/pipermail/kolab-format/2011-July/001415.html -- Christian Mollekopf Software Engineer Kolab Systems AG Z?rich, Switzerland e: mollekopf at kolabsys.com w: http://kolabsys.com pgp: EA657400 Christian Mollekopf -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: From vanmeeuwen at kolabsys.com Tue Aug 30 15:03:29 2011 From: vanmeeuwen at kolabsys.com (Jeroen van Meeuwen (Kolab Systems)) Date: Tue, 30 Aug 2011 14:03:29 +0100 Subject: Request for Input: Storing Searches In-Reply-To: <2522707.RBIlZQlMEe@myhost> References: <2232400.u9sBjAbLUj@katana.lair> <2522707.RBIlZQlMEe@myhost> Message-ID: <201108301403.42063.vanmeeuwen@kolabsys.com> Christian Mollekopf wrote: > Something along the lines of tagging the search results via the ANNOTATE > Extension and using an XML object will probably gives us the most > flexibility. > Searches may or may not include non-IMAP content (such as LDAP content). Why / How would a Kolab XML object help? Why not save the information in a "virtual" IMAP folder? This IMAP folder can handle the ACLs, thus be shared, it shows up empty for clients that are unable to handle KEP #9, and *at your option* could be prepopulated with the results of the search (OPTIONAL) - though the tradeoff would be data duplication. > There are however some problems existing: > > On the Akonadi side searches are afaik implemented as virtual folders which > are populated based on a nepomuk query(sparql). Searches don't belong to a > single resource however, since you can search accross various resources. This is a client-side (saved) search, that cannot be shared with any other client, correct? We are attempting to put the saved search on the server-side somehow. > This means we would have to implement that functionality from outside of > the kolab resource I reckon. > What do you mean by 'the kolab resource'? An account? A folder? A single Kolab XML object? The yet-to-be-defined Kolab saved-search XML object? > - The action would then pick the results from the search which are in this > resource, and tag them via ANNOTATE and create an xml object with the > search info. > Where would this XML object be put? > From here were going the same path if we are consumer or creator of the > search: > > - The resource creates a virtual-folder inside the resource i.e. "Shared > Searches/Last Search" (The name can be edited and is stored to the xml > object), based on the xml object. > "the resource" and "the resource" keep confusing me. Also, "Shared searches" implies the searches are shared... I think "saved searches" are more appropriate. > - The resource populates the virtual folder with virtual items based on the > tags. > > => - No data duplication This has always been "optional"; a saved search folder *could* be pre- populated. Imagine a saved search across 5 contact folders with 10.000 contacts on average. When NOT pre-populating the saved search folder with the search results, you pay the cost every time the folder is opened. Maybe this cost is not so great for a fat client with a local cache to query (Kontact/akonadi/nepomuk/Disconnected IMAP), but for a web-interface... well... When pre-populating the saved search folder with the search results, you pay the cost in "duplicate" storage (as explained, not on the server side, perhaps on the client side if it's not intelligent enough to de-duplicate). Another penalty in pre-populating the saved search folder could, arguably, be that perhaps there's results in said saved search folder that the person using the folder would otherwise not have access to. However, this can also be considered a feature; "Share all contacts from Vendors folder tagged with 'ict' with helpdesk personnel" Kind regards, Jeroen van Meeuwen -- Senior Engineer, Kolab Systems AG e: vanmeeuwen at kolabsys.com t: +44 144 340 9500 m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mollekopf at kolabsys.com Tue Aug 30 16:11:49 2011 From: mollekopf at kolabsys.com (Christian Mollekopf) Date: Tue, 30 Aug 2011 16:11:49 +0200 Subject: Request for Input: Storing Searches In-Reply-To: <201108301403.42063.vanmeeuwen@kolabsys.com> References: <2232400.u9sBjAbLUj@katana.lair> <2522707.RBIlZQlMEe@myhost> <201108301403.42063.vanmeeuwen@kolabsys.com> Message-ID: <1429331.OZmtLNvdxk@myhost> On Tuesday, August 30, 2011 02:03:29 PM you wrote: > Christian Mollekopf wrote: > > Something along the lines of tagging the search results via the ANNOTATE > > Extension and using an XML object will probably gives us the most > > flexibility. > > Searches may or may not include non-IMAP content (such as LDAP content). > > Why / How would a Kolab XML object help? Why not save the information in a > "virtual" IMAP folder? > > This IMAP folder can handle the ACLs, thus be shared, it shows up empty for > clients that are unable to handle KEP #9, and *at your option* could be > prepopulated with the results of the search (OPTIONAL) - though the tradeoff > would be data duplication. > True, I thought by using something else than an empty folder we don't have to hide the empty folder. Otherwise I guess an IMAP folder per search would do as well. > > There are however some problems existing: > > > > On the Akonadi side searches are afaik implemented as virtual folders > > which are populated based on a nepomuk query(sparql). Searches don't > > belong to a single resource however, since you can search accross > > various resources. > This is a client-side (saved) search, that cannot be shared with any other > client, correct? > Yes, that is correct. > We are attempting to put the saved search on the server-side somehow. Clear. > > > This means we would have to implement that functionality from outside of > > the kolab resource I reckon. > > What do you mean by 'the kolab resource'? An account? A folder? A single > Kolab XML object? The yet-to-be-defined Kolab saved-search XML object? > Sorry for not being more specific. Whenever I used the word resource I am referring to an akonadi resource (aka agent). The connection to kolab is in akonadi realised as a kolab-resource which handles all the kolab specific bits and then exposes the data with message/rfc822 messages. In a similar way there exist resources to access email from pop3 servers etc. you get the idea. See http://community.kde.org/KDE_PIM/Akonadi for more info. > > - The action would then pick the results from the search which are in > > this resource, and tag them via ANNOTATE and create an xml object with > > the search info. > > Where would this XML object be put? > I'd imagine that we put those objects in the rootfolder or in a "Saved Searches" folder. In the "Saved Searches" folder we could then also create the optional server-side populated search directories. > > From here were going the same path if we are consumer or creator of the > > search: > > > > - The resource creates a virtual-folder inside the resource i.e. "Shared > > Searches/Last Search" (The name can be edited and is stored to the xml > > object), based on the xml object. > > "the resource" and "the resource" keep confusing me. Also, "Shared searches" > implies the searches are shared... I think "saved searches" are more > appropriate. Yeah, my "shared" was more referring to shared among different machines, Saved searches are probably more appropriate. > > > - The resource populates the virtual folder with virtual items based on > > the tags. > > > > => - No data duplication > > This has always been "optional"; a saved search folder *could* be pre- > populated. > > Imagine a saved search across 5 contact folders with 10.000 contacts on > average. > > When NOT pre-populating the saved search folder with the search results, you > pay the cost every time the folder is opened. Maybe this cost is not so > great for a fat client with a local cache to query > (Kontact/akonadi/nepomuk/Disconnected IMAP), but for a web-interface... > well... Yes, since akonadi can create virtual folders it only has to populate them once AFAIK, and then result is then cached. For a webclient I guess you're right. But if we have the dataduplication and it should also be writable it looks somewhat error prone to me. Especially if we have to implement that for every client. I reckon using akonadi as a cache for the webinterface would solve that problem? > > When pre-populating the saved search folder with the search results, you pay > the cost in "duplicate" storage (as explained, not on the server side, > perhaps on the client side if it's not intelligent enough to de-duplicate). > Well, the argument for doing it server side would be that it is available on any client (i.e. smartphone), but then read-only is the only option i see. If it is on the client side, this ends up to be essentially the same as the akonadi virtual folders. > Another penalty in pre-populating the saved search folder could, arguably, > be that perhaps there's results in said saved search folder that the person > using the folder would otherwise not have access to. However, this can also > be considered a feature; "Share all contacts from Vendors folder tagged > with 'ict' with helpdesk personnel" > If it is being populated on the client side I don't see how you could get access to items you shouldn't have access to. With my best regards, Christian > Kind regards, > > Jeroen van Meeuwen -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: From vanmeeuwen at kolabsys.com Wed Aug 31 14:13:24 2011 From: vanmeeuwen at kolabsys.com (Jeroen van Meeuwen (Kolab Systems)) Date: Wed, 31 Aug 2011 13:13:24 +0100 Subject: Request for Input: Storing Searches In-Reply-To: <1429331.OZmtLNvdxk@myhost> References: <2232400.u9sBjAbLUj@katana.lair> <201108301403.42063.vanmeeuwen@kolabsys.com> <1429331.OZmtLNvdxk@myhost> Message-ID: <201108311313.25074.vanmeeuwen@kolabsys.com> Christian Mollekopf wrote: > > > - The action would then pick the results from the search which are in > > > this resource, and tag them via ANNOTATE and create an xml object with > > > the search info. > > > > Where would this XML object be put? > > I'd imagine that we put those objects in the rootfolder or in a "Saved > Searches" folder. In the "Saved Searches" folder we could then also create > the optional server-side populated search directories. > There's no such thing as a 'root folder'. If a 'Saved Searches' folder were to be used, all 'saved search' Kolab XML objects would go into that one folder. Sharing any particular saved search now becomes a problem, Clients not compatible with KEP #9 are now helpless, since a top-level folder of unknown type is encountered, but they have to descend in order to get to any sub-folder, The saved searches folder(s) cannot be pre-populated unless the Kolab XML object for saved searches also states where any pre-populating should go out to, which naturally is subject to too much change, Keeping 'reference' objects in a 'saved search' folder creates the same 'subject to change' problem... if a reference object where to say, user/john.doe/Contacts at example.org?uid=blabla, renaming the Contacts folder to Kontacten would create a reference issue; any reference should be completely referential (OLAP), etc. > > > - The resource populates the virtual folder with virtual items based on > > > the tags. > > > > > > => - No data duplication > > > > This has always been "optional"; a saved search folder *could* be pre- > > populated. > > > > Imagine a saved search across 5 contact folders with 10.000 contacts on > > average. > > > > When NOT pre-populating the saved search folder with the search results, > > you pay the cost every time the folder is opened. Maybe this cost is not > > so great for a fat client with a local cache to query > > (Kontact/akonadi/nepomuk/Disconnected IMAP), but for a web-interface... > > well... > > Yes, since akonadi can create virtual folders it only has to populate them > once AFAIK, and then result is then cached. > For a webclient I guess you're right. But if we have the dataduplication Again, the data duplication is *at one's option*. For web-interface, I say one should pre-populate the search folder. For clients like Kontact (with client- side, local caches), perhaps it's feasible to allow them to ignore the cached results and go with a real-time search. > and it should also be writable it looks somewhat error prone to me. A clause in the KEP for these types of folders can be, that the content of the folder SHOULD NOT be made writeable for any event other then 'update saved search'. > Especially if we have to implement that for every client. > We have to implement everything and anything for every client, in case you haven't noticed. > I reckon using akonadi as a cache for the webinterface would solve that > problem? > No, the caching layer is moot unless you also consider all clients use akonadi. One cache (in one location) to rule them all being akonadi is not necessarily the best way to go with this. This, however, is a different topic, and we should talk about caching separately from the saved searches topic. > > When pre-populating the saved search folder with the search results, you > > pay the cost in "duplicate" storage (as explained, not on the server > > side, perhaps on the client side if it's not intelligent enough to > > de-duplicate). > > Well, the argument for doing it server side would be that it is available > on any client (i.e. smartphone), but then read-only is the only option i > see. If it is on the client side, this ends up to be essentially the same > as the akonadi virtual folders. > Note that the "problem" or "difficulty" is not the editing of an object from within the saved search, not for a client and not for a user. It is the occurence of said object twice or more times in or across all readable folders that is the first problem. > > Another penalty in pre-populating the saved search folder could, > > arguably, be that perhaps there's results in said saved search folder > > that the person using the folder would otherwise not have access to. > > However, this can also be considered a feature; "Share all contacts from > > Vendors folder tagged with 'ict' with helpdesk personnel" > > If it is being populated on the client side I don't see how you could get > access to items you shouldn't have access to. > You want to avoid your 350.000 clients from each having to iterate and re- iterate against most of your infrastructure components themselves, just to pre-populate and update their saved (contact) searches cache(s), if you can do so periodically on the server-side, under your own control. Please note that the folders of type 'contact' that the user has access to are not the only things that are subject to change. Please also note that these folders and other resources can be *huge*. Saved searches are often not the smallest set of search results. They often include a type of query that is a little less specific then (mail=john.doe at example.org), and inherently can include a lot of attributes that are not (cannot be?) indexed anywhere. As such, saved searches are *hugely* expensive to execute. When they happen on the client side, and on one particular client only, configurable per client, then we're all fine. Kontact for instance can do saved searches in a reasonable fashion because of Akonadi. A web interface such as Horde or Roundcube however cannot. Please note a saved search for these interfaces will have to; 1) execute within the PHP execution timeout (30 seconds), 2) stay within the memory_limit (64/128/192 MB), 3) do not use the user's credentials / privileges to query resources other then IMAP Furthermore, interesting saved searches discussions can be held over particular types of Kolab clients such as Z-Push (and its clients). Not all Kolab-consuming applications have a need for, an interface for, or a place for saved searches or server-side caching of said saved searches. Similarly, not all clients have a need for, or place for, a server-side cache -such as Disconnected IMAP in Kontact. That said, a folder clearly configured as a 'saved_search' folder can be ignored (Z-Push?) or the contents thereof can be ignored (Kontact?), while the contents may be the periodically updated search results, or the folder be empty. Both cases serve all clients well, KEP #9 compatible or not. Kind regards, Jeroen van Meeuwen -- Senior Engineer, Kolab Systems AG e: vanmeeuwen at kolabsys.com t: +44 144 340 9500 m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08 -------------- next part -------------- An HTML attachment was scrubbed... URL: From vkrause at kde.org Wed Aug 31 15:25:40 2011 From: vkrause at kde.org (Volker Krause) Date: Wed, 31 Aug 2011 15:25:40 +0200 Subject: Request for Input: Storing Searches In-Reply-To: <2232400.u9sBjAbLUj@katana.lair> References: <2232400.u9sBjAbLUj@katana.lair> Message-ID: <2016124.HU9QVM6cYj@vkpc9> Hi, Christian already pointed out some implications on Kontact/Akonadi, but I have some additional remarks on that as well. On Saturday 27 August 2011 20:39:40 Georg C. F. Greve wrote: > Some of us have started tossing around thoughts about how to save searches > in one Kolab client in a way that they are re-usable in all others, > ideally. > > When giving it some brainspace, it turns out this is not a trivial issue, > for a variety of reasons, starting with there being a tradeoff decision > between being expensive for the CPU or storage, for instance. But it is a > little bit more complex than that, actually. > > Allow me to higlight a couple of scenarios with advantages and > disadvantages: > > - Scenario 1: Storage with a new KEP 9 based XML object > > One could attempt to model this as a "search" XML object that would > incorporate the fields of the object type searched, plus some special > fields, e.g. folders to search, as well as searches across multiple fields > and search logic (AND/OR etc), Here you have the first challenge already (we run in to the same issue during the work on Akonadi, and it's still not fully solved there): You need to define the exact semantics of the query language and make sure all clients can actually implement this. Seems easy at first ("person's name equals 'Georg'"), but you quickly end up with ugly details: - what kind of Boolean logic do you want to support, and in what nesting depth? - what kind of comparison operators do you want to support for strings? equal, contains, matches regexp, case sensitive vs. insensitive, etc - what comparison operators do you want to support for other types (numbers, date/time)? equal, greater/less than, occurs at, etc. (keep in mind the extra fun with DST/time zones and recurrences), also note the complexity implications of recurrence-related queries with negations (e.g. "event does never occur on a Friday 13th") - on what fields are you operating? Just the ones defined in the Kolab format spec, or is that too low-level for users and you want higher level composite fields? Eg. a person's name maps to a bunch of internal fields. - next to the low-level search operators, are you considering higher-level ones that depend on external context (eg. something along the lines of KMail's "mail from someone in my address book" or "mail from someone tagged as 'Friend'" operators, or your example of "events in Germany")? If you want client-side search with consistent results, this needs to be specified in detail. Unlike the format specification where we only display the most of content without interpretation (except most date/time fields), this actually requires agreement on exact semantics. > These objects would live in the regular folders for resources, and would > potentially even replace the list object in functionality, as they would > then model a list of recipients as list of address book entries, which is > something that Alain once suggested. [1] > > Advantages: Fairly close to existing functionality, and likely not too > hard to implement for most clients (in comparison, at least), no data > duplication anywhere. implementation complexity for us would mainly depend on the complexity of the query language and how well it maps to our existing search support. > Disadvantages: Expensive on the CPU, Does not work on all resources > because we cannot store these XML elements in email type mailboxes. what resources are you referring to here? > - Scenario 2: Creation of new folder type w/KEP #9 annotation for metadata, > create one folder per saved search > > In this approach we'd create a new folder of the corresponding resource > folder type for each search which would be identified as a stored search > folder by existence of the /vendor/kolab/saved-search annotation which > carries the metadata for the search in an array, e.g. > > { 'saved_search': > { 'search_locations': 'blabla', > 'params': 'blabla', > 'filter': 'blabla', > 'fuzzyness': 'blabla', > 'async': '0' > .... > } > } > > and the folder would be populated with the results of the search. > > This DOES mean data duplication on the client, but Cyrus does allow to > deduplicate entries on the server side, so it would not affect storage > there. I am sure something similar would be possible with Dovecot, so > we can for the moment assume data gets duplicated on the client only. This is actually very close to the Akonadi approach for searching. We also have dedicated folders for search results ("virtual folders"), which contain only references to the actual object (sort of like symlinks). This avoid duplication and de-synchronization issues etc.. If the Kolab server would allow us to retrieve information about the original location of an object in a pre-populated search folder, we could symlink that on the client as well. There are some pros and cons of the server-side search approach though: + there's a single implementation, so same semantics and expressive power everywhere + minimal client changes required - no search possible when offline - increasing de-synchronization between results and reality while being offline > Advantages: Allows clients without search functionality to use results, > can be automatically regenerated on the server if needs be, least CPU > usage, works on email. > > Disadvantages: Data duplication on the client, possible data set de- > synchronization (e.g. contact gets edited in search results, same contact > in main box and other search results boxes must be updated, this may be > hard to ensure), increases folder clutter, some folder sharing questions. > > > - Scenario 3: Map searches with tags > > As a Kolab object, each search will carry an ID. If we were to introduce > a new email header flag in storage that can carry an arbitrary number > of tags, we could tag each object with the ID of every search that it > matches. > > IMAP searching for header fields should make it comparatively easy > and fast to find all objects of type X that match a certain tag Y, > especially if we ask the server to cache this header field. > > This would be complemented by a KEP 9 compatible object to describe > the search, which could then be automatically applied to new objects on > the server, or performed by the client, based on the scenario. This assumes there is agreement on the exact semantics, otherwise you could end up with nasty tag ping-pong games between multiple clients and/or the server. > Advantages: Low CPU & storage requirements, allows Kolab clients to apply > a tag concept over all object types including email with potential server > side tagging of incoming email > > Disadvantages: New concept, some questions around shared folders, e.g. > what if a client sees a shared object tagged with an ID for a search it > does not know because it does not have access to the folder where that > search is defined? Unlike the other two option, this doesn't really fit well into the Akonadi way of dealing with searches (searching by tags is easy, but we are not prepared to tag and write back based on queries yet), so this would be the most expensive one for us to implement I think. > There may be other advantages and disadvantages that I did not list. > > Please help us identify them all, so we can come to a good decision. > > Likewise, if you can think of a scenario that should be considered in > addition to the ones listed here, please let me know. As for the scenarios > listed, there are two questions in particular that I wonder about: > > (a) Compatibility with clients, in particular: How will this integrate (or > not) with the new Nepomuk/Akonadi KDE Kontact basis The biggest challenge for Kontact would be the fact that these searches are limited to just one Kolab account (and would most likely be context-free). Both are limitations we have been working on getting rid of. Ie. we would like the user to see his data as one set, without having to worry where it's coming from and what capabilities follow from that. Also, we are moving to more higher-level context-aware query/search concepts (e.g. "persons near my current location"). We already have a similar problem with KMail local filters and server-side Sieve filters, the latter basically having the same limitations as a per-account cross-client saved search would have. Of course, if you look at this from the web client POV, you don't really have these kind of problems. regards Volker -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: This is a digitally signed message part. URL: From greve at kolabsys.com Wed Aug 31 20:10:02 2011 From: greve at kolabsys.com (Georg C. F. Greve) Date: Wed, 31 Aug 2011 20:10:02 +0200 Subject: NEW KEP: KEP #14: Non-conflicting edits of RFC5228/Sieve scripts by multiple editors Message-ID: <2367431.zDd980amMF@katana.lair> Dear all, Based on the initial idea circulated in May on kolab-devel, please be informed of a new Design KEP to enable non-conflicting mail filter edits on the server by multiple clients: KEP 14: Non-conflicting edits of RFC5228/Sieve scripts by multiple editors http://wiki.kolab.org/User:Greve/Drafts/KEP:14 This is more on the convention/application behaviour side and not really a format modification, so all comments/discussion should take place on kolab-devel at kolab.org for this particular KEP. Best regards, Georg -- Georg C. F. Greve Chief Executive Officer Kolab Systems AG Z?rich, Switzerland e: greve at kolabsys.com t: +41 78 904 43 33 w: http://kolabsys.com pgp: 86574ACA Georg C. F. Greve -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 308 bytes Desc: This is a digitally signed message part. URL: