KEP-0015.txt

Georg Greve greve at kolabsys.com
Tue Sep 6 16:14:29 CEST 2011


 KEP-0015.txt |  140 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 140 insertions(+)

New commits:
commit d167784a6623159600eaf7170ccc981435b2376d
Author: Georg Greve <greve at kolabsys.com>
Date:   Tue Sep 6 16:13:57 2011 +0200

    Very first draft. Still incomplete. Not all concepts already nailed down. But a start of channeling the discussion.

diff --git a/KEP-0015.txt b/KEP-0015.txt
new file mode 100644
index 0000000..f8b37bd
--- /dev/null
+++ b/KEP-0015.txt
@@ -0,0 +1,140 @@
+{{kep
+ |number=15
+ |ticketnumber=
+ |title=Saved searches and sharing searches across all clients
+ |author=Georg Greve
+ |author_email=greve at kolabsys.com
+ |status=draft
+ |type=design
+ |creation_date=
+ |obsoleted_by=
+ |related=
+}}
+
+== Abstract ==
+This Kolab Enhancement Proposal (KEP) <ref name="kep">[[KEP:1]] Bootstrapping the KEP process</ref> defines part of a changeset applied against Kolab XML version 1.0 of Kolab Format 2.0 to create provisions for (a) sharing the parameters of a search across clients, (b) sharing the parameters of a search across users, (c) sharing the results of a previously executed search across clients, (d) sharing the results of a previously executed search across users and (e) doing all of the above in a way that enables automatic asynchronous updates to search results to keep certain search results readily available. Searches can take place across '''all''' object types and the LDAP tree, as well as potential external resources tied into the Kolab Groupware Solution.
+
+In order to achieve that functionality, this KEP defines a new Kolab XML object and makes use of the {{rfc|5257}}<ref name="rfc5257">{{rfc|5257|title=Internet Message Access Protocol - ANNOTATE Extension}}</ref> and {{rfc|5464}}<ref name="rfc5464">{{rfc|5464|title=The IMAP METADATA Extension}}</ref> provisions set forth in <ref name="kep9">[[KEP:9]] Storage of configuration and application control information</ref>. 
+
+The KEP follows a brainstorming session<ref name="rfi">[http://kolab.org/pipermail/kolab-format/2011-August/001432.html kolab-format at kolab.org: Request for Input: Storing Searches]</ref> on Kolab Format started 27 August 2011.
+
+== Storing Searches ==
+For every stored search a new folder of [[#New_value_.27search.27_for_.27.2Fvendor.2Fkolab.2Ffolder-type.27|folder-type 'search']] '''SHALL''' be generated with the parameters of the search stored in the [[#New_annotation:_.27.2Fvendor.2Fkolab.2Fsearch.27|'search' annotation]]. This folder '''MAY''' hold objects of any object type, e.g. email, events, contacts.
+
+=== Client Scope and Conventions ===
+Clients '''MAY''' choose to implement display-only support for saved searches by following only the provisions in the [[#Display|Display]] section, below. Such clients '''SHALL''' be known as '''Passive Clients'''. All other clients '''MUST''' implement all rules of the [[#Passive|Passive]] and [[#Active|Active]] sections, below. Such clients '''SHALL''' be known as '''Full Clients'''.
+
+==== Passive ====
+Passive clients '''MUST''' be capable to parse for folders of [[#New_value_.27search.27_for_.27.2Fvendor.2Fkolab.2Ffolder-type.27|folder-type 'search']] and '''MUST''' be able to display at least one category of object type completely. 
+
+===== Caching Folders =====
+For folders with 'caching' enabled in the [[#New_annotation:_.27.2Fvendor.2Fkolab.2Fsearch.27|'search' annotation]], the client '''MUST''' consider only objects in this folder to be matching the search. 
+
+* Before displaying these objects, the client '''SHALL''' make use of the [[#New_annotation:_.27.2Fvendor.2Fkolab.2Foriginal.27|'/vendor/kolab/original']] message annotation to confirm the object has '''not''' been modified. If the object was modified, it will have changed its number in the IMAP folder, so the client '''MUST''' look for the updated object by its 'uid';
+* If the client cannot find an object with the same 'uid' by searching the Message-Id (Email) and Subject (Kolab XML Objects) header fields, clients '''SHALL''' assume the object has been deleted and remove it also in the search folder;
+* If the user does not have read-access to the folder containing the original object, the client '''SHALL''' assume the object is unmodified;
+* If the object was modified, the client '''SHALL''' first update the object in the search result folder accordingly, preserving all message annotations of the copy in the search folder. Secondly it '''SHALL''' make use of the [[#New_annotation:_.27.2Fvendor.2Fkolab.2Fmatches.27|'/vendor/kolab/matches/']] attribute to update all other folders in which this object is mirrored to which this client has write access.
+
+===== Non-Caching Folders =====
+For folders with disabled 'caching', the  client '''MUST''' search the entire IMAP tree visible to this user for occurences of the saved search in [[#New_annotation:_.27.2Fvendor.2Fkolab.2Fmatches.27|'/vendor/kolab/matches/']] message annotations among all object types the client is capable of displaying.
+
+===== User modification/deletion of object =====
+By default, client '''SHALL''' display all search results as 'read only' and '''MAY''' choose to do so even if permissions for modification and deletion exist. Where a client aims to provide modification or deletion of objects, client '''MAY''' only allow such modification or deletion where the client has sufficient permissions on '''BOTH''' the search folder containing the object '''AND''' the folder of the original object as referred to in the [[#New_annotation:_.27.2Fvendor.2Fkolab.2Foriginal.27|'/vendor/kolab/original']] message annotation.
+
+For modifications of an object, the client '''MUST''' 
+* directly modify the original object referenced in the [[#New_annotation:_.27.2Fvendor.2Fkolab.2Foriginal.27|'/vendor/kolab/original']] message annotation, preserving '''all''' its message annotations;
+* update the object in the activated search folder updating its [[#New_annotation:_.27.2Fvendor.2Fkolab.2Foriginal.27|'/vendor/kolab/original']] message annotation;
+* use the [[#New_annotation:_.27.2Fvendor.2Fkolab.2Fmatches.27|'/vendor/kolab/matches/']] message annotation of the original object to update the object in all other folders that contain it and to which the client has write access while updating the respective [[#New_annotation:_.27.2Fvendor.2Fkolab.2Foriginal.27|'/vendor/kolab/original']] message annotations.
+
+==== Active ====
+In addition to the provisions set forth in the [[#Passive|Passive]] section, Active Clients '''MUST''' also
+* be capable of executing a search specified in the [[#Query_Language|Query Language]] over all object types;
+* tag matching objects in the original folders with the appropriate [[#New_annotation:_.27.2Fvendor.2Fkolab.2Fmatches.27|'/vendor/kolab/matches/']] message annotation;
+* copy objects to the respective search folder(s);
+* tag cached objects in the search folders with the appropriate [[#New_annotation:_.27.2Fvendor.2Fkolab.2Foriginal.27|'/vendor/kolab/original']] message annotation.
+
+Upon opening a search folder, the client '''MUST''' perform all the steps outlined in the the [[#Passive|Passive]] section, followed by a search over objects newer than 'lastrun' in the [[#New_annotation:_.27.2Fvendor.2Fkolab.2Fsearch.27|'search' annotation]] and cache/tag objects accordingly. The 'lastrun' field in the [[#New_annotation:_.27.2Fvendor.2Fkolab.2Fsearch.27|'search' annotation]] '''SHALL''' be updated as the '''first''' thing when the client begins the search.
+
+=== Path Conventions ===
+The search functionality '''MUST''' be able to uniquely refer to a folder or message on an IMAP server regardless of which credentials are being used to access the data. This is necessary because a shared search by one user '''MUST''' be able to reference folders that ought to be searched, and it '''MUST''' be possible to reference to individual messages as search results or the original object will desynchronize. 
+
+The absolute path to an object '''SHALL''' be defined as
+ 
+  /<namespace>/<path>
+ 
+  where 
+ 
+  <namespace> = {user, shared}
+  <path>      = path to object, including user name in 'user' namespace
+ 
+
+==== Examples ====
+IMAP object 15 in the main Calendar of user joe would become the absolute path
+
+  /user/joe/Calendar/15
+
+and object 32 in the main Inbox of user alice would become the absolute path
+
+  /user/alice/32
+
+which would be the absolute path for ''all users'', joe & alice included. So if alice is connecting to the server, her client '''MUST''' translate the absolute path of '/user/alice/32' into the relative path of '/inbox/32' while retaining the full path '/user/joe/Calendar/15' for first example.
+
+Folders in the shared hierarchy, e.g. object 14 in a folder /company/meetings/
+
+ /shared/company/meetings/14
+
+are the same for all users.
+
+=== Query Language ===
+{{note|This is still an open question|We need to figure out which query language across clients we could/should be using as this needs to be consistent in order to get reproduceable results regardless of client.}}
+
+SPARQL? Subset of SPARQL? How to define something that works across all clients?
+
+=== RFC 5257 Annotations ===
+Based on {{rfc|5257}}<ref name="rfc5257" /> and KEP #9<ref name="kep9" />, this KEP defines the following new message annotations '/vendor/kolab/original' and '/vendor/kolab/matches'.
+
+==== New annotation: '/vendor/kolab/original' ====
+Objects in folders of [[#New_value_.27search.27_for_.27.2Fvendor.2Fkolab.2Ffolder-type.27|folder-type 'search']] '''MUST''' reference the object of which they are a copy by using the '/vendor/kolab/original' message annotation. The shared value of the annotation '''SHALL''' be the absolute path of the original object according to the [[#Path_Conventions|Path Conventions]] defined in this KEP.
+
+==== New annotation: '/vendor/kolab/matches' ====
+Original objects that match a saved search '''MUST''' have the shared value of their '/vendor/kolab/matches' message annotation set to a JSON ({{rfc|4627}}<ref name="rfc4627">{{rfc|4627|title=The application/json Media Type for JavaScript Object Notation (JSON)}}</ref>) UTF-8 encoded list of all the saved searches they match. The list '''SHALL''' reference saved searches as the absolute path to the corresponding search folders according to the [[#Path_Conventions|Path Conventions]] defined in this KEP.
+
+=== RFC 5464 Annotations ===
+Based on {{rfc|5464}}<ref name="rfc5464" /> and KEP #9<ref name="kep9" />, this KEP defines the following additional value for '/vendor/kolab/folder-type' and new folder annotation '/vendor/kolab/search'.
+
+==== New value 'search' for '/vendor/kolab/folder-type' ====
+Direct subfolders of the INBOX can have their '/vendor/kolab/folder-type' annotation set to 'search', declaring them folders that '''SHALL''' store search queries and '''MAY''' be prepopulated with the results of previously executed searches. 
+
+==== New annotation: '/vendor/kolab/search' ====
+Folders of [[#New_value_.27search.27_for_.27.2Fvendor.2Fkolab.2Ffolder-type.27|folder-type 'search']] '''MUST''' also carry the shared annotation '/vendor/kolab/search' detailing the parameters and status of the search. The value '''SHALL''' be a a JSON ({{rfc|4627}}<ref name="rfc4627" />) UTF-8 encoded array with the following key/value pairs:
+  
+  query:	The stored query in [[#Query_Language|Query Language]] format
+  		(Details what to search for, where, in which kinds of object types)
+  caching: 	Whether the folder shall be prepopulated with copies of the objects matching the query
+  		(1: yes, 0: no)
+  lastrun: 	KEP #2<ref name="kep2">[[KEP:2]] Modification of datetime: store local time, add 'tz' attribute</ref> compliant date-time of ''start'' of last search in UTC
+  		(for prepopulating search or non-prepopulating search alike)
+
+
+
+== Upgrade Path ==
+
+== Rationale ==
+{{note|Full writeup still missing|This will need to be fully written up, right now it is a mix of fully fledged out and shorthand points. It is also still quite incomplete.}}
+
+* Must support all object types: The mechanism in this KEP should work for all types of data stored in a Kolab Server, including email, calendar, address book, tasks, journal '''LEADS TO:''' own folder type, cannot use Kolab XML to store object specific metadata
+* Prepopulation should still allow deduplication on server '''LEADS TO:''' must not modify headers of emails objects should be kept as identical copies, use message annotations instead
+* Objects must not desynchronize and Akonadi only symlinks '''LEADS TO:''' link back with original message annotation
+* Objects should be faster found even without prepopulation (may be too space-expensive on client where there may be no deduplication) '''LEADS TO''' use message annotation to store matches, which can be searched with regular IMAP queries
+* All clients should find the same things when executing the same search, and saved searches should be executable by all clients '''LEADS TO''' define shared query semantics
+* It is not clear over how many users a query will be shared, it can be one or thousands '''LEADS TO''' absolute paths required
+
+
+
+== References ==
+
+{{Reflist}}
+
+== Copyright ==
+
+This document has been placed in the public domain.





More information about the commits mailing list