KEP-0014.txt

Georg Greve greve at kolabsys.com
Sun Sep 4 19:38:24 CEST 2011


 KEP-0014.txt |  200 ++++++++++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 157 insertions(+), 43 deletions(-)

New commits:
commit a1eeaff325f995b9d65d1e0acb28cfb8110bee00
Author: Georg Greve <greve at kolabsys.com>
Date:   Sun Sep 4 19:37:54 2011 +0200

    Draft No 2, after LOTS of updates and discussion. Beware, will contain glitches.

diff --git a/KEP-0014.txt b/KEP-0014.txt
index 502d2f6..a4fc59f 100644
--- a/KEP-0014.txt
+++ b/KEP-0014.txt
@@ -14,77 +14,171 @@
 
 == Abstract ==
 
-This Kolab Enhancement Proposal (KEP) <ref name="kep">[[KEP:1]] Bootstrapping the KEP process</ref> defines the conventions and application behaviour to enable non-conflicting edits of {{rfc|5228}}<ref name="sieve">{{rfc|5228|title=Sieve: An Email Filtering Language}}</ref> Sieve scripts on the server by multiple editors. It follows the basic ideas outlined in an email to the Kolab development mailing list in May 2011.<ref name="multisieve">Georg Greve: [http://kolab.org/pipermail/kolab-devel/2011-May/012846.html Enabling multiple sieve editors on Kolab installation]</ref>
+This Kolab Enhancement Proposal (KEP) <ref name="kep">[[KEP:1]] Bootstrapping the KEP process</ref> defines the conventions and application behaviour to enable non-conflicting edits of {{rfc|5228}}<ref name="sieve">{{rfc|5228|title=Sieve: An Email Filtering Language}}</ref> Sieve scripts on the server by multiple editors. It follows the basic ideas outlined in an email to the Kolab development mailing list in May 2011 <ref name="multisieve">Georg Greve: [http://kolab.org/pipermail/kolab-devel/2011-May/012846.html Enabling multiple sieve editors on Kolab installation]</ref> by making use of the 'include' functionality defined in the 'Sieve Email Filtering: Include Extension' draft<ref name="include">[https://datatracker.ietf.org/doc/draft-ietf-sieve-include/ Sieve Email Filtering: Include Extension]</ref>.
 
-== Non-Conflicting Sieve Editing ==
+The result is the ability to have different layers of Sieve management per user, with different levels of required authorization, and the ability to provide users with multiple active scripts which can be activated and deactivated as required, thus enabling splitting of the user-facing Sieve functionality into more conveniently managed files. For more information on what this KEP does or does not address, please see the [[#Rationale|Rationale]] section and its [[#Scope_of_this_KEP|Scope of this KEP]] subsection, in particular.
 
-In order to enable Non-Conflicting Sieve Editing according to this KEP, a [[#MASTER.siv|MASTER.siv]] file '''MUST''' be defined for all users during integration and adapted during deployment. This [[#MASTER.siv|MASTER.siv]] '''SHALL''' then be considered authoritative for that deployment. It '''SHALL''' be automatically deployed and activated for all users in that system that are to make use of the Non-Conflicting Sieve Editing.
+== Naming Conventions ==
+For the purpose of this KEP, all file names of Sieve<ref name="sieve" /> scripts '''SHALL''' be given in their 'basename' only, so without extension, which is customarily dependent upon the specific IMAP server used. Cyrus IMAP commonly uses '.siv', Dovecot commonly uses '.sieve', other servers or variations of these servers may use different conventions. These extensions '''MAY''' be added as required or customary for the particular installation, but only the basename '''SHALL''' be considered authoritative.
 
-=== Sieve Editor Requirements ===
+== Enabling Non-Conflicting Sieve Editing ==
+In order to enable Non-Conflicting Sieve Editing according to this KEP, a [[#MASTER|MASTER]] alongside an no-op [[#MANAGEMENT|MANAGEMENT]] and [[#USER|USER]] file '''MUST''' be defined during platform integration and adapted during deployment. The [[#MASTER|MASTER]] '''SHALL''' then be considered authoritative for that deployment. 
+
+It '''SHALL''' be automatically deployed and activated for all users in that system that are to make use of the Non-Conflicting Sieve Editing alongside the no-op [[#MANAGEMENT|MANAGEMENT]] and [[#USER|USER]] files.
 
-To be compliant with this KEP, Sieve editors '''MUST''' check for the presence of the [[#MASTER.siv|MASTER.siv]] file, and, if present, '''MUST''' do the following:
-* hide the [[#MASTER.siv|MASTER.siv]] script and all other [[#Protected Names|Protected Names]] from user selection;
-* protect the [[#Protected Names|Protected Names]] by choosing one of the following paths:
-** default to the editors' script '''ONLY''' and '''NOT''' allow users to have multiple scripts, '''--- OR ---''';
-** prevent the user from assigning any of the [[#Protected Names|Protected Names]]. Ways in which this '''MAY''' be achieved are through renaming or simply by informing the user that a certain script name is unavailable. Activating any given script '''MUST''' be done by '''copying''' it to carry the protected name assigned to this editor;
-* ensure that after finishing its modification(s), the [[#MASTER.siv|MASTER.siv]] is active.
+{{note|Why do we need dummy files?|The 'Sieve Email Filtering: Include Extension' draft specifies that the server MUST generate an error when trying to include non-existing files. Because [[#MASTER|MASTER]] is immediately active, typically before the management system or the user had any time to touch the Sieve scripts, empty files are required to avoid superfluous error messages and potentially break the functionality, e.g. when the user has created Sieve scripts, but the management system has not yet done so.}}
 
-At the choice of its author, a Sieve editor '''MAY''' visualize scripts that are executed before or after the script edited by the user. How this is implemented is largely up to the individual editors, but it should ensure that
+=== Sieve Editor Requirements ===
+To be compliant with this KEP, Sieve editors '''MUST''' check for the presence of the [[#MASTER|MASTER]] file, and, if present, '''MUST''' do the following:
+* hide [[#MASTER|MASTER]] and all other [[#Protected Names|Protected Names]] from user selection;
+* protect all [[#Protected Names|Protected Names]] by preventing users from choosing them for their own scripts;
+* define a consistent name and version to be used in the provisions outlined in [[#USER|USER]] and [[#User_specific_scripts|User specific scripts]];
+* upon startup read [[#USER|USER]] to get the number and order of active scripts;
+* regularly check for changes by other editors to [[#USER|USER]] or any of the active scripts;
+* edit scripts in line with the [[#User_specific_scripts|User specific scripts]] provisions;
+* overwrite [[#USER|USER]] to reflect activations/deactivations of [[#User_specific_scripts|User specific scripts]] by including/excluding them from the [[#USER|USER]] script;
+* ensure that after finishing its modification(s), [[#MASTER|MASTER]] is the active script.
+
+At the choice of its author, a Sieve editor '''MAY''' visualize scripts that are executed before or after the scripts edited by the user. How this is implemented is largely up to the individual editors, but it should ensure that
 * it is made clear to the user these rules cannot be edited within this editor, e.g. through graying out;
 * the user is given some indication as to where these rules can be edited, e.g. in the management console.
 
-If there is no [[#MASTER.siv|MASTER.siv]] file in the users list of Sieve scripts, editors '''MAY''' behave as they see fit, but '''SHALL''' always ensure its users do not use [[#Protected Names|Protected Names]] and '''MUST''' always ensure that users do not create a [[#MASTER.siv|MASTER.siv]] which would trigger the functionality according to this KEP.
+If there is no [[#MASTER|MASTER]] file in the users list of Sieve scripts, editors '''MAY''' behave as they see fit, but '''MUST''' always ensure its users do not use [[#Protected Names|Protected Names]] and '''MUST''' always ensure that users do not create a [[#MASTER|MASTER]] which would trigger the functionality according to this KEP.
 
-=== Integration Requirements ===
+=== Management system requirements ===
+To be compliant with this KEP, management systems that make use of Sieve for some of their functionality '''MUST''' check for the presence of the [[#MASTER|MASTER]] file, and, if present, '''MUST''' do the following:
+* limit themselves to the [[#MANAGEMENT|MANAGEMENT]] script;
+* define a consistent name and version to be used in the provisions outlined in [[#MANAGEMENT|MANAGEMENT]];
+* write out the [[#MANAGEMENT|MANAGEMENT]] script as defined.
 
-Integrators and Kolab Certified Professionals responsible for the deployment '''SHALL''' ensure that if the platform or deployment intends to make use of this feature, users which are supposed to make use of this feature will have their [[#MASTER.siv|MASTER.siv]] file created and activated upon account creation. 
+If there is no [[#MASTER|MASTER]] file in the users list of Sieve scripts, management systems '''MAY''' behave as they see fit, but '''MUST''' always ensure its users do not use [[#Protected Names|Protected Names]].
 
-This can be in the form of adding this to the default behaviour for all users, by tying this to some role or group in LDAP, or whichever other approach the platform takes on configuration.
+If the management system aims to also control (de-)activation of the functionality defined in this KEP through creation or deletion of the [[#MASTER|MASTER]] script, it '''MUST''' ensure that it writes the [[#MASTER|MASTER]] as defined authoritatively for this platform or group of users, and '''MUST''' always write the [[#MANAGEMENT|MANAGEMENT]] script as defined herein.
 
-Sieve editors '''MUST''' be able to rely on the proper creation of the [[#MASTER.siv|MASTER.siv]] script before they are called for the first time. It is the responsibility of the platform integrator or deployment engineer to ensure this is done properly.
+Sticking to the conventions for [[#MANAGEMENT|MANAGEMENT]] scripts is recommended for all management systems in order to facilitate easier deployment/integration with the Kolab Groupware Solution.
 
-=== MASTER.siv ===
+=== Integration Requirements ===
+Integrators and Kolab Certified Professionals responsible for the deployment '''SHALL''' ensure that if the platform or deployment intends to make use of this feature, users which are supposed to make use of this feature will have their [[#MASTER|MASTER]] file created and activated upon account creation alongside the no-op [[#MANAGEMENT|MANAGEMENT]] and [[#USER|USER]] files.
+
+This can be in the form of adding this to the default behaviour for all users, by tying this to some role or group in LDAP, or whichever other approach the platform takes on configuration.
 
-This is an example of the MASTER.siv script, which can act as a template and '''SHALL''' be modified during integration and deployment as required for the particular installation:
+Sieve editors '''MUST''' be able to rely on the proper creation of the [[#MASTER|MASTER]] script before they are called for the first time. It is the responsibility of the platform integrator or deployment engineer to ensure this is done properly.
+
+=== MASTER ===
+This is an example of the MASTER script, which can act as a template and '''SHALL''' be modified during integration and deployment as required for the particular installation. 
 
  #
- # MASTER.siv
+ # MASTER
  # 
- # This file is automatically generated and MUST BE the ACTIVE script.
+ # This file is authoritative for your system and MUST BE KEPT ACTIVE.
  #
- # Altering it by hand is likely to BREAK YOUR INSTALLATION!
+ # Altering it is likely to render your account dysfunctional and may
+ # be violating your organizational or corporate policies.
  # 
- # For more information, see http://wiki.kolab.org/KEP:14
+ # For more information on the mechanism and the conventions behind
+ # this script, see http://wiki.kolab.org/KEP:14
  #
  
  require ["include"];
  
- # Some global include for all users in the installation
- # include :global "all-users.siv";
- 
+ # OPTIONAL: Includes for all or a group of users
+ # include :global "all-users";
+ # include :global "this-group-of-users";
+
  # The script maintained by the general management system
- # include :personal "<MANAGEMENT>.siv";
+ include :personal "MANAGEMENT";
  
  # The script(s) maintained by one or more editors available to the user
- # include :personal "<EDITOR>.siv";
+ include :personal "USER";
+
+MASTER, [[#MANAGEMENT|MANAGEMENT]] and [[#USER|USER]] '''SHALL''' be receiving their file extension as required and appropriate for the IMAP server used.
+
+MANAGEMENT is the script written by the management system of the platform, USER is the script intended for controling the execution of individual scripts by a user. General sieve editors available to users and platform management systems '''MUST NOT''' alter this script.
+
+=== MANAGEMENT ===
+Management systems '''SHALL''' define the variables 'global.management_system' and 'global.management_system_version' to correspond to their name/platform and version somewhere in the MANAGEMENT script by using the Variables Extension according to {{rfc|5229}}<ref name="variables">{{rfc|5229|title=Sieve Email Filtering: Variables Extension}}</ref>.
+
+Otherwise management systems are free to write to each user's MANAGEMENT script as required and appropriate.
+
+General sieve editors available to users '''MUST NOT''' alter this script.
+
+=== USER ===
+Editors '''SHALL''' be reading in the USER script, parsing it for individual includes of user-defined scripts, which take the form of
+
+  ## <EDITOR NAME>
+  ## <EDITOR VERSION>
+  include :personal "<filename>"
+
+where <EDITOR NAME> is the name of the editor that wrote this include, <EDITOR VERSION> is its version, and <filename> is the file name of the script written by this editor of this version for the current user.
 
-<MANAGEMENT> and <EDITOR> '''SHALL BE''' placeholders for the actual names of the platform and/or editor.
+Parsing '''SHALL''' preserve all comments that do not conform to the above format and discard the rest of the file.
 
-<MANAGEMENT>.siv is the script written by the management system of the platform and there can be one or more <EDITOR>.siv scripts used by the canonical set of defined Sieve editors for users of the system. Because these editors will generally '''not''' see each other's rules, it is strongly recommended to use just one user-facing editor.
+When replacing the USER script on the server, editor '''SHALL''' first write out all preserved comments, followed by an empty line, and then the client side includes, each followed by an empty line. The order of entries '''SHALL''' always be preserved unless explicitly changed by the user.
+
+==== Sample USER Script ====
+
+  # USER Master Script
+  #
+  # This script includes the various active sieve scripts
+  # it is AUTOMATICALLY GENERATED. DO NOT EDIT MANUALLY!
+  # 
+  # For more information, see http://wiki.kolab.org/KEP:14#USER
+  #
+  
+  ## Roundcube Sieve Plugin
+  ## 0.6
+  include :personal "superspamrules"
+  
+  ## Roundcube Sieve Plugin
+  ## 0.6
+  include :personal "mailinglists"
+  
+  ## Horde Ingo
+  ## 2.0.5
+  include :personal "personalmail"
+  
+  ## Roundcube Sieve Plugin
+  ## 0.6
+  include :personal "workmail"
+  
+
+=== User specific scripts ===
+User specific scripts contain the actual Sieve commands as defined by the user through the editors. When reading in such a script, editors '''SHALL''' parse it for the existance of {{rfc|5229}}<ref name="variables" /> variables for 'global.editor' and 'global.editor_version'.
+
+If at least one of these variables is not set, editor '''SHALL''' assume that the file is not under control of any particular editor and '''SHOULD''' import it as best it can and claim authority over it by setting the variables to its own name and version when replacing the script on the server. Alternatively, the editor '''MAY''' consider the script immutable '''--- AND/OR ---''' suggest it for deletion by the user.
+
+If the variables are set propery, editor '''MUST''' consider the format of that editor authoritative and keep the file compliant to the format of the specified editor, although editor '''MAY''' update the format of the file to a newer version of that same editor when replacing the script on the server. Editor '''MAY''' only edit the script if it can fully parse and edit the file accordingly, otherwise editor '''MUST''' consider the script read-only.
+
+Manually edited scripts '''MUST''' be designated with the value 'MANUAL' for editor and 'NONE' for its version and '''MUST''' always be considered read-only by all editors.
+
+==== Sample user specific script ====
+
+A manual script example:
+
+  # Spam Rules, manually defined
+  require ["variables", "fileinto", "comparator-i;ascii-numeric", "imapflags", "regex"];
+  global "editor";
+  global "editor_version";
+  set "global.editor" "MANUAL";
+  set "global.editor_version" "NONE";
+ 
+  if anyof ( not exists ["From","Date"],
+	   header :comparator "i;ascii-casemap" :contains "X-Spam-Level" "*****",
+	   header :is ["X-Spam-Flag", "X-Spam-Status"] ["YES","Yes","yes"] ) {
+    setflag "\\Seen";
+    fileinto "INBOX/Spam";
+    stop;
+  }
+ 
 
 === Protected Names ===
 
-The convention is that platform / management scripts '''SHALL BE''' using names all in upper-case, similar to the '''MASTER.siv''' script, while user-facing editors '''SHALL BE''' using names all in lower-case.
+All protected names '''SHALL''' be protected in their basename and all extensions, e.g. the listing of MASTER in the table below '''MUST''' be read to also protect MASTER.sieve, MASTER.siv explicitly, and more generally MASTER.*.
 
-Protected names under this KEP '''SHALL BE'''
-* 'MASTER.siv' for the primary script, which '''SHALL''' be defined at integration stage authoritatively for all users and kept active for each user at all times;
-* 'BCS.siv' for the script written out by the Bynari Collaboration Suite<ref name="bcs">[http://www.bynari.net/bynari.php?page=collaboration Bynari Collaboration Suite]</ref>;
-* 'COREBIZ.siv' for the script written out by the CoreBiz Groupware Management Console<ref name="cb">[http://www.linux-ag.com/produkte/cb-groupware/ CoreBiz Groupware]</ref>;
-* 'UCS.siv' for the script written out by the Univention Management Console on the Univention Corporate Server (UCS)<ref name="UCS" />;
-* 'WEBADMIN.siv' for the script written out by the standard Kolab webadmin of the Kolab Groupware Server<ref name="webadmin">[http://kolab.org Kolab] Server Webadmin</ref>;
-* 'ingo.siv' for the script written out by the Horde<ref name="ingo">[http://www.horde.org/apps/ingo/ Horde: Ingo]</ref> Sieve editor 'Ingo';
-* 'kontact.siv' for the script written out by the KDE Kontact<ref name="kontact">[http://community.kde.org/KDE_PIM KDE Kontact]</ref> Sieve editor;
-* 'rc.siv' for the Roundcube<ref name="rc">[http://roundcube.net/ Roundcube]</ref> Sieve management plugin.
+Protected names under this KEP '''SHALL''' be
+* 'MASTER' for the primary script, which '''SHALL''' be defined at integration stage authoritatively for all users and kept active for each user at all times;
+* 'MANAGEMENT' for the script written out by management system of the system;
+* 'USER' for the control script of user editable scripts.
 
 == Upgrade Path ==
 
@@ -92,15 +186,35 @@ This KEP introduces new conventions.
 
 Older clients must be expected to ignore these conventions, consequently breaking the functionality in ways that will be undesirable to the user, but should normally not result in data loss.
 
+=== Updates of 'Sieve Email Filtering: Include Extension' draft ===
+
+This KEP '''SHALL''' remain authoritative for the latest version of the latest draft of 'Sieve Email Filtering: Include Extension' draft<ref name="include" /> and ultimately the resulting RFC, provided no revision introduces functionality incompatible with this KEP, in which case the ''last compatible version'' '''SHALL''' be authoritative until this KEP can be obsoleted and replaced by a subsequent Kolab Enhancement Proposal.
+
 == Rationale ==
+There are multiple issues around Sieve (RFC 5228<ref name="sieve" />) scripts. This KEP will provide a workaround for '''some''' of these issues, but it cannot address all of them.
+
+=== Issue No 1 ===
+A major issue is the incompatibility of multiple Sieve editors with each others' output. Due to the complexity of Sieve, editors typically only implement a subset of how certain rules are expressed, typically oriented around how the editing of these rules is presented to the user. When presented with an arbitrary Sieve script written by another editor or human being, editors typically '''CANNOT''' just parse this script and present it visually for editing.
+
+Reasons for this are likely the complexity and flexibility of the language, which make it hard for clients to parse existing scripts into their internal model and representation of what kind of filtering is supposed to take place how and when. To circumnavigate this issue and allow users to edit existing scripts, editors typically have some metainformation about the rules that have been expressed in Sieve which are modeled after their own approach to editing these scripts. That information is either stored in a separate data store, such as the file system or an SQL based data base, or injected into the Sieve script itself as comments.
+
+This fundamental incompatibility is the reason that it is very difficult for two different Sieve editors to work on the same script, and the reason why usually only one Sieve editor is authoritative in any one installation. 
+
+=== Subsequent Issue ===
+The Kolab Groupware Server often finds itself integrated into existing products which provide their own user and system administration infrastructure. This infrastructure typically also involves some level of vacation notice and/or spam management which may be the preferred way for customers to make use of such functionality. 
+
+That functionality is typically implemented through Sieve scripts. But because there can ever only be one active Sieve script, and because editors don't mix, this often means that installations have to disable all Sieve based filtering for their users in order to protect the managed functionality.
 
-The Kolab Groupware Server often finds itself integrated into existing products such as Univention Corporate Server<ref name="UCS">Univention: [http://www.univention.de/produkte/ucs/ UCS]</ref> which provide their own user and system administration infrastructure. This infrastructure typically also involves some level of vacation notice and/or spam management which may be the preferred way for customers to make use of such functionality. 
+=== Issue No 2 ===
+With the Sieve: Include Extension<ref name="include" />, Sieve knows two namespaces, a global one shared among all users, and a local one for each user. Global scripts can be included locally, but are never executed automatically, while local scripts are always editable by the user.
 
-Simultaneously, users should be able to filter their mail into mailboxes with server-side Sieve (RFC 5228) scripts.
+For corporate environments, this is not always desirable. Admins may want to set scripts from the global context to be executed or even have user specific scripts that are being executed in a way that does not allow the user to accidentally or intentionally disable these scripts. The inability of Sieve to protect local scripts from editing, or define global scripts that are always executed then results in a similar workaround to the one described in the [[#Subsequent_Issue|Subsequent Issue]], above: Access to Sieve is blocked in its entirety for all users.
 
-Because there can typically only be one sieve script active at any point in time, different editors for these scripts typically invalidate each other, as editing another editor's script is generally unsafe and a recipe for desaster, and thus an editor edits its own script and activates it, consequently deactivating the script of another editor.
+=== Issue No 3 ===
+Sieve scripts can get fairly complex. It would be much more convenient and easier to manage the functionality with different scripts that define certain subsets of functionality and get executed as required. This was the original intention behind the 'Sieve Email Filtering: Include Extension' draft<ref name="include" />.
 
-Providing conventions that will allow to avoid these mutually invalidating scenarios is the purpose of this KEP.
+=== Scope of this KEP ===
+This KEP addresses the [[#Subsequent_Issue|Subsequent Issue]] and [[#Issue_No_2|Issue No 2]] through a workaround that is largely based on convention and procedure in a way that keeps the resolution of [[#Issue_No_3|Issue No 3]] through the Include Extensiont<ref name="include" /> intact. It does nothing to address [[#Issue_No_1|Issue No 1]], which is out of its reach, although hopefully the workaround may help to bring some convergence in the field of editors and consequently contribute to a solution for that issue some time in the future.
 
 == References ==
 





More information about the commits mailing list