[Kolab-devel] Kolab 3.0: Mime Message Storage

[Christian Mollekopf]

On Wednesday 16 May 2012 21.58:07 Jeroen van Meeuwen wrote:
> On Wednesday, May 16, 2012 09:58:57 PM Christian Mollekopf wrote:
> > On Wednesday 16 May 2012 16.55:24 Jeroen van Meeuwen wrote:
> > > I suggest re-phrasing the entire section so that it is clear that;
> > > 
> > > - All folders with Kolab Groupware content MUST be annotated, while
> > > folders
> > > with mail contents MAY exist but without metadata, it is assumed folders
> > > without metadata are mail folders, and clients SHOULD set metadata for
> > > folders that have some special sort of mail (Drafts, Trash, Sent,
> > > etc...).
> > > 
> > > - the difference between ANNOTATEMORE (the draft) and METADATA (the RFC)
> > > doesn't matter for the verbiage of this part of the format definition,
> > > 
> > > - The shared namespace (/vendor/kolab/ value.shared in ANNOTATEMORE,
> > > /shared/vendor/kolab/ in METADATA) is good for groupware types, while
> > > sub-
> > > types are usually best set in the private namespace,
> > > 
> > > """All annotation MUST be set during the initial creation of a folder
> > > and
> > > cannot be altered afterwards"""
> > > 
> > > They MUST be set alright, but SHOULD NOT (or even MUST NOT) be altered
> > > afterwards. It's not like they "cannot be changed".
> > > 
> > > Also, consider that this would apply to the type only, and not (equally
> > > well) to the sub-type.
> > > 
> > > Folders without any metadata MUST (not SHOULD) be treated as (not
> > > assumed
> > > to be) mail folders.
> > > 
> > > ===
> > > 
> > > """All "default" Kolab-Folders MUST be located within the personal IMAP
> > > namespace. """
> > > 
> > > I don't think that MUST holds true, and I don't think we need to
> > > restrict
> > > '.default' sub-types to only ever be allowed in the personal IMAP
> > > namespace.
> > > 
> > > What must be considered is any personal IMAP namespace '.default' folder
> > > takes precendence over any other, and that there may only be one (or
> > > less,
> > > depending on opinions on the aforementioned) such personal IMAP
> > > namespace
> > > '.default' folder.
> > > 
> > > I would also consider adding that, if multiple .default folders exist,
> > > either "in the personal namespace" or "none in personal but multiple in
> > > the
> > > shared namespace" clients SHOULD pretend nothing has been marked as
> > > '.default' and behave accordingly (i.e. always pop up a "save to what
> > > folder?" type of dialog?).
> > 
> > I changed the draft accordingly. If I understand correctly, that means the
> > annotations are not really backwards compatible thought.
> 
> It might not be backwards compatible, but nor is the format. If a client
> fails to update its routines wrt. folder metadata, it's not necessarily
> prohibitively incompatible.
> 
> > However, I agree that it makes sense to have the groupware type as shared
> > annotations and the rest as private annotation, and now is the time to
> > make
> > the changes.
> 
> It's been one of those things that had to be refined it its definition.
> 
> > We might want to also allow all annotations (also default, inbox, draft,
> > etc.) as shared annotations, with private annotations taking precedence
> > over shared ones. This would allow for defaults in the folders in the
> > shared namespace.
> 
> I'm not sure what a shared folder marked as INBOX establishes exactly, but
> 'spam' and 'ham' come to mind as good use-cases for mail folder-type
> metadata in the shared namespace.
> 

Indeed.

> > > ===
> > > 
> > > """TODO: we probably should have a bit a firmer specification how
> > > multiple
> > > folders SHOULD be interpreted for each type to get a consistent user
> > > experience."""
> > > 
> > > This remark doesn't make much sense, perhaps you care to elaborate?
> > 
> > I mean we should maybe give a better specification/guideline how folders
> > should be interpreted.
> > 
> > In the Akonadi Resource it's not that interesting because akondi allows us
> > to map the imap folders 1:1 to the calendar folders. (We can have
> > subfolders for calendars, notes, and addressbooks, and we can even create
> > calendar folders for journals only).
> > 
> > I don't know if that is possible for all systems, because an ical calendar
> > is typically for events, todos and journals. So it might be that not all
> > systems provide such fine-grained control (especially when it comes to
> > mobile devices), and could make use of some guidline. e.g. display all
> > folders containing calendar items as a single calendar and store new items
> > to the folder  which was marked as default.
> > 
> > But since it is very straight forward for our main client, I suppose we
> > don't really need to think about that now and can just tackle it when we
> > are facing the issue.
> 
> Right... that doesn't make any more sense than it did already, sorry ;-)
> 
> Suppose we have one or more hierarchies of calendars, address books,
> notebooks, and such and so forth... which are the aspects of such
> hierarchies that could benefit from a better specification or set of
> guidelines, you reckon?
> 

Well, for instance you can interpret a folder containing tasks as a tasklist, 
or just as a folder, or you don't care about the folder structure anyways and 
merge all tasks together in one big list. There are different concepts, which 
could be expressed using folders, and if we were to heavily rely on such 
concepts we better specify them to have the chance of a consistent user 
experience.

But for now we can just leave it at that (I think).

> > > ===
> > > 
> > > As to the X-Kolab-* headers;
> > > 
> > > X-Kolab-Type is a duplicate of the Content-Type header to the
> > > attachment.
> > > I
> > > don't really see what purpose this serves, as in addition to the
> > > duplication, the object is also only supposed to be stored in a folder
> > > marked as containing rfc822 encapsulated payload of a particular type.
> > 
> > The Content-Type header is the same for all xcal/xcard headers and doesn't
> > give us this differentiation. And doing that by the xml is painful to say
> > the least.
> > 
> > Since we have separate folders for the content types, it would be fairly
> > easy to change our implementation, it could be a problem for other clients
> > though, depending on how the code is written. If we remove the
> > X-Kolab-Type
> > header it is also no longer to automatically convert a single mime message
> > (without the context of the folder), without specifying the type manually.
> > 
> > Generally I would rather keep it, as it IMO is a useful hint for client
> > implementors which don't have the foldertype available right away and
> > allows to double check the type. Also I'm not that fond of relying on the
> > folder type to "guess" the kolab object type.
> 
> I understand where it comes from, but I doubt whether it's needed.
> 
> When a client without any (current) caches opens up a folder that contains
> the same MIME content-type attachments, it does need to read all
> attachments in order to get to the display name, summary, duration and
> whatnot, right? What use does a X-Kolab-Type header have if all attachments
> need to be read regardless?

It contains the same Kolab-Objects stored as MIME message. Of each message it 
only needs to read the xml attachment. If you don't have the X-Kolab-Type 
header you need to rely on the folder, which you're currently processing to 
know what object to expect.

That is an unfortunate implementation detail, as ideally it would be possible 
to just get the Content-Type of the attachment, see it is 
"application/calendar+xml" and call readxCal(), which returns an 
event/todo/journal. But instead we have to call readEvent, readTodo, 
readJournal explicitly, which is what we need the exact type for.

> 
> Do we have clients that only ever want the distribution lists out of a set
> of contacts, or is it merely a step client applications have gotten used to
> making to separately query for all distribution lists and then separately
> query for all (individual) contacts?
> 

I'm not aware of such clients. But for contacts/dist-lists it's actually not 
even possible to rely on the folder type, so we need the X-Kolab-Type 
definitely.

> > > X-Kolab-Version is a duplicate of the version contained within the XML
> > > and
> > > should not be used.
> > 
> > It is not IMO, as it specifies the format of the mime message format. I'm
> > in favor of keeping it.
> 
> In that case, let's make it the full $x.$y.$z, so various version comparison
> routines can parse it similarly to how they (could) parse the $x.$y.$z in
> the XML, as opposed to an arbitrary string.
> 
> Let's suppose we don't allow the mime message format to be changed between
> teeny releases; only $x.$y then needs to be stored.
> 
> Also, wouldn't the version of libkolabmime be more appropriate to use here?
> 

That is libkolab right now, which is stored already as User-Agent. I think we 
need the version number of the specification there (I started with 3.0, as the 
Kolab Format 2.0 specification included the Mime Format specification).

> Perhaps, also, it's best to slap a name on the header that actually
> represents the fact it's about the mime message structure.
> X-Kolab-Mime-Version if you will.
> 

Sure, that is cleaner.

> > > ===
> > > 
> > > """The following headers/options MUST be used:"""
> > > 
> > > I made this a separate bullet point, and second, I see no reason to
> > > restrict the charset or encoding.
> > > 
> > > This brings me back to the bullet point just before this one, too. I
> > > suppose a localized version of said text could be allowed, as long as
> > > the
> > > English version is somewhere?
> > 
> > I thought that ascii-us is somewhat the industry standard for emails, I
> > might be wrong though. In any case I'm ok with allowing also other
> > charsets
> > as it's not a functional but puerly informal part anyways.
> 
> Right. My point was to not superfluously define things we shouldn't actually
> care all that much about.
> 

Agreed, good point.

> > > ===
> > > 
> > > """The following headers/options MUST be used:"""
> > > 
> > > I made this a separate bullet point, and second, again I see no reason
> > > to
> > > restrict the encoding to quoted-printable, nor a reason to restrict the
> > > filename to kolab.xml.
> > 
> > IMO there is no reason to not enforce the encoding, and as every client
> > must be able to read all supported encoding types, I'd rather just use
> > one. I don't really care about the filename, but see no reason not to
> > just specify one, which keeps it at least consistent.
> 
> Same as the above for the encoding. The filename question is more of a "save
> all to disk" thing to be honest, and unless everything's hard coded to look
> for a particular Kolab XML attachment by detecting which attachment has
> filename "kolab.xml", it makes little sense to specify and restrict.
> 

Agreed for the filename. For the encoding there is the same reason to only 
allow one encoding as there is for writing KEP#17 (firmer specification that is 
implementable).

> Perhaps, MUST can become SHOULD, and "here's an example" for both this one
> and the aforementioned one.
> 
> > > """MIMETYPE MUST be one of (as specified in KEP #17):"""
> > > 
> > > I think this is inconsistent with the aforementioned x vnd kolab
> > > headers?
> > 
> > No, respectively should be =)
> > The X-Kolab-Type specifies the exact Kolab Object type, where the
> > Content-Type only specifies the mime-type of the format with which the xml
> > object object is stored. That means if we have a v2 and a v3 event they
> > have the same X-Kolab- Type but different Content-Types.
> > 
> > > ===
> > > 
> > > Attachments SHOULD be stored separately but MUST be referenced? Perhaps
> > > a
> > > little fine-tuning of the verbiage is justified... ;-)
> > > 
> > > ===
> > > 
> > > The example is... large.
> > 
> > Yes, I just copied a testfile to be honest. I
> > 
> > > For demonstrative sake, can we insert a "may remove
> > > carriage returns and superfluous whitespace", if not simply a "base64
> > > encoding allowed" perhaps (even though it is kinda nice to be able to
> > > read
> > > as a human being) - just offering this one up for consideration.
> > 
> > I don't follow here:
> > "may remove carriage returns and superfluous whitespace" where?
> 
> At the end of every line and the indentation, respectively. The "markup" as
> it is only matters to human eye-balls glancing over the raw spool payload,
> no?

That is true.

Cheers,
Chrisitian

> > Regarding the base64, as I'm not in favor of allowing this, is there any
> > specific reason to do so?
> 
> Not really.
> 
> Kind regards,
> 
> Jeroen van Meeuwen
-- 
Christian Mollekopf
Software Engineer

Kolab Systems AG
Zürich, Switzerland

e: mollekopf at kolabsys.com
w: http://kolabsys.com

pgp: EA657400 Christian Mollekopf
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.kolab.org/pipermail/devel/attachments/20120517/bb16ad55/attachment.sig>