Extra Header field

[Martin Konold]

Am Montag, 12. Juli 2004 19:37 schrieb Stuart K. Bingë:

Hi Stuart,

> Once a UID is initially assigned to an object, it uniquely identifies that
> object over its lifetime. 

Yes.

> Throughout this period, due to modification of 
> the object, it may be contained in several different IMAP messages, all
> with different IMAP UIDs. 

Yes, but it is always one after the other. There shall not be two imap 
messages for the same UID in the common case (conflicts aside)

> However, the UID that we assign to the object on 
> its creation stays constant throughout all this - and this is in fact the
> only way I can reference that specific object.

The imap uid of an object only changes in case of a modification which 
technically results in a new IMAP message. In case the modification is done 
by the Horde client you already know about this and in case it sis done by 
some other client the server tells you (unsolicited server message). If your 
client cannot handle unsolicated messages you can "poll" the server for new 
messages. This avoids that you can miss any modification/change.

> When loading in the list of objects, all that I can do is pass Horde an
> array containing the UIDs of all the objects that were present, along with
> some cursory information about the objects so that a listing page can be
> displayed.

What is a cursory information?

> Once Horde has generated this page, and a user clicks on an 
> object of interest, Horde again calls my driver to "fully" load the object
> of interest - and all I am given to reference this object is its UID.

Yes, and in order to know which message you have to fetch from the server you 
need to keep a map of the UID,imap UID. If the message changed in the 
meantime (fortunately not so common) you must check for any new messages on 
the server. The later allows you to update your  "local mapping".

Only three things can happen:
1. No modification happened (this is the common case and the imap uid is 
already known and valid)
2. Some modification happened (less common case) and you must read all new 
messages with imap uid > max imap uid (FETCH MAXUID:*). When doing so two 
cases can happen
2.1 The object you are interested in got modified and you find the new imap 
uid in the newly fetched messages.
2.2 The object got deleted (and maybe also alread expunged though this does 
not matter). In this case an UID in the Subject header would also not help 
anymore.

> My scripts do not exist/are not running between these page loads, 

This is a traditional issue with web applications.

> therefore 
> for either of these calls I have no idea what was previously loaded. All I
> know is what I need to load.

I assume that you keep a session for that purpose. The actual uid,imap uid 
mapping is not much data. An imap UID is 32bit. 

Technically in order to be able to correctly present a calendar you must parse 
_every_ message of the calendar folder anyway. So doing the mapping while 
parsing is no big extra effort.

BTW: Please don't forget that you MUST also check the uid validity of a 
folder!

> Furthermore, when I modify a message (or any other client modifies it, for
> that matter), its IMAP UID changes - if this is the UID I then passed to
> Horde, and a user requests to view that specific object, I'll be stuck in
> the situation of loading the incorrect data *and* not knowing that I am
> doing so. This is why I need to ensure I have some sort of "true" UID that
> I can use to reference the object.

This true UID is embedded in the message. The imap uid is never reused and 
grows monotonically. So when fetching an imap message via the imap uid it is 
guaranteed that as long as this fetch succeeds that it is always the very 
same unmodified message.

> Having the UID in the subject would not solve the problem - that is what we
> have the <uid> tag in the XML definition for. The reason I would want the
> UID in the Subject or in an X-Header is purely for efficiency - both in my
> code and in terms of load on the server - as I can then IMAP SEARCH for the
> object of interest instead of having to manually parse every message in the
> mailbox to find the one I'm looking for.

When using the imap uid you can directly fetch the message without any need 
for a search. In case the message with this imap uid vanished you exactly 
know this it is a modified object. You would not know that the object 
actually changed in case you would do the "subject search" operation.

> This is in fact very similar to 
> the issue that Joon is having with the "content" header.

Hmm... I think the isssue is different as you try to identify individual 
messages and Joon only tries to avoid extra parsing in case he cannot rely on 
the contents annotation.

> > Basically in order to have the full information about a calendar you need
> > to read all messages and you must monitor all new messages. So how can
> > you "loose" an IMAP UID?
>
> I cannot monitor the mailbox. Every time a page, such as a mailbox listing
> page or a calendar view, is loaded, I have to then re-read the entire
> message list.

No, this is not required as the IMAP server guarentees that the UID will 
always only grow but never shrink. So instead of rereading everything you 
keep the uid,imap uid mapping in the session and fetch <last uid:*>.

> I am given no knowledge about what has already been loaded, 
> whether it was sufficient, whether anything has changed, etc. All I am told
> is "load everything" or "load the object with UID X".

1. Load everything --> not problematic for the issue beeing discussed
2. Load the object with UID X --> lookup the imap uid in the mapping and then 
fetch this message.  In case you do not succeed check for new messages. In 
case the message is found (requires parsing of all new messages, typically a 
low number) you are done.  In case the message id deleted (cannot be fetched) 
and none of the new messages contains the required UID the object is simply 
not available anymore.

> >  After all there is no modification in IMAP only new and delete.
>
> Exactly - you "modify" an object by creating a new one with the updated
> information, and delete the older one; with these operations you are then
> assigned a different IMAP UID for the same object, making it an essentially
> useless UID (for our purposes).

I tend to disagree with this. It remains useful. With the old imap UID you can 
check if the message got modified behind your back. And then you only need to 
fetch the new messages (<last uid:*>). 

The fetching and parsing of new messages is zero extra effort because you must 
do this anyway in order to keep your calendar uptodate.

Regards,
-- martin

Dipl.-Phys. Martin Konold

e r f r a k o n
Erlewein, Frank, Konold & Partner - Beratende Ingenieure und Physiker
Nobelstrasse 15, 70569 Stuttgart, Germany
fon: 0711 67400963, fax: 0711 67400959
email: martin.konold at erfrakon.de