[Kolab-devel] Cyrus problems with folder-type in annotations.db missing first 4 characters after upgrade from Kolab2

[Timotheus Pokorra]

Hello,
after more digging and debugging, I think I have understood the problem.

Have a look at the function that parses the values from the skiplist,
which is the format of annotations.db.
http://git.cyrusimap.org/cyrus-imapd/tree/imap/annotate.c?h=cyrus-imapd-2.4#n305
(by the way, cyrus-imapd-2.3 had the same implementation).
In the method split_attribs there are these lines:
    memcpy(&tmp, data, sizeof(unsigned long));
    attrib->size = (size_t) ntohl(tmp);
    data += sizeof(unsigned long); /* skip to value */

But this means, on a 32 bit machine, the sizeof(unsigned long) is 4,
but on a 64 bit machine, it is 8.
This means: when reading a file on a 64 bit machine, the first 4 bytes
of the string are skipped, if the file was written on a 32 bit
machine.

Using the script http://oss.netfarm.it/download/skiplist.py, I was
able to see the difference in the annotation value:
skiplist written on 32 bit system:
Data size 32 (32)
Data String ^@^@^@^Lnote.default^@text/plain^@I^E¨ç
Data String Hex:
00:00:00:0c:6e:6f:74:65:2e:64:65:66:61:75:6c:74:00:74:65:78:74:2f:70:6c:61:69:6e:00:49:05:a8:e7
skiplist written on 64 bit system:
Data size 40 (40)
Data String ^@^@^@^L^@^@^@^@note.default^@text/plain^@Q<9f>^S<9a>^@^@^@^@
Data String Hex:
00:00:00:0c:00:00:00:00:6e:6f:74:65:2e:64:65:66:61:75:6c:74:00:74:65:78:74:2f:70:6c:61:69:6e:00:51:9f:13:9a:00:00:00:00

By the way, in trunk, it has this comment:
http://git.cyrusimap.org/cyrus-imapd/tree/imap/annotate.c#n888
  /*
     * Sigh...this is dumb.  We take care to be machine independent by
     * storing the length in network byte order...but the size of the
     * length field depends on whether we're running on a 32b or 64b
     * platform.
     */
    memcpy(&tmp, data, sizeof(unsigned long));
    data += sizeof(unsigned long); /* skip to value */
Unfortunately, the problem has been recognised, but not been resolved.

I am thinking now of a solution for the upgrade, that does not require
to connect to the old server.
I am wondering to modify the skiplist.py, and call for each value
containing kolab/folder-type:
kolab set-mailbox-metadata "user/test/Notizen at sample.org"
"/shared/vendor/kolab/folder-type" "note.default"
But other values need to be fixed as well, eg /vendor/kolab/activesync etc
A first version of the script is available here:
https://gist.github.com/tpokorra/5643410, and attached to this email
as well.

After that, it might be good to convert the fixed skiplist to
berkeley, and back to skiplist, to make the annotations.db smaller
again, since the old values are still stored in the database
otherwise.
/usr/lib/cyrus-imapd/cvt_cyrusdb /var/lib/imap/annotations.db skiplist
`pwd`/annotations-berkeley.db berkeley
/usr/lib/cyrus-imapd/cvt_cyrusdb `pwd`/annotations-berkeley.db
berkeley /var/lib/imap/annotations.db skiplist

I should probably update the upgrade instructions in the Administrator
Guide, at least explain the sentence "The annotations database may not
have been upgraded correctly, causing some annotations to miss the
first 4 characters of their value".
I have attached a patch file for the docs, for file
https://git.kolab.org/kolab-docs/tree/Administrator_Guide/en-US/Upgrading_from_Kolab_2_on_OpenPKG.xml

All the best,
  Timotheus

On 22 May 2013 16:00, Timotheus Pokorra <timotheus at pokorra.de> wrote:
> Hello all,
>
> I am wondering about this issue:
> After you upgraded your Kolab2 IMAP data, on Kolab3 the command kolab
> list-mailbox-metadata will eg. show t.default instead of event.default
> for your calendar.
> Because of this, the calendar folder will not appear as the default
> calendar in Roundcube.
>
> This is documented in the Administrators Guide:
> http://docs.kolab.org/en-US/Kolab_Groupware/3.0/html/Administrator_Guide/chap-Administrator_Guide-Upgrading_from_Kolab_2_on_OpenPKG.html
> "The annotations database may not have been upgraded correctly,
> causing some annotations to miss the first 4 characters of their
> value"
>
> Following the advice in the guide to connect to the old server might
> work (where is the password for cyrus admin on Kolab2?), but I am
> still wondering about the reasons for this problem, and if there could
> be a cleaner solution.
>
> When I open on my Kolab3 server the file /var/lib/imap/annotations.db
> in vi, I can search for the folder types, and they seem ok to me,
> there I can see default.event etc.
>
> Related to this problem, I found also this post on the mailing list:
> http://www.intevation.de/pipermail/kolab-users/2012-March/013290.html
> "x32 to x64 convertion, Cyrus issue with annotations"
>
> When I try to convert the annotations.db from skiplist to flat:
> /usr/lib/cyrus-imapd/cvt_cyrusdb `pwd`/annotations.db skiplist
> `pwd`/test.db flat
> I get this error in maillog:
> cvt_cyrusdb[18569]: IOERROR: writing /root/annotations/test.db.NEW:
> Invalid argument
>
> I cannot find any suggestions for the reason of this error message on the web.
>
> Any ideas?
>
> Thanks,
>   Timotheus
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Upgrading_from_Kolab_2_on_OpenPKG.xml.patch
Type: application/octet-stream
Size: 1176 bytes
Desc: not available
URL: <http://lists.kolab.org/pipermail/devel/attachments/20130524/0c3fa131/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: skiplist_upgrade32_to_64bit.py
Type: application/octet-stream
Size: 5871 bytes
Desc: not available
URL: <http://lists.kolab.org/pipermail/devel/attachments/20130524/0c3fa131/attachment-0001.obj>