Need opinions about whether I should convert mailboxes.db and annotations.db to skiplist

[Thomas Arendsen Hein]

* Skip Morse <skipmorse at gmail.com> [20100519 21:12]:
> On Wed, May 19, 2010 at 2:03 AM, Thomas Arendsen Hein
> <thomas at intevation.de> wrote:
> > * skipmorse at gmail.com <skipmorse at gmail.com> [20100518 22:16]:
> >> I'm thinking about converting mailboxes.db and annotations.db to the
> >> skiplist format, I've upgraded from version 2.1 Kolab to 2.2.3 and just
> >> set it to use the berkley format since that's what they were in.
> >>
> >> I'ma bit paranoid that I'll end up with a corrupted annotations.db and
> >> there doesn't seem to be anything about how one might 'export' that as
> >> you would the mailboxes.db to a text file for backup.
> >
> > Quoting myself from https://issues.kolab.org/msg21516:
> >
> > | The script already exists and is downloadable from CVS:
> > | http://kolab.org/cgi-bin/viewcvs-kolab.cgi/*checkout*/utils/admin/kolab-mailboxinfo.pl
> > |
> > | (call it with option -c to get input suitable for cyradm)
> 
> Thanks, just tried the script on a test setup and it seemed to work
> great. So, with the "-c" option I recognize the cyradm commands for
> setting the ACL's, and i guess the mboxcfg will set the annotations
> for each folder.  Seems exactly what I was looking for.  As a process,
> I guess I would import a text version of the mailboxes.db, then import
> this to set the ACL's and annotations.  But, what command would I use
> to send that input into cyradm?  I've only ever used it in a 'console'
> type mode.

For a short list of commands you can use copy&paste to the console :)

For longer lists you can use some shell tricks, put all commands you
want to execute in a file, in this example "cyradm-commands", then
execute:
/kolab/bin/cyradm -u manager -password $(< /dev/tty ) localhost < cyradm-commands

After executing this line, enter the password (it will be visible!)
followed by <Enter> and <Ctrl-D>

You can test it by putting a simple command, e.g. just "lm" in the
cyradm-commands file.

> So maybe having a process that runs kolab-mailboxinfo.pl, exports the
> mailboxes.db to a text file, and then makes a copy of the actual
> mailboxes.db and annotations.db each day should provide enough backup
> of the mailbox data to allow for a restore if needed?

kolab-mailboxinfo.pl contains everything for annotations.db, but not
for mailboxes.db.

mailboxes.db contains more information, e.g. the current number of
the newest mail in each folder. If you restore from the output of
kolab-mailboxinfo.pl, you will need to cyrreconstruct all folders
and refresh the IMAP caches of all clients so they notice the new
numbering.

So yes, it would be a good idea to copy both files, too.

http://wiki.kolab.org/index.php/Backups_for_kolab2 additionally
mentions ctl_mboxlist, but I have not much experience with that.

> As a first resort with any DB corruption, I would just copy over the
> backed up mailboxes.db and annotations.db then start the imap process?

Yes.

> Also, is it necessary to stop the imap server when taking the backup?
> I assume it would be the safest bet to insure that no folders were
> changed between copying the mailboxes.db and annotations.db.

That would be ideal, but usually Cyrus copes fine with minor
inconsistencies between those two.

> Any guesses at the corruption timeline?  I've read that it might not
> show any corruption for a 'few days', I've heard a definite 4 days.
> Does that mean that a 2 week 'rotation' SHOULD be enough?

Some corruptions that we saw in the past might go unnoticed for
months and won't show up unless a user changes things on an affected
folder. But do you really want to install such an old database?

> > The conversion is documented in 1st.README.
> 
> The only thing that I saw in reference to this in the 1st.README is:
> "It is possible to convert existing berkeley database files to the now
> default skiplist format, but the necessary procedure is not well
> tested and therefore currently not recommended.  The documentation of
> the actual conversion is beyond the scope of this document."

Oh, right, I confused it with the 2.0->2.1 upgrading docs.
The relevant part is here:
http://wiki.kolab.org/index.php/Kolab2_IMAPD_annotations.db_Problems#Convert_the_database
(it describes skiplist -> berkeley, but with exchanging some words
it should work the other way round. We just did only test this on
very few installations, so we don't know about potential pitfalls)

> But, it sounds like with our current usage, it wouldn't be worth the
> time to look into conversion anyway (now that I have your script!) :)

:)

Regards,
Thomas Arendsen Hein

-- 
thomas at intevation.de - http://intevation.de/~thomas/ - OpenPGP key: 0x5816791A
Intevation GmbH, Neuer Graben 17, 49074 Osnabrueck - AG Osnabrueck, HR B 18998
Geschaeftsfuehrer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner