Need opinions about whether I should convert mailboxes.db and annotations.db to skiplist

[Skip Morse]

On Wed, May 19, 2010 at 2:03 AM, Thomas Arendsen Hein
<thomas at intevation.de> wrote:
> * skipmorse at gmail.com <skipmorse at gmail.com> [20100518 22:16]:
>> I'm thinking about converting mailboxes.db and annotations.db to the
>> skiplist format, I've upgraded from version 2.1 Kolab to 2.2.3 and just
>> set it to use the berkley format since that's what they were in.
>>
>> I'ma bit paranoid that I'll end up with a corrupted annotations.db and
>> there doesn't seem to be anything about how one might 'export' that as
>> you would the mailboxes.db to a text file for backup.
>
> Quoting myself from https://issues.kolab.org/msg21516:
>
> | The script already exists and is downloadable from CVS:
> | http://kolab.org/cgi-bin/viewcvs-kolab.cgi/*checkout*/utils/admin/kolab-mailboxinfo.pl
> |
> | (call it with option -c to get input suitable for cyradm)

Thanks, just tried the script on a test setup and it seemed to work
great. So, with the "-c" option I recognize the cyradm commands for
setting the ACL's, and i guess the mboxcfg will set the annotations
for each folder.  Seems exactly what I was looking for.  As a process,
I guess I would import a text version of the mailboxes.db, then import
this to set the ACL's and annotations.  But, what command would I use
to send that input into cyradm?  I've only ever used it in a 'console'
type mode.

>> I've noticed that there are scripts that will check for corruption and
>> also backup those files in case there it at some point finds any. I
>> haven't seen anything like that for the berkley format. Also, in the
>> upgrade notes, or the release notes (somewhere) I'd read that the
>> decision was made to move back to skiplist because of problems with
>> berkely, anyone know what those problems were?
>
> Mostly locking issues with larger setups.
>
>> So, my question, am I being paranoid or is this a valid concern (I talked
>> to someone the other day on IRC that had it happen after having to
>> hard-power off the system). I do keep about 3 days of backups of the
>> system, but I've heard it can take longer than that to see the fallout
>> from the corruption.
>
> Yes, keeping longer backups for mailboxes.db and annotations.db is a
> good idea.
>
>> Also, any tips for the conversion? I haven't looked into the process too
>> much, but on the wiki it discusses the converstion TO berkely from
>> skiplist, so I think it would likely be similar. But, when I did the
>> upgrade, I couldn't just copy over mailboxes.db and annotations.db, I
>> needed to copy the DB folder and the DB backup folders as well, anyone
>> foresee an issue there?
>
> This might be needed if there was an unclean shutdown of the imap
> server, but usually you only need the two files.

So maybe having a process that runs kolab-mailboxinfo.pl, exports the
mailboxes.db to a text file, and then makes a copy of the actual
mailboxes.db and annotations.db each day should provide enough backup
of the mailbox data to allow for a restore if needed?

As a first resort with any DB corruption, I would just copy over the
backed up mailboxes.db and annotations.db then start the imap process?

Also, is it necessary to stop the imap server when taking the backup?
I assume it would be the safest bet to insure that no folders were
changed between copying the mailboxes.db and annotations.db.

Any guesses at the corruption timeline?  I've read that it might not
show any corruption for a 'few days', I've heard a definite 4 days.
Does that mean that a 2 week 'rotation' SHOULD be enough?

> The conversion is documented in 1st.README.

The only thing that I saw in reference to this in the 1st.README is:
"It is possible to convert existing berkeley database files to the now
default skiplist format, but the necessary procedure is not well
tested and therefore currently not recommended.  The documentation of
the actual conversion is beyond the scope of this document."

But, it sounds like with our current usage, it wouldn't be worth the
time to look into conversion anyway (now that I have your script!) :)

>> PS Server has about 20 users, 3 group users for shared
>> calendar/contacts/folders and a handful of 'shared folders'...
>
> With these numbers it probably does not matter if you're using
> skiplist or berkeley.
>
> Regards,
> Thomas Arendsen Hein
>
> --
> thomas at intevation.de - http://intevation.de/~thomas/ - OpenPGP key: 0x5816791A
> Intevation GmbH, Neuer Graben 17, 49074 Osnabrueck - AG Osnabrueck, HR B 18998
> Geschaeftsfuehrer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner
>
> _______________________________________________
> Kolab-users mailing list
> Kolab-users at kolab.org
> https://kolab.org/mailman/listinfo/kolab-users
>