Advice needed: Migration Dovecot to Kolab Cyrus.

[Scott Lambert]

On Wed, Mar 20, 2013 at 12:30:11PM +0100, JP Horst wrote:
> I migrated to kolab3 on CentOS using imapsync.
> 
> imapsync --ssl1 --useuid --authmech1 PLAIN --ssl2 --authmech2 PLAIN --host1 
> 10.76.25.x --user1 x at y.de --passfile1 /root/migrate/pass_x --host2 10.76.25.y 
> --user2 xy --passfile2 /root/migrate/pass_xy
> 
> Unfortunately imapsync with kolab3 is extremely slow. With kolab 2 I 
> remembered it much faster. A big mailbox with 10 GB can take a day or more to 
> sync. (Message rate: 0.2 messages/s, Average bandwidth rate: 70.0 KiB/s)
> 
> I applied 3 steps:
> 
> 1. dryrun to see if all subfolders get synchronized. Due to errors I had users 
> to rename some folders
> 
> 2. Initial sync during productive use, which took several days.
> 
> 3. Final sync during swichting without productive use in the background. Since 
> the bulk of the mails is already transferred this third step will be quicker 
> and downtime shorter.
> 
> As a consequence there might be some mails doubled, but that should not be a 
> big problem.

There should not be any duplicates if the final sync is run with
the --delete2 and --expunge2 options enabled.

When I migrated from imap-uw to cyrus-imapd, I ran multiple syncs
prior to stopping the MTAs and blocking IMAP/POP3 access.  The first
sync takes so long that a lot changes happen to the imap spool while
it is running.  I actually just ran the process over and over, one
after another for a couple of weeks while I was verifying that what
was on the cyrus server was correct.

while true; do time run_imapsync.sh > imapsync.log; done;

This is what I used for each mailbox when migrating from imap-uw
to cyrus-imapd, assuming the username is the same on both sides:

------------------------------------------------------------
#DELETE="--delete2 --expunge2"
DELETE=""

/usr/local/bin/imapsync --host1 ${OLD_MAIL_IP} \
        --user1 $IMAPUSER1 --password1 $IMAPPASS1 --authmech1=LOGIN  \
        --host2 ${NEW_MAIL_IP} \
        --user2 $IMAPUSER1 --password2 $IMAPPASS1 --authmech2=LOGIN \
        --prefix1 mail/ \
        --include '^(INBOX|mail/)'  \
        --regextrans2 's#^INBOX/INBOX$#INBOX#' \
        --regextrans2 's#/INBOX\.#/#' \
        --regextrans2 's#/\.#/_#' \
        --regextrans2 's#[()<>!@]#_#g' \
        --regextrans2 's#\*#+#g' \
        --skipsize \
        --subscribe ${DELETE} \
        2>&1 | tee ${LOGFILES_DIR}/${IMAPUSER1}.log

------------------------------------------------------------

Coming from Dovecot will likely need different --regextrans2 options,
if any.  We migrated four servers to one and every server needed
different options to get a good end result.

The first serveral runs were with --justfolders so that we could
figure out which regextrans2 manipulations were needed before
starting to move e-mail.

We only had around 20G of imap data, 3700 user accounts.  It was
about three years ago.  I don't remember exactly how long the runs
were taking.  I think it was down to about 20 minutes per pass, at
night, after the first few runs, 2 - 4 hours during the local
business day.  The first run without --justfolders took a *long*
time.

I enabled delete after a week of testing.  The new server's imap
spool was growing quickly.  :-) We didn't have enough users removing
folders to worry about --delete2folders.

> Am Mittwoch, 20. März 2013, 09:56:32 schrieb Winfried Ritsch:
> >  I used imapsync from Fedora free source: https://fedorahosted.org/imapsync/
> > 
> >  with options:
> >   --tls1 --tls2 --syncinternaldates --delete2  --noexpunge --subscribe
> > 
> >   (no --delete2folders since it would erase default folders of Kolab)
> > 
> > Since mailboxes are quite big, I did a pre-sync during running system. after
> > that during switching (exim mailserver down, firewall blocked imap) I did a
> > new sync:
> >   - some of mails where doubled, folder structure was broken.
> > 
> > on this new imapsync a lot of mails are copied again, which will take more
> > then 8 hours, to long for a mailserver downtime.

It shouldn't be syncing the same e-mails, but there may have been
a lot of changes since the first run began.  Do several runs while
still in production on the exim/dovecot server.  Time the runs. See
if they get shorter.

To me, buying the new version of imapsync was worthwhile.

We are not running Kolab.  I want to see how v3 settles out before
I try to make it work on FreeBSD.

-- 
Scott Lambert                    KC5MLE                       Unix SysAdmin
lambert at lambertfam.org