Advice needed: Migration Dovecot to Kolab Cyrus.

[Winfried Ritsch]

Thanks for the answer.

The options --useuid options did a better job now to do not copy again same  
messages and the deleted messages where not expunged, so my partition hit the 
limit: --expunge2 was the solution.

I got on a 10GB imap storage 4,3msg/sec , so it will be ok do to with a 
presync avoiding downtime more than 2 hours.

I will test your additional options and report what is working best.

mfG 
 Winfried Ritsch



Am Mittwoch, 20. März 2013, 18:11:28 schrieb Scott Lambert:
> On Wed, Mar 20, 2013 at 12:30:11PM +0100, JP Horst wrote:
> > I migrated to kolab3 on CentOS using imapsync.
> > 
> > imapsync --ssl1 --useuid --authmech1 PLAIN --ssl2 --authmech2 PLAIN
> > --host1 10.76.25.x --user1 x at y.de --passfile1 /root/migrate/pass_x
> > --host2 10.76.25.y --user2 xy --passfile2 /root/migrate/pass_xy
> > 
> > Unfortunately imapsync with kolab3 is extremely slow. With kolab 2 I
> > remembered it much faster. A big mailbox with 10 GB can take a day or
> > more to sync. (Message rate: 0.2 messages/s, Average bandwidth rate:
> > 70.0 KiB/s)
> > 
> > I applied 3 steps:
> > 
> > 1. dryrun to see if all subfolders get synchronized. Due to errors I had
> > users to rename some folders
> > 
> > 2. Initial sync during productive use, which took several days.
> > 
> > 3. Final sync during swichting without productive use in the background.
> > Since the bulk of the mails is already transferred this third step will
> > be quicker and downtime shorter.
> > 
> > As a consequence there might be some mails doubled, but that should not
> > be a big problem.
> 
> There should not be any duplicates if the final sync is run with
> the --delete2 and --expunge2 options enabled.
> 
> When I migrated from imap-uw to cyrus-imapd, I ran multiple syncs
> prior to stopping the MTAs and blocking IMAP/POP3 access.  The first
> sync takes so long that a lot changes happen to the imap spool while
> it is running.  I actually just ran the process over and over, one
> after another for a couple of weeks while I was verifying that what
> was on the cyrus server was correct.
> 
> while true; do time run_imapsync.sh > imapsync.log; done;
> 
> This is what I used for each mailbox when migrating from imap-uw
> to cyrus-imapd, assuming the username is the same on both sides:
> 
> ------------------------------------------------------------
> #DELETE="--delete2 --expunge2"
> DELETE=""
> 
> /usr/local/bin/imapsync --host1 ${OLD_MAIL_IP} \
>         --user1 $IMAPUSER1 --password1 $IMAPPASS1 --authmech1=LOGIN  \
>         --host2 ${NEW_MAIL_IP} \
>         --user2 $IMAPUSER1 --password2 $IMAPPASS1 --authmech2=LOGIN \
>         --prefix1 mail/ \
>         --include '^(INBOX|mail/)'  \
>         --regextrans2 's#^INBOX/INBOX$#INBOX#' \
>         --regextrans2 's#/INBOX\.#/#' \
>         --regextrans2 's#/\.#/_#' \
>         --regextrans2 's#[()<>!@]#_#g' \
>         --regextrans2 's#\*#+#g' \
>         --skipsize \
>         --subscribe ${DELETE} \
>         2>&1 | tee ${LOGFILES_DIR}/${IMAPUSER1}.log
> 
> ------------------------------------------------------------
> 
> Coming from Dovecot will likely need different --regextrans2 options,
> if any.  We migrated four servers to one and every server needed
> different options to get a good end result.
> 
> The first serveral runs were with --justfolders so that we could
> figure out which regextrans2 manipulations were needed before
> starting to move e-mail.
> 
> We only had around 20G of imap data, 3700 user accounts.  It was
> about three years ago.  I don't remember exactly how long the runs
> were taking.  I think it was down to about 20 minutes per pass, at
> night, after the first few runs, 2 - 4 hours during the local
> business day.  The first run without --justfolders took a *long*
> time.
> 
> I enabled delete after a week of testing.  The new server's imap
> spool was growing quickly.  :-) We didn't have enough users removing
> folders to worry about --delete2folders.
> 
> > Am Mittwoch, 20. März 2013, 09:56:32 schrieb Winfried Ritsch:
> > >  I used imapsync from Fedora free source:
> > >  https://fedorahosted.org/imapsync/
> > >  
> > >  with options:
> > >   --tls1 --tls2 --syncinternaldates --delete2  --noexpunge --subscribe
> > >   
> > >   (no --delete2folders since it would erase default folders of Kolab)
> > > 
> > > Since mailboxes are quite big, I did a pre-sync during running system.
> > > after that during switching (exim mailserver down, firewall blocked
> > > imap) I did a
> > > 
> > > new sync:
> > >   - some of mails where doubled, folder structure was broken.
> > > 
> > > on this new imapsync a lot of mails are copied again, which will take
> > > more then 8 hours, to long for a mailserver downtime.
> 
> It shouldn't be syncing the same e-mails, but there may have been
> a lot of changes since the first run began.  Do several runs while
> still in production on the exim/dovecot server.  Time the runs. See
> if they get shorter.
> 
> To me, buying the new version of imapsync was worthwhile.
> 
> We are not running Kolab.  I want to see how v3 settles out before
> I try to make it work on FreeBSD.


-- 
---- Atelier Algorythmics ----
Winfried Ritsch
Leitnergasse 7a, 
A-8010 Graz
Austria
mobil: ++43 664 2439369
http://algo.mur.at/
--------------------------------------