Advice needed: Migration Dovecot to Kolab Cyrus.
Winfried Ritsch
ritsch at algo.mur.at
Fri Mar 22 00:31:07 CET 2013
Hello,
It now seems to work fast execpt one failure with nesting folders I cant fix,
e.g:
Host1:
...
Host1 folder [art]
Host1 Folder art: Could not examine: 995 NO Mailbox isn't selectable: art
Host1 folder [art/anfragen]
Size: 6401396 Messages: 17 Biggest: 5646848
...
What I saw was that the mailbox art (\HasChildren) but is no mailbox storage,
so it is not selectable, imapsync doesnt move it.
In Roundcubemail, the art mailbox is greyed out and when I try to subscibe it,
it says "SUBSCRIBE mailbox does not exist", which is true looking at kolab
listmailboxes. Even art/anfragen is there.
Also this would be no problem, but using KMAIL, the mailbox-folder and all
their children is not listed, so I cant use it.
I just put the script used below if someone interrested.
Any hint how to fix this ?
mfG
Winfried
--- cut ---
#!/bin/bash
#--- SETTINGS, edit here ---
SOURCE="algo.mur.at"
TARGET="algorythmics.at"
# --- set test flags before doing actual migration
TESTFLAGS="--dry --justfolders" # first try synthax and regex
TESTFLAGS="--justfolders" # second uncomment to try folderstructure
TESTFLAGS="" # last uncomment to do migration
#TESTFLAGS+=" --debug --debugimap1"
# --- autenthicate on servers: PLAIN, LOGIN, CRAM-MD5 etc.
AUTHS="--tls1 --authmech1=PLAIN --tls2 --authmech2=PLAIN"
# --- Use uid instead of header
MAILMATCH="--useuid --expunge2 --subscribe_all"
# --- CONVERT Namings on TARGET folders, uncomment if needed
REGEXTRANS="\
--regextrans2 's#^INBOX/INBOX$#INBOX#' \
--regextrans2 's#/INBOX\.#/#' \
--regextrans2 's#/\.#/_#' \
--regextrans2 's#[()<>!@]#_#g' \
--regextrans2 's#\*#+#g' \
"
# --- delete mails not on SOURCEhost and expunge on TARGET
DELETES="--delete2 --skipsize"
DELETES+=" --expunge2" # to reduce target size
#DELETES+=" --delete2folders" # ATTENTION: auto generated folders on Kolab3
server will be erased
# --- no need to edit below ---
OPTIONS="$TESTFLAGS $AUTHS $MAILMATCH $REGEXTRANS $DELETES"
if [ $# -lt 1 -o ! -f $1 ]; then
echo batch imapsync
echo usage: $0 \<usermigration_filename.txt\>
echo
echo argument is textfile with users to migrate, with following synthax:
echo " user1_1\;password1_1\;user1_2\;password1_2"
echo " user2_1\;password2_1\;user2_2\;password2_2"
echo " ..."
echo
exit -1
fi
echo Imap versions on $SERVER1 to $SERVER2 :
# for logging info:
imapsync --host1 $SERVER1 --host2 $SERVER2 --justconnect
if [ $? -lt 0 ]; then
echo Error $?
exit $?
fi
echo --- start migration ---
{ while IFS=';' read u1 p1 u2 p2; do
echo --- $SOURCE : $u1 -\> $TARGET : $u2 ---
imapsync $OPTIONS --host1 $SOURCE --user1 "$u1" --password1 "$p1" --host2
$TARGET --user2 "$u2" --password2 "$p2" ...
done ; } < $1
echo --- done ---
exit $?
--- cut ---
Am Donnerstag, 21. März 2013, 00:24:15 schrieb Winfried Ritsch:
> Thanks for the answer.
>
> The options --useuid options did a better job now to do not copy again same
> messages and the deleted messages where not expunged, so my partition hit
> the limit: --expunge2 was the solution.
>
> I got on a 10GB imap storage 4,3msg/sec , so it will be ok do to with a
> presync avoiding downtime more than 2 hours.
>
> I will test your additional options and report what is working best.
>
> mfG
> Winfried Ritsch
>
> Am Mittwoch, 20. März 2013, 18:11:28 schrieb Scott Lambert:
> > On Wed, Mar 20, 2013 at 12:30:11PM +0100, JP Horst wrote:
> > > I migrated to kolab3 on CentOS using imapsync.
> > >
> > > imapsync --ssl1 --useuid --authmech1 PLAIN --ssl2 --authmech2 PLAIN
> > > --host1 10.76.25.x --user1 x at y.de --passfile1 /root/migrate/pass_x
> > > --host2 10.76.25.y --user2 xy --passfile2 /root/migrate/pass_xy
> > >
> > > Unfortunately imapsync with kolab3 is extremely slow. With kolab 2 I
> > > remembered it much faster. A big mailbox with 10 GB can take a day or
> > > more to sync. (Message rate: 0.2 messages/s, Average bandwidth rate:
> > > 70.0 KiB/s)
> > >
> > > I applied 3 steps:
> > >
> > > 1. dryrun to see if all subfolders get synchronized. Due to errors I
> > > had users to rename some folders
> > >
> > > 2. Initial sync during productive use, which took several days.
> > >
> > > 3. Final sync during swichting without productive use in the
> > > background. Since the bulk of the mails is already transferred this
> > > third step will be quicker and downtime shorter.
> > >
> > > As a consequence there might be some mails doubled, but that should not
> > > be a big problem.
> >
> > There should not be any duplicates if the final sync is run with
> > the --delete2 and --expunge2 options enabled.
> >
> > When I migrated from imap-uw to cyrus-imapd, I ran multiple syncs
> > prior to stopping the MTAs and blocking IMAP/POP3 access. The first
> > sync takes so long that a lot changes happen to the imap spool while
> > it is running. I actually just ran the process over and over, one
> > after another for a couple of weeks while I was verifying that what
> > was on the cyrus server was correct.
> >
> > while true; do time run_imapsync.sh > imapsync.log; done;
> >
> > This is what I used for each mailbox when migrating from imap-uw
> > to cyrus-imapd, assuming the username is the same on both sides:
> >
> > ------------------------------------------------------------
> > #DELETE="--delete2 --expunge2"
> > DELETE=""
> >
> > /usr/local/bin/imapsync --host1 ${OLD_MAIL_IP} \
> >
> > --user1 $IMAPUSER1 --password1 $IMAPPASS1 --authmech1=LOGIN \
> > --host2 ${NEW_MAIL_IP} \
> > --user2 $IMAPUSER1 --password2 $IMAPPASS1 --authmech2=LOGIN \
> > --prefix1 mail/ \
> > --include '^(INBOX|mail/)' \
> > --regextrans2 's#^INBOX/INBOX$#INBOX#' \
> > --regextrans2 's#/INBOX\.#/#' \
> > --regextrans2 's#/\.#/_#' \
> > --regextrans2 's#[()<>!@]#_#g' \
> > --regextrans2 's#\*#+#g' \
> > --skipsize \
> > --subscribe ${DELETE} \
> > 2>&1 | tee ${LOGFILES_DIR}/${IMAPUSER1}.log
> >
> > ------------------------------------------------------------
> >
> > Coming from Dovecot will likely need different --regextrans2 options,
> > if any. We migrated four servers to one and every server needed
> > different options to get a good end result.
> >
> > The first serveral runs were with --justfolders so that we could
> > figure out which regextrans2 manipulations were needed before
> > starting to move e-mail.
> >
> > We only had around 20G of imap data, 3700 user accounts. It was
> > about three years ago. I don't remember exactly how long the runs
> > were taking. I think it was down to about 20 minutes per pass, at
> > night, after the first few runs, 2 - 4 hours during the local
> > business day. The first run without --justfolders took a *long*
> > time.
> >
> > I enabled delete after a week of testing. The new server's imap
> > spool was growing quickly. :-) We didn't have enough users removing
> > folders to worry about --delete2folders.
> >
> > > Am Mittwoch, 20. März 2013, 09:56:32 schrieb Winfried Ritsch:
> > > > I used imapsync from Fedora free source:
> > > > https://fedorahosted.org/imapsync/
> > > >
> > > > with options:
> > > > --tls1 --tls2 --syncinternaldates --delete2 --noexpunge
> > > > --subscribe
> > > >
> > > > (no --delete2folders since it would erase default folders of Kolab)
> > > >
> > > > Since mailboxes are quite big, I did a pre-sync during running
> > > > system. after that during switching (exim mailserver down, firewall
> > > > blocked imap) I did a
> > > >
> > > > new sync:
> > > > - some of mails where doubled, folder structure was broken.
> > > >
> > > > on this new imapsync a lot of mails are copied again, which will take
> > > > more then 8 hours, to long for a mailserver downtime.
> >
> > It shouldn't be syncing the same e-mails, but there may have been
> > a lot of changes since the first run began. Do several runs while
> > still in production on the exim/dovecot server. Time the runs. See
> > if they get shorter.
> >
> > To me, buying the new version of imapsync was worthwhile.
> >
> > We are not running Kolab. I want to see how v3 settles out before
> > I try to make it work on FreeBSD.
--
---- Atelier Algorythmics ----
Winfried Ritsch
Leitnergasse 7a,
A-8010 Graz
Austria
mobil: ++43 664 2439369
http://algo.mur.at/
--------------------------------------
More information about the users
mailing list