wilde: utils/admin skiplistsafe.sh, NONE, 1.1 skiplistsafe.conf, NONE, 1.1 skiplistsafe.README, NONE, 1.1 save-annotations.README, 1.2, 1.3

[cvs at kolab.org]

Author: wilde

Update of /kolabrepository/utils/admin
In directory doto:/tmp/cvs-serv31159

Modified Files:
	save-annotations.README 
Added Files:
	skiplistsafe.sh skiplistsafe.conf skiplistsafe.README 
Log Message:
Added skiplistsafe, a more generic replacement for save-annotations.
This can be used to backup/validate any skiplist db files
(annotations.db and mailboxes.db in case of Kolab).
save-annotations is now deprecated.


--- NEW FILE: skiplistsafe.sh ---
#!/bin/bash

# -------------------------------------------------------------------
# Copyright (C) 2006 by Intevation GmbH
# Author(s):
# Sascha Wilde <wilde at intevation.de>

# This program is free software under the GNU GPL (>=v2)
# Read the file COPYING coming with the software for details.
# -------------------------------------------------------------------
# $Id: skiplistsafe.sh,v 1.1 2006/08/30 15:22:47 wilde Exp $

# Description:

# Backup and check skiplist db files.
# The checks are done using validate-skiplist

# As the kolab/cyrus-imap skiplist files (especially annotations.db)
# tends to break on a regular basis, this script should be setup as a
# cronjob to backup them.

# This script:
# - backups the given skiplist file
# - checks if the backup file is a valid skiplist db
# - informs the admin if it isn't
# - removes old backups if everything worked fine

# Usage:
# skiplistsafe FILENAME

# For more details and hints on configuration please consult
# skiplistsafe.README

# Configuration can be done in /kolab/etc/imapd/skiplistsafe.conf
# Don't change anything in this script!

# -------------------------------------------------------------------

if [ ! $# -eq 1 ] ; then
  echo "Usage: $0 FILENAME"
  exit 1
fi

DBFILE="$1"

CONF_DIR=/kolab/etc/imapd
CONF_GENERIC="${CONF_DIR}/skiplistsafe.conf"
CONF_PERFILE="${CONF_DIR}/skiplistsafe-`basename $DBFILE`.conf"

# BEGIN default values:
ADMIN_MAIL=root at localhost
ERROR_SENDER=postmaster@`hostname -f`
SKIPLIST_DIR=/kolab/var/imapd
BACKUP_DIR=/tmp/backuptest
BACKUP_LIVESPAN=7
USER=kolab-r
CHECK_CYRUSDB=/kolab/bin/validate-skiplist
COPY="/usr/bin/rsync"
COMPRESS="/bin/gzip -9"
SENDMAIL=/kolab/sbin/sendmail
# END default values

# Read configuration files
[ -r "$CONF_GENERIC" ] && . "$CONF_GENERIC"
[ -r "$CONF_PERFILE" ] && . "$CONF_PERFILE"

# Setup absolute filenames
[ "${DBFILE:0:1}" != "/" ] && DBFILE="${SKIPLIST_DIR}/${DBFILE}"
BACKUP_FILE="${BACKUP_DIR}/`basename $DBFILE`-`date +%Y%m%d-%H%M%S`"

# Change identity if necessary
if [ `whoami` != "$USER" ] ; then
  if [ $UID -eq 0 ] ; then
    exec su -p - $USER "$0" "$1"
  else
    echo >&2 "$0 must be run as $USER or root!"
    exit 1
  fi
fi

# Error handling functions:
reporterr()
{
  $SENDMAIL $ADMIN_MAIL \
    <<EOF 
From: Skiplistsafe Script <${ERROR_SENDER}>
To: <${ADMIN_MAIL}>
Subject: Skiplistsafe Error
X-Skiplistsave-Error: generated on $HOSTNAME

$0 reported error:
$1
EOF
}

fatal()
{
  reporterr "$1"
  exit 1
}

# Worker functions:
backup-skiplist()
{
  [ -d "$BACKUP_DIR" ] || mkdir -p "$BACKUP_DIR"
  $COPY >/dev/null 2>&1 "$DBFILE" "$BACKUP_FILE" || \
    fatal "FATAL: \"$COPY $DBFILE $BACKUP_FILE\" failed with returncode $?"
}

check-backup()
{
  $CHECK_CYRUSDB "$BACKUP_FILE" >/dev/null 2>&1
}

compress-backup()
{
  if [ "$COMPRESS" ] ; then
    $COMPRESS >/dev/null 2>&1 "$BACKUP_FILE" || \
      reporterr "Compression failed: \"$COMPRESS $BACKUP_FILE\" returened $?"
  fi
}

expunge-old-backups()
{
  find "$BACKUP_DIR" \
    -type f -name "`basename $DBFILE`-*.gz" \! -mtime -$BACKUP_LIVESPAN \
    -exec rm \{\} \;
}

# -------------------------------------------------------------------
# Main routine:

if [ -f "$DBFILE" ] ; then
  backup-skiplist
  check-backup
  check_success=$?
  if [ ! $check_success -eq 0 ] ; then
    reporterr "BROKEN DB: check of $BACKUP_FILE failed.  
Kolab most likely stoped working!"
    mv "$BACKUP_FILE" "${BACKUP_FILE}_CORRUPT"
    BACKUP_FILE="${BACKUP_FILE}_CORRUPT"
  fi
  compress-backup
  [ $check_success -eq 0 ] && \
    expunge-old-backups
else
  fatal "FATAL: File \"$DBFILE\" does not exist."
fi

--- NEW FILE: skiplistsafe.conf ---
# --------------------------------------------------------------------
# skiplistsafe exsample configuration file
#
# $Id: skiplistsafe.conf,v 1.1 2006/08/30 15:22:47 wilde Exp $
# -------------------------------------------------------------------
# Copyright (C) 2006 by Intevation GmbH
# Author(s):
# Sascha Wilde <wilde at intevation.de>

# This program is free software under the GNU GPL (>=v2)
# Read the file COPYING coming with the software for details.
# -------------------------------------------------------------------


# --------------------------------------------------------------------
# Copy this file to /kolab/etc/imapd/skiplistsafe.conf
# And set variables acording to your needs.

# IMPORTAINT NOTE: 
# This is a shell script which is sourced everytime skiplistsafe is run!
# --------------------------------------------------------------------


# Error-reports will be send to this address:
# (Default: root at localhost)
ADMIN_MAIL=root at localhost

# Error-reports will have this from address:
# (Default: postmaster@`hostname -f`)
ERROR_SENDER=postmaster@`hostname -f`

# Default directory holding the skiplist files,
# if no absolute filename is given as argument to skiplistsafe 
# this directory is searched:
# (Default: /kolab/var/imapd)
SKIPLIST_DIR=/kolab/var/imapd

# Directory of the backups:
# This must be writeable for $USER (see below)
# (Default: /tmp/backuptest)
BACKUP_DIR=/tmp/backuptest

# Keep backup files for N days:
# (DEFAULT: 7)
BACKUP_LIVESPAN=7

# Run this script as user:
# The user must be able to read $ANNOTATIONSDB and to write to $BACKUP_DIR
# (Default: kolab-r)
USER=kolab-r

# Commands used:

# - to validate skiplist
# (Default: /kolab/bin/validate-skiplist)
CHECK_CYRUSDB=/kolab/bin/validate-skiplist

# - to copy annotations.db to backup
# (Default: "/usr/bin/rsync")
COPY="/usr/bin/rsync"

# - to compress file, for no compression leave empty
# (Default: "/bin/gzip -9")
COMPRESS="/bin/gzip -9"

# sendmail used to send error-reports
# THIS HAS TO BE CORRECT OR ANY ERROR-REPORTING WILL FAIL!!!
# (Default: /kolab/sbin/sendmail)
SENDMAIL=/kolab/sbin/sendmail

--- NEW FILE: skiplistsafe.README ---
======================================================================
                           skiplistsafe.sh
======================================================================

Backup skiplist db file and check if it's broken.

The cyrus imap skiplist db files (especially annotations.db) are known
to get corrupted easily and to be hard to recover
<https://intevation.de/roundup/kolab/issue840>. 

Therefor skiplistsafe was created to be installed on an kolab-server
as a cronjob.  It can backup skiplist files (annotations.db,
mailboxes.db in case of a kolab standard installation) and warn the
administrator if the current skiplist file is broken.

The warning will be sent by email, so make sure you can read your
mails even when the kolab-server fails!


Usage
-----

skiplistsafe.sh FILENAME

where FILENAME is the name of the skiplist db file which should be
backuped and validated.  FILENAME can be a absolute filename or only
the files base name (for example "annotations.db"), in which case
skiplistsafe uses the configured SKIPLIST_DIR to find it (defaults to
/kolab/var/imapd).


Prerequisites
-------------

validate-skiplist must be installed:

- Build it from the included source file:
  gcc -O2 validate-skiplist.c -o validate-skiplist

- Copy (as root) validate-skiplist anywhere you like.  The default
  configuration of save-annotations.sh expects it in /kolab/bin: 
  cp validate-skiplist /kolab/bin


Installation
------------

- Copy skiplistsafe.conf to /kolab/etc/imapd/skiplistsafe.conf and
  edit it according to your needs: read the comments and set the
  variables in the script to meaningful values.  Most defaults should
  be sensible for a standard kolab installation.

  In addition you can make extra configurations per skiplist db you
  want to backup (for example if you want to backup annotations.db and
  mailboxes.db in different directories).  To do this create a file
  /kolab/etc/imapd/skiplistsafe-DBFILE.conf where DBFILE is the name
  of the skiplist db you want to backup.  Put the settings specific
  for this backup into that file.

  Example: You have set BACKUP_DIR=/backup/kolab-imapd but want the
  backups for the annotations.db (and only them) to go to
  /backup/annotations, so you put the line
  "BACKUP_DIR=/backup/annotations" in
  /kolab/etc/imapd/skiplistsafe-annotations.db.conf and you are done.

- Make skiplistsafe.sh executable if it isn't:
  chmod 755 skiplistsafe.sh

- Copy (as root) skiplistsafe.sh anywhere you like, eg:
  cp skiplistsafe.sh /kolab/sbin

- Make an entry in /etc/crontab to run skiplistsafe.sh on a regular
  basis for all skiplist dbs wou want to backup.  If the last working
  update is not up to date, recovering the lost information will be
  difficult and painful, so more often is better.  

  This example runs skiplistsafe every hour on annotations.db and
  mailboxes.db: 

  0 * * * * kolab-r /kolab/sbin/skiplistsafe.sh annotations.db
  0 * * * * kolab-r /kolab/sbin/skiplistsafe.sh mailboxes.db


Now: read your mails, and if for example the annotations.db gets
corrupted put the last working backup in place:

zcat PATH/TO/BACKUP/annotations.db-20060124-08:01:01.gz >/kolab/var/imapd/annotations.db

Index: save-annotations.README
===================================================================
RCS file: /kolabrepository/utils/admin/save-annotations.README,v
retrieving revision 1.2
retrieving revision 1.3
diff -u -d -r1.2 -r1.3
--- save-annotations.README	29 Mar 2006 11:29:02 -0000	1.2
+++ save-annotations.README	30 Aug 2006 15:22:47 -0000	1.3
@@ -1,3 +1,8 @@
+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+              THE save-annotations SCRIPT IS DEPRECATED
+                   PLEASE USE skiplistsafe INSTEAD!
+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+
 save-annotations
 ================