[Kolab-devel] Introducing freebusyd: Solve freebusy performance problems

[Thomas Jarosch]

----------------------------------------------------------------
From: https://www.intevation.de/roundup/kolab/issue2554
All the files can be found in the issue tracker. Dunno why
it didn't generate an automatic email to kolab-devel.
----------------------------------------------------------------

Hello together,

last week I started to deeply analyze why the current freebusy performance
is so horrible for large calendar folders. Gunnar and I developed
simple patches that halved the memory and CPU usage of pfb.php.
Deleting an event (=IMAP mail) in a calendar folder takes less
than a second now on a 3 Ghz P4. That was a great improvement.

But this was only one part of the problem. I could easily kill modern
hardware by moving around calendar folders. The other part gets
fixed by "freebusyd". I tested it with importing/moving and deleting
10.000 calendar entries and everything felt really smooth.
Here's the introduction section from the README file:

------------------------------------------------------
###########################################################
# freebusyd - control daemon for Kolab's freebusy pfb.php #
###########################################################

Kolab provides server sided freebusy information for calendar folders.
A script called pfb.php connects to the IMAP server and processes the data.
Every change in a calendar folder will trigger a new session of pfb.php,
which results in many parallel running pfb.phps.
This easily kills even modern hardware.

freebusyd is a daemon to make sure the generation of freebusy data
doesn't eat all your CPU/memory on your server: All requests
for pfb.php will be queued and executed in a controlled manner.

Features:
* Only one pfb.php is running at a time per owner/folder combination.
* Global maximum concurrent job limit to relieve the system load.
* Persistent request queue. Never lose a request by server reboot etc.
* Requests for folders that get triggered too often in a configurable
  time period get delayed for some minutes. Speeds up large imports
  or move of calendar data between folders.
* Good logging facilities and runs without root privileges.

Optional features:
* A special patch for the cyrus imap server will notify freebusyd
  about deleted or renamed IMAP folders. freebusyd will then remove
  the cached freebusy data for you. No more freebusy ghost entries.

Technical details:
* Written in C++ and should be able to handle 1.000+ users.
* Very CPU friendly: The daemon usually sleeps and will only wake
  up when there is a new request in the queue (uses inotify).
* Tested with large (10.000 events) calendar folders
* Fully doxygen documented
------------------------------------------------------

Intra2net decided to release the daemon as GPL and it should
be very Kolab server compatible. It will need some tweaking
to the paths and users rights, but that should be no big deal.

If you decide to integrate the cyrus folder delete patch,
there'll be some work needed to support a separated
freebusy and IMAP server. The patch fixes
a Kolab bug that is around since 2005:
https://www.intevation.de/roundup/kolab/issue614

Enjoy,
Thomas Jarosch