[Kolab-devel] Introducing freebusyd: Solve freebusy performance problems

[Gunnar Wrobel]

Hi Thomas,

Thomas Jarosch <thomas.jarosch at intra2net.com> writes:

> ----------------------------------------------------------------
> From: https://www.intevation.de/roundup/kolab/issue2554
> All the files can be found in the issue tracker. Dunno why
> it didn't generate an automatic email to kolab-devel.
> ----------------------------------------------------------------
>
> Hello together,
>
> last week I started to deeply analyze why the current freebusy performance
> is so horrible for large calendar folders. Gunnar and I developed
> simple patches that halved the memory and CPU usage of pfb.php.
> Deleting an event (=IMAP mail) in a calendar folder takes less
> than a second now on a 3 Ghz P4. That was a great improvement.
>
> But this was only one part of the problem. I could easily kill modern
> hardware by moving around calendar folders. The other part gets
> fixed by "freebusyd". I tested it with importing/moving and deleting
> 10.000 calendar entries and everything felt really smooth.
> Here's the introduction section from the README file:
>
> ------------------------------------------------------
> ###########################################################
> # freebusyd - control daemon for Kolab's freebusy pfb.php #
> ###########################################################
>
> Kolab provides server sided freebusy information for calendar folders.
> A script called pfb.php connects to the IMAP server and processes the data.
> Every change in a calendar folder will trigger a new session of pfb.php,
> which results in many parallel running pfb.phps.
> This easily kills even modern hardware.
>
> freebusyd is a daemon to make sure the generation of freebusy data
> doesn't eat all your CPU/memory on your server: All requests
> for pfb.php will be queued and executed in a controlled manner.
>
> Features:
> * Only one pfb.php is running at a time per owner/folder combination.
> * Global maximum concurrent job limit to relieve the system load.
> * Persistent request queue. Never lose a request by server reboot etc.
> * Requests for folders that get triggered too often in a configurable
>   time period get delayed for some minutes. Speeds up large imports
>   or move of calendar data between folders.
> * Good logging facilities and runs without root privileges.
>
> Optional features:
> * A special patch for the cyrus imap server will notify freebusyd
>   about deleted or renamed IMAP folders. freebusyd will then remove
>   the cached freebusy data for you. No more freebusy ghost entries.
>
> Technical details:
> * Written in C++ and should be able to handle 1.000+ users.
> * Very CPU friendly: The daemon usually sleeps and will only wake
>   up when there is a new request in the queue (uses inotify).
> * Tested with large (10.000 events) calendar folders
> * Fully doxygen documented
> ------------------------------------------------------
>
> Intra2net decided to release the daemon as GPL and it should
> be very Kolab server compatible. It will need some tweaking
> to the paths and users rights, but that should be no big deal.
>
> If you decide to integrate the cyrus folder delete patch,
> there'll be some work needed to support a separated
> freebusy and IMAP server. The patch fixes
> a Kolab bug that is around since 2005:
> https://www.intevation.de/roundup/kolab/issue614

wow, thanks a lot for providing all this stuff. I didn't have much
time to look at this so far but I hope I find the time soon to give
some more feedback and to start incorporating the things. in any case
it sounds really nice.

Cheers,

Gunnar

>
> Enjoy,
> Thomas Jarosch
>
> _______________________________________________
> Kolab-devel mailing list
> Kolab-devel at kolab.org
> https://kolab.org/mailman/listinfo/kolab-devel

-- 
______ http://kdab.com _______________ http://kolab-konsortium.com _

p at rdus Kolab work is funded in part by KDAB and the Kolab Konsortium

____ http://www.pardus.de _________________ http://gunnarwrobel.de _
E-mail : p at rdus.de                                 Dr. Gunnar Wrobel
Tel.   : +49 700 6245 0000                          Bundesstrasse 29
Fax    : +49 721 1513 52322                          D-20146 Hamburg
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   >> Mail at ease - Rent a kolab groupware server at p at rdus <<                 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~