Kolab2 Slapd hanging - master/slave replication issues

[Dieter Kluenter]

Stephan Buys <list at codefusion.co.za> writes:

> Hi,
>
> On Tuesday 15 March 2005 10:08, Dieter Kluenter wrote:
>> > We have a big Kolab2 server with about 350 users on it and 2 slave servers
>> > in remote geographic locations. The slave servers have between 50 and 100
>> > users each.
>> 
>> These are not large numbers, in terms of directory speaking :-)
> I know :-) But Kolab installation-wise its not too bad for one P4 with an IDE HDD,
> and 1 GB memory. You have to remember that Kolab makes a _lot_ of LDAP
> request right through the message pipeline.
>
>> Please give some more details
>> - OpenLDAP versions of master and slaves
> openldap-2.2.17-2.2.0 (OpenPKG)
>
>> - Contents of DB_CONFIG
> D.N.E. As a default this is not configured on Kolab (1 and 2). I think this is
> a pretty big oversight.

If not present create a file DB_CONFIG in the database directory,
which is /kolab/var/openldap/openldap-data

,----[ DB_CONFIG ]
| set_cachesize   0       26214400        1
| # set_flags     DB_TXN_NOSYNC
| set_tmp_dir /dev/shm
`----

which will allow 26MB cache, (default is 256kB) which would be
sufficient for 25k entries. When addding bulk entries to slapd
activate DB_TXN_NOSNYC, which will speed up write activities but
database synchronisation will be delayed, but be careful only set this
flag when adding large ldif files.

>> - cachesize in slapd.conf
> Not configured, defaults in other words

the default is 1000 entries, this would be sufficient for you
>
>> - idlecachsize in slapd.conf
> Not configured, defaults in other words

default is zero, to speed up search for indexed entries, set a value
that will meet your requirements, I would say 2000 would be
sufficient.
see man slapd-bdb(5) and man slapd.conf(5) for mor information.
>> - indices in slapd.conf
> index   objectClass     eq
> index   uid             eq
> index   mail            eq
> index   alias           eq

I would add a presence index (pres) as you rarely do a equality
search. After adding pres rebuild your database with slapindex(8)

>> - database definition in slapd.conf
> database        bdb
> checkpoint      128 10
> directory       /kolab/var/openldap/openldap-data
>
>> - BerkeleyDB version and number of patches applied
> db-4.2.52.2-2.2.0 (OpenPKG)
>
>> > 2) When slapd hangs we need to do a db_recover to get it back up and running.
>> > After having to run db_recover on the slave as well as the master servers the
>> > databases have now become inconsistent.
>> 
>> A database corruption occurs only with heavy write load and
>> insufficiened cache size or a flag DB_TXN_NOSYNC set in DB_CONFIG.
> As I suspected. I have a persistent 200Mb free memory on the box and I'm
> thinking of making the cache nice and big. 
>
>> > So my questions:
>> >
>> > 1) What can I do to make slapd more robust? Pre-forking, more
>> >    children processes?
>> 
>> That depends on your OS and hardware as well as on the number of
>> connections in a given period.
> Is there a good way to measure number of connections per second using OpenLDAP?

Well back monitor would give you some hints. I don't know wether you
built OpenLDAP with back-monitor, if so, just add a database
declaration to your slapd.conf

database monitor

no further configuration. restart slapd and you can search with

ldapsearch -H ldap://my.host -b "cn=monitor" -s sub + -x
ldapsearch -H ldap://my.host -b cn=connections,cn=monitor" monitorCounter -s sub -x
ldapsearch -H ldap://my.host -b cn=total,cn=connections,cn=monitor"
monitorCounter -s one -x

As all attributes are operational attributes you have to add '+' to
the searchstring if no attribute is defined.

>> > 2) How easy is it to re-sync the master-slave databases? Can I stop the servers,
>> > copy the master dbs to the slaves and start up again?
>> 
>> You may slapcat the master and slapadd on the slave.

> Copying the database across is the recommended way in the OpenLDAP
> Admin Guide.

That must be an old version of Admin Guide :-)

> I have no issue with using slapcat and slapadd though. This can be done at any stage?

the database has to be idle, if you cannot secure this stop slapd and
slapcat, when slapadding the database has to be empty and slapd not
running. 

>> > Any tuning/optimisation tips would be greatly appreciated.
>> 
>> I would like to, but a few more informations would be helpful :-)
> Well, I hope these details help.

-Dieter

-- 
Dieter Klünter | Systemberatung
http://www.dkluenter.de
GPG Key ID:01443B53