[Kolab-devel] LDAP performance

[Buchan Milne]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Andreas Gungl wrote:
> Hi, 
>  
> after having managed the import of about 18.000 addresses into our Kolab 2 
> (test) server, I would like to provide some feedback and ask some 
> questions. 
>  
> The many contacts have been added as addresses (not users). Importing via 
> ldapadd was very slow. It took me more than 5 hours on our PIII-500 test 
> machine with 256 MB RAM plus slow HDD. But hey, import needs to be done 
> only once. :-) 

You should have done your tuning first, you box should be able to do
this substantially faster IMHO.

> Handling the data in the admin interface is problematic, there are issues 
> in the tracker (admin interface can't handle more addresses than in the 
> LDAP limit AFAICT). Well, I've set up some Perl scripts to support the 
> update of these addresses.

Personally, I use Evolution for this :-).

> So I don't have to care. 
>  
> The processing of queries is very different. I experience _very_ fast 
> searches out of KAddressbook / Kontact (less than 0.5 sec for serveral 
> names), while Thunderbird / Mozilla are really slow (10-15 sec for the 
> same queries). Using TB/Mozilla, the CPU load is above 90% for quite a 
> while. I think that's because of the way how those tools define the lookup 
> ("name plus email", but they search e.g. the givenName as well, at least I 
> think so after looking at it in ethereal). 

It may be due to Mozilla including more search filters, probably
including attributes you haven't indexed.

BTW, at least with kdepim-3.3 (on Mandrake, not sure which branch it
comes from), I found libkabc_ldap/Kontact to be way too slow, since it
seemed to run a search, and then run a subsequent search for every entry
it found in the first search.

Mozilla and Evolution have always performed adequately for me.

> Following the thread about tweaking the LDAP and DB backend configuration, 
> I've set (on Kolab 2 RC 1) in /kolab/var/openldap/openldap-data/DB_CONFIG 
>   set_cachesize 0 65000000 1 
>   set_lg_bsize  2097152 

Note that you need to run db_recover for these changes to take effect,
for example:

# su -c 'db_recover -h /kolab/var/openldap/openldap-data' -s /bin/bash -
ldap

>  
> I expect it to be 64 MB cache, is that okay this way? I must admit that I 
> couldn't completely follow the explanations of Dieter. Hm, I blame it to 
> my English. 

I don't know if my mail to this list last week with a tuning script made
it through, I'm cc'ing you directly and attaching the script ... but you
may need to customise it a bit.

> Another change was made in /kolab/etc/kolab/templates/slapd.conf.template 
>   idlecachesize 10000 
>   sizelimit 20000 
>  
> That should cache ~50% of the data, while I can query all records without 
> getting to the limit. Does that look fine? 
>  
> I'm really unsure about the config values. But changing them and trying to 
> measure is not as easy as it seems in the first place, so I thought I 
> might ask here. 
>  
> Anyway, the good news is that Kolab can handle that data quite fine. But 
> if you have Mozilla LDAP clients, you will need a strong CPU. 

I really don't find this to be the case myself (having use Mozilla as an
 LDAP client for over 3 years).

BTW, I don't usually browse the Address Book in Mozilla, but usually
just start typing into the "To"-type fields in a new mail ... which
results in Mozilla searching on the partial string. Browsing the entire
Mozilla Address Book may result in different behaviour (so it would be
useful if you reported exactly what you were doing).

Regards,
Buchan

- --
Buchan Milne                      Senior Support Technician
Obsidian Systems                  http://www.obsidian.co.za
B.Eng          RHCE (803004789010797),LPIC-1 (LPI000074592)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFCkdxHrJK6UGDSBKcRAoBpAJ0UVQlBW+uTRILVK3kSrNl71DOhLgCguB4d
VolWU6Xpg7wZoXZTQhWNQD4=
=V5Ng
-----END PGP SIGNATURE-----
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ldap-tune
URL: <http://lists.kolab.org/pipermail/devel/attachments/20050523/6e873780/attachment.ksh>