[Kolab-devel] Other kolab scalability issues

[Alain Spineux]

First we need to know what we are speaking about !

I tested kolab-2.2beta1.

postfix support at least 100000 domains with the format
extralargedomainXXXXX.loc.
the main.cf was more than 5Mo large !
imap failed at 68 domains, with line of about 4096 chars.

Second what we need.

Kolab components don't need to be fully dynamic (using lookup at every
request), kolabd daemon is here to detect changes, and restart other
daemons when needed.

Postfix dont need any changes now and also support other solution,
like table lookup, with table in flat file or compiled (berkley db
style).

I dont see any solution for cyrus.
cyrus could be patched to support unlimited config line or table
lookup, static (with admins and realms stored in a flat file)  or
dynamic with data in ldap....

I will start the discussion on cyrus mailing list.

Regards.







On 10/23/07, Fabio Pietrosanti <lists at infosecurity.ch> wrote:
>
>  Gunnar Wrobel wrote:
>  Fabio Pietrosanti <lists at infosecurity.ch> writes:
>
>
>
>  Dear all,
>
> i noticed one bad scalability issue of Kolab while you handle a lot of
> virtual domains.
>
> The main issues are the 'static' configuration that could be made
> dynamic of imapd.conf of Cyrus and main.cf of Postfix.
>
> For Cyrus imapd.conf the "loginrealms:" include the list of all virtual
> domains and it's generated by kolabd.
> If the list of loginrealms became too much long because you are using a
> lot of virtual domains bad things could appens.
>
>  What kind of limits are there and at which point are we going to hit
> these limits? What would cyrus and postfix do if this limit gets
> exceeded?
>
>  I still doesn't know BUT what i found is that loginrealms: is not required
> to have kolab working (at least on my kolab 2.1 installation).
>
>  Could we consider removing it or do you think it's required?
>
>
>
>  Solution would be to add something like this loginrealms dynamic lookup
> described here:
> http://www.irbs.net/internet/info-cyrus/0601/0244.html
>
> or i am wondering which is the real reasons for which we are requiring
> loginrealms, because here they removed it and it worked
> http://www.nabble.com/Multidomain-support-in-Cyrus-eGroupware-with-LDAP--solved--t3665722s3741.html
>
> Now on my multiple-domains (a lot of domains) email servers, i commented
> out the loginrealms and it works fine, i can create users, change
> password, login, change permissions, etc, etc . It seems to work.
>
>
> For postfix's main.conf it's the same for the following configuration
> line that contains the list of domains:
> mydestination
> masquerade_domains
>
> Imho we hould map this lists to dynamic lookups to the ldap directory in
> order to scale better with a large number of virtual domains.
>
>  Would the lookup only happen when the services are started? Or would
> the lookup occur associated with certain user actions? The second
> scenario would slow things down so it might be better if this is
> configurable or gets only activated if the server has more than a
> defined number of virtual domains managed.
>
>
>  I think that i found a good way to manage both situation in one shot.
>
>  The problem of dynamic lookups is that it would require a high number of
> queries under high email load.
>  The ldap client of postfix doesn't support lookup caching and require for
> each new query a new connection.
>
>  Still, the proxymap allow a single permanent connection to be established
> to the ldap server and reused by all lookups.
>
>  So, generally speaking, if we apply the proxymap instead of the ldap direct
> lookup to all the postfix lookup we can gain the advantage of connection
> caching for ALL the ldap lookups removing a great overhead.
>
> http://archives.neohapsis.com/archives/postfix/2004-03/0806.html
>
>  Imho this could give a 'good' performance improvement and allow the
> implementation of FULL DYNAMIC lookup of all postfix tables (also because
> openldap has it's own cache).
>
>  What do you think about those two proposals?
>
>
>  I am just evaluating how to remove what could be somehow a scalability
> issue and find out a "not in the choice of the sysadmin" solution.
>
>  Fabio
>
> _______________________________________________
> Kolab-devel mailing list
> Kolab-devel at kolab.org
> https://kolab.org/mailman/listinfo/kolab-devel
>


-- 
Alain Spineux
aspineux gmail com
May the sources be with you