[Kolab-devel] Debian Packages for Kolab

[Jeroen van Meeuwen (Kolab Systems)]

On 2013-10-24 14:20, Paul Boddie wrote:
> Hello,
> 
> I recently shared some thoughts on the Kolab packages with Torsten 
> Grote and
> Hugo Roy (and also in a blog article), and I thought I should post some 
> of
> them here for further discussion.
> 

Hi Paul,

thanks for providing your thanks and feedback, it is much appreciated. 
Please find some of my thoughts and comments inline.

> First of all, credit is due to Paul Klos for getting the packages into 
> a form
> that allowed me (and others) to try them out. I doubt that I would have 
> been
> so motivated to try Kolab if the software hadn't been available in 
> packaged
> form.
> 

Agreed, many kudos go out to Paul Klos.

Rest assured, we bought him dinner (very "gezellig" BTW), and I don't 
think he's going to need to pay for any beers during FOSDEM ;-)

> The setup-kolab script should be integrated into package 
> post-installation
> and/or be robust enough to be run several times in potentially 
> different ways.
> Consider the scripts provided with PostgreSQL on Debian for doing
> administration, for example: https://wiki.debian.org/PostgreSql
> 

Firstly, we have to acknowledge where "setup-kolab", the beast that it 
currently is, has come from.

In the days this utility was authored (I myself was and still am the 
author/maintainer), an entirely refactored stack for Kolab became 
available, driven by Kolab Systems itself (as an investment) as well as 
customers whom had gladly paid for the costs of Kolab Systems setting it 
all up for them otherwise (i.e., no "setup-kolab" was required from a 
commercial perspective, and in fact a "setup-kolab" bites chunks off of 
our revenue).

But, for us to be able to finally declare the OpenPKG distribution 
method to be obsolete (it occupied a lot of our corporate resources in 
those days), the community has obviously needed some assistance in 
setting up Kolab -- if not simply for the sense of being able to provide 
some set of similar features in setting up Kolab, like there was with 
the OpenPKG distribution method).

This spawned the need for (at least) a one-shot setup utility, which is 
the "setup-kolab" you are looking at today.

It's far from perfect, both in its level of applicability to various 
use-cases as well as deployment scenarios, and far from perfect in its 
current code model(ling) (perhaps).

> In other words, setup-kolab shouldn't be a "one shot" program which 
> doesn't
> like being run again, or where it isn't clear what it does if it is run 
> again.
> Although an all-in-one approach to setting stuff up can be beneficial, 
> I think
> it needs to be complemented by approaches that allow people to keep 
> their
> existing configuration. Maybe setup-kolab does this, but I didn't get 
> this
> impression. (I also wonder if a choice of sensible defaults might 
> reduce the
> need for setup-kolab, but I accept that various policy matters have to 
> be
> specified somewhere, and so setup-kolab may not be completely 
> redundant.)
> 

Now, "setup-kolab" is far from perfect in many regards, but it has 
always aimed to only touch the system settings that Kolab Groupware (the 
solution) would actually care about. Proof is in the fact, for example, 
that you can provide any /etc/postfix/main.cf and find only those 
settings changed that bear relevance for Kolab. In this regard, we 
intend to do preserve existing configuration to whatever extent possible 
-- improvements notwithstanding, of course.

There are few parts where "only touching the settings it cares about" 
are simply not feasible;

   - Augeas (the technology of choice here) does not support the level of 
complexity for Roundcube configuration,

   - Augeas does not support headerless .ini files (for example, 
/etc/imapd.conf),

   - Agueas is sometimes too outdated to provide the features we would 
otherwise use,

   - there are one or two more I can't remember from the top of my head 
right now.

As you have mentioned, the areas that "setup-kolab" right now does not 
serve as well, as it possibly could or even should, are;

   - it does not yet quite perfectly well set up a Kolab Groupware server 
in an existing infrastructure, where you may already have a central 
authentication, authorization and user/group information database (like 
LDAP), a database server (be it MySQL, or PostGres),

   - similarly, it really, really wants to use LDAP, and sets up the rest 
of all of it to do the same. As such, it sets you up without the CAS, 
Kerberos, etc.

   - it does not "check" the current system before actually going about 
trying to set anything up, and this problem is two-fold;

     - it doesn't recognize a previous setup-kolab job having failed 
mid-way, nor
     - does it check or recognize (for example) /dev/shm not being 
available with the right permissions.

   - it also doesn't manage, or even offers to manage, SSL/TLS related 
stuff,

   - it also doesn't manage, or even offers to manage, firewalling 
related stuff,

   - it also doesn't recognize a particular node that "setup-kolab" is 
being run on may have only a limited set of roles (such as, "this is an 
LDAP server").

These, among other things, are items that the openSUSE supporters of 
Kolab have circumvented with scripts to be executed prior to running 
"setup-kolab". It is our intention to facilitate them in bringing these 
nifty things back upstream, while bearing in mind that we do ship for 7 
entirely different Linux and GNU/Linux distributions.

> The authentication aspects of Kolab might be better supported using 
> things
> other than passwords where appropriate. I imagine that this can be 
> tricky for
> the Web applications, but for server-side tasks where the administrator 
> has
> root already (and can therefore read and write various configuration 
> files),
> having key-based logins (or even trust-based logins) might be cleaner 
> than
> having passwords in configuration files. Looking at Debian packages 
> with
> similar application architectures might be informative.
> 
> On this front, I aim to look a bit more at the other groupware systems
> provided by Debian [*] such as Citadel, SOGo and OpenChange. Other 
> kinds of
> systems are also relevant here because managing credentials is a 
> widespread
> problem. Suggestions are appreciated!
> 

Firstly, I have to say that Kolab is not at all locked in with 389 
Directory Server.

It may seem that way because a default "yum install kolab" or "aptitude 
install kolab" includes 389 DS, and there are reasons to do so, as I can 
list reasons why 389 DS is far superior to, say, OpenLDAP in 99% of 
deployment cases, but any such argument is moot, irrelevant and futile.

I have absolutely zero objections against abstracting the Debian and/or 
Ubuntu packages in such a way that in no way does an "aptitude install 
kolab" pull in 389 DS, as long as at least the equivalent of the current 
"setup-kolab" can be made to do the same it does with 389 DS today, but 
with OpenLDAP (or whichever).

> [*] https://wiki.debian.org/Groupware
> 
> It's a bit unfortunate that Kolab prefers 389 to other LDAP servers and 
> has
> preferences for other things that effectively make those things 
> unsupported.

When you say "other LDAP servers", I cannot but be very curious about 
what it is you regret about 389 DS, and prefer in other LDAP servers, as 
well as what LDAP servers are included with "other LDAP servers" 
exactly.

We actually support a wide variety of LDAP server technologies, 
including all the way back to Sun DS versions declared EOL, as well as 
OpenLDAP, Novell eDirectory and M$ Active Directory.

We also support a  wide variety of authentication databases and 
protocols, among which (preferably) FreeIPA, plain old Kerberos, (H)OTP, 
*SQL, plaintext databases, PAM -- it's just not something a 
"setup-kolab" has been made responsible for achieving.

> I imagine it could take a lot of work to support everything that could
> potentially support Kolab, but having seen issues with Exim and 
> Postfix, it
> might be nice to allow existing choices to be respected (and also not 
> require
> people to start from scratch).

I don't understand how switching an MTA from "foo" to "bar" in order to 
deploy a rather well featured Groupware environment is necessarily 
invoking people to "start from scratch", but I sure appreciate that 
Kolab should "Just Work(TM)" equally well with sendmail, exim, qmail or 
whatever else may already be on a system.

That said, the package management becomes a problem. Should I only 
require an MTA, then features for Kolab that we do not provide Exim 
integration with / for basically would mean that *ALL* "aptitude install 
kolab" deployments would lack those features, since Exim is the default 
for Debian -- regardless of whether the system was or was not some sort 
of mail server before running "setup-kolab".

> There shouldn't be any reason why installing
> Kolab wouldn't be adding the "icing to the cake" provided by 
> already-installed
> mail, IMAP and LDAP infrastructure, in other words adding the extra 
> facilities
> of Kolab, even if one might insist that only new "instances" (of 
> mailboxes,
> databases and so on) or migrated ones would be usable by Kolab.
> 

I've done this sort of exercise many, many times before, and they are 
likely to be a large part of my (and my colleague's) agenda for quite 
some while to come. Bearing in mind that it lacks the publicly 
documented HOWTO you might be looking for, it is (even with such HOWTOs) 
also rather complex to automate.

We DON'T require Cyrus IMAP, nor 389 DS, nor the apache webserver, nor 
MySQL.

This is *solely* what happens if you simply "aptitude install kolab" - 
the one entirely empty meta-package *solely* intended for the 
installation of a default Kolab Groupware stack on what is probably 
otherwise an unoccupied system.

Please note that *NONE* of the actual Kolab Groupware software 
components installed require another functional component to be of a 
particular type of technology (bugs in packaging notwithstanding). For 
example, 'kolab-server' does not require 'kolab-ldap' nor '389-ds', and 
in fact it requires no '*ldap*' thing or technology whatsoever.

Other issues (you may find pull in dependencies you were not expecting) 
are PHP lacking support for the getEffectiveRights LDAP controls 
extensions, pulling in Mozilla LDAP tools.

> Again, the other groupware packages may support choices of 
> dependencies, and
> it will be interesting to see how they manage this. For example, the 
> citadel-
> server package configuration mentions the following:
> 
> Suggests: postfix | exim4 | citadel-mta | mail-transport-agent
> 
> Of course, actually supporting multiple dependency options may require 
> more
> work because it's unlikely that all these things work in the same way 
> and will
> thus need explicit support in Kolab.
> 

This is work that, I have to admit, Kolab Systems does either for 
customers, or because it would enhance the entire groupware solution as 
a whole.

We do not actively seek integration with exim4, nor qmail, nor sendmail, 
for as far as we are concerned, there is no reason to do so.

Other than that, community's preferences (which I have to say are very 
valid(!), in their own merits(!), while I'm also saying you cannot 
expect us to do that for you[1]), are the community's to work on and try 
to implement.

[1] https://kolab.org/blog/grote/2013/10/23/call-participation

Frankly, with all due respect, Kolab Systems -- Ye Vendor and by far, 
far and away the largest contributor to all components that make up 
Kolab Groupware, including the packaging, including those you have for 
Debian/Ubuntu systems -- is actually looking more in to feature 
enhancements that go a little beyond whether or not Dovecot or Exim may 
be installed on anyone's given system. Just to name a few;

   - Instant Messaging,
   - Voice and Video (Conferencing),
   - Properly(!) implemented web-client decryption and signing of 
communications,
   - Relations between email threads, files, tasks, events and notes 
(which may get you to "Project Management").

These are chunks of effort a community (member) cannot (typically) 
afford to substantially contribute to, and therefore require the full 
attention of a custodian, a guardian, like Kolab Systems.

> Finally, I wonder whether targeting "sid" (unstable) rather than 
> "wheezy"
> (stable) isn't best for packaging work given that the Debian packaging
> community usually prefers this, and also things like the 389 packages 
> are
> already in "sid" but must presumably be backported to "wheezy" to 
> provide
> Kolab packages. Naturally, backported packages can be made available as 
> a
> consequence of the main packaging effort.
> 

For package integration, somebody (sunweaver, you know who you are), has 
already made valuable suggestions to the packaging that we do, and it 
might indeed be better to target Sid.

However, up and until now we target systems that might feasibly run 
actual Kolab Groupware deployments in real life (like we don't target 
Fedora Rawhide).

> Any comments on the above?
> 

Your feedback is very, very much appreciated.

Your thoughts are along the exact same lines of where ultimately I would 
like to see pykolab's "setup-kolab", but other things as well[2], end 
up, as well as where other people, elsewhere, have already been 
scripting some stuff[3].

[2] https://lists.kolab.org/pipermail/devel/2012-March/013280.html
[3] 
https://build.opensuse.org/package/show/server:Kolab:UNSTABLE/kolab-scripts

I'm a strong fan of multiple people collaborating to achieve the same or 
similar goals -- in over a decade of Free Software contributions, I have 
yet to actually decline a patch submitted to me, and I have yet to 
disallow SCM commit privileges to anyone who requested them.

To me, "no" is an invalid answer, but I believe "how?" is a valid 
question in return. It is about consensus about what your goals are, and 
matching those with my goals and other people's goals, in some way - 
possibly a difficult process in and by itself.

I think to achieve our goals in these cases, for "setup-kolab" and 
friends alike, we have to address the following;

   - What is the flow we would like to see "setup-kolab" complete?

     I'm thinking there's the obvious answer file seeding the dialog's 
answers, but perhaps also;

       - a check on the current system status (software available, etc.) 
in order to determine what individual components need to be set up for 
Kolab, possibly including the establishment of what roles this node may 
fulfill given that certain software may or may not be installed on the 
current system.

   - What can we do in each of the steps in such a flow, modeling what 
happens in the next step(s) of the flow?

     I.e. detect a previous run of setup-kolab (perhaps having failed) 
rolling forward (re-attempting) what else it needed to do, or

     I.e. replace and/or amend/extend "setup_mta" with more specific 
"setup_postfix" and "setup_sendmail", each of which to be executed only 
when appropriate.

   - etc., etc., (...)

If we could work together on at least defining what "setup-kolab" should 
actually be like, then I'm fairly confident we can also achieve that 
(I'm a pretty decent hacker after all, if I may say so myself). If we 
can't work together however, it is unlikely to ever happen.

Yours sincerely,

Kind regards,

Jeroen van Meeuwen

-- 
Systems Architect, Kolab Systems AG

e: vanmeeuwen at kolabsys.com
m: +44 74 2516 3817
w: http://www.kolabsys.com

pgp: 9342 BF08