[Kolab-devel] Debian Packages for Kolab

[Paul Boddie]

On Thursday 24. October 2013 22.28.17 Jeroen van Meeuwen (Kolab Systems) 
wrote:
> On 2013-10-24 14:20, Paul Boddie wrote:
> > The setup-kolab script should be integrated into package
> > post-installation
> > and/or be robust enough to be run several times in potentially
> > different ways.
> > Consider the scripts provided with PostgreSQL on Debian for doing
> > administration, for example: https://wiki.debian.org/PostgreSql

By the way, I'm not really criticising setup-kolab, but I am putting myself in 
the position of someone having a conversation with Debian experts and 
imagining what they might say about it.

> Firstly, we have to acknowledge where "setup-kolab", the beast that it
> currently is, has come from.
> 
> In the days this utility was authored (I myself was and still am the
> author/maintainer), an entirely refactored stack for Kolab became
> available, driven by Kolab Systems itself (as an investment) as well as
> customers whom had gladly paid for the costs of Kolab Systems setting it
> all up for them otherwise (i.e., no "setup-kolab" was required from a
> commercial perspective, and in fact a "setup-kolab" bites chunks off of
> our revenue).

Well, one can certainly have the perspective that if people cannot figure 
things out for themselves then there are support opportunities to be had in 
setting things up for them, but the risk is also there that people step down 
from the ladder at the first rung/step and choose an alternative to Kolab 
instead. That certainly doesn't drive business in the direction of Kolab 
providers.

> But, for us to be able to finally declare the OpenPKG distribution
> method to be obsolete (it occupied a lot of our corporate resources in
> those days), the community has obviously needed some assistance in
> setting up Kolab -- if not simply for the sense of being able to provide
> some set of similar features in setting up Kolab, like there was with
> the OpenPKG distribution method).
> 
> This spawned the need for (at least) a one-shot setup utility, which is
> the "setup-kolab" you are looking at today.
> 
> It's far from perfect, both in its level of applicability to various
> use-cases as well as deployment scenarios, and far from perfect in its
> current code model(ling) (perhaps).

I think it is good that you have embraced the distributions even if it makes 
for more work, and I'm happy to look into this, not only as someone who uses 
Debian but also as someone who is a Python developer.

[...]

> Now, "setup-kolab" is far from perfect in many regards, but it has
> always aimed to only touch the system settings that Kolab Groupware (the
> solution) would actually care about. Proof is in the fact, for example,
> that you can provide any /etc/postfix/main.cf and find only those
> settings changed that bear relevance for Kolab. In this regard, we
> intend to do preserve existing configuration to whatever extent possible
> -- improvements notwithstanding, of course.

I think that my only real complaint was that previously - not sure about new 
installations today - Kolab would object to the presence of Exim, which would 
itself be installed on a Debian system after some existing package "tasks" had 
been installed. As I mention below, this led to some confusion and a suggested 
remedy to start with a fresh installation of Debian.

> There are few parts where "only touching the settings it cares about"
> are simply not feasible;
> 
>    - Augeas (the technology of choice here) does not support the level of
> complexity for Roundcube configuration,
> 
>    - Augeas does not support headerless .ini files (for example,
> /etc/imapd.conf),
> 
>    - Agueas is sometimes too outdated to provide the features we would
> otherwise use,
> 
>    - there are one or two more I can't remember from the top of my head
> right now.

Understood.

> As you have mentioned, the areas that "setup-kolab" right now does not
> serve as well, as it possibly could or even should, are;
> 
>    - it does not yet quite perfectly well set up a Kolab Groupware server
> in an existing infrastructure, where you may already have a central
> authentication, authorization and user/group information database (like
> LDAP), a database server (be it MySQL, or PostGres),
> 
>    - similarly, it really, really wants to use LDAP, and sets up the rest
> of all of it to do the same. As such, it sets you up without the CAS,
> Kerberos, etc.

I can understand that this is done to minimise complexity, so there are no 
real complaints from me here.

>    - it does not "check" the current system before actually going about
> trying to set anything up, and this problem is two-fold;
> 
>      - it doesn't recognize a previous setup-kolab job having failed
> mid-way, nor
>      - does it check or recognize (for example) /dev/shm not being
> available with the right permissions.

This is perhaps where a lot of benefit could be had for relatively little 
effort.

>    - it also doesn't manage, or even offers to manage, SSL/TLS related
> stuff,
> 
>    - it also doesn't manage, or even offers to manage, firewalling
> related stuff,
> 
>    - it also doesn't recognize a particular node that "setup-kolab" is
> being run on may have only a limited set of roles (such as, "this is an
> LDAP server").
> 
> These, among other things, are items that the openSUSE supporters of
> Kolab have circumvented with scripts to be executed prior to running
> "setup-kolab". It is our intention to facilitate them in bringing these
> nifty things back upstream, while bearing in mind that we do ship for 7
> entirely different Linux and GNU/Linux distributions.

I think it is right to delegate these other things to the distribution and the 
user. There are plenty of choices around SSL/TLS, for example, that should be 
separate from Kolab configuration, and it would be a nightmare to try and tie 
the two things together.

[Storing passwords in configuration files]

> > On this front, I aim to look a bit more at the other groupware systems
> > provided by Debian [*] such as Citadel, SOGo and OpenChange. Other
> > kinds of
> > systems are also relevant here because managing credentials is a
> > widespread
> > problem. Suggestions are appreciated!
> 
> Firstly, I have to say that Kolab is not at all locked in with 389
> Directory Server.
> 
> It may seem that way because a default "yum install kolab" or "aptitude
> install kolab" includes 389 DS, and there are reasons to do so, as I can
> list reasons why 389 DS is far superior to, say, OpenLDAP in 99% of
> deployment cases, but any such argument is moot, irrelevant and futile.
> 
> I have absolutely zero objections against abstracting the Debian and/or
> Ubuntu packages in such a way that in no way does an "aptitude install
> kolab" pull in 389 DS, as long as at least the equivalent of the current
> "setup-kolab" can be made to do the same it does with 389 DS today, but
> with OpenLDAP (or whichever).

This was my misunderstanding, I think. Since Kolab preferred 389-ds, perhaps 
as a result of the repository priorities being set up to favour the Kolab 
repository (which provided its own 389-ds package), I got the impression that 
other LDAP solutions were not supported (and I think the documentation 
concurred with this).

Indeed, what I aim to discover now is whether I can set up Kolab on wheezy 
with anything providing ldap-server, and since 389-ds isn't in wheezy I should 
end up with something else in this role. This may not be a good solution, but 
I suppose it should demonstrate both what you write above and also that the 
packages already function as expected.

> > [*] https://wiki.debian.org/Groupware
> > 
> > It's a bit unfortunate that Kolab prefers 389 to other LDAP servers and
> > has
> > preferences for other things that effectively make those things
> > unsupported.
> 
> When you say "other LDAP servers", I cannot but be very curious about
> what it is you regret about 389 DS, and prefer in other LDAP servers, as
> well as what LDAP servers are included with "other LDAP servers"
> exactly.
> 
> We actually support a wide variety of LDAP server technologies,
> including all the way back to Sun DS versions declared EOL, as well as
> OpenLDAP, Novell eDirectory and M$ Active Directory.
> 
> We also support a  wide variety of authentication databases and
> protocols, among which (preferably) FreeIPA, plain old Kerberos, (H)OTP,
> *SQL, plaintext databases, PAM -- it's just not something a
> "setup-kolab" has been made responsible for achieving.

Again, this is my misunderstanding, for which I apologise. (I don't actually 
have any opinions about LDAP servers.)

> > I imagine it could take a lot of work to support everything that could
> > potentially support Kolab, but having seen issues with Exim and
> > Postfix, it
> > might be nice to allow existing choices to be respected (and also not
> > require
> > people to start from scratch).
> 
> I don't understand how switching an MTA from "foo" to "bar" in order to
> deploy a rather well featured Groupware environment is necessarily
> invoking people to "start from scratch", but I sure appreciate that
> Kolab should "Just Work(TM)" equally well with sendmail, exim, qmail or
> whatever else may already be on a system.
> 
> That said, the package management becomes a problem. Should I only
> require an MTA, then features for Kolab that we do not provide Exim
> integration with / for basically would mean that *ALL* "aptitude install
> kolab" deployments would lack those features, since Exim is the default
> for Debian -- regardless of whether the system was or was not some sort
> of mail server before running "setup-kolab".

I honestly cannot say what level of integration it is with Postfix that Kolab 
requires and if using something else would result in degraded capabilities. 
The thing I somewhat regret is that it appeared impossible to install Kolab 
without starting with a completely clean wheezy installation, although I 
suspect that it only really needed to be free of certain packages that were 
causing problems for the setup-kolab script. (I really have to think back to 
what really happened when it failed, which is a few months ago now.)

These days it isn't too much hard work to set up a fresh installation in some 
kind of virtual environment, and maybe everyone else does that to begin with, 
but I just found the troubleshooting of such difficulties and the suggested 
remedy to be a bit opaque. Naturally, I understand that it is easier for 
everyone if people start with a known configuration and eliminate sources of 
unnecessary errors.

> > There shouldn't be any reason why installing
> > Kolab wouldn't be adding the "icing to the cake" provided by
> > already-installed
> > mail, IMAP and LDAP infrastructure, in other words adding the extra
> > facilities
> > of Kolab, even if one might insist that only new "instances" (of
> > mailboxes,
> > databases and so on) or migrated ones would be usable by Kolab.
> 
> I've done this sort of exercise many, many times before, and they are
> likely to be a large part of my (and my colleague's) agenda for quite
> some while to come. Bearing in mind that it lacks the publicly
> documented HOWTO you might be looking for, it is (even with such HOWTOs)
> also rather complex to automate.

I accept this. I am also of the opinion that automation is better than 
complicated documentation, so I'm not sure I would want to see a HOWTO that 
covers the same ground that a script does. But I also wonder whether a 
collection of scripts might not be better than one that tries to do it all.

> We DON'T require Cyrus IMAP, nor 389 DS, nor the apache webserver, nor
> MySQL.
> 
> This is *solely* what happens if you simply "aptitude install kolab" -
> the one entirely empty meta-package *solely* intended for the
> installation of a default Kolab Groupware stack on what is probably
> otherwise an unoccupied system.
> 
> Please note that *NONE* of the actual Kolab Groupware software
> components installed require another functional component to be of a
> particular type of technology (bugs in packaging notwithstanding). For
> example, 'kolab-server' does not require 'kolab-ldap' nor '389-ds', and
> in fact it requires no '*ldap*' thing or technology whatsoever.
> 
> Other issues (you may find pull in dependencies you were not expecting)
> are PHP lacking support for the getEffectiveRights LDAP controls
> extensions, pulling in Mozilla LDAP tools.

Right. So my expectations of what the kolab metapackage can do are perhaps a 
bit inappropriate. Here, I can only refer to things like PostgreSQL and other 
systems I have some experience with in Debian, and also attempt to see what 
other unfamiliar packages do, too. Certainly, with something like PostgreSQL 
there are many policy and configuration decisions that could be taken on 
behalf of the user as well as others that should be left to the user, and I 
would even argue that the allocation of these decisions isn't necessarily 
optimal now (things like the default locale for databases, for example).

So I aim to try and see what other people do to find the right trade-offs 
between convenience and configurability.

[Dependency flexibility, particularly for MTAs]

> This is work that, I have to admit, Kolab Systems does either for
> customers, or because it would enhance the entire groupware solution as
> a whole.
> 
> We do not actively seek integration with exim4, nor qmail, nor sendmail,
> for as far as we are concerned, there is no reason to do so.
> 
> Other than that, community's preferences (which I have to say are very
> valid(!), in their own merits(!), while I'm also saying you cannot
> expect us to do that for you[1]), are the community's to work on and try
> to implement.
> 
> [1] https://kolab.org/blog/grote/2013/10/23/call-participation

Well, I'm not asking anyone to do that kind of work, especially if it is a 
non-trivial exercise to extend integration with a bunch of systems nobody 
really cares about. I've done this kind of thing before (not with mail 
systems) and there's only a limited amount of reward in merely showing that 
these things can be done when nobody wants them done.

> Frankly, with all due respect, Kolab Systems -- Ye Vendor and by far,
> far and away the largest contributor to all components that make up
> Kolab Groupware, including the packaging, including those you have for
> Debian/Ubuntu systems -- is actually looking more in to feature
> enhancements that go a little beyond whether or not Dovecot or Exim may
> be installed on anyone's given system. Just to name a few;
> 
>    - Instant Messaging,
>    - Voice and Video (Conferencing),
>    - Properly(!) implemented web-client decryption and signing of
> communications,
>    - Relations between email threads, files, tasks, events and notes
> (which may get you to "Project Management").
> 
> These are chunks of effort a community (member) cannot (typically)
> afford to substantially contribute to, and therefore require the full
> attention of a custodian, a guardian, like Kolab Systems.

Well, perhaps the community can fill in the gap with regard to packaging. I 
think that's why I offered my help in the first place. ;-)

> > Finally, I wonder whether targeting "sid" (unstable) rather than
> > "wheezy"
> > (stable) isn't best for packaging work given that the Debian packaging
> > community usually prefers this, and also things like the 389 packages
> > are
> > already in "sid" but must presumably be backported to "wheezy" to
> > provide
> > Kolab packages. Naturally, backported packages can be made available as
> > a
> > consequence of the main packaging effort.
> 
> For package integration, somebody (sunweaver, you know who you are), has
> already made valuable suggestions to the packaging that we do, and it
> might indeed be better to target Sid.
> 
> However, up and until now we target systems that might feasibly run
> actual Kolab Groupware deployments in real life (like we don't target
> Fedora Rawhide).

The reason I mention sid is because the first thing one gets told when 
submitting packaging work to Debian is to target sid because that's how the 
packages get into the system and end up in the next stable release. Obviously, 
real-world deployments probably won't use sid, which means that either people 
wait to see the packages appear in stable (or maybe testing) or the packages 
are backported to stable.

> > Any comments on the above?
> 
> Your feedback is very, very much appreciated.
> 
> Your thoughts are along the exact same lines of where ultimately I would
> like to see pykolab's "setup-kolab", but other things as well[2], end
> up, as well as where other people, elsewhere, have already been
> scripting some stuff[3].
> 
> [2] https://lists.kolab.org/pipermail/devel/2012-March/013280.html
> [3]
> https://build.opensuse.org/package/show/server:Kolab:UNSTABLE/kolab-scripts

Yes, I think this has the basis of an approach that could work for Debian, as 
I understand it and as I understand the demands of Debian.

> I'm a strong fan of multiple people collaborating to achieve the same or
> similar goals -- in over a decade of Free Software contributions, I have
> yet to actually decline a patch submitted to me, and I have yet to
> disallow SCM commit privileges to anyone who requested them.
> 
> To me, "no" is an invalid answer, but I believe "how?" is a valid
> question in return. It is about consensus about what your goals are, and
> matching those with my goals and other people's goals, in some way -
> possibly a difficult process in and by itself.
> 
> I think to achieve our goals in these cases, for "setup-kolab" and
> friends alike, we have to address the following;
> 
>    - What is the flow we would like to see "setup-kolab" complete?
> 
>      I'm thinking there's the obvious answer file seeding the dialog's
> answers, but perhaps also;
> 
>        - a check on the current system status (software available, etc.)
> in order to determine what individual components need to be set up for
> Kolab, possibly including the establishment of what roles this node may
> fulfill given that certain software may or may not be installed on the
> current system.

We should be able to lean on the dependency management of the system. You 
mentioned problems with /dev/shm, and that is perhaps something that needs 
additional controls, although it also sounds like something that a package 
should be testing for in one of its scripts.

>    - What can we do in each of the steps in such a flow, modeling what
> happens in the next step(s) of the flow?
> 
>      I.e. detect a previous run of setup-kolab (perhaps having failed)
> rolling forward (re-attempting) what else it needed to do, or
> 
>      I.e. replace and/or amend/extend "setup_mta" with more specific
> "setup_postfix" and "setup_sendmail", each of which to be executed only
> when appropriate.
> 
>    - etc., etc., (...)
> 
> If we could work together on at least defining what "setup-kolab" should
> actually be like, then I'm fairly confident we can also achieve that
> (I'm a pretty decent hacker after all, if I may say so myself). If we
> can't work together however, it is unlikely to ever happen.

As I said, I'm no expert at Debian packaging (although I'm proud of my own 
modest contribution to the mountain of packages in Debian), and I can only 
bring experience around other kinds of systems to bear on this problem, but I 
also see the opportunity to help improve the situation. So, as I noted above, 
I'll look at how others perform similar tasks and see if there's anything we 
can learn from their systems.

Paul