[Kolab-devel] How to test the Kolab server (was: Re: Brain Dump of the Week: Ideas, Plans and Realities ...)

[Gunnar Wrobel]

Hi Richard,

Quoting Richard Bos <ml at radoeka.nl>:

[snip]

> > 2. Packaging and Distribution Future
> >
> > The current official and only commercially supported platform
> for Kolab
> > is OpenPKG -- unfortunately the future of OpenPKG, technically
> and
> > politically (read business model), is unknown -- at least to
> us.  On the
> > other hand there is the contributed native packaging of Kolab
> server for
> > many distributions, with varying grades of functionality and
> stability.
> >
> > Currently we are not sure what to make of all this.  It is
> clear that we
> > will continue to need one defined distribution with stability
> and
> > support standards for our enterprise business -- but this
> might not
> > necessary be OpenPKG for ever.  Besides this we want to make
> Kolab
> > server as attractive and easy to setup for users and
> developers as
> > possible, which not at least would mean to support good native
> > packaging, but once again the strategy how to achieve this
> with limited
> > resources is work in progress.
>
> May I point your attention to the openSUSE build service?

[snip]

As pointed out by both Sascha and Bernhard before I also don't think  
that building the packages is our core problem. Let me try to explain  
why and then come back to the core problem we see: testing the server.

The openSUSE build service is of course interesting. It would provide  
one piece of a continuous integration setup - where code commits  
directly lead to an updated server release that could be tested. Using  
something like the openSUSE build service would help in producing  
releases for several platforms simultaneously. Compiling these for  
several distributions would certainly help the package quality.

We have had our problems with supporting the packaging efforts of the  
community in the past and many people from the native ports (of course  
including you) helped the Kolab project to improve in that area.  
Things are still not perfect but I would say they progressed quite  
well in the recent years.

But for the Kolab Konsortium the community cannot be the only driving  
factor. I'm already happy that the process is as open as it currently  
is. But in order to finance the Kolab server we are required to  
deliver a rock solid mail server for the customers. This is simply  
essential in the area of mail services. And that means we need to  
thoroughly test the server.

This has been one of the driving factors for the OpenPKG decision.  
This way the server runs on many Linux base distros while we can still  
test only one specific packaging setup and the corresponsing upgrade  
path.

Providing packages for several distributions won't really help with  
the testing problem. In fact we'd get more setups we'd need to test  
and we already have enough trouble testing the one setup we currently  
support.

So I think the core issue to discuss would be testing. With the manual  
testing scheme we are currently using we can only support one  
distribution. If that is OpenPKG or something else in the future does  
not really matter.

The only alternative I see - and that has also already been mentioned  
- is a test suite for the complete server (or even the Kolab server  
network if we consider slaves).

The advantages of such a test suite would be:

  1) It would be independant of the distribution and at one point we might be
     able to say: A Kolab Server is an installation that runs fine  
with this test
     suite. That would be cool - but also a lot of work.

  2) It would document the server functionality. Right now sometimes the best
     solution to determine how something works on the Kolab server is to check
     the OpenPKG variant. This is not really satisfying.

  3) It would speed up our development process. Having a test suite  
always helps
     developers to point out potential side effects of their commits that they
     did not think of.

  4) It would allow the communities from the native ports to help improving the
     testing. Right now issues from the native ports are problematic because it
     is difficult to determine if a reported issue is a distribution specific
     problem or not.

And the disadvantages:

  1) It would be a lot of work to provide a test suite that covers a lot of the
     Kolab Server functionality.

  2) If such a test suite exists it is tempting to write a new test  
for each new
     issue (if possible). This would slow down the development process.

  3) The test suite would have to be maintained. If it is a large test  
suite then
     it might be hard to do larger rearragements for the Kolab server  
as the test
     suite suddenly fails in many places. This is a particular problem of such
     high level testing as it would be necessary for the Kolab server.

  4) Makes adding new functionality hard as all the additional features need
     testing, too.

The current system also has some advantages:

  1) Tried and tested :) system.

  2) Works well with the available man power.

And the main disadvantages:

  1) Limits us to a single distribution.

  2) Makes adding new functionality hard as all the additional features need
     testing, too (see web client).

  3) Not reliable as we can't possibly test all the different features of the
     Kolab server manually for each release.


This is not an exhaustive overview and my personal view of the topic.  
I just wanted to highlight the difficulties I currently see.

As to a possible solution: The two systems are in no way mutually  
exclusive. There is no reason why we should not have both of them at  
the same time. So why not just start with a very small test suite?  
Something that tests just one area of the Kolab server.

I would suggest to go for the resource management  
(Kolab_FreeBusy/Kolab_Filter) of the Kolab server. In my experience  
this is one of the things that is hard to get right on the native  
distributions and that tends to break easily. So there should be a  
benefit for the OpenPKG version as well.

W could provide a defined way on how to set up and run this test suite  
as well as describing how people can add new tests. By supporting a  
small - or even tiny  - testsuite over several server releases we  
might get what we need to know in order to decide which way to go. We  
might get an idea on whether such a test suite is hard to maintain and  
how much effort adding new tests would be. We might be able to see if  
people are willing and able to contribute.

My 2cents on testing.

Cheers,

Gunnar


-- 
____ http://www.pardus.de[1] _________________
http://gunnarwrobel.de[2] _

E-mail : p at rdus.de                                 Dr. Gunnar Wrobel
Tel.   : +49 700 6245 0000                         Bundesstrasse 29
Fax    : +49 721 1513 52322                        D-20146 Hamburg
--------------------------------------------------------------------
    >> Mail at ease - Rent a kolab groupware server at p at rdus <<
--------------------------------------------------------------------

Links:
------
[1] http://www.pardus.de
[2] http://gunnarwrobel.de

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: Digitale PGP-Unterschrift
URL: <http://lists.kolab.org/pipermail/devel/attachments/20091123/ae94cc2b/attachment.sig>