ZfOS FTP/CVS Organiation (Was: Re: [Kolab-devel] Changes to ftp.kolab.org)

[Thomas Lotterer]

On Wed, Sep 01, 2004, Thomas Lotterer wrote:
> I wrote a document describing very much in detail why
> the structure is exactly designed *that* way. [...]
>
ZFOS FTP/CVS ORGANIZATION
============================
public draft, September 2004

May 2004
- original confidential draft
- review by closed user group

September 2004
- public draft
- typo fixing
- change examples to use not our intended city names

FIXME
- ISO CD/DVD naming/placement/content
- CURRENT/snapshot*
- CVS (tag) and FTP relationship

What is ZfOS
============
"Zen for Open Source" [1] (ZfOS) is a companion to OpenPKG [2] and
related projects, namely "Open Source Software Platform" [3] (OSSP),
ePaperwork [4] and Network SoftWare archive [5] (NetSW). OSSP is
about developing open source software, NetSW is about collecting and
archiving freely available software, OpenPKG is about packaging and
porting software and ePaperwork is about documentation and information
publishing. ZfOS is aimed to fill gaps in the practical application
of these and additional third party projects. It is a cauldron where
ingredients of other projects can be brewed together to create
solutions, experiments and reproducible lab setups that do either not
fit into a single other place or do currently not fit into the origin
concept. ZfOS also provides a platform for additional development where
needed. It should be noted that the ultimate goal of all ZfOS brewing
efforts is to bring back modified and value added components back into
their origin.

[1] http://www.zfos.org/
[2] http://www.openpkg.org/
[3] http://www.ossp.org/
[4] http://www.epaperwork.org/
[5] ftp://ftp.netsw.org/

Scope of this document
======================
This document is the attempt to organize and document a FTP and CVS
structure to achieve the following goals, some of which are incompatible
and require compromises:

- FTP easy navigation for end users
- FTP easy, powerful mirror/rsync capabilities
- FTP useful for low-bandwidth download sites
- FTP structures allowing useful and powerful include/exclude
- FTP support for files, archives and ISOs
- FTP additive changes with existing stuff nailed to stone
- FTP capable of supporting SRC, BIN and UPD equally well
- strong relationship between CVS components and FTP areas
- CVS easy navigation for developers
- CVS relationship to OpenPKG openpkg-src (packaging) module
- CVS support for brewery (poor man's release engineering)
- CVS support for development of own components
- naming useful for sorting, browsing and any kind of automatism
- naming useful for human to human communication
- foolproof and clean naming and numbering
- last but no least: art + beauty + neat => Zen ;-)

Kolab at ZfOS
=============
The Kolab project was the first to develop a solution based on OpenPKG
packages. The majority of packages were vanilla OpenPKG but some
packages required modifications to fulfill the needs of Kroupware, the
name of the commercial contract that brought the Kolab project into
life. Also additional packages were needed. This was all done with
little coordination between the Kolab and OpenPKG teams. ZfOS tries to
merge the Kolab changes back into OpenPKG.

CVS organization (generic)
==========================
Concurrent Version System (CVS) is used to maintain the unique parts of
ZfOS which can be modifications and changes to existing components or
new developments. Available modules are:

zfos-openpkg-src
this is a copy of the OpenPKG "openpkg-src" module. Modifications to
existing packages and new packages are stored here. Currently the
openpkg-src module needs to be imported as a vendor branch but the OSSP
project is working on a "CVSfusion" software which will allow even
better integration. Packages are uploaded to the ZfOS ftp service. [As
of 1-Sep-2004 the "CVSfusion" is still not available so ZfOS should not
wait for it]

zfos-develop
place where ZfOS specific development takes place, i.e. the home of
obmtool.

zfos-brewery
scripts, configs, READMEs and all the stuff required to brew a solution.
Treat this being the "release engineering" part                        .

FTP service (generic)
=====================
The top level folder structure explained:

/openpkg

    this resembles parts of the structure found at
    ftp://ftp.openpkg.org, however it is not a mirror. ZfOS contains
    data which does not exist on the OpenPKG ftp service (any more).
    Examples are binaries for PLUS packages and platforms not officially
    supported by OpenPKG. Also selected CURRENT packages are archived
    for getting brewed when the original vanished from the official
    OpenPKG ftp service.

/zfospkg

    this is the upload area for ZfOS packages whose spec files are
    maintained in the zfos-openpkg-src CVS module. These are additional
    or modified packages different from vanilla OpenPKG. It is the goal
    of ZfOS to merge the changes back into the original as soon as
    possible.

/lootpkg

    3rd party packages where spec files are neither under OpenPKG nor
    ZfOS control.

/brewery

    solutions, experiments and reproducable lab setups assembled
    together by scripts and information from within zfos-brewery CVS
    module.

Secondary folder structure explained:

/brewery/$TOPIC

    everything available for download on $TOPIC.
    Treat topic like "product".

/brewery/$TOPIC/$COLLECTION

    everything needed to setup $COLLECTION.
    Treat collection like "release".

Please note the names are chosen carefully to have an equal number
of characters, making a folder listing look proper. More names are
available which might be used in the future. Here is a brain dump:

    archive
    brewery OK, used
    concept
    contrib
    develop
    edition
    lootpkg OK, used
    openpkg OK, used
    service
    website
    zfospkg OK, used

FTP (kolab)
===========

/openpkg

    contains SRC and UPD packages required for various Kolab related
    collections. These are vanilla OpenPKG packages which can be
    tracked and reproduced in the OpenPKG openpkg-src CVS module. Kolab
    collections using them have them hardlinked.

/zfospkg

    reserved for SRC and UPD packages required for Kolab related
    collections. These are new or modified OpenPKG packages which can
    be tracked and reproduced in the ZfOS zfos-openpkg-src CVS module.
    Kolab collections using them have them hardlinked.

/lootpkg

    reserved for SRC and UPD packages required for Kolab related
    collections. These are new or modified OpenPKG packages which can
    only, if at all, tracked and reproduced by the origin vendor version
    control service.

/brewery/$TOPIC

    everything available for download on "kolab".
    Treat topic being a "product".

/brewery/$TOPIC/$COLLECTION

    everything needed to setup $COLLECTION.
    Treat collection being a "release".

COLLECTIONs

are used to group setup material. There is one highly experimental and
recent collection called "CURRENT". It contains packages from the last
attempt of making a collection work. The completeness and quality is
unpredictable and arbitrary. The other collections have a name following
a strict schema and every collection serves a special flavor which needs
to be documented and understood by every potential consumer. The name
follows a "YYYY-MM.<name>" format where the date resembles the initial
fermentation.

This prefix makes machines and esthete's happy as it
1.) ensures the collection has a unique name
2.) the name leads to sorted order and
3.) this part of the name makes up a proper fixed-width column

For the (caro)user the collection gets a human friendly name appended.
This one is easy to be referenced in human to human communication.
For the time being the names manifest home cities of well known beer
breweries. [Not in this public draft to ensure search engines do not
point people using the real names to this draft later] City names are
1.) changing infrequently
2.) free of patents and trademarks
3.) available in quantities allowing occupation of most if not all letters
4.) alphanumeric sortable if chosen appropriately
5.) somewhat predictable

/brewery/kolab/2003-10.athens/ (formerly kolab-1.0.14-20031126)
/brewery/kolab/2004-02.berlin/ (formerly kolab-20040217-2.0.0)
/brewery/kolab/CURRENT/

The folder structure below every collection discussed below. Note
that the names were chosen carefully to allow meaningful alpha
sort.

bulk/
genesis/
latest/BIN... -> update.001/BIN... || latest -> update.003
update.001/
update.002/
update.003/
update.dev-20040512234512/

"bulk" holds the majority of the files (most, if not all, RPMs) used
in all the other folders. In fact, the RPMs are hardlinks to /openpkg,
/zfospkg and /lootpkg.

"genesis" holds all the files (a majority of them being RPMs) used in
the initial fermentation. The files are actually symlinks to "../bulk/".
This is a static area that never changes. Resolve the symlinks into real
files, burn it to CD/DVD, write documentations for colleagues, peers,
readers etc. that remains valid.

"latest" is a symlink to the latest update within a collection. This is
initially a link to "genesis", later the target moves to the most recent
"update.###". In cases where a latest update is not (yet) available
for all platforms, "latest" might become a real folder with symlinks
redirecting the lower levels individually.

"update.###" contains updated versions of "genesis". Same features
apply.

"update.dev-YYYYMMDDhhmmss" is scratch area, usually (part of) the next
"update" being under construction. Intended for use by development,
release engineering, security services, quality assurance, field
testing etc. The fine granularity allows uploading at frequent rates if
necessary. It also allows parallel developments taking place.

The folder structure below every "bulk", "genesis" and "update.*" folder
discussed below. Note that BIN and SRC terminology is taken from OpenPKG
but the additional OpenPKG folder level for different <arch>-<os> is
replaced by a dot. There is no UPD folder because ZfOS updates exist for
both BIN and SRC and both genesis and update packages share the same
folder.

BIN.ix86-debian3.0.tar
BIN.ix86-debian3.0/
BIN.ix86-freebsd5.2.tar
BIN.ix86-freebsd5.2/
BIN.ix86.iso
BIN.sparc64-solaris8.tar
BIN.sparc64-solaris8/
SRC

Here are the binaries (BIN) for every available <arch>-<os> combination
and the sources (SRC) as well. Unlike OpenPKG, ZfOS/brewery/kolab
does not come with a dedicated update (UPD) area for SRC packages but
updates are integral part of both sources and binaries. The CFG folder
known from ancient ZfOS structures is gone as the core obmtool.conf is
included in the SRC and every BIN folder anyway.

Groups of folders are available for download as tape archive file. Such
archive files are named after the folder they contain with .tar suffix
appended. The archives contain all the folders and files below their
respective folders with symlinks revamped into real files. The .tar
suffix is not used for anything else inside a kolab collection so it can
be used for ftp mirror include/exclude lists.

Selected groups of folders might also be available for download as
CD/DVD ISO images. Such archive files are named to identify their
ingredients with a .iso suffix appended. The images contain folders and
files with symlinks revamped into real files. The .iso suffix is not
used for anything else inside a kolab collection so it can be used for
ftp mirror include/exclude lists.

SCENARIOS
=========

User asks for help thereby reports a bug which needs to be identified,
fixed and leads to a updated collection.

    User:
    I downloaded the ix86-debian3.0 binaries of the 2nd update to athens
    and "bootstrap -b" reports the following error ...

    Developer:
    $ cvs checkout anonymous at cvs.zfos.com:/zfos-brewery-kolab
    $ cvs log README | grep ATHENS
    ... look for the full name of the tag
    $ cvs up -rZFOS_KOLAB_2003_10_ATHENS_UPD_005
    ... hunt down the bug
        this might require checking out OpenPKG openpkg-src, ZfOS
        zfos-openpkg-src (modified/added packages), Erfrakon CVS
        (genuine Kolab/Kroupware packages), ZfOS zfos-develop
        (obmtool) ...
    $ cvs up -rZFOS_KOLAB_2003_10_ATHENS
    ... look if bug exists in the head of the branch
    [ optional, assume new update
    $ ./devtool release
    ]
    ... tell user about availabe update

Contributor wants to write a documentation including a step-by-step
procedure how to download, install, setup and maintain a certain
collection.

    Contributor:
    reference only "genesis" and specific "update.###" files, archives
    (.tar) and images (.iso) in download, install and setup examples.
    Mention "latest" but do not use it as it might change past
    documentation finish.

Administrator wants to setup a ZfOS kolab mirror.

    Administrator:
    $ rsync rsync.zfos.org:/brewery/kolab/2003-10.athens/ \
      --exclude .tar --exclude .iso --copysymlinks #FIXME

    $ rsync rsync.zfos.org:/brewery/kolab/2003-10.athens/ \
      --include latest \
      --exclude .tar --exclude .iso --resolvesymlinks #FIXME

Hacker finds security flaw which needs to be identified, fixed and leads
to a security advisory and updated collection.

    See "User" demo with assumed update. Mention new "3rd athens update"
    and the new included RPMs in security advisory.

--
Thomas.Lotterer at cw.com, Cable & Wireless