Kolab Service Does not Start/No Emails Can be Sent/Received

[Bob Jones]

Hello Kolab Community,

Our Kolab server has gone into some kind of funk, and we can't seem to receive/send any emails. The server is on CentOS 7/Kolab 16 and was recently updated (was healthy before the update). 

The relevant kolab packages updated were:
   Updated: kolab-freebusy-1.1.2-3.63.el7.kolab_16.noarch
   Updated: kolab-syncroton-2.3.19-1.3.el7.kolab_16.noarch
   Updated: roundcubemail-skin-kolab-0.4.1-10.19.el7.kolab_16.noarch
   Updated: roundcubemail-plugin-redundant_attachments-1.4.11.4-1.2.el7.kolab_16.noarch
   Updated: roundcubemail-plugin-newmail_notifier-1.4.11.4-1.2.el7.kolab_16.noarch
   Updated: roundcubemail-plugin-contextmenu-3.0-14.57.el7.kolab_16.noarch
   Updated: guam-0.9.9-2.8.el7.kolab_16.x86_64
   Updated: kolab-webadmin-3.2.15-3.1.el7.kolab_16.noarch
   Updated: wallace-0.8.19-1.8.el7.kolab_16.noarch
   Updated: cyrus-imapd-2.5.15.28-1.22.el7.kolab_16.x86_64
   Updated: kolab-server-0.8.19-1.8.el7.kolab_16.noarch
   Updated: kolab-saslauthd-0.8.19-1.8.el7.kolab_16.noarch
  Updated: postfix-kolab-0.8.19-1.8.el7.kolab_16.noarch


kolabd does not start. Here is what I see from status:

systemctl status kolabd.service

   ● kolabd.service - Kolab Groupware Server.
      Loaded: loaded (/usr/lib/systemd/system/kolabd.service; enabled; vendor preset: disabled)
      Active: failed (Result: exit-code) since Fri 2021-04-30 18:58:33 UTC; 1h 44min ago
     Process: 1944 ExecStart=/usr/sbin/kolabd $FLAGS --pid-file /run/kolabd/kolabd.pid (code=exited, status=0/SUCCESS)
Main PID: 1946 (code=exited, status=2)

   Apr xxx.yyy.zzz systemd[1]: Starting Kolab Groupware Server....
   Apr xxx.yyy.zzz systemd[1]: kolabd.service: Supervising process 1946 which is not our child. We'll most likely not notice when it exits.
   Apr xxx.yyy.zzz systemd[1]: Started Kolab Groupware Server..
   Apr xxx.yyy.zzz systemd[1]: kolabd.service: main process exited, code=exited, status=2/INVALIDARGUMENT
   Apr xxx.yyy.zzz systemd[1]: Unit kolabd.service entered failed state.
   Apr xxx.yyy.zzz systemd[1]: kolabd.service failed.


kolab lm reports

   cyruslib.CYRUSError: (99, 'LIST', 'Connection is not authenticated')


kolab-saslauthd seems to be running but testsaslauthd -u cyrus-admin -p fails with:

   0: NO "authentication failed"

dirsrv seems to be running (ns-slapd). Email clients are failing to log in perhaps because of (from /var/log/maillog):

   xxxxxx imaps[2258]: SASL Password verification failed 
   xxxxxx imaps[2258]: badlogin: [127.0.0.1] unknown AUTHENTICATE+PLAIN+TLS [SASL(-13): authentication failure: Password verification failed]

pykolab seems to be reporting many messages of the form:

   2021-04-XX 23:52:45,266 pykolab.auth ERROR [1289] Invalid DN, username and/or password for 'uid=kolab-service,ou=Special Users,dc=abc,dc=com'.          
   2021-04-XX 23:52:45,277 pykolab.auth ERROR [1289] Invalid DN, username and/or password for 'uid=yyy,ou=People,dc=abc,dc=com'.
   2021-04-XX 23:52:50,130 pykolab.auth ERROR [1289] Invalid DN, username and/or password for 'uid=kolab-service,ou=Special Users,dc=abc,dc=com'.
   2021-04-XX 23:52:50,142 pykolab.auth ERROR [1289] Invalid DN, username and/or password for 'uid=zzzzzz,ou=People,dc=abc,dc=com'.
   2021-04-XX 23:52:54,054 pykolab.auth ERROR [1476] Invalid DN, username and/or password for 'uid=kolab-service,ou=Special Users,dc=abc,dc=com'.
   2021-04-XX 23:52:54,055 pykolab.auth ERROR [1476] Invalid DN, username and/or password for 'uid=kolab-service,ou=Special Users,dc=abc,dc=com'.

What could be going wrong? Any help is appreciated.

Bob