More 3.4 calendar woes

[Homer Dokes]

Hello all,

1st let me say thank you to dsp3 for his response to my conundrum.  I 
appreciate your response tho I must say that it was unclear what to 
'remove'.

2nd let me broadcast to the world that I have found a site that I did 
not know existed nor could I find any reference to it in the past so I 
want to include it here for other 'would be' administrators of the Kolab 
environment.  The web site, http://users.kolab.narkive.com/, looks to 
actually consolidate all the emails associated with 
users at lists.kolab.org in a 'subject' organized manner providing a single 
source of kolab specific issues/solutions with an ability to search 
through them for specific information.  I wished I had found this 6 
months ago.

3rd, I am happy to say I have discovered success for this issue.

Long story short, my discoveries have revealed that roundcubemail 
derives it's information for display in the web client not from the 
mysql tables but rather from the cyrus.cache and cyrus.index files that 
exist under each user imap folder.

i.e. to display user contacts in the web client under the contacts menu 
selection, the data within is drawn from two files located in 
/var/spool/imap/domain/<1st initial of domain name>/<domain name>/<1st 
initial of mail user>/user/<username>/Contacts which are cyrus.cache and 
cyrus.index.

I have not been able to determine what purpose the mysql table 
'kolab_cache_contact' has but I'm confident it doesn't play a part in 
the normal operation emails.  If I'm wrong here someone please 
correct.   I have determined however that the records in 'kolab_folders' 
does play a role to the extent that it keeps track of what folders have 
been created and how many objects reside within.  I am having a separate 
issue where this objectcount is resetting to 0 but that's for another topic.

So the primary symptom here is that contacts and events disappeared for 
one user.  All the contact files and events existed in the associated 
imap user's folder... it wasn't as tho they were physically missing.  
Once I realized that roundcubemail derived it's information from the two 
files, cyrus.cache and cyrus.index, I had to assume they were corrupted 
in this case. These are binary files so there isn't any real way, that I 
have been able to determine, that they were or were not corrupted. Once 
I accepted that they must be I set out to try to determine if and how 
they could be reconstructed.  I began concentrating on cyrus and not the 
'kolab' umbrella.

Turns out there is a command that when executed from the command line in 
this format:

  'reconstruct -rf user.testuser'

If you have  the 'unixhierarchysep' parameter in the /etc/imapd.conf 
folder set to yes than you will need to use this format of the command: 
'reconstruct -rf user/testuser'.

It should also be noted that the format of 'testuser' is the actual user 
name with the domain reference, i.e. testuser at domain.com.

The initiation of this command line will reconstruct the cyrus.cache and 
cyrus.index files.

Now here is the kicker... it is important that you remove these two 
files BEFORE executing that command line as it will not reconstruct them 
if they already exists.  It only does so if the two files are missing.  
It should also be noted that all this assumes that the original (or 
recovered) individual files exist in the respective directories.  So 
this means you can 'choose' which folders, i.e. contacts, calendar, 
tasks, etc, have their cyrus.cache and cyrus.index rebuilt by only 
removing the cyrus.cache and cyrus.index you want rebuilt.  The others 
will be left alone.  This is important if you want to only effect a sub 
folder but leave the parent folders as they are.  The -r parameter in 
the command is recursive.

Running that command line recreates the cyrus.cache and cyrus.index 
files with information taken from the actual content files within the 
respective directory.  Once I had done this the contacts and calendar 
events reappeared in the users roundcubemail client.  Can you say.... 
HaLLelUiAh!!!!

Now... the one unfavorable, I still don't know what caused the original 
corruption of these two files for this one user.  All other users have 
been operating without issue.  This is disconcerting as I cannot insure 
it won't happen again.  It isn't as tho the user had direct access to 
these files.  The server is well protected against power issues and the 
like.

I hope this information is of use for anyone who may find that their 
roundcubemail content has suddenly vanished.

Thank you for indulging me.

hdokes



On 8/18/2015 1:45 PM, dsp3 wrote:
> Hello hdokes,
>
> Having been through a similar scenario, I believe the issue related to
> having a corrupt calendar event/events in a users calendar directory.
> Removing this/these resulted in all calendar items showing again as
> expected.
>
> Sorry I can't be more specific, but I'm 99% certain this is how I
> resolved the issue.
>
> On Tue, 2015-08-18 at 12:11 -0400, Homer Dokes wrote:
>> Hi Timotheus,
>>
>> Thank you so much for your prompt reply.
>>
>> I have enabled debugging from the initial installations (has been
>> running for about 6 months now).  I actually monitor the log files in
>> real time 24/7 for two configured kolab servers via ssh and tail
>> configured in multiple windows through tmux.  This allows us to
>> 'catch'
>> some of the issues as they occur and address them right away.
>> Unfortunately, I have been unable to determine why this particular
>> users
>> contacts and calendar events stopped showing in the roundcubemail web
>> client.  As indicated before, any new entries do show... just not all
>> the entries prior to it's hiccup.
>>
>> I appreciate the sources of information you have provided.  We will
>> spend some time and sift through them to know what is where and how
>> to
>> best utilize it.
>>
>> What I really need right now is to know where the mysql tables
>> 'kolab_cache_contact' and 'kolab_cache_calendar' get their records
>> from.  I had rebuilt them with the command below but that only
>> brought
>> back the newer entries after the hiccup.  The older entries do not
>> show
>> up in these tables even tho the actual 'files' exist in the users
>> respective Calendar and Contacts directories.
>>
>> Command used to rebuild tables
>> /⁠usr/⁠share/⁠roundcubemail/plugins/libkolab/bin/modcache.sh clear
>> -⁠u
>> <user at domain.org> -h localhost
>>
>> Within these directories are binary files that are named cyrus_cache,
>> cyrus.header, and cyrus.index.  For this particular users contact
>> directory, the cyrus.cache file size is 1.6mb, the cyrus index is
>> 155kb,
>> and the cyrus.header are 208 bytes.  I am making the assumption that
>> these files contain reference information to the user files within
>> the
>> directory but I can't be sure as the cache and index files are binary
>> and not reviewable. Judging by the size of the files I have to also
>> believe that they contain information related to all the user contact
>> files within that directory as I doubt the cache file would be 1.6mb
>> in
>> size if it only contained information on 18 contacts.  Do the
>> respective
>> mysql table for contacts derive their content (records) from these
>> cache
>> files?  If so, can these cache files be rebuilt from the user contact
>> files contained within the directory.  If not, how does one include
>> all
>> the contact files into the mysql kolab_cache_contact' table?  The
>> command line above does not include them all... only the newer files
>> that were added after the hiccup.  Should that command have included
>> all
>> the files from 1. to 1458. that exist within the contacts directory
>> for
>> that user?
>>
>> For clarification.... definition of a 'hiccup': The point at which
>> the
>> user realized something was amiss...  not ALWAYS consistent with when
>> the symptom actually manifested itself. :-)
>>
>> I have tried to provide as much information here as I can think is
>> pertinent.  If additional information would help please let me know
>> what
>> you would need to get a better understanding of the issues.
>>
>> Again, Thanks Tim for your reply, and to anyone else that may be able
>> to
>> shed some light on this issue.  In the mean time, We'll keep plugging
>> away for a viable solution.
>>
>> hdokes
>>
>>
>> _______________________________________________
>> users mailing list
>> users at lists.kolab.org
>> https://lists.kolab.org/mailman/listinfo/users