Kolab 16: Problems after Update

[Bartosiak-Jentys, Chris]

Hi Jeroen,

Thank you for the detailed response, its really interesting to learn 
more about Kolab's architecture and the work that goes into developing 
it. I just want to say its very much appreciated, thank you to you and 
all the other contributors.

I had no idea that Kolab cached expensive LDAP searches in SQL, thanks 
for taking the time to explain this. I admit I had to read over your 
response a few times to understand it (and still don't fully). Hopefully 
a better understanding will enable myself and others to submit better 
bug reports with more relevant logs in future as well as helping others 
on the list.

Thank you for your time.

Kind regards,

Chris Bartosiak-Jentys


On 2016-07-07 17:18, Jeroen van Meeuwen wrote:
> On Thu, 2016-07-07 at 14:56 +0100, Bartosiak-Jentys, Chris wrote:
>> Thanks Jeroen,
>> 
>> Dropping the policy_results table seems to have fixed the problem on 
>> both servers, thank you!
>> Seems its automatically regenerated on the next email I send.
>> 
>> Any idea what could have caused this issue? I did dump the kolab 
>> database before dropping the table as a precaution but the dump just 
>> contains the table schema, no data was in policy_result.
>> 
> 
> Well, yeah I know what caused the issue, in part because I wrote this
> stuff and in part because I'm responsible for you and others
> experiencing the issue in the first place;
> 
>   - An issue in caching the policy results of users delegated the
> authorization to send 'on behalf of' necessitated the introduction of
> additional information, as opposed to just 'envelope sender', 'sasl
> username', 'recipient' with a 'yay or nay' boolean -- this is the
> policy_result.data column.
> 
>   - Adding a column in the ORM (SQLAlchemy) is easy, but offering an
> automated path to migrate existing installations with a schema update
> is very expensive (in time and effort and otherwise).
> 
> In our eagerness to make sure Kolab 16 functions exactly the way we
> think it should until at least we do Kolab 17, I (personally) had
> (mistakenly) assumed this particular fix was already in Kolab 16,
> meaning no existing installation of Kolab 16 would require a schema
> upgrade. All-in-all, it's therefore my mistake.
> 
> Now, as to the reason that this is cached to begin with, it is
> primarily for larger hierarchies with (typically) more delegation
> happening and more frequently used distribution groups, where the
> number of searches against LDAP needed to establish the authorization
> for one single authenticated user to use the envelope sender address
> used in the message being submitted, taking in to account the
> recipient(s) might have policies in place prohibiting the (envelope)
> sender from sending them email, is ... well ... costly. The length of
> that sentence is a testimony to that fact.
> 
> Other than that single aspect of costly searches with potentially many,
> many individual recipients for a single message, this cache has no
> value whatsoever. It can be discarded at any time, and recreated as an
> environment returns to regular usage patterns (delays on initial
> submissions implied, though).
> 
> I should note that there's a means to 'drop, lather, rinse and repeat'
> in error recovery -- we've implemented such for the 'authentication
> cache' already -- though rest assured this doesn't actually cache
> credentials, it just caches search results (domain 'example.org' being
> 'dc=example,dc=org', and 'john.doe at example.org' being
> 'uid=doe,ou=People,dc=example,dc=org', if you will, with the password
> check still being an actual bind operation against LDAP).
> 
> I suppose we could implement something similar for the SMTP Access
> Policy cache (the policy_result table), which as you mentioned is
> restored 'automagically' (thanks to the ORM, actually).
> 
> Kind regards,
> 
> Jeroen van Meeuwen
> 
> --
> Senior Solutions Architect, Kolab Systems AG
> 
> e: vanmeeuwen at kolabsys.com
> m: +41 79 951 9003
> w:
> https://kolabsystems.com
> 
> pgp: 9342 BF08