[Kolab-devel] pykolab/format

Thu Mar 1 18:13:55 CET 2012

On Thursday 01 March 2012 16.22:42 Jeroen van Meeuwen wrote:
> On 2012-03-01 15:50, Christian Mollekopf wrote:
> > On Friday 24 February 2012 12.12:31 Jeroen van Meeuwen wrote:
> >> On 2012-02-24 10:58, Jeroen van Meeuwen (Kolab Systems) wrote:
> >> > On 2012-02-24 8:55, Christian Mollekopf wrote:
> >> >> pykolab/format/tests/test-thread_safety.py |   68
> >> >> +++++++++--------------------
> >> >> 
> >> >>  1 file changed, 22 insertions(+), 46 deletions(-)
> >> >> 
> >> >> New commits:
> >> >> commit cb63be089fce18dd9899526455022dad10f8e2ac
> >> >> Author: Christian Mollekopf <mollekopf at kolabsys.com>
> >> >> Date:   Fri Feb 24 09:54:15 2012 +0100
> >> >> 
> >> >>     Actually call the function from different threads instead of
> >> 
> >> the
> >> 
> >> >> main thread.
> >> > 
> >> > Hi Christian,
> >> > 
> >> > I'm reverting this change, renaming the resulting test (the
> >> > original),
> >> > and applying your version of the test as the test for thread
> >> 
> >> safety.
> >> 
> >> Let me rephrase, because maybe I'm not going to do any of that...
> >> Not
> >> Sure Yet(TM)
> >> 
> >> If I understand correctly, getSerializedUID() is called from within
> >> the
> >> child thread, even though the reason for it to be called is coming
> >> from
> >> the main thread's <thread>.uid() call.  It seems though the most
> >> upper-level thread causing the function to be called is the thread
> >> under
> >> which it runs - and thus getSerializedUID() returns that thread
> >> local
> >> value?
> > 
> > Hey Jeroen,
> > 
> > In test-thread_safety.py  there shouldn't be any call to
> > getSerializedUID()
> > from the main thread from what I see.
> 
> Yes, my statement confirms that for the *current* version, I was
> speaking about what (apparently) happened in the old version, before
> your latest commit that fixed the test.
> 

Ah, ok. In the previous version you just ran two empty threads (because you 
didn't implement run() => empty run() base implementation). And then you 
called the getUid() functions of the thread object  (if i recall correctly) 
from the main thread, meaning everything was running in the same thread in the 
end.

The misunderstanding is, I think, that the thread object somehow represents 
the thread border and that everything inside the thread object runs inside the 
thread. That is not the case.

Instead the Thread object is an ordinary object, and calls of methods are 
executed in the calling thread (sequentially). The run() (respectively the 
target runnable object), represents the thread instead (think of it as the 
threads main() function), and everything which is called from the run() method 
runs in the new thread.

That's also why you need synchronization if you access a value from inside the 
run() method of  a thread and from the main thread, because those two calls 
could happen at the same time.

> >> Now, I've attached a parallel_threads test, which succeeds, but does
> >> a
> >> full mesh of comparisons. Note I do not have to time.sleep() - but I
> >> added some tests that sleep anyway.
> >> 
> >> Could you please examine the output (including what I am assuming
> >> are
> >> the thread identifiers spit out to stdout/stderr by libkolabxml),
> >> and
> >> confirm/deny whether or not these tests are appropriate / do what we
> >> expect them to do?
> > 
> > Yes, the debug output (which you have to activate in utils.cpp,
> > ThreadLocal
> > class), are the thread-identifiers.
> > 
> > It seems to work sometimes, but sometimes also not. I suspect that
> > threads are being reused because you run the threads sequentially
> > instead of in parallel
> > (you sleep first).
> 
> Well... I've established an error in my reasoning so you're helping me.
> 
> Here's what I'm thinking I'm doing (with this new attachment); poke a
> hole in this reasoning if you will, please;
> 
> 1) I create two threads, that do not yet start.
Correct.
> 1a) As part of their __init__ though, they each create an in-memory
> representation of a freshly generated Event object.
> 
>      I don't really care about these objects, but I do care about when
> the get_uids() function is being called, whether that happens
> synchronously or asynchronously, and whether or not the main thread
> continues running (creating and running that other thread) while the
> individual threads are going through their motions.
> 

init is executed sequentially (in the main thread), run(), respectively the 
"target" runnable are the startpoints of the new thread. Therefore get_uids() 
gets called from within the new thread.

> 2) I start thread #1,
> 2a) meanwhile the main thread continues,
> 2b) thread #1 is serializing the event and getting its hands on a uid,
> 2c) here it should sleep() for longer than it takes thread #2 to be
> started and serialize it's event for the first time,
> 
>     This wasn't the case in the version of my ramblings you were looking
> at before, BTW. I think I've fixed it now though.
Yes, you just need to start both threads immediately without sleeping between 
the two calls to start. One or the other will be first and the other will run 
very likely while the other is sleeping. (There is no hard guarantee for that 
without explicit locks, but it should be a reasonable assumption).
> 
> 3) the result of get_uids() is two variables (self.uid1, self.uid2,
> supposed to be specific to the thread itself) containing a <uid/>, which
> we;
> 

Threads share the memory space (in most implementations), so it's just the 
variable of the object. Only processes have separate memory space, which is 
why you can communicate between threads so easily (wouldn't be possible that 
way using processes).

> 4) after joining back the threads in to the main thread,
> 
> 5) store and compare against one another using the full mesh of tests.
> 
> Furthermore I'm trying to push the timings to their extremes (by
> sleeping a little, and then later on, not at all).
> 

To avoid sleeping, and doing exactly what you want: Running the thread#1 first 
call to getSerializedUID() after the thread#2 finishied the call, repectively:

#1 writeEvent()
#1 getSerializedUID()
#2 writeEvent()
#2 getSerializedUID()
#1 getSerializedUID()
#2 getSerializedUID()

....you should use locking using Lock objects. This way you can synchronize 
thread#1 and #2 without sleeping, and being absolutely sure how the threads 
sequentially interact.

> I think especially the mixture of tests succeeding as well as the debug
> output shown, are particularly peculiar, as sometimes without sleeping
> it seems like libkolabxml is indicating it is using the same(?)
> thread-local(?) storage?
> 

Not in my test, because there the threads run in parallel and take some time 
(due to the for loop). In your test that can happen because it's possible that 
one thread finishes before the other even started, so python probably reuses 
the thread...

> > It's also not quite clear what you want to test with this test,
> 
> I've grown to become suspicious of things I do not completely
> understand, if you will ;-)
> 
> To phrase it differently, my gut feeling says something is wrong, or
> not quite right yet, or any variety of either - my gut feeling doesn't
> speak Dutch or English, it's hard to understand the insensible grunts.
> 

Sure, I don't think the problem lies in libkolabxml though ;-)

Cheers,
Christian

> Kind regards,
> 
> Jeroen van Meeuwen
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.kolab.org/pipermail/devel/attachments/20120301/9bb60109/attachment.sig>