Python vs. IronPython - threading behavior vis-a-vis underly…

Top Page

Reply to this message
Author: Andy Sy
Date:  
To: True Computer Science Mailing List, A Philippine Pythonista Hangout
Old-Topics: Re: "Understanding Threading in Python"
Subject: Python vs. IronPython - threading behavior vis-a-vis underlying machine
On 2005-Apr-1, Andy Sy wrote:

> Miguel A Paraz wrote:
>
>> So Python hasn't shifted to using kernel threads, yet? It still uses
>> interpreter-based threads?
>
> ... snip ...
>
> I take that to mean that that all this time Python has been mapping
> its threads to the underlying thread implementation of the environment
> it is running on. In the case of pthreads on an NPTL-enabled glibc,
> then that would presumably mean that Python threads _are_ being 'mapped'
> onto kernel threads.
>
> However, the fact that Python uses the OS' thread implementation (whether
> mapped to kernel threads or otherwise) does not necessarily mean the
> elimination of the GIL. I am guessing that python never used
> "interpreter-based" threads, in other words, thread scheduling has never
> been done by the interpreter. The GIL release and acquire, however, is
> being scheduled by the interpreter and this is what may have led to that
> impression.


Almost exactly 5 years later, on a quadcore machine running Windows 7, I
am able to test the above out. The results are interesting:

CPython - 1 thread spawned
==========================
CPU usage: ~15-20%
Core activity: mainly on one core (with some slight activity
on another, presumably the main Python thread).

CPython - 2 threads
===================
CPU usage: usually above 25%
Core activity: on 3 cores.

CPython - 3 threads
===================
CPU usage: usually above 25%
Core acitivty: on 3 cores.

CPython - 4 threads
===================
CPU usage: usually around 25%
Core activity: around 25% activity on three cores and 10-15% on the fourth core.

IronPython - 1 thread
=====================
CPU usage: 50%
Core activity: 75% activity on 2 cores, 25% on the other 2

IronPython - 2 threads
======================
CPU usage: 75%
Core activity: 100% on 1 core, around 60-75% on 2 cores, and 50% on the fourth

IronPython - 3 threads
======================
CPU usage: 100%
Core activity: 100% on all four cores

IronPython - 4 threads
======================
CPU usage: 100%
Core activity: 100% on all four cores


Comment
=======
So it would seem that python threads _are_ being mapped to
separate OS threads, but the GIL(?) is somehow preventing each
thread from running at full speed.

No matter how many threads you start on CPython, you
really can't utilize more than one core's worth of power even if
the threads do run on multiple cores. Not sure what the exact
explanation for this is, e.g. why the total CPU usage seems
to be the correct fraction for 1 core.

On IronPython, you happily get full cpu utilization just as with
Jython. Interesting how the JVM and CLR based Pythons just don't
seem to face the problems that need the GIL to solve for CPython.
Considering IronPython performance is on par with CPython, there
seems to be very little reason to stick with CPython outside
of being able to use C-specific stuff (and they're addressing
even this with ironclad).

What's puzzling about IronPython though is that you get full
cpu utilization across all four cores even with just three
threads, and the unexpected distribution of load across the
different cores.


The test program looks something like the below. I'm basically
running blackjack simulations on Python and need to run lots and
lots of iterations.

If someone can try on Linux (CPython vs. Fepy) let me know
what result you get!


SINGLE THREAD
=============
def worktodo():
do_some_stuff()
done_threads+=1

done_threads=0
thread.start_new_thread(worktodo)
while done_threads<1:
pass

TWO THREADS
===========
def worktodo():
do_some_stuff()
done_threads+=1

done_threads=0
thread.start_new_thread(worktodo)
thread.start_new_thread(worktodo)
while done_threads<2:
pass

and so on...



------------------------
http://www.neotitans.com
Web and IT Consulting
____________________________________________________
Philippine Pythonista Hangout
python@??? (#Python @ irc.free.net.ph)
http://lists.free.net.ph/mailman/listinfo/python
Searchable Archives: http://archives.free.net.ph