Tracing exceptions in multiprocessing in Python

I had problems with debugging my programme using multiprocessing.Pool.
[python]
Traceback (most recent call last):
File "src/homologies2mysql_multi.py", line 294, in <module>
main()
File "src/homologies2mysql_multi.py", line 289, in main
o.noupload, o.verbose)
File "src/homologies2mysql_multi.py", line 242, in homologies2mysql
for i, data in enumerate(p.imap_unordered(worker, pairs), 1):
File "/usr/lib64/python2.6/multiprocessing/pool.py", line 520, in next
raise value
ValueError: need more than 1 value to unpack
[/python]

I could run it without multiprocessing, but then I’d have to wait some days for the program to reach the point where it crashes.
Luckily, Python is equipped with traceback, that allows handy tracing of exceptions.
Then, you can add a decorator to problematic function, that will report nice error message:
[python]
import traceback, functools, multiprocessing

def trace_unhandled_exceptions(func):
@functools.wraps(func)
def wrapped_func(*args, **kwargs):
try:
return func(*args, **kwargs)
except:
print ‘Exception in ‘+func.__name__
traceback.print_exc()
return wrapped_func

@trace_unhandled_exceptions
def go():
print(1)
raise Exception()
print(2)

p = multiprocessing.Pool(1)

p.apply_async(go)
p.close()
p.join()
[/python]

The error message will look like:
[python]
1
Exception in go
Traceback (most recent call last):
File "<stdin>", line 5, in wrapped_func
File "<stdin>", line 4, in go
Exception
[/python]

Solution found on StackOverflow.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s