Multiprocessing in Python and garbage collection

Working with multiple threads in Python¬†often leads to¬†high RAM consumption. Unfortunately, automatic garbage collection in child processes isn’t working well. But there are two alternatives:

  • When using Pool(), you can specify no. of task after which the child will be restarted resulting in memory release.
p = Pool(processes=4, maxtasksperchild=1000)
  • If you use Process(), you can simply delete unwanted objects call gc.collect() inside the child. Note, this may slow down your child process substantially!