parallel processing - A memory issue with multiprocessing of Python: How to release the memory of process who ended their tasks -
when run multiprocessing several pools, how quit process after tasks , drop memory usage? include close() , join(). saw process still holds memory after jobs. how drop memory?
for example, code plays below
from multiprocessing import pool import pandas pd def sub_test(aux): return sum(aux.a)
at below 4 pools generated , holds several copies of aux.
def test(aux): pool = pool(processes=4) pool.map(sub_test, aux) pool.close() pool.join()
i expect process release memory after close , join, still holds.
#here main function aux = pd.dataframe() aux['a'] = [1,1,1] x in range(10): test(aux)
is there commands handle issue?
problems solved.
pool.close() , pool.join() work properly.
Comments
Post a Comment