Created
November 5, 2019 14:08
-
-
Save psinger/6e5f11981588378bc9316397131be66a to your computer and use it in GitHub Desktop.
Pandas groupby apply multiprocessing #python #pandas
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from joblib import Parallel, delayed | |
import multiprocessing | |
import pandas as pd | |
import time | |
def applyParallel(dfGrouped, func): | |
retLst = Parallel(n_jobs=multiprocessing.cpu_count())(delayed(func)(group) for name, group in dfGrouped) | |
return pd.concat(retLst) | |
def myfunc(df) | |
return df | |
start_time = time.time() | |
res = applyParallel(df.groupby(['id']), myfunc) | |
print(time.time() - start_time) |
if we want to parse som "args" to myfunc - how would we do that?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Thanks!