Created
April 10, 2020 14:40
-
-
Save RaulMedeiros/aeba02c20ea2b3f4b8be96eeabdec52e to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Example 1: List of lists | |
A list of multiple arguments can be passed to a function via pool.map | |
(function needs to accept a list as single argument) | |
Example: calculate the product of each data pair | |
import multiprocessing | |
import numpy as np | |
data_pairs = [ [3,5], [4,3], [7,3], [1,6] ] | |
# define what to do with each data pair ( p=[3,5] ), example: calculate product | |
def myfunc(p): | |
product_of_list = np.prod(p) | |
return product_of_list | |
if __name__ == '__main__': | |
pool = multiprocessing.Pool(processes=4) | |
result_list = pool.map(myfunc, data_pairs) | |
print(result_list) | |
[15, 12, 21, 6] | |
Example 2: using partial() | |
Parallel run of a function with multiple arguments | |
To use pool.map for functions with multiple arguments, partial can be used to set constant values to all arguments which are not changed during parallel processing, such that only the first argument remains for iterating. (The variable input needs to be always the first argument of a function, not second or later arguments). | |
Example: multiply all numbers in a list by 10 | |
import multiprocessing | |
from functools import partial | |
data_list = [1, 2, 3, 4] | |
def prod_xy(x,y): | |
return x * y | |
def parallel_runs(data_list): | |
pool = multiprocessing.Pool(processes=4) | |
prod_x=partial(prod_xy, y=10) # prod_x has only one argument x (y is fixed to 10) | |
result_list = pool.map(prod_x, data_list) | |
print(result_list) | |
if __name__ == '__main__': | |
parallel_runs(data_list) | |
[10, 20, 30, 40] | |
Partial creates a new simplified version of a function with part of the arguments fixed to specific values. | |
https://docs.python.org/3/library/multiprocessing.html | |
https://docs.python.org/3/library/functools.html#functools.partial |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment