with parallelization python module "multiprocessing"

In this article, we will see explore how we can generate random arrays of any shape using the method numpy.random.randint() in two phases: one running a single process and a second one with multiprocessing module. We aim here to construct a 2D numpy array of random int.

Random number does NOT mean a different number every time. Random means something that can not be predicted logically

Random arrays can be used for multiple purposes such as algorithm construction, investment strategy backtesting, statistical calculation and many other applications.

1. Generate random arrays with a single process

We want a to impose two conditions for the generated random numbers where:

• A minimum and maximum range for each number in the final output.
• The sum of the elements of one row of the final array should be equal to 1.

We define first the minimum and maximum ranges corresponding to the first condition:

ranges=[[0,99],[0,99],[0,99],[0,99],[0,99]]
m=10

The final array should be of shape m x len(ranges)

We initiate the arrays we want to fill later with random numbers:

df_ranges=pd.DataFrame(ranges,columns=['range low','range high']).transpose()
arr_ranges_low=df_ranges.values[0]
arr_ranges_high=df_ranges.values[1]
#
l=len(ranges)
rands=np.zeros(l)
weights=np.zeros((m,l))

For each element, we call numpy.random.randint() between the minimum and maximum ranges.  A while condition will be used in order to satisfy the second condition.

for k in range(m):
    while rands.sum()!=1000:
        rands=np.array([np.random.randint(
            arr_ranges_low[i],arr_ranges_high[i]
            ) for i in range(l)])
        if rands.sum()==100:
            weights[k]=rands
            break

We look for random elements satisfying the first conditions until the second condition is met and then we exit.

We group the previous code into one function rand_weights(ranges,m) and get a random output:

rand_weights can be executed for a very large number of iterations such as 100 000 or more, but the approach can be very consuming regarding memory usage and time optimization. Hence, we can execute it on multiple process using the package multiprocessing.

2. Generate random arrays using multiprocessing

Multiprocessing is package that supports spawning processes using an API similar to the threading module. The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads.

for more details, please visit:

https://www.journaldev.com/15631/python-multiprocessing-example

A prime example of this is the pool object which gives a convenient means of parallelizing the execution of a function across multiple input values, distributing the input data across processes (data parallelism). The following example demonstrates the common practice of defining such functions in a module so that child processes can successfully import that module.

Since our function rand_weights takes multiple arguments, we will use Multiprocessing method pool.starmap() which accepts a sequence of argument tuples, it then automatically unpacks the arguments from each tuple and passes them to our function.

We can build a function MP_a_func that use pool.starmap() method for any function where:

def MP_a_func(func,iterable,proc,chunk):
    pool=multiprocessing.Pool(processes=proc)
    Result=pool.starmap_async(func,iterable,chunksize=chunk)
    pool.close()
    return Result

We can test our function on a number of iterations n_iter=1000:

iterable=[[ranges,m] for n in range(n_iter)]
if __name__ == '__main__':
    results=MP_a_func(rand_weights,iterable,proc,chunk)

MP_a_func returns a list containing 1000 random numpy arrays generated using multiprocessing module in python.

We hope this interactive article from yacodata at www.yacodata.com helped you understand the basics of generating random numbers, so you can use for you proper applications.

If you liked this article and this blog, tell us about it our comment section. We would love to hear your feedback!