Python Performance: A Comparison

Articles From: QuantDare
Website: QuantDare

By:

QuantDare Blog https://quantdare.com/author/jgonzalezomega/

This article was first posted on QuantDare Blog.

When coding in any computer language, performance is always an important feature to take into consideration. But if it comes to Python, this factor becomes crucial. In this post, we will see how the way we develop a function and whether we’re using a library or not can make dramatical changes regarding performance.

Let’s look at two possible implementations of a simple function, which apply different transformations according to the input values:

def some_calcs1(x):

if x > 0.04:
return 0.4
elif x > 0.01:
return 10 * x
elif x > 0:
return 10 * x / 3.0 + 1 / 15.0
else:
return 0

This second function would obtain the same result as the previous one:

def some_calcs2(x):

return (x > 0.04) * 0.4 + \
((x > 0.01) & (x <= 0.04)) * x * 10.0 + \ ((x > 0) & (x <= 0.01)) * (x * 10.0 / 3.0 + 1.0 / 15.0) + \
0.0

Let’s define 3 equivalent variables in different types: a list of lists, a Numpy array and a DataFrame from Pandas:

import pandas as pd
import numpy as np

xarray = np.random.rand(1000,10)
xlist = xarray.tolist
xdf = pd.DataFrame(xarray)

Lists                                                           

If you apply the two functions defined above to a list of lists, the first function is 5 times faster than the second, just as a consequence of the way they are coded. In function 1 only the code inside the fulfilled condition is executed, while in function 2 all the calculations are done for every figure.

%timeit z1 = [list(map(some_calcs1, z)) for z in xlist]
%timeit z2 = [list(map(some_calcs2, z)) for z in xlist]

Numpy arrays

But, what if we worked with Numpy arrays instead of lists? Can we expect the same behaviour?

First of all, in order to “map” the first function in Numpy we would need to vectorize it (we will use a decorator), otherwise it would not work. Vectorize a function allows us to apply the function to the whole array, instead of using a loop.

import numpy as np

@np.vectorize
def some_calcs1_vec(x):
if x > 0.04:
return 0.4
elif x > 0.01:
return 10 * x
elif x > 0:
return 10 * x / 3.0 + 1 / 15.0
else:
return 0

At first sight, we realise that the performance has improved a little with the first function, and tremendously with the second one. But what amazes the most is that now the second function is much faster than the first one! But, were not we saying that the first implementation was faster? Let’s explain what is going on here:

%timeit z1 = some_calcs1_vec(xarray)
%timeit z2 = some_calcs2(xarray)

Numpy has what they call the universal functions (ufunc), which are functions that can receive array like inputs and return array output, but they operate over each element. It is quite the same that we do when vectorizing, but with faster results, since these functions look over the elements by loops in a lower level (C implementations). Besides, these functions broadcast (adjust) the input arrays when they have different dimensions.

Then, the first function is “generalized” to operate like the ufuncs, but the second function does use the ufuncs.

Alright, but I don’t see any ufunc at all! Well, the different operators you see in the formulas, like *, +, &, > are overloaded with the ufuncs multiply(), add(), logical_and() or greater(). Then, for example, x+4.0 would be the same as applying np.add(x, 4.0).

Pandas DataFrame

To conclude, we might wonder if Pandas library would obtain similar performance as Numpy, taking into account that Pandas makes use of Numpy arrays underneath.

If we apply a map operation with the functions defined, the performance would be slower than the one obtained by mapping a list and, obviously, much slower than Numpy:

%timeit z1 = xdf.applymap(some_calcs1)
%timeit z2 = xdf.applymap(some_calcs2) If we apply the functions directly over Pandas DataFrames, the first vectorized function performs slower than with a Numpy array; the second function loses all its potential when applying over a DataFrame, with a performance similar to the one obtained with a list.

%timeit z1 = some_calcs1_vec(xdf)
%timeit z2 = some_calcs2(xdf)

Finally, to complicate matters even further, we could use the “apply ” DataFrame method which applies the function specified to entire rows or columns; as we see below, the choice of the axis to operate on is a factor that makes a big difference in terms of performance.

In general, it is advisable not to use this kind of mapping in Pandas if you want an acceptable performance.

%timeit z1 = xdf.apply(some_calcs1_vec, axis=0)
%timeit z2 = xdf.apply(some_calcs1_vec, axis=1)

So, be warned: the way you implement your code and the choice of the right libraries and functions can make your programs fly or be as slow as molasses in January.

Disclosure: Interactive Brokers

Information posted on IBKR Campus that is provided by third-parties does NOT constitute a recommendation that you should contract for the services of that third party. Third-party participants who contribute to IBKR Campus are independent of Interactive Brokers and Interactive Brokers does not make any representations or warranties concerning the services offered, their past or future performance, or the accuracy of the information provided by the third party. Past performance is no guarantee of future results.

This material is from QuantDare and is being posted with its permission. The views expressed in this material are solely those of the author and/or QuantDare and Interactive Brokers is not endorsing or recommending any investment or trading discussed in the material. This material is not and should not be construed as an offer to buy or sell any security. It should not be construed as research or investment advice or a recommendation to buy, sell or hold any security or commodity. This material does not and is not intended to take into account the particular financial conditions, investment objectives or requirements of individual customers. Before acting on this material, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice.