Compare 2 pipelines performance
Budget: $30 – $250 USD
Introduction :
The goal is to create two pipelines written in Python with two different sets of tools and arrange step-by-step runtime performance comparison . The development should be performed in Colab's Jupyter notebook.
Procedure :
1. Create input data by simulation with Id + 10 continuous features and 1 million rows
2. Data report definition : is a pandas DataFrame contains a row per column in input data and columns for following statistics :
a. Number of rows with non-empty values
b. Number of rows with empty values
c. Number of distinct values
d. Top 2 frequent values as list , i.e. [Bin 3 , Bin 0]
e. Top 2 frequent counts (for values in d above) as list , i.e [80 , 70]
3. Discretization:
a. Use sklearn.preprocessing.KBinsDiscretizer - https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.KBinsDiscretizer.html
b. Initialize : est = KBinsDiscretizer(n_bins=3, encode='ordinal', strategy='uniform')
c. Given column for transformation the output should be of format "Bin XXX" Where XXX the number of bin returned by KBinsDiscretizer's fit_transform
4. Pipeline 1 - pandas package for data report + sequential discretization
a. Apply sequentially , with for loop , one column after another
b. Prepare Data report as described above using pandas
5. Pipeline 2 - Spark's pandas for data report + parallelized (with Pool.map) discretization
a. Apply in parallel with Pool, where number of columns to be processed in parallel is a parameter
b. Prepare Data report as described above using Spark's pandas
c. Pandas API for Spark : https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/index.html
Remarks :
1. Present clearly the packages were installed
2. Present clearly times and memory usages for 4.a , 4.b , 5.a, 5.b - Pipeline 2 should show lower runtime depending on number of cores with similar memory usage
3. KBinsDiscretizer was chosen as an arbitrary transformer. Most appropriate solution is to develop a function that transform single column and then run it sequentially (Flow 1) or in parallel (Flow 2).
4. Any other method to speed up calculation in Flow 2 comparing to Flow 1 also possible until the runtime gain is demonstrated correctly.
Milestones for review and payment:
1. Screenshots of Pipeline 1 - 20$
2. Screenshots of Pipeline 2 - 40$
3. Q&A on final code - 20$
4. Possible bonus for elegant code and ideas - 20$
The goal is to create two pipelines written in Python with two different sets of tools and arrange step-by-step runtime performance comparison . The development should be performed in Colab's Jupyter notebook.
Procedure :
1. Create input data by simulation with Id + 10 continuous features and 1 million rows
2. Data report definition : is a pandas DataFrame contains a row per column in input data and columns for following statistics :
a. Number of rows with non-empty values
b. Number of rows with empty values
c. Number of distinct values
d. Top 2 frequent values as list , i.e. [Bin 3 , Bin 0]
e. Top 2 frequent counts (for values in d above) as list , i.e [80 , 70]
3. Discretization:
a. Use sklearn.preprocessing.KBinsDiscretizer - https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.KBinsDiscretizer.html
b. Initialize : est = KBinsDiscretizer(n_bins=3, encode='ordinal', strategy='uniform')
c. Given column for transformation the output should be of format "Bin XXX" Where XXX the number of bin returned by KBinsDiscretizer's fit_transform
4. Pipeline 1 - pandas package for data report + sequential discretization
a. Apply sequentially , with for loop , one column after another
b. Prepare Data report as described above using pandas
5. Pipeline 2 - Spark's pandas for data report + parallelized (with Pool.map) discretization
a. Apply in parallel with Pool, where number of columns to be processed in parallel is a parameter
b. Prepare Data report as described above using Spark's pandas
c. Pandas API for Spark : https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/index.html
Remarks :
1. Present clearly the packages were installed
2. Present clearly times and memory usages for 4.a , 4.b , 5.a, 5.b - Pipeline 2 should show lower runtime depending on number of cores with similar memory usage
3. KBinsDiscretizer was chosen as an arbitrary transformer. Most appropriate solution is to develop a function that transform single column and then run it sequentially (Flow 1) or in parallel (Flow 2).
4. Any other method to speed up calculation in Flow 2 comparing to Flow 1 also possible until the runtime gain is demonstrated correctly.
Milestones for review and payment:
1. Screenshots of Pipeline 1 - 20$
2. Screenshots of Pipeline 2 - 40$
3. Q&A on final code - 20$
4. Possible bonus for elegant code and ideas - 20$