PySpark program to generate large dataset with 100 000 columns and 50 million rows
Budget: $10 – $30 USD
I need a PySpark program to generate a large dataset with 100 000 columns and 50 million rows. I should be able to set the number of dimension columns (i.e. columns with non-numeric values such as Country, State, Suburb, Product etc). The rest of the columns must be all numerically populated with random floating-point numbers. The program's output needs to save the data to a single parquet or CSV file.
I need to be able to set up the dimension values with CSV tables in the format below.
Dimension name: Country
File name: Country.csv
File contents:
1,United States
2,United Arab Emerates
3,Saudi Arabia
Random numbers must be picked from the file above to populate the dimensions.
I need to be able to set up the dimension values with CSV tables in the format below.
Dimension name: Country
File name: Country.csv
File contents:
1,United States
2,United Arab Emerates
3,Saudi Arabia
Random numbers must be picked from the file above to populate the dimensions.