PySpark program to generate large dataset with 100 000 columns and 50 million rows

Job ID: 34964959

Budget: $10 – $30 USD

I need a PySpark program to generate a large dataset with 100 000 columns and 50 million rows. I should be able to set the number of dimension columns (i.e. columns with non-numeric values such as Country, State, Suburb, Product etc). The rest of the columns must be all numerically populated with random floating-point numbers. The program's output needs to save the data to a single parquet or CSV file.

I need to be able to set up the dimension values with CSV tables in the format below.

Dimension name: Country
File name: Country.csv
File contents:
1,United States
2,United Arab Emerates
3,Saudi Arabia

Random numbers must be picked from the file above to populate the dimensions.
Related categories: Spark PySpark Apache Spark