Two Existing Public Datasets Combined in One
Budget: $10 – $30 USD
Combining 2 existing datasets, using a combination of programming tools and data manipulation techniques. Python with libraries like Pandas are well-suited for this task.
You'll use pd.read_csv() to load both datasets (in CSV format) into separate Pandas DataFrames. But Apache Spark can be more efficient for large data. You can use SparkSession to read the CSV files into Spark DataFrames.
After this step, use functions like df.head(), df.info(), df.describe() (Pandas) or df.show(), df.printSchema() (Spark) to inspect the structure and data types of each DataFrame.
Identify and handle missing values, inconsistencies, or incorrect data types for data cleaning: You might need to convert data types (e.g., from string to numeric), handle outliers, and potentially rename columns to ensure consistency between the datasets.
You'll likely need to transform the data to match the schemas. This might involve:
Selecting specific columns from each dataset, renaming columns to have the same names in both DataFrames, creating new columns based on existing ones (e.g., calculating discharge rates, or normalizing values) and filtering or aggregating data based on certain conditions.
Use df.concat() to combine the DataFrames vertically (if they have the same columns, for example, stacking the data from multiple files into a single DataFrame) or df.merge() for combining horizontally (if they have common columns). You can use df.unionByName() to combine DataFrames with the same schema or df.join() to combine DataFrames with common columns.
Data Storage:
Consider using other formats like JSON, which are more efficient for large datasets.
Tools Summarized:
Programming Languages: Python (with Pandas)
Data Manipulation Libraries: Pandas (Python)
Data Storage: JSON, JSONL, CVS. .
Provide a clear, concise and precise explanation of your approach, showing step by step how you went from point A to point B, and so on. Finally, please also provide me with all the work (Python code) and the new dataset as a separated file (CVS, JSON).
To save the dataset back to CVS, use df.to_csv() (Pandas) or df.write.csv().
Budget: $30-$50
The datasets will be provided
You'll use pd.read_csv() to load both datasets (in CSV format) into separate Pandas DataFrames. But Apache Spark can be more efficient for large data. You can use SparkSession to read the CSV files into Spark DataFrames.
After this step, use functions like df.head(), df.info(), df.describe() (Pandas) or df.show(), df.printSchema() (Spark) to inspect the structure and data types of each DataFrame.
Identify and handle missing values, inconsistencies, or incorrect data types for data cleaning: You might need to convert data types (e.g., from string to numeric), handle outliers, and potentially rename columns to ensure consistency between the datasets.
You'll likely need to transform the data to match the schemas. This might involve:
Selecting specific columns from each dataset, renaming columns to have the same names in both DataFrames, creating new columns based on existing ones (e.g., calculating discharge rates, or normalizing values) and filtering or aggregating data based on certain conditions.
Use df.concat() to combine the DataFrames vertically (if they have the same columns, for example, stacking the data from multiple files into a single DataFrame) or df.merge() for combining horizontally (if they have common columns). You can use df.unionByName() to combine DataFrames with the same schema or df.join() to combine DataFrames with common columns.
Data Storage:
Consider using other formats like JSON, which are more efficient for large datasets.
Tools Summarized:
Programming Languages: Python (with Pandas)
Data Manipulation Libraries: Pandas (Python)
Data Storage: JSON, JSONL, CVS. .
Provide a clear, concise and precise explanation of your approach, showing step by step how you went from point A to point B, and so on. Finally, please also provide me with all the work (Python code) and the new dataset as a separated file (CVS, JSON).
To save the dataset back to CVS, use df.to_csv() (Pandas) or df.write.csv().
Budget: $30-$50
The datasets will be provided