Project Related to Data
Budget: $30 – $250 USD
How to develop a project idea?
1] Pick and Study a Use Case: The first step towards running as stellar data engineering project is choosing a use case. This will give you a clarity about deliverables or the outcomes to evaluate. Read this full-length guide on how to pick your project.
2] Identify Data Storage Platform: What services will you use to store your data (e.g. data warehouse, Bigquery, Cassandra, or other NoSQL platforms)? Will you use any service or pre-processing before moving data to your data warehouse? Determine if you will use Bigquery to both store and analyze large volumes of unstructured data and identify which framework you will use (like Kafka, Spark etc).
3] Data transformation: Ask the following:
· What type of data (e.g. relational, machine data, social, JSON, etc.) are you ingesting?
· Will you ingest real-time streaming data?
· Which sources are you ingesting data from?
· What types of transformations will you use for this use case (e.g. complex file parsing, joins, aggregation, updates)?
4] Visualization and Analytics:
· Are you aiming for visualizations (e.g. bar charts, heat maps, bubble charts,
scatter plots, tree maps)?
· What types of analytic algorithms will you need to run (e.g. basic or advanced statistics, predictive analytics, machine learning)?
· What is the format of the data you will deliver to the visualization and analytics tools?
1] Pick and Study a Use Case: The first step towards running as stellar data engineering project is choosing a use case. This will give you a clarity about deliverables or the outcomes to evaluate. Read this full-length guide on how to pick your project.
2] Identify Data Storage Platform: What services will you use to store your data (e.g. data warehouse, Bigquery, Cassandra, or other NoSQL platforms)? Will you use any service or pre-processing before moving data to your data warehouse? Determine if you will use Bigquery to both store and analyze large volumes of unstructured data and identify which framework you will use (like Kafka, Spark etc).
3] Data transformation: Ask the following:
· What type of data (e.g. relational, machine data, social, JSON, etc.) are you ingesting?
· Will you ingest real-time streaming data?
· Which sources are you ingesting data from?
· What types of transformations will you use for this use case (e.g. complex file parsing, joins, aggregation, updates)?
4] Visualization and Analytics:
· Are you aiming for visualizations (e.g. bar charts, heat maps, bubble charts,
scatter plots, tree maps)?
· What types of analytic algorithms will you need to run (e.g. basic or advanced statistics, predictive analytics, machine learning)?
· What is the format of the data you will deliver to the visualization and analytics tools?