Analytics Dashboard with different data sources
Budget: €250 – €750 EUR
Requirements:
A dashboard for
inputs from different data sources feeding into a central repository, which then connects to analysis tools and results in actionable insights or outputs.
A Portal to access the data lake, including features like search engines, recommendation systems, and analytics dashboards.
Query Tools: query functionalities to fetch specific datasets
Alerts and Strategies: Notification services
1. Gather and connect the datasources
Unstructured Data Sources: mainly text, images, videos, text documents, audio files, and sensor data.
Structured Data Sources any
2. Data Processing:
Extraction: Crawling and scraping cleaning information from various repositories, academic journals, preprint servers
3.Analizes
4. Visualization
Show the data inside a dashboard to get actionable insights
5. Ask questions over the data with an LLM
The system should have monitoring built in showing the health of the datasources and the system.
Needs to be deployable on aws or other cloud.
As an example we would need
Amazon Web Services (AWS)**:
- **Storage**: Amazon S3 for scalable storage.
- **Databases**: Amazon Redshift for data warehousing, Amazon DynamoDB for NoSQL storage.
- **Data Processing**: AWS Glue for ETL jobs.
- **Machine Learning**: Amazon SageMaker for building, training, and deploying machine learning models.
- **Analytics**: AWS Athena for SQL queries on S3 data.
We could also take the open source route
Open Source Way:
Apache Hadoop: A framework that allows for the distributed processing of large data sets across clusters of computers.
Apache Spark: An open-source, distributed processing system used for big data workloads.
Apache Kafka: A distributed streaming platform that can be used for building real-time data pipelines.
Elasticsearch: A distributed search and analytics engine that can be used for real-time data applications.
Apache Airflow: A platform to programmatically author, schedule, and monitor workflows.
TensorFlow: An open-source machine learning framework.
Jupyter: An open-source tool that allows you to create and share documents that contain live code, equations, visualizations, and narrative text.
A dashboard for
inputs from different data sources feeding into a central repository, which then connects to analysis tools and results in actionable insights or outputs.
A Portal to access the data lake, including features like search engines, recommendation systems, and analytics dashboards.
Query Tools: query functionalities to fetch specific datasets
Alerts and Strategies: Notification services
1. Gather and connect the datasources
Unstructured Data Sources: mainly text, images, videos, text documents, audio files, and sensor data.
Structured Data Sources any
2. Data Processing:
Extraction: Crawling and scraping cleaning information from various repositories, academic journals, preprint servers
3.Analizes
4. Visualization
Show the data inside a dashboard to get actionable insights
5. Ask questions over the data with an LLM
The system should have monitoring built in showing the health of the datasources and the system.
Needs to be deployable on aws or other cloud.
As an example we would need
Amazon Web Services (AWS)**:
- **Storage**: Amazon S3 for scalable storage.
- **Databases**: Amazon Redshift for data warehousing, Amazon DynamoDB for NoSQL storage.
- **Data Processing**: AWS Glue for ETL jobs.
- **Machine Learning**: Amazon SageMaker for building, training, and deploying machine learning models.
- **Analytics**: AWS Athena for SQL queries on S3 data.
We could also take the open source route
Open Source Way:
Apache Hadoop: A framework that allows for the distributed processing of large data sets across clusters of computers.
Apache Spark: An open-source, distributed processing system used for big data workloads.
Apache Kafka: A distributed streaming platform that can be used for building real-time data pipelines.
Elasticsearch: A distributed search and analytics engine that can be used for real-time data applications.
Apache Airflow: A platform to programmatically author, schedule, and monitor workflows.
TensorFlow: An open-source machine learning framework.
Jupyter: An open-source tool that allows you to create and share documents that contain live code, equations, visualizations, and narrative text.