Steam Review Analysis Platform
Budget: £15 – £20 GBP
I am looking for a web developer to create an application that scrapes Steam game reviews, analyses sentiment and themes, and stores and displays this data by period.
1. Introduction
1.1. Project Overview
The Steam Review Analysis Platform is a web-based application designed to scrape user reviews from selected games on the Steam platform. The application will process these reviews weekly, storing historical data in a scalable SQL database. Users will be able to view review data, sentiment analysis, and visualisations through a web interface. Admins will manage user access and the list of games to be tracked.
1.2. Objectives
- Provide a platform for analysing Steam game reviews.
- Enable scheduled weekly scraping and processing of review data.
- Store and display historical data for weeks, months and user-selected time periods.
- Integrate with Microsoft Power BI for advanced data visualisation.
- Ensure data security and compliance with ISO 27001 standards.
2. System Requirements
2.1. User Management
- Roles:
- Admin: Manage user accounts, add/remove games.
- Standard User: View data and visualisations.
- Admin Capabilities:
- Add, edit, and delete user accounts.
- Add and remove games from the tracking list.
- Standard User Capabilities:
- Access the platform to view data.
- Authentication:
- Support for SSO with Microsoft 365 logins.
2.2. Steam Data Scraping
- Frequency: Weekly data scraping from Steam.
- Technology: No specific preference; use industry-standard frameworks for web scraping.
- Scope: Ability to add and remove games from the tracking list via the admin interface.
- Compliance: Ensure scraping complies with Steam’s terms of service.
2.3. Historical Data Storage
- Database: SQL database.
- Retention: No limit on historical data retention.
- Scalability: Database should be scalable to accommodate growing data over time.
- Backup: Weekly database backup schedule.
3. User Interface
3.1. General Requirements
- Responsiveness: Interface must be mobile-optimised and responsive.
- Design: No specific design requirements for the initial release; focus on functionality.
- Framework: No preference; choose a modern, reliable framework suitable for web applications.
3.2. Data Visualisation and Interaction
- Word Cloud:
- Interactive, displaying related reviews upon clicking a word.
- Filterable by overall, positive, negative.
- No specific library preference.
- Review Summaries:
- Display top 5 positive and top 5 negative points, ranked.
- Timeline View:
- Graphical representation of % positive, % negative, and NPS scores over time.
- Handle missing data by re-scraping and reprocessing.
4. Sentiment Analysis and AI Integration
4.1. Sentiment Analysis
- API: Use standard ChatGPT API for sentiment analysis.
- Processing Frequency: Perform sentiment analysis once per weekly data scrape, then save results in the database.
- Sentiment Metrics:
- Identify and count reviews with positive, neutral and negative sentiments.
- Calculate NPS as the difference between % positive and % negative reviews.
4.2. AI-Generated Summaries
- Content: Summarise and rank the top 5 positive and top 5 negative points from the reviews.
5. Integration with Power BI
5.1. Data Push
- Frequency: Scheduled weekly push to Power BI following data scraping and processing.
- Method: Use the Power BI REST API for integration.
- Data Scope: Include all relevant processed data, including sentiment scores, review summaries, and visualisations.
6. Security and Compliance
6.1. Security Measures
- Compliance: Adhere to ISO 27001 standards.
- Encryption: Use HTTPS for all data transfers.
- Authentication: SSO with Microsoft 365.
7. Deployment and Hosting
7.1. Hosting
- Environment: Deploy on Microsoft Azure.
- Scalability: No high availability or load balancing required for the initial release.
- Backup: Weekly database backups.
8. Project Timeline and Milestones
8.1. Timeline
- Total Duration: 1 month from project start to initial release.
8.2. Milestones
Week 1:
- Finalise system architecture.
- Set up the database and hosting environment.
- Begin development of user management and authentication.
Week 2:
- Implement data scraping and database storage.
- Develop the web interface for data visualisation.
- Begin integration with ChatGPT for sentiment analysis.
Week 3:
- Finalise sentiment analysis and AI-generated summaries.
- Complete Power BI integration.
- Conduct internal testing.
Week 4:
- Final bug fixes and optimisations.
- User acceptance testing (UAT).
- Prepare for deployment and release.
9. Deliverables
- Functional Web Application: With user management, data scraping, sentiment analysis, and visualisation features.
- Documentation: Include technical documentation.
- Deployment on Azure
1. Introduction
1.1. Project Overview
The Steam Review Analysis Platform is a web-based application designed to scrape user reviews from selected games on the Steam platform. The application will process these reviews weekly, storing historical data in a scalable SQL database. Users will be able to view review data, sentiment analysis, and visualisations through a web interface. Admins will manage user access and the list of games to be tracked.
1.2. Objectives
- Provide a platform for analysing Steam game reviews.
- Enable scheduled weekly scraping and processing of review data.
- Store and display historical data for weeks, months and user-selected time periods.
- Integrate with Microsoft Power BI for advanced data visualisation.
- Ensure data security and compliance with ISO 27001 standards.
2. System Requirements
2.1. User Management
- Roles:
- Admin: Manage user accounts, add/remove games.
- Standard User: View data and visualisations.
- Admin Capabilities:
- Add, edit, and delete user accounts.
- Add and remove games from the tracking list.
- Standard User Capabilities:
- Access the platform to view data.
- Authentication:
- Support for SSO with Microsoft 365 logins.
2.2. Steam Data Scraping
- Frequency: Weekly data scraping from Steam.
- Technology: No specific preference; use industry-standard frameworks for web scraping.
- Scope: Ability to add and remove games from the tracking list via the admin interface.
- Compliance: Ensure scraping complies with Steam’s terms of service.
2.3. Historical Data Storage
- Database: SQL database.
- Retention: No limit on historical data retention.
- Scalability: Database should be scalable to accommodate growing data over time.
- Backup: Weekly database backup schedule.
3. User Interface
3.1. General Requirements
- Responsiveness: Interface must be mobile-optimised and responsive.
- Design: No specific design requirements for the initial release; focus on functionality.
- Framework: No preference; choose a modern, reliable framework suitable for web applications.
3.2. Data Visualisation and Interaction
- Word Cloud:
- Interactive, displaying related reviews upon clicking a word.
- Filterable by overall, positive, negative.
- No specific library preference.
- Review Summaries:
- Display top 5 positive and top 5 negative points, ranked.
- Timeline View:
- Graphical representation of % positive, % negative, and NPS scores over time.
- Handle missing data by re-scraping and reprocessing.
4. Sentiment Analysis and AI Integration
4.1. Sentiment Analysis
- API: Use standard ChatGPT API for sentiment analysis.
- Processing Frequency: Perform sentiment analysis once per weekly data scrape, then save results in the database.
- Sentiment Metrics:
- Identify and count reviews with positive, neutral and negative sentiments.
- Calculate NPS as the difference between % positive and % negative reviews.
4.2. AI-Generated Summaries
- Content: Summarise and rank the top 5 positive and top 5 negative points from the reviews.
5. Integration with Power BI
5.1. Data Push
- Frequency: Scheduled weekly push to Power BI following data scraping and processing.
- Method: Use the Power BI REST API for integration.
- Data Scope: Include all relevant processed data, including sentiment scores, review summaries, and visualisations.
6. Security and Compliance
6.1. Security Measures
- Compliance: Adhere to ISO 27001 standards.
- Encryption: Use HTTPS for all data transfers.
- Authentication: SSO with Microsoft 365.
7. Deployment and Hosting
7.1. Hosting
- Environment: Deploy on Microsoft Azure.
- Scalability: No high availability or load balancing required for the initial release.
- Backup: Weekly database backups.
8. Project Timeline and Milestones
8.1. Timeline
- Total Duration: 1 month from project start to initial release.
8.2. Milestones
Week 1:
- Finalise system architecture.
- Set up the database and hosting environment.
- Begin development of user management and authentication.
Week 2:
- Implement data scraping and database storage.
- Develop the web interface for data visualisation.
- Begin integration with ChatGPT for sentiment analysis.
Week 3:
- Finalise sentiment analysis and AI-generated summaries.
- Complete Power BI integration.
- Conduct internal testing.
Week 4:
- Final bug fixes and optimisations.
- User acceptance testing (UAT).
- Prepare for deployment and release.
9. Deliverables
- Functional Web Application: With user management, data scraping, sentiment analysis, and visualisation features.
- Documentation: Include technical documentation.
- Deployment on Azure