Data loading and processing Application enhancements
Budget: £250 – £750 GBP
Project Type: Web Application
Core requirements:
Load CSV/parquet data, drop/edit columns, apply python APIs, job scheduling, data visualization (plotly).
Problem:
I am looking for a developer to enhance the performance of my data loading and processing application. The current application is designed to load CSV data and invoke Python APIs; however, it encounters difficulties when dealing with substantial data volumes, primarily due to its utilization of browser memory. I am in search of a solution that can efficiently manage extensive datasets while also applying Python APIs for data cleansing. It is not essential to display the entire dataset upfront, and only the initial and final few rows should be visible unless the user requests additional details. Once the user is content with the data, a background task should execute the Python function on the dataset without causing any performance disruptions to the application. The application's frontend should be lightweight and responsive, capable of handling errors gracefully without affecting the ongoing session. Users should experience swift page navigation, and the application should provide informative messages and a progress bar during data processing. I would also need a few additional features in the existing application related to data visualization.
NOTE: The application should be able to run on windows platform, doesn’t matter which O.S you choose for your development. I would expect full deployment instructions and support until its deployed and tested.
Example use case: Load and process CSV/parquet file of 800 MB that contains 585 columns and 2 million rows.
- The primary focus of the enhancements should be on improving the application's performance.
- The application primarily deals with structured data.
- I would also like to add new features and functionality to the application.
- Additionally, I am looking for improvements in the user interface to make it more user-friendly.
- The expected turnaround time for this project is 1-3 weeks.
Skills and Experience:
Tech Stack:
ReactJS (with Next.js), Django Rest Framework, React Query, Next UI Components, Zustand, SQLite (potentially upgrading to PostgreSQL), Celery for Background Tasks, knowledge of Python Pandas library.
Description:
- The project is a web application that combines modern front-end technologies with a Django-based back end to create a robust and responsive web experience.
- Frontend: Utilizes ReactJS with the Next.js framework, providing a dynamic and efficient user interface.
- Backend: Django Rest Framework serves as the API backend, handling data management and interaction with the database.
- API Calls & Caching: React Query is employed to streamline API requests and efficiently manage data caching.
- UI Components: The UI is built using Next UI components, ensuring a clean and polished design.
- State Management: Zustand is used for state management, facilitating smooth data flow within the application.
- Database: Currently using SQLite, with potential plans to upgrade to PostgreSQL for enhanced data scalability and performance.
- Background Tasks: Celery is integrated to run background tasks, allowing for asynchronous and scheduled processing.
P.S: A DEMO of existing application will be given before committing any work
Core requirements:
Load CSV/parquet data, drop/edit columns, apply python APIs, job scheduling, data visualization (plotly).
Problem:
I am looking for a developer to enhance the performance of my data loading and processing application. The current application is designed to load CSV data and invoke Python APIs; however, it encounters difficulties when dealing with substantial data volumes, primarily due to its utilization of browser memory. I am in search of a solution that can efficiently manage extensive datasets while also applying Python APIs for data cleansing. It is not essential to display the entire dataset upfront, and only the initial and final few rows should be visible unless the user requests additional details. Once the user is content with the data, a background task should execute the Python function on the dataset without causing any performance disruptions to the application. The application's frontend should be lightweight and responsive, capable of handling errors gracefully without affecting the ongoing session. Users should experience swift page navigation, and the application should provide informative messages and a progress bar during data processing. I would also need a few additional features in the existing application related to data visualization.
NOTE: The application should be able to run on windows platform, doesn’t matter which O.S you choose for your development. I would expect full deployment instructions and support until its deployed and tested.
Example use case: Load and process CSV/parquet file of 800 MB that contains 585 columns and 2 million rows.
- The primary focus of the enhancements should be on improving the application's performance.
- The application primarily deals with structured data.
- I would also like to add new features and functionality to the application.
- Additionally, I am looking for improvements in the user interface to make it more user-friendly.
- The expected turnaround time for this project is 1-3 weeks.
Skills and Experience:
Tech Stack:
ReactJS (with Next.js), Django Rest Framework, React Query, Next UI Components, Zustand, SQLite (potentially upgrading to PostgreSQL), Celery for Background Tasks, knowledge of Python Pandas library.
Description:
- The project is a web application that combines modern front-end technologies with a Django-based back end to create a robust and responsive web experience.
- Frontend: Utilizes ReactJS with the Next.js framework, providing a dynamic and efficient user interface.
- Backend: Django Rest Framework serves as the API backend, handling data management and interaction with the database.
- API Calls & Caching: React Query is employed to streamline API requests and efficiently manage data caching.
- UI Components: The UI is built using Next UI components, ensuring a clean and polished design.
- State Management: Zustand is used for state management, facilitating smooth data flow within the application.
- Database: Currently using SQLite, with potential plans to upgrade to PostgreSQL for enhanced data scalability and performance.
- Background Tasks: Celery is integrated to run background tasks, allowing for asynchronous and scheduled processing.
P.S: A DEMO of existing application will be given before committing any work