Propose Flask/Waitress configuration to avoid memory crash
Budget: $10 – $30 USD
We need help from an expert in configuring our Python based flask/waitress app for stability.
App: https://github.com/CSS-Electronics/canedge-grafana-backend
Context: The app is deployed on e.g. a PC or AWS EC2 instance. It exposes an endpoint that can be connected to via Grafana dashboards. When a user makes a change in the dashboard, it sends a query to the backend, which fetches meta info and data from disk/S3, processes the data and sends it to Grafana for visualization.
The problem: Currently, the app is stable if users perform "1 update at a time" in Grafana. But if a user sends many requests (e.g. by 'spamming' changes to time periods in Grafana), it will trigger a queue of requests to the backend. In some cases, this results in the app getting killed - in particular when deployed on EC2 with 2 GB RAM. The problem seems to be that too many requests are processed in parallel, causing a memory issue.
Task: We want to retain the same loading speed performance as currently - but ensure that the backend app cannot crash due to 'session request spam'.
To solve this, we believe the flask/waitress settings should be modified. A simplistic method would be to limit the #sessions sent to the backend to 1 - this way out-of-memory crashes should not be possible. However, some of the requests sent to the backend are light-weight "meta info" requests, which should ideally be processed in parallel to ensure speed. Therefore, we assume a better solution would be to somehow restrict the parallel processing based on available memory.
The task is to fork our github project and adjust the waitress configuration to solve the above. Once we verify that it solves the issue, the project is OK.
App: https://github.com/CSS-Electronics/canedge-grafana-backend
Context: The app is deployed on e.g. a PC or AWS EC2 instance. It exposes an endpoint that can be connected to via Grafana dashboards. When a user makes a change in the dashboard, it sends a query to the backend, which fetches meta info and data from disk/S3, processes the data and sends it to Grafana for visualization.
The problem: Currently, the app is stable if users perform "1 update at a time" in Grafana. But if a user sends many requests (e.g. by 'spamming' changes to time periods in Grafana), it will trigger a queue of requests to the backend. In some cases, this results in the app getting killed - in particular when deployed on EC2 with 2 GB RAM. The problem seems to be that too many requests are processed in parallel, causing a memory issue.
Task: We want to retain the same loading speed performance as currently - but ensure that the backend app cannot crash due to 'session request spam'.
To solve this, we believe the flask/waitress settings should be modified. A simplistic method would be to limit the #sessions sent to the backend to 1 - this way out-of-memory crashes should not be possible. However, some of the requests sent to the backend are light-weight "meta info" requests, which should ideally be processed in parallel to ensure speed. Therefore, we assume a better solution would be to somehow restrict the parallel processing based on available memory.
The task is to fork our github project and adjust the waitress configuration to solve the above. Once we verify that it solves the issue, the project is OK.