Modify Nginx Loadbalancing & Catching Solution

I have setup simple Loadbalancing solution for my user having his online video courses. They want some modification in the way its currently working.

Currently :
Loadbalancer directs users to 12 Edge Servers via roundrobin, the course contents are catched in Edge Server from Origin Server cache valid for 10hrs since the edge server diskspace is only 120GB. m3u8 ts video courses are permanently stored in 12TB Origin Server.
Configs of Current Loadbalancer & Edge Server In Bottom.

Current Setup - Nginx RoundRobin Loadbalancer <-> 12 Edge Servers <-> 1 Origin Server .

||||||||||||||||||||||||||||||||||

Issue With Current Setup.
Same Accessed file gets cached in all edge servers visited by users due to round robin setup sending users to split the load. The Edge Servers 120GB Disks gets filled up fast and having issue.
Tempruarly i have setup Generic Hash method to keep their site running. But it will not take load when they put a new course 1 edge server cannot handle it. so looking for a modification in loadbalancing.


||||||||||||||||||||||||||||||||||

Modification Needed.
1. When a user requests a file & it gets catched to a edge server, all the other requests for the same file should be sent to the same edge server Unless theres some bottleneck reached.
2. Max Network usage restriction & Disk Restriction for Edge servers.
- Some of their edge servers have 250Mbps bandwidth, if they use over that they get over usage cost. currently i have put weight for those servers in loadbalancer config.
- Their Edge Servers are 120GB SSD so would like to keep max 100GB, inorder to not delete already catched data due to diskfull.
- If these 2 set restrictions reached, user shouldnt be sent to that edge server till both the values drop below the Specified value.
3. Edge Server should read from 3 available origins, & take file from them, due to disk issue, user wants to split the permanent files in 3 origin server.


Needed Setup - Nginx RoundRobin Loadbalancer <-> 12 Edge Servers <-> 3 Origin Server.

Im guessing this can be achieved by introducing a Healthcheck in Loadbalancer & having some type of a log which loadbalancer can scan the last 10hrs which can be used by loadbalancer to determine whether the file is already catched in any server & send the other users requesting same file to already catched server or redirect to another server if that server network restriction reached.


||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||

@ Current Loadbalancer Servers Config.


upstream onlineacademy {
server edge1.examplesite.com:80 max_fails=3 fail_timeout=15s weight=3;
........................
server edge6.examplesite.com:80 max_fails=3 fail_timeout=15s weight=3;
server edge7.examplesite.com:80 max_fails=3 fail_timeout=15s;
........................
server edge12.examplesite.com:80 max_fails=3 fail_timeout=15s;
}


server {
listen 80;
server_name examplesite.com.com;

location / {
proxy_pass http://onlineacademy;
proxy_set_header Host www.examplesite.com;
proxy_set_header Forwarded www.examplesite.com;
}
}


||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||

@ Current Edge Servers, Catching Config.


proxy_cache_path /path/to/cache levels=1:2 keys_zone=my_cache:10m inactive=12h use_temp_path=off;
server {
# ...
location / {
proxy_cache my_cache;
proxy_cache_revalidate on;
proxy_cache_lock on;
proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
proxy_cache_background_update on;

proxy_pass http://my_upstream;
}
}


||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||

Price : $250 ( Negotiable )

Thank you.