Consultation for socket communication issues in a network

We have around 80 ESP32 based wireless devices in a center. These devices communicate with our local server which is assigned a static IP. We have a live TCP socket from each of the device to local server. This socket reattempts connection if device doesnt hear from server for 20sec. Everything runs smoothly for a period of time and then suddenly devices start disconnecting and reconnecting to this TCP socket. This goes on for around 1hr or so and then most of the time situation self resolves. If we restart the access point/router we see improvement in the situation but it does recur. During the period of 'reconnections' even HTTP requests from devices to server start to fail. Pings from server to device IP times out too. We can see in access point that the device is still connected to WiFI but somehow it cant reach the server. Also during this period a PC connected to the same WiFi AP/router also cant make server requests.
Our network architecture is-
Local Windows based Server PC-TP link router-(Cascaded routers/Multiple access points)