Remote Desktop problem in WindowsServer 2012 R2

We have a windows server 2012 R2, we have an office set up, where we connect to it using Remote desktop - LAN (Wired). Suddently, the remote desktop disconnects and we are not able to connect to the server. Then when we restart the server, we will be able to connect. Couldnt figure out what exactly the problem is. Need someone to trouble shoot and fix this asap.