Fixing the problem `kex_exchange_identification: read: Connection reset by peer`

I'm having trouble connecting to a specific server

I managed to login once with ssh me@server. Subsequent times I got the following error:

└╼ ssh -v me@server
OpenSSH_8.2p1 Ubuntu-4ubuntu0.2, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /home/infinity/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to server [the ip] port 22.
debug1: Connection established.
debug1: identity file /home/infinity/.ssh/id_rsa type 0
debug1: identity file /home/infinity/.ssh/id_rsa-cert type -1
debug1: identity file /home/infinity/.ssh/id_dsa type -1
debug1: identity file /home/infinity/.ssh/id_dsa-cert type -1
debug1: identity file /home/infinity/.ssh/id_ecdsa type -1
debug1: identity file /home/infinity/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/infinity/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/infinity/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/infinity/.ssh/id_ed25519 type -1
debug1: identity file /home/infinity/.ssh/id_ed25519-cert type -1
debug1: identity file /home/infinity/.ssh/id_ed25519_sk type -1
debug1: identity file /home/infinity/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/infinity/.ssh/id_xmss type -1
debug1: identity file /home/infinity/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.2
kex_exchange_identification: read: Connection reset by peer

It is a bit strange as I can connect to some server, but not that specific one