RNN debugging

I am making a small RNN application for learning purposes. This is a two-layer RNN in PyTorch, but not using the RNN or LSTM class, but written by me. The starting code was using an LSTM class and I rewrote it. Well, as you already can guess, it does not converge (there are no errors, but network does not learn anything useful). I want to have my little application fixed, so that I know what I did wrong. Together with the application you will find a few text files attached, as a database. Ah, one more thing, I want this application to work with a batch size = 1.