As a non-native English speaker I have to say that I think English wasn't all that difficult to learn in itself. A good friend of mine called it "Easy to learn, but difficult to master", which is probably more fitting.
But I have to say that in the end, it depends on the person (which is also what I gave my vote for in the poll), because of two reasons:
First of all, I taught English myself. We had it in school (where it was a mandatory class), but I didn't really care about it. I learned it relatively quickly because I wanted to, which is probably a pretty big factor why it worked in my case, since my motivation and the "willingness" of the brain to suck up every bit of new information is way bigger (as well as the chance that it will stick in the memory long-term) than with someone who isn't really keen on learning a new language (or a specific one, like English, in this case). Most of the time I was digging up texts I was interested in and translated word-by-word everything I didn't understand so far and with each time, I noticed that I understood more and more. Pronunciation and hearing I learned by hearing spoken English from movies and TV series in their original dub, as well as from web videos and radios and the like. (Which also explains why my "dialect" is some horrible hodgepodge of British, American and Oceanian English, with a good measure of what I considered the most probable pronunciation at the given time thrown in XD). The upside was that I was one of the best in English class in school without ever consciously learning for it, lol.
The second reason is that my native language is an Indo-European (specifically Germanic) one, so it's got relatively many parallels to English from the get-go, since both of them were derived from the same protolanguage less than a millenium ago and belong to the same group of this family, which probably made things significantly easier for me than for someone whose mother tongue is Finnish, Chinese, Japanese, Hindi, Swahili or another language that's remarkably different from this single branch of the language tree.
TL;DR: I don't think that English is one of the toughest languages out there and maybe there isn't one that objectively can be called that. It depends on what your native language is and how closely related it is to the language at hand you want to learn and how big your motivation and sound your reason for learning it is.