Help Request DDOS / Downtime Situation

Status
Not open for further replies.

Moo

ADISC Admin
Staff
Messages
5,444
Role
  1. Private
We were hit by a DDOS attack. Someone consumed all the bandwidth for our server's internet connection, knocking us offline until we could get a new connection installed.

Most people should be able to connect to ADISC again already.
Those that can't should only need to wait a few days (1-2 days for most ISPs, up to 5-6 for the slowest ISPs).

Please note, though:
  • Some website features may be disabled while I implement additional protections against these sort of attacks.
  • IRC no longer accepts connections via "adisc.org". You must connect to "irc.adisc.org" to use it. Yes, the difference is important.
  • Emails (like password resets) may deliver slowly, to spam folders, or in some cases, not at all. This may take a few days to fix. In the meantime, please post here (or in requests) if you have email-related issues.

Technical explanation, for geeks only (everyone else can skip this):
  • ADISC, like most websites of substantial size, is run off server(s) sitting in a giant warehouse of other servers. This is called a datacenter. They look like this. Every one of those rectangles with a pair of green lights on it is a server, and each of those servers probably hosts a bunch of different things. Generally, each of these small servers cost $50 to $1,000 per month, and are rented by corporations, not individuals.
  • This particular attack, however, was so large it downed the whole datacenter. Our datacenter has thousands of other customers, each with their own websites, all of which were completely down due to the attack targeted at one customer - us. Our ISP, the datacenter, was not happy about this at all. If even a few customers left as a result of this downtime they'd lose many thousands of dollars. In case you're wondering, this sort of $ damage to businesses is the reason why DDOS attacks like the one that hit us are considered a criminal act. Our ISP, the datacenter, did the only logical thing - they asked their ISP for help.
  • Their ISP, a sort of internet backbone company which provides connections to datacenters and other major internet users, then null routed us. This powerful technique, also known as "blackholing" is used by major ISPs to make whatever it is targeted at disappear from the internet. Remember when YouTube went down for a few hours across the globe? That happened because a small ISP in Pakistan put a nullroute on YouTube. It was meant to only block YouTube in Pakistan, but nullroutes are powerful things, and one typo later, the nullroute wiped out access to YouTube across the globe. Well, the same thing just happened to us. A nullroute was issued, and ADISC was put into a sort of internet limbo, unreachable to anyone.
  • To get ADISC back up, I had to do two things. First, we needed a new IP address (ie: new connection, one not affected by the nullroute) and second, this IP/connection had to be hidden (because otherwise whoever attacked us could just attack the new one). After getting the new IP set up, I put us behind CloudFlare (cloudflare.com), a sort of reverse proxy service which hides our real IP address and absorbs, as much as possible, attacks targeted at us, so they never even reach our datacenter, much less our server.
  • There's still a lot of work for me to do here. Attacks like the ones we faced typically cost $200-1,000 a month, plus a great deal of time spent by IT professionals, to deal with. We obviously don't have that kind of money or resources, so I need to find ways of dealing with it in my spare time, on a shoestring budget.
  • I do welcome tips/advice from the technical folks out there, but bear in mind I won't be taking every suggestion. There are a lot of things we can't do for various reasons (cost, time, etc).
  • Speaking of cost, since we're donation-funded I am in no hurry to run out and start spending, for example, $200 a month on a cloudflare "business" level plan (believe it or not, this is one of the cheapest options for dealing with these sort of attacks). I am more likely to deal with this incrementally - adding protection each time we get attacked - because the cost of adding everything all at once would be several times our current operating budget.

Super-Technical steps to regain ADISC access immediately, for hardcore nerds only:
 
M

Marka

Guest
Once again... thank you for your efforts, Moo! -Marka
May the force be with you...
 

AAO

Est. Contributor
Messages
1,005
Role
  1. Diaper Lover
Thanks for using Twitter to update us. That was very helpful.
 

ozbub

Est. Contributor
Messages
1,781
Role
  1. Adult Baby
  2. Little
Unbelievably happy that I'm even able to post this :) down under feels very down under when connections are lost. Ta :hugs:
 
S

Starrunner

Guest
I wish I had some suggestions that would be helpful, but I'm pretty much a luddite who has regular fights and battles with a simple tablet. So all I can do is offer my support and say a thousand thank-yous for all the hard work you do, Moo. We do appreciate it, even though we don't say it nearly as often as we should. :worshippy:
 

MickeyM

Est. Contributor
Messages
813
Role
  1. Adult Baby
  2. Diaper Lover
  3. Babyfur
  4. Little
I've also used Amazon Web Services Elastic Compute Cloud to mitigate attacks. You provision enough infrastructure to weather the attack, they will give up eventually (assuming there is no $$) and you spin down the extra capacity.

I do welcome tips/advice from the technical folks out there, but bear in mind I won't be taking every suggestion. There are a lot of things we can't do for various reasons (cost, time, etc).

I can do consultation about possible solutions, I would need to speak with you to make an informed opinion though.
 
Last edited:
Messages
2,229
Role
  1. Adult Baby
  2. Diaper Lover
I'm curious why we got attacked. I mean, it's probably just someone trolling, but it also seems quite random.

Regardless, thank you Moo for keeping this all up and running.
 

Snivy

Est. Contributor
Messages
2,654
Role
  1. Babyfur
  2. Carer
  3. Private
I don't think we were attacked individually. According to the information I found, other websites were hit. It was the main host that was attacked and ADISC happened to be on that host.

Example, if ADISC was supported by Gator Host and Gator was attacked, ADISC would also be attacked thus knocking us out.
 

Moo

ADISC Admin
Staff
Messages
5,444
Role
  1. Private
I've also used Amazon Web Services Elastic Compute Cloud to mitigate attacks. You provision enough infrastructure to weather the attack, they will give up eventually (assuming there is no $$) and you spin down the extra capacity.



I can do consultation about possible solutions, I would need to speak with you to make an informed opinion though.

I've been looking into AWS EC2 actually :)

I'm curious why we got attacked. I mean, it's probably just someone trolling, but it also seems quite random.

Regardless, thank you Moo for keeping this all up and running.

I don't know why we were targeted.

I do know it was targeted at us, though. We were the only users of the IP that was attacked.

If they were going after our hosting provider, they'd have been more likely to target that provider's routers, switches, or just their network in general.

According to the information I found, other websites were hit. It was the main host that was attacked and ADISC happened to be on that host.

Please PM me that information. I think it is incorrect, but I'd still like to take a look at it myself.
 

LilByte

Est. Contributor
Messages
1,036
Role
  1. Adult Baby
  2. Little
Glad it's back up for now, hope you can keep it up. Good luck.
 

INTrePid

Est. Contributor
Messages
403
Role
  1. Diaper Lover
  2. Private
I'm glad we've got the DDoS attack squashed (good work Moo) but I hope this doesn't mean we'll have to worry about hackers trying to pull off an Ashley Madison style data breach now too. It's fortunate no one registers with their real names but it would absolutely not be good if someone were to somehow get ahold of a list of all the usernames, passwords, and email accounts. With a little detective work, email accounts can in most cases be linked to real identities. The thing that worries me the most is that we have no idea who was behind this attack and what their motive was. A DDos attack large enough to take down an entire data center would require a massive amount of bandwidth and resources. I highly doubt that such a powerful and targeted attack like this was random. This could be anything, from the work of a misguided hacktivist collective trying to take down sites that they suspect of catering to pedophiles to a disgruntled former member special ordering a DDoS attack paid for with bitcoin to a group of Russian hackers. In any case though, you'll want to make sure the login data is secure and site is resilient as possible to any future attacks.
 
S

Speck

Guest
So who thinks this was a randomly generated attack or directed specifically at this site?

This reminds me of the furry fandom, FAF get DDOS attacked all the time.
 

Moo

ADISC Admin
Staff
Messages
5,444
Role
  1. Private
In any case though, you'll want to make sure the login data is secure and site is resilient as possible

All we've seen is DDOS, not hack attacks. That said, we're adding additional layers of security against hack attacks too, such as a Web Application Firewall. Just in case.

We're not a very attractive hacking target. We don't store real names, payment info, etc for users. If they're looking for someone to hack, we're not an attractive target.

So who thinks this was a randomly generated attack or directed specifically at this site?

This reminds me of the furry fandom, FAF get DDOS attacked all the time.

It was directed as us in particular. As I've explained earlier in the thread.

DDOS attacks are becoming twice as common every year. It is only a matter of time until pretty much every significant site has to deal with them.

We've been lucky not to suffer them thus far, but given the size of our site it was always just a matter of time.
 

dw2169

Est. Contributor
Messages
59
Role
  1. Adult Baby
  2. Diaper Lover
  3. Incontinent
Thank you MOO, so many do not have a clue about the amount of work that goes into keeping this group up and running. I don't underrstand why a group like this would be targeted. It had to cost them something to pull off the attack.
 

DTlogist

Est. Contributor
Messages
24
Role
  1. Adult Baby
I've been talking to Moo. At the rough time Adisc's IRC was attacked, EFnet and Freenode were attacked as well. It looks like that the attack was on multiple different IRC networks, that the attack was on IRC networks.

The IRC protocol makes DDOS mitigation effectively impossible; IRC predates 30gbit $5 canons existing by about 15 years.
 

Cottontail

Sailing, sailing, ...
Est. Contributor
Messages
5,281
Role
  1. Adult Baby
  2. Diaper Lover
  3. Sissy
Glad to be able to connect again--just now. ADISC was unreachable via both my home ISP as well as my cellular data plan until literally minutes ago. Wowzers. Thanks for all the efforts in getting things back online, Moo. Hopefully the defenses hold! :)
 

BabyToddler

Est. Contributor
Messages
1,361
Role
  1. Adult Baby
  2. Diaper Lover
  3. Incontinent
Thank you Moo! Thanks for keeping Adisc up! :thumbsup:
 
Status
Not open for further replies.
Top