[Derek Zeanah]

We were under attack by someone who was pretty much just throwing enough packets at our connection to try and make new connections impossible. This caused our firewall to run out of memory after the first 45 minutes, which was my fault for configuring it over-optimistically. After that the firewall (with reasonable settings) held together just fine for the next couple of hours, until my ISP decided the only way to block the attack was to drop incoming packets until the attack slowed down.

I'll try and get more info, but for now we're back up. :)

[RNB65]

Was this the same cause as last month's long outage or something different?

[Jamie C.]

Almost every gun board I visit was down last night.

Made for a long, boring night. :(


J.C.

[ZeSpectre]

Ah hah, we should ban packets! (yes that was a joke).

[Geno]

Alright...where were the Dems lastnight?! Ah huh?! Yeah...thought so!

[jlbraun]

I recall hearing that the server originating the packets was in Chicago. Hmmm...

:mad:

[OnDuty]

Alright...where were the Dems lastnight?! Ah huh?! Yeah...thought so!

LMAO

[yhtomit]

I dreamed last night that when the forum came back up, someone had immediately posted a joking, tasteless explanation (which the poster had intended as lighthearted, rather than cruel) that it was because one of the administrators (Derek) had died of unknown causes, and this was followed by a string of angry respondents berating that original poster for suggesting such a thing, and the original poster soon enough apologized for non-THR behavior.

Therefore, a good outcome ;)

Glad you're alive, Derek.

timothy

[M92FS]

it's great to be back at the High Road ! :)

big thank you to you Derek ! :)

[gyrfalcon16]

This caused our firewall to run out of memory after the first 45 minutes, which was my fault for configuring it over-optimistically. After that the firewall (with reasonable settings) held together just fine for the next couple of hours, until my ISP decided the only way to block the attack was to drop incoming packets until the attack slowed down.

Almost every gun board I visit was down last night.

If your firewall crashes you've got some major issues right off the bat. Do you know what sort of attack you were actually under? If you want any assistance feel free to contact me... I deal with issues like this quite often.

BTW: Your ISP is jerking you around if you run a commercial website with them. They didn't need to shutdown your site to deal with the attack and could have instated ACLs on their edge routes to stop it.

[Tob]

Glad we're back up. Thanks :D

[txgho1911]

Found some educational reading on reactions to different dos ddos drdos attacks.
http://www.grc.com/dos/drdos.htm
http://www.grc.com/dos/grcdos.htm
http://www.grc.com/dos/intro.htm

[bogie]

What we really need is a physical address...

Actually, I'll bet it wasn't just gun boards. I'll bet a lot of republican-leaning outfits got zapped last night...

The internet is probably the 2nd amendment activist's best weapon. You know, the more I think of it, that attack was essentially a terrorist attack.

[Derek Zeanah]

If your firewall crashes you've got some major issues right off the bat. Do you know what sort of attack you were actually under? If you want any assistance feel free to contact me... I deal with issues like this quite often.

BTW: Your ISP is jerking you around if you run a commercial website with them. They didn't need to shutdown your site to deal with the attack and could have instated ACLs on their edge routes to stop it.We were seeing sustained inbound of around 50 Mb/s. Once I set the state table to something reasonable the firewall handled the load just fine, and THR barely slowed (THR was the target, btw.)

The problem is that I get billed at the 95th percentile, and with bandwidth costing me $50/Mbit, I didn't want to let that pile up too much. It's not like THR is a profit making enterprise, after all. ;)

Re: my colo provider. I'm trying to get the details. If it was one host flooding us, you'd think it would be just as easy to have the border router drop packets based on source rather than destination; if it was distributed, then that's a bit harder do to and maintain.

I honestly don't know more than I've disclosed so far.

[RNB65]

It'd be interesting to get hold of a packet capture and take a look at what was being sent. What IP protocols were used and whether the packets have forged source addresses. Not sure if we'd learn anything useful, but it's always good to know what you enemy is up to.

Somebody doesn't like us. :(

[Gewehr98]

If it was one of those Distributed Reflected Denial Of Service attacks, finding the source IP(s) would be a herculean effort. :(

[skinnyguy]

It's great to see THR back up. I was wondering what was happening when I couldn't log on all day yesterday.

I guess it just proves a point. If someone has a truth you don't want known, you shut them up any way possible.

THANKS Derek. I appreciate you working on finding answers, and for the site.

[Jorg]

My guess is it was some kind of SYN flood with spoofed source addresses that caused the number of open connectios to skyrocket due to holding things open waiting for the ACK after sending a SYN/ACK off into the ether and bringing things to a crawl.

However, it could have been something more fancy. Hard to know without traces and even then, spoofing makes it tricky to figure out what really happened without access to more info than you probably have.

[.cheese.]

Derek - they were also performing a DOS attack on your nameserver btw and also were attacking your registrar's whois entry for your domain. It took me a while to figure out what was going on, but once I did I was trying to figure out the extent of it, and then I tried searching the "underground" websites to see if anybody was claiming responsibility for it.... but nope. The NS and WHOIS things could be unrelated though.

My guess, Clinton has a new laptop. ;)

and agreed that it was probably a SYN flood.

[helpless]

As I watched the debate last night all I could think of was how interesting that all my gun forums were down.

Total BS

[Fergy35]

to know about the technical side of it, but I was definitely going through a case of withdrawal. Glad you guys were able to get everything back up and running.

[Old Fuff]

The problem is that I get billed at the 95th percentile, and with bandwidth costing me $50/Mbit, I didn't want to let that pile up too much. It's not like THR is a profit making enterprise, after all.

If you need money to fight this sort of thing, ask for it. :fire:

I have no doubt that the membership will back you . The Old Fuff included.

[helpless]

Quote:
The problem is that I get billed at the 95th percentile, and with bandwidth costing me $50/Mbit, I didn't want to let that pile up too much. It's not like THR is a profit making enterprise, after all.
If you need money to fight this sort of thing, ask for it.

I have no doubt that the membership will back you . The Old Fuff included.

This is true

[Nematocyst]

...could have instated ACLs on their edge routes ...

If it was one of those Distributed Reflected Denial Of Service attacks...

they were also performing a DOS attack on your nameserver btw
and also were attacking your registrar's whois entry for your domain.Excuse me, please ... does anyone here speak English? :confused:

Only kidding. :D

I understand just enough about servers, packets and IP stuff to be dangerous,
but mostly follow this in a very broad sense.

(Seriously, though, some of you IP folks should start an education thread in Tech Support some day to educate the rest of us about these kinds of things. I'd definitely participate as a student. It's gun related because it's THR related, IMO.)

It's great to be part of a community that has not only superb education about firearms, but about how to keep the community connected in that global-scale non-linear network called "Internet", a virtual ecosystem where parasites and predators are common.

If you need money to fight this sort of thing, ask for it. :fire:

I have no doubt that the membership will back you .I, too, fully agree with that.

Derek, there are a BUNCH of us in here that would gladly send bucks your way to help with what you need. As a small business owner (and a fairly new one at that), I'm far from a rich man, and summer is my hardest time. But I'd always send what I can to help. And with a lot of member sending even a small amount, we could fund what you need.

Just let us know if you need anything.

Nem

[tyme]

Can anyone confirm that ar-15 was down at some point?
Was freerepublic hit?
Was bladeforums hit?

[cosine]

Bladeforums seemed to be mostly up and running. I visited it a couple of times when I couldn't get on THR or APS.

[Jorg]

tyme, I can confirm ar15 was down for a bit, there was even a thread in their general forum. It wasn't nearly as long. I'll see if I can find thread.

Edited to add:
Here we go...
http://www.ar15.com/forums/topic.html?b=1&f=5&t=577694

[tyme]

Interesting. TFL got hit right about when that thread on arfcom started (i.e. when the arfcom attack stopped).

[MikePGS]

You really should make a donation page for this site. I'm not saying that it would provide enough income to give you an excessive amount of bandwith or anything... but I for one really enjoy this site and would like to help suppourt it any way that I could. And i doubt that I'm the only one in that same boat...

[Nematocyst]

You really should make a donation page for this site. Second, third, fourth, fifth, sixth, seventh and eighth.

OK, OK, ninth.

<singing>

"I got money burning a hole in my pocket,
looking to support a good gun forum and RKBA..."

[Green Lantern]

Had me worried.

FWIW, Glock Talk (about the only other gun board I visit) was fine AFAIK during the downtime.

Hope you can find and prosecute the bums that crashed us!

[PercyShelley]

Are the police looking into this?

[RNB65]

Are the police looking into this?

Derek will have to answer that to be certain, but I'm sure they're not. DOS attacks are a dime a dozen on the 'net and could have been launched from anywhere in the world. Unless the attacker is a complete idiot, identifying him will be impossible.

Also, LE typically does not get involved unless there is substantial financial loss or some really important data is threatened. Since THR isn't a commercial website and no national secrets are housed here, LE won't touch it with the proverbial 10ft pole. Nobody gives a darn about THR being down other than THR users.

[Rocketman56]

They do track these things..

I used to be Network Manager for a large West Coast Research facility..
We dealt with these things all the time..


+1 on the donation, if you need it.. Don't hesitate to ask..

Steve

[exar]

I would be somewhat surprised if the attack had a hidden agenda. These DoS attacks have gotten ridiculously numerous over the net these days. However, the Ping of Death is nothing new. It's an easy fix by the sys admin. There are many types of attacks though and it can be a pain to stay on top of it if you're constsntly targeted. I'm a telecomm tech by trade and I like to ping things to death in order to test equipment durability.

[Geronimo45]

Wondered why things were running so slow.

[gyrfalcon16]

We were seeing sustained inbound of around 50 Mb/s. Once I set the state table to something reasonable the firewall handled the load just fine, and THR barely slowed (THR was the target, btw.)

The problem is that I get billed at the 95th percentile, and with bandwidth costing me $50/Mbit, I didn't want to let that pile up too much. It's not like THR is a profit making enterprise, after all.

You might want to look at moving to a different provider if they're going to count DoS attacks as traffic.

The Planet (http://theplanet.com) uses Arbor networks peakflow to identify DoS attacks and then blocks them with TippingPoint and Cisco Guard systems. They're an okay host if you want to think about moving.

[Green Lantern]

WHY'D I go and open my mouth? Now GT is down...:banghead:

[Phyphor]

If you need money to fight this sort of thing, ask for it.

Actually, given an actual home address of the instigator(s), I'd say a baseball bat and access to the offending originating machine would suffice...... :evil:

[DrLaw]

What happened this week. Could not get THR to come up since Saturday.

The Doc is out now.

[hceptj]

thx!!!

[benedict1]

You really should make a donation page for this site. I'm not saying that it would provide enough income to give you an excessive amount of bandwith or anything... but I for one really enjoy this site and would like to help suppourt it any way that I could. And i doubt that I'm the only one in that same boat...

I'll donate, just tell me how!

[Car Knocker]

I'll donate, just tell me how!

http://www.thehighroad.org/showthread.php?t=277958

[benedict1]

You can send him a Pay Pal donation to at derek@zeanah.com

He posted this but put a couple of extra spaces in his email so it wouldn't work with cut and paste. The above is his correct address.

Pony up boys--this is the best forum around and when it is down I get depressed!

[1 old 0311]

Glad to see you back up. Thanks for al your effort.

[dasmi]

I recall hearing that the server originating the packets was in Chicago. Hmmm...
That really doesn't matter. The attacking machine or machines could've been compromised. The attacker could be Cairo for all we know.
Also, the reason Derek didn't make his email a clickable link, was so that spam bots don't suck it up from the thread and add it to their list.

[chongfa]

It is nice to be back on again. Thanks Derek!!

[AmbulanceDriver]

Yaaaaay. I'm so glad to have THR and APS back.... Me happy now....

Thanks Derek!

[RNB65]

Downtime? We've had downtime?? What downtime??? ;)

[BigRobT]

I'll bet that he added the extra spaces to confuse the spambots.

[metallic]

Can we get a time line of when all the different boards went down? Some interesting timing here. Maybe the person attacking THR is targeting different boards at different times.

[RNB65]

THR and APS are hosted on the same server. You take one down with a DOS attack, you take the other down also. The attacks are THR specific. It's someone who has a grudge against Oleg and/or THR.

Thank God for TFL.

[.cheese.]

ok.... so just to confirm - THR and APS were in fact down for what seems like close to a week?

Sounds like I wasn't the only one.... but I want to make sure. I was worried maybe to counter whatever attack was going on, Derek or Oleg maybe had blocked a range of IP addys that I coincidentally fell under. Then I ran across the THR statistics during a google search and noticed that it wasn't logging new posts for days, so that seemed to confirm that it wasn't just me - but I just want to double check.

Also, if this was another DOS attack, or part of the same one, I say get the FBI involved at this point.

If there is anything I can do to help, don't hesitate to ask.

[middlechainringguy]

Glad we are back on line, very concerned about the terrorist censorship if that is what it was.

[twenty711]

What would you guys say to a proposed alternate form of THR if such an attack happens again? I'm not sure what it would be exactly. Maybe a designated chat room or an email newsletter? What do you guys think?

[.cheese.]

I will start up an IRC room/server soon, unless Derek or somebody objects. Are there any objections from the admin or mods for me to have an html/cgi, or html/java interface with a domain going to it that includes the letters "THR" (with of course a disclaimer on the page that it is unofficial). I know stuff like this falls under fair use with the disclaimer and all, but still I won't do it if the admins would rather me not as it would be disrespectful.

[dasmi]

If you could go ahead and make the IRC channel on irc.oftc.net, that would be great, since I'm connect there all day anyway :)

[RNB65]

What would you guys say to a proposed alternate form of THR if such an attack happens again?

We already have that. It's called TFL. Works great. :)

http://thefiringline.com/forums/

[.cheese.]

or actually.... hmmm... instead of that, how about this:

I started #thrchat on freenode.

If you have an IRC chat client, I'm getting it set up to be a permanent channel in case the site goes down so we can all make sure it's not just us! lol

ETA: Sorry dasmi... didn't see your post in time, but it should be ok, your client lets you connect to multiple servers right?

[dasmi]

Sure, no problem, I was just trying to be lazy about it :)