What's Up With Firehawk.Org?
here you go, its a sticky over there right now
Site Outage
Post new topic Reply to topic The Official Firehawk Message Board Forum Index -> General Discussion
View previous topic :: View next topic
Author Message
Firehawk
Joined: 11 Jan 2002
Posts: 656
Location: Indianapolis, IN
PostPosted: Thu Aug 30, 2007 4:49 pm Post subject: Site Outage Reply with quote
Sorry for the outage, everything should be back to normal.
Will explain more later tonight after I get something to eat.
Brian
Back to top
View user's profile Send private message Send e-mail Visit poster's website Yahoo Messenger
Firehawk
Joined: 11 Jan 2002
Posts: 656
Location: Indianapolis, IN
PostPosted: Thu Aug 30, 2007 8:46 pm Post subject: Reply with quote
[SHORT NON-TECH VERSION]
Brian hosed the box and spent 24+ hours trying to fix it.
Its fixed and back to normal now.
[/SHORT NON-TECH VERSION]
So here the longer story of that happened.
Yesterday the server was getting hammered by some machine over seas trying to send a boat load of mail through it.
I setup some filtering, but found a problem in sendmail that would allow that machine to make things even worse. I figure it was time to finally setup the firewall on the server, but found I didn't have the kernel built with firewall support. Turns out it was built as a shared library and I stupidly tried to load it (mistake #1). Trying to load it made the kernel panic and lock the box.
I got a hold of the colo guys and got them to reboot the box. It takes a while to boot since it has to check 500G of disk when it is not rebooted cleanly. It boots up and everything is looking good and then it disappears again. Turns out the colo guys didn't see it come up the first time and figures it needed another kick and rebooted it again (mistake #2). When it cam up this time it checked the 500G and found minor problems, but wanted human intervention.
My machine has serial console access enable, this mean that I can get all the boot up messages and control remotely (just like being at the keyboard and monitor). It seems the boot loader I am running has some bugs because there is no way to redirect the output from the serial console back to the keyboard and monitor output. This normally isn't a problem, but currently it is because console access isn't working right now (mistake #3). So the machine was booting up, detects the machine wasn't cleanly rebooted and checks the disk. While checking the disk it finds a few problems which is enough to force human intervention, but it is sending the prompt to the console port (which isn't working) instead of the keyboard and monitor.
So I have spent the last 24+ hours trying to figure out how to disable serial access and get keyboard and monitor working so I could literally type:
Code:
enter
fsck -y
y
y
and wait 20mins while the disks are being checked and then reboot the machine cleanly.
Needless to say, it has been a long day.
Brian
Back to top
View user's profile Send private message Send e-mail Visit poster's website Yahoo Messenger
The Silverdude
Joined: 03 Oct 2006
Posts: 27
Location: Picayune, MS
PostPosted: Fri Aug 31, 2007 3:08 pm Post subject: Reply with quote
Bummer about those problems! What OS are you running? I'm guessing something Linux based? Hopefully you have the firewall going now, and those mad overseas spammers won't be taking over the site again!
_________________
Black 2002 Trans Am Firehawk # 1133, Pioneer AVIC-N1 DVD nav system, custom Audiobahn sound system, Wolf custom cover, custom plates "O2 FHWK"
Back to top
View user's profile Send private message AIM Address Yahoo Messenger MSN Messenger
Matt B
Joined: 19 Mar 2002
Posts: 754
Location: Des Moines, Ia
PostPosted: Fri Aug 31, 2007 3:47 pm Post subject: Reply with quote
Don't worry about it Brian.. we appreciate your time and effort to keep it all going!!!
Matt
Back to top
View user's profile Send private message Send e-mail Visit poster's website AIM Address Yahoo Messenger MSN Messenger
Firehawk
Joined: 11 Jan 2002
Posts: 656
Location: Indianapolis, IN
PostPosted: Fri Aug 31, 2007 4:48 pm Post subject: Reply with quote
The Silverdude wrote:
What OS are you running? I'm guessing something Linux based?
Running FreeBSD. At the time it had the best 64bit and multiple processor support. Used to run Linux (and still do at home), but need to take another look at it.
The Silverdude wrote:
Hopefully you have the firewall going now, and those mad overseas spammers won't be taking over the site again!
Nope, no firewall. I really don't need it as I think I will just hack sendmail to do what I want it to. Currently when I configure sendmail to deny connections from a ip address it will be nice and still allow it to connect, but deny all commands. I think I will change that to just deny the connection all together and not let it consume any resources.
Brian
Post new topic Reply to topic The Official Firehawk Message Board Forum Index -> General Discussion
View previous topic :: View next topic
Author Message
Firehawk
Joined: 11 Jan 2002
Posts: 656
Location: Indianapolis, IN
PostPosted: Thu Aug 30, 2007 4:49 pm Post subject: Site Outage Reply with quote
Sorry for the outage, everything should be back to normal.
Will explain more later tonight after I get something to eat.
Brian
Back to top
View user's profile Send private message Send e-mail Visit poster's website Yahoo Messenger
Firehawk
Joined: 11 Jan 2002
Posts: 656
Location: Indianapolis, IN
PostPosted: Thu Aug 30, 2007 8:46 pm Post subject: Reply with quote
[SHORT NON-TECH VERSION]
Brian hosed the box and spent 24+ hours trying to fix it.
Its fixed and back to normal now.
[/SHORT NON-TECH VERSION]
So here the longer story of that happened.
Yesterday the server was getting hammered by some machine over seas trying to send a boat load of mail through it.
I setup some filtering, but found a problem in sendmail that would allow that machine to make things even worse. I figure it was time to finally setup the firewall on the server, but found I didn't have the kernel built with firewall support. Turns out it was built as a shared library and I stupidly tried to load it (mistake #1). Trying to load it made the kernel panic and lock the box.
I got a hold of the colo guys and got them to reboot the box. It takes a while to boot since it has to check 500G of disk when it is not rebooted cleanly. It boots up and everything is looking good and then it disappears again. Turns out the colo guys didn't see it come up the first time and figures it needed another kick and rebooted it again (mistake #2). When it cam up this time it checked the 500G and found minor problems, but wanted human intervention.
My machine has serial console access enable, this mean that I can get all the boot up messages and control remotely (just like being at the keyboard and monitor). It seems the boot loader I am running has some bugs because there is no way to redirect the output from the serial console back to the keyboard and monitor output. This normally isn't a problem, but currently it is because console access isn't working right now (mistake #3). So the machine was booting up, detects the machine wasn't cleanly rebooted and checks the disk. While checking the disk it finds a few problems which is enough to force human intervention, but it is sending the prompt to the console port (which isn't working) instead of the keyboard and monitor.
So I have spent the last 24+ hours trying to figure out how to disable serial access and get keyboard and monitor working so I could literally type:
Code:
enter
fsck -y
y
y
and wait 20mins while the disks are being checked and then reboot the machine cleanly.
Needless to say, it has been a long day.
Brian
Back to top
View user's profile Send private message Send e-mail Visit poster's website Yahoo Messenger
The Silverdude
Joined: 03 Oct 2006
Posts: 27
Location: Picayune, MS
PostPosted: Fri Aug 31, 2007 3:08 pm Post subject: Reply with quote
Bummer about those problems! What OS are you running? I'm guessing something Linux based? Hopefully you have the firewall going now, and those mad overseas spammers won't be taking over the site again!
_________________
Black 2002 Trans Am Firehawk # 1133, Pioneer AVIC-N1 DVD nav system, custom Audiobahn sound system, Wolf custom cover, custom plates "O2 FHWK"
Back to top
View user's profile Send private message AIM Address Yahoo Messenger MSN Messenger
Matt B
Joined: 19 Mar 2002
Posts: 754
Location: Des Moines, Ia
PostPosted: Fri Aug 31, 2007 3:47 pm Post subject: Reply with quote
Don't worry about it Brian.. we appreciate your time and effort to keep it all going!!!
Matt
Back to top
View user's profile Send private message Send e-mail Visit poster's website AIM Address Yahoo Messenger MSN Messenger
Firehawk
Joined: 11 Jan 2002
Posts: 656
Location: Indianapolis, IN
PostPosted: Fri Aug 31, 2007 4:48 pm Post subject: Reply with quote
The Silverdude wrote:
What OS are you running? I'm guessing something Linux based?
Running FreeBSD. At the time it had the best 64bit and multiple processor support. Used to run Linux (and still do at home), but need to take another look at it.
The Silverdude wrote:
Hopefully you have the firewall going now, and those mad overseas spammers won't be taking over the site again!
Nope, no firewall. I really don't need it as I think I will just hack sendmail to do what I want it to. Currently when I configure sendmail to deny connections from a ip address it will be nice and still allow it to connect, but deny all commands. I think I will change that to just deny the connection all together and not let it consume any resources.
Brian
