I woke up this morning and checked my email.
Tried to, anyway. IMAP server wasn't responding. Tried to SSH in. Nothing. No ping, either. This is not how I envisioned the morning going.
Hooked a monitor and keyboard to the Mac mini. What was up: load average through the roof and a series of messages stating mclpool limit reached: increase NMBCLUSTERS.
Groggily, I figured something weird must have happened to my normally stable server, and I'd gladly sacrifice a four-month uptime to avoid having to think about it. Rebooted. Went back to bed.
I woke up this afternoon and once again failed to check my email. Same symptoms. Slightly more awake, I thought: what have I changed recently? Couldn't imagine how they'd affect anything, but I rolled back greetdelay and dovecot to yesterday's versions. No help. Then I noticed a whole bunch of httpd processes. Hmmmm. That didn't look right. Stopped Apache and things gradually got better. Hmmmm.
I got on IM in case any of my friends had noticed that nothing was working and wanted to get a hold of me. davidj was online and filled in my mental blank: it wasn't that I'd changed the server configuration in a way that made it less stable, it was that he'd gotten a story on his web site featured on Digg, and our web server was getting hammered.
I started Apache and watched the number of processes grow until things went predictably awry, then entered a smaller number for MaxClients in httpd.conf and restarted. Lo and behold, web requests were handled a little sluggishly, but the machine's other services were unaffected.
I'm glad this happened: the next time one of the mini's users makes it big, we'll be better prepared. And the rest of my day was blissfully uneventful.
I wondered what was happening. I figured you were getting as many pages as I was, though, so I didn’t worry about it.
I had assumed that the problem was something with your surrounding network, because whenver I logged in “top” showed minimal usage. Probably because apache was down
Either way, cool that we can now handle a slashdotting!
P.S. – did you consider increasing NMBCLUSTERS? Was the processor or disk spiked, or was it just the network?
Dear Mistre Arthru: Yes, when you were able to log in it was precisely because I had stopped Apache! Ultimately I did also bump NMBCLUSTERS by a factor of 2. The specific numbers for MaxClients and NMBCLUSTERS may not be quite optimal, but we’ll know where to tweak “next time.”
I was both happy and sorry at the same time. It was a great weekend.
I think when you get /. ‘ed your server just melts so the config file wont help like with Digg. But it wont matter to DJ, he’d be so happy to be on slashdot he wont realize anything is wrong.