Ok, maybe that was it? Bottlenecks! They just keep moving.

[Mo1]

I have no clue of what you just said .. but thank you for getting us back up and running

[John Robinson]

MySQL, it's been good to us and has a choice of storage engines each with different features. We're using the MyISAM storage engine for everything right now, but it's starting to show signs of contention (MyISAM is really good at heavy writing or heavy reading, but not both) so I'll probably start migrating some tables to InnoDB which handles contention better (and does transactions too.)

[John Robinson]

Oh, the DB is around 30 GB, with another 7 GB of compressed archives which are flat files (actually just HTML.) Those are the older /forum/ URLs you might see from time to time. 2001 and earlier.

[Squantos]

Lets have another fund raiser.........last one was too fast !

[razorback-bert]

I understand, a hugh moose bit your sister in the shower.

[John Robinson]

I was real tempted to install XP and Doom III on one, but I'd still need to acquire the PCI-X vid card and Doom III. Plus I really didn't have the time, if I did, I probably wouldn't have given the machine up. :-)

[tame]

um...okay...john...i think i...got...it.

so, the external clastoid mastoid hyberchronifiar, dismachifnegated the hypostatic ekenosinator, thereby vitiatimating the vortocuticlastical shmagtrofinator...right?

:o)

[swheats]

This is kinda scary I actually understood what you said.

Thanks for being so speedy on the repair.

[plushaye]

Thanks for the info. A 30gb db. The biggest single db of the many sybase servers I work with is a mission-critical one about 45 gb, but we have bigger ones in Oracle, which I don't work with. Interesting you have MySQL for a large db and for such a heavily hit application (FR). I'm surprised but impressed. I have to do some research on MySQL dbs.

[toupsie]

Get a Mac!!! -- Just kidding! (sort of)

Wow! That is some load. I have only seen that on an overloaded multinode Nagios monitoring system with 7,000+ checks. But this shows the power of Unix/Unix-like OSes. They might get overloaded but they don't go down. Throw in a little more hardware and a foundry switch to balance it out and you are golden.

[lara]

No. What's really scary is that I understood what tame was saying. : )

[weegee]

Thanks for getting it fixed in short order (and offering alternate server addresses in the meantime).

[tame]

No. What's really scary is that I understood what tame was saying. : )

:o)

[swheats]

:) jabberwocky

[melkor]

That looks fun. I have no experience with AMD but have been meaning to give them a try.

---

unless your looking to get one of the SUN AMD servers with the insane cooling systems stay away from AMD for a server.. you think you have fire/thermal problems now hehehehhehe.

AMDs make really fast workstations. but they have never made a decent server. they will have to do alot of thermal work before someone can expect them to take the load they need to. Their whole thermal design on the 64 is "well you dont run it at 100% all the time. so we let it cool off between those times"

yours would never cool off :P

thanks for the hard work.

[John Robinson]

I plan to do a complete write-up sometime later. Everything is now clustered into three distinct groups (frontend, backend, database.) All machines are dual processor. Frontend now runs on 933 MHz and 1.4 GHz machines, backend runs on three 2.8 GHz machines, database runs on 933 MHz and 1.4 GHz machines. We have a dual 450 MHz machine for misc. tasks (it's incapable of any serious work these days.) We've upgraded to Giga-Ethernet for our internal network, the forward facing network is constrained to 10 Mbits/sec (a point-to-point connection to Verio, our co-location host.)

My goal is to memcache everything, to hit the database for as little as possible. Right now the two frontends have 2 GB of memory each, with a 1 GB memcache on each. The backends have 1 GB of memory each, with a piddly 256 MB memcache on each. The database machines have 3 GB of memory all to themselves.

The only disks that matter are the DB, they're 15K RPM 74 GB SCSIs for the data, one on each server. I have another 15K RPM 36 GB SCSI for the O/S and log files and some of the tables (just to balance load.)

Oh, and this guy (PDF) is my hero.

The system is capable of syncing across the Internet. In fact, I sync the database over a SSH forwarded port, for backup and development. However, we rely on a master write database, which means any child node would need to communicate with the master database to store anything. Site redundancy isn't yet an option. I can address that with a complete rewrite of the database and software, including some of the concepts we know and love, which is something I both look forward to doing and dread.

[John Robinson]

Thanks for that warning.

Speaking of cooling, those Dells have 7 screaming fans. They really blow. And they're REALLY LOUD! Multiplied by 3. They sat behind me for the duration of their configuration, about 10 days. You will have to speak louder, I'm now partially deaf.

(And I thought my home file server was noisy with 4 HD coolers and misc fans.)

Sure could hear the quiet when the breakers blew. (The breakers blew twice here at home when I was compiling Gentoo on all three boxes. My 650 watt UPS powered them along for literally 5 seconds before it gave up in disgust. LOL, that really sucked!)

[tahoeblue]

That's about what I figured. I could do the hardware but would have issues on the bandwidth.

Right now I have a half a dozen dual Xeon's with a few GB each, 2 dual Opterons with a few GB each, and a number of single CPU machines - P4 and Alpha mostly, and my trust Sparc. A few TB of disk, internal GigE network. I may have to pick up another pair of dual or even quad Opterons soon for a new project.

If you ever get to the point of looking at a live backup site, feel free to send me email. At some point I'm going to have to upgrade my connectivity.

Nice to have the hardware, but it's the power bill that sucks.

-tb

[Pete'sWife]

That might do it.

Thanks!

[shezza]

THANK YOU, Jim, for everything you do to keep this invaluable forum up and running. You're a gem.