L337 Server Scalability - How to build a kick ass FAST server that is db intensive

Replication is a possibility but you can only have one master node - and that is the only node you can write too. The mysql (4.x) series has clustering capabilities now, although it may not be ready for production.

Take a look at http://jeremy.zawodny.com/blog/archives/001868.html.

..."Benchmarks on a 72 CPU SunFire box hit 380,000 write txns/sec and 1,500,000 read txns/sec. Nice! More numbers in the slides, but I can't type that fast."

I haved designed/built the supercomputer clusters for dna research at some of the top universities in the U.S. Hit me up if you have any questions.

Hehe - ehm, me too, except that I am a scientific programmer working in a research establishment. Personally, I work on whole-genome databases for specific organisms, writing dbase progs to collate and present all data known about every gene in that organism - not dna BLAST alogrithms per se but presenting any and all data known about a specific gene (developmental expression patterns, promoter regulation, exon/intron patterns blah blah)

Not that I'm doubting in any way the validity of your statements, but since when did universities freelance programming work? Scientific programmers are amongst the best in the world and are more than capable of designing/implmenting their own clusters and/or algorithms? I mean, probably what you are talking about, BLAST is fully OS to the scientific community and has some exquisite C Functions for higher-level (none MySQL queries) arguments for interfacing with MySQL....

As far as the original poster's quagmire, the important Q is out of those millions of daily queries, what % are db entries/modifications? File locking is obviously essential, but how about some time-delay checking on the availability of the dbase? I mean if the dbase is currently being modified and cannot be written to, spool the write to some process which will become modified at the next available time? That way the website doesn't hang, the person who executed the mysql command doesn't have a browser hanging forever and everyone is happy.
'cept you'd need some process control to make sure things don't get out of hand and if too many processes are spawned to start writing to disk to update/insert when things calm down.
Also, the updates wouldn't be live-live but the whole thing should clear within a few seconds or worst-case scenario after a few minutes.

I work for a bioinformatics consulting company. You are correct, most of the software used is open source, but what about special cases? Also, do you really expect some grad students to be able to assemble a 1000+ node beowulf cluster? That just doesn't work, it takes real world experience that grad students do not have. The simplest of issues, such as heat buildup, never crosses their mind, not to mention things like network file storage systems. Good luck running NFS with 1000+ machines.

BTW - it is interesting to run across someone else on this board with similar skills...didn't think many bioinformaticians were in the porn business

Like I said, I never doubted your comments and it was a long time ago when I was a grad student. Thank god!

But I doubt that if we were to ever built such a cluster, the grad students would even get a look in! I'd be hands-on in there giving the company responsible for building it a hand

You're right though - hardware clustering needs to be outsourced, but the software that uses those clusters is very much in-house built. At least it has been for all the labs I've been it, and I know it is for the big-dog NCBI crew.

Here's the system structure I used for a system for dreamworks that was getting between 15-20 million page views a month.

The system used mod-perl, mysql and template-toolkit all running on debian servers with apache. Our image servers used thttpd (a super fast light weight server ideal for images). We used mod-perl because of how it integrates with apache, once you startup apache all your perl modules are in memory.

- 3 Web servers (load balanced w/internal and external NIC)
- 3 Image Servers (load balanced w/internal and external NIC)
- 1 Image Repository NFS mounted to both the web servers and image servers
- 1 Database server

the servers werent anything amazing, the DB servers was a dual proc p3 at the time with 1gig of ram. The key to the systems speed was the caching of the data. We introduced object/item and page level caching with the concept of clean/dirty cache on the web servers. The cache lived on the Image Repository server so once an item got cached on www1, it was also cached on www2 or www3.

listen to the man... i don't think there's anyone here that knows as much as you do.... Uber-Programmer_Guru