Hosted SQL: is there such a thing?

[rowan]

I'm not talking about an overworked mysql server looking after thousands of virtual accounts, more something that's scalable over multiple machines and multiple customers. Whether I insert 1MB or 10TB of data, and do 1000 or 5 million queries per day, it can handle it. The backend hardware is transparent to the customer, all I get is an address to connect to, a username and password for my db.

Does this exist yet?

[Damian_Maxcash]

It must be - you dont want to pay for something you dont need at first though....

I would have thought a host with experiance is what you are looking for.

[rowan]

It must be - you dont want to pay for something you dont need at first though....

I would have thought a host with experiance is what you are looking for.

I'm talking about a model that differs from the "pay us to lease one mega spec'd machine" setup, rather the customer pays for storage and queries with no worries about how it works.

I don't know if SQL can be run over multiple machines.

[Damian_Maxcash]

I am sure GFY is run on more than one machine - maybe ask Lens or Eric.

[MattO]

It's possible, I have a server with MSSQL that I make database calls to from other servers. It's pretty easy with Windows machines because it took me no time to figure it out when I set it up.

[dozey]

Sounds like something one would accomplish on a mainframe, I believe they accommodate that sort of transparency and reliability.

Hosting options are apparently scarce, mostly limited to government or university clients. Makes sense I suppose, the virtualisation paradigm is only just beginning to take off again.

[teksonline]

mysql clustering, you dont need a service what you need is a tech, unfortunately you or your HOST actually sold out the techs the same way tube sites sold out the affiliate industry... there are no more... so you can ask your host and well most of em dont care for your kind of client

[rowan]

I presume mysql clustering is replicating which is aimed at optimizing lots of reads; I want to do lots of writes.

Basically I'm looking for a service that can scale without needing to put up capital of tens of thousands of dollars for my own equipment.

[teksonline]

maybe if i turn the hard drives upside down for you? <blink>

[djscrib]

Amazon's Simple DB solution is supposed to solve this niche if your data is a fit for the model. They charge you based on traffic and CPU usage, and you can scale as large as yo uwant.

[rowan]

Amazon's Simple DB solution is supposed to solve this niche if your data is a fit for the model. They charge you based on traffic and CPU usage, and you can scale as large as yo uwant.

Hmm... it says data size is limited to 10GB during beta, and "Amazon SimpleDB is designed to store relatively small amounts of data and is optimized for fast data access and flexibility in how that data is expressed."

Still, something to keep an eye on. It was Amazon's EC2 service that got me thinking about scalable third party database providers.

[rowan]

FWIW my current mysql db size is on the order of a billion rows in total, and is likely to grow to at least 10 times that. The data should be able to fit on a 1TB drive, but who knows where it will go in the future.

[raymor]

You'll want to have the MySQL server(s) physically located right there
next to the web servers for best performance. That gives a "database
as a service" a major disadvantage if it will be used much. If you just
wanted to STORE a lot of data in a database that didn't get used much
we could set you up on some special systems we have for doing something
similar. Assuming the database will be actively used, your best option is
probably to get your web host or an outside sysadmin such as myself to
set it up for you and provide the layer of "the customer pays for storage
and queries with no worries about how it works".

You mentioned mega speced machines. I should mention that certain database
design decisions, like what indexes to use, can easily make a database ten
times faster or slower, as can the way that the most frequently used queries
are crafted, so check on that stuff before spending big bucks on hardware.
The same database may well run faster on a 500 Mhz Pentium when properly
indexed than it does on a 4 GHz P4 without the right indexes. (My office
has servers with P300, P667, and quad core 3.2GHz CPUs.) After the database
is optimized, get lots and lots of RAM and a good sysadmin can set you up
a great SQL server that will "just work" without you having to worry about it
any further.

[djscrib]

Hmm... it says data size is limited to 10GB during beta, and "Amazon SimpleDB is designed to store relatively small amounts of data and is optimized for fast data access and flexibility in how that data is expressed."

Still, something to keep an eye on. It was Amazon's EC2 service that got me thinking about scalable third party database providers.

I've been told the 10GB limit will go away outside of beta.

By small amounts of data they are really comparing it with S3. 1k chunks or smaller, go in SimpleDB, larger binaries go into S3.

Really what this means is you could build a data driven website with your database taken care of by Amazon, and the static files also hosted by them with infinite scalability. You just need to provide the business logic that holds the two together.

I can't help but assume that EC2 will be upgraded to fill that middle tier at some point.

The basic premise is all the same though, why buy enough hardware to process 100 widgets/second, if you only need that throughput 1% of the time.

Lastly, most databases for this industry are very suited for the SimpleDB layout. you're talking about mostly read-access databases, and a lot of the functionality relational databases provide is overkill.

Lastly, the S3 security model for securing urls is widely underappreciated. You can generate download links which are only valid for say 30 minutes. A ghetto use of this would be to prevent hotlinking completely. If anyone copy/pastes one of your image urls onto their site, the url is worthless after 30 minutes. The url generation code runs locally on your server, so it is pretty much painless to add from a performance perspective.

Anyhow, there are plenty of S3 haters on this thread, but if anyone wants to have a serious discussion feel free to track me down via my website contact info.