Question about storing several TBs of images ...

[NALEM]

We are working on a new project that will eventually host several TBs of images, no video.
If you have experience in managing such a large DB, I would greatly appreciate it if you would get in touch with me. We are looking for a bit of advise, to make sure we are going in the right direction with things.

Also we are looking for Server Admin (Linux) to work with us remotely.

Thanks in advance.

[SilentKnight]

Several TB's of just pics?

Whew...what are they - RAW format?

[PuppetMaster]

That's a lot of images.

[NALEM]

Several TB's of just pics?

Whew...what are they - RAW format?

Nope

That's a lot of images.

Yes it is.

[L-Pink]

That's a lot of ......

[Pac Man!]

Nalem.com does dvtimes photo shoot

[SBJ]

I'd just store them on floppys and be done with it

[j3rkules]

Please... no him again.

[AdultKing]

We are working on a new project that will eventually host several TBs of images, no video.
If you have experience in managing such a large DB, I would greatly appreciate it if you would get in touch with me. We are looking for a bit of advise, to make sure we are going in the right direction with things.

You haven't really provided enough information to give you any useful advice.

However generally speaking if you are dealing with massive databases then using a standard dedicated server model is not the way to go.

You would be using some kind of object storage that was flexible so that it could size up or down according to your needs.

Amazon, OVH, IBM, Google, HP and countless other companies provide highly scalable and low latency storage solutions. The OVH Public Cloud is an interesting product to look at because it's lower cost than many competitors.

Then you need to engineer a way to store and retrieve your data objects from the data store.

Will you need a high level of redundancy? Will you need to index the data before returning it to the end user ? Do you need to scale up and down easily or are you going to provide a fixed retrieval solution on a dedicated machine ?

Engineering big data projects is non trivial and you really need to put in a great deal of work to design a system that will continue to meet your needs as you grow and as user demand scales up and down. You don't want to be paying for compute or storage that you're not using so something cloud based where you pay for what you use is desirable.

Some products to look at

https://aws.amazon.com/solutions/

IBM - Cloud Computing for Builders & Innovators

https://www.ovh.co.uk/cloud/

[SilentKnight]

I'd just store them on floppys and be done with it

Yeah, go with the 5.25" dual-sided.

[NALEM]

I'd just store them on floppys and be done with it

Will this be enough? This is all I have left.

[NALEM]

You haven't really provided enough information to give you any useful advice.

However generally speaking if you are dealing with massive databases then using a standard dedicated server model is not the way to go.

You would be using some kind of object storage that was flexible so that it could size up or down according to your needs.

Amazon, OVH, IBM, Google, HP and countless other companies provide highly scalable and low latency storage solutions. The OVH Public Cloud is an interesting product to look at because it's lower cost than many competitors.

Then you need to engineer a way to store and retrieve your data objects from the data store.

Will you need a high level of redundancy? Will you need to index the data before returning it to the end user ? Do you need to scale up and down easily or are you going to provide a fixed retrieval solution on a dedicated machine ?

Engineering big data projects is non trivial and you really need to put in a great deal of work to design a system that will continue to meet your needs as you grow and as user demand scales up and down. You don't want to be paying for compute or storage that you're not using so something cloud based where you pay for what you use is desirable.

Some products to look at

https://aws.amazon.com/solutions/

IBM - Cloud Computing for Builders & Innovators

https://www.ovh.co.uk/cloud/

"Design a system that will continue to meet your needs" - this is exactly what we all are paying close attention to now, well before launch. Use of a cloud based storage system combined with traditional servers is what has been the plan.

I have already forwarded your links for them to look into.

Thanks AdultKing.

[AdultKing]

Use of a cloud based storage system combined with traditional servers is what has been the plan.

Depending upon the use case, traditional bare metal servers may be an expensive option, there are so many better alternatives now that you'd really want to justify using standard dedicated servers.

It really depends what you're building though.

[NALEM]

Depending upon the use case, traditional bare metal servers may be an expensive option, there are so many better alternatives now that you'd really want to justify using standard dedicated servers.

It really depends what you're building though.

Lead programmer sent me a message, that they have us set up on AWS.
Whatever he sees up there, he is loving it. He went from being stressed to
"this will work great". So a big thank you Adult King.

[Ferus]

One of our clients have +600TB images, with two Windows 2012R2 servers (on Hyper-V with 8Gig of Ram and 8vCPU). The Application frontend sees a SMB3 share that I can just add servers/LUN's to if needed. But so far it have a throughput of about 1,5mill IOPS.

The Storage is on a Tintri solution, but we are planning to set up a tiered storage for a nearline Archive/backup on the cheaper(slower) SAN.



It's really really simple to set up today, but it depends on what you want/need in regards to CDN or Clustering

[MiamiBoyz]

Everything goes away in the end...why bother

[muthisdev]

We are working on a new project that will eventually host several TBs of images, no video.
If you have experience in managing such a large DB, I would greatly appreciate it if you would get in touch with me. We are looking for a bit of advise, to make sure we are going in the right direction with things.

Also we are looking for Server Admin (Linux) to work with us remotely.

Thanks in advance.

What kind of redundancy are you looking for?
Also, more important than how big the images are... How many images? 6 million rows isn't a huge database.