Question for Server and Programer Gurus. "Soft Update"

[boneprone]

Ive been having a political debate over the past 12 hours with my own programers and staff over wheather I should have "soft updates" activated on my server?

The discussion and arguent has been between my head programer, my script programer, and my Noc-Server Programer.

The deal is my out cgi links seem to be laggin on my server with a new trade script. Its a fucking AWSOME script! AWSOME! But i notice it lagging a bit when links out are clicked. Not much but a bit to where someone picky like me notices.

They reconmned I do a softupdate.
However the stabilty of this has my NOC programer concerned.
I guess if shit crahes all hell breaks loose ,a nd i can loose data and even corupt other files.

My main progrmer is like "what the fuck are soft updates" and like me are trying to figure out what the hell to do?

Does anyone here have softupdates on or off and why?

Like i said, the out links are a bit slow, but really not that bad!
But, im told if i do this soft update it will be Fast! Which really turns me on.

Anyone know anything about this shit? and have an opinon?
It seems to be split 50-50 between my people.

Anyhow Im enabling the ost update for the next 24 hours to see how it goes and decide from there.

[salsbury]

softupdates is safe. safer than not using it.

basically, softupdates guarantees that the data written to your disk is exactly what it should be. it'll write the data to the disk and only after that it'll update the "metadata" to reflect that new data has been written. if the power goes out after data is written but the metadata isn't updated (a very, very quick update, btw) then that data is basically "discarded" because it's not part of any file.

without softupdates, the data is written basically sequentially, and it's possible that the data could be written half-way when the power goes out.

it also improves disk performance, to varying degrees.

it's only available for FreeBSD, btw.

[boneprone]

from what i understand it is just the opposite?

[boneprone]

that softupdates on is more efficient in its writing, but performance is greater becasue it meta stores the info and writes it peridiocally.

So if you have an outage inbetween this meta and writing you loose everything.
And can also run the possibility of corrupting other files.

There seems to be 2 very strong arguments over this.

[salsbury]

Originally posted by boneprone
from what i understand it is just the opposite?

there are basically 3 ways to use disks. the normal way, asynchronous, and w/ soft-updates.

asynchronous might be the one you're thinking of - that one is not safe but is very fast. there's no guarantee that the data will be written to the disk, ever, with that.

[boneprone]

here is one argument against.


Noc@ (3:33 PM) :
what soft updates does is basically every time you edit a file, add a file, move a file, or do any kind of operation that would require a harddrive to write data, it forces the harddrive not to write that data

Noc(3:33 PM) :
Instead it stores it as meta data, and then when enough is stored it writes it all out at once

Noc@(3:33 PM) :
This makes opening,closing,unpacking files quicker, because you are decreasing the amount of times you write to the disk

Noc@(3:34 PM) :
Its like a buffer

Noc@(3:34 PM) :
So first edits goto the buffer, then whent he buffer has enough it writes to the disk

Noc@3:34 PM) :
If you have a power failure, crash, or harddrive crash, then this meta data is lost

Noc@ (3:35 PM) :
This can mean anything from losing files that you updated in the last hour, to losing directories if they were moved but had not yet been written, in rare cases even partitions can be come corrupted

Boneprone (3:35 PM) :
ahh so the potential we can loose on a crash is just that date in between the time of writing?


Noc@:
no, you can essentially lose everything, because meta data when it gets messed up it may force errors on neighboring files

[salsbury]

Originally posted by boneprone
that softupdates on is more efficient in its writing, but performance is greater becasue it meta stores the info and writes it peridiocally.

So if you have an outage inbetween this meta and writing you loose everything.

you'd only lose the data that was not yet completely written to the disk. given that it wasn't complete, you don't even know if the data was correct yet. it's better to just "pretend" that data was never written than to have to worry if it was corrupted.

you certainly don't lose everything.

And can also run the possibility of corrupting other files.

There seems to be 2 very strong arguments over this.

i've never heard of this, except perhaps very early on when softupdates was being introduced. it's very stable now. i've never seen a single problem with it.

it's to the point that FreeBSD 5.0 will have background fsck'ing - filesystem checking - because regular fsck'ing will no longer be necessary.

[salsbury]

Originally posted by boneprone
here is one argument against.
Noc@ (3:35 PM) :
This can mean anything from losing files that you updated in the last hour, to losing directories if they were moved but had not yet been written, in rare cases even partitions can be come corrupted

it's more like 15-60 seconds, not an hour.

Boneprone (3:35 PM) :
ahh so the potential we can loose on a crash is just that date in between the time of writing?


Noc@:
no, you can essentially lose everything, because meta data when it gets messed up it may force errors on neighboring files

it really sounds like Noc is thinking of async rather than softupdates.

i'll bow out and let other people comment now.

[boneprone]

thanks man, this helps

[boneprone]

And its just not this one NOC who is saying this.
Its several.


And I also have several programers tellling me to do the softupdate.

Like i said its a political debate in the family.

[frankfortuna]

Here are some rules to live by:

1. Use a journaling filesystem.
2. Spindles, spindles, spindles. If your concerned about disk performance, stripe your data across multiple spindles.
3. Mirror. Mirror your data / stripes. 1+0 or 0+1 schemes are recommended depending on the amount / type of data you're talking about. RAID-5 schemes are not an effective option in my opionion.
4. Stop fucking around with Linux / FreeBSD solutions and get yourself a real OS. You'll do a better mitigating risk and managing performance using Solaris with Veritas Volume Manager and Veritas File System (VxVM and VxFS respectively).

[boneprone]

Solaris ?????????
WTF?

[frankfortuna]

Yes Solaris. But that issue aside, it sound like you need a better way of managing your strorage. Trust me, if you evaluate the I/O of the application in question and create stripe set(s) with a width that aligns with the I/O operations being performed, you'll notice a dramatic increase in performance.

If I may ask, what is the configuration of the box in question? Number of I/O controllers, type, and # of disks?

[boneprone]

I have no fucking idea!
I barely know html man.

Im like George Bush, I have no fuckign idea whats going on, thats why i surround myself with people in the know.

I actually dont even know what you are asking?
But i can tell you when to double down, split, and hit.

Just give me a yes or no on the softupdate idea, and why.

[vending_machine]

Originally posted by frankfortuna
Here are some rules to live by:

4. Stop fucking around with Linux / FreeBSD solutions and get yourself a real OS. You'll do a better mitigating risk and managing performance using Solaris with Veritas Volume Manager and Veritas File System (VxVM and VxFS respectively).

hahahahaha
(not gonna dignify that a real answer)

[frankfortuna]

You laugh, but I doubt you know the difference between a Unix kernel and Colonel Sanders.

[vending_machine]

Originally posted by frankfortuna
You laugh, but I doubt you know the difference between a Unix kernel and Colonel Sanders.

edit; never mind.

[frankfortuna]

To answer your questions, soft updates can be a good way of improving small I/O performance, but there are some risks to be assumed. Mainly, if you have a panic or power loss before the syncer can commit the data to disk, you're hosed.

[salsbury]

Originally posted by frankfortuna
To answer your questions, soft updates can be a good way of improving small I/O performance, but there are some risks to be assumed. Mainly, if you have a panic or power loss before the syncer can commit the data to disk, you're hosed.

this is the case with regular filesystems. softupdates makes it so you're not "hosed". rather, your file is left "untouched" if the system loses power during a write.

the "syncer"'s last step is to mark that the data is complete after it is all written to the disk.

and yeah i said i'd bow out, but ya gotta fight FUD as you see it.

[vending_machine]

Originally posted by frankfortuna
Here are some rules to live by:

4. Stop fucking around with Linux / FreeBSD solutions and get yourself a real OS. You'll do a better mitigating risk and managing performance using Solaris with Veritas Volume Manager and Veritas File System (VxVM and VxFS respectively).

Ok, while you're laughing at me, since apparantly I'm a comedian, answer me this:

How much would a system like that cost? $20+k at the very least?


How would it compare to buying two pIII 1Ghz w/ FreeBSD and then put them in a round robin with failover. This is the adult internet, not NASA. Who cares if Joe Blo clicks on a link and the one round robin server crashes causing one click to go amiss.

[frankfortuna]

You're absolutely correct, I confused this somewhat with pure write buffering. I missed a few hours of sleep yesterday and spoke out of turn.

My only other advice with soft updates is to watch memory utilization if this is already an existing concern.

In summary, I would recommend soft updates if we're talking about hundreds of short-lived, small I/O operations such as temporary files.

[frankfortuna]

Originally posted by vending_machine


Ok, while you're laughing at me, since apparantly I'm a comedian, answer me this:

How much would a system like that cost? $20+k at the very least?


How would it compare to buying two pIII 1Ghz w/ FreeBSD and then put them in a round robin with failover. This is the adult internet, not NASA. Who cares if Joe Blo clicks on a link and the one round robin server crashes causing one click to go amiss.

Actually it's not as much as you'd think. Netra T1's are quite cost effective and plentiful on the used market for around $1500 or less. If you are concerned about the availability of web services and load balancing, Sun Cluster 3.0 does an excellent job at $2500 list per system.

In summary, you can do more for less with Sun systems in my opinion, hence, an overall cost reduction is achieved. And for those who aren't comfortable with the notion, my suggestions can be applied to Linix and BSD.

[boneprone]

so frank, you say no?

[frankfortuna]

I say go for it. It can be implemented in a matter of minutes, however, you will need to completely unmount / remount your filesystems, which will create a brief outage.

One last question, is this script using mySQL by chance?

[frankfortuna]

Hey, I'm going to step out for a bit. I sent you an email with contact details. Do not hesistate to email me if you have additional questions / concerns.

[boneprone]

mySQL by chance?

No.

[notjoe]

Originally posted by boneprone
thanks man, this helps

I run softupdates on all my freebsd servers, i can do shit with them and pull the plug out anytime i want.

The question is this, Is you NOC not that sure they can provide power to you 100% of the time? i know network outages happen sometimes but power? with all the generators & UPS's?

I dunno, but i would suggest running softupdate.

[Script Dude]

Soft updates delays writes.

By delaying the writes, the writes can be sequenced so that interrelated data is always written out consistently.

Thus, if the sequence is interrupted, your hard disk is left in better condition than if you had done the writes in the original sequence.

Soft updates basically changes the order that writes occur in. While doing this, it can sometimes combine writes that would have otherwise been done seperately, resulting in a performance increase.

The draw back is that if a crash occurs, the capacity of the hard drive is reduced. (The locations of the most recent disk writes are literaly lost.)

The repair process will find this lost space and return it to the free space, so this dimished capacity is temporary.

The actual data will still be gone, but only the most recently written data.

Everything that remains will be usuable. (from a file system level point of view) This may not be the case without soft updates.

Long down times for disk repair are no fun.

I suspect you will probably be better off with it than without it.

[AbuckOr2]

no softupdates and shit. you will leave so many holes open when you do that man. I hope you know what i'm talking about.

my

[Script Dude]

Originally posted by boneprone
[B]The deal is my out cgi links seem to be laggin on my server with a new trade script. Its a fucking AWSOME script! AWSOME! But i notice it lagging a bit when links out are clicked. Not much but a bit to where someone picky like me notices.
/B]

Also, soft updates effect disk performance. Do you know that disk performance is causing the delays you are seeing?

I have found that most programmers are rarely correct about performance issues.

I am not saying that this is the case here, but performance is complicated and guessing is no substitute for measurement.