boneprone

Ive been having a political debate over the past 12 hours with my own programers and staff over wheather I should have "soft updates" activated on my server?

The discussion and arguent has been between my head programer, my script programer, and my Noc-Server Programer.

The deal is my out cgi links seem to be laggin on my server with a new trade script. Its a fucking AWSOME script! AWSOME! But i notice it lagging a bit when links out are clicked. Not much but a bit to where someone picky like me notices.

They reconmned I do a softupdate.
However the stabilty of this has my NOC programer concerned.
I guess if shit crahes all hell breaks loose ,a nd i can loose data and even corupt other files.

My main progrmer is like "what the fuck are soft updates" and like me are trying to figure out what the hell to do?

Does anyone here have softupdates on or off and why?

Like i said, the out links are a bit slow, but really not that bad!
But, im told if i do this soft update it will be Fast! Which really turns me on.

Anyone know anything about this shit? and have an opinon?
It seems to be split 50-50 between my people.

Anyhow Im enabling the ost update for the next 24 hours to see how it goes and decide from there.

__________________

Industry Hall Of Fame Legend Mike Jones
Bow to the Power - Still BP4L
http://gfyawards.com/hall-of-fame
Learn about it kids.

salsbury

softupdates is safe. safer than not using it.

basically, softupdates guarantees that the data written to your disk is exactly what it should be. it'll write the data to the disk and only after that it'll update the "metadata" to reflect that new data has been written. if the power goes out after data is written but the metadata isn't updated (a very, very quick update, btw) then that data is basically "discarded" because it's not part of any file.

without softupdates, the data is written basically sequentially, and it's possible that the data could be written half-way when the power goes out.

it also improves disk performance, to varying degrees.

it's only available for FreeBSD, btw.

__________________

boneprone

from what i understand it is just the opposite?

__________________

Industry Hall Of Fame Legend Mike Jones
Bow to the Power - Still BP4L
http://gfyawards.com/hall-of-fame
Learn about it kids.

boneprone

that softupdates on is more efficient in its writing, but performance is greater becasue it meta stores the info and writes it peridiocally.

So if you have an outage inbetween this meta and writing you loose everything.
And can also run the possibility of corrupting other files.

There seems to be 2 very strong arguments over this.

__________________

Industry Hall Of Fame Legend Mike Jones
Bow to the Power - Still BP4L
http://gfyawards.com/hall-of-fame
Learn about it kids.

salsbury

there are basically 3 ways to use disks. the normal way, asynchronous, and w/ soft-updates.

asynchronous might be the one you're thinking of - that one is not safe but is very fast. there's no guarantee that the data will be written to the disk, ever, with that.

__________________

boneprone

here is one argument against.


Noc@ (3:33 PM) :
what soft updates does is basically every time you edit a file, add a file, move a file, or do any kind of operation that would require a harddrive to write data, it forces the harddrive not to write that data

Noc(3:33 PM) :
Instead it stores it as meta data, and then when enough is stored it writes it all out at once

Noc@(3:33 PM) :
This makes opening,closing,unpacking files quicker, because you are decreasing the amount of times you write to the disk

Noc@(3:34 PM) :
Its like a buffer

Noc@(3:34 PM) :
So first edits goto the buffer, then whent he buffer has enough it writes to the disk

Noc@3:34 PM) :
If you have a power failure, crash, or harddrive crash, then this meta data is lost

Noc@ (3:35 PM) :
This can mean anything from losing files that you updated in the last hour, to losing directories if they were moved but had not yet been written, in rare cases even partitions can be come corrupted

Boneprone (3:35 PM) :
ahh so the potential we can loose on a crash is just that date in between the time of writing?


Noc@:
no, you can essentially lose everything, because meta data when it gets messed up it may force errors on neighboring files

__________________

Industry Hall Of Fame Legend Mike Jones
Bow to the Power - Still BP4L
http://gfyawards.com/hall-of-fame
Learn about it kids.

salsbury

you'd only lose the data that was not yet completely written to the disk. given that it wasn't complete, you don't even know if the data was correct yet. it's better to just "pretend" that data was never written than to have to worry if it was corrupted.

you certainly don't lose everything.

i've never heard of this, except perhaps very early on when softupdates was being introduced. it's very stable now. i've never seen a single problem with it.

it's to the point that FreeBSD 5.0 will have background fsck'ing - filesystem checking - because regular fsck'ing will no longer be necessary.

__________________

salsbury

it's more like 15-60 seconds, not an hour.

it really sounds like Noc is thinking of async rather than softupdates.

i'll bow out and let other people comment now.

__________________

boneprone

thanks man, this helps

__________________

Industry Hall Of Fame Legend Mike Jones
Bow to the Power - Still BP4L
http://gfyawards.com/hall-of-fame
Learn about it kids.

boneprone

And its just not this one NOC who is saying this.
Its several.


And I also have several programers tellling me to do the softupdate.

Like i said its a political debate in the family.

__________________

Industry Hall Of Fame Legend Mike Jones
Bow to the Power - Still BP4L
http://gfyawards.com/hall-of-fame
Learn about it kids.

frankfortuna

Here are some rules to live by:

1. Use a journaling filesystem.
2. Spindles, spindles, spindles. If your concerned about disk performance, stripe your data across multiple spindles.
3. Mirror. Mirror your data / stripes. 1+0 or 0+1 schemes are recommended depending on the amount / type of data you're talking about. RAID-5 schemes are not an effective option in my opionion.
4. Stop fucking around with Linux / FreeBSD solutions and get yourself a real OS. You'll do a better mitigating risk and managing performance using Solaris with Veritas Volume Manager and Veritas File System (VxVM and VxFS respectively).

boneprone

Solaris ?????????
WTF?

__________________

Industry Hall Of Fame Legend Mike Jones
Bow to the Power - Still BP4L
http://gfyawards.com/hall-of-fame
Learn about it kids.

frankfortuna

Yes Solaris. But that issue aside, it sound like you need a better way of managing your strorage. Trust me, if you evaluate the I/O of the application in question and create stripe set(s) with a width that aligns with the I/O operations being performed, you'll notice a dramatic increase in performance.

If I may ask, what is the configuration of the box in question? Number of I/O controllers, type, and # of disks?

boneprone

I have no fucking idea!
I barely know html man.

Im like George Bush, I have no fuckign idea whats going on, thats why i surround myself with people in the know.

I actually dont even know what you are asking?
But i can tell you when to double down, split, and hit.

Just give me a yes or no on the softupdate idea, and why.

__________________

Industry Hall Of Fame Legend Mike Jones
Bow to the Power - Still BP4L
http://gfyawards.com/hall-of-fame
Learn about it kids.

vending_machine

hahahahaha
(not gonna dignify that a real answer)

frankfortuna

You laugh, but I doubt you know the difference between a Unix kernel and Colonel Sanders.

vending_machine

edit; never mind.

frankfortuna

To answer your questions, soft updates can be a good way of improving small I/O performance, but there are some risks to be assumed. Mainly, if you have a panic or power loss before the syncer can commit the data to disk, you're hosed.

salsbury

this is the case with regular filesystems. softupdates makes it so you're not "hosed". rather, your file is left "untouched" if the system loses power during a write.

the "syncer"'s last step is to mark that the data is complete after it is all written to the disk.

and yeah i said i'd bow out, but ya gotta fight FUD as you see it.

__________________

vending_machine

Ok, while you're laughing at me, since apparantly I'm a comedian, answer me this:

How much would a system like that cost? $20+k at the very least?


How would it compare to buying two pIII 1Ghz w/ FreeBSD and then put them in a round robin with failover. This is the adult internet, not NASA. Who cares if Joe Blo clicks on a link and the one round robin server crashes causing one click to go amiss.

frankfortuna

You're absolutely correct, I confused this somewhat with pure write buffering. I missed a few hours of sleep yesterday and spoke out of turn.

My only other advice with soft updates is to watch memory utilization if this is already an existing concern.

In summary, I would recommend soft updates if we're talking about hundreds of short-lived, small I/O operations such as temporary files.

frankfortuna

Actually it's not as much as you'd think. Netra T1's are quite cost effective and plentiful on the used market for around $1500 or less. If you are concerned about the availability of web services and load balancing, Sun Cluster 3.0 does an excellent job at $2500 list per system.

In summary, you can do more for less with Sun systems in my opinion, hence, an overall cost reduction is achieved. And for those who aren't comfortable with the notion, my suggestions can be applied to Linix and BSD.

boneprone

so frank, you say no?

__________________

Industry Hall Of Fame Legend Mike Jones
Bow to the Power - Still BP4L
http://gfyawards.com/hall-of-fame
Learn about it kids.

frankfortuna

I say go for it. It can be implemented in a matter of minutes, however, you will need to completely unmount / remount your filesystems, which will create a brief outage.

One last question, is this script using mySQL by chance?

frankfortuna

Hey, I'm going to step out for a bit. I sent you an email with contact details. Do not hesistate to email me if you have additional questions / concerns.

boneprone

mySQL by chance?

No.

__________________

Industry Hall Of Fame Legend Mike Jones
Bow to the Power - Still BP4L
http://gfyawards.com/hall-of-fame
Learn about it kids.

notjoe

I run softupdates on all my freebsd servers, i can do shit with them and pull the plug out anytime i want.

The question is this, Is you NOC not that sure they can provide power to you 100% of the time? i know network outages happen sometimes but power? with all the generators & UPS's?

I dunno, but i would suggest running softupdate.

Script Dude

Soft updates delays writes.

By delaying the writes, the writes can be sequenced so that interrelated data is always written out consistently.

Thus, if the sequence is interrupted, your hard disk is left in better condition than if you had done the writes in the original sequence.

Soft updates basically changes the order that writes occur in. While doing this, it can sometimes combine writes that would have otherwise been done seperately, resulting in a performance increase.

The draw back is that if a crash occurs, the capacity of the hard drive is reduced. (The locations of the most recent disk writes are literaly lost.)

The repair process will find this lost space and return it to the free space, so this dimished capacity is temporary.

The actual data will still be gone, but only the most recently written data.

Everything that remains will be usuable. (from a file system level point of view) This may not be the case without soft updates.

Long down times for disk repair are no fun.

I suspect you will probably be better off with it than without it.

AbuckOr2

no softupdates and shit. you will leave so many holes open when you do that man. I hope you know what i'm talking about.

my

Script Dude

Also, soft updates effect disk performance. Do you know that disk performance is causing the delays you are seeing?

I have found that most programmers are rarely correct about performance issues.

I am not saying that this is the case here, but performance is complicated and guessing is no substitute for measurement.