WowzaMediaServer
 

What a frikken excellent configurable, modulator, everything to the hilt media server it is

:thumbsup

just thought I'd say. Now back to work.

It rocks! Which is why webair.com is a wowza partner!

http://www.wowzamedia.com/partners.html

deploying standard installs of wowza pro?

you know the standard install (as mentioned in the user guide) is made for development platforms.... each install requires tweaking per server setup...

all the partners fall down on this from what I've heard, leasing their servers with wowza on, only for client to get into tload problems when any serious traffic comes in.

just saying, like you were.

This thread is full of lies.

Who writes an IO heavy application in Java? Wowza is a horrible product that will run your equipment costs up thousands of dollars above & beyond what is needed if you actually push any traffic to it. (Speaking about VOD)

Or, perhaps WebAir uses that to their advantage to sell more servers?

I'm not a big fan of Java myself, but Wowza does a good job at keeping its CPU usage low. The older 1.x version had some issues, but 2.x is a totally different. If you want to complain about CPU usage and java streaming servers, take a look at Red5...

What we've found to be the most common limiting factor for connections/second and what usually requires the most tweaks are low level network buffers or sysctls, which is what you'd hope for.

Wowza is not the only streaming service we offer, it's just another product that we've integrated into our standard offering that can be enabled via a simple on/off command to work with a server's existing configuration.

We also offer FMS, pseudo streaming, and RTMP streaming directly via our CDN. Wowza will also be offered on a per minute basis as part of our upcoming cloud computing initiative.

Carry on...

At the end of the day you don't program anything that requires heavy IO (network, disk, etc) in Java. As for the CPU usage comment. See a little dstat snippit of mine below. We are only running Wowza and nothing else. The system is tweaked and all the media is served off dedicated SSD drives (Notice the 0 IOWait for 1.2Gbits of transfer?)

I'd love to really see Wowza push even more traffic. But the software is at it's technical limits, no amount of OS level tweaks will do anything.

(And this is the latest & greatest build as offered by Wowza, all of our libraries are updated, yadda yadda yadda.)

Post up some Flash Media Server stats and lets chat.

This stands to be an awesome thread.

Yes guys, please get into any strain it may put on a server or its limitations. I'm all eyes and ears.

I am trying to decide between Wowza and Adobe. I was leaning towards Wowza but lately I'm more towards Adobe.

Definitely be checking back on this thread.

senior spank - your stats look really odd - for one, for each usr (how many max connections), you have massive system CPU usage, with lots of idles and waiting connections. Also your disk reads/total compared to amount sent on the network says something is very wrong - for each client you are reading more than sending... dsk read/total should be pretty much 0

can you copy/paste your files:

/usr/local/WowzaMediaServer/bin/startup.sh
/usr/local/WowzaMediaServer/conf/VHost.xml

as well as output from

cat /proc/cpuinfo
and
df -h

or email then (email in sig)

it could be a wms config problem or a bad nfs mount... but they are not right stats for sure

but your disk reads per client are still 10x more than what is being sent over the net... something is wrong here

Actually you are horribly horribly wrong which makes me wonder about the way you brag in your first post. Forgive me if I go about the rest of my comment with a bit of arrogance, but just because you teach a dog to play dead does not mean you are Cesar Millan.

Like I said; everything on our system has been configured with huge amounts thought and foresight. The reads are that way because of the IOScheduler we use combined with the massive read-aheads we have set. The system has the capacity, why not use it. Also, the fact that there is 0 IO load, it shouldn't matter how much it's reading because it's not the bottleneck.

We do not use NFS mounts, It's { Storage => Wowza => Internet } to keep overhead and possible complications as limited as possible.

Now, the way I see it is I have one solution should I want to keep using this software while keeping up with growth. An NFS, and a bunch of servers with a shitload of RAM and a shitload of CPU power. Put it behind a load balancer (or round-robin the DNS to a series of IPs) and for every gigabit of capacity I want to add, I need an additional $3,000 to $4,000 in hardware overhead. (This of course doesn't include power overhead, additional network ports used on the switch, whatever)

If you like to wipe your ass with $20's, then Wowza is the software for you. For those of you who live in a world with budget's and do not enjoy responding to server alerts at 3 AM (Don't get me started on the fact it doesn't "crash" but just stops serving video files at random.) then I'd recommend skipping Wowza.

On a positive note about WowzaMedia - Their software does work, and their support team actually seems to be very responsive to customers. If you are running a smaller operation and want very good streaming and scrubbing. Then Wowza may be the solution for you. My comments are only speaking to people who plan to be serving multiple gigabits of content.

And lastly, a bit of why I've even had to deal with this headache...

We were able to saturate a 2 gigabit connection with a modest $2,000 server with 4 gigs of RAM and using nginx. The only problem of course, is the MP4 atom at the beginning of the video causing slow load times for long videos. Wowza (and other FMS servers) solves this problem with rtmp.

In our efforts to get this working we have spent over $6,000 in additional equipment. Dual Quad-Core processors, 32GB of RAM, SSD drives, etc.

Ask a Java engineer :)

are you using the java server or the java client on startup??? big difference

also, are you transcoding your files eg even though wowza can stream .mov files to .mp4 doesn't mean it's optimal....

lots of different problems can arise I'm sure, but my own experience with a hundred or so connections is minimal/zero disk reads and lots of net throughput. a "top" only ever picks up the odd instance here and there, with the load-balanced httpd server taking top slot always. However, wowza, hasnt been opened up to the tube streaming side yet...

Finally, any protections in place to prevent stream ripping/hotlinking? Another source of unnecessary load....

hate to tell you but Adobe FMS is written in java too.

I can give you a hint. We have had atoms engineered "correctly" with only 4G of ram push out 1.5G without any problems. Konrad can contest to our server setups.

We have spent countless hours engineering streaming solutions and have perfected a proven solution. You don't need 32g of ram or dual quads. But a simple Atom server.


Wowza is a great product we have several dozen machines running it. It fixes problems that are left open that lighttpd and nginx would fail at. The biggest one that I can think of? Indexed keyframes in flv files. wowza takes care of it. Lighttpd and nginx use the same backend modules to handle these file types and both have identical issues. Skipping through videos just wont happen. Also with MP4 files that are not correct they will just cause nginx/lighttpd to hang and stall out.

Best advice keep things simple, it works the best.

how much more expensive is FMS than Wowza?

I never said anything about FMS, So why do you hate to tell me about it?

At the end of the day, Java is not capable of handling huge amounts of IO.

As for the rest of your comments, You say you are pushing out 1.5Gbits with Wowza or you are pushing 1.5Gbits in general? Are you trying to bait some potential customers in here with your "mystery" solution?

This is a fascinating thread that is largely above my head but I would like to ask Senior Spank why he continues to use wowza if he cannot get it to perform as he would like.....is it because he also has a problem with Adobe on his setup and Wowza for all its "limitations" is the best solution he can find?

I work for the customer. He has my recommendations but would like to continue to work with Wowza for as long as possible.

Dissing Wowza "because it's Java-based, therefore crap" is childish. You won't show any config options (which I'm sure are well-tuned), what you are serving (live, static), what protocol (rtmpt urgh), video framerate vs WMS and flash client bufferLength setup, system setups, RAID configuration, more dstat outputs (-c -d -g -i -l -m -n -p -s -t -y), system OS, kernel vers, java version nothing.

Javas performace is often as comparable as anything compiled with C. Where it falls short in comparison is on I/O, since C's buffered I/O is filled internally by large file reads (although Java's FileChannel with a wrapped array ByteBuffer isn't too shabby).

[total-memory-mb] = ([stream-bitrate-kbps]/([1024-kb-per-mb]*[8-bits-per-byte])) * [chunkDurationTarget-seconds] * [maxChunkCount]

so if memory isn't a problem, which with 32GB RAM doesn't seem to be, you can assign max RAM to the java server on startup and start to offload disk input and network output by filling up the ram with more chunks.

your problem clearly lies with disk input as those values are huge and as wowza can only deliver as fast as it can get the data, the limiting point seems to be the reading of the files, pushing up cpu cycles. Having WMS sitting on SSD's has no beneficial effect in this case - having your content on SSDs would.

Apache Tomcat (Java) performance is 100% comparable with Apache HTTP (C) when configured correctly for the needs it serves, but can also be 0% crap if badly configured (as can HTTP for that matter). I'd say WMS (or adobe's flavour) is far more suited for this type of media since it is far more configurable that anything else out there in C and given the ever increasing platforms/stream types that need to be served from the same file base, configuring the server for each stream type can only be a good thing.

1.5G off a single box with bonded nics. We push a lot more than that. Yes thats it you are exactly right. I'm fishing for potential customers with our mystery configuration. You are sherlock homes himself.

No these little boxes are not using wowza or fms. But nginx like you use. Our hardware is just.. different.

My point is with the proper configuration you can get a lot more bang for your buck. I'm sure if we swapped it to Wowza these machines might be able to pump out the same.

I wouldn't crack to much on java. Much of oracle itself is done in java now. This was one of the main reasons why oracle bought sun to take oracle into itself and create more of a database appliance to compete with EMC and likes. Since they were missing a hardware "solution". Much like a lot of ERP, JD Edwards/Peoplesoft, SAP software and a likes are all done in java.

Overall though Yes, Wowza creates more disk IO than traditional psudo-streaming methods using lighttpd or nginx. You are completely 100% correct on this statement. They have done things to improve this and its memory caching, and the more memory you have the more it can serve faster due to well.. yes! Caching into memory.

The only advantage wowza had over nginx/lighttpd was its ability to stream at the bit rate level properly. But any admin that knows which way is up can configure nginx/lighttpd to do almost the same. But wowza will win on that level but a trade off vs hardware.

iirc, adobe fms was 1k. Wowza was that maybe a bit cheaper now. But most people license it per month. Theres another version of FMS thats about 3x the cost of the lower end version. Unless they got rid of it or merged it/whatever. I have not kept up with Adobe FMS in a while though. But they do make one of the best/free live encoding engines for live streams/broadcasting to rtmp servers.

Bookmarking this thread to revisit when I'm finished with my current java studies...

The basic FMS is still at $1K; the interactive FMS with unlimited adaptors and other features is $4.5K. There is also the free development version, of course.