WowzaMediaServer

[borked]

What a frikken excellent configurable, modulator, everything to the hilt media server it is



just thought I'd say. Now back to work.

[webair]

It rocks! Which is why webair.com is a wowza partner!

http://www.wowzamedia.com/partners.html

[borked]

deploying standard installs of wowza pro?

you know the standard install (as mentioned in the user guide) is made for development platforms.... each install requires tweaking per server setup...

all the partners fall down on this from what I've heard, leasing their servers with wowza on, only for client to get into tload problems when any serious traffic comes in.

just saying, like you were.

[Senior_Spank]

This thread is full of lies.

Who writes an IO heavy application in Java? Wowza is a horrible product that will run your equipment costs up thousands of dollars above & beyond what is needed if you actually push any traffic to it. (Speaking about VOD)

Or, perhaps WebAir uses that to their advantage to sell more servers?

[webair]

I'm not a big fan of Java myself, but Wowza does a good job at keeping its CPU usage low. The older 1.x version had some issues, but 2.x is a totally different. If you want to complain about CPU usage and java streaming servers, take a look at Red5...

What we've found to be the most common limiting factor for connections/second and what usually requires the most tweaks are low level network buffers or sysctls, which is what you'd hope for.

Wowza is not the only streaming service we offer, it's just another product that we've integrated into our standard offering that can be enabled via a simple on/off command to work with a server's existing configuration.

We also offer FMS, pseudo streaming, and RTMP streaming directly via our CDN. Wowza will also be offered on a per minute basis as part of our upcoming cloud computing initiative.

Carry on...

[Senior_Spank]

Originally Posted by webair

I'm not a big fan of Java myself, but Wowza does a good job at keeping its CPU usage low. The older 1.x version had some issues, but 2.x is a totally different. If you want to complain about CPU usage and java streaming servers, take a look at Red5...

At the end of the day you don't program anything that requires heavy IO (network, disk, etc) in Java. As for the CPU usage comment. See a little dstat snippit of mine below. We are only running Wowza and nothing else. The system is tweaked and all the media is served off dedicated SSD drives (Notice the 0 IOWait for 1.2Gbits of transfer?)

----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
 53  17   9   3   1  17| 162M  192k|   0     0 |  81k   75k|  69k   17k
 67  21   0   0   0  12| 247M    0 |2622k   88M|   0     0 |  55k   13k
 53  24   0   0   1  23| 191M   24k|1651k   97M|   0    12k|  85k   18k
 51  23   0   0   1  25| 194M   36k|3876k  148M|   0    36k|  94k   18k
 54  23   2   0   0  22| 101M   32k|2761k  157M|4096B   32k|  92k   12k
 69  10   0   0   1  20| 146M    0 |3392k  114M|   0     0 |  59k 5654
 61  23   0   0   1  16| 195M   16k|1691k   83M|   0     0 |  65k   18k
 50  22   0   0   1  27| 177M   16k|3287k  129M|   0  4096B|  96k   18k
 50  22   0   0   0  28| 170M    0 |2723k  166M|   0     0 |  96k   20k
 60  15   2   0   0  24|  54M 3180k|4266k  145M|  60k   28k|  76k 7941
 61  20   1   0   3  16| 236M    0 |1962k   94M| 188k    0 |  55k   11k
 58  24   0   0   1  18| 204M    0 |2161k   89M|   0     0 |  77k   19k
 53  17   0   0   1  30| 190M   44k|2239k  136M|   0     0 |  93k   23k
 52  21   0   0   0  27| 172M 4096B|4596k  161M|   0  4096B|  93k   18k
 69  14   1   0   0  16|  19M   28k|2486k  126M|   0    28k|  61k 1374
 59  24   0   0   2  15| 298M    0 |2089k   74M|   0     0 |  57k   17k
 56  23   0   0   1  21| 213M   12k|1661k  103M|   0     0 |  87k   18k
 49  26   0   0   1  24| 179M   12k|4101k  159M|   0     0 |  95k   18k

I'd love to really see Wowza push even more traffic. But the software is at it's technical limits, no amount of OS level tweaks will do anything.

(And this is the latest & greatest build as offered by Wowza, all of our libraries are updated, yadda yadda yadda.)

[react]

Post up some Flash Media Server stats and lets chat.

[DWB]

This stands to be an awesome thread.

Yes guys, please get into any strain it may put on a server or its limitations. I'm all eyes and ears.

[mynameisjim]

I am trying to decide between Wowza and Adobe. I was leaning towards Wowza but lately I'm more towards Adobe.

Definitely be checking back on this thread.

[borked]

senior spank - your stats look really odd - for one, for each usr (how many max connections), you have massive system CPU usage, with lots of idles and waiting connections. Also your disk reads/total compared to amount sent on the network says something is very wrong - for each client you are reading more than sending... dsk read/total should be pretty much 0

can you copy/paste your files:

/usr/local/WowzaMediaServer/bin/startup.sh
/usr/local/WowzaMediaServer/conf/VHost.xml

as well as output from

cat /proc/cpuinfo
and
df -h

or email then (email in sig)

it could be a wms config problem or a bad nfs mount... but they are not right stats for sure

[borked]

Originally Posted by Senior_Spank

The system is tweaked and all the media is served off dedicated SSD drives

but your disk reads per client are still 10x more than what is being sent over the net... something is wrong here

[Senior_Spank]

Originally Posted by borked

senior spank - your stats look really odd - for one, for each usr (how many max connections), you have massive system CPU usage, with lots of idles and waiting connections. Also your disk reads/total compared to amount sent on the network says something is very wrong - for each client you are reading more than sending... dsk read/total should be pretty much 0

Actually you are horribly horribly wrong which makes me wonder about the way you brag in your first post. Forgive me if I go about the rest of my comment with a bit of arrogance, but just because you teach a dog to play dead does not mean you are Cesar Millan.

Like I said; everything on our system has been configured with huge amounts thought and foresight. The reads are that way because of the IOScheduler we use combined with the massive read-aheads we have set. The system has the capacity, why not use it. Also, the fact that there is 0 IO load, it shouldn't matter how much it's reading because it's not the bottleneck.

We do not use NFS mounts, It's { Storage => Wowza => Internet } to keep overhead and possible complications as limited as possible.

Now, the way I see it is I have one solution should I want to keep using this software while keeping up with growth. An NFS, and a bunch of servers with a shitload of RAM and a shitload of CPU power. Put it behind a load balancer (or round-robin the DNS to a series of IPs) and for every gigabit of capacity I want to add, I need an additional $3,000 to $4,000 in hardware overhead. (This of course doesn't include power overhead, additional network ports used on the switch, whatever)

If you like to wipe your ass with $20's, then Wowza is the software for you. For those of you who live in a world with budget's and do not enjoy responding to server alerts at 3 AM (Don't get me started on the fact it doesn't "crash" but just stops serving video files at random.) then I'd recommend skipping Wowza.

On a positive note about WowzaMedia - Their software does work, and their support team actually seems to be very responsive to customers. If you are running a smaller operation and want very good streaming and scrubbing. Then Wowza may be the solution for you. My comments are only speaking to people who plan to be serving multiple gigabits of content.

And lastly, a bit of why I've even had to deal with this headache...

We were able to saturate a 2 gigabit connection with a modest $2,000 server with 4 gigs of RAM and using nginx. The only problem of course, is the MP4 atom at the beginning of the video causing slow load times for long videos. Wowza (and other FMS servers) solves this problem with rtmp.

In our efforts to get this working we have spent over $6,000 in additional equipment. Dual Quad-Core processors, 32GB of RAM, SSD drives, etc.

[Senior_Spank]

Originally Posted by borked

but your disk reads per client are still 10x more than what is being sent over the net... something is wrong here

Ask a Java engineer

[borked]

Originally Posted by Senior_Spank

Ask a Java engineer

are you using the java server or the java client on startup??? big difference

also, are you transcoding your files eg even though wowza can stream .mov files to .mp4 doesn't mean it's optimal....

lots of different problems can arise I'm sure, but my own experience with a hundred or so connections is minimal/zero disk reads and lots of net throughput. a "top" only ever picks up the odd instance here and there, with the load-balanced httpd server taking top slot always. However, wowza, hasnt been opened up to the tube streaming side yet...

Finally, any protections in place to prevent stream ripping/hotlinking? Another source of unnecessary load....

[Spudstr]

Originally Posted by Senior_Spank

This thread is full of lies.

Who writes an IO heavy application in Java? Wowza is a horrible product that will run your equipment costs up thousands of dollars above & beyond what is needed if you actually push any traffic to it. (Speaking about VOD)

Or, perhaps WebAir uses that to their advantage to sell more servers?

hate to tell you but Adobe FMS is written in java too.

[Spudstr]

Originally Posted by Senior_Spank

Actually you are horribly horribly wrong which makes me wonder about the way you brag in your first post. Forgive me if I go about the rest of my comment with a bit of arrogance, but just because you teach a dog to play dead does not mean you are Cesar Millan.

Like I said; everything on our system has been configured with huge amounts thought and foresight. The reads are that way because of the IOScheduler we use combined with the massive read-aheads we have set. The system has the capacity, why not use it. Also, the fact that there is 0 IO load, it shouldn't matter how much it's reading because it's not the bottleneck.

We do not use NFS mounts, It's { Storage => Wowza => Internet } to keep overhead and possible complications as limited as possible.

Now, the way I see it is I have one solution should I want to keep using this software while keeping up with growth. An NFS, and a bunch of servers with a shitload of RAM and a shitload of CPU power. Put it behind a load balancer (or round-robin the DNS to a series of IPs) and for every gigabit of capacity I want to add, I need an additional $3,000 to $4,000 in hardware overhead. (This of course doesn't include power overhead, additional network ports used on the switch, whatever)

If you like to wipe your ass with $20's, then Wowza is the software for you. For those of you who live in a world with budget's and do not enjoy responding to server alerts at 3 AM (Don't get me started on the fact it doesn't "crash" but just stops serving video files at random.) then I'd recommend skipping Wowza.

On a positive note about WowzaMedia - Their software does work, and their support team actually seems to be very responsive to customers. If you are running a smaller operation and want very good streaming and scrubbing. Then Wowza may be the solution for you. My comments are only speaking to people who plan to be serving multiple gigabits of content.

And lastly, a bit of why I've even had to deal with this headache...

We were able to saturate a 2 gigabit connection with a modest $2,000 server with 4 gigs of RAM and using nginx. The only problem of course, is the MP4 atom at the beginning of the video causing slow load times for long videos. Wowza (and other FMS servers) solves this problem with rtmp.

In our efforts to get this working we have spent over $6,000 in additional equipment. Dual Quad-Core processors, 32GB of RAM, SSD drives, etc.

I can give you a hint. We have had atoms engineered "correctly" with only 4G of ram push out 1.5G without any problems. Konrad can contest to our server setups.

We have spent countless hours engineering streaming solutions and have perfected a proven solution. You don't need 32g of ram or dual quads. But a simple Atom server.


Wowza is a great product we have several dozen machines running it. It fixes problems that are left open that lighttpd and nginx would fail at. The biggest one that I can think of? Indexed keyframes in flv files. wowza takes care of it. Lighttpd and nginx use the same backend modules to handle these file types and both have identical issues. Skipping through videos just wont happen. Also with MP4 files that are not correct they will just cause nginx/lighttpd to hang and stall out.

Best advice keep things simple, it works the best.

[Mutt]

how much more expensive is FMS than Wowza?

[Senior_Spank]

Originally Posted by Spudstr

hate to tell you but Adobe FMS is written in java too.

I never said anything about FMS, So why do you hate to tell me about it?

At the end of the day, Java is not capable of handling huge amounts of IO.

As for the rest of your comments, You say you are pushing out 1.5Gbits with Wowza or you are pushing 1.5Gbits in general? Are you trying to bait some potential customers in here with your "mystery" solution?

[lagcam]

This is a fascinating thread that is largely above my head but I would like to ask Senior Spank why he continues to use wowza if he cannot get it to perform as he would like.....is it because he also has a problem with Adobe on his setup and Wowza for all its "limitations" is the best solution he can find?

[Senior_Spank]

Originally Posted by lagcam

This is a fascinating thread that is largely above my head but I would like to ask Senior Spank why he continues to use wowza if he cannot get it to perform as he would like.....is it because he also has a problem with Adobe on his setup and Wowza for all its "limitations" is the best solution he can find?

I work for the customer. He has my recommendations but would like to continue to work with Wowza for as long as possible.

[borked]

Dissing Wowza "because it's Java-based, therefore crap" is childish. You won't show any config options (which I'm sure are well-tuned), what you are serving (live, static), what protocol (rtmpt urgh), video framerate vs WMS and flash client bufferLength setup, system setups, RAID configuration, more dstat outputs (-c -d -g -i -l -m -n -p -s -t -y), system OS, kernel vers, java version nothing.

Javas performace is often as comparable as anything compiled with C. Where it falls short in comparison is on I/O, since C's buffered I/O is filled internally by large file reads (although Java's FileChannel with a wrapped array ByteBuffer isn't too shabby).

[total-memory-mb] = ([stream-bitrate-kbps]/([1024-kb-per-mb]*[8-bits-per-byte])) * [chunkDurationTarget-seconds] * [maxChunkCount]

so if memory isn't a problem, which with 32GB RAM doesn't seem to be, you can assign max RAM to the java server on startup and start to offload disk input and network output by filling up the ram with more chunks.

your problem clearly lies with disk input as those values are huge and as wowza can only deliver as fast as it can get the data, the limiting point seems to be the reading of the files, pushing up cpu cycles. Having WMS sitting on SSD's has no beneficial effect in this case - having your content on SSDs would.

Apache Tomcat (Java) performance is 100% comparable with Apache HTTP (C) when configured correctly for the needs it serves, but can also be 0% crap if badly configured (as can HTTP for that matter). I'd say WMS (or adobe's flavour) is far more suited for this type of media since it is far more configurable that anything else out there in C and given the ever increasing platforms/stream types that need to be served from the same file base, configuring the server for each stream type can only be a good thing.

[Spudstr]

Originally Posted by Senior_Spank

I never said anything about FMS, So why do you hate to tell me about it?

At the end of the day, Java is not capable of handling huge amounts of IO.

As for the rest of your comments, You say you are pushing out 1.5Gbits with Wowza or you are pushing 1.5Gbits in general? Are you trying to bait some potential customers in here with your "mystery" solution?

1.5G off a single box with bonded nics. We push a lot more than that. Yes thats it you are exactly right. I'm fishing for potential customers with our mystery configuration. You are sherlock homes himself.

No these little boxes are not using wowza or fms. But nginx like you use. Our hardware is just.. different.

My point is with the proper configuration you can get a lot more bang for your buck. I'm sure if we swapped it to Wowza these machines might be able to pump out the same.

I wouldn't crack to much on java. Much of oracle itself is done in java now. This was one of the main reasons why oracle bought sun to take oracle into itself and create more of a database appliance to compete with EMC and likes. Since they were missing a hardware "solution". Much like a lot of ERP, JD Edwards/Peoplesoft, SAP software and a likes are all done in java.

Overall though Yes, Wowza creates more disk IO than traditional psudo-streaming methods using lighttpd or nginx. You are completely 100% correct on this statement. They have done things to improve this and its memory caching, and the more memory you have the more it can serve faster due to well.. yes! Caching into memory.

The only advantage wowza had over nginx/lighttpd was its ability to stream at the bit rate level properly. But any admin that knows which way is up can configure nginx/lighttpd to do almost the same. But wowza will win on that level but a trade off vs hardware.

[Spudstr]

Originally Posted by Mutt

how much more expensive is FMS than Wowza?

iirc, adobe fms was 1k. Wowza was that maybe a bit cheaper now. But most people license it per month. Theres another version of FMS thats about 3x the cost of the lower end version. Unless they got rid of it or merged it/whatever. I have not kept up with Adobe FMS in a while though. But they do make one of the best/free live encoding engines for live streams/broadcasting to rtmp servers.

[munki]

Bookmarking this thread to revisit when I'm finished with my current java studies...

[VGeorgie]

Originally Posted by Spudstr

iirc, adobe fms was 1k. Wowza was that maybe a bit cheaper now. But most people license it per month. Theres another version of FMS thats about 3x the cost of the lower end version. Unless they got rid of it or merged it/whatever.

The basic FMS is still at $1K; the interactive FMS with unlimited adaptors and other features is $4.5K. There is also the free development version, of course.