Background
91.3 FM WYEP, an independent AAA format radio station based in Pittsburgh Pennsylvania, needed to consider updating our streaming methods.
Urgent Need
We were getting reports from listeners that the stream was cutting out, and it was sometimes not working at all.
We were running on a relatively inexpensive ($280 / mo) service from a company called Streamguys.
This provided up to 450 concurrent listeners an Icecast based stream.
We had access to a rudimentary log analyzer that generated relatively opaque reports. Unfortunately we had no monitoring system in place to let us know if we were in danger of reaching out limit, or over it.
We simply did not have the tools to monitor and react to the streaming needs of our listeners in real time.
Fresh Eyes
I had no experience in broadcasting before I joined WYEP in December, 2015.
My background is in product management, web development, and content management. I've worked in software development and for scrappy budget constrained startups.
So I asked: can't we serve our own stream?
I had (and have) a lot to learn. And since I work for a non profit, I am happy to be able to share our methods and results!
Technology Stack
Summary:
The Barix Instreamer converts the audio from our mixing board to a 320kbps CBR Shoutcast stream. We already had this lying around. This could easily be swapped with a cheap computer running ffserver.
We chose to use Ubuntu 14.04 LTS to install Wowza and ffserver on. It's fast, secure and stable, and LTS stands for Long Term Service. It's also a tool I'm familiar with and can easily find documentation to make it do what I want. All of this could be done with Windows if you wanted.
The interesting bits
Wowza is a great piece of software. It has its quirks, but overall it does a fantastic job. The customer support is fantastic, and there is seamless integration with Wowza Cloud. As long as you have your firewall set properly... but more on that in another post.
The main thing Wowza gives us is the ability to stream in HLS. Not only does HLS sound better at 128kbps than our prior stream, it also allows us to take advantage of a sexy p2p service I came across, Peer5.
Peer5 can enable us to provide an increase of 80% or more service to listeners by using P2P. It's like Bittorrent for streaming. Here's a typical dashboard snapshot for one day:
The blue bars in the top left indicate bandwidth that is being served by P2P instead of direct or CDN.
With all of these fancy new technologies, we still had holes in our service. During my research I discovered that the BBC went through a similar transition about a year ago. Change is disruptive, and can make consumers angry. What could we learn from the BBC? What could we improve on?
Can't hear us? We are listening.
We have decided to adopt a "No Listener Left Behind" policy. What that means in practical terms is: we will do everything we can to support our listeners ability to tune in.
Which brings us to ffserver (part of ffmpeg). This open source tool has been around for years, and is a Swiss army knife for audio and visual formats. It's fast, tried, and true. We are using this to serve up mp3 and Windows Media Player compatible streams, among others. This helps us cover the bases for players not supported by Wowza.
Now let's talk about scaling. Wowza Cloud offered to waive the $35 monthly fee for 12 months (a bargain anyway), which comes with 1 free terabyte of streaming. After that the services offered through their partnership with Akamai provide very affordable and rock solid scaling capability. Adding this to our mix was a no-brainer.
Practical Application
After making the switch, we are serving a peak of over 300 concurrent listeners(as of this writing, and growing). About 250 of those are consuming the HLS stream, and about half of those are enhanced by P2P through Peer5.
For listeners who come in through our website, we are able to scale with Wowza Cloud to whatever Akamai can handle. That's big. We are also using Wowza Cloud to serve streams to listeners behind restrictive firewalls. There are more elegant solutions, but this works for us right now, and means I can focus on our more pressing digital needs. We currently use less than half of our free 1TB per month.
Disruption
Sadly, some listeners have not been able to tune in for a few days. They are on smart TVs or internet radios, hardware that gets our stream URL from a service like Tunein or Vcast. We are updating as many of these as quickly as we can, but the process has not gone as quickly as we would like. Still, it's a small number of listeners who are affected by this. Every week I get about 2 "your stream stopped working since the update" emails. I promptly chase down the service that has our outdated stream info and send them the latest.
Reports
We need to find a solution for consolidating reports. We now have Wowza Cloud, and two servers in our VM. Peer5 also has a dashboard. I have to go to 5 different places to get a picture of our streaming metrics (if you add in Google Analytics).
Results... so far
91.3 FM WYEP, an independent AAA format radio station based in Pittsburgh Pennsylvania, needed to consider updating our streaming methods.
Urgent Need
We were getting reports from listeners that the stream was cutting out, and it was sometimes not working at all.
We were running on a relatively inexpensive ($280 / mo) service from a company called Streamguys.
This provided up to 450 concurrent listeners an Icecast based stream.
We had access to a rudimentary log analyzer that generated relatively opaque reports. Unfortunately we had no monitoring system in place to let us know if we were in danger of reaching out limit, or over it.
We simply did not have the tools to monitor and react to the streaming needs of our listeners in real time.
Fresh Eyes
I had no experience in broadcasting before I joined WYEP in December, 2015.
My background is in product management, web development, and content management. I've worked in software development and for scrappy budget constrained startups.
So I asked: can't we serve our own stream?
I had (and have) a lot to learn. And since I work for a non profit, I am happy to be able to share our methods and results!
Technology Stack
Summary:
- 75mbps FiOS
- Barix Instreamer
- Ubuntu 14.04 (running in a virtual machine)
- Wowza Engine
- Wowza Cloud
- ffserver
- Peer5
The Barix Instreamer converts the audio from our mixing board to a 320kbps CBR Shoutcast stream. We already had this lying around. This could easily be swapped with a cheap computer running ffserver.
We chose to use Ubuntu 14.04 LTS to install Wowza and ffserver on. It's fast, secure and stable, and LTS stands for Long Term Service. It's also a tool I'm familiar with and can easily find documentation to make it do what I want. All of this could be done with Windows if you wanted.
The interesting bits
Wowza is a great piece of software. It has its quirks, but overall it does a fantastic job. The customer support is fantastic, and there is seamless integration with Wowza Cloud. As long as you have your firewall set properly... but more on that in another post.
The main thing Wowza gives us is the ability to stream in HLS. Not only does HLS sound better at 128kbps than our prior stream, it also allows us to take advantage of a sexy p2p service I came across, Peer5.
Peer5 can enable us to provide an increase of 80% or more service to listeners by using P2P. It's like Bittorrent for streaming. Here's a typical dashboard snapshot for one day:
The blue bars in the top left indicate bandwidth that is being served by P2P instead of direct or CDN.
With all of these fancy new technologies, we still had holes in our service. During my research I discovered that the BBC went through a similar transition about a year ago. Change is disruptive, and can make consumers angry. What could we learn from the BBC? What could we improve on?
Can't hear us? We are listening.
We have decided to adopt a "No Listener Left Behind" policy. What that means in practical terms is: we will do everything we can to support our listeners ability to tune in.
Which brings us to ffserver (part of ffmpeg). This open source tool has been around for years, and is a Swiss army knife for audio and visual formats. It's fast, tried, and true. We are using this to serve up mp3 and Windows Media Player compatible streams, among others. This helps us cover the bases for players not supported by Wowza.
Now let's talk about scaling. Wowza Cloud offered to waive the $35 monthly fee for 12 months (a bargain anyway), which comes with 1 free terabyte of streaming. After that the services offered through their partnership with Akamai provide very affordable and rock solid scaling capability. Adding this to our mix was a no-brainer.
Practical Application
After making the switch, we are serving a peak of over 300 concurrent listeners(as of this writing, and growing). About 250 of those are consuming the HLS stream, and about half of those are enhanced by P2P through Peer5.
For listeners who come in through our website, we are able to scale with Wowza Cloud to whatever Akamai can handle. That's big. We are also using Wowza Cloud to serve streams to listeners behind restrictive firewalls. There are more elegant solutions, but this works for us right now, and means I can focus on our more pressing digital needs. We currently use less than half of our free 1TB per month.
Disruption
Sadly, some listeners have not been able to tune in for a few days. They are on smart TVs or internet radios, hardware that gets our stream URL from a service like Tunein or Vcast. We are updating as many of these as quickly as we can, but the process has not gone as quickly as we would like. Still, it's a small number of listeners who are affected by this. Every week I get about 2 "your stream stopped working since the update" emails. I promptly chase down the service that has our outdated stream info and send them the latest.
Reports
We need to find a solution for consolidating reports. We now have Wowza Cloud, and two servers in our VM. Peer5 also has a dashboard. I have to go to 5 different places to get a picture of our streaming metrics (if you add in Google Analytics).
Results... so far
- increased insight in real time
- greater flexibility
- higher quality audio
- ability to instantly scale on demand
- reliability
- reduced costs
Great read Antonio. Any concerns about redundancy? What about Sound Exchange reporting? Are you able to pull a report our of wowza to file your quarterly SX report?
ReplyDeleteThanks Charles! The report we generated for SX was very basic. I am going to have to merge logs from ffserver and Wowza, and then parse those for the final report. That is undoubtedly the subject of a future post.
ReplyDelete