SonyLiv mobile application serving in India

With the advance of internet technology, we could see more and more use cases of serving the video content over-the-top. Live events, like football matches, Olympic games, award ceremonies and music concerts, are the most valuable shows among all forms of video contents. Services providers are willing to put huge budget bidding the regional exclusive streaming rights and try to make a profit through advertising, subscription and even the virtual e-ticket (pay-per-view for the live events). Recently, we got a big football event, the Euro 2016.

The Euro 2016 incidents on LeTV Hong Kong

At the early days of the Euro 2016 events, many subscribers flock to LeTV Hong Kong’s facebook page and complaint. Many angry subscribers claimed that they could not access to the live streaming at all. They missed their favorite team matches and yelled for refunds. Representative from LeTV initially claimed that there were some technical issues brought about by UEFA and therefore their service was affected. That provoked the angers of the audience as it was obvious that other service provider (e.g. now TV in Hong Kong and, our client, SonyLiv in India) are not affected at all. LeTV eventually admitted the fault and committed refunds for affected subscribers.

Audience’s complains on LeTV’s facebook page on the first day of Euro 2016 match

Live Events OTT Challenges

Serving Live events over the internet environment is challenging. Unlike cable and IPTV which dedicated channels would be served as the medium to transmitting the information from one single point to many end-points in one direction as a way of broadcasting/multicast. OTT live streaming would require the establishment of multiple connections and the server loading hugely piled up when users access to the service at the same time.

The front-end interface (e.g. mobile apps / web pages) would have multiple dependencies on different services (e.g. analytics, video streaming, 3rd party libraries, caching, middleware and other plug-in). When the access to the front-end shoot up at a short time, a little items on the critical path may fail and pull down the whole service. A 404 error when retrieving a tiny library on remote domain could get the whole service down. It is very important to identify the weakest link(s) before we get the services to the market.

StarHub Go in Singapore

Lessons learnt regarding Live Events OTT

Although there seems to be lots of OTT live event fail cases in the market, I didn’t see rolling out a Live Event OTT service is a risky bet and there are some preventive measures to ensure a smooth and successful launch.

1. System Architecture and Fall-back Mechanisms

   Some service providers used to be doing typical OTT services (e.g. catch-up, video on-demand) and the system architecture and the backend infrastructure are not designed for the high amount of concurrent users.After winning the bid of the exclusive right, service providers shouldn’t expect the existing infrastructure could always be capable of handling the big audience for live events. Investigate of the system architecture, setting up load balancer, multiple layers of fall-back sites and back-up server should be considered accordingly.

    

2. Performance Test

   We should have some fire drill before we are actually getting on fire! In order to find out the bottlenecks, weaknesses and potential consequences of failure. Performance tests, with multiple amount of concurrent users simulating the various activities on the services, should be use frequently while preparing the service for Live Events OTT. There are lots of tools available on the market and JMeter is the one that is open source and could adopt to different use cases. There are some service providers that could provide the scalable testing and performance report.After multiple rounds of testing, you should be able to tackle the problems spotted before the site is getting to serve the public audience.

    

3. “Live” supports

   After you have revamped the system structure and get assurance by the performance test, you should be “fine” for facing all the known issues with huge traffic. However, there could always be some “black swans” that could break your services. The cost of the down time during live events are so high as the advertisement, audience are all coming for that precise hours. Angry customers will definitely file complains when they miss the only goal for the whole football match. Even the live events would usually be held at night outside office hour, it is still worthwhile to assign technical staff to stand-by and provide any necessary support to the service. Instant remedy actions could be taken if there is any unexpected issue. (And, from developer’s point of view, who doesn’t want to get extra pay to “watch” the Euro matches?)