Content-Aware Playback Speed Control for Low-Latency Live Streaming of Sports

ABSTRACT

There are two main factors that determine the viewer experience during the live streaming of sports content: latency and stalls. Latency should be low and stalls should not occur. Yet, these two factors work against each other and it is not trivial to strike the best trade-off between them. One of the best tools we have today to manage this trade-off is the adaptive playback speed control. This tool allows the streaming client to slow down the playback when there is a risk of stalling and increase the playback when there is no risk of stalling but the live latency is higher than desired. While adaptive playback generally works well, the artifacts due to the changes in the playback speed should preferably be unnoticeable to the viewers. However, this mostly depends on the portion of the audio/video content subject to the playback speed change. In this paper, we advance the state-of-the-art by developing a content-aware playback speed control (CAPSC) algorithm and demonstrate a number of examples showing its significance. We make the running code available and provide a demo page hoping that it will be a useful tool for the developers and content providers.

References

1. CAPSC - MMSys’21 Demo Page. [Online] Available: http://streaming.university/demo/mmsys21-capsc/. Accessed on June 1, 2021.
2. ISO/IEC 14496-12:2020 Information technology -- Coding of audio-visual objects -- Part 12: ISO base media file format. [Online] Available: https://www.iso.org/standard/74428.html. Accessed on June 1, 2021.
3. ISO/IEC 23000-19:2020 Information technology -- Multimedia application format (MPEG-A) -- Part 19: Common media application format (CMAF) for segmented media. [Online] Available: https://www.iso.org/standard/79106.html. Accessed on June 1, 2021.
4. ISO/IEC 23001-18 Information technology -- MPEG systems technologies -- Part 18: Event message tracks in the ISO base media file format. Under development. [Online] Available: https://www.iso.org/standard/82529.html. Accessed on June 1, 2021.
5. ISO/IEC 23009-1:202x Information technology -- Dynamic adaptive streaming over HTTP (DASH) -- Part 1: Media presentation description and segment formats. Under development. [Online] Available: https://www.iso.org/standard/83314.html. Accessed on Aug. 1, 2021.
6. I. Abibouraguimane, K. Hagihara, K. Higuchi, Y. Itoh, Y. Sato, T. Hayashida, and M. Sugimoto. CoSummary: adaptive fast-forwarding for surgical videos by detecting collaborative scenes using hand regions and gaze positions. In Int. Conf. Intelligent User Interfaces, 2019.
7. A. C. Begen, M. N. Akcay, A. Bentaleb, and A. Giladi. Adaptive streaming of content-aware-encoded videos in dash.js. In Int. Broadcasting Convention Conf. (IBC), 2021.
8. A. Bentaleb, M. N. Akcay, M. Lim, A. C. Begen, and R. Zimmermann. Catching the moment with LoL+ in Twitch-like low-latency live streaming platforms. IEEE Trans. Multimedia, pages 1--1, 2021
9. K.-Y. Cheng, S.-J. Luo, B.-Y. Chen, and H.-H. Chu. SmartPlayer: user-centric video fast-forwarding. In Conf. Human Factors in Computing Systems, 2009.
10. DASH-IF. DASH Reference Client. [Online] Available: https://reference.dashif.org/dash.js/. Accessed on June 1, 2021.
11. DASH Industry Forum (DASH-IF). Low-Latency Modes for DASH. [Online] Available: https://dashif.org/guidelines/. Accessed on June 1, 2021.
12. K. Durak, M. N. Akcay, Y. K. Erinc, B. Pekel, and A. C. Begen. Evaluating the performance of Apple's low-latency HLS. In IEEE MMSP, 2020
13. M. Kalman, S. T. Eckehard, and B. Girod. Adaptive playout for real-time media streaming. In IEEE ISCAS, 2002.
14. C.-T. Kao, Y.-T. Liu, and A. Hsu. Speeda: Adaptive speed-up for lecture videos. In ACM Symp. User Interface Software and Technology, 2014.
15. W. Kooij. Playout delay of tv broadcasting. [Online] Available: https://essay.utwente.nl/64744/1/Kooij_MA_EEMCS.pdf, 2014. Accessed on June 1, 2021.
16. R. Pantos, Ed. HTTP Live Streaming 2nd Edition. [Online] Available: https://datatracker.ietf.org/doc/draft-pantos-hls-rfc8216bis/. Accessed on June 1, 2021.
17. E. Steinbach, N. Farber, and B. Girod. Adaptive playout for low latency video streaming. In IEEE ICIP, 2001.
18. L. Sun, T. Zong, S. Wang, Y. Liu, and Y. Wang. Tightrope walking in low-latency live streaming: optimal joint adaptation of video rate and playback speed. In ACM MMSys, 2021.
19. T. Friedman, Ed. and R. Caceres, Ed. and A. Clark, Ed. RTP Control Protocol Extended Reports (RTCP XR). [Online] Available: https://datatracker.ietf.org/doc/html/rfc3611. Accessed on June 1, 2021.
20. WHATWG. HTML Living Standard. [Online] Available: https://html.spec.whatwg.org/multipage/web-sockets.html. Accessed on June 1, 2021.
21. WHATWG. HTML Living Standard. [Online] Available: https://html.spec.whatwg.org/multipage/server-sent-events.html. Accessed on June 1, 2021.

Index Terms

• Information systems

  • Information systems applications

    • Multimedia information systems

      • Multimedia streaming

• Networks

  • Network protocols

    • Application layer protocols