Real-time Visual Recap and Game-flow Visualization
Sports Visualization, Visual Summary, Game-flow Visualization.
With a focus on online streaming sites, in this work, we have developed an intuitive real-time visual summary to recap the key events in a match. Using state-of-the-art computer vision algorithms, we visualize game-flow to enable a tactical understanding of the match. We develop an approach that we intend to extend to other non-visual modalities in the future.
Despite the growing popularity of online streaming for sports content, these platforms are not user-friendly. The content lacks accessibility and personalization. For example, when a viewer tunes in late to a broadcast, they are dependent on the producers of the TV feed for a quick recap of the action they have missed. While most online streaming websites include live text about the action, it is cumbersome to scroll through the text for a quick recap. Hence, a fast and intuitive real-time summary of the action would improve viewer experience. Soccer is one of the most watched sports in the world. While a variety of statistics are used to summarize the match, and quantify which team dominated. Statistics seldom paint a picture of how the game panned out. They do not show if the game was end-to-end or mainly played in one half. In other words, the statistics do not capture game-flow. In this work, we provide a fast and intuitive real-time visual summary of the action for the benefit of viewers who tune-in late to a broadcast. We also developed a visualization for game-flow which could also provide tactical insights into the match.
Real-time visual recap
As mentioned, a real-time recap of the action would improve the viewer experience. Fig 1 shows a real-time visual summary recapping a few key events in the match. A separate timeline is indicated for each half. The red/blue color indicates a goal scored by the corresponding team. The length of the color bar indicates the duration of the broadcast corresponding to the goal. On clicking the colored regions, the video snippet of the goal can be seen. While not indicated in Fig 1, it is trivial to extend the timeline to include yellow/red cards, disallowed goals, injuries or any other key events in the match based on the viewer preferences. This intuitive visualization helps the viewer quickly recap the key events while not disturbing the live video feed. After the match, the viewers can use the same visual timeline summary to quickly catch up on the key events of the match.
Figure 1: The timeline prototype for real-time visual recap of the action is shown. The colored regions indicate the goals. On clicking the colored regions, the corresponding video can be seen.
Visualization of game flow
The statistics such as possession and ‘shots on target’ try to quantify the dominance of a team and paint a picture of how the game panned out. Sometimes, a team can pad up the statistics and hence, it can be misleading. A robust method is necessary to visualize the game-flow, i.e., if the match was end-to-end or was it mainly played in one half? This kind of approach for storyboarding in sports has been previously explored for soccer  and snooker . Unlike these previous works, we use non-invasive state-of-art computer vision algorithms that can track the ball  and players  accurately. Using the position of the ball along the length of the field, a line chart of the ball location can be visualized as shown in Fig 2. The visualization provides an intuition about game-flow. In Fig 2, it is observed that as the game started, one team dominated and scored immediately. When this visualization is extended to a full game, as shown in Fig. 3, it paints a picture of how the game panned out. In Fig 3(a), the red team won 2-0 while in Fig 3(b), the blue team won 3-2. Though the first game finished 2-0, it was end-to-end and both teams were attacking. In contrast, after the flurry of goals, the second game petered out. Looking only at the statistics of the game, a likely assumption is that the 3-2 game would have been more end-to-end. Thus, through this visualization, not only do we capture and understand game-flow, we also capture ironies that statistics would not reflect.
Figure 2: The figure shows the game-flow visualization using the ball location along the length of the pitch. The red color indicates the few frames corresponding to a goal.
Figure 3: The visualization of game-flow for two games is shown. In Fig (a), the game was end-to-end and the red team scored two late goals to win 2-0. In Fig (b), after a flurry of early goals, the game petered out and the blue team won 3-2. The statistics would not capture the irony that the 2-0 game was more end-to-end than a 3-2 game.
The ball line chart visualization shown in Fig 3, can be easily extended to player positions as well. These line charts can be used to tactically analyze the play of a team, such as the role of various players. In Fig 4(a), it is observed that the player mainly played in the opponent’s half, indicating the role of a striker. In Fig 4(b), the player mainly played through the middle and dropped back sometimes to defend, indicating the role of a holding midfielder.
Combining the visual timeline summary, shown in Fig 1, and the ball line chart, shown in Fig 3, we have developed a prototype for visualizing the game as shown in Fig 5 (The use of 3D-render visualization is only meant to help illustrate the use of space and time). Fig 5(a) shows the visual timeline summary, with the pink color indicating a goal. Upon zooming in, we see the ball positions in individual frames leading up to the goal, as shown in Fig 5(b). On clicking one of the frames, the position of the ball and players can be seen, as shown in Fig 5(c). Upon clicking on a player or the ball, the visualizations shown in Fig 3 and Fig 4 can be obtained. We are hopeful that this visualization would enable tactical understanding of the play. We are in the process of developing the software for this visualization.
Figure 4: The figure shows the player positions for a game. In Fig (a), is it seen that the player mainly played in the opponent’s half, indicating the role of a striker. In Fig (b), the player mainly played through the middle and dropped back sometimes to defend, indicating the role of a holding midfielder.
Figure 5: The prototype of our visualization which combines the visual timeline summary and ball line chart visualization, is shown in the figure. The visual timeline summary is shown in Fig (a). On zooming in, the location of the ball in individual frames can be seen, as shown in Fig (b). On clicking on the frame, the position of the ball and players can be seen, as shown in Fig (c). Upon clicking on a player or the ball, the visualization shown in Fig 3 and Fig 4 can be obtained. We are hopeful that this would enable tactical understanding of the play.
In this project, we have developed an intuitive real-time visual summary of key events in a game to provide a recap of the action to a viewer who is late to a broadcast. Using the position of the ball, we visualize the game-flow. We have shown that the visualization can be extended to understand the role of various players, thus, enabling tactical understanding of the play. With focus on accessibility our approach can be easily extended to other non-visual modalities.
 J. Allebach. Binary display of images when spot size exceeds step size. Applied Optics, 15:2513–2519, August 1980.
 E. Catmull. A tutorial on compensation tables. In Computer Graphics, volume 13, pages 1–7. ACM SIGGRAPH, 1979.
 Peter Litwinowicz and Lance Williams. Animating images with drawings. Proceedings of SIGGRAPH, 1994.
 Andrii Maksai, Xinchao Wang and Pascal Fua. What Players do with the ball: a physically constrained interaction modeling. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2016.
 Charles Perin, Romain Vuillemot, Jean-Daniel Fekete. SoccerStories: A Kick-off for Soccer Visual Analysis. Proceedings of the IEEE Transactions on Visualization and Computer Graphics 2013.
 Jingchen Liu, Peter Carr and Robert Collins. Tracking sports players with context-conditioned motion models. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2013.
 Matthew Parry, Philip Legg, David Chung, Iwan Griffiths and Min Chen. Hierarchical Event Selection for Video Storyboards with a Case Study on Snooker Video Visualization. Proceedings of the IEEE Transactions on Visualization and Computer Graphics 2011.
Presentation at IEEE InfoVis '17
October 1-6, 2017