Xingyu Bruce Liu

selected publications

1. CHI

   Visual Captions: Augmenting Verbal Communication with On-the-fly Visuals

   Xingyu "Bruce" Liu, Vladimir Kirilyuk, Xiuxiu Yuan, Alex Olwal, Peggy Chi, Xiang "Anthony" Chen, and Ruofei Du

   In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems 2023

   PDF Abs BibVideo

   Computer-mediated platforms are increasingly facilitating verbal communication, and capabilities such as live captioning and noise cancellation enable people to understand each other better. We envision that visual augmentations that leverage semantics in the spoken language could also be helpful to illustrate complex or unfamiliar concepts. To advance our understanding of the interest in such capabilities, we conducted formative research through remote interviews (N=10) and crowdsourced a dataset of 1500 sentence-visual pairs across a wide range of contexts. These insights informed Visual Captions, a real-time system that we integrated into a videoconferencing platform to enrich verbal communication. Visual Captions leverages a fine-tuned large language model to proactively suggest relevant visuals in open-vocabulary conversations. We report on our findings from a lab study (N=26) and a two-week deployment study (N=10), which demonstrate how Visual Captions has the potential to help people improve their communication through visual augmentation in various scenarios.

   @inproceedings{liu2023visualcaptions,
     author = {Liu, Xingyu "Bruce" and Kirilyuk, Vladimir and Yuan, Xiuxiu and Olwal, Alex and Chi, Peggy and Chen, Xiang "Anthony" and Du, Ruofei},
     title = {Visual Captions: Augmenting Verbal Communication with On-the-fly Visuals},
     year = {2023},
     isbn = {},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {},
     doi = {},
     booktitle = {Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems},
     articleno = {},
     numpages = {},
     keywords = {Computer Mediated Communication ; Artifact or System ; Dataset ; Empirical study that tells us about how people use a system},
     location = {Hamburg, Germany},
     series = {CHI '23},
     video = {https://youtu.be/sL_YeHtQt44},
   }

2. UIST

   CrossA11y: Identifying Video Accessibility Issues via Cross-Modal Grounding

   Xingyu "Bruce" Liu, Ruolin Wang, Dingzeyu Li, Xiang "Anthony" Chen, and Amy Pavel

   In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology 2022

   🏆 Best Paper Award

   PDF Abs BibVideo Website

   Authors make their videos visually accessible by adding audio descriptions (AD), and auditorily accessible by adding closed captions (CC). However, creating AD and CC is challenging and tedious, especially for non-professional describers and captioners, due to the difficulty of identifying accessibility problems in videos. A video author will have to watch the video through and manually check for inaccessible information frame-by-frame, for both visual and auditory modalities. In this paper, we present CrossA11y, a system that helps authors efficiently detect and address visual and auditory accessibility issues in videos. Using cross-modal grounding analysis, CrossA11y automatically measures accessibility of visual and audio segments in a video by checking for modality asymmetries. CrossA11y then displays these segments and surfaces visual and audio accessibility issues in a unified interface, making it intuitive to locate, review, script AD/CC in-place, and preview the described and captioned video immediately. We demonstrate the effectiveness of CrossA11y through a lab study with 11 participants, comparing to existing baseline.

   @inproceedings{liu2022crossa11y,
     author = {Liu, Xingyu "Bruce" and Wang, Ruolin and Li, Dingzeyu and Chen, Xiang "Anthony" and Pavel, Amy},
     title = {CrossA11y: Identifying Video Accessibility Issues via Cross-Modal Grounding},
     year = {2022},
     isbn = {9781450393201},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     doi = {10.1145/3526113.3545703},
     url = {https://doi.org/10.1145/3526113.3545703},
     booktitle = {Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology},
     articleno = {43},
     numpages = {14},
     keywords = {audio description, video, accessibility, closed caption},
     location = {Bend, OR, USA},
     series = {UIST '22},
     award = {Best Paper Award},
     video = {https://youtu.be/HDqjnHOZ7J8},
   }

3. CHI

   What Makes Videos Accessible to Blind and Visually Impaired People?

   Xingyu Liu, Patrick Carrington, Xiang "Anthony" Chen, and Amy Pavel

   In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems 2021

   PDF Abs BibVideo Website

   User-generated videos are an increasingly important source of information online, yet most online videos are inaccessible to blind and visually impaired (BVI) people. To find videos that are accessible, or understandable without additional description of the visual content, BVI people in our formative studies reported that they used a time-consuming trial-and-error approach: clicking on a video, watching a portion, leaving the video, and repeating the process. BVI people also reported video accessibility heuristics that characterize accessible and inaccessible videos. We instantiate 7 of the identified heuristics (2 audio-related, 2 video-related, and 3 audio-visual) as automated metrics to assess video accessibility. We collected a dataset of accessibility ratings of videos by BVI people and found that our automatic video accessibility metrics correlated with the accessibility ratings (Adjusted R2 = 0.642). We augmented a video search interface with our video accessibility metrics and predictions. BVI people using our augmented video search interface selected an accessible video more efficiently than when using the original search interface. By integrating video accessibility metrics, video hosting platforms could help people surface accessible videos and encourage content creators to author more accessible products, improving video accessibility for all.

   @inproceedings{liu2021what,
     author = {Liu, Xingyu and Carrington, Patrick and Chen, Xiang "Anthony" and Pavel, Amy},
     title = {What Makes Videos Accessible to Blind and Visually Impaired People?},
     year = {2021},
     isbn = {9781450380966},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3411764.3445233},
     doi = {10.1145/3411764.3445233},
     booktitle = {Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems},
     articleno = {272},
     numpages = {14},
     keywords = {visual impairments, blind, online videos, accessibility},
     location = {Yokohama, Japan},
     series = {CHI '21},
     video = {https://youtu.be/n2enrJJZdTs},
   }