Okamoto Lauren ; Parmar Paritosh

Abstract

Action quality assessment (AQA) applies computer vision to quantitativelyassess the performance or execution of a human action. Current AQA approachesare end-to-end neural models, which lack transparency and tend to be biasedbecause they are trained on subjective human judgements as ground-truth. Toaddress these issues, we introduce a neuro-symbolic paradigm for AQA, whichuses neural networks to abstract interpretable symbols from video data andmakes quality assessments by applying rules to those symbols. We take diving asthe case study. We found that domain experts prefer our system and find it moreinformative than purely neural approaches to AQA in diving. Our system alsoachieves state-of-the-art action recognition and temporal segmentation, andautomatically generates a detailed report that breaks the dive down into itselements and provides objective scoring with visual evidence. As verified by agroup of domain experts, this report may be used to assist judges in scoring,help train judges, and provide feedback to divers. Annotated training data andcode: https://github.com/laurenok24/NSAQA.