Majeedi Abrar ; Gajjala Viswanatha Reddy ; GNVV Satya Sai Srinath Namburi ; Li Yin

Abstract

The ability to quantify how well an action is carried out, also known asaction quality assessment (AQA), has attracted recent interest in the visioncommunity. Unfortunately, prior methods often ignore the score rubric used byhuman experts and fall short of quantifying the uncertainty of the modelprediction. To bridge the gap, we present RICA^2 - a deep probabilistic modelthat integrates score rubric and accounts for prediction uncertainty for AQA.Central to our method lies in stochastic embeddings of action steps, defined ona graph structure that encodes the score rubric. The embeddings spreadprobabilistic density in the latent space and allow our method to representmodel uncertainty. The graph encodes the scoring criteria, based on which thequality scores can be decoded. We demonstrate that our method establishes newstate of the art on public benchmarks, including FineDiving, MTL-AQA, andJIGSAWS, with superior performance in score prediction and uncertaintycalibration. Our code is available at https://abrarmajeedi.github.io/rica2_aqa/