Nwoye Chinedu Innocent ; Mutter Didier ; Marescaux Jacques ; Padoy Nicolas

Abstract

Purpose: Real-time surgical tool tracking is a core component of the futureintelligent operating room (OR), because it is highly instrumental to analyzeand understand the surgical activities. Current methods for surgical tooltracking in videos need to be trained on data in which the spatial positions ofthe tools are manually annotated. Generating such training data is difficultand time-consuming. Instead, we propose to use solely binary presenceannotations to train a tool tracker for laparoscopic videos. Methods: Theproposed approach is composed of a CNN + Convolutional LSTM (ConvLSTM) neuralnetwork trained end-to-end, but weakly supervised on tool binary presencelabels only. We use the ConvLSTM to model the temporal dependencies in themotion of the surgical tools and leverage its spatio-temporal ability to smooththe class peak activations in the localization heat maps (Lh-maps). Results: We build a baseline tracker on top of the CNN model and demonstratethat our approach based on the ConvLSTM outperforms the baseline in toolpresence detection, spatial localization, and motion tracking by over 5.0%,13.9%, and 12.6%, respectively. Conclusions: In this paper, we demonstrate that binary presence labels aresufficient for training a deep learning tracking model using our proposedmethod. We also show that the ConvLSTM can leverage the spatio-temporalcoherence of consecutive image frames across a surgical video to improve toolpresence detection, spatial localization, and motion tracking. keywords: Surgical workflow analysis, tool tracking, weak supervision,spatio-temporal coherence, ConvLSTM, endoscopic videos