Chart question answering (CQA) is a task used for assessing chartcomprehension, which is fundamentally different from understanding naturalimages. CQA requires analyzing the relationships between the textual and thevisual components of a chart, in order to answer general questions or infernumerical values. Most existing CQA datasets and models are based onsimplifying assumptions that often enable surpassing human performance. In thiswork, we address this outcome and propose a new model that jointly learnsclassification and regression. Our language-vision setup uses co-attentiontransformers to capture the complex real-world interactions between thequestion and the textual elements. We validate our design with extensiveexperiments on the realistic PlotQA dataset, outperforming previous approachesby a large margin, while showing competitive performance on FigureQA. Our modelis particularly well suited for realistic questions with out-of-vocabularyanswers that require regression.