Picek Lukáš ; Šulc Milan ; Matas Jiří ; Heilmann-Clausen Jacob ; Jeppesen Thomas S. ; Læssøe Thomas ; Frøslev Tobias

Abstract

We introduce a novel fine-grained dataset and benchmark, the Danish Fungi2020 (DF20). The dataset, constructed from observations submitted to the Atlasof Danish Fungi, is unique in its taxonomy-accurate class labels, small numberof errors, highly unbalanced long-tailed class distribution, rich observationmetadata, and well-defined class hierarchy. DF20 has zero overlap withImageNet, allowing unbiased comparison of models fine-tuned from publiclyavailable ImageNet checkpoints. The proposed evaluation protocol enablestesting the ability to improve classification using metadata -- e.g. precisegeographic location, habitat, and substrate, facilitates classifier calibrationtesting, and finally allows to study the impact of the device settings on theclassification performance. Experiments using Convolutional Neural Networks(CNN) and the recent Vision Transformers (ViT) show that DF20 presents achallenging task. Interestingly, ViT achieves results superior to CNN baselineswith 80.45% accuracy and 0.743 macro F1 score, reducing the CNN error by 9% and12% respectively. A simple procedure for including metadata into the decisionprocess improves the classification accuracy by more than 2.95 percentagepoints, reducing the error rate by 15%. The source code for all methods andexperiments is available at https://sites.google.com/view/danish-fungi-dataset.