CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scale

View a PDF of the paper titled CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scale, by ZeMing Gong and 6 other authors

View PDF
HTML (experimental)

Abstract:Measuring biodiversity is crucial for understanding ecosystem health. While prior works have developed machine learning models for taxonomic classification of photographic images and DNA separately, in this work, we introduce a multimodal approach combining both, using CLIP-style contrastive learning to align images, barcode DNA, and text-based representations of taxonomic labels in a unified embedding space. This allows for accurate classification of both known and unknown insect species without task-specific fine-tuning, leveraging contrastive learning for the first time to fuse DNA and image data. Our method surpasses previous single-modality approaches in accuracy by over 8% on zero-shot learning tasks, showcasing its effectiveness in biodiversity studies.