Penta ML at EXIST 2024: Tagging Sexism in Online Multimodal Content With Attention-enhanced Modal Context

Abstract

Content moderation at scale warrants automated systems that are capable of understanding nuance from the text and images being posted online. Transformer-based models have been shown to perform with these preconditions in mind, but the additional complexities originating from multimodality and multilinguality mandate the need for better tuned systems that can capture more enriched representations of the context. This can essentially translate to downstream tasks such as sexism identification in online content, which is the forefront of the EXIST 2024 shared tasks. This paper, as part of the EXIST challenge at CLEF 2024, investigates an attention-based approach to improve performance over baseline multimodal models by assigning separate importance to the textual and visual representations. The proposal is evaluated against CLIP and ViLT, two established multimodal models, while achieving state-of-the-art performance in multi-label classification tasks in the hard-hard evaluation context. The study is further augmented by the inclusion of different forms of ablations, involving confusion metrics for the applicable tasks.