Your Sidebar Name

Your Sidebar Name

Short biography for the left-hand sidebar

Portfolio

Multi-modal Prompting for Open-vocabulary Video Visual Relationship Detection (AAAI 2024)

Abstract

Open-vocabulary video visual relationship detection aims to extend video visual relationship detection beyond annotated categories by detecting unseen relationships between objects in videos. Recent progresses in open-vocabulary perception, primarily driven by large-scale image-text pre-trained models like CLIP, have shown remarkable success in recognizing novel objects and semantic categories. However, directly applying CLIP-like models to video visual relationship detection encounters significant challenges due to the substantial gap between images and video object relationships. To address this challenge, we propose a multi-modal prompting method that adapts CLIP well to open-vocabulary video visual relationship detection by prompt-tuning on both visual representation and language input. Specifically, we enhance the image encoder of CLIP by using spatio-temporal visual prompting to capture spatio-temporal contexts, thereby making it suitable for object-level relationship representation in videos. Furthermore, we propose visual-guided language prompting to leverage CLIP’s comprehensive semantic knowledge for discovering unseen relationship categories, thus facilitating recognizing novel video relationships. Extensive experiments on two public datasets, VidVRD and VidOR, demonstrate the effectiveness of our method, especially achieving a significant gain of nearly 10\% in mAP on novel relationship categories on the VidVRD dataset.

METOR: A Unified Framework for Mutual Enhancement of Objects and Relationships (IJCAI 2025)

Abstract

Open-vocabulary video visual relationship detection aims to detect objects and their relationships in videos without being restricted by predefined object or relationship categories. Existing methods leverage the rich semantic knowledge of pre-trained vision-language models suchas CLIP to identify novel categories. They typically adopt a cascaded pipeline to first detect objects and then classify relationships based on the detected objects, which may lead to error propagation and thus suboptimal performance. In this paper, we propose Mutual EnhancemenT of Objects and Relationships (METOR), a query-based unified framework to jointly model and mutually enhance object detection and relationship classification in open-vocabulary scenarios. Under this framework, we first design a CLIP-based contextual refinement encoding module that extracts visual contexts of objects and relationships to refine the encoding of text features and object queries, thus improving the generalization of encoding to novel categories. Then we propose an iterative enhancement module to alternatively enhance the representations of objects and relationships by fully exploiting their interdependence to improve recognition performance. Extensive experiments on two public datasets, VidVRD and VidOR, demonstrate that our framework achieves state-of-the art performance.

End-to-end open-vocabulary video visual relationship detection using multi-modal prompting (TPAMI 2025)

Abstract

Open-vocabulary video visual relationship detection aims to expand video visual relationship detection beyond anno tated categories by detecting unseen relationships between both seen and unseen objects in videos. Existing methods usually use trajectory detectors trained on closed datasets to detect object trajectories, and then feed these trajectories into large-scale pre-trained vision-language models to achieve open-vocabulary classification. Such heavy dependence on the pre-trained trajec tory detectors limits their ability to generalize to novel object categories, leading to performance degradation. To address this challenge, we propose to unify object trajectory detection and relationship classification into an end-to-end open-vocabulary framework. Under this framework, we propose a relationship aware open-vocabulary trajectory detector. It primarily consists of a query-based Transformer decoder, where the visual encoder of CLIP is distilled for frame-wise open-vocabulary object detec tion, and a trajectory associator. To exploit relationship context during trajectory detection, a relationship query is embedded into the Transformer decoder, and accordingly, an auxiliary relationship loss is designed to enable the decoder to perceive the relationships between objects explicitly. Moreover, we propose an open-vocabulary relationship classifier that leverages the rich semantic knowledge of CLIP to discover novel relationships. To adapt CLIP well to relationship classification, we design a multi-modal prompting method that employs spatio-temporal visual prompting for visual representation and vision-guided language prompting for language input. Extensive experiments on two public datasets, VidVRD and VidOR, demonstrate the effectiveness of our framework. Our framework is also applied to a more difficult cross-dataset scenario to further demonstrate its generalization ability.

Multi-modal Open-vocabulary Video Visual Relationship Detection (Under Review)

Abstract

Open-vocabulary Video Visual Relationship Detection (OV-VidVRD) aims to detect and localize subject-predicate-object relationship triplets in videos, even for relationships unseen during training. While pivotal for comprehensive video understanding, existing methods are largely confined to the visual modality. This uni-modal focus neglects rich contextual cues from other modalities, which are often vital for disambiguating complex relationships in real-world videos. To bridge this gap, we introduce the new task of Multi-modal Open-vocabulary Video Visual Relationship Detection (MM-OV-VidVRD) and present the first-of-its-kind benchmark VaM-VidVRD to catalyze research in this area. Our benchmark features numerous videos where each instance pairs the visual stream with a heterogeneous set of auxiliary modalities, such as audio, textual descriptions, and 3D data. This design is crucial as it mirrors the diverse and source-dependent nature of real-world multi-modal data. To tackle this new challenge, we propose a novel and flexible framework that effectively processes arbitrary combinations of modalities. The core of this framework is a Multi-modal Synergistic Prompting (MMSP) mechanism, which learns to dynamically align and fuse features from diverse sources into a unified, relationship-aware representation. Extensive experiments on our benchmark show that our method significantly outperforms vision-only approaches.