Multi-modal Prompting for Open-vocabulary Video Visual Relationship Detection (AAAI 2024)

Abstract

Open-vocabulary video visual relationship detection aims to extend video visual relationship detection beyond annotated categories by detecting unseen relationships between objects in videos. Recent progresses in open-vocabulary perception, primarily driven by large-scale image-text pre-trained models like CLIP, have shown remarkable success in recognizing novel objects and semantic categories. However, directly applying CLIP-like models to video visual relationship detection encounters significant challenges due to the substantial gap between images and video object relationships. To address this challenge, we propose a multi-modal prompting method that adapts CLIP well to open-vocabulary video visual relationship detection by prompt-tuning on both visual representation and language input. Specifically, we enhance the image encoder of CLIP by using spatio-temporal visual prompting to capture spatio-temporal contexts, thereby making it suitable for object-level relationship representation in videos. Furthermore, we propose visual-guided language prompting to leverage CLIP’s comprehensive semantic knowledge for discovering unseen relationship categories, thus facilitating recognizing novel video relationships. Extensive experiments on two public datasets, VidVRD and VidOR, demonstrate the effectiveness of our method, especially achieving a significant gain of nearly 10\% in mAP on novel relationship categories on the VidVRD dataset.

Methodology

The framework focuses on two key adaptations of the CLIP architecture:

1. Dual-side Prompt Tuning: We introduce learnable prompt tokens into both the image encoder and text encoder. This allows the model to align video features with semantic descriptions more flexibly.
2. Vision-guided Language Prompting: Instead of static text prompts, we use visual context to guide the generation of language prompts, helping the model “see” the relationship before naming it.
3. Spatial-Temporal Visual Prompting: Learnable conditional prompts are used to capture the dynamic changes in video frames.

Experiments

Datasets and Evaluation

We evaluate our method on two large-scale benchmarks:

• VidVRD: 1,000 videos covering 35 object and 132 predicate categories.
• VidOR: 10,000 videos covering 80 object and 50 predicate categories.
• Settings: Training on Base categories and testing on Novel and All splits.

Main Results

Our method consistently outperforms existing works like RePro, VidVRD-II, and ALPro. Notably, we achieve a nearly 10% mAP improvement on the Novel split of VidVRD.

Qualitative Analysis

T-SNE visualizations demonstrate that our spatio-temporal visual prompting effectively adapts the CLIP image encoder, pulling features of the same categories closer while pushing different categories apart for both base and novel predicates.

Citation

@inproceedings{yang2024multi,
  title={Multi-modal prompting for open-vocabulary video visual relationship detection},
  author={Yang, Shuo and Wang, Yongqi and Ji, Xiaofeng and Wu, Xinxiao},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={38},
  number={7},
  pages={6513--6521},
  year={2024}
}