Multi-modal Open-vocabulary Video Visual Relationship Detection (Under Review)

Abstract

Open-vocabulary Video Visual Relationship Detection (OV-VidVRD) aims to detect and localize subject-predicate-object relationship triplets in videos, even for relationships unseen during training. While pivotal for comprehensive video understanding, existing methods are largely confined to the visual modality. This uni-modal focus neglects rich contextual cues from other modalities, which are often vital for disambiguating complex relationships in real-world videos. To bridge this gap, we introduce the new task of Multi-modal Open-vocabulary Video Visual Relationship Detection (MM-OV-VidVRD) and present the first-of-its-kind benchmark VaM-VidVRD to catalyze research in this area. Our benchmark features numerous videos where each instance pairs the visual stream with a heterogeneous set of auxiliary modalities, such as audio, textual descriptions, and 3D data. This design is crucial as it mirrors the diverse and source-dependent nature of real-world multi-modal data. To tackle this new challenge, we propose a novel and flexible framework that effectively processes arbitrary combinations of modalities. The core of this framework is a Multi-modal Synergistic Prompting (MMSP) mechanism, which learns to dynamically align and fuse features from diverse sources into a unified, relationship-aware representation. Extensive experiments on our benchmark show that our method significantly outperforms vision-only approaches.

Dataset

VaM-VidVRD is the first benchmark specifically designed for Multi-modal Open-vocabulary Video Visual Relationship Detection (MM-OV-VidVRD). It addresses the limitations of existing uni-modal datasets by providing video streams synchronized with a heterogeneous set of auxiliary modalities, including Audio, Text, and 3D data.

📊 Data Curation and Composition

The benchmark is constructed by aggregating diverse data sources to reflect various real-world scenarios. It comprises 3,000 high-quality videos (2,400 for training and 600 for testing) curated based on dynamic interactions and modal synchronization.

🛠️ High-Fidelity Annotation Pipeline

To ensure the highest data quality, VaM-VidVRD employs a rigorous four-stage, manually-validated annotation process:

1. Frame-level Object Annotation: Combines YOLO-World proposals with meticulous manual refinement (refining boundaries, deleting false positives, and adding missing objects).
2. Trajectory Linking: Annotated objects are linked into temporally-consistent trajectories using a 30-frame interval strategy to prevent identity switches.
3. Predicate Annotation: Targets pairs co-occurring for more than 30 frames. Annotators identify specific predicate categories within 30-frame segments to ensure focused accuracy.
4. Multi-modal Alignment: Rigorous synchronization of auxiliary modalities:
   • Audio: Segmented into 30-frame intervals.
   • Text Descriptions: Aligned at the clip-level.
   • 3D Data: Matched at the frame-level with video streams.

📈 Dataset Statistics & Vocabulary Split

VaM-VidVRD features a large-scale collection with significant complexity:

• Total Instances: 13,742 high-precision object trajectories and 80,581 predicate instances.
• Vocabulary: A comprehensive set of 54 object categories and 44 predicate categories.
• Open-Vocabulary Split: Following the natural long-tail distribution of real-world data:
  • Base Categories: 37 objects and 29 predicates (frequent categories used for training).
  • Novel Categories: 17 objects and 15 predicates (unseen categories used to test generalization).

This split defines the core challenge: models must learn from base categories and generalize to detect all valid relationships, including those involving novel objects and interactions in multi-modal environments.

Methodology

Our framework introduces a unified pipeline to handle the Multi-modal Open-vocabulary Video Visual Relationship Detection (MM-OV-VidVRD) task. It leverages pre-trained vision-language models (CLIP) and dynamic multi-modal fusion to detect relationships involving both seen (base) and unseen (novel) categories.

🏗️ Three-Component Pipeline

The framework follows a sequential structure to process multi-modal video data:

1. Trajectory Localizer: Parsed from the visual stream to generate class-agnostic object trajectories, identifying salient objects without pre-defined category labels.
2. Multi-modal Object Classifier: Assigns semantic categories to each trajectory by comparing multi-modal-aware visual features with textual category representations.
3. Multi-modal Predicate Classifier: Analyzes temporally co-occurring trajectory pairs using a three-stream architecture (Subject, Object, and Global Context) to identify interaction predicates.

⚡ Core Mechanism: Multi-modal Synergistic Prompting (MMSP)

The MMSP mechanism is the central fusion engine that bridges the gap between vision, language, and auxiliary modalities (Audio, Text, 3D):

• Synergistic Fusion: It uses a cross-attention module to aggregate initial visual features with all available auxiliary data into a unified context vector ($p_{syn}$).
• Dual Prompt Projection: This unified vector is projected into two specialized instructions:
  • Visual Prompts ($p_v$): Injected into the CLIP visual encoder to guide feature extraction dynamically.
  • Textual Prompts ($p_t$): Integrated with category tokens to refine the CLIP text encoder’s output.

🎯 Open-Vocabulary Classification

By guiding a frozen CLIP model with learnable, multi-modal prompts, the framework maintains the broad knowledge of large-scale pre-training while adapting to specific video interaction scenarios.

• Object Scoring: Determined by the similarity between multi-modal-aware visual features and prompt-enhanced textual category embeddings.
• Predicate Scoring: Uses feature aggregation (concatenation of subject, object, and context) followed by projection through MLPs to capture complex inter-object relationships.

📑 Training Strategy

The model is optimized via a three-stage strategy:

1. Pre-training the trajectory localizer.
2. Training classifiers (Object then Predicate) while keeping the CLIP backbone and auxiliary encoders frozen to preserve generalization capabilities.
3. Joint Fine-tuning of the entire pipeline to harmonize localization and classification.

Experiments

We conducted comprehensive evaluations on our newly introduced VaM-VidVRD benchmark. Our experiments demonstrate that multi-modal signals significantly enhance open-vocabulary generalization compared to state-of-the-art vision-only methods.

🔬 Experimental Setup

• Tasks: Evaluated on three standard protocols: PredCls (Predicate Classification), SGCls (Scene Graph Classification), and SGDet (Scene Graph Detection from raw video).
• Metrics: Performance is measured using mean Average Precision (mAP), Recall@K (R@K), and object trajectory precision (mAP_o).
• Generalization: Results are reported on the Novel-split (unseen categories during training) and the All-split (seen + unseen) to test open-vocabulary robustness.

🏆 Quantitative Results: Outperforming Baselines

Our method consistently outperforms state-of-the-art vision-only (V-Only) methods, including CLIP, RePro, and EOV-MMP.

• Key Insight: Leveraging auxiliary modalities provides substantial gains (e.g., +3.81% mAP on the Novel SGDet split), proving that non-visual cues are vital for disambiguating complex interactions.

🔍 Ablation Studies: The Power of Fusion

• Modality-Specific Strengths: Each auxiliary modality addresses specific visual ambiguities:
  • Audio: Boosts sound-related actions (e.g., speaking).
  • Text Descriptions: Helps clarify complex action-oriented predicates (e.g., shaking hands).
  • 3D Data: Significantly improves spatial reasoning (e.g., above).
• Efficacy of MMSP: Our Multi-modal Synergistic Prompting (MMSP) mechanism effectively prevents “modal conflict.” While naive fusion methods can actually degrade CLIP’s performance, MMSP successfully preserves and enhances its generalization capabilities.
• Dual-Component Impact: Applying multi-modal fusion to both the Object Classifier and Predicate Classifier yields the best results, validating our holistic framework design.

🖼️ Qualitative Comparison

Qualitative analysis highlights our model’s ability to “see” what vision-only models miss: