Yonsei University · Image and Video Pattern Recognition Lab
TL;DR
We address the object-pair distribution shift in weakly-supervised VSGG by refining pseudo-labels with vision-language grounding and suppressing non-interactive pairs throughout relational reasoning.
Weakly-supervised video scene graph generation relies on off-the-shelf detectors that produce numerous interaction-irrelevant objects, creating a substantial object-pair distribution shift from fully-supervised settings. We address this shift at both the pseudo-label generation and relational reasoning stages. Relation-Aware Matching (RAM) uses vision-language grounding to refine ambiguous detection-triplet matching, while Pair Affinity Learning and Scoring (PALS) learns pair interactivity for prediction ranking and Pair Affinity Modulation (PAM) suppresses non-interactive pairs during spatial-temporal reasoning. Our framework consistently improves different baselines and backbones on Action Genome, achieving state-of-the-art weakly-supervised VSGG performance.
Conventional WS-VSGG trains only on class-matched pairs and discards unmatched pairs, even though both appear at inference. Our pipeline refines the matched partition and learns predicate classes together with pair affinity from matched and unmatched pairs.
RAM converts each unlocalized triplet into a relation-aware text query and extracts cross-modal attention from a vision-language model. Reliable attention grounds the interaction-consistent instance; uncertain cases safely fall back to class-level matching.
PALS learns whether a subject-object pair is interactive and uses that affinity in final triplet ranking. PAM carries the same signal inside spatial-temporal attention, preventing non-interactive pairs from diluting relational context.
Scene graph detection on Action Genome with the STTran backbone. Higher Recall@K is better.
| Backbone | Supervision | Method | With Constraint | No Constraint | ||||
|---|---|---|---|---|---|---|---|---|
| R@10 | R@20 | R@50 | R@10 | R@20 | R@50 | |||
| STTran [1] | Fully | Vanilla | 25.20 | 34.10 | 37.00 | 24.60 | 36.20 | 48.80 |
| Weakly | PLA [2] | 15.39 | 21.44 | 26.24 | 15.83 | 22.83 | 31.74 | |
| PLA + Ours | 22.24 | 26.48 | 28.00 | 23.20 | 30.24 | 37.47 | ||
| TRKT | 15.11 | 20.63 | 26.02 | 15.73 | 22.64 | 31.23 | ||
| TRKT + Ours | 19.63 | 24.29 | 27.55 | 21.13 | 27.95 | 35.89 | ||
Select a clip and toggle any combination of ground truth and model predictions. Every panel shares the same decoded video frame, playback position, and seek control.
Predictions are produced on sparsely sampled frames and held until the next prediction timestamp.
@inproceedings{kang2026paws,
title = {Revisiting Weakly-Supervised Video Scene Graph Generation via Pair Affinity Learning},
author = {Kang, Minseok and Lee, Minhyeok and Kim, Minjung and Lee, Jungho and Kim, Donghyeong and Woo, Sungmin and Jeon, Inseok and Lee, Sangyoun},
booktitle = {European Conference on Computer Vision},
year = {2026}
}