SSync: Selective Synergistic Learning
for Video Object-Centric Learning

Typical video object-centric learning (VOCL) approaches employ slot-based frameworks that rely on reconstruction-driven encoder–decoder architectures, where learning is mediated by two spatial maps: attention maps from the encoder and object maps from the decoder. As these two distinct maps exhibit different properties, a recent dense alignment strategy attempted to reconcile this discrepancy by enforcing agreement across all spatio-temporal patches. However, this indiscriminate alignment inadvertently propagates the inherent weaknesses of each module—noisy encoder predictions and blurred decoder boundaries—while incurring a cost quadratic in the number of spatio-temporal patches.

Motivated by this, we propose Selective Synergistic Learning (SSync). Instead of exhaustive patch-to-patch alignment, SSync prevents error propagation by selectively distilling only the most reliable cues: leveraging the encoder strictly for boundary refinement and the decoder for interior denoising, realized via a pseudo-labeling scheme with linear complexity. To prevent the reinforcement of architectural biases like slot redundancy, we further introduce a transitive pseudo-label merging that consolidates overlapping slots based on spatio-temporal activation consistency. Extensive studies show that SSync improves decomposition quality, serves as a versatile plug-and-play module, and remains exceptionally robust to slot configurations.

Each dataset has its own slider—drag it to blend between the input video and the predicted slot map (each slot shown in a distinct color).