PRISM: Predictive Recomposition via Semantic Latent Decomposition

Abstract

Cross-view video representation learning aims to capture viewpoint-invariant action semantics despite substantial appearance changes across egocentric and exocentric videos. However, existing methods encode each video as a unified embedding, where view-invariant and view-variant semantics inevitably entangle under co-occurrences: a failure mode we show persists even in cross-view methods explicitly trained for view-invariance. Our key insight is that a view-invariant feature is truly disentangled when it can be sufficiently recomposed with an arbitrary view-variant feature while preserving their independent semantics. Building on this, we propose PRISM, that decomposes video into view-invariant and view-variant latents and recompose them under language supervision encouraging clean decomposition of the two streams. PRISM achieves state-of-the-art results on EgoExo4D, EgoExoLearn, AE2, even surpassing in-domain models under zero-shot setting.

Insight

A truly view-invariant representation must be independently decomposed from view-variant semantics.

Previous works collapse a video into a single unified representation, which fails to prevent view-variant information from leaking in when a dataset exhibits a strong correlation between view-invariant (V-I) semantics and the underlying view-variant (V-V) content. If the V-I action “playing tennis” predominantly co-occurs with the V-V context “tennis court”, unified representations may rely on the shared contextual semantics rather than the action identity itself.

Figure 1: comparison of unified and compositional representations — **Figure 1. Comparison of unified and compositional representations.** (a) Unified representations entangle actions (*playing tennis*) with co-occurring context (*tennis court*), failing under novel compositions. (b) PRISM decomposes video into view-invariant and view-variant latents and recomposes them under language supervision, enforcing clean disentanglement.

Key insight
Such decomposition is only achieved when a model can freely recompose a V-I representation with an off-distribution V-V counterpart while still preserving the correct semantic meaning.

Which is what language is good for

⟦ ⟧

Language is compositional

Thanks to its inherently compositional structure, language naturally provides semantically controllable V-I / V-V compositions, enabling seamless decomposition and novel semantic recombination beyond observed visual co-occurrences.

◉

Language is independent of viewpoint

Language captures high-level conceptual identity independently of the low-level visual variations induced by viewpoint changes.

Method

Decompose, then recompose.

The core principle of PRISM is that visual information in a video can be decomposed into a view-invariant component V-I and a view-variant component V-V. If this decomposition is achieved cleanly, the V-I of one video should retain its semantic identity even when recomposed with the V-V of another video.

Decompositional Encoder θ

We feed an input video v into our Decompositional Encoder θ to separate the primary interaction V-I from the view-variant context V-V, producing two frame-level representations.

(z^V-I_v, z^V-V_v) = θ(v) (1)

The decomposition happens per frame, and the two streams are then carried through time by a causal temporal transformer, so each keeps its own timeline.

video v

Decompositional Encoder θ

z^V-I

z^V-V

t = 1T

Decompose-and-recompose schema

Once decomposed, we deliberately break each video’s natural co-occurrence through cross-composition. Given two independently sampled videos A and B, we perform cross-view concatenation by pairing z^V-I_A with z^V-V_B. This cross-composed pair is fed into the Compositional Latent Predictor φ to synthesize a compositional semantic latent.

s_A,B = φ(z^V-I_A, z^V-V_B) (2)

Language supervised decomposition

We combine the view-invariant description T^V-I_A of video A with the view-variant description T^V-V_B of video B, both generated through a pre-trained LVLM. A text embedding model ℰ then maps the recombined text into a target semantic embedding.

e_A,B = ℰ(T^V-I_A ⊕ T^V-V_B) (3)

We then apply a contrastive objective that aligns the compositional latent s_A,B with the recombined text embedding e_A,B.

ℒ_decomp = 𝔼_A,B∈ℬ [ −log 𝕂(s_A,B, e_A,B) ∑_C,D∈ℬ 𝕂(s_A,B, e_C,D) ] (4)

Intuitively, if φ relies on shortcut correlations in z^V-V to infer the semantics of z^V-I (e.g., kitchen → cooking), such dependencies become inconsistent under cross-composition and therefore cannot match the recomposed language target.

video A

z^V-I_A

z^V-V_A

video B

z^V-I_B

z^V-V_B

cross-view concatenation

Compositional Latent Predictor φ

s_A,B

↔ℒ_decomp

T^V-I_A ⊕ T^V-V_B

playing tennis in a ballpark

Text Embedding ℰ

e_A,B

Internalizing temporal dynamics

Clip-level linguistic supervision only provides coarse semantic summaries, such as “chopping an onion”. Consequently, it is insufficient for learning representations capable of capturing fine-grained frame-level temporal dynamics, such as “lifting a knife → placing it on the onion → slicing downward”.

To compensate for this limited temporal resolution, we introduce a self-predictive signal that tasks the predictor φ with forecasting the future outputs of the encoder θ.

(ẑ^V-I_t+1, ẑ^V-V_t+1) = φ(z^V-I_≤t, z^V-V_≤t) (5)

Since θ is continuously updated during training, using it directly as the target encoder would yield unstable supervisory signals and risk representational collapse. To provide stable learning targets, we utilize a target encoder θ̄, parameterized by an Exponential Moving Average (EMA) of θ’s weights.

θ̄ ← αθ̄ + (1 − α)θ (6)

(z̄^V-I_v, z̄^V-V_v) = θ̄(v) (7)

The temporal objective maximizes the cosine similarity between predicted and target representations, allowing each stream to capture its own temporal dynamics independently without relying on external alignment signals.

ℒ_temp = 𝔼_{c∈{V-I, V-V}} [−sim(ẑ^c_t, z̄^c_t)] (8)

This objective incentivizes the encoder θ to embed temporal foresight into its representation space. Consequently, rather than merely capturing the present state, the representations at each time step naturally anticipate upcoming transitions.

z^V-I

z^V-V

t = 1T

observed ≤ t predicted ẑ_t+1

predictor φ

EMA target θ̄

ẑ_t+1 predicted

↔cosine

z̄_t+1 EMA target

We use orthogonal throughout in this semantic sense: the two streams are required to carry the disjoint language-level semantics of V-I and V-V, rather than to be geometrically orthogonal vectors.

Figure 2: overview of PRISM — **Figure 2. Overview of PRISM.** PRISM decomposes videos into view-invariant (V-I) and view-variant (V-V) latent streams, cross-composes them across videos, and aligns the resulting compositional latent with recombined language semantics. A self-predictive temporal objective further internalizes fine-grained temporal dynamics.

Results

PRISM demonstrates the most significant improvements in Retrieval and Association, tasks that directly measure semantic equivalence across different viewpoints.

0.00

EgoExo4D Retrieval R@5 (avg)

+10.4 vs. ViewpointRosetta

0.00

EgoExo4D Recognition top-1

+7.46 vs. ViewpointRosetta

0.00

EgoExoLearn Association (avg)

+11.5 vs. ViewpointRosetta

0.000

AE2 Phase Ordering Kendall’s τ

zero-shot, above in-domain AE2 (0.562)

Method	EgoExo4D						EgoExoLearn
	Retrieval (R@5)			Recognition		Assessment	Association_test			Anticipation (R@5)					Assessment
	ego2exo	exo2ego	avg	top-1	top-5	Assessment	ego2exo	exo2ego	avg	ego-V	ego-N	exo-V	exo-N	avg	Assessment
Image-Language Model
CLIP	19.11	12.24	15.68	10.49	29.90	54.93	16.00	15.64	15.82	33.50	37.40	39.60	44.30	38.70	73.48
SigLIP2	35.08	19.72	27.40	13.86	37.84	55.57	25.00	28.10	26.6	64.70	71.70	56.90	65.00	64.60	76.03
Video-Language Model
TimeSformer	6.68	6.95	6.82	5.18	14.14	51.58	15.00	17.64	16.32	68.09	75.18	51.57	61.92	64.19	75.47
InternVideo	23.64	20.67	22.16	13.67	38.99	52.58	30.60	21.70	26.20	63.76	74.28	56.67	65.92	65.16	68.88
Ego-Centric Methods
EgoVLP	29.33	13.47	21.40	20.33	46.32	54.37	31.50	28.90	30.20	67.70	72.50	50.40	52.00	60.65	69.04
LaViLa	34.91	12.02	23.47	26.43	55.01	54.10	28.70	25.70	27.20	75.67	76.58	48.03	51.05	62.83	68.44
Cross-View Methods
ActorObserverNet	29.50	24.85	27.18	15.70	38.45	54.10	11.91	11.36	11.64	63.50	61.60	49.10	50.30	56.13	68.59
VI Encoder	29.53	24.40	26.97	14.85	35.39	53.83	-	-	-	-	-	-	-	-	-
EgoInstructor	46.04	31.68	38.86	24.15	51.40	54.73	-	-	-	-	-	-	-	-	-
SUM-L	47.14	32.77	39.96	24.83	52.08	55.10	5.18	4.09	4.64	61.4	61.70	34.70	24.20	45.50	65.31
ViewpointRosetta	58.14	47.21	52.68	34.47	64.85	55.82	33.36	31.27	32.32	66.44	72.21	52.82	57.10	62.14	73.70
PRISM	75.89	50.27	63.08	41.93	72.92	55.28	44.36	43.36	43.86	72.78	77.15	62.58	65.33	69.46	68.53

Cross-view semantic alignment on EgoExo4D and EgoExoLearn. We compare VLMs, ego-centric and cross-view methods. PRISM achieves new SoTA on Retrieval, Recognition, Association, and Anticipation. Bold = best, underline = second best.

On Skill Assessment, which evaluates execution proficiency within the same action class, PRISM yields a performance (55.28) comparable to ViewpointRosetta (55.82) and SigLIP2 (55.57). We attribute this to the fact that proficiency cues (e.g., hand tremors, movement fluidity) rely heavily on subtle visual nuances in execution style rather than the core semantic identity of the action. As a result, such fine-grained visual details are likely allocated to the V-V component during the decomposition process.

Method	Frame Retrieval (mAP@10)				Kendall’s τ	Action Phase Classification (F1)				Phase Progression
Method	regular	ego2exo	exo2ego	avg	Kendall’s τ	regular	ego2exo	exo2ego	avg	Phase Progression
Random	53.97	51.68	51.20	52.28	0.004	32.90	33.44	33.46	33.27	−0.069
Trained w/ AE2 videos (In-Domain)
ActorObserverNet	50.47	42.70	41.29	44.82	0.002	36.14	36.40	31.00	34.51	−0.052
TCN	58.25	47.37	42.48	49.37	0.046	56.80	35.92	41.40	44.71	−0.227
CARL	56.44	51.14	47.86	51.81	0.025	52.22	40.85	43.19	45.42	−0.124
TCC	70.58	62.08	65.84	66.17	0.400	67.17	55.90	52.27	58.45	0.322
GTA	72.42	66.39	65.45	68.08	0.464	69.63	63.29	70.41	67.77	0.322
AE2	75.78	72.58	71.25	73.20	0.562	75.96	71.00	76.44	74.47	0.480
Trained w/o AE2 videos (Zero-Shot Transfer)
ResNet-50	58.26	44.87	42.94	48.69	0.025	53.53	31.47	45.25	43.41	−1.215
CLIP	53.70	47.35	41.27	47.44	0.047	55.83	39.39	37.46	44.23	−1.212
SigLIP2	49.87	45.58	41.23	45.56	0.020	52.39	38.00	41.33	43.91	−1.322
SUM-L	56.20	39.70	41.50	45.80	0.110	63.00	43.80	46.00	50.93	0.030
ViewpointRosetta	60.80	53.50	48.20	54.17	0.047	53.20	39.00	48.60	46.93	−0.150
PRISM	77.70	68.90	65.00	70.53	0.601	79.60	69.80	71.30	73.57	0.647

Fine-grained temporal modeling on the AE2 benchmark. We compare methods trained w/ and w/o AE2 videos. PRISM achieves new SoTA among methods trained w/o AE2 data. Bold = best among methods trained w/o AE2 videos.

PRISM surpasses the best in-domain model AE2 on Phase Ordering (0.601 vs. 0.562) and Phase Progression (0.647 vs. 0.480), while remaining within 3 and 1 points on Frame Retrieval and Action Phase Classification, respectively. This confirms that the self-predictive temporal objective internalizes fine-grained temporal dynamics without requiring any in-domain supervision.

Method	#Params	CLIP ViT-L/14		SigLIP2		Qwen3Embed		Overall
Method	#Params	R@10	RSA	R@10	RSA	R@10	RSA	R@10	RSA
Vision Foundation Models
V-JEPA 2	1 B	7.1	0.055	7.7	0.046	7.0	0.046	7.3	0.049
DINOv2	1 B	13.8	0.121	14.2	0.107	14.2	0.209	14.1	0.146
Cross-View Methods
ActorObserverNet	58 M	6.6	0.066	6.9	0.065	6.7	0.091	6.7	0.074
SUM-L	126 M	4.0	0.039	3.9	0.029	3.9	0.056	3.9	0.041
ViewpointRosetta	177 M	7.5	0.039	7.7	0.064	7.3	0.191	7.5	0.098
PRISM	108 M	14.7	0.114	14.9	0.128	15.1	0.301	14.9	0.181

Robustness to action-scene disentanglement on the UNSCENE benchmark. PRISM surpasses both vision foundation models and cross-view methods. Bold = best, underline = second best.

UNSCENE features videos where the action contradicts the background context (e.g., fishing inside a bedroom). PRISM nearly doubles the best cross-view baseline ViewpointRosetta in both metrics (14.90 vs. 7.50 in R@10; 0.181 vs. 0.098 in RSA), while matching DINOv2 with only one-tenth of its parameters. This confirms that disentanglement fundamentally requires an explicit decomposition mechanism and cannot be trivially acquired by scaling alone.

Settings	Cross-view Alignment			Temporal Alignment
Settings	w/o ℒ_temp	w/ ℒ_temp	Avg	w/o ℒ_temp	w/ ℒ_temp	Avg
Unified	38.8	32.4	35.6	64.0	69.2	66.6
Decomposed	50.9	53.5	52.2	65.9	72.1	69.0
Avg	44.9	43.0	-	65.0	70.7	-

Contribution of training objectives. Unified encodes a video into a single embedding, whereas Decomposed splits it into V-I and V-V streams that are cross-composed across videos. Decomposition drives cross-view alignment while ℒ_temp drives temporal alignment, and combining both achieves the best on both axes.

Settings	Cross-view Alignment			Temporal Alignment
Settings	ego2exo	exo2ego	Avg	Frm. retrieve	Act. phase	Avg
Supervision w/ Ego-Exo Pairing
ActorObserverNet	20.7	18.1	19.4	44.8	34.5	39.7
ViewpointRosetta	45.8	39.2	42.5	54.2	46.9	50.6
Supervision w/o Pairing (PRISM)
Exo only + Gemini	52.2	44.3	48.2	69.4	73.0	71.2
Ego + Exo + Qwen	59.4	46.3	52.9	69.2	74.5	71.8
Ego + Exo + Gemini	60.1	46.8	53.5	70.5	73.6	72.1

Sensitivity to training views and captioner choice. PRISM with exo-only data still surpasses all baselines on cross-view alignment, and substituting the captioner introduces only noise-level differences.

Analysis

Figure 5: correlation between representation similarity and textual V-I / V-V similarity — **Figure 5. Correlation between representation similarity and textual V-I / V-V similarity.** The x- and y-axes denote V-V and V-I textual similarity for every clip pair in EgoExoLearn, and color indicates representation similarity (we use z^V-I for PRISM). Only PRISM shows similarity governed by the V-I axis and flat along V-V.

Figure 4: t-SNE of frame-level embeddings over time — **Figure 4. t-SNE of frame-level embeddings over time.** Temporally adjacent frames are connected by lines. PRISM is the only model exhibiting both semantic separation and temporal continuity.

Figure 3: DeepLift attribution of encoder output patches to the V-I and V-V streams — **Figure 3. DeepLift attribution of encoder output patches to the V-I and V-V streams.** PRISM consistently attends to hands and interacting objects for V-I, and to background regions for V-V, across both ego and exo views.

Limitations stated in the paper

Captioner ceiling. Our semantic decomposition is inherently upper-bounded by the quality of the pre-trained LVLM captioner that provides the language-level supervisory signal. Swapping the captioner introduces only noise-level differences, but this robustness holds only across individual model choices. If a systematic bias is shared across LVLMs, such bias would propagate directly into the decomposed supervision targets and impose a quality ceiling on the encoder’s disentanglement.

Activity domain. All benchmarks evaluated in this work center on procedural human activities involving physical object manipulation (e.g., cooking, sports). This leaves non-human activities and human interactions that lack tangible physical manipulation unexplored.

Two minutes on how PRISM works

A truly view-invariant representation must be independently decomposed from view-variant semantics.

Which is what language is good for

Language is compositional

Language is independent of viewpoint

Decompose, then recompose.

Decompositional Encoder θ

Decompose-and-recompose schema

Language supervised decomposition

Internalizing temporal dynamics