A creative video agent needs to preserve a subject across shots. A 3D editing agent needs to change a material without breaking the geometry. A planning agent needs to distinguish a plausible prediction from an action that actually reaches its goal.
Multimodal intelligence becomes useful through an agent loop: understand the context, reason about the next step, act, and verify the outcome. Computer vision and representation learning supply the state; generative and predictive models expand the available actions; evaluation tells us whether those actions worked.
This technical report explains that architecture through concrete implementations, established methods, and our published evaluation study. It distinguishes the browser demonstration, product interfaces, and measured experiments so each artifact can be assessed on its own terms.
The architecture: observation, state, prediction, action
The architecture separates five responsibilities:
- Observation. Images, video frames, audio, documents, and tool responses carry source and time information. Perception extracts entities, relationships, geometry, and events without discarding the original evidence.
- Persistent state. Observations update a representation of objects and their current attributes. The representation distinguishes what was observed, what was inferred, and what may have become stale.
- Prediction. A temporal model estimates possible future states. Action-conditioned prediction adds the candidate action as an input and retains uncertainty over its consequences.
- Agent decision. The agent selects a response, a request for more evidence, or an action within its permissions. State uncertainty can change the decision: observe again, ask for clarification, or defer to a reviewer.
- Verification. Outcome checks compare the resulting state with the task contract. A convincing explanation and a correct action are evaluated separately.
In a compact notation, z_t = update(z_(t-1), observation_t) represents state maintenance, while p(z_(t+1) | z_t, action_t) represents action-conditioned prediction. These describe interfaces between capabilities, not a claim that the current products share one trained model.
The difficult work is inside those interfaces: maintaining correspondence across views, separating camera motion from scene change, handling occlusion, calibrating uncertainty, and limiting compounding prediction error.
Entity resolution: preserve the reference before acting
A request such as “send it to Alex” can name several valid candidates. The next action determines what must be resolved: notifying a person needs a specific identity, while routing a ticket may need only a shared team. The system should preserve that distinction when the user corrects a name or changes the task.
SuperER connects conversational references to catalog entities and carries typed state with provenance. It combines deterministic rules, retrieval, and optional model adapters; unresolved identity can produce a structured clarification instead of guessed tool parameters. The application remains responsible for authorization and execution.
This is one part of the broader perception-to-action problem. Identifying a person in a catalog and tracking an object across video frames require different methods, but both depend on preserving the reference as context changes. Useful tests include ambiguous aliases, corrections, stale memory, missing candidates, and the parameters proposed for the next action.
An executable agent loop
The homepage demonstration runs a complete, small planning task in the browser. A 6 × 5 grid exposes a start cell, a goal, and occupied cells. A breadth-first search planner computes a shortest route; the executor advances one cell per step; the verifier checks the final location and tests every visited cell against the obstacles.
“Change scene” inserts another obstacle and resets the task. Running the agent again recomputes the path from the changed state. The transition from observation to plan to action is visible, and the final move count is calculated from the executed route.
This implementation uses structured scene state and deterministic search. It makes the control loop inspectable without claiming learned visual perception or neural dynamics. Replacing the state encoder or transition model with a learned component introduces specific tests: reconstruction error, prediction error, calibration, and downstream task success under an aligned budget.
The accompanying original video visualizes two representation problems: connecting appearance to spatial structure, and preserving a generated scene across camera movement. It is generated media, not an experimental reconstruction result.
Video intelligence: preserve events across time
A video agent should answer “Where did this happen?” with a grounded interval and “What happened next?” with a clear distinction between observation and prediction.
A temporal-understanding pipeline combines clip-level features, temporal retrieval, and object or event memory. The important comparison is whether explicit temporal state improves on a retrieval baseline under the same observation budget. Extra frames, extra context, and extra inference compute must not silently become the explanation for a better result.
Measurement: measure temporal localization, relevant-event retrieval, cross-view consistency, unsupported-event rate, and latency. Recognition and forecasting use separate scores. Hold out sequences and environments; analyze occlusion, identity switches, and evidence gaps explicitly.
Ego-Exo4D provides synchronized first- and third-person views of skilled activity and benchmarks including keystep recognition and pose estimation. Ego4D provides complementary episodic-memory and forecasting tasks. These public foundations help study temporal understanding; editorial video and learning workflows need their own task-specific acceptance protocols.
Coherent 3D content: preserve structure across views
A generated scene should remain the same scene when the camera moves. An editing agent should be able to change an object or material while preserving the constraints the designer asked to keep.
This separates three questions: does the image look plausible, does the geometry agree across views, and did the edit preserve the intended content? An attractive render answers only the first.
The scene-editing architecture connects visual conditioning, camera poses, an explicit or learned scene representation, and an edit history. Useful baselines include established reconstruction tools, pretrained image-to-3D systems, and direct multi-view generation. An agent assembles candidate edits, checks constraints, and presents the alternatives to a designer.
Measurement: novel-view quality, geometry consistency, object preservation, requested-edit fidelity, and cost per accepted asset. Use held-out viewpoints and separate reconstruction scores from generation and editing scores.
The homepage’s Video → 3D geometry lab makes a narrower part of this problem executable. Two calibrated synthetic views share known correspondences sampled from visible surfaces. Closest-ray midpoint triangulation estimates 3D positions; the interface recomputes RMSE against the authored geometry as camera separation and pixel noise change. The downloaded measurement includes reconstructed and reference points. Surface and wireframe views display the reference mesh; only the point-cloud view displays triangulated positions. This isolates geometric conditioning without claiming feature extraction, pose estimation, or learned reconstruction from arbitrary video.
ScanNet++ offers high-fidelity captures and benchmarks for novel-view synthesis and 3D semantic understanding, with public tooling and baselines. It can support geometry and view-consistency checks. Creative editing needs an additional protocol with explicit edit instructions and preserved attributes.
World models: move from recall to prediction
In a simulated task, retrieval locates a past observation. Temporal memory retains task context. A predictive world model goes further: it estimates how a candidate action changes the environment.
A useful baseline ladder is frame retrieval, a policy with temporal memory, and a learned action-conditioned predictor. Compare them with aligned observation budgets, held-out sequences, and increasing gaps between observations. Evaluate future-state error together with uncertainty, rather than reporting only a plausible-looking generated continuation.
The downstream test: does a better state representation improve a decision? Evaluate the agent with and without the predictive component, holding the policy, task set, and compute budget as consistent as possible. Report where prediction helps, where it fails, and its additional latency.
Here, “world model” refers to a learned predictive representation. An event-sourced business record is operational memory; it does not by itself demonstrate learned physical dynamics.
Planning in simulation: compare candidate actions
A planner can compare candidate actions using predicted states, select an action, observe the resulting transition, and update its plan. Long-horizon prediction introduces compounding error: a plausible first transition can still lead to an unsuccessful plan.
The baseline comparison separates a reactive policy, a policy with temporal memory, and a planner using predicted future states. Inputs, actions, task chains, and compute budgets need to be aligned. Collision and other constraint checks belong in the outcome contract.
CALVIN provides an open-source simulated benchmark for long-horizon, language-conditioned manipulation. Its environment, task definitions, and baselines provide a foundation for testing completed task chains, compounding error, calibration, and compute per successful sequence. Simulation results require separate validation before any claim of transfer to physical robots.
Controllable generation: vary conditions, preserve facts
A useful visual edit changes a requested attribute while preserving identity, geometry, and the rest of the scene. For video, that requirement extends across frames.
An edit-preservation comparison pairs pretrained editing baselines with lightweight adaptation and explicit spatial or identity conditioning. Controlled variations in lighting, viewpoint, or layout can also expose an agent's reliance on superficial cues. Each generated test case needs an independent check that the task-relevant facts survived the edit.
Measurement: requested-edit fidelity, object identity, geometric consistency, temporal artifacts, and inference cost. VBench provides established video-quality dimensions. Edit-preservation probes and targeted human review complement those dimensions; perceptual similarity alone does not verify the intended change.
Visual workflow agents: carry evidence into an action
Consider a document-to-record task. An agent reads a public document, constructs structured state, prepares a record change, and verifies the result. The failure taxonomy distinguishes perception, numerical reasoning, tool arguments, permissions, and final-state correctness.
The workflow architecture preserves an evidence pointer with each field. A review step can inspect the source before a consequential write. Post-action verification checks the actual record rather than a screenshot of an apparently successful interaction.
Measurement: compare a direct vision-language agent, extraction plus tools, and explicit grounding plus verification. Measure final-state correctness, grounding accuracy, permission violations, latency, and cost per success across repeated runs.
OSWorld provides an environment and evaluation approach for executable computer-use tasks. TAT-QA provides table-and-text financial reasoning. Supplied TAT-QA text and tables do not test PDF perception; rendered-document variants need a separately identified protocol.
Agent evaluation: inspect the outcome and the path
An evaluation contract should identify the task, the allowed actions, the evidence requirement, and the final-state checks. Deterministic checks, model graders, and human adjudication answer different questions and can disagree.
Useful signals include repeated-run task success, false acceptance, evidence quality, policy adherence, and cost. A single aggregate score can conceal a system that performs well on easy cases while accepting consequential errors.
τ²-bench provides a foundation for tool-agent-user interaction. Keep official benchmark scores separate from added evidence or cost checks, and pin the revision, model version, tools, budgets, and scoring rules.
Multivon contributes existing evaluation infrastructure. Native video, 3D, and robotics evaluation require additional adapters and domain-specific scorers; they are not implied by the current library's feature set.
Application patterns: agents that act on visual evidence
The common workflow is observe → maintain state → take a permitted action → verify the result.
- Creative video. Convert a brief into shots, revise selected attributes, and review consistency. Evaluate subject preservation, temporal artifacts, requested-edit fidelity, and cost per accepted sequence.
- Editable 3D worlds. Generate a spatial asset from visual references and apply controlled edits. Evaluate held-out views, geometry, identity preservation, and constraint satisfaction.
- Agents in simulation. Compare reactive policies with predictive planning on reproducible tasks. Evaluate completed task chains, prediction error, constraint violations, and compute per success.
- Video knowledge. Retrieve moments from lectures, interviews, and training material. Evaluate temporal grounding, unsupported claims, coverage, and human review agreement.
- Visual software agents. Connect document or screen understanding to a permitted software action. Verify the resulting application state independently of the agent's explanation.
- Multimodal regression. Produce validated visual variations, rerun agents, and assemble failure reports. Evaluate invariant preservation, fault discovery, false acceptance, and reproducibility.
Public tools provide useful starting points. World Labs' Marble combines multimodal world generation with editing and export for spatial-content workflows. TwelveLabs' enterprise video-agent example shows how video analysis can enter an agent workflow. VBench contributes established dimensions for evaluating generated video.
These references inform the technical choices; they do not establish customer demand for OmniTensorLabs or imply partnerships. Our application focus is creative and knowledge software, simulated planning, and evaluation infrastructure. The common technical thread is the connection between representation quality and downstream outcomes.
Products and public evidence
Saathi is a personal AI assistant in private preview. Voice, camera reading, messages, and persistent personal context make the perception–memory–response loop concrete. Its case study describes the system and evaluation contract; field results remain unpublished.
OmniManas brings operational context and permissioned workflows. The developer tools address distinct parts of building these systems: SuperER resolves references and conversational state, SuperReview reviews code changes inside a coding agent, and Multivon evaluates answers, trajectories, and outcomes. They share a research direction; availability and integration requirements are described on each product’s site.
In Multivon's published TAT-QA financial evidence study, jointly requesting answers and evidence locations reduced exact-match accuracy from 74.74% to 72.22% across 1,663 questions and increased estimated model cost by 41.7%. This is evidence for one model and protocol using supplied table and text contexts, not a video, 3D, or customer-performance result. The study links its protocol, metrics, and execution artifacts.
Reproducible artifacts
Use established datasets, official scorers, pretrained models, reconstruction tooling, and simulators where they answer the question. Respect access and redistribution terms. Keep training and development data separate from held-out evaluation, and report contamination risks for public benchmarks.
A complete experimental report includes the baseline configuration, ablations, unsuccessful runs, uncertainty, failure analysis, and resource costs. New representations, adaptation methods, and verifiers earn their place through measured outcomes.
The website's original films, diagrams, and synthetic examples explain the architecture. They are illustrative media, not recordings of deployed systems or experimental evidence. Public benchmark claims are attached to a corresponding study and its artifacts.