For years, “video understanding” in large AI models has mostly relied on one blunt approach: sample a video into frames, stuff those frames into the model’s context window, and hope it notices the moments that matter.
That works reasonably well for a 30-second product demo.
It becomes wildly inefficient for a 90-minute lecture, a three-hour meeting, or a tutorial where the answer you need appears for 12 seconds near the end.
Google is now taking a different approach.
On September 1, 2026, Google introduced agentic video understanding, a new mode that gives Gemini tools for navigating video and lets the model decide what to watch, how closely to watch it, and which signals to use.
Instead of processing an entire video at a fixed rate such as one frame per second, Gemini can move through the timeline using a think–act–observe loop. It can scan quickly, jump to relevant sections, inspect frames at different speeds, read the transcript, isolate audio, and revisit specific moments when it needs more evidence.
Google says the result can be dramatic: up to 88% fewer tokens, up to 66% lower cost, and up to 7% higher accuracy, with the biggest benefits appearing on long-form video.
The feature is available now through the Gemini API, Google AI Studio, and Gemini Enterprise Agent Platform, with support for Gemini 3.7 Flash, 3.6 Flash, and 3.5 Flash-Lite. Google says it is also coming to the Gemini app and YouTube’s “Ask YouTube” experience.
Traditional video processing is straightforward.
The model samples frames at a fixed rate — typically around 1 FPS by default, although developers can adjust it — combines those frames with audio, and reasons across the resulting representation.
The longer the video, the more expensive that becomes.
A dense lecture can consume a huge amount of context before the model even reaches the slide containing the answer.
Developers have already built workarounds for this. A common approach is to transcribe the video first, break it into chunks, run retrieval over the transcript, identify promising timestamps, and then send only those sections back to a vision model.
It works.
But it also requires a lot of glue code.
Agentic video understanding moves much of that search process inside the model itself.
Gemini receives a reference to the video and the user's question, then decides which internal tools to call. It can retrieve transcript sections, request targeted sequences of frames at adaptive frame rates, isolate audio, scan quickly through unimportant sections, and slow down when something deserves closer inspection.
In other words, instead of forcing the model to watch everything equally, Google is giving it something closer to a remote control.
It is also the video equivalent of agentic vision, which Google introduced earlier in 2026. With agentic vision, Gemini can write and execute code to crop, rotate, zoom into, and inspect images. With agentic video, those actions happen across a timeline instead.
Google’s examples focus on tasks where fixed-rate sampling is inherently inefficient.
Finding sub-second moments
A quick cut, state change, visual glitch, or missed step might exist for only a fraction of a second.
At 1 FPS, that frame may never enter the model’s context at all.
Agentic processing can identify the relevant window and resample it at a much higher frame rate. That could be particularly useful for video editing, quality assurance, and visual debugging.
Searching for a needle in a multi-hour haystack
Imagine asking:
“When does the speaker contradict the earlier claim about pricing?”
With a traditional approach, the model may need to process an enormous portion of a multi-hour recording.
With agentic video, Gemini can search the transcript first, identify likely timestamps, and then inspect only the relevant visual segments to confirm what actually appeared on screen.
Detecting anomalies
If Gemini notices something unusual, it can return to that section and inspect it again at a higher frame rate instead of accepting its first low-resolution view.
That makes it better suited for detecting visual glitches, strange motion, or inconsistencies that would otherwise be easy to miss.
Counting fast actions
Google also demonstrated Gemini 3.7 Flash counting repeated exercise movements.
Rather than relying on a single fixed sampling rate, the model scans the footage, identifies the important sequence, and replays it at different frame rates until it reaches a stable count.
That is much closer to how a person would review the footage.
Google also compared the same model with and without agentic processing on LongVideoBench. With agentic mode enabled, token usage dropped significantly while accuracy improved.
That is an important distinction.
This is not necessarily about creating a completely new model family.
It is about giving existing models a smarter way to decide where to spend their attention and tokens.
Static processing remains the default.
Developers opt into agentic processing on a per-video basis.
In the Interactions API, that means setting "processing": "agentic" on the video input:
from google import genai
client = genai.Client()
interaction = client.interactions.create(
model="gemini-3.7-flash",
input=[
{
"type": "video",
"uri": "https://youtu.be/7Z5Vy9JBANs",
"processing": "agentic",
},
{
"type": "text",
"text": "What are the 3 most important announcements in this keynote?",
},
],
)
print(interaction.output_text)
Google also exposes the processing trace rather than hiding the entire process behind the final answer.
That trace can include:
processing_call — Gemini requests a transcript section, video segment, or sequence of framesprocessing_result — the requested content is returnedthought — reasoning summaries, when enabledmodel_output — the final responseThat opens up interesting possibilities for interfaces.
A video-analysis app, for example, could show users messages such as:
“Scanning transcript…”
“Checking 00:14:20–00:14:45…”
“Replaying at 8 FPS…”
In multi-turn or stateless workflows, those processing steps also need to be passed back to the model so it retains the video context. Google notes that dropping them can reduce quality.
The economics are slightly different as well. Media loaded through the tools counts toward tool-use tokens, while the model’s navigation and reasoning count as thought tokens.
Developers can even mix processing modes within the same request.
You might use agentic processing for a one-hour lecture, static processing for a 30-second laboratory clip where every frame matters, and then ask Gemini to compare the two.
Controls such as manually specifying clipping intervals or custom FPS remain primarily part of the static workflow. In agentic mode, Gemini makes those decisions itself.
Agentic processing is not automatically better for every video.
Google’s guidance is essentially to start with agentic processing when long-video quality or cost matters, but stay with static processing when:
Agentic processing introduces planning and tool calls before the final response begins.
For a 20-second GIF, that overhead may accomplish very little.
For a two-hour webinar, it can fundamentally change the economics.
The token math helps explain why.
Google documents static video processing at roughly 100 tokens per second at lower resolution and around 300 tokens per second at higher resolution once video and audio are considered.
Multiply that across an hour of footage and the model can consume a huge number of tokens before generating a single word.
Agentic mode changes the equation because the model pays primarily for the sections it actually loads and examines.
That is also why Google's “up to 88% fewer tokens” figure should be interpreted correctly: it reflects workloads where selective viewing provides a major advantage, particularly long videos. It is not a promise that every video request will use 88% fewer tokens.
Available now:
Coming soon:
That last use case could be especially interesting.
Instead of answering questions primarily from captions or metadata, Gemini could ground an answer in what is actually happening on screen.
Google positions Gemini 3.7 Flash near the accuracy-to-cost sweet spot for agentic video understanding: strong enough for demanding analysis while remaining inexpensive enough to use as a general workhorse.
The smaller Flash models use the same basic agentic loop with lighter reasoning budgets.
The usual Gemini video-input rules still apply. Large or reusable videos can go through the Files API, smaller clips can be sent inline, and YouTube support is limited to public videos.
Google's million-token context window can already accommodate very long videos under static processing, but agentic processing changes the goal.
Instead of asking, “Can this entire video fit inside the context window?”
developers can ask:
“How little of this video does the model actually need to watch?”
Video may be the perfect use case for agentic AI because video is incredibly uneven.
A two-hour recording might contain only three minutes that actually matter to the question being asked.
Most of the footage is informationally cheap.
A few moments are extremely valuable.
Any system that treats every second as equally important will eventually lose to one that knows when to skip ahead.
That is the bigger shift Google is making.
Gemini is no longer simply consuming a predetermined stream of sampled frames.
It can navigate the video itself.
Give it a two-hour keynote and ask for the three announcements that matter, and the model can scan the transcript, jump to the relevant slides, inspect the surrounding footage, and spend its tokens only where additional evidence could change the answer.
For developers building video search, education tools, meeting intelligence, sports analytics, screen-recording support systems, automated QA, or highlight generation, the implementation change may be surprisingly small:
Use "processing": "agentic" for the long videos.
Keep static processing where comprehensive frame coverage matters.
Then compare accuracy, latency, and token usage against your existing pipeline.
The important change is not that Gemini can watch more video.
It is that Gemini no longer has to watch everything. It can watch on purpose.
