Google Shipped Gemini 4 Hydrogen to Global Search Without Publishing One Benchmark

The newest Gemini is already powering Search globally, but Google published zero benchmarks, no MMLU, no SWE-bench, nothing. You can't diff a vibe against your eval set.

Google Shipped Gemini 4 Hydrogen to Global Search Without Publishing One Benchmark

A major version jump went straight into global Search with zero eval numbers. That omission is the story, and it should shape your adoption call.

Here is what we actually learned today: Google renamed its reasoning line from 2.0 to 4, dropped the word “Hydrogen” on it, and wired it into Search for what the blog calls a global rollout. Here is what we did not learn: how it does on anything. No MMLU-Pro, no GPQA, no SWE-bench, no coding pass rates, not even one of those stacked bar charts where the competitor always comes in second by a convenient margin. A company that has historically published benchmark tables at launch chose not to this time. Read that as data.

When OpenAI shipped GPT-4, you got a technical report thick with evals you could argue about. When Anthropic shipped Claude 3.5 Sonnet, you got agentic coding numbers and a public SWE-bench figure you could reproduce against your own harness. Imperfect, gameable, selectively chosen, sure. But a number you can check is a commitment. Hydrogen arrives with a demo reel and a Search feature, which is the launch posture of a product, not a measured capability. You can’t diff a vibe against your eval set.

The Search integration isn’t a side quest here; it’s the actual delivery vehicle. Putting the new model in front of billions of queries is cheap distribution and expensive honesty. It means the first real benchmark is production traffic, and you are the harness. Google gets a planet-scale A/B test on quality, latency, and hallucination rate before a single page of it is written down. That’s a smart way to launch a model you’re not yet confident enough to score. It is a lousy way to decide whether to rebuild your pipeline on it.

So let me be the architect who has to make the call. The question I get asked this week is “do we move our Gemini 2.0 workloads to Hydrogen.” The answer is not yet, and not because the model is bad, I have no idea if it’s bad, which is precisely the problem. Migration isn’t free. Prompts drift, tool-calling formats shift between major versions, structured-output schemas that were stable on 2.0 come back subtly malformed, and your retry logic starts eating cost you didn’t budget. You pay all of that up front against a capability delta nobody has quantified. A rational trade needs a number on both sides of the ledger. Right now one side is blank.

If and when the model lands in the Gemini API as a selectable version, the only move that makes sense is a shadow deployment: mirror a slice of real traffic, keep 2.0 as the control, and score both against the evals you already trust. Something like this, cheap to stand up:

snippet.pyPython
import google.generativeai as genai
​
CONTROL = "gemini-2.0-pro"      # your current baseline
CANDIDATE = "gemini-4-hydrogen" # swap to the exact API id when published
​
def shadow(prompt, tools=None):
    results = {}
    for name in (CONTROL, CANDIDATE):
        model = genai.GenerativeModel(name, tools=tools)
        r = model.generate_content(prompt)
        results[name] = r.text
    return results  # diff against your own graded eval set, not the demo

Grade the candidate on your tasks, the tool call that has to emit valid JSON, the retrieval question where 2.0 confidently invents a citation, the long-context summarisation where it quietly drops the middle third. If Hydrogen wins there by a margin that clears your migration tax, move. If it ties, stay. Don’t rearchitect on marketing.

Watch for the quiet things, too. Pricing and context window details weren’t disclosed, which matters because Search-scale inference has to be cheap, and cheap usually means a smaller or more aggressively quantised model than the headline number suggests. A reasoning jump big enough to earn a whole new major version should be loud about its reasoning. This one is loud about its search box.

The benchmarks will come. Vendors always publish eventually, once the production A/B test tells them which charts flatter them. Until then, treat Hydrogen as a Search feature that happens to have an API attached, and keep 2.0 in the control group. The model that ships without numbers is telling you to measure it yourself.