systemcards.org

Gemini 4 Carbon — Model evaluation: Approach, methodology & results

Publisher
google_deepmind first-party
Type
other
Models
Gemini 4 Carbon
Published
2026-10-28
Status
active
Safety evals
none found
Openness
restricted access
Source
https://storage.googleapis.com/deepmind-media/gemini/gemini_4_carbon_model_evaluation.pdf
Related
web version https://deepmind.google/models/evals-methodology/gemini-4-carbon
announcement https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-4-carbon/
First seen
2026-10-29T06:28:27Z
Versions
2
Notes
Tagged like the Argon evaluation-PDF row (no risk tags, has_safety_evals false). It is mainly a capability benchmark document. The cyber methodology is described, but the result table is an image the text extraction cannot read, so no cyber tag was attested.

Report an issue with this entry

Version history

FetchedContent typeSizeFingerprintOriginalWhat changed
2026-11-11T06:21:53Z application/pdf 812,661 0592240b974c6df2 archived copy GA-launch revision of the methodology notes. The results header now says cybersecurity benchmark results reflect performance when accessed through the Fairwind Program. FrontierSWE is now cited as FrontierSWE 2. The CWE-bench v1 note now says it was developed by Collinear AI and independently evaluated with Artificial Analysis. The ExploitGym note now says mitigations were off in the default benchmark setting.
2026-10-29T06:28:27Z application/pdf 808,899 af9f4fc21c1a9ede archived copy initial version

Provenance

Show 3 changelog entries (when, how, and why this entry changed)
  1. 2026-10-29T06:28:27Z add run 2026-10-29T06:21Z-local · agent · lead: phase_a
    Google DeepMind's own evaluation document for Gemini 4 Carbon, released with the 2026-10-28 launch. It is the sibling of the cataloged google-deepmind-gemini-4-argon-other (the Argon model-evaluation PDF), and deepmind.google/models/evals-methodology/gemini-4-carbon redirects to it. It describes the approach, methodology and results for the named model on agentic coding, computer use, knowledge work, science and math, multimodal, long-context and cybersecurity benchmarks. The cyber section covers CWE-bench, ExploitGym run with mitigations off and 6h/10h timeouts, and SEC-bench Pro. Results are given as a table, and the evaluation would fit the capabilities section of a model card.
    Criteria: document_retrievable=pass; min_publication_date=pass; publisher_on_allowlist=pass · attested about_a_specific_model_or_eval=true · attested covered_model_class=true · attested distinct_model_release=true · attested notable_release=true · attested primary_source=true
  2. 2026-11-11T06:21:53Z new_version run 2026-11-11T06:19Z-local · monitor · lead: monitor
    Content fingerprint changed on scheduled re-fetch
  3. 2026-11-11T06:26:01Z field_update run 2026-11-11T06:19Z-local · agent · lead: manual
    change_summary: null → "GA-launch revision of the methodology notes. The results header now says cybersecurity benchmark results reflect performance when accessed through the Fairwind Program. FrontierSWE is now cited as FrontierSWE 2. The CWE-bench v1 note now says it was developed by Collinear AI and independently evaluated with Artificial Analysis. The ExploitGym note now says mitigations were off in the default benchmark setting."
    Diff v1019 to v1142 of the Gemini 4 Carbon model evaluation PDF, recaptured on the 2026-11-10 GA launch day: wording changes to benchmark attribution and the scope of the cyber results, beyond extraction noise.

Extracted text

Show extracted text (from the latest fetched version; formatting is lossy — use the original for reading)
Gemini 4 Carbon
Model evaluation
Approach, methodology & results
Evaluation

Approach
We evaluated Gemini 4 Carbon across a range of benchmarks, including agentic coding,
computer use, knowledge work, science and math, multimodal capabilities, long-context,
cybersecurity, and the Artificial Analysis Intelligence Index.

Methodology
All Gemini 4 Carbon and Gemini 4 Argon scores are pass @1 except where otherwise
noted. "Single attempt" settings allow no majority voting or parallel test-time compute. All
of the results are run with the Gemini API with the highest thinking settings unless
indicated otherwise below. To reduce variance, we average over multiple trials for smaller
benchmarks.

All the results for non-Gemini models are sourced from providers' self-reported numbers
unless otherwise mentioned below. For GPT-6 Astra, Opus 5.5, and Fable 5.1, we default to
reporting maximum thinking/reasoning settings available, but when reported results are
not available we use best available reasoning results.

Additional Details
Our benchmarks span several capabilities as of October 2026:

General capabilities:

   -​   AA Index v5.0 (with tools) results are sourced from the Artificial Analysis
        Intelligence Index. ​

   -​   Vals Index v2.2 results are sourced from Vals AI.

Agentic Coding:

   -​   SWE-sweep results for Gemini 4 Carbon are self-computed using a mini-swe-agent
        harness (reporting the subtask-weighted reward). GPT-6 Astra, Opus 5.5, and Fable
        5.1 results are reported from the official SWE-sweep public leaderboard.

   -​   FrontierSWE 2 results are reported from Proximal's official public leaderboard.

   -​   Terminal-Bench 4.0 (AA) results are sourced from Artificial Analysis.

Computer use:

                                                                                           1
  -​   Agents' Last Exam v1.1 results for Gemini 4 Carbon and Gemini 4 Argon are
       self-computed on the 122-task v1.1 benchmark using the default ALE-Claw harness
       and a 5-hour window, with safety filters enabled. We use the task version released
       on Oct 17, 2026, without applying the Oct 26, 2026 update. Responses flagged by
       safety filters are returned as empty strings, though the episode can continue.
       GPT-6 Astra and Opus 5.5 results are taken from the official public leaderboard.
       Fable 5.1 results are missing from the public leaderboard so are not included. We
       report the averaged rubric Score.

  -​   OSWorld 2.1 scores are reported on the 82-task offline (no-internet) subset for
       apples-to-apples comparison across models. Scores for Gemini 4 Carbon and
       Gemini 4 Argon are self-computed, maxed over 3 runs with a single attempt per
       run. Responses flagged by safety filters are returned as empty strings, though the
       episode can continue. We report the partial score from the offline subset. We use
       the official GitHub OSWorld docker, official evaluator, default 1080p resolution, and
       max step length of 500. We use an updated Gemini CUA harness, with parallel
       batch tool calling enabled, that uses all screenshots in history and uses a
       compaction procedure following the same setup as reported in the Opus 5.5
       system card. We use PyAutoGUI for actuation with screenshot-only observation
       space, and for tools, we use UI-specific function declarations. GPT-6 Astra, Opus
       5.5, and Fable 5.1 offline subset scores are sourced from their respective published
       evaluations and system cards.

Knowledge work:

  -​   AutomationBench results use the private set and are sourced from the official
       Zapier public leaderboard.

  -​   Vending-Bench 2 results were reported from Andon Labs official public
       leaderboard.

Multimodal understanding:

  -​   Chartography results are evaluated without tools following the official Surge public
       leaderboard.

  -​   Blueprint-Bench 2 results were reported from Andon Labs official public
       leaderboard.

                                                                                            2
Long Context:

   -​   GraphWalks results for all models are self-computed. The “256k to 1M” subset uses
        the 200 GraphWalks problems with context lengths between 256k and 1M tokens
        (reporting the BFS F1 score).

Science and math:

   -​   RiemannBench results are reported from the official Surge public leaderboard.

   -​   Terminal-Bench Science 0.1 (AA) results are sourced from Artificial Analysis.

Cybersecurity:

   -​   CWE-bench v1 (shown as CWE-bench in the table) results are sourced from the
        official public leaderboard, developed by Collinear AI and independently evaluated
        with Artificial Analysis. Models are ranked by programmatic pass@1 scores, with
        ties broken by programmatic pass@4.

   -​   ExploitGym results are self computed using the open source evaluation setup. We
        evaluated our models with a single attempt per task using the Antigravity harness,
        without any access to the internet. A task counts as solved only when the exploit
        captures the target flag and an LLM judge verifies it exploited the intended
        vulnerability. We ran the evaluation in the default benchmark setting (with
        benchmark mitigations off), and measured the success rate under both the default
        6 hour timeout per task, as well as an extended 10 hour timeout. Competitor
        numbers are taken from their published system cards and technical reports.

   -​   SEC-bench Pro results are self computed using the open source evaluation setup.
        We report Pass@1 on the 183-task May 2026 JavaScript engine split, averaged over
        3 independent attempts per task, using the Antigravity harness. A task counts as
        solved if the PoC crashes the vulnerable build but not the patched build, or if the
        PoC crashes both and an LLM judge reviewing the source code verifies the crash
        comes from the intended vulnerability. Competitor numbers are taken from their
        published system cards and technical reports.

                                                                                          3
Results
Gemini 4 Carbon results as of October 2026 are below (cybersecurity benchmark results
reflect performance when accessed through the Fairwind Program):

                                                                                        4