Evaluation record ยท nemotron-3-ultra

Nemotron 3 Ultra

v20260604

NVIDIA

Modelopen-sourcemixture-of-expertshybrid-mamba-transformerlong-context
85
Strong
About This Model

NVIDIA's open frontier reasoning model, released 2026-06-04 to complete the Nemotron 3 rollout (Nano Dec 2025, Super Mar 2026): a 550B total / 55B active LatentMoE hybrid Mamba-Transformer under OpenMDW-1.1 with open weights, training data, and recipes. 1M-token context, 71.9% SWE-bench Verified (vendor), Artificial Analysis Index 48 โ€” the top-scoring US open-weight model โ€” with ~140 tok/s decode and the best non-hallucination score in its comparison set (78.7 AA-Omniscience).

Last Evaluated: July 9, 2026
Official Website

Trust Vector Analysis

Dimension Breakdown

๐Ÿš€Performance & Reliability
+

Strongest US open-weight model on the Artificial Analysis Intelligence Index (48, #9 overall) with class-leading decode speed (~140 tok/s) and a 1M-token context. Vendor benchmark peaks (SWE-bench 71.9) run a few points above independent multi-harness reproductions (65.0-70.4). Verbosity (~2.3x peer output tokens) is the main efficiency caveat.

task accuracy code

Vendor coding benchmarks cross-checked against independent multi-harness SWE-bench reproductions

Evidence
MarkTechPost - Nemotron 3 Ultra release coverage โ€” SWE-bench Verified 71.9, IOI 2025 570.0 (vendor-reported)
Digital Applied independent analysis โ€” SWE-bench Verified 65.0-70.4 across five agent harnesses in independent runs, slightly below the 71.9 vendor peak; Terminal-Bench 2.1 56.4 (vendor-stated)
mediumVerified: 2026-07-09
task accuracy reasoning

Independent aggregate intelligence index plus vendor reasoning benchmarks; some vendor numbers await third-party replication

Evidence
Artificial Analysis Intelligence Index (via Digital Applied) โ€” Intelligence Index 48 (#9 of 89): highest-scoring US open-weight model, 6 points behind open leader Kimi K2.6 (54)
MarkTechPost - Nemotron 3 Ultra release coverage โ€” PinchBench 90.0 held-out; post-trained with SFT, RLVR, and Multi-teacher On-Policy Distillation for long-running agent reasoning
mediumVerified: 2026-07-09
task accuracy general

Knowledge and long-context retrieval benchmarks from launch materials, corroborated by independent press analysis

Evidence
MarkTechPost - Nemotron 3 Ultra release coverage โ€” AA-Omniscience 78.7, the highest non-hallucination score in its comparison set; RULER 94.7 at 1M tokens
mediumVerified: 2026-07-09
output consistency

Cross-harness benchmark variance and community reports; only one month of production usage available

Evidence
Digital Applied independent analysis โ€” Stable results across agent harnesses but notably verbose: ~2.3x more output tokens than peer models
lowVerified: 2026-07-09
latency p50

Independent hosted-endpoint speed measurements plus vendor throughput comparisons

Evidence
Digital Applied independent analysis โ€” 140.3 tokens/s output (#7 of 89 models) with 1.33s time-to-first-token; several times faster than Kimi K2.6 (50-100 tok/s)
MarkTechPost - Nemotron 3 Ultra release coverage โ€” 5.9x throughput vs GLM-5.1 and 4.8x vs Kimi K2.6 at 8K input / 64K output (vendor)
mediumVerified: 2026-07-09
latency p95

Qualitative assessment from early independent benchmarking; p95 distributions not yet published one month post-launch

Evidence
Digital Applied independent analysis โ€” High decode speed offset by 2.3x output-token verbosity on reasoning tasks; per-provider tail latencies not yet broadly characterized
lowVerified: 2026-07-09
context window

Official specification from model card; note that individual API providers may cap served context

Evidence
Hugging Face Model Card โ€” 1M-token context window; RULER 94.7 at 1M tokens (some hosts serve reduced limits)
highVerified: 2026-07-09
uptime

Multi-provider availability assessment; only one month of hosted operating history

Evidence
OpenRouter model listing โ€” Day-zero availability across 25+ platforms (NVIDIA NIM, OpenRouter, Together AI, Fireworks, Perplexity, SageMaker JumpStart) plus self-hosting redundancy
mediumVerified: 2026-07-09
๐Ÿ›ก๏ธSecurity
+

Reasonable default safety stack including a dedicated companion content-safety model, but almost no independent red-team literature exists yet given the June 2026 launch. Open weights shift guardrail responsibility to deployers who fine-tune.

prompt injection resistance

Review of vendor safety documentation; independent OWASP LLM01 testing not yet available for this release

Evidence
Hugging Face Model Card โ€” Safety post-training documented; no published third-party injection red-team results one month post-launch
lowVerified: 2026-07-09
jailbreak resistance

Assessment of vendor guardrail stack; accounts for open-weight modifiability and absence of independent adversarial evaluations

Evidence
Digital Applied independent analysis โ€” NVIDIA ships a companion Nemotron 3.5 Content Safety model (4B, 23 safety categories, 12 languages, released 2026-06-02); base alignment is removable downstream given open weights
lowVerified: 2026-07-09
data leakage prevention

Analysis of deployment options: self-hosting gives full data isolation, hosted routes depend on provider policies

Evidence
Hugging Face Model Card โ€” Open weights allow fully self-hosted deployment with complete data control; hosted access follows each provider's data handling terms
mediumVerified: 2026-07-09
output safety

Review of default-weight safety behavior and the vendor's companion moderation stack

Evidence
NVIDIA Nemotron research page โ€” Safety-aligned release with documented post-training; pairs with the Nemotron 3.5 Content Safety classifier covering 23 categories across 12 languages
mediumVerified: 2026-07-09
api security

Review of API security features on NVIDIA NIM and principal third-party hosts

Evidence
OpenRouter model listing โ€” NVIDIA NIM and major hosts provide API-key authentication, HTTPS, and rate limiting via OpenAI-compatible endpoints
mediumVerified: 2026-07-09
๐Ÿ”’Privacy & Compliance
+

US-jurisdiction provider with unusually flexible residency thanks to open weights and 25+ hosts, including hyperscaler routes (SageMaker JumpStart) that carry their own certifications. No model-level HIPAA/FedRAMP; regulated buyers should deploy via certified infrastructure.

data residency

Review of hosting options and open-weight licensing; residency is fully controllable via self-hosting

Evidence
Digital Applied independent analysis โ€” Day-zero availability on OpenRouter, NVIDIA NIM, Together AI, Fireworks, Perplexity, and Amazon SageMaker JumpStart; OpenMDW weights allow deployment anywhere
highVerified: 2026-07-09
training data optout

Analysis of NVIDIA hosted-service terms plus the self-hosting option

Evidence
NVIDIA Privacy Policy โ€” NVIDIA hosted services do not train on API data by default; self-hosting removes the concern entirely
mediumVerified: 2026-07-09
data retention

Review of hosted-platform retention policies across the deployment spectrum

Evidence
NVIDIA Terms of Service โ€” Hosted retention follows NVIDIA or third-party host policies; open weights make retention fully deployment-dependent
mediumVerified: 2026-07-09
pii handling

Review of data protection capabilities and deployer responsibilities

Evidence
Hugging Face Model Card โ€” Customer responsible for PII redaction; NVIDIA documents data provenance and provides NeMo Guardrails tooling in the surrounding stack
mediumVerified: 2026-07-09
compliance certifications

Verification of provider infrastructure certifications versus model-service-level compliance

Evidence
NVIDIA Trust Center โ€” NVIDIA infrastructure holds SOC 2 Type II and GDPR programs; no model-service HIPAA/FedRAMP path โ€” regulated deployments inherit certifications from the chosen host (e.g., SageMaker) or self-hosted environment
mediumVerified: 2026-07-09
zero data retention

Review of data handling across NVIDIA NIM, third-party hosts, and self-hosting

Evidence
Hugging Face Model Card โ€” No formal zero-retention guarantee on hosted endpoints; self-hosting provides true zero external retention
mediumVerified: 2026-07-09
๐Ÿ‘๏ธTrust & Transparency
+

Transparency is this release's standout: weights, 20T-token training data, and recipes are all published under OpenMDW-1.1 โ€” materially beyond typical open-weight disclosure. Independent bias and calibration audits are still pending one month post-launch.

explainability

Evaluation of reasoning-trace accessibility and weight/recipe inspectability

Evidence
MarkTechPost - Nemotron 3 Ultra release coverage โ€” Reasoning model exposing chain-of-thought traces; fully inspectable weights and recipes when self-hosted
mediumVerified: 2026-07-09
hallucination rate

Non-hallucination benchmark results from launch materials, pending broader independent replication

Evidence
MarkTechPost - Nemotron 3 Ultra release coverage โ€” 78.7 on AA-Omniscience โ€” the highest non-hallucination score in its comparison set
mediumVerified: 2026-07-09
bias fairness

Review of vendor responsible-AI disclosures; third-party fairness evaluations pending given launch recency

Evidence
Hugging Face Model Card โ€” NVIDIA Responsible AI documentation and data provenance disclosed; independent bias audits not yet published for this release
lowVerified: 2026-07-09
uncertainty quantification

Qualitative assessment of confidence expression in outputs

Evidence
Hugging Face Model Card โ€” Expresses uncertainty within reasoning traces; calibration data not yet independently characterized
lowVerified: 2026-07-09
model card quality

Review of model card and technical documentation completeness

Evidence
Hugging Face Nemotron v3 collection โ€” Detailed cards for BF16, Base-BF16, NVFP4, and GenRM variants covering architecture (LatentMoE, MTP), training phases, data cutoffs (pre-train Sep 2025, post-train May 2026), and deployment recipes
highVerified: 2026-07-09
training data transparency

Review of published training datasets and recipes; best-in-class disclosure among frontier-scale models

Evidence
Hugging Face Base model card โ€” Openly released training corpora: 20T-token pre-training mix documented (Nemotron-CC, Common Crawl snapshots CC-MAIN-2013-20 through 2025-13), plus 10M SFT samples and 1M RL tasks published under OpenMDW-1.1
highVerified: 2026-07-09
guardrails

Analysis of built-in safety mechanisms in default weights and companion tooling

Evidence
Digital Applied independent analysis โ€” Safety alignment in released weights plus the companion Nemotron 3.5 Content Safety 4B classifier; removable by downstream fine-tuning
mediumVerified: 2026-07-09
โš™๏ธOperational Excellence
+

Strong launch operations: day-zero support across 25+ hosts and all major inference frameworks, plus a single NVFP4 checkpoint spanning Ampere to Blackwell. Ecosystem and monitoring scores held conservative pending more production history; OpenMDW-1.1 is permissive but newer than Apache 2.0.

api design quality

Review of API design and feature completeness across NIM and principal hosts

Evidence
OpenRouter model listing โ€” OpenAI-compatible endpoints via NVIDIA NIM and 25+ hosts with function calling and reasoning controls
mediumVerified: 2026-07-09
sdk quality

Review of inference-framework support and deployment tooling at launch

Evidence
Digital Applied independent analysis โ€” Day-zero support in vLLM, SGLang, and TensorRT-LLM; Hugging Face Transformers integration; single NVFP4 checkpoint spans Ampere through Blackwell
highVerified: 2026-07-09
versioning policy

Review of release cadence, family roadmap execution, and weight-availability guarantees

Evidence
NVIDIA Nemotron 3 family announcement โ€” Predictable staged rollout: Nemotron 3 Nano (Dec 2025), Super 120B-A12B (2026-03-11), Ultra 550B-A55B (2026-06-04)
NVIDIA Nemotron research page โ€” Open weights remain permanently downloadable, softening deprecation risk; note NVIDIA retired the prior Llama-3.1-based Nemotron line within about a year
mediumVerified: 2026-07-09
monitoring observability

Review of monitoring tools across deployment options

Evidence
OpenRouter model listing โ€” Usage dashboards on NIM and third-party hosts; full observability when self-hosting on vLLM/SGLang/TRT-LLM stacks
mediumVerified: 2026-07-09
support quality

Assessment of enterprise support tiers, documentation, and community responsiveness

Evidence
NVIDIA AI Enterprise โ€” Enterprise support with SLAs available via NVIDIA AI Enterprise; strong developer documentation and community channels
mediumVerified: 2026-07-09
ecosystem maturity

Analysis of derivative models, third-party hosting breadth, and tooling integrations; scored conservatively given launch recency

Evidence
Hugging Face Nemotron v3 collection โ€” Complete family ladder (Nano 30B-A3B, Super 120B-A12B, Ultra 550B-A55B) with early community quantizations (e.g., Unsloth GGUF); derivative ecosystem still nascent one month post-launch
mediumVerified: 2026-07-09
license terms

Review of licensing terms and restrictions

Evidence
Hugging Face Model Card โ€” OpenMDW License 1.1 (Linux Foundation): permissive, covers weights, data, and recipes, with patent-termination clause; less legally familiar to enterprises than Apache 2.0
highVerified: 2026-07-09
Strengths
  • +Top-scoring US open-weight model: Artificial Analysis Intelligence Index 48 (#9 of 89)
  • +Radical transparency: weights, 20T-token training data, and recipes all released under OpenMDW-1.1
  • +1M-token context with RULER 94.7 at full length
  • +Fast for its class: ~140 tok/s decode, 1.33s TTFT; vendor-reported 4.8-5.9x throughput vs open peers
  • +Best non-hallucination score in its comparison set (78.7 AA-Omniscience)
  • +Single NVFP4 checkpoint runs across Ampere, Hopper, and Blackwell; day-zero vLLM/SGLang/TRT-LLM support
  • +Completes a predictable family ladder (Nano 30B-A3B, Super 120B-A12B, Ultra 550B-A55B)
Limitations
  • !Very verbose: ~2.3x more output tokens than peers, inflating cost and end-to-end latency on reasoning tasks
  • !Trails the open-weight leader Kimi K2.6 by 6 Intelligence Index points (48 vs 54)
  • !Vendor benchmark peaks exceed independent reproductions (SWE-bench 71.9 vendor vs 65.0-70.4 across harnesses)
  • !Text-only: no image, audio, or video input
  • !Launched 2026-06-04: minimal independent red-teaming, bias audits, or production track record
  • !550B total parameters require serious multi-GPU infrastructure to self-host
  • !OpenMDW-1.1 license is permissive but less familiar to enterprise legal teams than Apache 2.0
Metadata
pricing
input: Free weights (OpenMDW-1.1); hosted ~$0.50 per 1M tokens (OpenRouter reference)
output: Hosted ~$2.20 per 1M tokens (OpenRouter reference); free-tier variant available
notes: Self-hosting is infrastructure-cost-only. Budget for ~2.3x peer output-token verbosity on reasoning tasks, which erodes headline per-token savings. Rates vary by host.
last verified: 2026-07-09
context window: 1000000
languages
0: English
1: German
2: Spanish
3: French
4: Italian
5: Japanese
modalities
0: text
api endpoint: https://integrate.api.nvidia.com/v1
open source: true
architecture: LatentMoE hybrid Mamba-Transformer: interleaved Mamba-2, MoE (512 experts, top-22 active), and select attention layers across 108 layers, with Multi-Token Prediction and NVFP4 4-bit pretraining
parameters: 550B total / 55B active

Use Case Ratings

code generation

71.9% SWE-bench Verified (vendor; 65.0-70.4 independent) with fast decode suits agentic coding; verbosity raises per-task output cost.

customer support

Capable but text-only, verbose, and primarily English-optimized; oversized for most support tiers.

content creation

Solid structured writing; reasoning verbosity requires prompt discipline for concise content.

data analysis

1M-token context with RULER 94.7 handles very large datasets and logs; text-only, so charts/images need preprocessing.

research assistant

1M context plus the best non-hallucination score in its set (78.7 AA-Omniscience) suits long-document research and long-running agents.

legal compliance

Compliance is deployment-dependent: viable via certified hosts (e.g., SageMaker) or self-hosting; no model-service certifications.

healthcare

No HIPAA path on NVIDIA-hosted endpoints; deploy in compliant infrastructure via open weights.

financial analysis

Strong reasoning and long-context document processing; independent calibration data still limited.

education

Clear reasoning traces aid tutoring, but verbosity and English-first coverage limit fit versus multilingual alternatives.

creative writing

Reasoning-optimized rather than prose-optimized; competent but not distinctive creative output.