IT-Bench SRE benchmark
43 AI models ranked on IT-Bench SRE. Each row shows the model’s score next to its cheapest API price per 1M tokens where it has one, so you can weigh quality against cost.
Top model today
GPT-5.6 Sol (OpenAI) leads at 56%, from $2.00 per 1M input OpenRouter routed quote.
Data via arXiv:2502.05352 →Models ranked
43
on this benchmark
Top score
56%
current leader
Type
Single eval
benchmark
Updated
Oct 14, 2026
benchmark scores
Cost vs IT-Bench SRE
42 priced modelsEach labelled name links to that model’s page. Curated to the cost-efficiency frontier, top scorers and cheapest so labels stay readable; the full field is in the leaderboard table below (and via “select all”). Reasoning models (extended “thinking” before answering) carry a ring around the dot.
Models ranked on IT-Bench SRE
highest score first · 43 models| # | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5.6 Sol |  OpenAI | 56% | $2.00 Routed quote | $10.00 Routed quote | |||||||||||||
| 2 | Step 5 Preview | Stepfun | 56% | n/a | n/a | |||||||||||||
| 3 | Gemini 3.8 Flash |  Google | 53% | $0.75 via Google | $3.75 via Google | |||||||||||||
| 4 | GLM 5.3 Flash |  Zhipu | 51% | $0.11 via Sail | $0.35 via Sail | |||||||||||||
| 5 | GPT-5.6 Terra | OpenAI | 51% | $2.00 via OpenAI | $12.00 via OpenAI | |||||||||||||
| 6 | Claude Fable 5.1 |  Anthropic | 50% | $10.00 via Anthropic | $50.00 via Anthropic | |||||||||||||
| 7 | GPT-6 Sol | OpenAI | 49% | $2.00 via OpenAI | $10.00 via OpenAI | |||||||||||||
| 8 | GPT-6 Astra | OpenAI | 49% | $10.00 via OpenAI | $50.00 via OpenAI | |||||||||||||
| 9 | Kimi K3 |  Moonshot | 48% | $0.70 Routed quote | $12.50 via Sail | |||||||||||||
| 10 | DeepSeek V4.1 Flash |  DeepSeek | 47% | $0.01 via Coralbricks | $0.60 via Sail | |||||||||||||
| 11 | Claude Opus 4.7 | Anthropic | 47% | $5.00 via Anthropic | $25.00 via Anthropic | |||||||||||||
| 12 | GLM 5.3 | Zhipu | 46% | $0.079 Routed quote | $3.08 via Sail | |||||||||||||
| 13 | GPT-5.5 | OpenAI | 46% | $5.00 via OpenAI | $30.00 via OpenAI | |||||||||||||
| 14 | GLM 5.2 | Zhipu | 43% | $0.05 Routed quote | $1.98 via Scx AI | |||||||||||||
| 15 | Qwen3.7 Max |  Alibaba | 42% | $1.25 via Novita | $3.75 via Novita | |||||||||||||
| 16 | Grok 4.7 |  xAI | 42% | $2.00 via xAI | $6.00 via xAI | |||||||||||||
| 17 | Gemini 3.5 Flash | Google | 40% | $1.50 via Google | $9.00 via Google | |||||||||||||
| 18 | GPT-5.6 Luna | OpenAI | 40% | $0.20 via OpenAI | $1.20 via OpenAI | |||||||||||||
| 19 | GLM 5.1 | Zhipu | 40% | $1.05 via DeepInfra | $3.50 via DeepInfra | |||||||||||||
| 20 | Qwen3.8 Max (0902) | Alibaba | 40% | $2.00 Routed quote | $6.00 Routed quote | |||||||||||||
| 21 | Claude Sonnet 4.6 | Anthropic | 40%reasoning: adaptive | $3.00 via Databricks | $15.00 via Anthropic | |||||||||||||
| 22 | DeepSeek V4 Pro 0423 | DeepSeek | 38% | $0.435 via Tencent | $0.87 via Tencent | |||||||||||||
| 23 | Claude Opus 5.5 | Anthropic | 38% | $4.00 via Anthropic | $20.00 via Databricks | |||||||||||||
| 24 | MiMo-V2.5-Pro |  Xiaomi | 38% | $0.435 Routed quote | $0.87 Routed quote | |||||||||||||
| 25 | Gemma 4 31B | Google | 37% | $0.14 via Amazon Bedrock | $0.40 via Amazon Bedrock | |||||||||||||
| 26 | Qwen3.5-27B | Alibaba | 35% | $0.26 via DeepInfra | $2.40 via Novita | |||||||||||||
| 27 | GPT-5.4 Mini | OpenAI | 35% | $0.75 via Databricks | $4.50 via OpenAI | |||||||||||||
| ||||||||||||||||||
| 28 | Qwen3.5 397B A17B | Alibaba | 34% | $0.164 via Aihubmix | $0.986 via Aihubmix | |||||||||||||
| 29 | Muse Spark 1.3 | Other | 33% | $1.25 via Azure | $4.25 via Azure | |||||||||||||
| 30 | Grok 4.3 | xAI | 33% | $1.25 via xAI | $2.50 via xAI | |||||||||||||
| 31 | DeepSeek V4 Flash 0423 | DeepSeek | 32% | $0.09 via DeepInfra | $0.18 via DeepInfra | |||||||||||||
| 32 | Kimi K2.6 | Moonshot | 31% | $0.65 via Wandb | $3.40 via Novita | |||||||||||||
| 33 | Gemini 3.1 Pro Preview | Google | 30% | $2.00 via Google | $12.00 via Google | |||||||||||||
| 34 | Step 3.7 Flash | Other | 30% | $0.20 via DeepInfra | $1.15 via DeepInfra | |||||||||||||
| 35 | Claude 4.5 Haiku | Anthropic | 27%reasoning: true | $1.00 via Replicate | $5.00 via Replicate | |||||||||||||
| 36 | MiniMax M2.7 |  MiniMax | 26% | $0.25 via DeepInfra | $0.55 via Pinstripes | |||||||||||||
| 37 | GPT-5.4 Nano | OpenAI | 24% | $0.20 via Databricks | $1.25 via Databricks | |||||||||||||
| ||||||||||||||||||
| 38 | Gemma 4 26B A4b | Google | 24% | $0.13 via Amazon Bedrock | $0.40 via Amazon Bedrock | |||||||||||||
| 39 | Qwen3.5-35B-A3B | Alibaba | 22% | $0.14 via DeepInfra | $1.00 via DeepInfra | |||||||||||||
| 40 | Grok 4.1 Fast | xAI | 18% | $1.25 via xAI | $2.50 via xAI | |||||||||||||
| 41 | gpt-oss-120b | OpenAI | 6% | $0.03 via Wandb | $0.17 via DeepInfra | |||||||||||||
| 42 | Nvidia Nemotron 3 Super 120B A12b |  NVIDIA | 1% | $0.085 via DeepInfra | $0.40 via DeepInfra | |||||||||||||
| 43 | Llama 3.3 70B Instruct |  Meta | 1% | $0.12 via Hyperbolic | $0.20 via Crusoe | |||||||||||||
| — | GPT-5.4 | OpenAI | 35%Artificial Analysis did not report this score in its latest run on Oct 14, 2026 | $2.50 via Databricks | $15.00 via OpenAI | |||||||||||||
| — | Nemotron 3 120B A12b | NVIDIA | 1%Artificial Analysis did not report this score in its latest run on Oct 14, 2026 | $0.50 via Cloudflare | $1.50 via Cloudflare | |||||||||||||
Frequently asked questions
What is the IT-Bench SRE benchmark?
IBM's IT-Bench benchmarks AI agents on real-world IT automation; its SRE (Site Reliability Engineering) subset presents Kubernetes-based incident scenarios that must be diagnosed and fully resolved, scored as a binary pass rate. State-of-the-art agents resolved only about 14% in the original paper.
Which AI model scores highest on IT-Bench SRE?
GPT-5.6 Sol (OpenAI) leads with 56%, from $2.00 per 1M input tokens as an OpenRouter routed quote, while OpenAI lists it at $4.00 per 1M input.
How many models are ranked on IT-Bench SRE?
43 models carry an IT-Bench SRE score. Scores come from Artificial Analysis; where a model has a listed API price it is drawn from LiteLLM and OpenRouter, and models with no current price are listed without one.
Other benchmarks
compare the same models on a different evalEvery weekday
AI moves fast. Here's your debrief.
News, analysis, tools, and more.
For people who build with AI