IT-Bench SRE benchmark
35 AI models ranked on IT-Bench SRE. Each row shows the model’s score next to its cheapest API price per 1M tokens where it has one, so you can weigh quality against cost.
Top model today
GPT-5.6 Sol (OpenAI) leads at 56%, from $2.00 per 1M input.
Data via arXiv:2502.05352 →Models ranked
35
on this benchmark
Top score
56%
current leader
Type
Single eval
benchmark
Updated
29 Aug 2026
last source fetch
Cost vs IT-Bench SRE
33 priced modelsEach labelled name links to that model’s page. Curated to the cost-efficiency frontier, top scorers and cheapest so labels stay readable; the full field is in the leaderboard table below (and via “select all”). Reasoning models (extended “thinking” before answering) carry a ring around the dot.
Models ranked on IT-Bench SRE
highest score first · 35 models| # | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5.6 Sol |  OpenAI | 56% | $2.00 | $10.00 | |||||||||||||
| 2 | GPT-5.6 Terra | OpenAI | 51% | $2.00 | $12.00 | |||||||||||||
| 3 | Kimi K3 |  Moonshot | 48% | $2.85 | $14.25 | |||||||||||||
| 4 | Claude Opus 4.7 |  Anthropic | 47% | $5.00 | $25.00 | |||||||||||||
| 5 | GPT-5.5 | OpenAI | 46% | $5.00 | $30.00 | |||||||||||||
| 6 | GLM 5.2 |  Zhipu | 43% | $0.61 | $1.98 | |||||||||||||
| 7 | Qwen3.7 Max |  Alibaba | 42% | $1.25 | $3.75 | |||||||||||||
| 8 | Gemini 3.5 Flash |  Google | 40% | $1.50 | $9.00 | |||||||||||||
| 9 | GPT-5.6 Luna | OpenAI | 40% | $0.20 | $1.20 | |||||||||||||
| 10 | GLM 5.1 | Zhipu | 40% | $1.05 | $3.50 | |||||||||||||
| 11 | Claude Sonnet 4.6 | Anthropic | 40%reasoning: adaptive | $3.00 | $15.00 | |||||||||||||
| 12 | DeepSeek V4 Pro 0423 |  DeepSeek | 38% | $0.43 | $0.87 | |||||||||||||
| 13 | DeepSeek V4 Pro (max) | DeepSeek | 38% | n/a | n/a | |||||||||||||
| 14 | MiMo-V2.5-Pro |  Xiaomi | 38% | $0.43 | $0.87 | |||||||||||||
| 15 | Gemma 4 31B | Google | 37% | $0.14 | $0.40 | |||||||||||||
| 16 | Qwen3.5-27B | Alibaba | 35% | $0.20 | $1.56 | |||||||||||||
| 17 | GPT-5.4 Mini | OpenAI | 35% | $0.75 | $4.50 | |||||||||||||
| ||||||||||||||||||
| 18 | GPT-5.4 | OpenAI | 35% | $2.50 | $15.00 | |||||||||||||
| 19 | Qwen3.5 397B A17B | Alibaba | 34% | $0.39 | $2.34 | |||||||||||||
| 20 | Grok 4.3 |  xAI | 33% | $1.25 | $2.50 | |||||||||||||
| 21 | DeepSeek V4 Flash 0423 | DeepSeek | 32% | $0.09 | $0.17 | |||||||||||||
| 22 | DeepSeek V4 Flash (max) | DeepSeek | 32% | n/a | n/a | |||||||||||||
| 23 | Kimi K2.6 | Moonshot | 31% | $0.65 | $3.40 | |||||||||||||
| 24 | Gemini 3.1 Pro Preview | Google | 30% | $2.00 | $12.00 | |||||||||||||
| 25 | Step 3.7 Flash | Other | 30% | $0.20 | $1.15 | |||||||||||||
| 26 | Claude 4.5 Haiku | Anthropic | 27%reasoning: true | $1.00 | $5.00 | |||||||||||||
| 27 | MiniMax M2.7 |  MiniMax | 26% | $0.25 | $0.55 | |||||||||||||
| 28 | GPT-5.4 Nano | OpenAI | 24% | $0.20 | $1.25 | |||||||||||||
| ||||||||||||||||||
| 29 | Gemma 4 26B A4b | Google | 24% | $0.13 | $0.40 | |||||||||||||
| 30 | Qwen3.5-35B-A3B | Alibaba | 22% | $0.14 | $1.00 | |||||||||||||
| 31 | Grok 4.1 Fast | xAI | 18% | $0.20 | $0.50 | |||||||||||||
| 32 | gpt-oss-120b | OpenAI | 6% | $0.03 | $0.17 | |||||||||||||
| 33 | Nvidia Nemotron 3 Super 120B A12b |  NVIDIA | 1% | $0.09 | $0.40 | |||||||||||||
| 34 | Nemotron 3 120B A12b | NVIDIA | 1% | $0.50 | $1.50 | |||||||||||||
| 35 | Llama 3.3 70B Instruct |  Meta | 1% | $0.12 | $0.20 | |||||||||||||
Frequently asked questions
What is the IT-Bench SRE benchmark?
IBM's IT-Bench benchmarks AI agents on real-world IT automation; its SRE (Site Reliability Engineering) subset presents Kubernetes-based incident scenarios that must be diagnosed and fully resolved, scored as a binary pass rate. State-of-the-art agents resolved only about 14% in the original paper.
Which AI model scores highest on IT-Bench SRE?
GPT-5.6 Sol (OpenAI) leads with 56%, from $2.00 per 1M input tokens.
How many models are ranked on IT-Bench SRE?
35 models carry a IT-Bench SRE score. Scores come from Artificial Analysis; where a model has a live API price it is verified against LiteLLM and OpenRouter, and models with no current price are listed without one.
Other benchmarks
compare the same models on a different evalEvery weekday
AI moves fast. Here's your debrief.
News, analysis, tools, and more.
For people who build with AI