Research

Inference Benchmarks

Most AI benchmarks test things nobody actually does at work. We tested three model configurations on 11 scenarios pulled from real operations: classifying emails, drafting responses, extracting structured data, generating code, and reasoning through multi-step problems.

February 2026 | Dedicated NVIDIA A100 80GB | Qwen3 30B-A3B MoE Architecture

Methodology

Why these benchmarks exist

MMLU and HumanEval scores look great on a spec sheet. They don't tell you whether a model can triage your inbox or draft a reply that sounds like a human wrote it. So we built our own test suite.

11 scenarios, all pulled from daily operations. Every model gets run through these before it goes anywhere near production. We tested three configurations of the same architecture to see exactly what quantization and instruction tuning change in practice.

Model Configurations

Three configurations, one architecture

Same architecture across all three: 30 billion total parameters, only 3 billion active per token. You get the knowledge of a large model at the speed of a small one. We tested full precision, quantized, and instruction-tuned variants to see where the tradeoffs actually are.

Baseline10/10

Qwen3 30B-A3B

BF16 (Full Precision)

PrecisionBF16

VRAM~75 GB

Medium throughput25.4 tok/s

Long throughput26.1 tok/s

Max concurrent4 streams

Peak aggregate89 tok/s

Classification latency~16,000ms

Classification tokens500+

Optimized10/11

Qwen3 30B-A3B

GPTQ-Int4 (Quantized)

PrecisionInt4 (GPTQ)

VRAM~18 GB

Medium throughput31.3 tok/s

Long throughput31.8 tok/s

Max concurrent16 streams

Peak aggregate456 tok/s

Classification latency~10,000ms

Classification tokens300+

Production11/11

Qwen3 30B-A3B Instruct-2507

GPTQ-Int4 (Quantized)

PrecisionInt4 (GPTQ)

VRAM~18 GB

Medium throughput28.0 tok/s

Long throughput30.3 tok/s

Max concurrent16 streams

Peak aggregate402 tok/s

Classification latency334ms

Classification tokens2-3

The big one: 60x faster classification

The base model would think out loud for 300-500 tokens before giving you a one-word answer. The instruction-tuned model just answers. Two tokens. Done.

Email classification went from ~16 seconds to 334 milliseconds. Same accuracy. That's the gap between AI that disappears into your workflow and AI that makes you stare at a spinner.

Quality Results

11 business tasks, head to head

Pass/fail accuracy and latency (ms) across all three configurations. BF16 baseline was tested for quality separately; latency shown for GPTQ and Instruct variants.

Test	BF16	GPTQ-Int4	Instruct-2507	GPTQ Latency	Instruct Latency
JSON Extraction (Contact)	✓	✓	✓	12,262ms	1,539ms
JSON Extraction (Meeting Notes)	✓	✗	✓	16,206ms	3,481ms
Email Classification (Urgent)	✓	✓	✓	16,590ms	492ms
Email Classification (Newsletter)	✓	✓	✓	8,704ms	501ms
Email Classification (Action Required)	✓	✓	✓	10,444ms	334ms
Financial Summarization	✓	✓	✓	9,113ms	2,869ms
Instruction Following (Format)	✓	✓	✓	11,581ms	3,728ms
Instruction Following (Constraint)	✓	✓	✓	17,092ms	878ms
Multi-step Reasoning	✓	✓	✓	13,312ms	819ms
Code Generation	✓	✓	✓	16,464ms	8,398ms
Email Drafting	✓	✓	✓	7,907ms	1,536ms
Total	10/10	10/11	11/11

Concurrency

Scaling under load

Aggregate throughput and average latency as we add concurrent streams. Full precision topped out at 4 before running out of VRAM. Quantized models handle 16 without breaking a sweat.

Streams	BF16 tok/s	BF16 latency	GPTQ tok/s	GPTQ latency	Instruct tok/s	Instruct latency
1	25	5119ms	30.1	4246ms	27.2	3891ms
2	46.3	5324ms	56.8	4508ms	56.2	3582ms
4	89.3	5683ms	112.1	4390ms	109.3	3665ms
8	—	—	216.6	4548ms	205.8	3824ms
16	—	—	456.2	4423ms	401.8	3959ms

Concurrency increase from BF16 to quantized

456 tok/s

Peak aggregate throughput (GPTQ, 16 streams)

Near-linear

Throughput scaling with minimal latency increase

Architecture

Why Mixture-of-Experts

30B knowledge, 3B cost

MoE architecture packs 30 billion parameters of knowledge but only fires 3 billion per token. You get the depth of a large model without paying the compute bill of one.

Single GPU, production scale

Quantized, the whole thing fits on a single A100 80GB with headroom for 16 concurrent streams. One GPU. No multi-node coordination. No distributed inference headaches.

Impact

What this means for clients

Real-time operations

Sub-second classification. Triage, routing, and flagging happen as events come in, not minutes later.

Scale without degradation

16 concurrent streams. Latency stays flat as load climbs. Nobody waits because someone else asked first.

Private infrastructure

These numbers come from dedicated hardware. Your data never touches a shared GPU pool.

Benchmarks conducted February 2026 on dedicated NVIDIA A100 80GB PCIe infrastructure. Business task suite includes 11 real-world scenarios across structured output, classification, summarization, instruction following, reasoning, code generation, and email drafting. All tests run under production conditions with vLLM 0.15.1.

We use cookies to improve your experience. Cookie Policy · Privacy Policy