Research
We run our own inference infrastructure. Not because it's trendy, but because sending client data to third-party APIs creates dependencies we're not willing to accept. Here's what the self-hosted landscape looks like right now, what we run in production, and the numbers behind our decisions.
February 2026 | Dry Ground AI Research
The Case
Every API call to a cloud provider is a bet. You're betting the price won't change. You're betting the model won't get deprecated. You're betting latency stays acceptable. You're betting your data handling meets your clients' compliance requirements.
For quick prototypes, those bets are fine. For production systems that process thousands of client communications daily, they're not.
Self-hosted inference gives us four things we can't get from APIs:
Client emails, documents, and communications never leave our infrastructure. No third-party data processing agreements to worry about. No shared GPU pools.
No rate limits. No noisy neighbors. No "the API is slow today" surprises. We control the hardware, we control the throughput.
We test, benchmark, and swap models on our own schedule. No waiting for a provider to support the model we need. No vendor lock-in on architecture decisions.
Per-token API pricing gets expensive fast at volume. Dedicated hardware is a fixed cost. The more you use it, the cheaper each inference gets.
Landscape
The story of 2025-2026 is Mixture-of-Experts (MoE). These architectures pack the knowledge of a massive model but only activate a fraction of the parameters per token. The result: big model quality at small model speeds.
Here's what we evaluated and what we're watching:
| Model | Architecture | VRAM (Quantized) | Self-Hostable | Notes | Our Status |
|---|---|---|---|---|---|
| Qwen3-30B-A3B | MoE (30B total, 3B active) | ~18 GB (Int4) | Yes | Best all-rounder. 90%+ of flagship quality at a fraction of the compute. | Our production model |
| Qwen3-Coder-Next | MoE (80B total, 3B active) | ~47 GB (Int4) | Yes | Sonnet 4.5-level coding. 70.6% SWE-bench Verified. Overkill for general ops work. | Watching |
| GLM-4.7-Flash | MoE | TBD | Yes | New contender generating community buzz. Benchmarks still maturing. | Watching |
| Gemini 3.1 Pro | Dense (cloud only) | N/A | No | 1M token context. 77.1% ARC-AGI-2. Cloud-only reference point. | Cloud baseline |
The community consensus is clear: Qwen3-30B-A3B is the default recommendation for most teams in 2026. It delivers 90%+ of flagship model quality at a fraction of the compute cost. For general-purpose business operations (classification, extraction, summarization, drafting), nothing else in the self-hosted space matches the ratio of quality to resource requirements.
Our Stack
We didn't just pick the popular model and call it a day. We tested three configurations of the same architecture in production conditions and let the numbers decide. Here's the progression:
Started here. 30B MoE at full precision. Quality was solid: 10/10 on our test suite. But VRAM consumption ate the entire GPU. Only 4 concurrent streams before we hit memory limits. Classification took 16 seconds because the base model would generate 500 tokens of reasoning before answering.
Good quality, terrible efficiency.
Dropped from BF16 to 4-bit integer quantization. VRAM fell from ~75 GB to ~18 GB. Throughput jumped. We went from 4 concurrent streams to 16 on the same hardware. Quality stayed almost identical: 10/11 tests passed. One edge case in meeting notes extraction broke.
75% less VRAM, 4x concurrency, near-identical quality.
Swapped the base model for the instruction-tuned release (Instruct-2507). Same quantization. This fixed the one failing test AND solved our biggest production headache: classification latency dropped from 10+ seconds to 334 milliseconds. The model learned to just answer instead of thinking out loud.
11/11 quality, 60x faster classification. This is production.
The model uses ~18 GB of the available 80 GB. That leaves 62 GB for KV cache, concurrent request batching, and headroom. This is why we can serve 16 concurrent streams without degradation. Most teams running dense models on the same hardware would max out at 1-2 concurrent users.
Benchmarks
These are not synthetic benchmarks. Every test maps to something our system actually does in production: triaging emails, extracting data from documents, drafting communications, generating code. We ran the full suite on our production hardware under normal operating conditions.
11/11
Quality Score
tests passed (100%)
28-30
Single Stream
tok/s throughput
16
Concurrent Users
simultaneous streams
402
Aggregate Throughput
tok/s at full load
334ms
Classification
email triage latency
~18 GB
VRAM Usage
on dedicated GPU
| Category | Tests | Result |
|---|---|---|
| Structured Output | JSON extraction from contacts, meeting notes | 2/2 passed |
| Classification | Email triage: urgent, newsletter, action-required | 3/3 passed |
| Summarization | Financial document summarization | 1/1 passed |
| Instruction Following | Format compliance, constraint adherence | 2/2 passed |
| Reasoning | Multi-step problem solving | 1/1 passed |
| Code Generation | Functional code output | 1/1 passed |
| Communication | Email drafting quality | 1/1 passed |
Medium generation (~128 tokens)
28 tok/s
Long generation (~512 tokens)
30.3 tok/s
For context: 30 tok/s is roughly 2,000 words per minute. Faster than any human reads.
The real test isn't single-user speed. It's what happens when 16 people hit the system at the same time.
| Concurrent Streams | Aggregate tok/s | Per-User tok/s | Avg Latency |
|---|---|---|---|
| 1 | 27.2 | 27.2 | 3,891ms |
| 2 | 56.2 | 28.1 | 3,582ms |
| 4 | 109.3 | 27.3 | 3,665ms |
| 8 | 205.8 | 25.7 | 3,824ms |
| 16 | 401.8 | 25.1 | 3,959ms |
Per-user throughput drops only 8% from 1 to 16 streams. Latency stays nearly flat. This is the MoE advantage: the model is small enough that vLLM can batch efficiently without memory contention.
This is the number that changed how our product works. Email classification (urgent / newsletter / action-required) dropped from ~16,000ms with the base model to 334ms with Instruct-2507. Same accuracy. 60x faster.
The base model would generate 300-500 tokens of chain-of-thought reasoning before giving a one-word answer. The instruction-tuned variant just answers: 2-3 tokens total. The thinking still happens internally, but the output is what you asked for.
At 334ms, classification is effectively real-time. Emails get triaged as they arrive. Actions get flagged immediately. The difference between "AI that helps" and "AI you have to wait for" lives in this gap.
Insights
Three months of testing, three configurations, thousands of inference runs. Here's what actually matters:
The biggest quality jump didn't come from a bigger model. It came from switching to an instruction-tuned variant of the same architecture. The Instruct-2507 release fixed the one test the base GPTQ model failed, and it did it with no increase in VRAM or meaningful change in throughput. If you're choosing between a larger base model and a smaller instruction-tuned one, pick the tuned one.
Going from BF16 to GPTQ-Int4 cut VRAM by 75% and increased throughput. Quality dropped by one test out of eleven, and that test was later recovered with instruction tuning. The community debates about quantization quality loss are largely academic for business workloads. At Int4, we can't find a meaningful difference in outputs a human would notice.
Going from 16 seconds to 334 milliseconds isn't an optimization. It's a different product. At 16 seconds, you batch classification and run it periodically. At 334ms, classification happens inline with every incoming event. The system stops being "AI that processes your email" and becomes "AI that triages your inbox in real time." This single improvement changed our product architecture.
30B parameters of knowledge, 3B parameters of compute. That ratio is what makes single-GPU deployment realistic. Dense models with comparable quality need multi-GPU setups, which means distributed inference coordination, higher failure surface area, and more operational complexity. Every top-performing self-hostable model released in the last six months has been MoE. This isn't a trend. It's the new baseline.
MMLU scores and SWE-bench numbers tell you something about general capability. They tell you nothing about whether a model can extract a phone number from a meeting transcript in valid JSON. Our test suite exists because standard benchmarks don't measure the things our product actually does. If you're choosing a model for production, build your own eval suite first.
Looking Ahead
Our current stack works. 11/11 quality, sub-second classification, 16 concurrent streams. We're not chasing new models for novelty. But the landscape is moving fast, and we have a process for staying current.
80B total, 3B active. Coding-focused MoE that scores 70.6% on SWE-bench Verified, putting it in Sonnet 4.5 territory. At ~47 GB quantized, it won't fit our current single-GPU setup alongside production workloads. But if we add a second GPU or a dedicated coding instance, this becomes interesting for code generation tasks.
Another MoE contender getting attention in the community. Early benchmarks look promising, but we're waiting for the dust to settle on quantized variants and real-world testing before we invest evaluation time. The model needs to prove itself on business workloads, not just academic benchmarks.
We don't swap models because something new came out. We swap when a new model passes all 11 quality tests, maintains sub-second classification, fits our VRAM budget, and shows measurable improvement on at least one dimension we care about. The bar is high on purpose.
We run our full 11-test quality suite automatically every week. Not because the model changes, but because everything around it does: vLLM updates, system configurations, driver versions. If quality regresses, we know about it before any client does. When a promising new model drops, we add it to the pipeline and let the numbers speak.
Methodology
Transparency matters. Here's exactly how these numbers were produced.
11 tests across 7 categories: structured output (JSON extraction), classification (email triage), summarization, instruction following (format and constraint compliance), multi-step reasoning, code generation, and email drafting. Each test has defined pass/fail criteria based on output correctness, format compliance, and factual accuracy. Tests are deterministic: same input, validated output structure.
Single-stream throughput measured at two generation lengths: medium (~128 output tokens) and long (~512 output tokens). We report tokens per second as measured by vLLM's internal counters, not wall-clock estimation. Each measurement is averaged across multiple runs.
We ramp from 1 to 16 simultaneous streams, each generating medium-length output. We measure aggregate throughput (total tok/s across all streams), per-user throughput, and average latency per request. All streams start simultaneously to simulate realistic burst patterns.
Measured end-to-end from request submission to complete response. This captures vLLM queue time, prompt processing, and token generation. The 334ms figure is the median across all three classification tests (urgent, newsletter, action-required).
All benchmarks run on dedicated GPU infrastructure with 80 GB VRAM. Inference engine: vLLM. Tests are automated and run weekly to catch regressions. The February 23, 2026 run reported in this article is a standard weekly benchmark, not a cherry-picked result.
Bottom Line
A year ago, self-hosted inference meant compromises. Smaller models, worse quality, operational headaches. That's no longer true. MoE architectures and mature quantization have closed the gap. You can run a model that passes 100% of real-world business tests, serves 16 concurrent users, and classifies in under 400 milliseconds. On a single GPU.
The tradeoff isn't quality anymore. It's operational maturity. You need to know how to evaluate models on your workloads, not on leaderboards. You need automated testing pipelines. You need to understand quantization tradeoffs and MoE architectures well enough to make informed decisions.
We've been running this for our clients through CompanyClaw. The numbers in this article come from the same infrastructure that processes real operations daily. If you want the full comparison tables across all three model configurations, check the detailed benchmark page.
Benchmarks conducted February 23, 2026 on dedicated GPU infrastructure (80 GB VRAM). Business task suite includes 11 real-world scenarios across structured output, classification, summarization, instruction following, reasoning, code generation, and communication. All tests run under production conditions with vLLM. External benchmark references (SWE-bench Verified, ARC-AGI-2) sourced from public model documentation as of February 2026.
We use cookies to improve your experience. Cookie Policy · Privacy Policy