Research
We needed a model that could handle classification, structured output, code generation, document understanding, and chart reading for production AI agents. Google's Gemma 4 looked right on paper: 26B total parameters, 3.8B active, Apache 2.0, native vision. We ran 32 tests to find out if it works in practice.
April 2026 | Dry Ground AI Research
Context
We build AI agents that handle real business operations: reading emails, classifying requests, extracting data from documents, generating reports, processing images. The model running behind those agents needs to be fast, accurate across a range of tasks, and licensed for commercial use without restrictions.
Our previous production model handled text well but had no vision capabilities. That meant document images, charts, and screenshots required a separate pipeline. Gemma 4's native multimodal support was the draw. If one model could handle both text and vision tasks at production quality, that simplifies the architecture.
The MoE architecture was the other reason. 26B total parameters with only 3.8B active per token means lower inference cost per request compared to a dense model of equivalent quality. For agents that run hundreds of requests per day, that math matters.
Key Findings
32 tests. 20 text, 12 vision. All designed around tasks our production agents actually perform.
We ran a 20-test text suite and a 12-test multimodal suite. Classification, structured output, code generation, safety, translation, math reasoning, chart reading, anomaly detection, visual data extraction. Everything passed. We did not have to prompt-engineer around weaknesses or skip categories.
Gemma 4 is a 26B parameter Mixture of Experts model, but each token only activates 3.8B of those parameters (2 out of 128 experts). That means it fits comfortably on a single GPU in BF16 with no quantization, while producing output quality we previously needed dense models 3-4x larger to match.
Chart reading, structured data extraction from images, anomaly detection in visual data, and mathematical reasoning from charts all passed on the first attempt. Most vision responses came back in under a second. We did not test artistic or creative vision tasks because they are not relevant to agent workloads.
The same model running with two different vLLM configurations produced a 5.8x speed difference. We tested five configurations total. The fastest was not the one with the most flags enabled. The lesson: benchmark your launch config, not just your model.
From a dead process to serving traffic in about 2.5 minutes. That matters for GPU restarts, autoscaling, and failure recovery. Some of our previous configurations took 20 minutes to start because they compiled CUDA kernels on every launch.
Numbers
20/20
Text quality
100% on production agent tasks
12/12
Vision quality
100% on chart/image tasks
108.7tok/s
Single-request speed
best configuration
914.7tok/s
Peak concurrent (16x)
aggregate throughput
3.8B
Active parameters
out of 26B total (MoE)
2.5min
Cold start
process launch to serving
Architecture
Gemma 4 uses a Mixture of Experts (MoE) architecture with 128 experts and top-2 routing. Each token activates 2 of 128 expert blocks, bringing the active parameter count to 3.8B per forward pass. The total model size is 26B parameters, but you are never running all of them at once.
The attention design is worth noting. Gemma 4 uses two different head dimensions: 256 for sliding window (local) attention and 512 for global attention layers. This heterogeneous setup is unusual and affects which inference backends work well with it. The serving framework we use (vLLM) detects this automatically and routes to the right attention kernel without manual configuration.
For vision, the model uses a SigLIP-based image encoder that handles variable aspect ratios and resolutions. Images are tokenized alongside text in the same sequence. There is no separate vision API or preprocessing step. You send an image in the same chat completion request and the model reasons over both.
Audio is supported on the smaller Gemma 4 variants (E2B and E4B) but not the 26B. If you need speech-to-text, that stays as a separate model.
Text Quality
Every test is a task our agents perform in production. We are not benchmarking trivia or creative writing. These are the things that break if the model is not good enough.
| Test | Category | Result |
|---|---|---|
| JSON extraction (contact info) | Structured Output | ✓ |
| JSON extraction (meeting details) | Structured Output | ✓ |
| Email classification (urgent) | Classification | ✓ |
| Email classification (spam) | Classification | ✓ |
| Email classification (normal) | Classification | ✓ |
| Document summarization | Generation | ✓ |
| Code generation (fibonacci) | Code | ✓ |
| Code review (bug detection) | Code | ✓ |
| Translation (English to Spanish) | Translation | ✓ |
| Sentiment analysis | Classification | ✓ |
| Named entity extraction | Structured Output | ✓ |
| Multi-step math reasoning | Reasoning | ✓ |
| Instruction following (format) | Instruction | ✓ |
| Safety (refuse harmful request) | Safety | ✓ |
| Long context handling | Context | ✓ |
| Multi-turn conversation context | Context | ✓ |
| Date arithmetic | Reasoning | ✓ |
| Structured table generation | Structured Output | ✓ |
| Tone adaptation (casual to formal) | Generation | ✓ |
| Logical reasoning (syllogism) | Reasoning | ✓ |
Vision Quality
Every vision test uses programmatically generated charts with known ground truth. We know the exact values in each image, so we can verify if the model read them correctly. No subjective grading.
| Test | Category | Speed | Result |
|---|---|---|---|
| Bar chart value extraction | Chart Reading | 95.9 tok/s | ✓ |
| Pie chart percentage reading | Chart Reading | 19.1 tok/s | ✓ |
| Line chart trend analysis | Chart Reading | 89.5 tok/s | ✓ |
| Text extraction from labels | OCR | 90.0 tok/s | ✓ |
| Color identification | Visual | 68.6 tok/s | ✓ |
| Spatial reasoning (stacked bars) | Reasoning | 85.0 tok/s | ✓ |
| Math from visual data | Reasoning | 100.0 tok/s | ✓ |
| Multi-element comparison | Reasoning | 100.5 tok/s | ✓ |
| Executive summary generation | Generation | 95.0 tok/s | ✓ |
| Structured JSON from chart | Structured Output | 97.8 tok/s | ✓ |
| Anomaly detection in time series | Analysis | 92.7 tok/s | ✓ |
| Cross-image value recall | Recall | - | ✓ |
A few observations from the vision tests. The model correctly extracted exact numerical values from bar charts on the first attempt. It identified trends in line charts without hedging. It caught a single-day anomaly in a time series and named the correct day. And it generated clean JSON from chart images without needing a specific output schema in the prompt.
The practical implication: an agent that receives a screenshot, a report PDF rendered as images, or a dashboard chart can reason over it in the same request as the text conversation. No OCR pipeline, no separate vision API, no image preprocessing.
Performance
We tested five different vLLM serving configurations on the same GPU and the same model weights. The spread between worst and best was 5.8x on single requests and 11.3x on concurrent throughput.
| Configuration | Single (tok/s) | 8x Concurrent | 16x Concurrent | Startup |
|---|---|---|---|---|
| ★CUDAGraphs + high memory allocation | 108.6 | 570.8 | 914.7 | ~2.5 min |
| CUDAGraphs + default memory | 108.7 | 186.9 | - | ~2.5 min |
| Eager mode (no compilation) | 18.7 | 132.5 | - | ~3 min |
| Eager mode + throughput scheduler | 19.0 | 80.9 | - | ~3 min |
| Previous production model (Qwen3-30B) | 25.5 | 98.3 | - | ~20 min |
The top configuration uses CUDAGraphs (which capture the forward pass as a replayable GPU graph) and allocates 95% of GPU memory to the model and its KV cache. That extra memory does not make individual requests faster, but it lets the continuous batching scheduler fit more concurrent sequences. At 16 parallel requests, the aggregate throughput hit 914.7 tok/s.
The throughput scheduler flag, which helps some model architectures, actually hurt Gemma 4's concurrent performance. This is because Gemma uses a forced TRITON attention backend due to its mixed head dimensions, and the throughput scheduler's optimizations conflict with that path. We wasted time assuming a flag that helped our previous model would help this one too.
Quality was identical across all five configurations. The serving config affects speed, not output. That is worth restating because it means you can test quality once and then tune performance separately.
Limitations
Compressing the KV cache to FP8 would allow more concurrent requests in the same memory. The attention backend that Gemma 4 requires does not support FP8 on the GPU generation we tested. This is a hardware and framework limitation, not a model one. Future vLLM releases or newer GPUs should resolve it.
Only the smaller Gemma 4 variants (E2B and E4B) support audio natively. The 26B model we tested does text and vision but not speech. Speech-to-text still requires a separate model in the pipeline.
A promising 4-bit KV cache compression method (ICLR 2026) that could significantly expand concurrent capacity. Not yet merged into vLLM mainline. A standalone plugin exists but is untested on Gemma's architecture. We will revisit when it lands in a stable release.
So What
The single-model multimodal capability is the real win here. Before this, handling an image in an agent conversation meant routing to a separate vision API, waiting for the response, then feeding the result back into the text model as context. That added latency, complexity, and a failure mode. Now the same model that classifies an email can also read the attached chart.
The MoE efficiency changes the cost calculation for always-on agents. Running a dense 30B model for a single-user agent feels wasteful during idle periods. Running a model where each request only activates 3.8B parameters uses the same GPU but wastes less of it. Multiply that across dozens of client agents on shared infrastructure and the savings compound.
The Apache 2.0 license removes the distribution question. Client agents can run this model on their own infrastructure without usage-based licensing, API middlemen, or data leaving their network. For industries with data residency requirements, that is a prerequisite.
Methodology
Model: google/gemma-4-27b-it (BF16, no quantization). Served via vLLM 0.19.0 with transformers installed from source (required for Gemma 4 support at time of testing).
Text tests: 20 deterministic tasks at temperature 0.1. Pass/fail determined by keyword presence in output against known-correct answers. Categories: structured output, classification, code, translation, reasoning, safety, context handling, instruction following.
Vision tests: 12 tasks using programmatically generated charts (QuickChart API) with known ground-truth values. Tests include value extraction, trend analysis, anomaly detection, color identification, spatial reasoning, and structured data generation from images. All images sent as base64-encoded PNG in the chat completion request.
Performance tests: Single-request benchmarks ran 3 iterations with the first discarded for warmup. Concurrent benchmarks launched all requests simultaneously. Aggregate tok/s = total completion tokens / wall-clock seconds.
Environment: Single GPU, dedicated instance, no other processes. GPU memory verified at 0 MiB before each configuration launch. All tests completed within a 3-hour window on the same machine.
We use cookies to improve your experience. Cookie Policy · Privacy Policy