IBM's MIA plugin for vLLM: steering model internals with CUDA graphs still on

On Oct. 21, 2026, IBM Research open-sourced MIA ("model internals aperture"). It is a plugin for the vLLM inference engine that lets developers read a model's internal states and steer them while the model generates text, without switching vLLM into its slower "eager" mode. IBM's own tests found it up to 20 times faster than its predecessor, vLLM-Hook. In those tests, steering added about 2% latency in some cases on small models and about 6% on Llama 70B. Those figures come from IBM alone. No outside benchmark has been published.

What changed on Oct. 21

IBM's blog post of Oct. 21, 2026, written by Kim Martineau, describes MIA as the newest version of vLLM-Hook. IBM is releasing it as open source at github.com/IBM/vLLM-Hook, under the Apache-2.0 license according to the repository. IBM's stated aim is to make "last-minute safety checks in vLLM more practical". Its examples include refusing malicious prompts, flagging hallucinated facts and adjusting a response's tone and accuracy.

The problem MIA solves

vLLM speeds up inference by recording GPU work as CUDA graphs and replaying them. IBM says most interpretability tools need custom Python code to run inside each model layer. That only works if graph capture and replay are disabled, which pushes vLLM into eager mode and gives up the speed benefit.

MIA works the other way round. IBM says it:

• captures and steers model internals before the graph is recorded, so replay stays on;
• uses a scheduler that maps the developer's selections to routing entries at fixed GPU memory addresses;
• uses a "courier" that moves the captured data to CPU memory together with the model's output.

IBM's performance figures

All figures below are as IBM reported them on Oct. 21, 2026. Each one answers a different question.

The repository README makes a more general claim. It says capture and steering reduce decode throughput by "a few percent" while keeping the engine's CUDA graphs.

IBM also says MIA outperformed two other tools:

• vLLM-Lens, a plugin built by the UK government that also uses Python hooks;
• EasySteer, which forks vLLM. IBM notes that a fork has to be kept in sync with vLLM's frequent releases.

The blog gives no numbers for either comparison. It also does not describe the workloads or batch sizes behind the 20x and 14x figures. IBM says those figures come from a technical report published in the repository.

What MIA can capture and steer

According to the README:

• Capture: attention patterns, attention heads, hidden states, and Q/K values.
• Steer: activation steering through configurable interventions.
• Demos: attention tracker (prompt-injection detection), CoRe reranker, activation steering, hidden-state probe, and hallucination and schema-linking tools.
• Parallelism: tensor parallelism works for hidden-state capture, Q/K capture and steering. Pipeline parallelism is not supported.
• Modes: offline use through MiaLLM, or served use through vllm serve with MiaClient.

Requirements before you try it

These come from the README, read on Oct. 25, 2026:

• Platform: Linux and an NVIDIA GPU with a CUDA 13 driver. Validated on an H100 80 GB.
• Software: validated on vLLM 0.29.0 with torch 2.13.0. Python 3.11 to 3.14 is supported, and 3.12 is validated.
• Memory: capture keeps a 4 GiB GPU buffer outside vLLM's gpu_memory_utilization share. The demos use 0.7 utilization and models up to 8B, so they need a 40 GB card or larger. On smaller cards, lower the utilization setting.
• Disk: about 12 GB for the environment and about 33 GB for the demos, plus the models.
• Gated models: Llama and Mistral require hf auth login.

CUDA graph modes

• The default cudagraph_mode is FULL_AND_PIECEWISE: full graphs for decode and piecewise graphs for mixed steps.
• FULL_DECODE_ONLY is accepted.
• FULL is accepted with a warning, because it can compute wrong attention on FlashAttention 3 under vLLM 0.29.
• PIECEWISE alone is refused.
• enforce_eager=True (vllm serve --enforce-eager) opts out of CUDA graphs. That brings back the eager-mode slowdown MIA is meant to avoid.

Install

Clone the repository, create a Python 3.12 environment, then run pip install -r requirement.txt and pip install -e . --no-deps. The plugin registers with every vLLM engine in that environment. To run stock vLLM there, set VLLM_PLUGINS=''.

Background

vLLM-Hook began as an attention-tracking tool for detecting prompt-injection attempts, built by IBM researchers Pin-Yu Chen and Irene (Ching-Yun) Ko. Their first paper, "vLLM Hook v0: A Plug-in for Programming Model Internals on vLLM" (arXiv 2603.06588, submitted Feb. 2, 2026), describes two kinds of use. "Passive" use probes internal states without changing the output. "Active" use changes those states to steer generation. IBM says vLLM-Hook is also used in its Steerability toolkit. It is linked to the Steerability Challenge, a NeurIPS 2026 competition offering $25,000 in prizes for steering methods that suppress dishonest behavior without degrading general capabilities.

Practical takeaways

• Decide what you need. If you need to monitor or steer internals in production vLLM serving, MIA is built for that. If you only analyze offline, the speed gain matters less.
• Check fit first. It is validated only on Linux, a CUDA 13 driver, vLLM 0.29.0 and an H100. Other setups are untested by IBM.
• Benchmark on your own workload. The 2% and 6% figures cover only the models IBM tested. The 20x and 14x figures compare MIA with other tools, not with plain vLLM.

Limitations: all performance numbers are self-reported by IBM, and no independent benchmark had been published at the time of writing.