> ## Content Index > Fetch the complete content index at: https://zipmex.com/llms.txt > Use this file to discover other available public pages before exploring further. # IBM's MIA plugin for vLLM: steering model internals with CUDA graphs still on - URL: https://zipmex.com/blog/ibm-says-its-mia-plugin-lets-developers-steer-vllm-model-internals-without-turning-off-graph-replay/ - Published: 2026-10-21T13:29:03.000Z - Updated: 2026-10-24T23:53:19.000Z - Description: IBM's open-source MIA plugin reads and steers vLLM model internals without eager mode. What it does, IBM's speed claims and what it requires. - Author: ZIPMEX - Tags: AI, #zipmex-newsroom On Oct. 21, 2026, IBM Research open-sourced MIA ("model internals aperture"). It is a plugin for the vLLM inference engine that lets developers read a model's internal states and steer them while the model generates text, without switching vLLM into its slower "eager" mode. IBM's own tests found it up to 20 times faster than its predecessor, vLLM-Hook. In those tests, steering added about 2% latency in some cases on small models and about 6% on Llama 70B. Those figures come from IBM alone. No outside benchmark has been published. ![Steering overhead per IBM: Llama 8B and Qwen 1.5B 2% in some cases; Llama 70B 6%. Bars compare the two model sizes.](https://storage.ghost.io/c/d4/f3/d4f34881-0300-4bbe-b8df-78a26d28be4d/content/images/2026/10/zipmex-aac1ffaf0fc0c6d6.png) IBM reports steering overhead of 2% in some cases for an 8B Llama and Qwen 1.5B, and 6% for a Llama 70B model. data: IBM Research blog, Oct. 21, 2026 · ZIPMEX chart; data: research.ibm.com ## What changed on Oct. 21 IBM's blog post of Oct. 21, 2026, written by Kim Martineau, describes MIA as the newest version of vLLM-Hook. IBM is releasing it as open source at [github.com/IBM/vLLM-Hook](https://github.com/IBM/vLLM-Hook?ref=zipmex.com), under the Apache-2.0 license according to the repository. IBM's stated aim is to make "last-minute safety checks in vLLM more practical". Its examples include refusing malicious prompts, flagging hallucinated facts and adjusting a response's tone and accuracy. ## The problem MIA solves vLLM speeds up inference by recording GPU work as CUDA graphs and replaying them. IBM says most interpretability tools need custom Python code to run inside each model layer. That only works if graph capture and replay are disabled, which pushes vLLM into eager mode and gives up the speed benefit. MIA works the other way round. IBM says it: - captures and steers model internals **before** the graph is recorded, so replay stays on; - uses a scheduler that maps the developer's selections to routing entries at fixed GPU memory addresses; - uses a "courier" that moves the captured data to CPU memory together with the model's output. ## IBM's performance figures All figures below are as IBM reported them on Oct. 21, 2026\. Each one answers a different question. | Figure (IBM's wording) | What it compares | | ---------------------------------- | ------------------------------------------------------------------------------------ | | Up to 20x faster | MIA vs. the first vLLM-Hook | | Up to 14x faster | MIA vs. a similar method that inserts code into the model | | About 2% overhead, "in some cases" | Steering cost on an 8B Llama and a Qwen 1.5B model | | About 6% overhead | Steering cost on Llama 70B | | 1.7x faster | First vLLM-Hook vs. the Hugging Face transformers library, for the attention tracker | The repository README makes a more general claim. It says capture and steering reduce decode throughput by "a few percent" while keeping the engine's CUDA graphs. IBM also says MIA outperformed two other tools: - **vLLM-Lens**, a plugin built by the UK government that also uses Python hooks; - **EasySteer**, which forks vLLM. IBM notes that a fork has to be kept in sync with vLLM's frequent releases. The blog gives no numbers for either comparison. It also does not describe the workloads or batch sizes behind the 20x and 14x figures. IBM says those figures come from a technical report published in the repository. ## IBM's four reported figures Which IBM figure measures which comparison or model size? | IBM figure | Figure as IBM states it | What it measures | Source | | -------------------------------- | -------------------------------- | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | MIA vs vLLM-Hook | up to 20 times faster | Speed against its predecessor | Source: [A new plugin can enable fast runtime monitoring and control (research.ibm.com), accessed Oct 21, 2026](https://research.ibm.com/blog/fast-interpretability-plugin-vllm?ref=zipmex.com) | | MIA vs code-insertion method | up to 14 times faster | Speed against a similar method that inserts code | Source: [A new plugin can enable fast runtime monitoring and control (research.ibm.com), accessed Oct 21, 2026](https://research.ibm.com/blog/fast-interpretability-plugin-vllm?ref=zipmex.com) | | Steering, 8B Llama and Qwen 1.5B | delayed by just 2% in some cases | Inference delay from steering, small models | Source: [A new plugin can enable fast runtime monitoring and control (research.ibm.com), accessed Oct 21, 2026](https://research.ibm.com/blog/fast-interpretability-plugin-vllm?ref=zipmex.com) | | Steering, Llama 70B | increased to only 6% | Steering overhead, larger model | Source: [A new plugin can enable fast runtime monitoring and control (research.ibm.com), accessed Oct 21, 2026](https://research.ibm.com/blog/fast-interpretability-plugin-vllm?ref=zipmex.com) | All four figures are IBM's self-reported results, and the table cannot estimate overhead on your own workload. > MIA (model internals aperture) is designed for vLLM, the main open-source inference engine serving enterprise workloads today. research.ibm.com, Oct 21, 2026 · [Read the source](https://research.ibm.com/blog/fast-interpretability-plugin-vllm?ref=zipmex.com) ## What MIA can capture and steer According to the README: - **Capture:** attention patterns, attention heads, hidden states, and Q/K values. - **Steer:** activation steering through configurable interventions. - **Demos:** attention tracker (prompt-injection detection), CoRe reranker, activation steering, hidden-state probe, and hallucination and schema-linking tools. - **Parallelism:** tensor parallelism works for hidden-state capture, Q/K capture and steering. Pipeline parallelism is not supported. - **Modes:** offline use through `MiaLLM`, or served use through `vllm serve` with `MiaClient`. ## Requirements before you try it These come from the README, read on Oct. 25, 2026: - **Platform:** Linux and an NVIDIA GPU with a CUDA 13 driver. Validated on an H100 80 GB. - **Software:** validated on vLLM 0.29.0 with torch 2.13.0\. Python 3.11 to 3.14 is supported, and 3.12 is validated. - **Memory:** capture keeps a 4 GiB GPU buffer outside vLLM's `gpu_memory_utilization` share. The demos use 0.7 utilization and models up to 8B, so they need a 40 GB card or larger. On smaller cards, lower the utilization setting. - **Disk:** about 12 GB for the environment and about 33 GB for the demos, plus the models. - **Gated models:** Llama and Mistral require `hf auth login`. ### CUDA graph modes - The default `cudagraph_mode` is `FULL_AND_PIECEWISE`: full graphs for decode and piecewise graphs for mixed steps. - `FULL_DECODE_ONLY` is accepted. - `FULL` is accepted with a warning, because it can compute wrong attention on FlashAttention 3 under vLLM 0.29. - `PIECEWISE` alone is refused. - `enforce_eager=True` (`vllm serve --enforce-eager`) opts out of CUDA graphs. That brings back the eager-mode slowdown MIA is meant to avoid. ### Install Clone the repository, create a Python 3.12 environment, then run `pip install -r requirement.txt` and `pip install -e . --no-deps`. The plugin registers with every vLLM engine in that environment. To run stock vLLM there, set `VLLM_PLUGINS=''`. ## Background vLLM-Hook began as an attention-tracking tool for detecting prompt-injection attempts, built by IBM researchers Pin-Yu Chen and Irene (Ching-Yun) Ko. Their first paper, "vLLM Hook v0: A Plug-in for Programming Model Internals on vLLM" (arXiv 2603.06588, submitted Feb. 2, 2026), describes two kinds of use. "Passive" use probes internal states without changing the output. "Active" use changes those states to steer generation. IBM says vLLM-Hook is also used in its Steerability toolkit. It is linked to the Steerability Challenge, a NeurIPS 2026 competition offering $25,000 in prizes for steering methods that suppress dishonest behavior without degrading general capabilities. ## Practical takeaways - **Decide what you need.** If you need to monitor or steer internals in production vLLM serving, MIA is built for that. If you only analyze offline, the speed gain matters less. - **Check fit first.** It is validated only on Linux, a CUDA 13 driver, vLLM 0.29.0 and an H100\. Other setups are untested by IBM. - **Benchmark on your own workload.** The 2% and 6% figures cover only the models IBM tested. The 20x and 14x figures compare MIA with other tools, not with plain vLLM. Limitations: all performance numbers are self-reported by IBM, and no independent benchmark had been published at the time of writing. Promotion · ZIPMEX ### Trade perpetual futures on ZIPMEX 135 marketsCrypto · stocks · gold · oil · FXPTS on every tradePTS → ZIPMEX tokens [See all markets](https://trade.zipmex.com/perps/markets?utm%5Fsource=zipmex-news&utm%5Fcampaign=perps&utm%5Fmedium=article&utm%5Fcontent=story-192961) 135 markets on trade.zipmex.com Perpetual futures are leveraged derivatives. Prices can move fast and you can lose all of your margin. Not available in every jurisdiction. Not investment advice.