AstaBrief 8B is an open-weights model from Ai2 that turns a research question plus retrieved paper excerpts into a cited report. Ai2 released it on October 2, 2026, under Apache 2.0, together with its training data. It powers Fast mode in Asta's "Generate a report" feature and averages 51.1 seconds per report, against 178.5 seconds for Asta's Claude-powered Thinking mode: about 3.5 times faster. On quality, the results are mixed. It is close to the other systems on Ai2's main benchmark and last of three on DeepScholarBench.
This report rests on Ai2's own announcement and model card. No independent evaluation has been published that we could find.
Key facts
- Model: 8 billion parameters, fine-tuned from Qwen3-8B with supervised fine-tuning (SFT) and then offline direct preference optimization (DPO).
- License: Apache 2.0 for the weights, on Hugging Face as
allenai/AstaBrief_8B. - Speed: 51.1 s vs 178.5 s per report across the full Asta pipeline.
- Quality: 87.0 on the SQA-CS2 test split, between DR-Tulu-8B (88.8) and Asta ScholarQA (86.2). On DeepScholarBench it scores 53.50, the lowest of the three.
- Intended use: research and education, under Ai2's Responsible Use Guidelines.
Why it is faster
Asta's Thinking mode runs a multi-stage pipeline that summarizes and clusters the retrieved snippets before writing. AstaBrief skips those stages and writes the whole report in one pass from the question and the snippets. Ai2 says this was possible "without sacrificing performance" on its tests.
Ai2 also mentions "nearly an order-of-magnitude reduction in report generation time" against the proprietary models it tracked. The end-to-end figure it reports for Asta is 3.5x.
Benchmark results
The model card compares three systems. DR-Tulu-8B is an earlier Ai2 model that showed reinforcement learning can improve long-form reports from open-weights models. Asta ScholarQA is the pipeline that underpins Asta's report feature, typically backed by Claude 3.5 Sonnet or 3.7 Sonnet.
| Measure | AstaBrief-8B | DR-Tulu-8B | Asta ScholarQA |
|---|---|---|---|
| SQA-CS2 dev | 86.3 | 86.5 | 87.6 |
| SQA-CS2 test | 87.0 | 88.8 | 86.2 |
| DeepScholarBench | 53.50 | 56.26 | 60.25 |
| Win rate vs Asta ScholarQA (dev) | 55% | 36% | n/a |
| Win rate vs Asta ScholarQA (test) | 72% | 54% | n/a |
SQA-CS2 is a set of computer science questions written by real users. The test split has 100 questions. On it, the three systems are within 2.6 points of each other. On the dev split, AstaBrief is last, by 1.3 points. DeepScholarBench, a benchmark from Stanford and UC Berkeley researchers built from recent arXiv papers, uses 63 queries in Ai2's evaluation. AstaBrief trails both rivals there. The two benchmarks use different metrics, so their scores cannot be compared with each other.
The win rates are pairwise judgments by an LLM against Asta ScholarQA. AstaBrief's DPO stage trained on pairwise preferences, so this test sits close to what it was optimized for. Treat the 72% with that in mind.
In a small human study, three researchers contributed 14 questions and ranked the three systems' reports. DR-Tulu won on overall preference. Two of the three researchers preferred AstaBrief on citation accuracy.
How AstaBrief 8B scores against two other systems
Sort Ai2's reported scores to see where AstaBrief 8B leads, where it trails, and how far apart the three systems are.
| System | SQA-CS2 test (100 questions) | SQA-CS2 dev | DeepScholarBench (63 queries) | LLM-judged win rate vs Asta ScholarQA (test) | LLM-judged win rate vs Asta ScholarQA (dev) | Source |
|---|---|---|---|---|---|---|
| AstaBrief-8B | 87.0 | 86.3 | 53.50 | 72% | 55% | |
| DR-Tulu-8B | 88.8 | 86.5 | 56.26 | 54% | 36% | |
| Asta ScholarQA | 86.2 | 87.6 | 60.25 | n/a (baseline) | n/a (baseline) |
These are Ai2's own 2025 results, not an independent evaluation. SQA-CS2 and DeepScholarBench use different metrics, so their scores can't be compared with each other. The table leaves out speed, the small human study and current frontier models.
How Ai2 trained it
Ai2 skipped reinforcement learning, which it says "can be unstable and expensive", and used SFT followed by DPO.
- Queries: user queries submitted to Ai2's ScholarQA-based system, filtered for quality, relevance and privacy, leaving about 90,000 research questions.
- SFT data: reports generated with Claude 3.5 Sonnet, Claude 3.7 Sonnet, o3, o4-mini and GPT-4.1. About 47,000 usable examples remained.
- DPO data: pairs of reports per query, from generators including o3, o4-mini, DeepSeek-V3 and DeepSeek-R1. GPT-4.1 and DeepSeek-R1 judged each pair. Ai2 says the judges agreed with human preferences 95% of the time, and it kept only pairs where both judges agreed. About 6,000 pairs remained.
The filter that mattered
Ai2 tested four signals for spotting weak training reports: output-to-input token ratio, citation relevance, citation density and citation diversity. Citation density gave the strongest gains. Ai2 reports that more aggressive filtering, combined filters and learning-rate sweeps added little.
Gains by stage
Scores on the SQA-CS2 test set, from the model card:
| Model | Average | Citation precision | Citation recall | Answer precision |
|---|---|---|---|---|
| Qwen3-8B (base) | 77.3 | 76.2 | 64.6 | 90.6 |
| AstaBrief-8B-SFT | 83.7 | 87.7 | 71.3 | 90.4 |
| AstaBrief-8B (DPO) | 87.0 | 90.5 | 78.2 | 89.0 |
Most of the gain is in citations. Answer precision slipped slightly across both stages.
Early usage in Asta
Ai2 reports that 374 Asta users have tried Fast mode. 29.1% used it on two or more days, and users created 3.67 report threads with it on average. 23% never switched back to Thinking mode. Another 18% switched between modes and used Fast mode for about 40% of their threads. Positive feedback was 84.2% for Fast mode and 85.2% for Thinking mode, which Ai2 calls "too sparse to draw strong conclusions."
How to run it
- Try it: Fast mode under "Generate a report" in Asta (asta.allen.ai).
- Download: the weights, the SFT checkpoint and the SFT, DPO and prompt datasets are in the
allenai/astabriefcollection on Hugging Face. - Build on it: Ai2's ai2-scholarqa-lib repository includes an example workflow that generates reports from your own PDFs.
The model does no retrieval. You supply the excerpts. The model card warns that "a different prompt or interaction format may lead to degraded or inconsistent behavior", so use its recommended prompt. In BF16, 8 billion parameters need roughly 16 GB for the weights alone (our estimate), plus memory for long inputs. Ai2 notes that institutions can run it behind their own firewall, which matters when research questions reveal sensitive or unpublished work.
Limitations
Ai2 ran its evaluations in 2025 and has not rerun them against current frontier models, and its metrics did not test whether a report overstates a study's scope.
