Meta’s Galactica was trained on 106 billion tokens from 48 million scientific papers, then pulled from public access after just 72 hours in November 2022. The 120 billion-parameter model beat GPT-3 on LaTeX equation accuracy by 19.2 percentage points, yet drew criticism for generating fabricated citations. This article breaks down Galactica’s training data, benchmark performance, model architecture, and lasting effect on scientific AI development.
Galactica Statistics in 2026 – TL;DR
Galactica trained on 106 billion tokens across 4.25 epochs, processing roughly 450 billion total tokens.
The largest model variant had 120 billion parameters and required 128 NVIDIA A100 nodes.
On LaTeX equation prediction, Galactica scored 68.2% versus GPT-3’s 49.0%.
The public demo launched on November 15, 2022, and was withdrawn on November 18 — a 72-hour window.
Galactica models remain open source on GitHub and Hugging Face, where they have accumulated 2.74k stars.
Galactica statistics in 2026 reflect a model that no longer has a public-facing product but still gets research attention. The model outperformed PaLM 540B on the MATH benchmark [20.4% vs. 8.8%] and set state-of-the-art results on PubMedQA at 77.6%. Its architecture, a decoder-only Transformer with a 2,048-token context window, was sized so the flagship 120B model could fit on a single NVIDIA A100 node with 80GB memory. That decision was deliberate — Meta wanted the model accessible to academic labs with limited compute. Despite the demo removal, Galactica’s backlash directly shaped Meta’s release strategy for Llama, which has since crossed 650 million downloads.
How Much Data Was Galactica Trained On?
Galactica’s training corpus contained 106 billion tokens drawn from academic and scientific sources. The dataset mixed natural language content with domain-specific sequences — protein structures, chemical compounds, and mathematical notation. Meta called it “curated humanity’s scientific knowledge,” separating it from the web-scraped approach used by most general-purpose LLMs.
| Data Source | Content Type | Scale |
|---|---|---|
| Scientific Papers | Full-text academic papers | 48 million |
| In-Context Citations | Reference links within papers | 360+ million |
| Unique References | Normalized citation entries | 50+ million |
| Protein Sequences | UniProt amino acid sequences | 0.5 million |
| Chemical Compounds | PubChem SMILES strings | 2 million |
| Task-Specific Prompts | QA, summarization, entity extraction | 358 million tokens |
Source: Meta AI / Galactica research paper (arXiv: 2211.09085)
The corpus was restricted to open-access papers, so a large portion of science published behind paywalls was excluded. Only 2 million of 110 million PubChem compounds and 0.5 million of 227 million UniProt sequences made it into the training set. The model processed the full corpus across 4.25 epochs, totaling about 450 billion tokens seen during training.
Galactica Model Architecture and Variants
Meta released five Galactica variants. All used a decoder-only Transformer architecture with a 2,048-token context window and a vocabulary of 50,000 tokens. The models ranged from 125 million to 120 billion parameters, and performance scaled smoothly with size across scientific benchmarks.
| Model | Parameters | Training Hardware |
|---|---|---|
| GAL 125M | 125 million | Single GPU |
| GAL 1.3B | 1.3 billion | Multi-GPU node |
| GAL 6.7B | 6.7 billion | Multi-GPU node |
| GAL 30B | 30 billion | Multi-node cluster |
| GAL 120B | 120 billion | 128 NVIDIA A100 nodes (512 GPUs, 16,384 GB total memory) |
Source: Meta AI / Galactica research paper
The 120B model was sized to fit on a single A100 node with 80GB memory at inference time. Meta trained it using AdamW optimization with a linear learning rate decay to 10% of peak value. The architecture included specialized tokenization for SMILES (chemistry), amino acid sequences, DNA sequences, LaTeX, and citation markers.
Galactica Benchmark Performance Compared to Other Models
Galactica beat several larger models on scientific reasoning and knowledge tasks. The 30B variant outperformed PaLM 540B on mathematical reasoning despite being 18 times smaller. On medical QA tasks, the 120B model set new state-of-the-art scores.
| Benchmark | Galactica 120B | Comparison Model | Comparison Score |
|---|---|---|---|
| LaTeX Equations (434 eqs.) | 68.2% | GPT-3 | 49.0% |
| Chemical Reactions | 43.1% | GPT-3 | 35.1% |
| MATH (5-shot CoT) | 20.4% | PaLM 540B | 8.8% |
| Mathematical MMLU | 41.3% | Chinchilla | 35.7% |
| PubMedQA | 77.6% | Previous SOTA | 72.2% |
| MedMCQA Dev | 52.9% | Previous SOTA | 41.0% |
Source: Galactica paper (arXiv: 2211.09085)
Galactica also outperformed Bloom and OPT-175B on the BIG-Bench benchmark for general language tasks, despite not being optimized for them. The researchers attributed this to the higher quality of the curated scientific corpus compared to web-scraped data used by other open LLMs.
Galactica Citation Prediction Accuracy
Citation prediction was one of Galactica’s core features. Accuracy ranged from 36.6% to 69.1% depending on the evaluation dataset. The model’s weight memory outperformed traditional sparse and dense retrieval methods on citation tasks. A documented bias toward highly cited papers emerged — larger model variants showed stronger preference for well-known references, raising questions about reinforcing existing academic hierarchies.
Why Was Galactica Shut Down After 3 Days?
Meta launched the Galactica public demo on November 15, 2022, and pulled it on November 18. Critics quickly showed the model could generate plausible but wrong scientific content. Michael Black, Director of the Max Planck Institute for Intelligent Systems, warned the outputs could produce “deep scientific fakes.”
The model fabricated citations attributed to real researchers and generated fictitious abstracts that read like genuine papers. Yann LeCun, Meta’s chief AI scientist, argued Galactica was meant as a writing aid, not an autonomous content generator. Joelle Pineau, Meta’s VP of AI Research, later confirmed Galactica “was never meant to be a product.”
The backlash had lasting consequences. Yann LeCun said the cautious rollout of Meta’s later AI models was a direct result of the Galactica experience. Llama, released three months later in February 2023, used a much more controlled distribution strategy.
Galactica’s Influence on Scientific AI Development
Galactica’s models remain available on GitHub and Hugging Face under the Apache 2.0 license. The Hugging Face repository has 2.74k stars. OPI-Galactica-6.7B, a variant fine-tuned for protein-related tasks, was accepted at the NeurIPS 2024 Workshop on Foundation Models for Science.
GeoGalactica, a 30 billion-parameter derivative built on Galactica’s base, was trained on 65 billion tokens of geoscience text and fine-tuned with 1 million instruction pairs. It remains the largest language model built for the geoscience domain.
The model introduced several design ideas that later scientific LLMs adopted: specialized tokenization for chemical and biological sequences, a working memory token for step-by-step reasoning, and task-specific prompts baked directly into pre-training rather than added as a separate fine-tuning stage.
Scientific AI Adoption and the Problem Galactica Tried to Solve
The information overload problem Galactica aimed to address has only grown. ArXiv, the largest open-access preprint repository, crossed 3.05 million total submissions as of May 2026, based on arXiv’s own statistics page. Monthly submissions hit nearly 28,000 by late 2025, up from about 20,700 in October 2023.
Over 80% of published researchers now report using LLMs in at least one area of their work, according to a 2025 study cited in recent AI adoption research. AI paper growth in materials science increased 241-fold between 1980 and 2024. In physics, the increase was 307-fold over the same period.
| Metric | Value | Year |
|---|---|---|
| ArXiv total submissions | 3.05 million | May 2026 |
| ArXiv monthly submissions (peak) | ~28,000 | Late 2025 |
| Researchers using LLMs | 80.9% | 2025 |
| AI papers in materials science growth | 241x | 1980–2024 |
| AI papers in physics growth | 307x | 1980–2024 |
| LLM market projected value | $259.8 billion | 2030 |
Source: arXiv.org, Liao et al. (2025), Kong Enterprise Survey
Galactica Statistics vs. Current Frontier Models
Galactica’s benchmarks from 2022 have been overtaken. On MMLU, GPT-5.3 Codex scored 93% as of February 2026 — most top models now sit above 88%. On GPQA Diamond, a PhD-level science benchmark, Gemini 3.1 Pro Preview reached 94.3%. The gap between frontier models has narrowed considerably, with the Chatbot Arena Elo gap between first and tenth place shrinking from 11.9% in 2024 to 5.4% by early 2025.
| Model | MMLU Score | GPQA Diamond | Year |
|---|---|---|---|
| Galactica 120B | ~45% (science subset) | N/A | 2022 |
| GPT-4 | 86.4% | ~50% | 2023 |
| Claude Opus 4.5 | ~90% | ~70% | 2025 |
| GPT-5.3 Codex | 93% | ~85% | 2026 |
| Gemini 3.1 Pro Preview | ~91% | 94.3% | 2026 |
Source: llm-stats.com, Artificial Analysis, vals.ai (March 2026)
Galactica’s MATH benchmark score of 20.4% was state-of-the-art in late 2022. By 2025, frontier models were scoring above 70% on that same test. The rate of improvement has been steep — AI systems improved by 48.9 percentage points on GPQA and 18.8 points on MMMU in a single year across 2024.
Meta’s AI Investment After Galactica
Meta’s capital expenditures reached $39.2 billion in 2024, with projections of $60–65 billion for 2025. Meta AI monthly users reached 1 billion by May 2025 and grew to 1.2 billion by Q1 2026. Llama models have crossed 650 million downloads, averaging one million daily since February 2023.
In June 2025, Meta formed Meta Superintelligence Labs with 3,000 employees focused on artificial superintelligence research. The division is headquartered in Menlo Park. Meta’s broader AI investment strategy scaled rapidly — the $39.2 billion in 2024 capex was a 130% increase over two years.
FAQs
How many parameters does Galactica have?
Meta released five Galactica variants: 125M, 1.3B, 6.7B, 30B, and 120B parameters. The largest 120B model was sized to fit a single NVIDIA A100 node at inference.
Is Galactica still available in 2026?
Yes. The public demo was removed in November 2022, but all model weights remain open source on GitHub (galai repository) and Hugging Face under the Apache 2.0 license.
What was Galactica trained on?
Galactica trained on 106 billion tokens from 48 million scientific papers, textbooks, lecture notes, protein sequences, chemical compounds, and encyclopedias.
Why was the Galactica demo shut down?
Meta pulled the demo after 72 hours due to criticism that the model generated fabricated citations, fictitious abstracts, and plausible but incorrect scientific content.
How does Galactica compare to GPT-4 or newer models?
Galactica’s 2022 benchmarks have been surpassed. GPT-5.3 Codex scored 93% on MMLU in 2026 versus Galactica’s roughly 45% on the science subset. Frontier models have improved rapidly across all scientific reasoning tests.
Sources:
https://arxiv.org/abs/2211.09085
