Sources are selected for direct architectural, empirical, theoretical, or interpretability relevance. Every entry records the version reviewed and its main limitation.
Last reviewed: September 3, 2026
16primary and official sources
9peer-reviewed
5preprints
2official statements
01 / Indexed sources
Research library
Showing 16 of 16 primary sources
PEER-REVIEWEDRecurrent neural network
Architectural Complexity Measures of Recurrent Neural Networks
Saizheng Zhang, Yuhuai Wu, Tong Che, Zhouhan Lin, Roland Memisevic, Ruslan Salakhutdinov, Yoshua Bengio
Summary. Defines recurrent depth as an over-time nonlinear-complexity measure for RNN architectures, alongside feedforward depth and recurrent skip coefficient.
Why it matters. It establishes that the phrase predates the current Transformer discussion and had a distinct formal meaning.
Main caveat. Its graph-theoretic RNN definition is not interchangeable with every present-day depth-recurrent Transformer design.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, Łukasz Kaiser
Summary. Applies a recurrent self-attention transition across depth and introduces a per-position adaptive halting mechanism.
Why it matters. It is an important shared-computation and adaptive-depth precursor to modern looped architectures.
Main caveat. Universal Transformers have their own recurrence and halting details; they are related to, not identical with, all later recurrent-depth language models.
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut
Summary. Studies cross-layer parameter sharing as one of two parameter-reduction techniques for BERT-style encoders.
Why it matters. It shows a major Transformer lineage in which layer parameters can be reused across depth.
Main caveat. Cross-layer sharing alone is not proof of the inference-time recurrence, dynamic stopping, or latent-reasoning claims studied in later systems.
Looped Transformers are Better at Learning Learning Algorithms
Liu Yang, Kangwook Lee, Robert Nowak, Dimitris Papailiopoulos
Summary. Tests looped Transformers on in-context data-fitting tasks and reports comparable performance to standard Transformers with substantially fewer parameters in the studied settings.
Why it matters. It connects recurrence with the inductive structure of iterative learning algorithms.
Main caveat. The experiments concern selected data-fitting tasks and should not be generalized to all language-model workloads.
Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Tom Goldstein
Summary. Introduces the 3.5-billion-parameter Huginn proof-of-concept, trained from scratch on 800 billion tokens, and studies scaling inference work through a recurrent core.
Why it matters. It is the central empirical reference for the current prelude–recurrent-core–coda design and variable test-time recurrence.
Main caveat. Results are model- and benchmark-specific; equivalent-compute comparisons are not evidence that extra passes universally outperform larger models or improve every task.
Summary. Analyzes looped models on synthetic reasoning and one-billion-parameter language-model experiments, with theory relating effective depth and iterative algorithms.
Why it matters. It distinguishes parameter count from effective depth and offers controlled evidence for reasoning-oriented inductive bias.
Main caveat. Its theoretical simulation result does not mean a trained model’s hidden states automatically form a readable chain of thought.
Latent Chain-of-Thought? Decoding the Depth-Recurrent Transformer
Wenquan Lu, Yuechuan Yang, Kyle Lee, Yanshu Li, Enqi Liu
Summary. Uses several probing methods on Huginn arithmetic behavior and reports inconsistent evidence for a clean, token-like latent reasoning trajectory.
Why it matters. It directly cautions against drawing a readable hidden chain-of-thought from recurrence alone.
Main caveat. The analysis is limited to one model family, arithmetic tasks, and the probes used; it does not settle every interpretability question.
Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence
Sean McLeish, Ang Li, John Kirchenbauer, Dayal Singh Kalra, Brian R. Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Jonas Geiping, Tom Goldstein, Micah Goldblum
Summary. Studies converting existing pretrained Transformers into recurrent models using architectural surgery and a recurrence curriculum.
Why it matters. It asks whether recurrence can be added without repeating full pretraining from scratch.
Main caveat. Reported gains focus on selected mathematics settings and models around one billion parameters.
Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers
Harsh Kohli, Srinivasan Parthasarathy, Huan Sun, Yuekun Yao
Summary. Reports systematic-generalization and depth-extrapolation results in controlled multi-hop tasks, including gains from increasing recurrence at inference.
Why it matters. It directly studies the train-short, test-deeper question and documents an overthinking failure mode.
Main caveat. The models are trained from scratch on controlled tasks; excessive recurrence can degrade predictions.
Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers
Ying Fan, Anej Svete, Kangwook Lee
Summary. Trains looped hidden representations with parallel supervision aligned to explicit reasoning steps and tests math reasoning up to three-billion-parameter scale.
Why it matters. It shows that latent-state interpretability can depend on explicit training choices, not recurrence alone.
Main caveat. Its reported latency and accuracy gains are tied to the selected math tasks, supervision recipe, hardware, and baselines.
Looped Transformers under the Jacobian Lens: Does the Global Workspace Survive Recurrence?
Wenlong Wang, Fergal Reid
Summary. Adapts a Jacobian-lens analysis to looped architectures and compares representation access patterns in Ouro and Huginn with an untied baseline.
Why it matters. It is a current example of recurrence-specific mechanistic inquiry available immediately before this review cutoff.
Main caveat. The work is a new preprint, studies selected models, and its proposed workspace framing remains subject to replication and debate.
These articles explain the present attention around Astra. They are displayed separately because reporting—especially reporting attributed to an unnamed source—is not a public technical specification.
REPUTABLE REPORTING
OpenAI Technique in “Astra” Model Sparks Security Concerns
The publication reported, citing a person with knowledge of development, that Astra uses a limited recurrent-depth or looped-Transformer technique.
Caveat. This is reporting based on an unnamed source, not a public architecture specification from OpenAI.