From Backpropagation to AlphaFold: What Deep Learning Stood On

Why You Need to Know

A 1989 paper experimented with handwritten postal code digits provided by the US Postal Service (DOI: 10.1162/neco.1989.1.4.541). A single neural network, which is a computational structure made of very many simple calculation units linked together, learned the entire recognition task from end to end—taking in neat pictures of digits and figuring out what numbers they were.

Some 30 years later, an evaluation paper for a protein structure prediction competition included this sentence: a research group's deep learning method, which is a way of training artificial neural networks with multiple layers stacked up, consistently calculated structures with an accuracy matching experimental structures, which amounts to a solution to the classical protein folding problem, at least for single proteins (DOI: 10.1002/prot.26237).

What happened between postal codes and proteins? The history of deep learning is usually told through report cards. Instead of report cards, this article follows a different question: what did each paper stand on, and what did they say they solved?

This chain is not just something imagined afterward. The skeleton is revealed when you follow what each paper actually put in its reference list. The ResNet paper we will look at in turn later cites the 1989 postal code paper, the Transformer paper cites ResNet, and the AlphaFold paper cites both. This citation relationship was verified in the reference records of OpenAlex, an open academic database. A citation is merely a record of "having referenced," and does not imply the magnitude of influence or the ownership of an idea.

Along with the citation chain, what we will follow is the bottleneck. In their abstracts, each paper points to a specific bottleneck where the mainstream method of the time was stuck, and reports solving it. Sometimes that bottleneck was left by the immediately preceding link in the chain and sometimes not, so this article tracks the citation chain and the bottlenecks separately. The bottlenecks differed at each stage, such as whether training works, how computations are structured, how many examples are needed, and how high accuracy goes.

Starting Point: It Begins in Winter

The starting line of the lineage is right after failure.

In the late 1940s, a few people realized that computers were more than simple automated calculators. They believed that by running the right algorithms, computers could do things that were thought to require human intelligence. In 1952, Christopher Strachey completed the first artificial intelligence program to play checkers, and IBM's Arthur Samuel added to it the ability to learn from experience, namely machine learning. An algebra program built by a team at Carnegie Mellon reproduced 38 out of 52 proofs from a classical mathematics textbook. Intoxicated by this success, scientists poured out optimistic declarations. However, projects failed one after another, and the first "AI winter" arrived (DOI: 10.1093/oso/9780198853732.003.0005).

According to a study reconstructing the early history of artificial intelligence research at the University of Edinburgh, figures like Donald Michie and Christopher Longuet-Higgins established Edinburgh as a hub for artificial intelligence in Europe, and Edinburgh made progress in several fields despite the AI winter triggered by the Lighthill report (DOI: 10.1177/30504554261417567).

The foundation for computation had been laid earlier than that. John von Neumann's 1945 draft report on the EDVAC was influential enough that Alan Turing cited it in his proposed Automatic Computing Engine as a decisive source for understanding the nature and design of a universal digital computer (DOI: 10.1109/85.238389). In 1948, Claude Shannon incorporated into the general theory of communication the share that can be saved thanks to channel noise and the statistical structure of the original message (DOI: 10.1002/j.1538-7305.1948.tb01338.x). There were machines and there was the mathematics of information. What was missing was a way to make machines learn representations on their own, where representation refers to what numbers the machine internally converts and holds its inputs as.

1989: The Root of the Lineage — Pre-Narrowing What Can Be Learned Through Neural Network Architecture

The postal code paper that opened this primer is the root of the deep learning chain, and the later stages are places we return to repeatedly.

The paper makes two claims. One is that the generalization ability of a learning neural network can be greatly improved by embedding constraints derived from the task domain. Generalization is the ability to perform well on new data not seen during training, and constraints are prior knowledge that "for this task, the answer is likely to take this form." The other is that the pathway for putting those constraints into a backpropagation neural network is the architecture (DOI: 10.1162/neco.1989.1.4.541). Architecture is the structure of a neural network, a design determining how many layers to have and which computational units connect to which units.

Backpropagation is, as defined in "Neural Networks and Parameters — What It Means to Nudge Hundreds of Billions of Dials", a method that calculates the effect (gradient) that each parameter has on the loss by tracing backward from the loss measured at the output—in other words, how far the model's answer strayed from the correct answer—and this method itself was already known.

What this paper added is the idea of pre-narrowing what can be learned through the architecture. For instance, if connections are arranged so that each unit in a layer only receives points close to each other in an image, that layer focuses on patterns between nearby points from the start. Instead of writing knowledge down as rules, it plants them into the structure. The AlphaFold paper about 30 years later also notes that it embedded knowledge about protein structures into the design of the deep learning algorithm. This article reads that as a variation of the same idea.

1998: From Handwriting to Bank Checks — A Neural Network That Scans Images Through a Small Window

Nine years later, the same research stream returned with a much bigger claim. Multi-layer neural networks trained by backpropagation are among the most successful examples of gradient-based learning, and given the right architecture, they can automatically build complex boundaries capable of separating patterns like handwritten digits with almost no human tuning. And comparing several handwriting recognition techniques on a standard digit recognition task, they reported that a convolutional neural network specifically designed to handle the jaggedness of two-dimensional shapes outperformed all other compared techniques (DOI: 10.1109/5.726791).

A convolutional neural network is a network that moves a small window across an image, repeating the same calculation with the same weights at every position. Weights are numbers multiplied by each input. Whether a digit is on the left or the right side of the image, the same window recognizes the same shape. This is what the zip code paper meant by "building constraints into the architecture."

The check-reading system in the paper combined a convolutional neural network character recognizer with a technique for training the entire system all at once, and it was deployed commercially to process millions of checks a day.

The references in this paper include the 1989 zip code paper. This is the first record that the lineage is not just imagination.

The In-Between Years: When Depth Was the Problem

The path forward was not smooth. If neural networks that learn representations on their own had already won, the story would have ended in the 2000s.

The problem was depth. Adding more layers should have made it possible to represent more complex relationships, but as a 2016 paper stated in its opening sentence, deeper neural networks were harder to train (DOI: 10.1109/cvpr.2016.90). One workaround during this period was to change the training method. A Boltzmann machine is a type of neural network that learns what data looks like in order to generate similar data, and a 2012 study training these machines in multiple layers showed that a layer-wise pre-training step set the initial weights reasonably, making training efficient. Features found this way were very effective as initial values for the middle layers of a standard neural network that passes calculations only forward, which could then be refined further toward the correct answer (DOI: 10.1162/neco_a_00311). Here, pre-training shares only a name with the pre-training of language models; it is a step that trains layers one by one beforehand to establish a starting point.

On the theoretical side, regarding generative neural networks called deep belief networks, a proof emerged that a deep but narrow neural network requires no more parameters than a shallow one to acquire the ability to mimic any data distribution as closely as desired (universal approximation) (DOI: 10.1162/neco.2010.08-09-1081). This was a demonstration that depth was not disadvantageous in principle. However, theory and actual training succeeding were different problems, and this gap remained homework until the appearance of residual networks, which we will see later.

Earlier, in the 1980s, the neural network approach received a cold shoulder from the academic community studying language. Connectionism is an approach that attempts to explain the mind and language by connecting many simple units, like a neural network. A 1992 article looking back at the lukewarm reception language researchers gave to the arrival of connectionism notes that connectionism was regarded as "at best irrelevant to the goals of linguistic theory and at worst actively hostile to them." However, the author of that article themselves believed that the two approaches had more in common than conventional wisdom held, and stood on the side of describing their points of contact (DOI: 10.1093/oso/9780195076653.003.0004).

Constraints existed outside of neural networks as well. Early artificial intelligence explored symbolic, neural, and probabilistic approaches alike, but was tied down by hardware and data limitations. Microchips, GPUs, and later TPUs boosted computing power, and the internet resolved a long-standing data bottleneck, but the conclusion of an overview summarizing this period is that the biggest turning point came from innovations in neural network architectures (DOI: 10.1017/s1062798725100379).

2013~2014: Parts Called Word Representations, Games, and Learning Tools

What came next was not a single giant invention, but parts.

The first part is word representation. A 2013 preprint refined word embedding, a method of turning words into bundles of numbers to capture meaning relationships between words. It reported that training while discarding some frequent words (subsampling) speeds up training and makes representations more regular, and also introduced a simple alternative (negative sampling) to replace one of the existing methods. This paper clearly noted its own limitations. Word representations are blind to word order and fail to represent idioms, so even if you combine the meanings of "Canada" and "Air", you do not get "Air Canada" (DOI: 10.48550/arxiv.1310.4546). The Transformer paper four years later pointed to recurrent and convolutional structures in its abstract, and there is no mention in that abstract of targeting this word-order limitation.

The second part is the path from seeing to acting. Another preprint in the same year presented the first deep learning model to learn control rules (control policies) directly from game-screen pixels using reinforcement learning, which scores actions tried and adjusts actions in the direction of higher scores. The output was an estimate of the reward to be received in the future, and the structure combined a convolutional neural network with a variant of Q-learning, a method of reinforcement learning. It reports applying this to seven Atari 2600 games without changing the architecture or the learning algorithm (DOI: 10.48550/arxiv.1312.5602). Both the word representation paper and this paper are preprints that have not undergone peer review, and the performance comparisons with other methods were reported by the authors themselves.

The third part is the tool that keeps learning running. Adam in 2014 is one of the first-order optimization methods that uses only gradients—a tweaked version of gradient descent, which looks at the gradient and moves parameters little by little in the direction where loss decreases—and it adjusts step sizes by estimating values like the mean and spread of gradients during training. The authors report that Adam is simple to implement, computationally efficient, and uses little memory, fits well with problems with large data or parameters and problems where gradients are noisy or sparse, and generally requires almost no tweaking of human-preset settings (hyperparameters) (DOI: 10.48550/arxiv.1412.6980, also a preprint). This article views Adam not as a new idea, but as a tool that makes trying ideas cheap.

2016: The Wall Where Stacking Layers Deeply Stopped Training

The opening sentence of the 2016 residual neural network paper describes the bottleneck of this era. Deeper neural networks are harder to train.

Residual learning is a method where, instead of the layers making a brand-new output from scratch, they are made to learn only the difference (residual) to be added based on the input they receive. The input passes straight through to the next, and the layers take charge only of the share to be added to it. The authors presented broad empirical evidence that networks built this way are easier to optimize and that accuracy increases even when the depth is significantly increased.

They evaluated up to 152 layers on ImageNet, an image classification dataset, which was 8 times deeper than the VGG neural network while having lower complexity. An ensemble combining the answers of multiple networks recorded a 3.57% error on the ImageNet test set, taking first place in the ILSVRC 2015 classification task. On CIFAR-10, a small photo dataset, they also presented analyses of 100 layers and 1,000 layers, and report obtaining a 28% relative improvement on the COCO task of finding objects in photos (DOI: 10.1109/cvpr.2016.90).

These figures are all benchmark scores directly reported by the authors—in other words, the test results where multiple models compete for scores on the same problem. In the lineage, this paper's place lies in its diagnosis rather than its ranking. If a previous detour helped training by properly capturing the initial values of a deep network, residual learning changed what the layers themselves learn into the difference to be added to the input, and the authors reported that deep networks structured this way are easier to optimize.

And in the references of this paper, there is also the 1989 zip code paper. It is a citation that spans 27 years.

2017: Abandoning the Structure That Reads Words One by One

The 2017 transformer paper begins by pointing out the dominant architecture of the time. In tasks like translation that turn a line of words into another line of words, the prevailing models were complex recurrent neural networks or convolutional neural networks built with an encoder to read the input and a decoder to write the output. A recurrent neural network is a neural network that reads words one after another in order, passing the content read so far to the next step. The top-performing models also connected the encoder and decoder with attention. Attention is a mechanism that calculates on the fly how much attention to pay to which parts of the input, as explained in "How Generative AI Answers — A Sentence Built One Piece at a Time".

The proposal was radical: a simple structure built entirely out of attention while completely discarding recurrence and convolution, namely the transformer, which uses attention as its backbone. In experiments on two machine translation tasks, this model was reported to have better quality, to make it easier to split calculations into multiple branches and run them at the same time (parallelization), and to require significantly less training time. It scored 28.4 on WMT 2014 English-German using the BLEU translation score, beating previous best results including ensembles by more than 2 BLEU, and set a new single-model record of 41.8 on English-French by training for 3.5 days on 8 GPUs, with a training cost that was very small compared to the literature's best models. It was also successfully applied to English constituency parsing (DOI: 10.65215/2q58a426). This performance is also as reported by the authors.

From a lineage perspective, the word "parallelization" carries more weight than the translation scores. A recurrent structure requires processing the previous word before processing the next one. Breaking that chain allows calculations to run simultaneously across multiple branches, and the authors reported that training time was much shorter. The point that attention made parallelization possible is also highlighted by a survey preprint reviewing large language models (DOI: 10.36227/techrxiv.176784493.33333564/v1). The previous deep learning revolution overview also notes that transformer-based models underpin the large language models of the time when that review was written (DOI: 10.1017/s1062798725100379). This article reads this as the moment the door to the age of scale was opened.

And among the references of this paper is ResNet.

2020: A Grown Language Model Does New Tasks with Just a Few Examples

The 2020 GPT-3 preprint reports the results of scaling up the size of a language model. The standard at the time was to pretrain on a large corpus and then fine-tune for each task. This pretraining is the first step of learning, as defined in "How Generative AI Answers — A Sentence Built One Piece at a Time", where a model repeatedly guesses the next token across an immense amount of text, while fine-tuning is the work of polishing by training once more on curated examples. This approach still demanded fine-tuning data consisting of thousands to tens of thousands of examples per task. By contrast, humans can tackle new language tasks with just a few examples or a simple instruction.

The paper reports that scaling up the size of a language model dramatically improves few-shot performance across tasks. Few-shot means doing a task by looking at just a few examples, and that performance was at times able to rival the previous best fine-tuning methods. GPT-3 is an autoregressive language model with 175,000,000,000 parameters—meaning a language model that looks at preceding tokens and pieces together the next token one by one—and was 10 times larger than any previous non-sparse model. Without adjusting parameters using gradients and without any fine-tuning, tasks and examples were handed over written solely in text (DOI: 10.48550/arxiv.2005.14165). This phenomenon of tackling a new task using only examples in the text without altering parameters is in-context learning, as discussed in "How Guessing the Next Word Becomes a Conversation".

The same abstract did not shy away from limitations. There are datasets where few-shot learning still struggles, and datasets that suffer from methodological problems originating from training on web corpora. The authors themselves pointed out that this model can generate news articles that humans find hard to distinguish from human-written text, carrying social implications. An overview tracing the history of language models from the 1990s summarizes that scaling up deep neural language models brought about qualitative changes, making a new name—Large Language Models (LLMs)—necessary (DOI: 10.1007/978-3-031-54827-7_1).

2021: A 50-Year-Old Open Problem in Biology

The terminus of the lineage lies outside the testing ground of artificial intelligence, in a long-standing problem of biology.

The AlphaFold paper summarizes the problem thus: despite tremendous experimental effort, only roughly 100,000 distinct proteins have had their structures determined, which is a tiny fraction of the billions of known sequences. This is because determining a single structure takes months to years. The problem of predicting three-dimensional structures from amino acid sequences alone had been an open research problem for over 50 years. Existing methods fell far short of atomic-level accuracy, especially when homologous structures—that is, similar proteins whose structures are already known—were absent.

What the paper presented was the first computational method to consistently predict structures with atomic-level accuracy even when no similar structure is known. Validation took place outside its own laboratory, at the 14th Critical Assessment of Protein Structure Prediction (CASP14), where it significantly outperformed other methods with an accuracy rivaling experimental structures in a majority of cases. The foundation for this lay in embedding physical and biological knowledge about protein structures and multiple sequence alignment—that is, lining up similar protein sequences from multiple organisms side by side at matching positions—into the design of the deep learning algorithm (DOI: 10.1038/s41586-021-03819-2).

CASP is a blind test where participants are given only the sequences of proteins whose structures have not yet been revealed, and an independent evaluator grades them. The sentence quoted in the second paragraph of this article came from this evaluation. The same evaluation paper notes that other research groups also drastically improved their performance, and that there are cases where these models have already provided solutions to tricky crystal structure problems (DOI: 10.1002/prot.26237). The fact that this was not a grade self-assigned by the authors but a blind score by independent evaluators is what distinguishes this achievement from the other benchmark figures in this article.

And in the references of the AlphaFold paper, both ResNet and the transformer are present. The idea that crossed the wall of depth in image recognition and the idea that broke the chain of sequential reading in translation meet in the references of a paper tackling a half-century-old biological problem.

What the Lineage Tells Us

If you lay the chain out in a single line, it looks like this.

1989 backpropagation and architecture constraints → 1998 convolution and commercial deployment → 2013–2014 representation, behavior, and optimization components → 2016 breaking the depth barrier with residuals → 2017 ditching recurrence to secure parallelism → 2020 scale becomes capability → 2021 a fifty-year problem in a different field.

This chain is a chain of citations. Bottlenecks must be read separately. The bottleneck identified by each paper in its abstract was that of the mainstream method at the time. Deep networks were hard to train, and residuals eased that. Recurrent and convolutional neural networks were the mainstream, and Transformers discarded both to make parallelization easier. Fine-tuning required thousands to tens of thousands of examples per task, and GPT-3 scaled up to rival fine-tuning sometimes with just a few examples. AlphaFold surpassed the limits of existing methods that fell short of atomic resolution when homologous structures were absent. The Transformer cites ResNet, but the bottleneck the Transformer pointed to was that of recurrent and convolutional translation models. The bottleneck AlphaFold pointed to was also that of existing structure prediction methods. The word-representation paper's own notes on the limitations of word order do not lead to the next link within the abstracts cited in this article.

Citation graphs are the victor's record. Bibliographies show only the paths that survived. Countless branches that took different paths at the same time do not appear in this chain. The long tradition of trying to learn high-level representations in human-readable symbols from data is one of them, and a paper reviewing that tradition forecasts that symbolic approaches and neural network approaches can learn from each other to solve today's machine learning problems (DOI: 10.24963/ijcai.2020/678). Nor are the dozen or so papers featured here all of deep learning's turning points. A lineage is a narrative organized after the fact, and it differs from the terrain of that era.

Open Questions

Scaling laws are not laws of nature. In the words of "Neural Networks and Parameters", a scaling law is an observed trend in how much the loss drops when you increase model size, data volume, and compute. Reviewing the literature on compute-optimal training for transformers, a review preprint summarizes that performance improves predictably with the scale of three resources—number of parameters, data volume, and compute budget—while emphasizing that scaling laws are empirical tools for budget allocation and risk reduction, and their coefficients vary by situation (DOI: 10.2139/ssrn.7142719).

Emergence is still debated. Emergent abilities are, in the words of "Neural Networks and Parameters", abilities that small models lack but large models possess. A survey preprint covering abilities that seem to appear only at scale, such as multi-step reasoning and in-context learning, summarizes that this phenomenon is hard to understand and requires more rigorous study (DOI: 10.36227/techrxiv.176784493.33333564/v1). The objection that what looks like emergence may arise from the scoring metric is discussed in "Neural Networks and Parameters".

The gap with the brain remains. In the phrasing of one overview, artificial neural networks have reached human-level capabilities, but the way they acquire symbolic knowledge is fundamentally different from the brain, and they usually require vast amounts of learning to form stable internal representations. Humans exhibit "fast mapping", instantly pairing a new word with the object it refers to. Among computational models trying to reproduce this, models reflecting neurobiological principles have taken a first step, but they still fall short of explaining fast mapping across various communication and pragmatic contexts (DOI: 10.1515/lingvan-2024-0249).

Even the endpoint is not an endpoint. According to a preprint reviewing CASP15, AlphaFold2 remains superior to other approaches for single protein structures, but standard protocols and default settings yield top-quality results for only about two-thirds of the targets—that is, the proteins assigned in that round. The rest require extensive sampling, running predictions multiple times to pick candidates. The biggest advance in this round was a large jump in the calculation accuracy of complexes with multiple protein chains, but complexes generally fell short of the single-protein level. In RNA structures and protein-ligand complexes (small molecules attached to proteins), which were covered for the first time in the same round, classical methods matched experiments better than deep learning methods (DOI: 10.22541/au.169658651.11658182/v1).

A review preprint summarizing the limitations of AlphaFold2 notes the prediction of multiple forms adopted by the same sequence, the effects of point mutations, and difficulties in antigen-antibody interactions. It lists protein-DNA and protein-RNA complexes, nucleic acid structures, ligand and ion binding, post-translational modifications, and membrane planes of transmembrane regions as failing in prediction (DOI: 10.20944/preprints202601.0708.v1). Topological entanglements where chains are looped together were found in about 0.7% of the structures predicted by AlphaFold-Multimer, a shape that has never been observed in experimental structures to the authors' knowledge (DOI: 10.1101/2022.09.14.507881).

At the same time, the impact is real. The fraction of human proteins with known structures was 48% using only experimental structures and template-based modeling (models imitating similar proteins with known structures), but it rises to 76% when including AlphaFold predictions, and the proportion of the "dark proteome" with almost no known structure drops from 26% to 10% (DOI: 10.1101/2021.08.03.454980). In structural biology practice, it is used to design expression constructs, design de novo proteins, and interpret cryo-EM data. However, it is tied to training data, its usefulness is limited for predicting structural variants and folding flexibility, and it is also noted that cofactors, post-translational modifications, and complexes with oligonucleotides are missing (DOI: 10.1101/2022.04.07.487522). The AlphaFold follow-up literature cited in this paragraph and the preceding two paragraphs consists entirely of preprints prior to peer review.

The last cell of the lineage is always empty. The authors of the 1989 ZIP code paper could not have known that their paper would be cited in a paper on protein structure prediction some 30 years later. What we call the end of the chain right now is probably just someone else's intermediate term.

Source Papers

• "Backpropagation Applied to Handwritten Zip Code Recognition," Neural Computation (1989) — DOI: 10.1162/neco.1989.1.4.541 — Integrating task-domain constraints into the architecture to improve generalization, single network from normalized image to final classification.
• "Gradient-based learning applied to document recognition," Proceedings of the IEEE (1998) — DOI: 10.1109/5.726791 — Backpropagation multilayer neural networks, superiority of convolutional neural networks for 2D shape variability, graph transformer networks, commercial deployment processing millions of checks daily.
• "Distributed Representations of Words and Phrases and their Compositionality" (2013, preprint) — DOI: 10.48550/arxiv.1310.4546 — Skip-gram extensions, subsampling and negative sampling, inherent limitations such as word-order indifference and inability to express idioms.
• "Playing Atari with Deep Reinforcement Learning" (2013, preprint) — DOI: 10.48550/arxiv.1312.5602 — First deep learning model to learn control policies directly from raw pixels, CNN + Q-learning.
• "Adam: A Method for Stochastic Optimization" (2014, preprint) — DOI: 10.48550/arxiv.1412.6980 — Adaptive estimation of lower-order moments, simple implementation, memory efficiency, and minimal tuning.
• "Deep Residual Learning for Image Recognition," CVPR (2016) — DOI: 10.1109/cvpr.2016.90 — "Deeper neural networks are harder to train", reformulation into residual functions, 152 layers, 1st place in ILSVRC 2015 (author-reported).
• "Attention Is All You Need" (2017) — DOI: 10.65215/2q58a426 — Attention-only architecture completely omitting recurrence and convolution, parallelization and training time reduction, WMT14 results (author-reported).
• "Language Models are Few-Shot Learners" (2020, preprint) — DOI: 10.48550/arxiv.2005.14165 — 175 billion parameters, few-shot without fine-tuning, self-critique on methodological issues of web corpus training and societal impacts.
• "Highly accurate protein structure prediction with AlphaFold," Nature (2021) — DOI: 10.1038/s41586-021-03819-2 — Atomic-level accuracy without homologous structures, CASP14 validation, deep learning integration of physics and biology knowledge with multiple sequence alignments.
• "First draft of a report on the EDVAC," IEEE Annals of the History of Computing (1945) — DOI: 10.1109/85.238389 — Decisive source for general-purpose digital computer design.
• "A Mathematical Theory of Communication," Bell System Technical Journal (1948) — DOI: 10.1002/j.1538-7305.1948.tb01338.x — General theory of communication including channel noise and statistical structure of messages.
• "Artificial Intelligence Emerges," Poems That Solve Puzzles (2020) — DOI: 10.1093/oso/9780198853732.003.0005 — Strachey's 1952 checkers program, Samuel's machine learning, exaggerated optimism and the first AI winter.
• "A History of the Early Years of Artificial Intelligence at the University of Edinburgh," European Journal on AI (2026) — DOI: 10.1177/30504554261417567 — AI winter triggered by the Lighthill report, Edinburgh hub with Michie and Longuet-Higgins.
• "The Deep Learning Revolution in AI," European Review (2025) — DOI: 10.1017/s1062798725100379 — Two-phase transition driven by hardware and data constraints, GPUs, TPUs, internet data, and architectural innovation.
• "Connectionism without Tears," Connectionism Theory and Practice (1992) — DOI: 10.1093/oso/9780195076653.003.0004 — Cold reception of connectionism by the linguistics community in the 1980s and its reasons.
• "An Efficient Learning Procedure for Deep Boltzmann Machines," Neural Computation (2012) — DOI: 10.1162/neco_a_00311 — Weight initialization via layer-wise pretraining, effect of initializing feedforward hidden layers.
• "Deep Belief Networks Are Compact Universal Approximators," Neural Computation (2010) — DOI: 10.1162/neco.2010.08-09-1081 — Deep, narrow generative networks do not require more parameters than shallow ones.
• "From Deep Neural Language Models to LLMs," Large Language Models in Cybersecurity (2024) — DOI: 10.1007/978-3-031-54827-7_1 — History of language models since the 1990s, contribution of self-attention, qualitative changes born from scale.
• "Critical assessment of methods of protein structure prediction (CASP)—Round XIV," Proteins (2021) — DOI: 10.1002/prot.26237 — Blind tests and independent evaluation, resolution of the classical folding problem limited to single proteins.
• "Critical Assessment of Methods of Protein Structure Prediction (CASP) – Round XV" (2023, preprint) — DOI: 10.22541/au.169658651.11658182/v1 — Only about two-thirds of targets reach top quality with baseline protocols, major progress in complex accuracy.
• "The Evolution of the AlphaFold Architecture" (2026, preprint) — DOI: 10.20944/preprints202601.0708.v1 — Systematic organization of limitations such as multiple conformations, point mutations, antigen-antibody complexes, and nucleic acid complexes.
• "Topological Links in Predicted Protein Complex Structures Reveal Limitations of AlphaFold" (2022, preprint) — DOI: 10.1101/2022.09.14.507881 — Topological entanglements absent in experimental structures found in roughly 0.7% of predicted complexes.
• "The structural coverage of the human proteome before and after AlphaFold" (2021, preprint) — DOI: 10.1101/2021.08.03.454980 — Coverage 48%→76%, dark proteome 26%→10%.
• "The impact of AlphaFold on experimental structure solution" (2022, preprint) — DOI: 10.1101/2022.04.07.487522 — Utilization in expression construct design, de novo design, and cryo-EM interpretation, alongside limitations tied to training data.
• "Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice" (2026, preprint) — DOI: 10.2139/ssrn.7142719 — Scaling laws are empirical tools for budget allocation rather than laws of nature.
• "Emergent Abilities in Large Language Models: In-Context Learning, Chain-of-Thought Reasoning, and Scaling Laws" (2026, preprint) — DOI: 10.36227/techrxiv.176784493.33333564/v1 — Unpredictability of emergent abilities and the need for more rigorous study.
• "From neural matter to rapid symbolic learning in brains and artificial neural networks," Linguistics Vanguard (2025) — DOI: 10.1515/lingvan-2024-0249 — Differences in symbol acquisition between artificial neural networks and brains, failure to reproduce fast mapping.
• "A Brief History of Learning Symbolic Higher-Level Representations from Data (And a Curious Look Forward)," IJCAI (2020) — DOI: 10.24963/ijcai.2020/678 — Genealogy of various approaches attempting to learn symbolic higher-level representations from data.