Fetching Is Slower Than Computing — The Memory Bottleneck and HBM

The Chef Is Fast, but the Ingredients Are Slow

"Surging AI chip demand, HBM supply shortages." If you follow artificial intelligence news, you run into headlines like this often. Artificial intelligence is a job that does an immense amount of calculation, yet the news talks about memory just as much as it talks about the chips that calculate. Memory is a chip that stores data to be used in calculation and hands it out when needed. The HBM in the headline is a memory used attached right next to the calculating chip. What it looks like, we will see after finishing the restaurant story. This article covers why memory holds back the calculating work, and why we bother to attach memory right next to the calculating chip.

For the explanation, we will use a single restaurant. There is a very skilled chef in the kitchen. This chef can make 10 plates in 1 minute. The ingredients are in a warehouse across the street, and a single narrow hallway runs between the warehouse and the kitchen. Assuming 1 plate is made with 1 box of ingredients, 2 boxes come through this hallway in 1 minute.

Then, how many plates does this restaurant serve in 1 minute? By the chef's skill it is 10 plates, but since there are only 2 boxes of ingredients, it is 2 plates. The chef spends most of 1 minute empty-handed, looking toward the hallway.

This restaurant example and those numbers were made to help understanding. We will carry this restaurant to the end of the article. The side that determines the restaurant's speed is the hallway where ingredients come in, rather than the chef's skill. The exact same thing happens inside a computer, and HBM is a way to widen that hallway.

Chips That Compute and Chips That Hold

Inside computers and mobile phones, chips with different jobs live separately. Let's look first at the chip that acts like a restaurant chef. A processor is a chip in charge of computation. Both the central processing unit of a computer and chips in charge of AI computation are processors.

The chip that corresponds to a warehouse is the memory mentioned earlier. There are several kinds of memory, and the side that holds data used while computing is mostly DRAM. DRAM is a memory that holds data while the computer works, and when the power is turned off, the contents held inside disappear. Things that must remain even when the power is turned off, like photos or apps, are stored in other types of memory.

Since the two chips are separate, to compute, data must first be moved from memory to the processor. Data held in DRAM must cross over to the processor before computation begins, and that road is a small connecting bridge linking the chip and the outside—in short, a passage with a limited number of pins. This moving work takes both time and energy (arXiv:1802.00320). No matter how fast the chef is, the ingredients must cross the road to reach the cutting board.

To compute, you must first haul it over, and the road for hauling is narrow. This single line is the starting point of this article.

Only the Chef Got Faster

Let's go back to the restaurant. Suppose we replace the chef with someone more skilled who can make 20 plates a minute. If the ingredients coming down the hallway remain at just 2 boxes a minute, the restaurant still puts out only 2 plates a minute. Only the waiting time increases in proportion to how much faster the chef became.

There is a name for how much comes down the hallway in a minute. Bandwidth is the amount of data that can actually be moved in a unit of time. In terms of our restaurant, it is the number of boxes entering the hallway per minute, and in computers, it is written as "gigabytes per second." One gigabyte is an amount that can hold roughly 1 billion English alphabet letters.

The history of computing has flowed much like this restaurant. Over many years, DRAM technology developed slowly, and as a result, memory became an increasingly large bottleneck for the entire computer (arXiv:1802.00320). A bottleneck refers to the narrowest spot that holds back the overall flow, much like the narrow neck of a bottle. In our restaurant, the hallway is the bottleneck. Even as the processors used for AI computation keep growing in their computing power, they cannot raise their performance to match when tied down by memory bandwidth (arXiv:2602.18568). Simply adding more computing units is not the answer either. As computing units increase, they demand correspondingly more bandwidth, sometimes making the problem even worse (arXiv:2310.18181). It is just like hiring one more chef only to double the amount of ingredients that need to enter the hallway.

There is a name for this state as well. The memory wall refers to a state where computing units keep getting faster, but the speed of bringing data over from memory fails to keep up, so waiting rather than computation determines performance. Our restaurant is in that exact state. Whether you change the chef or hire more, it is still 2 plates a minute.

Countertop Within Reach

The first way to reduce waiting has been used for a long time. It is keeping frequently used ingredients within the cook's reach. If you keep ingredients that go into almost every dish, like salt or soy sauce, on the countertop, you do not have to walk to the pantry every time.

In computers, this countertop is called a cache. A cache is a small but fast storage space placed inside or right next to the processor, holding data that will be used again soon. A countertop usually has multiple tiers. A very small compartment right in front of your hands, a slightly larger compartment a bit further away, and a still larger compartment behind that—they are arranged in layers, with closer ones being smaller and faster.

Here we need to untangle an assumption laid down earlier. The restaurant example assumed "one box of ingredients per dish." A countertop proves useful when you can make multiple dishes with ingredients brought over just once. If you place a jar of salt on the countertop, you only make one trip down the hallway for the salt and share it across a hundred dishes. The larger the share of ingredients used repeatedly like this in a single dish, the fewer boxes travel back and forth down the hallway. Conversely, if every dish needs new ingredients and they cannot all fit on the countertop, the cook looks back down the hallway even with a countertop in place.

Actual calculations are just like this. Data brought from memory often has to cross the road directly without benefiting from the processor's cache (arXiv:1802.00320). In calculations searching for gravitational wave signals, when the data that must be handled at once exceeded cache capacity, instances of the processor grinding to a halt and waiting increased significantly (arXiv:2601.18835). If data does not fit on the countertop, the path back and forth to the pantry determines the speed all over again.

Transporting takes electricity along with time. Moving data from memory to the processor comes with a heavy energy burden (arXiv:1802.00320). When running large artificial intelligence models, data movement takes up the largest share of energy rather than computation, according to a 2026 review (arXiv:2608.28048, preprint not peer-reviewed). In a restaurant like this, the effort the cook spends carrying ingredients is greater than the effort spent cutting with a knife.

The countertop only works for ingredients used multiple times. The more ingredients that are used once and done with, the more the width of the hallway matters again.

Why Artificial Intelligence Is Thirsty for Moving Data

Now let's move to the star of the news, conversational artificial intelligence. When such an AI receives a question, it does not output the answer all at once; it builds it by stringing words together one by one. The information the AI has learned is contained in a huge amount of numbers, and this bundle of numbers is called a model. The data loaded from memory while generating words contains these numbers of the model (arXiv:2502.10659).

The problem is that the amount of computation performed is small compared to the amount fetched. Every time a single word is generated, almost all the numbers of the model must be loaded from memory again, and the computation done with each loaded number amounts to only a few multiplications and additions. When generating the next word, the exact same numbers are loaded yet again. Therefore, the stage of generating answers word by word involves little calculation using the fetched data, and performance relies heavily on memory bandwidth (arXiv:2604.04253). To put it in terms of a restaurant, it is closer to the case of "one box for a single dish" than keeping salt at the prep station and using it across a hundred plates. Because the model's numbers are too numerous to fit entirely on the prep station—meaning the cache—it is like hauling fresh ingredients from the warehouse for every single plate. The decisive constraint in running such large AI models quickly is memory rather than computation, and the previous review concludes that specifically the word-by-word generation stage is bound by bandwidth (arXiv:2608.28048).

There is an experiment that demonstrates this ceiling almost directly. First, to point one thing out, the bandwidth written on a chip is the maximum possible under ideal conditions, and actually using that much in practice is difficult. Small devices usually have a bandwidth of under 20 gigabytes per second. An AI model was loaded onto one such device, and the pathway through which data travels was tuned as much as possible. The result was around 5 words per second, and this speed was 85% of the theoretical limit permitted by the device's memory bandwidth (arXiv:2502.10659, single measurement taken on a small device). This means the device was receiving and using almost all the ingredients the hallway allowed. Any room to further increase speed on this device lies on the hallway side.

What if we make the chef faster? In a simulation where the processor's operating speed was increased, the stage of reading the entire question all at once before writing an answer became faster, but the gains in the word-by-word generation stage stopped at external memory bandwidth (arXiv:2512.22066, simulation). The question-reading stage is bound to the computation side because a single loaded number is used across multiple words of the question all at once, whereas the word-by-word generation stage is not. The authors noted that memory bandwidth acts like a performance ceiling. This looks just like our restaurant where upgrading the chef to produce 20 dishes a minute still resulted in only 2 dishes coming out.

The speed at which conversational artificial intelligence generates answers hits the limit of the width for hauling data from memory before it hits computing power. This is why AI news talks about memory so often.

There is also the opposite direction. If the hallway becomes wide enough, the chef determines the speed once again. In our restaurant, if the hallway brings in 20 boxes a minute, a chef who can make 10 dishes a minute becomes the limit once more. There is an analysis that in designs where bandwidth is greatly expanded to compute right beside stacked memory, many of the operations that generate words become bound to the computation side once again (arXiv:2604.04253).

Stacking a Warehouse Right Beside the Kitchen — HBM

There are two ways to widen a hallway. You can make the people carrying boxes run faster, or you can build more doors. In computer terms, that means increasing the speed sent over a single line, or laying down more lines connecting the memory and the processor.

Lying down many lines hit a wall. Ordinary memory sticks plugged into a computer motherboard, which are memory modules, struggle to exceed 50 gigabytes per second due to limits on the number of pins they can connect and problems with signal distortion (DOI: 10.1109/isscc.2018.8310257, background description from a 2018 paper). If you try to use multiple fast memories (graphics memory) placed alongside a graphics chip at the same time to increase bandwidth, power consumption rises and the wiring on the circuit board becomes densely tangled (DOI: 10.1109/isscc.2016.7418034). Even if you build hundreds of doors in a warehouse across the street, you cannot pave hundreds of hallways from those doors out onto the road to the kitchen.

HBM solves this problem by moving the warehouse. The name stands for High Bandwidth Memory, meaning high-bandwidth memory. Several layers of DRAM are stacked on top of each other, and this stacked block is attached right next to the computing chip, inside the same component (DOI: 10.23919/date51398.2021.9474024). In restaurant terms, rather than using a warehouse across the street, you have newly built a multi-story warehouse building right next to the kitchen wall.

The stacked layers must be connected vertically. Through-silicon vias are wiring created by drilling vertical holes that pierce a chip from top to bottom and filling them with a conductive metal such as copper (DOI: 10.1002/9781118760475.ch06). They are called TSVs for short. It is like drilling through the floor on every level of a warehouse building to make an elevator shaft, so that materials from upper floors come straight down without traveling around the perimeter of the building. HBM connects multiple layers of DRAM through these vertical pathways to increase the amount contained in a single block. That block sits side-by-side with the processor on a thin baseplate laid with dense wiring. As of a 2020 paper, the number of lines connecting the two is 1,024 (DOI: 10.1109/isscc19947.2020.9063110).

When there are this many lines, individual lines can be slow. Even though HBM has lower speed per line than ordinary DRAM, its total bandwidth is much higher (DOI: 10.1109/isscc19947.2020.9062977). In restaurant terms, instead of making people run, you put over a thousand doors in the kitchen wall and pass boxes slowly through each door.

The results come out in numbers. An announcement was made in 2024 that a single HBM block with 16 stacked layers of DRAM carries 1,280 gigabytes per second (DOI: 10.1109/isscc49657.2024.10454440). Because both the timing and the target differ from the earlier 50 gigabytes per second, it cannot be read as a simple multiple. Most of the numbers for HBM chips appearing in this article come from the International Solid-State Circuits Conference, a semiconductor circuit society, and are values measured and announced by the chipmakers themselves.

Would it not be closer if the warehouse were put right on top of the kitchen? The method of stacking memory directly on top of the processor yields higher performance but runs hotter, shortening its lifespan, whereas placing them side-by-side offers lower bandwidth but suffers less heat-induced aging and lasts much longer (DOI: 10.1109/irps.2016.7574618). The authors of that paper considered that stacking on top could also find a balance between the two if operated at lower voltage and speed. The HBM we saw earlier is of the side-by-side placement type on a baseplate (DOI: 10.1109/isscc19947.2020.9063110).

HBM is a method of stacking a warehouse, attaching it right beside the kitchen, and opening over a thousand doors to carry things widely all at once.

The Bill — Heat, Cost, and Capacity

Stacking warehouses and attaching them to the side comes with three bills attached.

The first bill is heat. When you pack multiple layers tightly into a narrow space, the paths for heat to escape narrow down. A three-dimensional stacked structure has the property of making heat difficult to escape, and poor thermal management threatens the reliability of components (DOI: 10.1115/1.4025531). Even in HBM, the higher the bandwidth, the more power consumption concentrates in one place, creating hot spots, which makes active thermal countermeasures necessary to manage them. This paper presents a circuit that adjusts memory operation according to the chip's temperature distribution (DOI: 10.1109/isscc.2016.7418034). The middle floors of a multi-story warehouse do not get good ventilation.

The second bill is cost. In stacked structures, defects occur where the vertical passages connecting layer to layer do not connect properly, and problems accumulate as the voltage drops slightly every time it passes through a layer. These problems drive up the total cost of HBM (DOI: 10.1109/isscc19947.2020.9062977). It is also pointed out that HBM is expensive and the manufacturing process is complex, making its yield—meaning the proportion of usable products among those manufactured—lower than that of ordinary DRAM (arXiv:2501.09605, the stance of the authors proposing a new memory).

The third bill is capacity. Memory stacked inside a component has a large bandwidth and requires little energy to retrieve data, but the amount it can hold is limited (DOI: 10.1145/3286475.3286484). A 2018 paper noted that the memory of the HBM-equipped graphics chip considered state-of-the-art at the time was only 4 gigabytes (DOI: 10.1109/tvlsi.2018.2791442). Since then, the number of layers has increased so that a single 16-layer block can hold 48 gigabytes (DOI: 10.1109/isscc49657.2024.10454440), but there is an analysis that large language models are growing in size so quickly that they have outpaced even the advancement of computing power and memory bandwidth (arXiv:2507.02456). A warehouse next to the kitchen is wide open, but it cannot be built as large as a warehouse across the street.

The remaining question is this: Can we expand capacity together while keeping bandwidth? The 2018 study concluded that more research is needed on the path of stacking DRAM higher, pointing to alternative bonding and stacking technologies that attach and connect layers as the key (DOI: 10.1145/3286475.3286484), and a 2026 outlook paper points to researching ways to obtain bandwidth comparable to HBM and 10 times the capacity using flash memory as a research task (arXiv:2601.05047). Flash memory is the type of memory mentioned earlier where photos and apps remain even when the power is turned off.

Let's Read a News Headline

Now, let's read a hypothetical article headline: "Expansion of 12-Layer HBM Supply, Competition for the Spot Beside the AI Chip."

The first question is what "12-layer" means. The H in notations like 8H and 12H used when referring to HBM stands for height, meaning how many tiers of DRAM are stacked (DOI: 10.1109/irps48228.2024.10529424). 12 layers means a warehouse with 12 floors. As the floors increase, the amount contained in a single unit increases, and problems like voltage drop that accumulate each time you pass a floor increase right along with it.

The second question is "Why the spot beside the AI chip?" While conversational artificial intelligence generates words one by one, what largely determines the speed is the width of the hallway, and a warehouse attached right next door widens that hallway. As the use of HBM surges due to demand from AI servers and supercomputers, the number of HBMs going into a single component is rising from around two to four to six to eight (DOI: 10.1109/irps48228.2024.10529424, 2024 paper). It is a picture of multiple warehouse buildings attached around the perimeter of a kitchen.

The third question is "What do we pay?" The headline does not answer this question, but the bill from the previous section does. The heat trapped between the stacked floors, the cost driven up by defects and voltage issues in the vertical pathways, and the limit on the amount that can be contained in a single unit.

When you hear the word HBM, you can think of three things: How many layers are stacked, how widely it carries things, and what price is paid in exchange.

Where to Look Next

This article used a single restaurant to cover only the meaning of the memory bottleneck and what HBM looks like. The article that traces, using numbers from actual papers, how HBM has widened with each generation over the past 10 years and the bills for heat, voltage, and capacity that come due the moment you stack it is "The Bottleneck Was Never Compute — Ten Years of Stacking Memory Beside the Processor".

The question of what it would be like to make memory do the math inside itself without moving data at all is continued by "Don't Move It, Compute Where It Sits — Six Questions Left After We Made Memory Do the Math". Why DRAM itself is getting harder to shrink and the new memories eyeing what comes next are covered by "Next-Generation Memory — The DRAM Wall and the Race for a "Dream Memory"".

Other articles in "Chips from Scratch" cover what a transistor, the smallest component of a chip, is ("A Transistor Is a Switch — A Tiny Faucet You Open and Close with Electricity"), what the "3 nanometers" in the news is the length of ("What Is "3 Nanometers" the Length Of? — Measuring a Chip's Name with a Tape Measure"), and how a chip's circuits are drawn with light ("Lithography — Painting Circuits with a Brush of Light").

Source Papers

• "Enabling the Adoption of Processing-in-Memory: Challenges, Mechanisms, Future Research Directions" (2018) — arXiv:1802.00320 — Years of slow DRAM development turned memory into an increasingly large bottleneck. Data must be moved to the CPU through pin-limited pathways before computation can begin, carrying heavy latency and energy burdens while often failing to benefit from caches.
• "RPU -- A Reasoning Processing Unit" (2026) — arXiv:2602.18568 — While GPUs continue to scale up compute volume, performance expansion is difficult in tasks bound by memory bandwidth. Proposes HBM that trades away capacity to gain energy and cost (simulation).
• "An Energy-Efficient Near-Data Processing Accelerator for DNNs that Optimizes Data Accesses" (2023) — arXiv:2310.18181 — Accelerators with increased computing units and on-chip buffers failed to solve the memory wall, and increasing compute units sometimes magnified the problem by also raising bandwidth demands.
• "Beyond FINDCHIRP: Breaking the memory wall and optimal FFTs for Gravitational-Wave Matched-Filter Searches with Ratio-Filter Dechirping" (2026) — arXiv:2601.18835 — The bottleneck in gravitational-wave matched-filter searches is memory bandwidth rather than compute throughput, and processors stall heavily when data exceeds cache capacity.
• "AI Hardware Accelerators for Large Language Models: Architectures and the Memory Wall" (2026) — arXiv:2608.28048 — The decisive constraint in large language model acceleration is memory, not arithmetic. The word generation stage is bound by bandwidth, and data movement dominates energy consumption (review paper, pre-peer-review).
• "Rethinking Compute Substrates for 3D-Stacked Near-Memory LLM Decoding: Microarchitecture-Scheduling Co-Design" (2026) — arXiv:2604.04253 — Word generation in language models has low arithmetic intensity, making performance extremely sensitive to memory bandwidth. If bandwidth is significantly widened through 3D-stacked near-memory processing, a substantial portion of word generation operations becomes compute-bound once again (analysis).
• "Pushing up to the Limit of Memory Bandwidth and Capacity Utilization for Efficient LLM Decoding on Embedded FPGA" (2025) — arXiv:2502.10659 — Measured roughly 5 words per second with a 7-billion-parameter model on a device under 4GB and 20GB/s, reaching 85% of the theoretical bandwidth limit.
• "Prefill vs. Decode Bottlenecks: SRAM-Frequency Tradeoffs and the Memory-Bandwidth Ceiling" (2025) — arXiv:2512.22066 — Raising the operating frequency speeds up the context input stage, but gains in the word generation stage are blocked by external memory bandwidth. Bandwidth acts as a performance ceiling (simulation).
• "A 1.2V 64Gb 341GB/S HBM2 stacked DRAM with spiral point-to-point TSV structure and improved bank group data control" (2018) — DOI: 10.1109/isscc.2018.8310257 — Conventional DIMM-style memory struggles to exceed 50GB/s due to pin-count constraints and signal integrity issues. HBM with TSVs and wide inputs/outputs is a solution, but power and reliability limits remain.
• "18.2 A 1.2V 20nm 307GB/s HBM DRAM with at-speed wafer-level I/O test scheme and adaptive refresh considering temperature distribution" (2016) — DOI: 10.1109/isscc.2016.7418034 — Using multiple GDDRs together causes problems with power and board routing congestion. HBM achieves wide inputs/outputs through TSVs and interposers, and increased bandwidth demands active thermal countermeasures against hot spots caused by concentrated power consumption, proposing a circuit technique (ART) that adjusts refresh according to temperature distribution.
• "Understanding power consumption and reliability of high-bandwidth memory with voltage underscaling" (2021) — DOI: 10.23919/date51398.2021.9474024 — HBM is a structure where multiple DRAM layers are stacked and placed right beside the compute chip within the same package, and the stacked memory layers consume a significant portion of the package power budget.
• "3D Stacking of RAM–Processor Chips Using TSV" (2014) — DOI: 10.1002/9781118760475.ch06 — TSVs are wiring channels created by filling vertical holes in a chip with a conductive metal like copper. Challenges of three-dimensional stacking include heat dissipation and mechanical and thermal stress.
• "22.1 A 1.1V 16GB 640GB/s HBM2E DRAM with a Data-Bus Window-Extension Technique and a Synergetic On-Die ECC Scheme" (2020) — DOI: 10.1109/isscc19947.2020.9063110 — HBM achieves 1,024 inputs/outputs via a 2.5D silicon interposer and large capacity through TSV stacking.
• "22.3 A 128Gb 8-High 512GB/s HBM2E DRAM with a Pseudo Quarter Bank Structure, Power Dispersion and an Instruction-Based At-Speed PMBIST" (2020) — DOI: 10.1109/isscc19947.2020.9062977 — HBM delivers much higher bandwidth at lower pin speeds than ordinary DRAM. Three-dimensional stacking causes TSV connection failures and cumulative voltage drops, increasing total cost, and as bandwidth increases, power and thermal issues also grow.
• "13.4 A 48GB 16-High 1280GB/s HBM3E DRAM with All-Around Power TSV and a 6-Phase RDQS Scheme for TSV Area Optimization" (2024) — DOI: 10.1109/isscc49657.2024.10454440 — 16-layer stacked 48GB, 1280GB/s HBM3E (manufacturer announcement).
• "Reliability-performance tradeoffs between 2.5D and 3D-stacked DRAM processors" (2016) — DOI: 10.1109/irps.2016.7574618 — Stacking DRAM on top of a compute chip yields high performance but makes it hotter and shortens lifespan, whereas placing it side-by-side using an interposer yields lower bandwidth but less thermal aging and a much longer lifespan.
• "Solving Thermal Issues in a Three-Dimensional-Stacked-Quad-Core Processor by Microprocessor Floor Planning, Microchannel Cooling, and Insertion of Through-Silicon-Vias" (2013) — DOI: 10.1115/1.4025531 — Three-dimensional stacking provides bandwidth and energy gains through short wiring, but high thermal resistance means insufficient thermal management threatens reliability.
• "Managed-Retention Memory: A New Class of Memory for the AI Era" (2025) — arXiv:2501.09605 — HBM is not optimal for artificial intelligence workloads and has lower yield than DRAM due to high cost and manufacturing complexity (position paper proposing a new memory class).
• "Challenges of High-Capacity DRAM Stacks and Potential Directions" (2018) — DOI: 10.1145/3286475.3286484 — Three-dimensional stacked memory inside a package offers high bandwidth and low access energy, but capacity is limited. Increasing capacity with higher stacks while simultaneously boosting bandwidth requires more research, and alternative bonding and stacking technologies may be the key.
• "Performance Evaluation and Optimization of HBM-Enabled GPU for Data-Intensive Applications" (2018) — DOI: 10.1109/tvlsi.2018.2791442 — The in-package memory capacity of state-of-the-art HBM-equipped GPUs at the time was limited to 4GB.
• "System-performance and cost modeling of Large Language Model training and inference" (2025) — arXiv:2507.02456 — The growth in size and complexity of language models has outpaced advancements in computing power, memory bandwidth, network performance, and cost efficiency.
• "Challenges and Research Directions for Large Language Model Inference Hardware" (2026) — arXiv:2601.05047 — The primary challenges in language model inference are memory and interconnects rather than compute. Proposes four research directions, such as high-bandwidth flash memory that expands capacity 10-fold with HBM-equivalent bandwidth.
• "Thermo-Mechanical Reliability Characteristics of 8H HBM3" (2024) — DOI: 10.1109/irps48228.2024.10529424 — Demand from artificial intelligence and HPC servers has significantly increased the use of HBM. HBM stacks DRAM using TSVs from 8 layers (8H) to 12 layers (12H), where H stands for height. The number of HBMs per package is increasing from 2 to 4 up to 6 to 8.