The Bottleneck Was Never Compute — Ten Years of Stacking Memory Beside the Processor

An 8-Gigabit Lump at 128 Gigabytes Per Second

In 2014, a semiconductor circuit journal reported an 8-gigabit memory lump made by stacking DRAM vertically. As defined in "Fetching Is Slower Than Computing — The Memory Bottleneck and HBM", DRAM is the memory that holds data while a computer works, and this lump is multiple of those DRAM chips piled on top of each other. It was made through a process of placing chips on a wafer, which is a thin, round silicon plate used to make chips, and tested with rapid signals by applying probes while still in the wafer state. It produced 128 gigabytes per second from a 1.2 V (volt) power supply (DOI: 10.1109/jssc.2014.2360379, measurement).

This number is the bandwidth. As defined in the same primer, bandwidth is the amount of data that can actually be moved in a unit of time, and is therefore written as "so many gigabytes per second."

Where ordinary DRAM stood around that time has been recorded in two papers respectively. According to a 2016 paper, conventional DRAM devices such as stick-shaped memory plugged into computer boards (DDR4 DIMM) or DRAM for graphics chips (GDDR5) were pegged at a bandwidth of under 30 gigabytes per second (DOI: 10.1109/isscc.2016.7418034, background statement). A 2018 paper wrote that DIMM-type memory is hard to exceed 50 gigabytes per second, citing pin count limitations and signal integrity problems as the reasons (DOI: 10.1109/isscc.2018.8310257, background statement). Pins are small connection terminals linking the chip to the outside, and signal integrity is the degree to which signals are delivered properly without being disrupted.

The two numbers have different targets and different points in time. Above all, neither of them is a result obtained by those papers through experiment. They are sentences describing the conditions those papers stood upon, and this article marks such sentences as a "background statement." This is because they are positions easily misread as research results of the paper.

The papers cited by this article are largely divided into two layers. One is papers reporting devices actually made. Numbers such as how many tiers the lump of piled DRAM chips, or stack, has, how many gigabits per second a single pin produces, and what the power supply voltage is come from here. The other is papers that simulated structures or analyzed workloads. The two layers are not values measured with the same ruler. Therefore, at every position with a heavy load, which of measurement, simulation, model, standard, review, or background statement it belongs to is written in parentheses. Since they diverge further within layers, the labels default to measurement, simulation, model, standard, review, and background statement, and are written in more detail where needed. Labels are attached to sentences, not papers. The same paper may be a measurement in one place and a background statement in another. Papers whose numbers in parentheses begin with arXiv and papers published on zenodo are materials released before undergoing peer review, and arXiv-side ones are called preprints.

What Kind of Tasks Hit the Memory Wall

As processors get faster, the speed is determined by the road that carries the data, a dynamic explained by the primer "Fetching Is The Slower Than Computing" using a single restaurant. In that restaurant, the chefs are the processors, the warehouse is the memory, the hallway between the warehouse and the kitchen is the path where data travels, and the number of boxes coming into the hallway is the bandwidth. By the definition in that article, the memory wall is a state where processors keep getting faster, but the speed of loading data from memory cannot keep up, so waiting—rather than computing—determines performance.

There is a scale that tells us which tasks hit that wall. Arithmetic intensity is a count of how many calculations are performed with the data for every 1 byte loaded from memory. In terms of the primer's restaurant, it is how many plates you make with a single box of ingredients. Keeping a canister of salt on the countertop—the small, fast storage space right next to the processor called the cache—and sharing it across multiple plates makes this value large. Tasks that require carrying new ingredients from the warehouse for every plate have a small value for this, and no matter how fast you swap out the chefs, they get stuck in the hallway.

When you draw this value on the horizontal axis and performance on the vertical axis, you can see at a glance from which point performance flattens out, blocked by bandwidth. This drawing is called the roofline model, or the roofline diagram. A 2024 survey paper used this diagram as a common framework to organize techniques for generating answers with large language models. The authors wrote that this framework explains why language models are bound by memory, how much memory and computation each require, and what hardware should be chosen (arXiv:2402.16363, review/analytical framework). A large language model is a model that serves as the foundation for conversational artificial intelligence.

A 2026 review uses the same framework to reach a neater conclusion. In accelerating large language models, the decisive bottleneck is not arithmetic, but memory. The grounds are three: the autoregressive stage, where words are pieced together one by one, is bound by bandwidth; the storage space holding the context can grow to a size comparable to the model weights; and energy is dominated by data movement rather than computation (arXiv:2608.28048, preprint review). Model weights are the numbers containing what the model has learned.

This diagnosis is not new. By the definition in the primer, HBM is memory used attached right next to the computing chip, and the name stands for High Bandwidth Memory. A 2022 device paper wrote in its opening sentence that since its emergence, HBM has been cited as a prominent solution to the memory wall problem. At the same time, it immediately added a caveat. Even so, the emergence of high-performance systems such as supercomputers, ultra-large data centers, and machine learning accelerators continues to demand even higher-performance memory (DOI: 10.1109/isscc42614.2022.9731562, background statement). Every time you climb over the wall, another one rises.

How Do You Know It Has Hit the Ceiling

The claim that "it is bound to memory" can be confirmed through measurement. In 2025, a research team loaded a 7-billion-parameter language model entirely onto a small FPGA board equipped with a 4-gigabyte DRAM. An FPGA is a chip whose internal circuitry can be reconfigured even after manufacturing, and parameters are the individual numbers that make up a model's weights. They did not install an operating system at all. This was to avoid losing a single byte of capacity needed to hold the model weights and intermediate results. They even manually wove together the flow of data, bundling operations into a single stream and rearranging the data into an array that memory could read easily. The result was just around 5 words per second. The key is the number attached right beside it: using 93.3% of the memory capacity, it achieved a speed corresponding to 85% of the theoretical memory bandwidth limit (arXiv:2502.10659, measured — preprint result measured on a single board).

Subtracting the remaining 15% means there is almost nothing left to squeeze out of the hallway on this board. On this board, it is difficult to significantly increase speed simply by making the compute units larger, and the remaining paths are either widening the hallway or reducing the very amount loaded up for transport. The model in this experiment had also already shrunk the numbers into fewer bits for storage. The passage in this paper noting that typical small devices usually have a 4-gigabyte capacity and a bandwidth of under 20 gigabytes per second is the context upon which that paper stands, not a measured value from this experiment (arXiv:2502.10659, background statement).

Where the bottleneck lies is also revealed by the fact that the same optimization works differently on different machines. Memory-bound means performance is tied to bandwidth, and compute-bound means performance is tied to calculation capability. A 2026 study analyzed the case where weights were compressed to 1/16th using the roofline model. On a central processing unit (CPU) bound by bandwidth, as the memory-bound operation shifted toward the compute-bound side, the speed of the core computational piece—that is, the kernel—grew 29.6 times faster, whereas there was almost no gain on a graphics processing unit (GPU). The abstract recorded both of these values together as the conclusion of the roofline analysis (arXiv:2604.20913, analysis). The values measured by running it from start to finish are separate. On a single Intel Xeon, it delivered an end-to-end speed of 32.4 words per second, which was 1.24 times faster than the 4-bit quantization implementation of llama.cpp, which compresses numbers into fewer bits for storage (arXiv:2604.20913, measured).

The fact that the same technique works on only one side means the bottlenecks of the two machines lie in different places. A 2025 simulation study describes this relationship more generally. Increasing the operating frequency of the compute unit reduces the latency of the phase that reads the context all at once, but in the phase that generates words one by one, that effect is capped by external memory bandwidth. The authors expressed this by stating that "bandwidth acts as a performance ceiling" (arXiv:2512.22066, simulation).

Bored Holes Instead of Lengthening Lines

As the primer "Fetching Is Slower Than Computing" puts it, there are two ways in principle to widen the corridor. You can make each individual line faster, or you can lay down more lines. Laying down more lines on a circuit board increases power consumption and routing congestion at the same time. This combination of problems was what a 2016 paper pointed to as the issue with attaching multiple graphics DRAMs simultaneously (DOI: 10.1109/isscc.2016.7418034, background statement).

The answer that emerged was to stack memory upward and pierce the layers vertically. As defined in the primer, a through-silicon via is a wiring created by filling a vertical hole drilled straight through a chip from top to bottom with a conductive metal such as copper, and is called TSV for short. The starting point of this approach is that properly arranging memory and processors with this structure drastically shortens the data pathways on the chip (DOI: 10.1002/9781118760475.ch06, review).

Placing the chips side by side requires a separate base plate. An interposer is a thin plate with multiple chips laid out side by side on it and only dense wiring laid between them. The distance between chips becomes much shorter than taking a detour around the circuit board.

This creates a fork in the road. You can sit the processor and the memory stack side by side on an interposer, or you can stack the DRAM directly on top of the processor. A 2016 reliability study compared the two side by side. The upward-stacked version yielded higher performance but operated at a higher temperature, shortening its lifespan. The side-by-side version had lower bandwidth between the processor and memory, but suffered less thermal degradation, resulting in a significantly longer lifespan. Instead of choosing one side, the authors concluded that running a 3D-stacked system at a lower voltage and frequency could strike a balance between reliability and performance (DOI: 10.1109/irps.2016.7574618, modeling and analysis).

The HBM device and packaging papers cited by this article consistently describe the side-by-side approach. That is true of a device paper noting that silicon interposer technology enabled 1,024 input/output pins (DOI: 10.1109/isscc19947.2020.9063110, background statement), and also of a system report placing a compute chip and a memory stack side by side on an interposer (DOI: 10.1109/ectc.2016.246, measurement and fabrication). At the bottom of that stack sits a base chip, not DRAM. A 2014 paper listed four jobs for this base chip: lowering the electrical load on input/output terminals, fixing connection failures between chips, making testing easier, and improving reliability (DOI: 10.1109/isscc.2014.6757501, design rationale). The DRAM pieces stacked on top of it each have independent bandwidth and are connected to the base plate via very tightly spaced microbumps (DOI: 10.1109/jssc.2014.2360379, measurement). A microbump is a tiny metal pillar with solder on the end that is sandwiched between two chips. The path of eliminating this spacer and bonding the two surfaces directly is covered in "Nothing In Between — How Bonding Two Surfaces Directly Became the Center of the Process".

The actual physical size is as follows. In a system built on a plastic-based organic interposer in 2016, a single HBM stack measured 5.5 × 7.7 × 0.48 millimeters, and contained one base buffer die and four DRAM core dies. A die is a single piece of chip cut out from a wafer. The main compute chip on the same board measured 19.1 × 24 millimeters (DOI: 10.1109/ectc.2016.246, measurement and fabrication). The reason the authors used an organic interposer instead of silicon was size. Silicon interposers have an upper limit on size tied to the area that lithography equipment can expose at once. Lithography is the work of shining light to transfer circuit patterns onto silicon ("Lithography — Painting Circuits with a Brush of Light"), and this ceiling for the area exposed at one time is called the reticle limit. The circumstances in which this ceiling forced chips to be made in pieces are covered in "They Chose Not to Make It One Chip — The Bill for Cutting It Apart".

Laying Down 1,024 Pins Means One Pin Can Be Slow

Reduce the distance, and you reduce the energy. When the wiring becomes shorter, the unwanted electrical bowl that comes along with it—in other words, parasitic capacitance—decreases, and the energy needed to push signals decreases by that much (DOI: 10.5281/zenodo.17652880, preprint public review and design discussion). When the wiring is short, you can make the circuit driving it small, which reduces the energy consumption of the wiring stage one more time (DOI: 10.1115/1.4025531, simulation and design study).

What made it possible to lay down an enormous number of lines were packaging technologies like through-silicon vias, microbumps, and silicon interposers. A 2020 device paper summarized the result produced by the advancement of these technologies in a single sentence. HBM has a lower speed per pin than conventional DRAM while having a much higher bandwidth (DOI: 10.1109/isscc19947.2020.9062977, background statement).

Expressed in numbers, it looks like this. The HBM3 specification supports 6.4 gigabits per second per pin, and uses 1,024 input/output pins. Multiply the two and convert bits to bytes, and you get 819 gigabytes per second (DOI: 10.1109/a-sscc58667.2023.10348011, specification value). This number is not something someone measured, but a value fixed by the specification document. The bandwidth in the device paper of the next section is a value actually manufactured and tested, so its nature differs from this 819.

We also widened the width by splitting channels. A channel is an independent pathway through which memory exchanges data separately. A 2016 paper proposed a method where a single existing channel is split into two so that command and address pins are used together, allowing one HBM to have 16 pseudo-channels instead of 8 channels (DOI: 10.1109/isscc.2016.7418034, design and measurement).

Numbers from a Decade

Below are the values that device papers actually built, tested, and reported, placed in chronological order. Do not mix them with the specification values in the previous section.

• 2014 — 8 gigabits, 8 channels, 128 gigabytes per second at 1.2 V (DOI: 10.1109/jssc.2014.2360379, measured).
• 2016 — 20 nanometer process, 1.2 V, 307 gigabytes per second. The goal this paper stated for itself was "doubling 128 gigabytes per second to 256 gigabytes per second or more," and it also presented a structure that supported 8-tier stacks and variable capacities from 16 gigabits to 64 gigabits with the same bandwidth (DOI: 10.1109/isscc.2016.7418034, measured).
• 2018 — 64 gigabits, 1.2 V, 341 gigabytes per second. It used a spiral point-to-point through-silicon via structure (DOI: 10.1109/isscc.2018.8310257, measured).
• 2020 — 128 gigabits, 8 tiers, 512 gigabytes per second (DOI: 10.1109/isscc19947.2020.9062977, measured). Because eight bits make one byte, this capacity is equal to 16 gigabytes. In the same year, 16 gigabytes, 1.1 V, and 640 gigabytes per second (5 gigabits per second per pin) were also reported. This included data bus window expansion, a high-speed power supply network, in-die error correction, and a large-capacity high-speed test circuit (DOI: 10.1109/isscc19947.2020.9063110, measured).
• 2022 — 192 gigabits, 12 tiers, 896 gigabytes per second at 1.0 V. It reported that power efficiency improved by up to 15%, but the abstract does not state what it was compared against (DOI: 10.1109/jssc.2022.3193354, measured).
• 2024 — 48 gigabytes, 16 tiers, 1,280 gigabytes per second. The point this paper emphasized was that by keeping the arrangement of connection terminals (bumps), the number of channels and input/outputs, and the operating voltage identical to the previous generation, it could be swapped in without modifying the system (DOI: 10.1109/isscc49657.2024.10454440, measured).

The number of tiers in a stack counts the number of DRAM sheets. A stacking technology paper from 2020 actually built and demonstrated 4-tier, 8-tier, and 12-tier structures, writing that each placed 4, 8, or 12 DRAM dies on a single base die, respectively (DOI: 10.1109/ted.2020.3021358, measured). Therefore, "16 tiers" means sixteen sheets of DRAM.

A tutorial from 2022 summarized this trajectory like this: alongside advanced packaging technology, HBM has been the only scalable DRAM bandwidth solution over the past decade, starting from 128 gigabytes per second and coming past 800 gigabytes per second (DOI: 10.1109/isscc42614.2022.11005811, tutorial). A review paper from 2017 introduced the same technology as a solution that exceeds 256 gigabytes per second while also reducing power consumption, listing reliability, heat dissipation capacity, the maximum allowable package size, and high-throughput stacking as challenges that must be solved together (DOI: 10.1109/imw.2017.7939084, review).

One caveat comes with these numbers. A paper from 2020 described the industry trend at the time as "speed improves by 15% to 20% every year and capacity becomes 1.5 to 2 times every 2 years" (DOI: 10.1109/isscc19947.2020.9063110, background statement). This is a trend observed by that paper, not a law of physics or a measurement result. The value "HBM2 is 341 gigabytes per second at 8 tiers and 8 gigabits" written in the same paper was also brought in by citing another paper's results with a reference number.

Bill ① Stack It and Heat Gets Trapped

Stacking things comes with a bill. The primer summarized these bills as heat, cost, and capacity, and here we follow four bills—heat, voltage, reliability that varies by position, and capacity—using the numbers from the papers. The first is heat, and this problem was known from the beginning. That three-dimensional stacking using through-silicon vias brings heat dissipation, mechanical stress, and thermal stress is an item that literature introducing this technology lists as challenges (DOI: 10.1002/9781118760475.ch06, review). A 2013 simulation study noted that a structure stacking a processor and DRAM in three dimensions has high thermal resistance—meaning how hard it is for heat to escape outside—so that poor thermal management threatens device reliability, and it issued a warning along with a prescription. It is to rearrange the core layout, cool with microfluidic channels, and insert through-silicon vias dedicated to removing heat to lower internal thermal resistance (DOI: 10.1115/1.4025531, simulation and design study).

Even inside device papers, this problem appears bundled with bandwidth. When bandwidth increases, power consumption concentrates in a narrow space, creating locally hot spots, so active thermal countermeasures are necessary (DOI: 10.1109/isscc.2016.7418034, design and measurement). A paper from the same year, 2016, proposed a method of changing the refresh cycle by considering the temperature distribution as a solution. By the definition in "Next-Generation Memory — The DRAM Wall and the Race for a "Dream Memory"", refresh is an operation where DRAM periodically reads and refills stored contents so it does not lose its values. The hotter DRAM is, the faster it loses stored values.

There is also research on the side of lowering heat. A 2025 simulation study calculated that if an interposer is made of hexagonal boron nitride—which does not conduct electricity but conducts heat well—instead of silicon, the temperature of hot spots becomes 20 degrees Celsius lower compared to a silicon interposer. The authors added that with that level of improvement, the chip's leakage power could be reduced by 22% (arXiv:2510.11461, simulation). Leakage power is the power lost to current leaking out even when a transistor is in the off state ("A Transistor Is a Switch — A Tiny Faucet You Open and Close with Electricity"). This is an estimate with a conditional clause attached, not a measured value.

There are also attempts to predict heat in advance. A 2025 study generated results for 13,494 combinations of conditions through a simulation that breaks down the structure finely to calculate heat flow, namely finite element analysis, and trained a neural network on them. The surrogate model made that way quickly estimates the junction temperature that circuits actually experience in a 3D HBM structure and the location of the hottest spot, without running slow simulations again (arXiv:2503.04049, model).

The moment you stack, the bill called heat comes along. How the logic chip side handles the heat trapped in layers, thermal resistance, and hot spots is continued in "Turn the Power Around and the Heat Gets Trapped — The Bill for Grinding Away the Silicon Under the Transistors".

Bill ② When You Stack Them, the Voltage Wobbles

A 2020 device paper first listed two problems created by three-dimensional stacked structures: the failure of through-silicon via connections, and the phenomenon where voltage drops occurring in the power delivery network accumulate layer by layer. A voltage drop is when electricity loses strength as it passes through a wire with resistance and arrives at a lower voltage than when it started, and this phenomenon itself is a story on the logic chip power network side, which "Turn the Power Around and the Heat Gets Trapped — The Bill for Grinding Away the Silicon Under the Transistors" covers in detail. The authors wrote that these two increase the total cost of HBM. The abstract does not end there. The authors added "Furthermore," connecting it to the point that as bandwidth increases, structural challenges in DRAM arise along with it, and power consumption and the heat problems that come with it also grow (DOI: 10.1109/isscc19947.2020.9062977, background and problem statement). The list does not close with two items.

What happens when the voltage wobbles was detailed in a 2023 power circuit paper. Because there is not enough space on the base die, HBM borrows cell capacitors inside the DRAM core dies stacked on top to use as noise buffers across the through-silicon vias. A cell capacitor is a small electrical bowl where DRAM stores values. However, the more core dies are stacked, the greater the resistance of that path—namely, the equivalent series resistance. When the resistance increases, the buffering effect drops, increasing power supply noise. That noise appears as signal jitter in the clock circuit that distributes timing signals, cutting into the time margin to read signals on time (DOI: 10.1109/a-sscc58667.2023.10348011, design and problem analysis). This paper does not stop at stating the problem. From its title, it is a high-speed voltage regulator for HBM3—meaning a circuit that receives a wobbling power supply and outputs a constant voltage—and what the authors presented is a compact, ultra-high-speed regulator that uses active inductors, self-adaptive bias, and a two-stage technique to handle the various operating modes of the DRAM structure (DOI: 10.1109/a-sscc58667.2023.10348011, design and proposal).

How much margin there is in the voltage has been measured with an actual chip. In 2021, a research team ran an actual HBM chip at a voltage lower than normal. Manufacturers leave a safety margin in the voltage so that it operates correctly under any environment. In the chip they tested, that margin was 19% of the reference voltage set by the manufacturer, namely the nominal voltage. When they lowered the voltage within that margin, power consumption dropped by 1.5 times regardless of bandwidth utilization. Lowering it by another 11% from there reduced it by up to 2.3 times, but at the cost of unwanted bit flips appearing (DOI: 10.23919/date51398.2021.9474024, actual measurement).

If you remove the caveat "In the chip they tested," this sentence becomes a claim about all HBMs. The authors did not write it that way. The authors characterized the frequency and type of bit flips that occurred at the lowered voltage to produce a defect map, and wrote that this map enables a three-way trade-off between power, capacity, and error rate (DOI: 10.23919/date51398.2021.9474024, actual measurement).

Bill 3: The Same Part Varies by Position

The third is a somewhat unfamiliar kind of problem revealed by the stacked structure.

Two phenomena defined in "Next-Generation Memory" appear here again. Rowhammer is a phenomenon where repeatedly opening and closing a single row of DRAM physically flips bits in the adjacent row, and rowpress is a phenomenon where simply keeping a row open for a long time disturbs the adjacent row and flips bits. Both are problems originally known from planar DRAM, and together they are called read disturbance.

In 2024, a research team plugged six actual HBM2 chips into two different types of FPGA boards and characterized this phenomenon in detail. What stood out in the results was the variance. Not only did vulnerability vary greatly from chip to chip, but it also varied significantly across vertically stacked channels within a single chip. Rows located at the ends and the middle of banks, which are bundles of cells inside the DRAM, endured better. The HBM2 chips tested included undocumented defense mechanisms that tracked rows activated multiple times (DOI: 10.1109/dsn58291.2024.00022, measurements).

An earlier report from the same research line put numbers to that variance. Within a single HBM2 chip, the bit error rate across vertically stacked channels differed by up to 79% (arXiv:2305.17918, measurements).

Even within the same product, properties vary greatly across vertically stacked channels. From a design perspective, this can be a burden. If properties vary significantly across channels, the margin may need to be tailored to the weakest channel. This is the interpretation of this article.

However, the purpose of these two studies was to understand attacks using read disturbance and their defenses. How to set the reliability margin was not the question of those studies, and the authors wrote that their findings could be used to build stronger attacks and more efficient defenses (DOI: 10.1109/dsn58291.2024.00022, measurements). This article read the fact that properties differ across channels as a design problem, while the original papers read it as a security problem.

Bill 4 — Capacity and Bandwidth Do Not Grow Together

The fourth bill is capacity. This method restricts capacity while buying bandwidth.

A 2018 study tackled this point head-on. Memory stacked in three dimensions inside a package has low access energy and high bandwidth, but limited capacity. The authors examined the challenges of the path toward stacking more DRAM dies to make higher stacks, and concluded the final two sentences like this: Higher stacks can be an interesting approach to increase capacity, but more research is needed to enable high-capacity stacks while scaling bandwidth alongside. They pointed to alternative bonding and stacking technologies as candidates that could hold the key (DOI: 10.1145/3286475.3286484, analysis and outlook). The path of attaching them in a different way continues in "Nothing In Between — How Bonding Two Surfaces Directly Became the Center of the Process".

How the capacity restriction actually manifests is shown by a 2018 empirical study. The starting point was that the memory capacity inside the package of the latest HBM-equipped GPU at the time was only 4 gigabytes. The authors bypassed this constraint by splitting neural network training into a software pipeline, meaning they divided the work into multiple stages and flowed them sequentially. As a result, it was 1.63 times faster than the highest-performance GPU available on the market. In breadth-first search, which scans graphs sequentially starting from the nearest points, they used two techniques together to achieve up to 24.5 times (9.8 times and 2.5 times, respectively) compared to a conventional implementation (DOI: 10.1109/tvlsi.2018.2791442, measurement). The former multiple is compared to market GPUs, and the latter to a conventional implementation, so the two multiples have different comparison targets.

There are also proposals to choose a direction outright between capacity and bandwidth. A 2026 structural study took high-bandwidth memory in a form that trades capacity for energy and cost as a design element, and jointly proposed a chiplet structure that allocates bandwidth first. A chiplet is a piece made by splitting a chip into multiple separate pieces and connecting them inside a package, and its meaning and cost are covered in "They Chose Not to Make It One Chip — The Bill for Cutting It Apart". As a result of simulation, they reported that in a specific large model, it achieved up to 45.3 times lower latency and 18.6배 higher throughput compared to an existing top-tier GPU system under the same heat limit (thermal design power) condition (arXiv:2602.18568, simulation). This is a simulation value, not a physically fabricated device.

There is also the cost. According to a 2016 packaging paper, high cost of ownership has been cited as the main reason why the adoption of interposer-using methods has been slow. The authors cited as the background that, lacking standards, process choices split into several, and each choice failed to reach economies of scale. And after working with four process choices for four years, they proposed adopting one as a baseline process, and viewed that method as expandable to an economical TSV-free packaging platform under development (DOI: 10.1109/ectc.2016.148, process development conclusion). A 2017 review paper places testing technology and testability as separate items, discussing them targeting verified individual stacks and 2.5D system-in-packages containing multiple chips in one package (DOI: 10.1109/imw.2017.7939084, review). In this structure, it means that what to test and how to test it takes up a separate chapter.

What Is Thirsty for Bandwidth — The Key-Value Cache

This is where the hardware side of the story ends. The reason for the demand focused on this technology lies on the software side. A 2024 device paper wrote at the beginning that with the emergence of large language models and generative artificial intelligence requiring a massive number of model parameters, the memory bandwidth and capacity needed for high-performance systems are increasing unprecedentedly (DOI: 10.1109/isscc49657.2024.10454440, background statement).

A key-value cache is a storage space where an language model accumulates intermediate results calculated for every word it has read so far, in order to reuse them when generating the next word. Although it has "cache" in its name, it is different from the small cache near the processor mentioned in the primer, and it occupies the GPU's memory (arXiv:2512.11920, proposal and evaluation). As the conversation gets longer, this storage space keeps growing, and every time a single new word is generated, the entire space must be read again. The larger the storage space, the more data needs to be fetched from memory.

The diagnosis of the 2026 review cited earlier reaches this very point. The step of generating words one by one gets bound by bandwidth, this cache can grow to a size comparable to the model weights, and energy is dominated by data movement (arXiv:2608.28048, preprint review).

Another analysis from the same year looks at this by dividing it into two metrics. Looking at operations per byte and how much capacity is occupied together explains aspects that the classic roofline model misses. The authors wrote that these two values vary significantly depending on tasks such as conversation, coding, and web usage, and concluded that long-context key-value caches strongly make the word generation phase memory-bound (arXiv:2601.22001, analysis).

The same wall appears in acceleration techniques that pre-generate multiple words and verify them all at once. The verification step must pass through the entire model once, so it is strictly bound by bandwidth. Starting from this point, a 2026 study used low-bit precision only for the verification step to effectively cut memory traffic in half, and reported a 1.28-fold improvement in end-to-end throughput (arXiv:2603.01399, experiment).

There are also attempts to offload the cache to another device. A 2025 proposal moved this cache to a separate accelerator-side memory and attached compression and decompression circuits to lower the memory bandwidth requirement by up to 4 times, and reported achieving up to 3.2 times the throughput compared to a GPU-only baseline (arXiv:2512.11920, proposal and evaluation).

Give It Bandwidth and the Bottleneck Moves

When bandwidth becomes generous, the bottleneck packs its bags and moves somewhere else. A 2026 study reached this conclusion while analyzing a structure that processes word generation right next to stacked 3D memory. The bandwidth advantage of such a structure pulls many computations right back into the compute-bound region (arXiv:2604.04253, analysis and simulation — preprint). The approach of computing near or inside memory is covered in "Don't Move It, Compute Where It Sits — Six Questions Left After Made Memory Do the Math".

The place the bottleneck moves to is not just one spot. In 2018, a research team plugged a prototype of the Hybrid Memory Cube—a 3D-stacked memory approach different from HBM—onto an accelerator board to characterize its access latency and bandwidth. The authors' goal was not to rank performance, but to investigate what its behavior meant for system and software design. The result was that performance is not determined by a single bandwidth number. The nature of the internal communication network connecting the components inside the stack, the packet-based protocol that sends and receives data in bundles, the characteristics of the internal queues, and the traffic conditions all together influence application performance (DOI: 10.1109/ispass.2018.00018, measurement and characterization).

A 2026 perspective paper defined the main challenge of language model inference as memory and interconnect rather than computation, and pointed out four research directions. They are high-bandwidth flash with 10 times the capacity while maintaining HBM-equivalent bandwidth, near-memory processing, 3D stacking of memory and logic, and low-latency interconnects (arXiv:2601.05047, perspective and review). Flash is memory that retains its contents even when the power is turned off.

A Fork in the Road Before the Next Generation — Standard or Custom

A design discussion from 2025 compares next-generation HBM side by side: the path of sticking to the standard specification versus modifying it to fit the system. The standard side offers proven reliability, supply chain accessibility, a fast integration schedule, and a predictable cost structure. The customized side brings performance differentiation, but significantly increases development complexity, lengthens schedules, and raises initial development costs (DOI: 10.5281/zenodo.17652880, preprint review and design discussion).

The integration challenges listed by the same literature overlap with the bills this article has been counting. They are the co-design of package substrates, power delivery networks to handle sudden current surges, thermal conduction in vertically stacked structures, signal and power integrity verification with almost no design margin, and the overall testing and verification procedures. In the same place, this literature notes that through-silicon vias enable vertical wiring densities beyond conventional methods, but simultaneously bring mechanical stress and keep-out zones where nothing can be placed around them.

A 2026 review cited earlier summarized its investigation with the same phrase as this article's title: in large language models, the memory system has become the computer. The same authors noted that no single architecture is optimal for every workload. Computing inside memory is acting as a complementary approach in heterogeneous systems as the most promising short-term response to this wall, while neuromorphic chips mimicking the brain's neural structure and optical computing calculating with light are not yet ready for production at the frontier scale (arXiv:2608.28048, preprint review). All of these are judgments as of the time that review was written.

The Number of Pins Stayed the Same, One Pin Got Faster

Between 128 in 2014 and 1280 in 2024, more than one thing changed. Packaging technologies like interposers and stacking made it possible to lay down 1024 of them, and circuits made each of those 1024 pins steadily faster.

Looking at the number of pins first, a 2020 paper noted that 2.5D interposer technology enabled 1024 input/output connections (DOI: 10.1109/isscc19947.2020.9063110, background statement), and the HBM3 standard also specifies 1024 input/output pins (DOI: 10.1109/a-sscc58667.2023.10348011, specification value). A 2024 paper records that the number of channels and input/output pins is identical to the previous generation (DOI: 10.1109/isscc49657.2024.10454440, measured data). Over this interval, the number of pins did not increase and remained the same.

What moved was the speed of a single pin. The previous-generation value that the 2020 paper pulled from another paper as background was 2.7 gigabits per second per pin (DOI: 10.1109/isscc19947.2020.9063110, background statement), and the value that the paper itself reported was 5 gigabits per second per pin (DOI: 10.1109/isscc19947.2020.9063110, measured data). The HBM3 standard is 6.4 gigabits per second per pin (DOI: 10.1109/a-sscc58667.2023.10348011, specification value). Because the first two are values produced by devices and the last one is a value set by a specification document, these three are also not measured with the same ruler. The industry trend noted by that previously cited paper was precisely that "speed improves by 15 to 20 percent every year."

The manufacturing process for transistors did not stand still either. Viewed through the node—the name attached to a process generation ("What Is "3 Nanometers" the Length Of? — Measuring a Chip's Name with a Tape Measure")—even just the ones recorded in this article show that the 2014 device used a 29-nanometer process (DOI: 10.1109/isscc.2014.6757501, measured data), the 2016 device used a 20-nanometer process (DOI: 10.1109/isscc.2016.7418034, measured data), and the 16-gigabit device from 2020 reported securing stable cell operation in the second generation of a 10-nanometer-class process (DOI: 10.1109/isscc19947.2020.9063110, measured data). Although the story started at a point where miniaturization became difficult, the manufacturing process kept moving within this technology.

Thus, this story cannot be summarized by a single line of "stack it and it gets fast," nor by "you can do it while leaving transistors as they are." Packaging technology that stacks and attaches side-by-side made 1024 lines possible, and on top of that, circuits and processes each played their continuous part. And the moment you stack, heat gets trapped, voltage fluctuates across layers, properties vary depending on the location even within the same product, and widening the bandwidth ties down the capacity. The ten years spent wrestling with those four things, and the share we have not yet fully paid back, are the actual substance of this technology.

The path to paying off that bill is continued by the sibling pieces of this series. The path of attaching layers directly without spacers is covered by "Nothing In Between — How Bonding Two Surfaces Directly Became the Center of the Process", the path of cutting a chip into pieces and attaching them by "They Chose Not to Make It One Chip — The Bill for Cutting It Apart", the path of changing the base to glass and seating light beside the chip by "If the Board Warps, Nothing Above It Helps — The Bill for Glass Base and In-Package Light", trapped heat and voltage drop by "Turn the Power Around and the Heat Gets Trapped", and the path of moving computation straight toward the memory side by "Don't Move It, Compute Where It Sits — Six Questions Left After We Made Memory Do the Math".

Source Papers

• "8Gb 8-Channel 128GB/s HBM Stacked DRAM" (2014) — DOI: 10.1109/jssc.2014.2360379 — Measured 128 GB/s at 1.2 V, independent bandwidth per slice.
• "8Gb HBM Stacked DRAM Using 29nm Process and TSVs" (2014) — DOI: 10.1109/isscc.2014.6757501 — Four roles of the base logic die.
• "Three-Dimensional Integration of RAM-Processor Chips Using TSVs" (2014) — DOI: 10.1002/9781118760475.ch06 — TSV definition, heat dissipation and stress challenges.
• "20nm 307GB/s HBM DRAM and Adaptive Refresh" (2016) — DOI: 10.1109/isscc.2016.7418034 — 16 pseudo-channels, hotspots called by bandwidth.
• "Reliability-Performance Trade-Offs in 2.5D and 3D Stacked DRAM" (2016) — DOI: 10.1109/irps.2016.7574618 — 3D means higher performance and lower lifespan; low voltage and low frequency may be the balance point.
• "Organic Interposer 3D SiP" (2016) — DOI: 10.1109/ectc.2016.246 — HBM stack 5.5×7.7×0.48 mm, bypassing the reticle limit.
• "Comparison of TSV Interposer Die Stacking Methods" (2016) — DOI: 10.1109/ectc.2016.148 — High cost of ownership and lack of standards, proposed baseline process.
• "HBM DRAM Technology and Architecture" (2017) — DOI: 10.1109/imw.2017.7939084 — Exceeding 256 GB/s and low power, test technology is a separate item.
• "1.2V 64Gb 341GB/s HBM2 Stacked DRAM" (2018) — DOI: 10.1109/isscc.2018.8310257 — Spiral point-to-point TSVs. The 50 GB/s wall is background statement.
• "Optimization of Data-Intensive Applications on HBM-Equipped GPUs" (2018) — DOI: 10.1109/tvlsi.2018.2791442 — 4 GB constraint, 1.63x and 24.5x (different denominators).
• "Internal Interconnect Impact of 3D Stacked Memories" (2018) — DOI: 10.1109/ispass.2018.00018 — Prototype characterization, bandwidth alone does not determine performance.
• "Challenges and Directions for High-Capacity DRAM Stacks" (2018) — DOI: 10.1145/3286475.3286484 — Capacity limits, different bonding and stacking technologies are the key.
• "Low-Temperature SoIC Bonding and Stacking for 12- and 16-Layer HBM" (2020) — DOI: 10.1109/ted.2020.3021358 — Fabricated 4-, 8-, and 12-layer versions (1 base + N core dies).
• "128Gb 8-Layer 512GB/s HBM2E DRAM" (2020) — DOI: 10.1109/isscc19947.2020.9062977 — Higher bandwidth at lower pin speeds, TSV failures and IR drop drive up costs.
• "1.1V 16GB 640GB/s HBM2E DRAM" (2020) — DOI: 10.1109/isscc19947.2020.9063110 — 5 Gb/s per pin, 10nm-class second generation. Industry trend is background statement.
• "Power and Reliability of HBM Under Low-Voltage Operation" (2021) — DOI: 10.23919/date51398.2021.9474024 — 19% guardband, 1.5x, 2.3x, and bit flips in tested chips.
• "192Gb 12-Layer 896GB/s HBM3 DRAM" (2022) — DOI: 10.1109/jssc.2022.3193354 — Measured 1.0 V, power efficiency up to 15% (denominator unstated).
• "HBM3 with Automated TSV Calibration" (2022) — DOI: 10.1109/isscc42614.2022.9731562 — Background statement that it has been cited as a memory wall solution.
• "Tutorial: HBM DRAM and 3D Stacked Memory" (2022) — DOI: 10.1109/isscc42614.2022.11005811 — The only scalable solution for a decade, going beyond 128 to 800 GB/s.
• "High-Speed LDO for HBM3" (2023) — DOI: 10.1109/a-sscc58667.2023.10348011 — Spec 6.4 Gbps × 1024 pins = 819 GB/s, noise caused by increased ESR and its solution.
• "Experimental Study of Read Disturb in HBM2 Chips" (2024) — DOI: 10.1109/dsn58291.2024.00022 — 6 physical chips, channel-to-channel variance. Goal is to understand attacks and defenses.
• "48GB 16-Layer 1280GB/s HBM3E DRAM" (2024) — DOI: 10.1109/isscc49657.2024.10454440 — Backward compatible by maintaining channel count, I/O count, and voltage.
• "Thermal Management of 3D Stacked Quad-Cores" (2013) — DOI: 10.1115/1.4025531 — Short wiring benefits and high thermal resistance; layout, microchannels, and thermal TSVs are the prescriptions.
• "Standard HBM4 vs. Custom HBM4" (2025) — DOI: 10.5281/zenodo.17652880 (pre-peer-review) — Integration challenge list, TSV stress and keep-out zones.
• "AI Accelerators and the Memory Wall for LLMs" (2026) — arXiv:2608.28048 (preprint) — The decisive bottleneck is memory, not arithmetic. No single architecture is a panacea.
• "LLM Inference Survey and Roofline Model" (2024) — arXiv:2402.16363 (preprint) — Common analytical framework for why language models are bound by memory.
• "Prefill vs. Decode Bottlenecks" (2025) — arXiv:2512.22066 (preprint) — Even when frequency is raised, the generation phase is capped by external bandwidth.
• "Bandwidth and Capacity Utilization Limits in Embedded FPGAs" (2025) — arXiv:2502.10659 (preprint) — 93.3% capacity, measured at 85% of theoretical bandwidth limits.
• "Multiplication-Free LLM Inference on CPUs via Fused Ternary Kernels" (2026) — arXiv:2604.20913 (preprint) — Kernel 29.6x (stated alongside roofline analysis), measured end-to-end 1.24x.
• "Junction Temperature Surrogate Model for 3D HBM Chiplets" (2025) — arXiv:2503.04049 (preprint) — Trained on 13,494 finite element data combinations.
• "3D Thermal Analysis of Boron Nitride Interposers" (2025) — arXiv:2510.11461 (preprint) — Simulation, hot spot reduced by 20 °C compared to silicon.
• "Rowhammer Experimental Analysis in HBM2 Chips" (2023) — arXiv:2305.17918 (preprint) — Up to 79% difference in bit error rates across channels.
• "Inference Processing Units" (2026) — arXiv:2602.18568 (preprint) — Memory traded for energy and cost by reducing capacity, up to 45.3x in simulation.
• "Heterogeneous Computing for AI Agent Inference" (2026) — arXiv:2601.22001 (preprint) — Long context caches make the generation phase memory-bound.
• "Quantization-Assisted Speculative Acceleration" (2026) — arXiv:2603.01399 (preprint) — Verification phase is strictly bandwidth-bound, end-to-end 1.28x.
• "Distributed FPGA Speculative Key-Value Caching" (2025) — arXiv:2512.11920 (preprint) — Bandwidth demand reduced by up to 4x, 3.2x compared to GPU alone.
• "Reevaluating Compute Substrates for 3D Near-Memory Decoding" (2026) — arXiv:2604.04253 (preprint) — Bandwidth advantages push many operations back to being compute-bound.
• "Challenges and Research Directions for LLM Inference Hardware" (2026) — arXiv:2601.05047 (preprint) — The challenge is memory and connectivity, not computation.