Don't Move It, Compute Where It Sits — Six Questions Left After We Made Memory Do the Math

256 Store-Bought DRAM Chips

There is an experiment where instruction sequences not supported by the manufacturer were fed into 256 store-bought DDR4 DRAM chips. DRAM is the memory that temporarily holds data while the computer works, and its contents disappear when the power goes off ("Fetching Is Slower Than Computing — The Memory Bottleneck and HBM"). This specification name refers to a generation of DRAM. The chips did things they were not asked to do. They performed the NOT operation, among the simplest logic rules dealing with only two values of true and false, with an average success rate of 98.37%. NAND, NOR, AND, and OR operations receiving up to sixteen inputs were also performed with average success rates of 94.94%, 95.87%, 94.94%, and 95.85%, respectively. When the data patterns were randomized, the success rate dropped by 1.39 to 1.98%, and even when the temperature was raised from 50 to 95 degrees Celsius, the fluctuation was at most 1.66% (DOI: 10.1109/hpca57654.2024.00030, measured on commercial chips).

The word to watch here, rather than 98.37, is "success rate." Success rate is the proportion of commanded operations that finished correctly. When you make a part do calculations that was never built to calculate, this word gets attached to the result.

Most computers have been built on a single framework. The von Neumann architecture is a computer structure where the part that calculates and the part that holds data are separated from each other, working by carrying data back and forth between them. In this framework, the way to get faster is to widen the path for carrying data. Stacking memory and attaching it next to the processor ("The Bottleneck Was Never Compute"), attaching the faces of two chips directly with nothing in between them ("Nothing In Between"), breaking chips into pieces and attaching them ("They Chose Not to Make It One Chip"), changing the base to glass and seating light next to the chip ("If the Board Warps, Nothing Above It Helps"), and routing electricity to the backside of the wafer ("Turn the Power Around and the Heat Gets Trapped") are all construction projects within this framework.

The proposal examined in this article is to send calculations to where the data is, instead of widening the path.

The evidence for this article mixes four layers that were not measured with the same yardstick: ⓐ directly poking and prodding commercial chips, ⓑ measuring a single block of circuitry, ⓒ measuring across an entire chip or a physical system, and ⓓ calculating with simulators and models. Therefore, the rating labels are attached to sentences rather than papers. The baseline for "N times faster" also differs from paper to paper. Whether it is the central processing unit (CPU) of a computer, a graphics processing chip (GPU) widely used for artificial intelligence calculations, a digital accelerator, or an older technique of the same kind is noted each time.

The unit this field enjoys using when writing down efficiency is "tera-operations per watt (TOPS/W)," meaning how many trillion operations per second are performed with one watt of electricity. How many bits are used for calculation refers to how many binary digits are used to write down a single number, meaning how fine-grained the scale is for the calculation. The number of operations per watt cannot be compared unless it is revealed how many bits were used for the calculation. As we will see later, the same circuit yields 117.3 at 1 bit and 2.06 at 16 bits. The circuit is the same, and the 57-fold difference comes from the scale.

An Idea Already Around in the 1950s

The roots of this idea run deep. According to a technical history study in a dissertation, Dudley Buck proposed a "recognizer" in the 1950s (DOI: 10.18130/q5ar-1c61, a technical history study in a dissertation). It was a device that found things by the stored content itself, which influenced later content-addressable memory—memory that finds data not by its location number, but by the content it holds. In the 1960s, "distributed logic memory" emerged, and when the memory wall became a hot topic in the 1990s, it returned once more with EXECUBE and IRAM. And in that era, it subsided.

The memory wall is the state described in "Fetching Is Slower Than Computing — The Memory Bottleneck and HBM". It is a state where processing units keep getting faster, but the speed of loading data from memory cannot keep up, so waiting rather than computing determines performance. To use the restaurant analogy from that article, no matter how much you change the chef (the processor), the hallway where ingredients come in from the warehouse (the memory) determines the speed. The width of the hallway is the bandwidth, which is the amount of data that can actually be moved per unit time.

A review paper that reexamined this idea notes that systems at that point were designed to move data toward computation, and that moving things is much more expensive than computing in terms of both energy and latency (arXiv:1905.04376v1, a review and position paper, preprint). The solution is to place computing devices inside or right beside the spot where data is stored. The spots are three: inside the memory chip, on the logic layer of a 3D stacked structure, and in the memory controller. The logic layer is a non-DRAM chip laid out at the very bottom of a stack of multiple DRAM layers (the base plate covered in "The Bottleneck Was Never Compute"). The memory controller is the circuit that directs the reading and writing of memory.

Here the name splits into two. In-memory computing is a method where, instead of hauling data over to a calculator, you finish the calculation inside the memory where the data is already placed. Near-memory processing is a method where you do not touch the memory array itself, but attach a small calculation circuit right beside that array to finish calculations before the data goes out of the chip.

A memory array is the part where cells holding values are lined up densely in rows and columns. To use the restaurant from "Fetching Is Slower Than Computing — The Memory Bottleneck and HBM", in-memory computing is finishing the cooking inside the warehouse, and near-memory processing is finishing the prep right in front of the warehouse door to reduce the load sent into the hallway. Either way, the amount passing through the hallway decreases.

The branches that the same review paper picked as viable point to these two as well. Doing large-scale parallel computations inside memory using the analog operations of DRAM, and providing a wide path to the calculation circuits inside 3D stacked memory using its logic layer.

What kind of work suits this approach is gauged by operational intensity. Operational intensity is a counted value of how many calculations are done with that data every time 1 byte of data is loaded from memory ("The Bottleneck Was Never Compute"). A restaurant that makes only one dish with a box of ingredients gains nothing from changing the chef.

Attaching Calculators Close to Memory

A 2015 design did not create new DRAM components. It used commercial DRAM and standard memory modules as-is, and adding through-silicon vias to stack an accelerator in 3D on top of the DRAM was almost the entire change. Through-silicon vias are wiring made by drilling vertical holes piercing a chip from top to bottom and filling them with a conductive metal such as copper (Fetching Is Slower Than Computing — The Memory Bottleneck and HBM). An accelerator is a dedicated circuit built to make only specific calculations fast.

The comparison target was a system with the same accelerator logic packed inside the processor. Compared to that, the total energy was 46% lower, the data transfer energy was 68% lower, and the performance was 1.67 times higher (DOI: 10.1109/hpca.2015.7056040, the design and its evaluation — not actual measurements of a fabricated chip). Since the baseline was neither a CPU nor a GPU, but a system with the same accelerator placed in a different spot, this number shows only the effect of moving the location.

A bank is a compartment made by dividing a single memory chip into multiple pieces, allowing each piece to be opened and closed separately. Because each bank moves independently, while one bank is retrieving data, another bank can work at the same time. Near-memory processing usually attaches a calculator right next to this bank.

Actual physical products have also emerged. A company named UPMEM released a product containing a commercial DRAM array and general-purpose sequential cores on a single chip so anyone can obtain it. A sequential core is a simple calculation unit that processes instructions one by one in the order they are written. Researchers ran sixteen memory-bound tasks on two physical systems with 640 and 2,556 cores. According to the authors, evaluations in this field up to that point were simulations or, at best, simplified prototypes (DOI: 10.1109/access.2022.3174101, measurements on physical systems).

If you use commercial parts as-is, you must also follow those parts' rules. Those rules are the invoice for this approach.

In-memory elements next to commercial banks usually wake up all banks at once to maximize parallelism, but then the instantaneous power can exceed the system's limits. This is especially true when stacking multiple such elements for language model inference. To keep power constraints, you can send out the entire instruction only after waking the banks one by one, and during that time, empty cycles occur where no work can be done. One design separated the instruction queue so it does not wait for all banks to wake up. As a result, it was 2.03 to 3.33 times faster than an A100 GPU and 1.11 to 1.52 times faster than the method of waking banks sequentially, and its peak power was 21.3 to 41.7% lower than the simultaneous activation method (DOI: 10.23919/date64628.2025.10992691, design and evaluation).

What commercial in-memory components cannot do is clear in a commercial platform that puts calculators into HBM (HBM-PIM). HBM is memory made by stacking multiple layers of DRAM, used attached right next to the calculating chip (Fetching Is Slower Than Computing — The Memory Bottleneck and HBM). The commercially available HBM-PIM platform has a limited instruction set and lacks a dedicated addition instruction to combine multiplied results inside memory. Therefore, one study redesigned the data flow using an outer product method without an addition stage, which is a method of filling a table by multiplying two lines of numbers across all pairs. In doing so, they actually measured up to 14.9 GFLOP/s per second (billion floating-point operations per second, 59.4 times per cycle) in a pseudo-channel of Samsung's Aquabolt-XL, which is a single independent pathway inside HBM (DOI: 10.1145/3801487.3806067, measurements on commercial hardware).

There is an insurmountable ceiling to the approach of placing calculators on the base layer. According to a 2019 study, most research up to that point placed calculators only as accelerators on the base layer of stacked DRAM. This study designed a method to put calculators inside the DRAM to operate in tandem with standard DRAM operations, and notes that it was 406% faster than the all-banks method and 35.2% faster than the per-bank method in 8-bit integer matrix-vector multiplication. The baseline is not the CPU, but a case where it only reads operands assuming it uses the entire bandwidth of external DRAM. And contrasting it with the base-layer method, it adds: The performance of a calculator placed on the base layer of 3D stacked memory can under no circumstances be better than what that bandwidth provides (DOI: 10.1109/access.2019.2924240).

The approach of processing materials in front of the warehouse door—that is, on the base layer of stacked memory—cannot exceed the rate at which materials come out from inside the warehouse to the door. This is why the same paper brought calculators inside the DRAM.

Memory Solves Logic on Its Own

A 2016 paper introduces a technique that copies an entire row—a single horizontal line of cells in an array—directly to another row inside DRAM. During the copy, not a single byte comes out of the chip. The second technique in the same paper performs a massive amount of AND and OR operations inside memory. Both directly utilize the analog operations that actually occur when DRAM reads values. In existing systems, the only things connections outside memory were allowed to do were reading and writing (DOI: 10.48550/arxiv.1610.09603, preprint).

Bitwise operations are the process of overlapping two very long lines of bit strings entirely and applying the same logic rules to each position all at once. Placing two checklists on top of each other and picking out only the boxes checked on both is an AND operation. When a database filters rows matching a condition, when intersecting neighbor sets in a graph, or when aligning genomes, all of them take this shape.

This bitwise operation is what the 256 chips that opened this article did. However, commercial DRAM does not officially support this operation, so the chips must be pushed beyond the timing recommended by the manufacturer. This is the reason why the term success rate is attached. Consequently, the preceding techniques were exposed to noise and process variation—a phenomenon where devices made with the same design have slightly different properties—so operations were correct only in some columns (a single vertical line of cells in an array). A preprint published before the previous chip experiment addressed this limitation. It discovered that by carefully arranging the instruction sequence, up to 32 rows could be opened simultaneously, and duplicated the same input multiple times to raise the success rate. Across 120 commercial DDR4 chips from two major manufacturers, for seven arithmetic and logic operations, the success rate was 24.18% higher and performance was 121% higher compared to the preceding technique (DOI: 10.48550/arxiv.2312.02880, preprint, measured on commercial chips).

The same thing happens in NAND flash. NAND flash is a memory where contents remain even when the power is turned off, and a block must be erased entirely before rewriting ("Next-Generation Memory — The DRAM Wall and the Race for a "Dream Memory""). A technique incorporating both a method of processing dozens of operands with a single sensing and programming that reduces errors had its operation verified in 160 physical 3D NAND chips. In three real-world applications, average performance and energy efficiency are reported to be 3.5 times and 3.3 times compared to the latest in-NAND processing techniques, and 32 times and 95 times compared to processing methods outside the storage device (DOI: 10.1109/micro56248.2022.00069). The reason why two numbers for the same performance differ by anywhere from ten to nearly thirty times lies in what they were compared against.

The next study went a step further. Until then, in-NAND operations were possible only within the same vertical line, but operations crossing vertical lines were added. As a result, error correction became possible inside NAND, and density increased 1.8 times by putting multiple bits in a single cell instead of the conventional method of putting only one bit. Basic database queries were 2.1 times faster and energy efficiency was 2.5 times compared to the latest technique. However, running the benchmark's queries entirely from end to end yields 1.7 times. Only this value's comparison target is missing from the abstract (DOI: 10.1145/3725843.3756055). The number for a kernel—a single chunk of core computation that runs repeatedly—and the number for the entire program are different.

Non-volatility is the property where stored values remain intact even when power is cut off ("Next-Generation Memory"). Among newly emerging non-volatile memories, some write values as resistance, and here that property is used as-is. When two or more rows are turned on simultaneously, the current flowing into the read circuit contains the result of the logic operation of those rows. A paper proposing this structure in 2016 modified only the read circuit instead of putting complex logic inside memory (DOI: 10.1145/2897937.2898064). In 2024, another research team built the same structure with actual PCRAM devices and demonstrated it through experiments. PCRAM (Phase-Change Memory) is a memory that stores information by changing a material between a crystalline state and an amorphous state ("Next-Generation Memory"). The difference between resistance states reached up to four orders of magnitude. However, the target that the team compared their actual measurements against was the simulation data of the original research (DOI: 10.1109/vlsid60093.2024.00103, experimental demonstration).

There is also research that built digital systems using memristor logic—which allows devices whose resistance changes to perform logical operations—and ran actual database queries (the strict meaning of the name memristor is covered in "Next-Generation Memory"). The baseline is an equivalent in-memory database on the same host—the processor that assigns tasks. "In-memory" here means loading data entirely into memory, not that memory calculates. Filter operations became 1.6 to 18 times faster, and entire queries 56 to 608 times faster. Energy decreased by 1.7 to 18.6 times in filters, and by 0.81 to 12 times in entire queries. 0.81 times means that in some cases more energy was actually used, which the authors wrote down as-is in the abstract. The required device lifespan was within the range of existing ReRAM. ReRAM (Resistive Random-Access Memory) is a memory that changes resistance by creating or breaking conductive paths inside an insulating material ("Next-Generation Memory") (DOI: 10.1109/tetc.2023.3315189, whole-system simulation).

Once memory starts calculating, a new problem arises. Operations that software commands directly to memory are a new category of memory operations alongside reads and writes, but to the authors' knowledge, ordering rules between them and other operations have been barely addressed in existing research. Without rules, a program cannot guarantee whether the value read by memory is the value from before or after the calculation inside memory. Focusing on massive bitwise operation methods, this paper points out that existing related solutions actually broke the host's original ordering rules, making it impossible to guarantee any correctness, and designed and evaluated four rules ranging from strict to loose. The execution time cost to guarantee correctness was at most 6% and in many cases it was actually faster, while the additional hardware cost was less than 0.22% (DOI: 10.1109/hpca56546.2023.10071007, whole-system simulation).

The conclusion of this section lies in a single sentence written by the aforementioned 2016 non-volatile memory logic paper. In graph processing and database applications, measuring only the bit operations separately showed they were about 500 times faster and saved about 28,000 times the energy. The same sentence continues as follows. Overall, compared to the conventional processor, it was 1.12 times faster and saved 1.11 times (DOI: 10.1145/2897937.2898064, proposal and simulation evaluation).

The front number is the gain of the bit operation portion moved to memory, and the back number is the gain of the entire program. The abstract does not state the reason, but according to textbook explanation (Amdahl's Law), no matter how fast the moved portion becomes, the remaining unmoved part runs at its original speed and bounds the overall gain. This direction is not always the case; in the database simulation earlier, the entire query sped up much more than the filter operation. 500 and 1.12 are in the same sentence, and the rest of this article follows that gap.

Electric Current Does Multiplication On Its Own

The analog approach does not break values down into two discrete states; it represents and computes values using the magnitude of voltage or current itself. This is the conclusion of a paper that analyzes the energy efficiency limits of digital and analog in-memory architectures alongside ordinary machines that process data one piece at a time. Analog in-memory architectures can see their energy efficiency rise arbitrarily as the problem size and processor size grow together. Because this conclusion is based on an analysis of theoretical limits, it does not mean actual chips have achieved those values. The three axes the authors followed are the size of the problem, the amount of computation per byte, and the number of bits used for calculation (arXiv:2302.06417v1, theoretical limit analysis, preprint).

A crossbar array is a structure built by laying horizontal and vertical lines in a grid, with a single device whose resistance can be changed placed at each intersection. When different voltages are applied to each horizontal line, a current flows across each intersection equal to the voltage applied multiplied by that device's conductance (the inverse of resistance). Because these currents naturally combine on the vertical lines, the current received at the end of a vertical line becomes the sum of those products.

Most of what neural networks do fits this shape. A neural network is a computational model that repeats the process of multiplying inputs by weights and adding them up layer by layer, where the weights are numbers determined through training. If you store the weights as the conductance of the devices and apply the inputs as voltages, multiplications and additions finish physically in the time it takes for the signal to pass through the circuit once. No instructions or multipliers are needed.

The answer obtained this way is a current, so to pass it on to the next layer, it must be converted into a number. An analog-to-digital converter is a circuit that translates continuously varying values, like voltage or current, into numbers that a computer can handle. You must first decide how many digits to use; every time you add a digit, the electricity consumed and the area occupied by the circuit increase.

A review paper summarizing this field points to three challenges: the converter bottleneck, variations in analog computation, and non-idealities where devices deviate from ideal behavior (DOI: 10.1109/mcas.2021.3092533, review). The answer from a study digging into how precise the converter needs to be was 6 bits. This is the value that provides the best trade-off between performance and area while maintaining accuracy even in large arrays (DOI: 10.1109/mdat.2021.3050715).

Even with design exploration results showing that 6 bits are enough, converters consume a large amount of power and area. According to one study, the area occupied by a single converter determines how many converters can be attached to a single crossbar, and that count constrains throughput. That study moved toward eliminating converters. Reducing the intermediate sum down to 1 bit or 1.5 bits removes the need for converters. Instead, a flood of real-number scaling factors are created that need to be multiplied, and those multiplications eat away at the gains again. The authors' answer was to divide the work. The sum of products is handled by the analog crossbar, while the scaling factors are handled by the digital in-memory array. Compared to an analog baseline structure using 7-bit and 2-bit converters, energy was reduced by up to 28 times and 11 times, with small accuracy losses (DOI: 10.1145/3658617.3697572).

A study that recalculated converter precision suggests that previous studies have overestimated the required precision. It argues that prior studies treated the error caused by fitting values to coarse-grained scales (quantization error) as input-independent noise, while ignoring the fact that the pre-conversion signal is already discrete in nature. When solved again using a metric tailored to computational accuracy, the required precision for a 256-dimensional binary dot product could be reduced by 3 bits while increasing the signal-to-noise ratio—the ratio indicating how much larger the desired signal is than the noise—by 6 decibels (arXiv:2507.09776v2, 28nm circuit-aware behavioral model, preprint).

The Price Analog Actually Pays

There is a fully analog structure that processes not just sums of products, but also nonlinear operations—calculations that change values disproportionately to the input between the layers of a neural network—in the same array, completely eliminating data converters. There is a study that solved this structure using the circuit simulator SPICE. When they loaded a small neural network running from 400 inputs to 10 outputs, the place where problems surfaced was the wiring. The resistance and capacitance of the metal lines forming the grid, which is the property of temporarily holding electricity, disrupt the calculations. When they split the array into 16, 8, and 8 horizontally and 8, 8, and 1 vertically per layer, the handwritten digit recognition accuracy rose up to 94.84%. In the authors' phrasing, it is a value comparable to about 97% produced by a digital implementation on a CPU. The toll is also in the same abstract: extra power is consumed because of the circuits attached to do the splitting (arXiv:2201.12480v1, circuit simulation, preprint).

There is also a price on the accuracy side. Analog computation relies on charge conservation, an operation with infinite resolution. Because even the tiniest shake goes straight into the result, it is exposed to error. The conclusion of a study that used simulations to check whether large-scale neural networks can be loaded onto analog non-volatile devices was that a neural network trained for high-precision digital cannot withstand the uncertainty of analog devices. The same paper does not end there. When they used analog floating-point representation and a new training algorithm, the accuracy improved significantly without adding cost to the inference hardware that generates answers with the trained neural network (arXiv:2008.02400v1, simulation, preprint).

Because effective precision is limited, this field has moved toward mixing analog compute tiles and digital processors. A preprint that reviewed existing methods for deciding which part of a neural network to assign to where applied that procedure to the language model GPT-2. Out of 49 matrices, 4 dominated the sensitivity to precision, and a single attention output in the first block overwhelmed the rest by orders of magnitude (arXiv:2606.02672v1, preprint). Attention is the calculation where the language model assigns weights to figure out which of the preceding words to reference. It is sensitivity measured in that single model, and the authors consider it important to arrange things in units of projection matrices and run the matrices in the front blocks and output side digitally.

A study implementing kernel approximation on a phase-change-memory-based analog in-memory chip showed hardware results within 1% accuracy drop in kernel ridge classification and within 1% in kernelized attention benchmarks. The kernel here is a function that measures how similar two pieces of data are in machine learning, which is a different meaning from the compute chunk earlier. However, in the same abstract, the energy efficiency advantage is written as an estimate rather than a measurement (arXiv:2411.03375v1, physical chip demonstration + estimation, preprint).

A study that built an attention accelerator for Transformers, the neural network architecture used by conversational artificial intelligence, assigned only tasks where the exact magnitude of values does not matter as long as the order is correct to the analog ReRAM array, which is sensitive to circuit non-idealities. This is the job of choosing which tokens—the word fragments handled by language models—are more relevant. Calculations that need to be accurate are handled by digital. The results were 13.5 to 28.5 times faster and 291.6 to 1894.3 times more energy efficient compared to a GPU. Yet, the energy efficiency compared at the same throughput with a state-of-the-art Transformer-dedicated accelerator in the same paper is 1.2 to 14.9 times (DOI: 10.1109/tvlsi.2023.3337777). When the baseline was changed from a GPU to a dedicated accelerator, the hundreds to thousands of times dropped down to single- or double-digit multiples.

Why Not Just Go Digital?

"The analog approach may not be suited for some applications requiring high accuracy." This sentence appears in the abstract of a 2021 conference paper that built a fully digital in-memory circuit using a 22-nanometer process. The authors' diagnosis is as follows: in-memory computing research has focused on the energy-efficient analog side, but a lack of accuracy caused by a low signal-to-noise ratio is the main drawback of that approach (DOI: 10.1109/isscc42613.2021.9365766). The title of this paper stamps 89TOPS/W and 16.3TOPS/mm² right on it, but the abstract does not explain those values.

The digital side's argument is clearest in the abstract of another circuit that processes bits one position at a time in sequence. The authors write that a fully digital implementation is free from process variation, noise sensitivity, and the burden of data conversion, and they write the price in the same abstract. The method of processing bits one by one significantly reduces the space burden, but sacrifices latency.

When that circuit was built as a 128×128 test chip using a 65-nanometer process and measured, the same circuit delivered 117.3 tera-operations per watt when used as 1-bit, and 2.06 tera-operations when used as 16-bit (DOI: 10.1109/jssc.2021.3061508, test chip measurements). The circuit diagram and the manufacturing process are identical, yet there is a nearly 57-fold difference. "Tera-operations per watt" that does not disclose how many bits were used for the calculation cannot be compared.

The second scale problem is where the counting stopped. A study that built and measured a 65-nanometer attention accelerator mixing analog and digital reports that the peak energy efficiency is 14.8 for the analog core and 1.65 for the entire chip (in tera-operations per watt). The performance per area also splits into 976.6 and 79.4 (arXiv:2409.04940v2, test chip measurements, preprint). One paper, one chip, both measured values, yet they differ by nearly nine times. The abstract does not disclose the bit count, and the analog core handles approximate calculations to select tokens to filter out, while the digital processor handles precise calculations for the remaining tokens. The two values measured circuits with different tasks and different scopes. A figure extracted from just a single circuit block contains neither the peripheral circuits nor the data movement between blocks.

Another study that built and measured a 22-nanometer in-memory neural network accelerator makes this problem harder. It states that the 4-bit converted efficiency of the entire accelerator is 813 tera-operations per watt, and a few lines down, extracting just the circuit block yields 4.2 peta-operations per watt. Peta is a thousand times tera. But the abstract does not state which bit standard the latter number is based on. Even when written side by side, they are not values that can be placed side by side. The core device of this chip was a multi-bit analog normalization circuit that bypasses the information lost in the converter's quantization (DOI: 10.1109/jssc.2023.3269098, test chip measurements).

The analysis that "as size scales up together, analog efficiency can theoretically become arbitrarily high" and the diagnosis that "analog is weak in accuracy" do not refute each other. The former is a statement about energy, and the latter is a statement about accuracy. The two papers neither cite nor refute each other, and this article has placed them side by side to show that there are multiple axes that must be measured.

What Remains

According to the authors, because there was no comprehensive tool to evaluate memory access patterns for each section, it was difficult to choose which sections to offload to memory. One study created a tool to view access patterns per section and data movement across sections together using static analysis, a method of reading only code without running the program to figure out its properties (DOI: 10.23919/date58400.2024.10546698, static analysis and evaluation). When dividing as chosen by that tool, it was on average 2.63 times faster compared to using only the CPU, and on average 4.45 times faster compared to offloading everything to memory. The maximums are 7.14 times and 10.64 times, respectively.

Comparing the two numbers side by side, on average in this evaluation, "all-in-memory" was even slower than using only the CPU. Sections suited for offloading to memory and sections not suited are mixed within the program.

A study that ran sparse matrix-vector multiplication—a calculation multiplying a single row of numbers into a large number table where most cells are empty—on 2,528 cores of a physical system also says the answer is not singular (DOI: 10.48550/arxiv.2201.05072, preprint, physical system measurement). The authors designed the data partitioning method in two directions. One is finishing all calculations at the memory-side cores, and the other is striking a balance between calculation cost and data transfer cost. We read the very fact that they released twenty-five kernels and four compression formats together as meaning that the answer changes depending on the shape of the matrix. Which partition won is not in the abstract.

There are cases where the bandwidth cannot be fully utilized even after offloading. A study that accelerated sparse-sparse matrix multiplication beside the bank reports through its own experimental analysis that previous in-memory studies failed to use the banks simultaneously and sufficiently, thereby wasting 80% of the memory bandwidth (DOI: 10.1109/tc.2025.3624920). When finding non-overlapping columns and processing them simultaneously across multiple banks, it became up to 17.88 times faster than the latest dedicated accelerator and up to 8.19 times faster than the latest in-memory method.

The gaps in software were written down most specifically by the team that built a 28-nanometer in-memory microcontroller. A microcontroller is a low-power processor that goes into small devices like home appliances or sensors. Products up to that point diagnosed by the team are as follows. The in-memory circuits were large and consumed area, most were analog mixed-signal so robustness against process, voltage, and temperature variations was limited, and not a single one supported practical development frameworks like TensorFlow Lite. Therefore, those products could not produce standard benchmark scores and were difficult to compare with the latest non-in-memory products. The achievements of the chip made by that team are also reported compared to existing in-memory products (DOI: 10.1109/jssc.2024.3362274, prior research diagnosis + actual prototype measurement).

"Difficult to compare" is different from saying performance is bad. It means that even though there is a common ruler called standard benchmarks, those products could not post scores on it.

Attempts to fill this spot are numerous. A study that attached in-memory operations to a standard execution environment reports that the reference platform was on average 1.64 times faster than an x86 CPU and 1.71 times faster than an ARM CPU (DOI: 10.1109/iceic54506.2022.9748444). Two studies targeting the physical UPMEM system are more specific. A framework that automated data movement and task distribution—jobs that humans previously had to write by hand—using a pattern-based interface was on average 2.1 times faster than a hand-tuned implementation while using 94% fewer lines of code (DOI: 10.48550/arxiv.2310.10168, preprint, physical system measurement). A compiler that stripped away reliance on hand-tuned libraries and generated code through automated search reports performance improvements of up to 6.18 times for kernels and up to 8.21 times for language model layers on physical hardware. This value does not disclose what it is compared against (DOI: 10.1145/3695053.3731096).

That automated tools beat hand-written code by 2.1 times is read as meaning humans had not been utilizing this hardware properly up to that point. This is our reading.

The programming model is also a stumbling block. Because in-memory accelerators bundle memory and computation into a single mass, they break the standard programming model premised on the von Neumann architecture which separates calculation and storage, and thus are difficult to integrate into existing software stacks. A study that attached an in-memory arithmetic unit as an execution unit to a RISC-V processor and even built dedicated instructions, a compiler, an assembler, and a profiler writes down the problem like this (DOI: 10.3389/felec.2022.898273). RISC-V is a processor instruction specification made public for anyone to use. As a result of such integration, a single multiplication kernel became 17.63 times faster, and the overall benchmark became 2.74 times faster against the same baseline.

The baseline is the same and the hardware is the same, but it is 17.63 and 2.74. The gap that existed between 500 times for bit operations and 1.12 times for the whole program appears here as well.

Two Sides of the Same Wall

"The performance of a processor placed on top of a 3D-stacked memory's base die can under no circumstances exceed what its bandwidth provides." This sentence quoted earlier was added right after a paper that brought processors inside DRAM to break past the base-die ceiling reported a 406% result. When you place a processor on the base die, the width of the path is the upper limit (DOI: 10.1109/access.2019.2924240).

The proposal to send compute to memory does not compete with construction to widen the path. One of the two branches picked by the prior overview paper was also to provide a wide path to the calculation circuits inside it using the logic layer of 3D-stacked memory (arXiv:1905.04376v1). Half of the proposals to eliminate the path are built upon technologies that widened it. Even a near-memory design from 2015 that used off-the-shelf DRAM almost as-is was an exception only in that it added through-silicon vias (DOI: 10.1109/hpca.2015.7056040). Without that wiring piercing top and bottom, this design does not exist either.

The path only gets shorter, not gone. A study that went down to the internal structure of DRAM and explored the computation method through simulation across the entire design space writes this after running additions, subtractions, bit counts, shifts, and multiplications: Even when executed right next to memory, data movement accounted for more than one-third of the total execution time, and thus was a major bottleneck (DOI: 10.18130/q5ar-1c61, dissertation and simulation framework).

Construction to widen the path and proposals to send compute to memory are two sides of the same wall. Sticking to the restaurant analogy from "Fetching Is Slower Than Computing", one side is construction to widen the hallway between the warehouse and the kitchen, while the other side is finishing dishes inside the warehouse or at the door to reduce the cargo moving back and forth through the hallway. However, the path construction actually stepped on by the items in this article is only one stacking, and the store-bought chips that opened this article did not even need that. The term "two sides" points to the fact that there is a single wall, not that all path construction overlaps here.

The prior technology history study summarizes why this idea appeared in the 1950s, sank, returned in the 1990s, and sank again as follows. Research ideas reappear when surrounding technologies, economic conditions, and application demands have ripened enough to support them. That study considers AI computation, three-dimensional stacking, and growing data movement costs to have made this idea worth revisiting (DOI: 10.18130/q5ar-1c61).

What then is left? From here on, this is not citations, but our own synthesis putting what we read earlier in one place. You cannot answer this proposal with "wins/loses"; what you can answer are the following six questions.

① How many calculations are performed per byte of data? What this approach mainly targets is work bound to memory due to low operational intensity per byte. Which work is like that differs from task to task.

② Which parts should be sent down? On average, sending everything down was the slowest case, and a tool that automatically selected segments outperformed that choice by an average of 4.45 times.

③ How many bits are needed? The same circuit is 117.3 at 1 bit and 2.06 at 16 bits. Efficiency figures that do not disclose precision cannot be compared.

④ Do those numbers belong to a circuit block or the entire chip? Within a single chip, analog cores were 14.8, while the entire chip was 1.65.

⑤ Who writes that code? A diagnosis was published in 2024 that none of the in-memory microcontrollers up to that point supported a practical development framework, and the team that wrote that diagnosis built chips themselves.

⑥ Are there ordering rules for new operations? Ordering rules between in-memory operations and outside operations were barely covered, and the cost of maintaining those rules in a bulk bit-operation method came out to a maximum of 6% of execution time in one simulation.

In our synthesis, for ③ and ④, what needs to be disclosed alongside the figures became clear through the evidence in this article, and for ⑥, one simulation from one branch of methods produced the scale of the cost. For ① and ②, the answers differ depending on the task and program, and for ⑤, at the time of that diagnosis, it was difficult to compare them using standard benchmarks. Therefore, the question fitting this technology is not "When does it replace the von Neumann architecture?" but rather "Which tasks should be brought inside the wall?"

The fact that 256 store-bought DRAM chips performed a NOT operation with a success rate of 98.37% means that physics capable of computation was built into those chips from the start. The authors view this result as showing the potential to use DRAM as a computing substrate. However, that operation occurred outside the timings recommended by the manufacturer.

Source Papers

• "Functional Completeness of Commercial DRAM Chips for Boolean Logic" (2024) — DOI: 10.1109/hpca57654.2024.00030 — Measured on 256 commercial DDR4 chips. NOT 98.37%, 16-input logic 94.94~95.87%, 50~95 degrees 1.66%.
• "PULSAR — Simultaneous Activation of Multiple Rows in Commercial DRAM" (2023) — DOI: 10.48550/arxiv.2312.02880 (preprint) — Measured on commercial chips. Compared to prior techniques, achieving what used to succeed only in some columns at a success rate of 24.18% and performance of 121%.
• "Pragmatic Near-Memory Computing" (2019) — arXiv:1905.04376v1 (preprint) — Overview and position paper. Two branches (analog properties of DRAM / 3D-stacked logic layers) and adoption challenges.
• "Horizontal Near-Memory Design Space Exploration and Brief History" (2026) — DOI: 10.18130/q5ar-1c61 — Dissertation and simulation. Even in near-distance execution, data movement accounts for over a third of execution time. 1950s–60s perception devices, 1990s EXECUBE.
• "NDA — Near-Data Acceleration with Commercial DRAM and Standard Modules" (2015) — DOI: 10.1109/hpca.2015.7056040 — Compared to a system placing the same accelerator inside the processor, 46% lower energy, 68% lower transfer, 1.67x performance.
• "Benchmarking New Paradigms" (2022) — DOI: 10.1109/access.2022.3174101 — Physical system measurements. 640 and 2,556 cores, 16 memory-bound workloads.
• "Processing "Inside" Memory Tailored to DRAM Operation" (2019) — DOI: 10.1109/access.2019.2924240 — 406% overall bank and 35.2% per-bank (baseline = reading operands saturating the bandwidth). An approach placing arithmetic units deep inside DRAM. A sparse sentence by comparison: An arithmetic unit on the base die can under no circumstances be better than that bandwidth.
• "DEAR-PIM — Split Execution of All-Bank Requests" (2025) — DOI: 10.23919/date64628.2025.10992691 — Design and evaluation. 2.03~3.33x compared to A100, 1.11~1.52x compared to sequential, 21.3~41.7% lower peak power compared to simultaneous activation.
• "AME-PIM — Can Memory Become a Tensor Accelerator" (2026) — DOI: 10.1145/3801487.3806067 — Measurements on commercial hardware. Limited instruction set, no native accumulation → outer product flow without accumulation. Up to 14.9 GFLOP/s per pseudo-channel.
• "Processing in Memory — The Tipping Point" (2016) — DOI: 10.48550/arxiv.1610.09603 (preprint) — Memory interfaces only read and write. Row copy using the analog behavior of DRAM.
• "Flash-Cosmos — In-Situ Compute Power of NAND Flash Memory" (2022) — DOI: 10.1109/micro56248.2022.00069 — Verified with 160 physical 3D NAND chips. 3.5 and 3.3 times compared to internal techniques, 32 and 95 times compared to processing outside storage.
• "CrossBit — In-NAND Computation via Inter-Bitline Communication" (2025) — DOI: 10.1145/3725843.3756055 — 1.8x density via internal error correction. 2.1x basic queries, overall 1.7x (control group not stated).
• "Pinatubo — A Framework for Enabling Massive Bit-Wise Operations in Non-Volatile Memories" (2016) — DOI: 10.1145/2897937.2898064 — Proposal and simulation evaluation. Roughly 500x bit operations and roughly 28,000x energy, while overall being 1.12x and 1.11x (baseline = conventional processor).
• "Bit-Wise Logic in Phase Change Memory via the Pinatubo Architecture" (2024) — DOI: 10.1109/vlsid60093.2024.00103 — Experimental demonstration. Contrasted against simulation values from the original study. Four-order-of-magnitude difference in resistance states.
• "Massive Bit-Wise Processing in Memory from a Database Perspective" (2023) — DOI: 10.1109/tetc.2023.3315189 — Full-system simulation. 1.6~18x filters, 56~608x queries, 1.7~18.6x energy, and 0.81~12x (0.81 = used more).
• "Consistency for Massive Bit-Wise Processing in Memory" (2023) — DOI: 10.1109/hpca56546.2023.10071007 — Full-system simulation. There were no ordering rules, and existing solutions broke host rules. Up to 6% guarantee overhead.
• "Analog In-Memory Computing Architectures for AI" (2023) — arXiv:2302.06417v1 (preprint) — Theoretical limit analysis. Analog efficiency can scale arbitrarily high. Axes are size, compute per byte, and precision.
• "In-Memory Computing Chips for Deep Learning" (2021) — DOI: 10.1109/mcas.2021.3092533 — Review. Challenges are converter bottleneck, analog computation variability, and device non-idealities.
• "Converter Design Exploration for In-Memory Accelerators" (2021) — DOI: 10.1109/mdat.2021.3050715 — Design exploration. With 6 bits, there is no accuracy loss even in large arrays.
• "HCiM — Hybrid Converter-Less In-Memory Accelerator" (2025) — DOI: 10.1145/3658617.3697572 — Converter area bounds throughput. Reducing intermediate sums can eliminate converters, but floating-point scaling offsets the gains → scaling is digital. 28 and 11 times compared to 7 and 2 bits.
• "Optimal SNR Converters for Analog In-Memory Computing" (2025) — arXiv:2507.09776v2 (preprint) — Circuit-aware behavioral model at 28nm. Prior studies overestimated precision requirements. 3 bits ↓, 6 dB ↑.
• "Parasitic Wiring and Partitioning in Fully Analog Architectures" (2022) — arXiv:2201.12480v1 (preprint) — Circuit simulation. Partitioning achieves 94.84% (comparable to ~97% for digital). Partitioned circuitry incurs higher power.
• "Accuracy and Resilience of Analog In-Memory Inference" (2020) — arXiv:2008.02400v1 (preprint) — Simulation. Neural networks trained for digital cannot withstand analog uncertainty. Improved via new representation and training.
• "Heterogeneous Mapping for Analog In-Memory Accelerators" (2026) — arXiv:2606.02672v1 (preprint) — In GPT-2, 4 out of 49 projections dominate sensitivity.
• "Kernel Approximation Using Analog In-Memory Computing" (2024) — arXiv:2411.03375v1 (preprint) — Physical chip demonstration. Accuracy drop less than 1% and within 1%. Energy advantage is an estimate, not a measurement.
• "HARDSEA — Hybrid Analog-Digital Attention Accelerator" (2023) — DOI: 10.1109/tvlsi.2023.3337777 — Assigning only tasks requiring monotonicity to analog. 291.6~1894.3x energy compared to GPU, 1.2~14.9x compared to dedicated accelerators.
• "A 22nm All-Digital Full-Precision In-Memory Computing Macro" (2021) — DOI: 10.1109/isscc42613.2021.9365766 — Diagnosis from the abstract: Analog's primary drawback is insufficient accuracy caused by low signal-to-noise ratio. ⚠️ The 89 TOPS/W in the title is not explained by the abstract.
• "Colonnade — Digital Bit-Serial In-Memory Computing Macros" (2021) — DOI: 10.1109/jssc.2021.3061508 — Measurements on a 65nm test chip. Sacrificing latency to reduce area. 117.3 tera per 1-bit watt / 2.06 tera per 16-bit watt.
• "IMPACT — A 22nm In-Memory Computing Accelerator" (2023) — DOI: 10.1109/jssc.2023.3269098 — Prototype measurements. 813 TOPS/W converted for 4 bits across the entire accelerator, 4.2 POPS/W for the macro alone (the precision baseline for the latter figure is absent from the abstract).
• "Charge-Based In-Memory Attention Accelerator" (2024) — arXiv:2409.04940v2 (preprint) — 65nm prototype measurements. Highest efficiency analog core 14.8 / chip-wide 1.65 TOPS/W.
• "AI-PiM — RISC-V with In-Memory Execution Units" (2022) — DOI: 10.3389/felec.2022.898273 — Breaking the standard von Neumann model to prevent stack integration. 17.63x kernels / 2.74x benchmarks against the same baseline.
• "iMCU — A 28nm Digital In-Memory MCU" (2024) — DOI: 10.1109/jssc.2024.3362274 — Prior research diagnosis + prototype measurements. Prior products had robustness limited by variability and lacked practical framework support, making it hard to compare against standard benchmarks — a product built by the team that wrote that diagnosis. Performance compared to existing in-memory products.
• "ONNX Runtime Extension for In-Memory Execution" (2022) — DOI: 10.1109/iceic54506.2022.9748444 — Abstracting in-memory operations in a standard execution environment. 1.64x compared to x86, 1.71x compared to ARM as baseline platforms.
• "DaPPA — Data-Parallel Programming Framework" (2023) — DOI: 10.48550/arxiv.2310.10168 (preprint) — Physical system measurements. Average 2.1x compared to hand-tuned, 94% fewer lines of code.
• "ATiM — Automatic Tensor Program Tuning for In-Memory DRAM" (2025) — DOI: 10.1145/3695053.3731096 — Physical hardware measurements. 6.18x kernels, 8.21x language model layers (baseline not specified).
• "A3PIM — In-Memory Offloader" (2024) — DOI: 10.23919/date58400.2024.10546698 — 2.63x compared to CPU-only, 4.45x compared to running entirely in memory (average. Up to 7.14 and 10.64x).
• "SparseP — Sparse Matrix-Vector Multiplication on Physical In-Memory Architectures" (2022) — DOI: 10.48550/arxiv.2201.05072 (preprint) — Physical measurements, 2,528 cores. Partitioning that finishes entirely in memory versus partitioning that balances computation and transfer costs. Win-loss outcomes are not in the abstract.
• "HighP — High Bank-Parallelism Sparse Matrix Multiplication" (2025) — DOI: 10.1109/tc.2025.3624920 — Internal experimental analysis showed 80% of bandwidth was being wasted. 17.88x compared to dedicated accelerators, 8.19x compared to in-memory.