Samsung's Processing-in-Memory (PIM)
Points and comments are a snapshot, not live.
Samsung's LPDDR5X-PIM embeds MAC units in DRAM banks for in-memory LLM inference.
Samsung's LPDDR5X-PIM places a processing block with a MAC tree at each of 16 DRAM banks, exploiting internal bandwidth of 614 GB/s versus 76.8 GB/s externally. It supports INT8/FP8 and 4-bit operations, delivering 2.4 TOPS per chip (9.6 with eight chips). PIM mode uses special row addresses and broadcasts commands across all banks, acting as a constrained SIMD processor. The chip works with standard memory controllers, but software challenges include mandatory uncacheable memory, disabling prefetching and out-of-order execution, and complex OS isolation to prevent interference between PIM and normal accesses.
Each PIM block can only quickly access its local bank; inter-bank data must move through the external interface. Samsung internally achieved large performance gains over standard LPDDR5X, but the article notes steep software hurdles: no concurrent PIM and regular DRAM use, no caching, no speculative loads, and no easy multitasking without blocking all other threads or disabling interrupts.
What commenters are saying
Commenters see LLM inference as the killer app, especially for next-token prediction's vector-matrix multiplies. Some argue PIM's constrained SIMD model suits GEMV but not GEMM, while others note that tiling across banks for larger layers is feasible if input/output vectors are small. A few recall past attempts at in-memory compute via standard DRAM string instructions, which failed due to tradeoffs. Concerns center on the fragmented memory space and inability to use caches, though some note JEDEC's upcoming LPDDR6-PIM standard could let CPU vendors make caches PIM-aware. Several question whether the complexity justifies the modest per-chip TOPS, especially given GPUs' flexible dequantization and established programming models.