Three weeks earlier, at "FMS 2026," the world's largest memory and storage event, the three memory firms had unveiled new technologies to break through memory bottlenecks; Hot Chips was a venue to explain those technologies in a more specialized and academic manner. To reduce memory bottlenecks, memory companies are pursuing three-dimensional (3D) stacking and hybrid bonding (directly connecting metal pads between chips without an intervening material) as breakthroughs.
HBM stacks multiple DRAM memories to raise bandwidth and speed up the computation of AI processors such as graphics processing units (GPUs). While the current mainstream HBM has 8 or 12 stacked layers, Samsung Electronics and SK hynix are honing technology to stack 16, 20, or more layers. Attendees fired pointed questions, pressing on why they are increasing the number of layers even at the cost of heat problems, and whether there are anticipated issues. The keynote presentations were given by Han Sang-wook, Technical Leader (TL) of the DRAM Design Team at Samsung Electronics' Memory Business Division; Lee Jae-sik, Vice President of SK hynix America; and Raghu Sriramaneni, Micron Fellow. Below are the key questions and the answers from the three memory makers' presenters.
Q. The bandwidth (data transfer per second) of a single DRAM reaches about 20 terabytes (TB) per square centimeter. In the presentation you said you would stack HBM memory up to 20 layers. The thickness would be twice that of HBM4, but one can calculate that throughput per memory drops sharply to about 4TB. Stacking 20 layers reduces bandwidth to 4/20, or 20% — a large drop. Is this the right choice? Why focus on stacking higher rather than going faster? HBM also requires more capacity than DDR (double data rate DRAM) or LPDDR (low-power DRAM). Stacking 20 layers makes the average speed slower than DDR5. Why do you see it as better to use HBM stack height instead of intelligently placing cheaper memory around it?
▷SK hynix: The answer must be found based on the requirements for training and inference. Some people say this: there are cases that want a sudden surge in workload (processing volume) and high bandwidth. They want to use the KV cache (which distributes and stores previously performed computations in short-term memory to avoid repeating them for faster computation) only on very high-bandwidth memory, and then place memory with lower performance, like LPDDR, next. I think that's all the architecture (structural design) decisions we have to make. On the training side, there are tasks that require much higher bandwidth, and cases that require more processing not only in capacity but also in workload. So that's why we're going in that direction. Especially in inference workloads too, customers want us to develop high-density, high-bandwidth memory. Beyond that (after high workload processing), other tasks can be distributed to other memory such as LPDDR. This is an issue we'll need to keep thinking about going forward.
Q. Where do you see the limit of HBM stacking? At how many layers can you no longer stack, and because of what problem? Is it a heat generation issue, or another factor?
▷Micron: Technically, many problems can be solved depending on how thin the DRAM can be made. When stacking DRAM, heat issues must be considered to avoid affecting quality. That's because each added layer creates a barrier in heat generation and dissipation. As stacking increases, there are more challenges to solve in terms of heat management and mechanical issues. There is a good path to 16 layers now, and 20 or more layers may be the target. Beyond 16 layers, there's still a lot to solve to achieve higher stacking.
Q. You explained that the silicon (wafer) consumption when manufacturing HBM3E per gigabyte (GB) is three times that of DDR5. Will this ratio be maintained even as new technologies are applied? Will the numbers get better, or worse?
▷Micron: This number is actually a somewhat old version. This figure doesn't support future data, but it certainly won't get better. There are many challenges: we have to increase bandwidth more, increase parallel throughput more, increase die size more, and raise stack height more. The only way to make it better is to lower performance expectations, but we can't do that. We need to change the curve shown in the graph, but we haven't yet found a way to do that.
Q. You said heat issues will become increasingly important for performance. How do you plan to solve the heat problem?
▷Micron: You can make a list of all the problems and each party can take a share of the role. It ranges from cooling solutions to ways of dissipating more heat. Many innovations are underway to reduce thermal resistance. Hybrid bonding will also help. Our understanding of how heat flows is also getting better. Better understanding lets us design accordingly. On the chip design side, we also want to find ways to reduce power density. Managing hotspots where very high power density is concentrated is really important. So we have to take a multidimensional approach to how to manage it more efficiently in terms of packaging and cooling technology. The design teams are working to reduce thermal resistance from the perspective of materials science and physical design. There's also the issue of how to plan power density from a fundamental perspective. At the basic design stage, placement planning that considers heat is becoming increasingly important.
