SILICON VALLEY — Samsung Electronics (005930.KS) and SK hynix (000660.KS) said they will sharply accelerate artificial intelligence performance with next-generation memory.
Kim In-dong, vice president of memory product planning at Samsung Electronics' U.S. unit, Device Solutions America (DSA), explained zHBM performance in a presentation at the AI Infra Summit held at the Santa Clara Convention Center in California on the 16th. zHBM, a next-generation high-bandwidth memory, is a technology that stacks memory directly on top of an AI accelerator.
"The response speed of current conversational AI systems is around 100 tokens per second per user," Kim said in the presentation. "Going forward, for agentic AI, we are aiming for a quantum jump that raises this tenfold to 1,000 tokens per second." He stressed that vertically stacked zHBM is the only breakthrough that can reach that goal.
SK hynix is moving beyond the limits of graphics processing units and high-bandwidth memory with processing-in-memory (PIM) technology, which carries out computation directly inside the memory chip. PIM handles simple data in a computing unit inside the memory rather than routing it through the GPU.
Lim Eui-chul, senior vice president in charge of Solution AT at SK hynix, noted in his presentation that the industry mainly uses SRAM when ultrafast inference is required. "SRAM-based systems deliver high bandwidth, but they have clear limits in capacity and cost," he said. He added that "PIM provides roughly 300 times the capacity of SRAM in the same area while maintaining high internal bandwidth."
Lim said the industry must move past an era of single-track infrastructure in which every AI service is handled by one combination of GPUs and HBM. He stressed the need for innovation across memory, including PIM and high-bandwidth flash (HBF), along with software optimization and integrated design.
