SILICON VALLEY — "A company that has never touched a machine says it can build a chip overnight? Well. Will that really work?"
A developer who has worked on artificial intelligence chip design in California's Silicon Valley for nearly a decade offered that blunt assessment when asked about the companies that have recently entered the AI chip market. "Software and hardware are completely different," the developer said. "It is impossible for a company that does not understand infrastructure to immediately catch up with firms that have spent 10 or 20 years working on chips."
The developer made an exception for Google and Amazon Web Services, both of which began chip development long ago. "Google started out as a search service, but it has already spent more than 10 years developing chips, and now even Nvidia is watching out for TPUs," the developer said. "Unless you are at Google's level, chip companies do not pay much attention. You cannot dismiss accumulated know-how. Chip development is never easy."
Nvidia has dominated the AI infrastructure market. Graphics processing units, once used to improve computer graphics performance, proved well suited to AI training, and demand for Nvidia's GPUs surged. As a result, Nvidia captured 90% of the market for AI accelerators, the core infrastructure in data center AI servers. Software companies that had relied on Nvidia GPUs came to realize how important chips are and concluded that they needed their own silicon to escape dependence on Nvidia.
Many software companies have since entered the AI chip market. Amazon Web Services launched its cloud computing service in 2006 and began building custom silicon in 2012. Google released its first tensor processing unit in 2015 and has accumulated more than a decade of expertise in custom AI chip development. AWS now leads in inference chips with Inferentia, and Google with its TPUs. Apart from those two companies, which entered the infrastructure business long ago, most began chip development only recently.
Microsoft unveiled its first AI chip, the Maia 100, in November 2023 and released the Maia 200, with improved inference capabilities, in January this year. The Maia 200 reflects a shift in the AI industry from large language models toward AI agents that make their own judgments, making inference performance more important.
Microsoft is expanding its chip effort aggressively to counter Nvidia's GPUs and catch up with Google's TPUs. The Information reported on Aug. 10 that Microsoft has set a target of sharply increasing production of its next-generation AI chip, the Maia 300, next year to attract large cloud customers, and is in talks with Taiwanese foundry TSMC.
Microsoft could unveil the Maia 300 as early as this month and is negotiating to secure more than 300,000 units next year, according to the report. That would be a sharp increase given that Maia 200 output is in the tens of thousands. Morgan Stanley expects Google to produce more than 3 million TPUs this year and 5 million next year. Microsoft Chief Executive Satya Nadella stressed the need for in-house chips in an email disclosed through a 2022 court filing, saying the company was only a thin layer on top of Nvidia and that all of the intellectual property belonged to OpenAI.
Meta, the operator of Facebook, unveiled four chips in its Meta Training and Inference Accelerator lineup — the MTIA 300, 400, 450 and 500 — in March this year and declared that it would release new chips every six months. MTIA is a custom chip, or ASIC, that Meta developed internally to run its data centers. At the March announcement, Meta Vice President of Engineering Jiwon Song said releasing a new chip every six months was an unusually fast cycle, adding that the company was expanding capacity very quickly and investing heavily in capital spending so that it would be ready to use cutting-edge chips at any time.
Anthropic and OpenAI have also joined the race for in-house chips alongside hyperscalers, the operators of large-scale computing infrastructure. The two companies, which developed their Claude and GPT models by leasing data centers equipped with Nvidia GPUs and Amazon and Google chips, are now preparing for chip independence.
Reuters reported in April, citing sources, that Anthropic was considering developing its own chip. The Information reported in July that Anthropic had begun early-stage work on an in-house AI chip and was in discussions with Samsung Electronics' foundry business as a potential manufacturing partner.
OpenAI, which said in October last year that it would work with Broadcom on custom chip development, unveiled its first inference-only chip, Jalapeño, in July. Jalapeño is expected to be installed in data centers OpenAI is building, with the chip designed to OpenAI's own specifications.
The Achilles' heel of the latecomers is their limited experience with hardware — with machines. Microsoft, which built the Windows operating system; Meta, which runs social media services including Facebook and Instagram; and Anthropic and OpenAI, which build AI models, are all software companies. Microsoft runs its Azure cloud service through data centers but remains heavily dependent on GPUs from Nvidia and AMD.
AI chip leaders such as Nvidia and AMD are drawing startups into their orbit to solidify ecosystems built around their own products and block challenges from latecomers. Late last year Nvidia carried out an indirect acquisition of Groq for $20 billion, or 28.3 trillion won, the largest deal in its history. Rival AMD agreed last month to acquire Talas, a Canadian startup that makes chips for AI inference.
The companies that "have never touched a machine" are well aware of their own weakness. Arm, the world's largest semiconductor intellectual property firm, presented its first in-house chip, the Arm AGI CPU, in March, 35 years after the company was founded, and disclosed key specifications last month at Hot Chips, a semiconductor industry conference. A company that had only drawn up design blueprints broke with its tradition of not competing with customers and challenged makers of data center server chips.
Arm Chief Financial Officer Jason Child said in an interview with The Information last month that supplying silicon is decidedly more complex than obtaining a design license, describing chip supply as no easy challenge. Even with more than 90% of the market for application processor architecture — the instruction set that determines how a chip operates — in smartphones, actually designing a semiconductor is a different kind of business, he said. Because it is making a chip for the first time, the production capacity allocated to it by contract manufacturers will be quite small and it could take several years to raise output, Child added, saying it would also take years to secure memory capacity.
Despite the challenge posed by limited chip development experience and obstruction by Nvidia and AMD, some argue that building an alliance could turn the effort into a David-and-Goliath fight. That is because companies such as Broadcom and Marvell help with chip design. Broadcom and Marvell take on design work in response to customer requests to build chips tailored to their AI models. They oversee silicon manufacturing, power and thermal management and memory integration, as well as final preparations for foundry production. As the AI market has shifted since last year from a focus on training to agentic AI, in which inference matters more, demand for custom chips specialized for inference has surged, and Broadcom and Marvell, which design ASICs, have been flooded with requests. The head of a Korean AI startup said one reason the Nvidia-centered structure is breaking down is that Broadcom has built its platform so well. "Broadcom is doing so well that chips with virtually no difference in performance are coming out the same way," the executive said.
