Artificial Intelligence Stakeholders Target Vertical Integration as They Attempt to Build a Proposition Capable of Competing with NVIDIA

Since 2020, NVIDIA stock price has grown by more than 7X, and it briefly even joined the US$1 trillion valuation club. This success has been built on its long-term, vertically integrated bet on Artificial Intelligence (AI). Its approach has always been “platform” or ecosystem-focused aiming to position itself as a “one-stop shop” for enterprises and Independent Software Vendors (ISVs) to build, develop, deploy, and scale AI. Its market-leading Graphics Processing Units (GPUs) (A100 and H100), in combination with its CUDA software has made this strategy an undeniable success. CUDA is a closed-source parallel computing platform and Application Programming Interface (API) that enables usage of its GPUs. It has acted as a protective moat keeping developers locked into its GPUs. NVIDIA has not finished, as it expanded to support Large Language Model (LLM) deployment by rolling out NeMo (a developer framework for generative AI model deployment) and has even recently integrated with Hugging Face.

All of this has left NVIDIA riding high at the top of the market. Now, NVIDIA’s competitors are trying to bridge the hardware-software gap in the hope that they can catch up. Intel has been very active as it looks to build out its software footprint. In late 2022, Intel acquired Granulate, while it recently made Intel Developer Cloud public, enabling easy access for ISVs and developers to build and run applications on Intel hardware and software. AMD has focused on expanding the Ryzen AI platform with the acquisitions of Mipsology (optimization tools for AI workloads) in 2022 and Nodi.ai (an open-source software team that has already worked on AMD Instinct data center accelerators and other chips) this year. Following less capital-intensive strategies, MediaTek and Qualcomm have partnered with Meta to support LLM optimization for on-device generative AI deployment.

Not only are chip vendors looking to compete with NVIDIA, but hyperscalers are looking to reduce their reliance on expensive NVIDIA GPUs through in-house development of optimized AI chipsets. Amazon Web Services (AWS) is no stranger to chips with Inferentia and Trainium custom cloud chips; and neither is Google Cloud with its Cloud Tensor Processing Units (TPUs). But others are now joining the party as Meta unveiled MTIA, a generative AI inference accelerator for internal use, while Microsoft is reportedly working with AMD on its new Athena chip. In addition, Chinese cloud giants are accelerating investment in proprietary chips. In 2022, Alibaba Cloud unveiled the development of a System-on-Chip (SoC) with RISC-V, while Huawei continues to invest in its Ascend processor range.

A multitude of commercial and technical factors are driving chip vendors and hyperscalers toward vertically integrated strategies that bridge the gap between hardware and software:

• Commercial
  • Build “Gated” ISV Ecosystem with Access to Hardware and Software: Chip vendors are recognizing the importance of attracting ISVs and developers into their AI ecosystem as enterprises/consumers look to find “killer applications.” Building a vertically integrated ecosystem that provides Machine Learning Operations (MLOps) tools, models, and optimization will resonate strongly with key partners as it significantly lowers the barriers for deploying applications, especially if these vendors remain committed to open source.
  • Limit Exposure to AI Accelerator Supply Chain and Expensive NVIDIA GPUs: NVIDIA GPUs are the hottest commodity in the market, and procurement is challenging and expensive. For hyperscalers, building chips in-house reduces (but does not eliminate) reliance on these GPUs. NVIDIA’s chips will remain critical for “giant” model training, given their performance and subsequent time/power saving, but hardware build in-house will at least limit this exposure for certain workloads. On top of this, hyperscalers can gain greater supply chain visibility, which is especially important given the U.S.-China chip war and supply chain disruption caused by geopolitical challenges.
  • Building an Additional Channel to Market: Chipset vendors building out their software portfolio through acquisition will hope to retain existing users and introduce them into their hardware ecosystem. In addition, new customers using software platforms for MLOps will find it easier to leverage hardware that is optimized for the tools/models that they are using.
  • Improve Economies of Scale beyond Commoditized Technology Approaches: Scaling AI deployments using third-party hardware is expensive, while building in-house enables you to scale production in line with requirements, enabling the company to benefit from economies of scale.
• Technical
  • Performance and Energy Optimization: Price/performance is the key technical factor that motivates hardware/software integration. Vertical integration allows stakeholders to optimize resources to run chosen models/software/applications more efficiently. Memory, latency, accuracy, and throughput can all be tailored to the type of workload and models run. This will have a significant impact on training and inferencing time, which will reduce energy consumption (probably the biggest challenge for generative AI) and create significant cost savings for hyperscaler running free-to-access applications like Bard or ChatGPT.
  • Explore New Deployment Types beyond the Data Center: Edge/device AI will continue to expand, so targeting these environments requires strong alignment between hardware and software, especially around power consumption and memory.

Building vertically integrated platform strategy will certainly be sensible, but it will not be simple. Stakeholders will still face an uphill struggle and the right commercial decisions must be made:

Hyperscalers

Building chips in-house seems like a good idea. They have available capital, and it will lessen their reliance on NVIDIA. However, chipset foundries remain capacity constrained, and very few plans, except Intel’s IDM 2.0, are being worked on to alleviate this supply-side constraint. Gaining capacity at TSMC or other foundries will be challenging, and we certainly wouldn’t advise in-house manufacturing. Working with existing chip vendors (e.g., AMD, Intel) to build custom chips may be the “smart play,” as they will have existing contracts with access to foundry capacity. In addition, hyperscalers should not look to eliminate NVIDIA GPUs from their cloud environments, as they remain necessary for compute-intensive training and inference workloads for “giant” models and performance-sensitive workloads. Instead, they should leverage chipsets developed in-house to lessen their burden by splitting duties between training “giant” models with NVIDIA, and then using lower-power processors for other training and inferencing workloads.

Chipset Vendors

NVIDIA may dominate the market, but ISVs/application developers remain concerned about CUDA-constrained “closed” environments due to vendor lock-in fears. This fear offers chip vendors hope, as they can offer an open-source alternative that alleviates much of this vendor lock-in concern. Intel is a big proponent of open-source, having built Open Vino, while AMD’s acquisition of Nodi.ai highlights its commitment to this community. Given NVIDIA fears and wider market dynamics, making its commitment to the open-source community a core tenet of its “go-to-market” strategy will be a clear differentiator. In addition, competing with NVIDIA on the data center high-performance GPU for “giant” models training will be nearly impossible.

Instead, these players should position their new integrated value proposition to target different deployment environments or workload types. Two areas to target are: 1) edge and on-device AI, which is likely to grow substantially over the next 7 years with associated MLOps software revenue increasing over 12X by 2030 (see ABI Research’s Artificial Intelligence (AI) Software report (MD-AISOFT-101)); and 2) fine-tuning and inference by “tailored” models with less than 15 billion parameters. NVIDIA hardware is not the best fit for these types of workloads and deployment environments due to energy consumption, cooling, and cost. This is a key opportunity for competitors, as it will highlight their differentiated value proposition.

Overall, even with a vertically integrated hardware-software platform, it does not look like NVIDIA’s market leadership is in too much danger, but ABI Research expects that it will improve the market outlook for competing hyperscalers and chipset vendors.