AIHS 23E6 Podcast: Vast.ai, Intel CPU Max, Perceive, Groq, Lightelligence, NeuReality

The paragraph introduces Vast AI, a company that specializes in AI chips. There's a debate on the pronunciation of the company's name, with suggestions that it might be pronounced as 'Vast AI' or 'Vasti'. The company was founded by two former AMD fellows and has been operating in stealth mode. While not much is known about their architecture, it's clear they're focused on data center inference acceleration and have video decoding capabilities on-chip. The discussion also touches on the broader trend of companies incorporating 'AI' in their names, often capitalizing the 'A' and 'I' or using a dot, and the challenges faced by smaller accelerator companies, especially those in China with limited visibility, in finding their niche in the market.

This section discusses Intel's Sapphire Rapids, a CPU now named 'Max CPU' due to its integration with HBM memory. It represents a significant step forward as the first consumer CPU with HBM. The conversation delves into the history of Intel's Project Larabe, which aimed to create a GPU-like x86 processor with big Vector units but ultimately failed to produce a gaming GPU. However, it did contribute to compute-focused high-end CPUs like the KNL series. Sapphire Rapids with HBM is expected to be particularly beneficial for AI workloads in data centers, offering high memory bandwidth and supporting various memory tiers, including DRAM and Optane. The potential for these CPUs in HPC and AI spaces is highlighted, with the possibility of them being used in systems with multiple sockets.

The conversation shifts to Perceive, a startup that has developed a chip named Ergo. Perceive is known for its impressive performance figures at low power consumption, achieved by reinventing neural network mathematics. The approach aims to perform operations more efficiently by reducing the number of calculations needed. While the specifics of this method are not publicly known, it's suggested that it could involve rearranging calculations to optimize memory usage and power efficiency. The potential applications of this technology in consumer electronics are also discussed.

The discussion in this paragraph centers on Groq, a company led by one of the original TPU designers, John Ross. Groq's architecture is based on a data flow model where the duration of each operation is deterministic, allowing for a fixed latency that is known at compile time. This is in contrast to other AI hardware where latency can vary based on data input. The technology is particularly suited for real-time applications that require consistent latency, such as recommendation engines. The conversation also includes a personal account of a visit to Groq's offices to better understand their technology.

Lightelligence, another company spun out of MIT, is the focus of this section. They are working on optical computing, similar to Lightmatter, but using a different approach to modulate light. Instead of using MEMS for physical waveguide manipulation, Lightelligence injects electrons to change the refractive index, which alters the phase of light. This method is compared to Lightmatter's electromechanical approach. The potential challenges and applications of this technology are discussed, including its use in solving complex computational problems like the Ising problem.

The final paragraph discusses New Reality, a company that has partnered with IBM to develop AI acceleration IP. Unlike traditional AI accelerators that focus on neural network computations, New Reality's approach is to accelerate the infrastructure around AI solutions. Their card plugs into a PCI slot and helps manage various aspects of AI infrastructure, such as job scheduling and database management. This unique approach is aimed at improving the efficiency and throughput of AI systems by optimizing the non-computational aspects of AI operations.