Update your settings

Modular, a startup creating a platform for developing and optimizing AI systems, has raised $100 million in a funding round led by General Catalyst with participation from GV (Google Ventures), SV Angel, Greylock and Factory.

Bringing Modular’s total raised to $130 million, the proceeds will be put toward product expansion, hardware support and the expansion of Modular’s programming language, Mojo, CEO Chris Lattner says.

“Because we operate in a deeply technical space that requires highly specialized expertise, we intend to use this funding to support the growth of our team,” Lattner said in an email interview with TechCrunch. “This funding will not be primarily spent on AI compute, but rather improving our core products and scaling to meet our incredible customer demand.”

Lattner, an ex-Googler, cofounded Palo Alto-based Modular in 2022 with Tim Davis, a former Google colleague in the tech giant’s Google Brain research division. Both Lattner and Davis felt that AI was being held back by an overly complicated and fragmented technical infrastructure, and founded Modular with a focus on removing the complexity of building and maintaining AI systems at large scale.Modular provides an engine that tries to improve the inferencing performance of AI models on CPUs — and beginning later this year, GPUs — while delivering on cost savings. Compatible with existing cloud environments, machine learning frameworks like Google’s TensorFlow and Meta’s PyTorch and even other AI accelerator engines, Modular’s engine, currently in closed preview, lets developers import trained models and run them up to 7.5 times faster versus on their native frameworks, Lattner claims.

Modular’s other flagship product, Mojo, is a programming language that aims to combine the usability of Python with features like caching, adaptive compilation techniques and metaprogramming. Currently available in preview to “hundreds” of early adopters, Modular plans to release Mojo in general availability early next month.

“Our developer platform enables our customers, and the world’s developers, to defragment their AI technology stacks — pushing more innovations into production faster and realizing more value from their investment in AI,” Lattner said. “We’re attacking the complexity that slows AI development today by solving the fragmentation issues that plague the AI stack, starting with where AI software meets AI hardware.”Ambitious much? Perhaps. But none of what roughly-70-employee Modular is proposing is out of the realm of possibility.

Deci, backed by Intel, is among the startups offering tech to make trained AI models more efficient — and performant. Another in that category is OctoML, which automatically optimizes, benchmarks and packages models for an array of different hardware.

In any case, to Lattner’s point, AI demand is fast approaching the limits of sustainability — making any tech to cut down on its compute requirements hugely desirable. The generative AI models in vogue today are 10 to 100 times bigger than older AI models, as a recent piece in The Wall Street Journal points out, and much of the public cloud infrastructure wasn’t built for running these systems — at least not at this scale.

It’s already had an impact. Microsoft is facing a shortage of the server hardware needed to run AI so severe that it might lead to service disruptions, the company warned in an earnings report. Meanwhile, the sky-high appetite for AI inferencing hardware — mainly GPUs — has driven GPU provider Nvidia’s market cap to $1 trillion. But Nvidia’s become a victim of its own success; the company’s best-performing AI chips are reportedly sold out until 2024.For these reasons and others, more than half of AI decision makers in top companies report facing barriers to deploying the latest AI tools, according to a 2023 poll from S&P Global.

“The compute power needed for today’s AI programs is massive and unsustainable under the current model,” Lattner said. “We’re already seeing instances where there is not enough compute capacity to meet demand. Costs are skyrocketing and only the big, powerful tech companies have the resources to build these types of solutions. Modular solves this problem, and will allow for AI products and services to be powered in a way that is far more affordable, sustainable and accessible for any enterprise.”