SaaS22 Nov 20262 min read

Inference now takes two-thirds of AI workloads as spending moves to smaller sites

AI workloads have flipped to two-thirds inference, driving chip and server buyers away from giant training clusters toward smaller, modular facilities.

Listen to this storyAbout 3 min, read by your device

Artificial intelligence infrastructure spending is pivoting away from massive model training clusters toward smaller facilities that deploy quickly. InferenceInferenceThe process of running live data through a trained artificial intelligence model to make predictions or generate outputs. Open in glossary, the process of running trained AI models to generate answers, now accounts for about two-thirds of AI workloads, according to Positron AI Inc. In 2023, training made up about 85% of workloads while inference represented just 15%.

Commodity memory reduces power and supply limits

As inference dominates computing demand, hardware builders are shifting strategies. Positron, which raised $875 million at a $5 billion valuationValuationWhat a company is judged to be worth. Open in glossary in September, builds inference systems that use standard commodity memory instead of high-bandwidth memory. High-bandwidth memory remains scarce and expensive across the semiconductor industry.

By using standard memory, the hardware consumes less power. A single Positron server uses around 4,500 watts, and an entire rack of its upcoming Titan servers operates within 40 kilowatts. This power level allows operators to run systems in standard air-cooled environments without specialized cooling setups.

Modular data centers cut deployment timelines

Lower power requirements are driving demand toward modular data centers that need between 0.5 and 2.5 megawatts. These sites can secure power and avoid heavy regulatory delays more easily than massive cloud campuses.

Hardware buyers are using these smaller footprints to set up operations and begin generating tokens within 90 to 120 days from securing land and electricity. Positron’s first-generation Atlas server used a programmable chip, and Oracle Corp. is bringing roughly 340 of those units live in its cloud network this quarter. Positron intends to tape outTape outThe final stage of the chip design cycle before sending blueprints to a manufacturing foundry. Open in glossary its own custom silicon early next year for the Titan line, which has already received commitments from an unnamed neocloudNeocloudA cloud provider focused specifically on specialised computing hardware for AI tasks. Open in glossary operator.

How does this story land?

One tap per reader.

Quick poll

Where should AI infrastructure spending focus over the next two years?

One tap. No sign-up.

Ask this story

Still have a question?

Answers use only what is in this story. They are written by AI and can be wrong.

Sources

  1. SiliconANGLE, Inference now drives two-thirds of AI workloads, changing where chips get deployed
Up next in SaaSStuttgart supercomputer center offers AI power to small firms with portable workloads2 min read

Keep reading

All SaaS stories