Skip to content
0MESubscribe
Chips17 Nov 20262 min read

SlipStream cuts flash memory delay to boost AI chip capacity by 16x

Researchers from Nvidia, Stanford, and Georgia Tech unveil SlipStream, a system that lets AI processors use dense flash memory without severe latency penalties.

Listen to this storyAbout 3 min, read by your device
Picture made with AI

Running large language models requires vast amounts of memory to hold billions of parameters. Today, artificial intelligence chipsArtificial intelligence chipsSpecialized microchips designed to quickly process complex tasks required by modern machine learning systems. Open in glossary rely on high-bandwidth memory made of dynamic RAM dies. High-Bandwidth Flash replaces these dies with NAND flash memory, providing up to 16 times more capacity at similar read speeds. However, flash memory suffers from high latency. It takes nearly 20 microseconds to read data, which is 20 times slower than standard memory and can slow down model inferenceInferenceRunning a trained AI model to get an answer. Open in glossary by up to 12.8 times.

Pacing data retrieval to fix lag

To solve this delay, researchers from Georgia Tech, Nvidia Research, and Stanford University developed SlipStream. It is a combined hardware and software prefetching system. Traditional methods try to guess what data is needed next, but long flash delays require massive memory buffers to track incoming traffic. Earlier tensor-streaming designs required hundreds of megabytes of extra buffer space because they fetched data based on address order rather than compute demand.

SlipStream solves this by creating a producer-consumer setup. In software, it maps out the exact order that computing tasks request model weights. Dedicated GPUGPUA chip that does many small sums at once. First made for graphics, now the main chip for AI. Open in glossary worker threads, called staging warps, trigger non-blocking retrieval instructions ahead of time. Crucially, tracking entries are only cleared when a computing task actually touches the data for the first time. This ensures that data retrieval stays directly tied to the actual computing pace.

Smaller buffers and higher throughput

The system tracks 4-kilobyte memory pages instead of tiny cache lines. It only reserves internal buffer space when the requested data actually arrives from storage. This decouples the required buffer size from the total volume of memory traffic.

  • In tests across seven dense language models ranging from 16 to 800 gigabytes, SlipStream using a 40-megabyte buffer reached 92.4% of the speed of an ideal flash system that has zero extra delay.
  • A competing design called H3 achieved only 43.1% performance while needing an eight times larger 320-megabyte buffer.
  • SlipStream also works with sparse Mixture-of-ExpertsMixture-of-ExpertsAn artificial intelligence design that activates only a small subset of neural networks for each task to save computing power. Open in glossary models, delivering 91.2% of ideal performance with the same 40-megabyte buffer.

How does this story land?

One tap per reader.

Quick poll

Will NAND flash replace high-bandwidth DRAM as the primary memory in future AI accelerators?

One tap. No sign-up.

Ask this story

Still have a question?

Answers use only what is in this story. They are written by AI and can be wrong.

Sources

  1. Semiconductor Engineering, Reducing High-Bandwidth Flash Latency In GPU Systems (Georgia Tech, Nvidia, Stanford)
Up next in ChipsArm powers the fastest and greenest supercomputers in latest rankings2 min read

Keep reading

All Chips stories