AI Infrastructure Guide: Compute, Data, Cloud and Cost

AI infrastructure is everything needed to build and run AI: the computers (especially GPUs), the data pipelines that feed them, the storage and networks that connect them, and the place they live, whether that is the cloud, your own data centre or both. For most businesses, the hard part isn't buying hardware. It is getting clean data in and keeping the running costs under control.

This guide covers the basics in plain terms, then explains NVIDIA's new Rubin platform and what it means (and doesn't mean) for a typical company.

The building blocks of AI infrastructure

Think of it in four layers:

1. Compute: the chips that do the maths.
2. Data: where your data comes from, how it is cleaned and where it is stored.
3. Platform: the software that runs models, such as model-serving tools, vector databases and monitoring.
4. Location: cloud, on-premises (your own servers) or a mix.

Compute: CPUs and GPUs

A CPU is the general-purpose chip in every computer. It handles a few tasks at a time very well.

A GPU (graphics processing unit) has hundreds or thousands of small cores that do many simple calculations at once. NVIDIA explains that this parallel design is why GPUs suit AI, which is mostly huge amounts of repeated maths.

AI work splits into two jobs:

• Training: teaching a model from large amounts of data. This needs a lot of GPU power, often for days or weeks. Most businesses never do this from scratch.
• Inference: using a trained model to answer questions or make predictions. This is what most businesses actually run, every day.

For many companies, the real need is inference: running an existing model, perhaps fine-tuned on their own documents. That needs far less hardware than training.

GPU memory matters as much as speed. A model has to fit in memory to run quickly, so larger models need GPUs with more memory, or several GPUs working together.

Data pipelines: the part people underestimate

An AI system is only as good as the data it receives. A data pipeline is the set of steps that moves data from where it is created to where AI can use it.

A typical pipeline:

1. Collect data from systems like your ERP, CRM, website and documents.
2. Clean it: remove duplicates, fix formats, handle missing values.
3. Protect it: mask personal data and control who can access what.
4. Store it in a form the AI can use, such as a data warehouse or a vector database for document search.
5. Monitor it, so you know when a feed breaks or goes stale.

That last step is often skipped. A broken pipeline doesn't crash; it quietly feeds the AI old or wrong data. Our guide to observability vs monitoring covers data observability in more detail.

Cloud vs on-prem vs hybrid

Where should your AI run? There is no single right answer.

	Cloud	On-premises	Hybrid
Upfront cost	Low	High (buy hardware)	Medium
Speed to start	Days	Months	Varies
Scaling up	Easy	Slow, needs new hardware	Burst to cloud
Data control	Shared with provider	Full	Sensitive data stays local
Best for	Pilots, variable demand	Steady heavy use, strict data rules	Most growing companies

Cloud is the easiest start. You rent GPUs by the hour and stop paying when you stop using them. For Malaysian firms, AWS opened a Malaysia region in August 2024, and other providers have regions nearby.

On-premises means buying and running your own servers. It gives full control over data, which matters to banks, hospitals and government agencies. But you pay upfront and need staff, power and cooling.

Hybrid combines the two. A common pattern: keep sensitive data and day-to-day inference on site, and use the cloud for occasional heavy jobs.

Data rules can tip the choice. Under Malaysia's PDPA, as amended, transfers of personal data abroad have been allowed since 1 April 2025 under conditions such as the destination having substantially similar law or an equivalent level of protection. Check this with your legal adviser before sending personal data to an overseas cloud region.

Controlling cost

AI infrastructure bills can grow quickly. Practical ways to keep them in check:

• Start in the cloud for pilots. Don't buy hardware until you know your steady usage.
• Right-size models. A smaller model that does the job is cheaper to run than the biggest one available.
• Turn things off. Idle GPUs in the cloud still cost money.
• Watch usage per use case, so you know which AI features earn their keep.
• Compare over time. If a workload runs flat out every day, owning hardware can become cheaper than renting. If demand is spiky, renting usually wins.
• Count the hidden costs: data cleaning, staff time, security and support often outweigh the hardware.

NVIDIA's Rubin platform: what it is

Rubin is NVIDIA's next-generation AI data centre platform, following Blackwell. NVIDIA announced it on 5 January 2026 as a set of six chips designed to work together:

• Vera CPU
• Rubin GPU
• NVLink 6 Switch (connects GPUs together)
• ConnectX-9 SuperNIC (network card)
• BlueField-4 DPU (offloads networking, storage and security tasks)
• Spectrum-6 Ethernet Switch

NVIDIA's own claims, compared with Blackwell, are up to 10x lower cost per token for large mixture-of-experts model inference, and training such models with 4x fewer GPUs. These are vendor figures, not independent benchmarks.

When it ships

In January, NVIDIA said Rubin-based products would be available from partners in the second half of 2026, naming AWS, Google Cloud, Microsoft and OCI among the first cloud providers. On 31 May 2026, NVIDIA said the platform was ramping into full production, with production shipments set to begin "this fall" (autumn 2026). In its August 2026 quarterly filing, NVIDIA said Vera Rubin began production shipments in its third fiscal quarter, and its results announcement said racks were already running at partners including CoreWeave, Google Cloud, Microsoft Azure, OCI and Nebius. NVIDIA also said it faces some supply constraints, so as of October 2026 most companies should expect access through cloud providers first.

What it means for most businesses

Probably not much directly. Rubin systems are rack-scale machines aimed at cloud providers and very large AI builders. Most companies will meet Rubin, if at all, as a new instance type from a cloud provider.

The useful takeaway is the trend: the cost of running AI keeps falling. That is a reason to avoid locking into long hardware commitments too early, and to design systems that can move to newer, cheaper capacity.

The bottom line

Good AI infrastructure starts with a clear use case and clean data, not with the biggest GPUs. Start in the cloud, prove value, watch costs, and move to hybrid or on-prem when steady demand or data rules justify it. For the wider rollout plan, see our enterprise AI roadmap.

Quick answers

What is AI infrastructure?

It is the hardware, software, data pipelines and hosting needed to build and run AI systems. GPUs, storage, networks and the cloud or data centre all count.

Do I need my own GPUs to use AI?

Usually not. Most businesses start by renting GPUs in the cloud or using AI services through an API, and only buy hardware when usage is steady and high.

Is cloud or on-prem better for AI?

Cloud is better for starting fast and handling changing demand. On-prem suits steady heavy use and strict data control. Many companies end up with a hybrid.

When is NVIDIA Rubin available?

It is shipping now. NVIDIA said in August 2026 that Vera Rubin had begun production shipments, with racks running at partners such as Google Cloud, Microsoft Azure and OCI. For most businesses, access will come through cloud providers.