When your team should consider GPU over CPU

GPU (Graphics Processing Unit) isn’t something new. It started a long time ago, after the advent of graphics accelerators like the Voodoo. The first use of cards with a GPU installed was gaming.
In mid 2010, the IT world began to consider GPUs for a new use case: mining cryptocurrencies like Bitcoin.

Recent years show even more powerful use cases for GPU-powered cards – Artificial Intelligence (AI).

In this article, we will explore why GPUs are used in AI workloads, their pros and cons, and the considerations when selecting the core infrastructure for our applications.

Comparison between CPU and GPU

Before we answer the question “what to choose,” we need a high-level understanding of the differences between GPUs and CPUs.

CPU – the workhorse

CPU stands for Central Processing Unit. It is the core of every computer. Today, it is built with multiple cores, each of which can perform various sequential tasks, including decision-making logic. Versatility is a primary goal in the design of CPUs.

CPUs are the heart of our systems. Realizing all tasks from running operating systems and applications, handling users and their requests, managing hardware and software, and so on.

GPU – one goal, many cores

GPU (Graphics Processing Unit) was created to simplify and streamline operations to achieve a single goal. Instead of a complex approach, its construction contains simpler cores but more of them. They are designed to perform the same operation across large amounts of data simultaneously. GPUs sacrifice single-core performance for sheer parallel throughput. This is the reason why GPUs are so good at rendering graphics. And as it turned out, there is complex math behind machine learning and Artificial intelligence.

TPU – more specialization

We also need to understand that CPU and GPU are not the only players here. TPU, which means Tensor Processing Unit, takes this specialization even further. This custom chip, created by Google, is focused on tensor operations, which are required for deep learning. General operations are no longer important; here we have a chip designed solely for efficiency and flexibility in machine learning and inference.

The difference between machine learning and inference

We have seen these terms a couple of times so far. But what do they really mean? Let’s debunk the true meaning behind them to better understand the types of AI workloads.

Machine learning

Machine learning (or ML for short) is software that learns patterns from data, rather than being programmed with explicit rules. Instead of writing “do this, and then do that”, we use ML to learn patterns through millions of similar activities. For example, how to write an email.
It shines especially when the problems are too complex to write explicit rules.

Deep learning

Deep learning (DL) is a more sophisticated and more powerful form of machine learning. It can handle far more complex tasks and is more accurate in its actions. Understanding languages, recognizing patterns in images, text generation – everything that consumers see as “AI” today needs deep learning to create.

Inference

In simple terms, it is the way we, users, interact with trained models. Inference belongs to its own phase. The first phase is training, where Machine Learning and Deep Learning belong, and Inference, where we put that learning to work.

Do I need a GPU?

We now know the hardware and design differences among CPU, GPU, and TPU, and we are also familiar with the main focus of Artificial Intelligence processes. It is time to discuss when to actually choose the specific architecture.

The reason to use a GPU

The truth is that in most cases, CPU is enough for all our workloads. GPU should be considered only when one or more aspects fall into one of these categories:

• Training new models. Teaching new models from scratch on large data. It is one of the most parallel jobs for GPU use cases.
• Fine-tune the existing model. Lighter process than training, however, mathematically similar to training.
• Inference. For large-scale systems and large models, a GPU is required. Small models can run on a CPU, but performance is better with a GPU.
• Heave parallel workloads. GPUs are great for more than just AI. We said this at the beginning: the GPU is here to help with graphics, and it does a great job in this area. Simulations, rendering, media processing, gaming – these areas are natural environments for GPUs.

GPU costs much more than a CPU

The truth is that GPU cards are much more expensive, even several times more expensive than CPUs. Why is that?

First, when we talk about CPU, we talk mainly about the chip itself; often, we dismiss that there is other hardware. In the GPU case, we refer to the entire card. Which is, in some cases, configured on the same hardware as a CPU.

Second, GPU chips are bigger and harder to make. A GPU consists of billions of transistors and has many more cores. With this increase in elements, the defect error rate per wafer is much higher than in CPUs.

Third, the memory used in GPU cards is more sophisticated than the memory in standard units. HBM memory used in GPUs is much more expensive than standard RAM.

These three are the reasons why GPUs are more expensive. But this is only the technical angle. There is also an economic angle.

First – limited competition. Today, one vendor – NVIDIA – dominates this sector.

And second – the demand. The demand for GPU cards is massive. It started several years ago, in the cryptocurrency mining era. When AI entered the game, demand rose even more.

We need to be aware of these reasons. As we can’t really make a huge impact on proxies, we have to find a way to avoid paying for a GPU when we don’t really need it.

Making the decision

When we justify the decision to buy this specialized hardware (be cause GPU is specialized), we should follow this simple framework:

• Measure. First, we need to know what kind of workload we want to run and how to measure effects. Is the specialized GPU required for the specific task? Do we experience bottlenecks? What kind? Performance, execution time, latency, scalability?
• Start where the problem is the biggest. We don’t need to redesign the stack. Your team probably already uses proper software libraries and approaches.
• Pilot the change. We already know, GPUs are expensive. Don’t burn thousands of Euros without validating the approach.

Decision matrix – when to choose CPU, GPU, or TPU

The table below should help us to make better decisions. We can use it as a “quick reference guide” for selecting the proper hardware for our needs.

Area to consider	CPU	GPU	TPU
Workloads	Varied, general, sequential work	Parallel math, rendering, graphics, Machine Learning, Inference	Tensor math for deep learning
Chip design	Powerful multipurpose cores	A huge number of simple cores	Custom, AI-specific cores
Best workload types	General use, web apps, API, databases, business logic, orchestration	Model training, fine tuning, inference, rendering, simulations	Large-scale deep learning, training and inference
Should be considered when	General workloads	AI training, inference, rendering	Deep learning, large scale operations
Should be avoided when	Parallel execution is a bottleneck	Spiky workload, general use cases, suited for CPU	No specific operations at scale
Costs / availability	Lowest costs, high availability on the market	Several times higher price than CPU. Limited availability, expected long time for delivery, even in cloud vendors	Comparable to GPU, availability limited and often dedicated to specific vendor (for example AWS or GCP)

This framework provides a solid foundation for making the right decision.

When to consider GPU for AI workloads?

Not all AI models need a GPU to run. SLM (Small Language Models), hallucination check frameworks, and so on, can be run on a CPU with good enough performance.

In case of AI workloads, like training, deep learning, fine-tuning, or even inference, different aspects must be considered as well. Mainly, the memory management and size.

We have to remember that these processes require a large amount of memory. The math for the needed memory size is more complicated than just how many parameters the model has. For example, precision is important to consider when planning memory. The model with 12 billion parameters will require around 20G of memory for standard precision (FP16), or only around 10G for heavily quantized precision (INT4). With 30B parameters models, it will be up to 85G and 24G accordingly, for FP16 and INT4.

When we consider a GPU, in most cases it means this memory size needs to be available on a GPU card. This is the most common industry approach; we do not talk here about the Apple approach implemented in MacBooks, where the memory is shared between CPU and GPU.

Use cases ready for GPU

Let’s quickly place a few scenarios when we should consider a GPU for our workload.

• Training models from scratch. This (together with deep learning) is the heaviest and most parallel work for hardware. In theory, the CPU can do that too, but would you like to wait a few weeks instead of a few hours?
• Fine-tuning. Less heavy process, but the arguments for using GPU here are the same as for GPU.
• Inference. Inference requires dedicated memory to run models, depending on the model’s design. Especially for scale and performance, GPU has only the advantages.
• Large models hosting. Same case as for inference. LLMs require a lot of memory and use parallel processing.
• Iterating fast and fast Go To Market goals for workloads. Trying new models, new architectures, new schemas for models, render the content faster – GPU is a simple choice here.

Conclusion

The question about utilizing a GPU isn’t about hardware, really. It is about the process, discipline, utilization, and workload.

CPUs still run most of the software, and this will not change. GPUs and TPUs are specialized, designed for specific workloads. In the age of AI, machine learning, and inference are the strongest incentives to choose them.

But we must also consider utilization. If the workload we plan will utilize 5% of the monthly plan and we will pay thousands of Euros for mostly idle hardware, we need to consider different options. Options like shared, on-demand offers, hourly used hardware, or even serverless, pay-as-you-go models.

If we can ask ourselves two questions about the reason to select a GPU, these questions should be:

• What GPU will buy us?
• Can we keep it busy?

When we have answers to these questions, we will know what direction to follow.

Do you want to know how easy it is to run your GPU virtual machine at UpCloud? Check this tutorial. If you are interested in our offerings and when to choose which GPU type to use, you can learn it in this article.

Do you want to know more? Our team is ready to help and answer all your questions!