GPU (Graphics Processing Unit) isn\u2019t something new. It started a long time ago, after the advent of graphics accelerators like the Voodoo. The first use of cards with a GPU installed was gaming.
In mid 2010, the IT world began to consider GPUs for a new use case: mining cryptocurrencies like Bitcoin.<\/p>\n\n\n\n
Recent years show even more powerful use cases for GPU-powered cards – Artificial Intelligence (AI).<\/p>\n\n\n\n
In this article, we will explore why GPUs are used in AI workloads, their pros and cons, and the considerations when selecting the core infrastructure for our applications.<\/p>\n\n\n\n
Before we answer the question \u201cwhat to choose,\u201d we need a high-level understanding of the differences between GPUs and CPUs.<\/p>\n\n\n\n
CPU stands for Central Processing Unit. It is the core of every computer. Today, it is built with multiple cores, each of which can perform various sequential tasks, including decision-making logic. Versatility is a primary goal in the design of CPUs.<\/p>\n\n\n\n
CPUs are the heart of our systems. Realizing all tasks from running operating systems and applications, handling users and their requests, managing hardware and software, and so on.<\/p>\n\n\n\n
GPU (Graphics Processing Unit) was created to simplify and streamline operations to achieve a single goal. Instead of a complex approach, its construction contains simpler cores but more of them. They are designed to perform the same operation across large amounts of data simultaneously. GPUs sacrifice single-core performance for sheer parallel throughput. This is the reason why GPUs are so good at rendering graphics. And as it turned out, there is complex math behind machine learning and Artificial intelligence.<\/p>\n\n\n\n
We also need to understand that CPU and GPU are not the only players here. TPU, which means Tensor Processing Unit, takes this specialization even further. This custom chip, created by Google, is focused on tensor operations, which are required for deep learning. General operations are no longer important; here we have a chip designed solely for efficiency and flexibility in machine learning and inference.<\/p>\n\n\n\n
We have seen these terms a couple of times so far. But what do they really mean? Let\u2019s debunk the true meaning behind them to better understand the types of AI workloads.<\/p>\n\n\n\n
Machine learning (or ML for short) is software that learns patterns from data, rather than being programmed with explicit rules. Instead of writing \u201cdo this, and then do that\u201d, we use ML to learn patterns through millions of similar activities. For example, how to write an email.
It shines especially when the problems are too complex to write explicit rules.<\/p>\n\n\n\n
Deep learning (DL) is a more sophisticated and more powerful form of machine learning. It can handle far more complex tasks and is more accurate in its actions. Understanding languages, recognizing patterns in images, text generation – everything that consumers see as \u201cAI\u201d today needs deep learning to create.<\/p>\n\n\n\n
In simple terms, it is the way we, users, interact with trained models. Inference belongs to its own phase. The first phase is training, where Machine Learning and Deep Learning belong, and Inference, where we put that learning to work.<\/p>\n\n\n\n
We now know the hardware and design differences among CPU, GPU, and TPU, and we are also familiar with the main focus of Artificial Intelligence processes. It is time to discuss when to actually choose the specific architecture.<\/p>\n\n\n\n
The truth is that in most cases, CPU is enough for all our workloads. GPU should be considered only when one or more aspects fall into one of these categories:<\/p>\n\n\n\n
The truth is that GPU cards are much more expensive, even several times more expensive than CPUs. Why is that?<\/p>\n\n\n\n
First, when we talk about CPU, we talk mainly about the chip itself; often, we dismiss that there is other hardware. In the GPU case, we refer to the entire card. Which is, in some cases, configured on the same hardware as a CPU.<\/p>\n\n\n\n
Second, GPU chips are bigger and harder to make. A GPU consists of billions of transistors and has many more cores. With this increase in elements, the defect error rate per wafer is much higher than in CPUs.<\/p>\n\n\n\n
Third, the memory used in GPU cards is more sophisticated than the memory in standard units. HBM memory used in GPUs is much more expensive than standard RAM.<\/p>\n\n\n\n
These three are the reasons why GPUs are more expensive. But this is only the technical angle. There is also an economic angle.<\/p>\n\n\n\n
First – limited competition. Today, one vendor – NVIDIA – dominates this sector.<\/p>\n\n\n\n
And second – the demand. The demand for GPU cards is massive. It started several years ago, in the cryptocurrency mining era. When AI entered the game, demand rose even more.<\/p>\n\n\n\n
We need to be aware of these reasons. As we can\u2019t really make a huge impact on proxies, we have to find a way to avoid paying for a GPU when we don\u2019t really need it.<\/p>\n\n\n\n
When we justify the decision to buy this specialized hardware (be cause GPU is specialized), we should follow this simple framework:<\/p>\n\n\n\n
The table below should help us to make better decisions. We can use it as a \u201cquick reference guide\u201d for selecting the proper hardware for our needs.<\/p>\n\n\n\n This framework provides a solid foundation for making the right decision.<\/p>\n\n\n\n Not all AI models need a GPU to run. SLM (Small Language Models), hallucination check frameworks, and so on, can be run on a CPU with good enough performance.<\/p>\n\n\n\n In case of AI workloads, like training, deep learning, fine-tuning, or even inference, different aspects must be considered as well. Mainly, the memory management and size.<\/p>\n\n\n\n We have to remember that these processes require a large amount of memory. The math for the needed memory size is more complicated than just how many parameters the model has. For example, precision is important to consider when planning memory. The model with 12 billion parameters will require around 20G of memory for standard precision (FP16), or only around 10G for heavily quantized precision (INT4). With 30B parameters models, it will be up to 85G and 24G accordingly, for FP16 and INT4.<\/p>\n\n\n\n When we consider a GPU, in most cases it means this memory size needs to be available on a GPU card. This is the most common industry approach; we do not talk here about the Apple approach implemented in MacBooks, where the memory is shared between CPU and GPU.<\/p>\n\n\n\n Let\u2019s quickly place a few scenarios when we should consider a GPU for our workload.<\/p>\n\n\n\n The question about utilizing a GPU isn\u2019t about hardware, really. It is about the process, discipline, utilization, and workload.<\/p>\n\n\n\n CPUs still run most of the software, and this will not change. GPUs and TPUs are specialized, designed for specific workloads. In the age of AI, machine learning, and inference are the strongest incentives to choose them.<\/p>\n\n\n\n But we must also consider utilization. If the workload we plan will utilize 5% of the monthly plan and we will pay thousands of Euros for mostly idle hardware, we need to consider different options. Options like shared, on-demand offers, hourly used hardware, or even serverless, pay-as-you-go models.<\/p>\n\n\n\n If we can ask ourselves two questions about the reason to select a GPU, these questions should be:<\/p>\n\n\n\n When we have answers to these questions, we will know what direction to follow.<\/p>\n\n\n\n Do you want to know how easy it is to run your GPU virtual machine at UpCloud? Check this tutorial. If you are interested in our offerings and when to choose which GPU type to use, you can learn it in this article.<\/p>\n\n\n\n Do you want to know more? Our team is ready to help and answer all your questions!<\/p>\n","protected":false},"excerpt":{"rendered":" GPU (Graphics Processing Unit) isn\u2019t something new. It started a long time ago, after the advent of graphics accelerators like the Voodoo. The first use […]<\/p>\n","protected":false},"author":100,"featured_media":84447,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_relevanssi_hide_post":"","_relevanssi_hide_content":"","_relevanssi_pin_for_all":"","_relevanssi_pin_keywords":"","_relevanssi_unpin_keywords":"","_relevanssi_related_keywords":"","_relevanssi_related_include_ids":"","_relevanssi_related_exclude_ids":"","_relevanssi_related_no_append":"","_relevanssi_related_not_related":"","_relevanssi_related_posts":"907,3430,445,898,3280,328","_relevanssi_noindex_reason":"Blocked by a filter function","footnotes":""},"categories":[22,25,76],"tags":[],"class_list":["post-8105","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cloud-infrastructure","category-comparisons","category-gpus"],"acf":[],"_links":{"self":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/posts\/8105","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/users\/100"}],"replies":[{"embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/comments?post=8105"}],"version-history":[{"count":0,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/posts\/8105\/revisions"}],"wp:attachment":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/media?parent=8105"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/categories?post=8105"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tags?post=8105"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Area to consider<\/th> CPU<\/th> GPU<\/th> TPU<\/th><\/tr><\/thead> Workloads<\/strong><\/td> Varied, general, sequential work<\/td> Parallel math, rendering, graphics, Machine Learning, Inference<\/td> Tensor math for deep learning<\/td><\/tr> Chip design<\/strong><\/td> Powerful multipurpose cores<\/td> A huge number of simple cores<\/td> Custom, AI-specific cores<\/td><\/tr> Best workload types<\/strong><\/td> General use, web apps, API, databases, business logic, orchestration<\/td> Model training, fine tuning, inference, rendering, simulations<\/td> Large-scale deep learning, training and inference<\/td><\/tr> Should be considered when<\/strong><\/td> General workloads<\/td> AI training, inference, rendering<\/td> Deep learning, large scale operations<\/td><\/tr> Should be avoided when<\/strong><\/td> Parallel execution is a bottleneck<\/td> Spiky workload, general use cases, suited for CPU<\/td> No specific operations at scale<\/td><\/tr> Costs \/ availability<\/strong><\/td> Lowest costs, high availability on the market<\/td> Several times higher price than CPU. Limited availability, expected long time for delivery, even in cloud vendors<\/td> Comparable to GPU, availability limited and often dedicated to specific vendor (for example AWS or GCP)<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n When to consider GPU for AI workloads?<\/h2>\n\n\n\n
Use cases ready for GPU<\/h3>\n\n\n\n
\n
Conclusion<\/h2>\n\n\n\n
\n