{"id":8026,"date":"2026-07-10T16:03:52","date_gmt":"2026-07-10T15:03:52","guid":{"rendered":"https:\/\/upcloud.com\/global\/?post_type=tutorial&#038;p=8026"},"modified":"2026-07-10T16:03:52","modified_gmt":"2026-07-10T15:03:52","slug":"run-llm-locally-using-upclouds-gpu-server-docker-model-runner","status":"publish","type":"tutorial","link":"https:\/\/upcloud.com\/global\/resources\/tutorials\/run-llm-locally-using-upclouds-gpu-server-docker-model-runner\/","title":{"rendered":"Run LLM locally using UpCloud\u2019s GPU server and Docker Model Runner"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Using Large Language Models (LLM) no longer requires you to be forced to use SaaS products like Claude, Cursor, or OpenAI. With decent GPU offerings and the ability to run models on these machines, we gain flexibility, greater security, and the ability to build cheaper solutions.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In this tutorial, you\u2019ll learn how to run LLMs on an UpCloud GPU server, using an open source product from Docker &#8211; Docker Model Runner.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Prerequisites<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We need to have a few things done before we can start to follow this tutorial.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>GPU instance is up and running. UpCloud offers four GPU models, all from NVIDIA. From the smallest one &#8211; NVIDIA L4, through the inference-optimized NVIDIA L40S and the dedicated NVIDIA H100 for heavy inference or machine learning, up to the powerful NVIDIA B200, dedicated for heavy workloads.<\/li>\n\n\n\n<li>To connect to the machine, we need to have SSH keys provisioned and configured on this machine.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Prepare the machine<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">GPU Servers from UpCloud already have Docker installed. Also, it is worth noting that UpCloud\u2019s team provides rebaked images with all the NVIDIA tools needed to use this machine to its full potential.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/1-nvidia-cuda-ubuntu.png\" alt=\"-\" class=\"wp-image-84492\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">For this tutorial, we are using the smallest NVIDIA-powered machine, using NVIDIA L4.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/2-gpu-server-ready-1024x102.png\" alt=\"-\" class=\"wp-image-84494\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Log in to the machine<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We have SSH keys added, so we can log in using the command below.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">ssh root@&lt;server_ip&gt;<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">After logging in, we might want to check whether the GPU is actually available and usable. To do so, NVIDIA provides a set of tools that we will use with nvidia-smi.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">nvidia-smi<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">We should see something similar to the picture below:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/3-nvidia-smi-status-1024x564.png\" alt=\"-\" class=\"wp-image-84497\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">This screenshot confirms that the NVIDIA L4 chip is available, how much power it consumes, and what memory is available.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Checking Docker<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Let\u2019s run the <code>docker<\/code> command to check if Docker is installed:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">docker --version<\/code><\/pre>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/4-docker-version.png\" alt=\"-\" class=\"wp-image-84499\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">It should be the newest version of the package. It might be different as newer versions are released.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Install Docker Model Runner (DMR)<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">There is one component missing, though. Docker Model Runner package.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The reason is simple: not everyone needs DMR, so those who do will install the package.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Docker Model Runner is available as a Docker plugin. It is installed as a package, but requires Docker to run.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Let\u2019s execute these commands:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">sudo apt update<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">sudo apt install docker-model-plugin<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">We don\u2019t need to add any additional registry. As mentioned, UpCloud\u2019s GPU machines already have a proper Docker registry installed.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The picture below shows the installation process.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/5-install-docker-model-plugin-1024x650.png\" alt=\"-\" class=\"wp-image-84502\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Let\u2019s finally check if DMR is installed and working:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">docker model version<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">docker model status<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If a version is returned, it means that the plugin is installed:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/6-docker-model-version.png\" alt=\"-\" class=\"wp-image-84503\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">When run for the first time, the command status needs to pull the image for the Docker model runner. This is a one-time execution. After that, we will see if DMR is enabled:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/7-docker-model-status-926x1024.png\" alt=\"-\" class=\"wp-image-84505\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">The most important information is \u201c<strong>Docker Model Runner is running<\/strong>\u201d.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>Note: Sometimes we can find information that we should use the command docker model install-runner \u2013gpu=cuda, but it is not needed. Using the way we followed, it will be enabled.<\/em><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Pulling the first model<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Now that we have DMR ready and running, it is time to pull our first LLM. Docker provides its Hub, which offers literally hundreds of models to pull. This <a href=\"https:\/\/hub.docker.com\/search?type=model\" target=\"_blank\" rel=\"noopener\">registry is available on the Hub<\/a> and allows us to search for models and view their information.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We will use the qwen3 model with 8b parameters.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Docker Hub gives us the specific command to use. We type in our CLI:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">docker model pull ai\/qwen3:8B-Q4_0<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The pull process for models is very similar to that for images.<br>After some time, depending on our connection speed and quality, the model will be stored locally.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/8-docker-model-pull-1024x192.png\" alt=\"-\" class=\"wp-image-84508\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Let\u2019s check what we have, running:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">docker model list<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This command gives us the list of pulled models, like in the image below:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/9-docker-model-list-1024x92.png\" alt=\"-\" class=\"wp-image-84510\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">What do we see here:<br>We see all models pulled so far and not removed. In our case, we have only one, qwen3. Let\u2019s understand each field in this list.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Model name<\/strong> &#8211; this is simple. It is the name of the model.<\/li>\n\n\n\n<li><strong>Parameters<\/strong> &#8211; this tells us how many parameters the model has. It describes the number of trainable weights in the neural network. To put it simply, it tells us how big the model&#8217;s brain is. Not how smart it is.<\/li>\n\n\n\n<li><strong>Quantization<\/strong> &#8211; describes precision. In simple terms, it means how much we compress the brain to fit the available memory. FP16 provides better reasoning but requires 16 bits per parameter. INT4 will reduce the amount of reasoning compared to FP16 but requires 4 bits per parameter.<\/li>\n\n\n\n<li><strong>Architecture<\/strong> &#8211; internal design of the neural network and processing information.<\/li>\n\n\n\n<li><strong>Model ID<\/strong> &#8211; like for image ID. It pinpoints the specific version of the model.<\/li>\n\n\n\n<li><strong>Created<\/strong> &#8211; when the model was prepared and stored in the hub.<\/li>\n\n\n\n<li><strong>Context<\/strong> &#8211; provides information about the maximum context window available for this model.<\/li>\n\n\n\n<li><strong>Size<\/strong> &#8211; How much space the package takes on the disk.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Use the model<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Finally! We know how to get the DMR and models, and we are ready to run our first inference!<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Let\u2019s run this command:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">docker model run ai\/qwen3:8B-Q4_0 \"Explain how Docker Model Runner Works\"<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">With L4, the model&#8217;s response starts almost immediately.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/10-docker-model-run-1024x397.png\" alt=\"-\" class=\"wp-image-84512\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">And now we are equipped with the local tool to run our own inference!<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you want to learn more about Docker model Runner, then\u2026 well\u2026 you know how to learn it locally!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Upcloud\u2019s GPU offering gives us an opportunity to run powerful hardware for our local inference applications. Docker Model Runner is a tool we can use to run inference locally in a very convenient way.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Read more about our GPU offering on our <a href=\"https:\/\/upcloud.com\/global\/products\/gpu-servers\/\">GPU landing page<\/a>. Contact our team if you wish to know more!<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">What will you build with this approach? Let us know!<\/p>\n","protected":false},"author":100,"featured_media":0,"comment_status":"open","ping_status":"closed","template":"","community-category":[244,241],"class_list":["post-8026","tutorial","type-tutorial","status-publish","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/8026","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial"}],"about":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/types\/tutorial"}],"author":[{"embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/users\/100"}],"replies":[{"embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/comments?post=8026"}],"version-history":[{"count":12,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/8026\/revisions"}],"predecessor-version":[{"id":8093,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/8026\/revisions\/8093"}],"wp:attachment":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/media?parent=8026"}],"wp:term":[{"taxonomy":"community-category","embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/community-category?post=8026"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}