{"id":8293,"date":"2026-08-06T23:06:59","date_gmt":"2026-08-06T22:06:59","guid":{"rendered":"https:\/\/upcloud.com\/global\/?post_type=tutorial&#038;p=8293"},"modified":"2026-08-06T23:06:59","modified_gmt":"2026-08-06T22:06:59","slug":"start-running-llms-locally-with-ollama","status":"publish","type":"tutorial","link":"https:\/\/upcloud.com\/global\/resources\/tutorials\/start-running-llms-locally-with-ollama\/","title":{"rendered":"Start running LLMs locally with Ollama"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Large Language Models (LLMs) are no longer limited to cloud and SaaS services (such as Anthropic or OpenAI). With tools like Ollama, we can run quite powerful models directly on our machines, even laptops, within minutes.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For experiments, internal chatbots, or processing sensitive data, getting started with models hosted by us is surprisingly easy.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In this tutorial, we will install Ollama, download one of the available models, and we will chat with it. All from one single terminal.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Why Run LLMs Locally?&nbsp;<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Before we dig deeper into the terminal, we should understand why we might want to run AI inference locally.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Privacy:<\/strong> In many scenarios, such as processing sensitive data, using SaaS products is simply a no-go. Regulations might also require us to ensure our data does not leave our infrastructure. Running the model locally ensures no prompts, data, or responses ever leave your infrastructure.<\/li>\n\n\n\n<li><strong>Low latency:<\/strong> With proper infrastructure, a local setup can be much faster than any SaaS platform. This is especially true during peak traffic hours. Of course, this requires a relatively powerful setup to provide benefit.<\/li>\n\n\n\n<li><strong>No usage fees:<\/strong> A very important element from the perspective of cost management. The infrastructure and, in some cases, the model will incur costs, but inference tokens won&#8217;t be charged, ensuring predictable running costs.<\/li>\n\n\n\n<li><strong>Full control:<\/strong> We control who has access, which models are used, how those models are deployed, and the guardrails within which the models must run.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">What is Ollama?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Ollama is an open-source platform for managing and running LLMs. It allows us to download, manage, and use different models from the model library, and allows us to run inferences from the console or through an API.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The platform allows us to run not only LLMs but also agents or systems like OpenClaw. It is owned and maintained by <a href=\"https:\/\/ollama.com\/\" target=\"_blank\" rel=\"noopener\">Ollama Inc.<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Install Ollama<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Okay! Now that we know why a local LLM workload makes sense, it\u2019s time to start our journey and install Ollama.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Prerequisites<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">To follow this tutorial, we need a machine to run the model. Given that models are designed to fit relatively small infrastructure, we don\u2019t need to use the most modern servers with the most powerful GPU cards.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The steps in this tutorial can be completed on most Linux systems. We will use the <a href=\"https:\/\/upcloud.com\/global\/products\/gpu-servers\/\">NVIDIA L4 instance offered by UpCloud<\/a> as an example.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To run the install script successfully, you will need to have the following packages installed.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">curl awk grep sed tee xargs<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Installation of Ollama on Ubuntu<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The installation is very simple. Ollama provides a simple, platform-agnostic installation script that supports most Debian- and RPM-based distributions.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">curl -fsSL https:\/\/ollama.com\/install.sh | sh<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">After a short time, we should see a similar message to the one in the image below:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/image-318-1024x307.png\" alt=\"-\" class=\"wp-image-85092\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">And\u2026 that\u2019s it!<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Let\u2019s verify the installation:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">ollama --version<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If Ollama returns the version number, we are ready to run our workloads!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Download the first LLM for Ollama<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Time to download the first model. We will pull the llama3 model for this tutorial. All available models are listed in the <a href=\"https:\/\/ollama.com\/search\" target=\"_blank\" rel=\"noopener\">library<\/a> page on the Ollama website.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To pull the model, we use this command:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">ollama pull llama3<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">It takes some time, depending on the connection speed and size of the model. This particular model is almost 5GB, so it takes a few moments to complete.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/image-319-1024x393.png\" alt=\"-\" class=\"wp-image-85095\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Once your model pull is complete, let&#8217;s learn more about it and validate the download.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">ollama list<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This command shows all the pulled models stored on our local volume.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/image-317.png\" alt=\"-\" class=\"wp-image-85091\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">We see here the model name and version, the image ID, the package size, and the last modification time.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To learn more about the model, you can use the following command:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">ollama show llama3<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This command will return basic information about the model. Its parameters are size, quantization (precision), and context window length.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/image-319-1024x265.png\" alt=\"-\" class=\"wp-image-85094\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Running the LLM model<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">It is time now to run our first prompt! This is equally easy as every step so far.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">First, run the model by using the command below:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">ollama run llama3<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">After a few moments, we will receive a prompt from Ollama, and we are ready to ask anything!<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/image-319-1024x393.png\" alt=\"-\" class=\"wp-image-85096\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">When we are finished with our chat, we can leave the Ollama by using the <strong>\/bye<\/strong> exit command.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">However, if you want to leave the model running to complete a longer prompt, we can return to our system prompt by using the <strong>ctrl<\/strong> + <strong>d<\/strong> combination to detach from the process.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>ctrl<\/strong> + <strong>d<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This will leave the model available for future use.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Monitoring the usage<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Ollama provides commands to monitor system performance by running:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">ollama ps<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Since we left the model running, we can see it is still loaded into memory.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/image-318-1024x112.png\" alt=\"-\" class=\"wp-image-85093\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">What\u2019s next?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We can download more models, like the European Mistral, using<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">ollama pull mistral<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Or, start with agentic workloads, using, for example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">ollama launch openclaw<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">However, for agent execution, we might need some dependencies installed, like Node.js and OpenCLAW, in this example.<\/p>\n\n\n\n<h1 class=\"wp-block-heading\">Conclusion<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Running LLM locally was never easier. Ollama, or Docker Model Runner (and many other applications), allows us to build an entire application powered by AI inference in our local environment, using only a handful of commands. This makes it an excellent choice for developers exploring generative AI.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Whether you are building prototypes, experimenting with AI, or developing privacy-sensitive applications, running Ollama with local LLMs gives you a fast, reliable, simple, and fully controlled environment to do so.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you would like to try, use <a href=\"https:\/\/upcloud.com\/global\/products\/gpu-servers\/\">UpCloud\u2019s GPU Servers<\/a> powered by NVIDIA GPU chips. If you have any <a href=\"https:\/\/upcloud.com\/global\/contact\/\">questions<\/a>, our team is always happy to help!<\/p>\n","protected":false},"author":100,"featured_media":0,"comment_status":"open","ping_status":"closed","template":"","community-category":[244,253],"class_list":["post-8293","tutorial","type-tutorial","status-publish","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/8293","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial"}],"about":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/types\/tutorial"}],"author":[{"embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/users\/100"}],"replies":[{"embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/comments?post=8293"}],"version-history":[{"count":5,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/8293\/revisions"}],"predecessor-version":[{"id":8317,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/8293\/revisions\/8317"}],"wp:attachment":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/media?parent=8293"}],"wp:term":[{"taxonomy":"community-category","embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/community-category?post=8293"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}