“Open weights” in the context of AI refers to large language models (LLMs) where the trained model parameters (the “weights”) are publicly released and accessible. This is distinct from “open source” in software, where the source code is open. While the training code for an LLM might be proprietary, open weights allow researchers, developers, and businesses to:<\/p>\n\n\n\n
- \n
- Innovate Freely:<\/strong> Build upon, fine-tune, and adapt models for specific use cases without vendor lock-in.<\/li>\n\n\n\n
- Promote Transparency:<\/strong> Enable inspection and understanding of how models behave, contributing to explainable AI and responsible development.<\/li>\n\n\n\n
- Foster Competition:<\/strong> Democratize access to powerful AI capabilities, preventing a few large corporations from monopolizing the field.<\/li>\n\n\n\n
- Ensure Data Privacy:<\/strong> Run models locally or on private cloud infrastructure without sending sensitive data to external APIs.<\/li>\n<\/ul>\n\n\n\n
Mistral AI, a prominent European AI company, has been a key player in advocating for and releasing models with open weights, such as the Mistral 7B and Mixtral models. This approach empowers a broader community to utilize and improve AI technologies.<\/p>\n\n\n\n
Why This Matters for Your UpCloud GPU Instance<\/strong><\/h2>\n\n\n\n
By combining an UpCloud GPU instance with open-weights models like Mistral run via Ollama, you gain:<\/p>\n\n\n\n
- \n
- Control and Privacy:<\/strong> You retain full control over your data and the inference process, aligning with data sovereignty requirements.<\/li>\n\n\n\n
- Cost-Efficiency:<\/strong> Running models on your own infrastructure can be more cost-effective for sustained use compared to API-based services.<\/li>\n\n\n\n
- Customization:<\/strong> The ability to fine-tune and adapt open-weights models to your specific needs is significantly enhanced.<\/li>\n\n\n\n
- Performance:<\/strong> Direct access to dedicated GPU resources on your instance provides optimal inference performance<\/li>\n<\/ul>\n\n\n\n
Step-by-Step Tutorial:<\/strong><\/h2>\n\n\n\n
1.<\/strong> Deploy an UpCloud GPU instance<\/strong><\/h4>\n\n\n\n
Log in to https:\/\/hub.upcloud.com<\/a> and select GPU servers from the left-side panel, then click \u201cDeploy server\u201d from the right side.<\/p>\n\n\n\n
![\"-\"](\"https:\/\/upcloud.com\/media\/gpu1-1024x215.png\")
<\/figure>\n\n\n\nSelect a GPU plan that fits your use case. You can choose up to 3 GPUs per Cloud Server, up to 20 cores and 256 GB of RAM.<\/p>\n\n\n\n
![\"-\"](\"https:\/\/upcloud.com\/media\/gpu2-1024x758.png\")
<\/figure>\n\n\n\nSelect Ubuntu 24.04 (with NVIDIA drivers & CUDA) <\/strong>as the Operating System. This template has built-in capabilities to immediately get started by running GPU workloads.<\/p>\n\n\n\n
Remember to add your SSH keys in the process, so you can access the server in the next step.<\/p>\n\n\n\n
Finally, press deploy!<\/p>\n\n\n\n
2. Connect to Your UpCloud Instance via SSH<\/strong><\/h4>\n\n\n\n
Open your terminal and connect to your UpCloud instance using its IP address and your SSH key:<\/p>\n\n\n\n
ssh ubuntu@your_instance_ip_address<\/code><\/pre>\n\n\n\nReplace
your_instance_ip_address<\/code> with the instance public IP address.
<\/p>\n\n\n\n3. Pull the Ollama Docker Image<\/strong><\/h4>\n\n\n\n
Once logged in, pull the official Ollama Docker image. This is a popular inference runtime for LLMs.<\/p>\n\n\n\n
docker pull ollama\/ollama<\/code><\/pre>\n\n\n\n3. Run the Ollama Container with GPU Support<\/strong><\/h4>\n\n\n\n
Now, run the Ollama container, ensuring it has access to your GPU resources. The
--gpus=all<\/code> flag is crucial for this. We’ll map a local port11434<\/code> (Ollama’s default port) to your host.<\/p>\n\n\n\ndocker run -d --gpus=all -v ollama:\/root\/.ollama -p 127.0.0.1:11434:11434 --name ollama ollama\/ollama<\/code><\/pre>\n\n\n\nAbout the parameters:<\/p>\n\n\n\n
- \n
-d:<\/code> Runs the container in detached mode (in the background).<\/li>\n\n\n\n--gpus=all<\/code>: Provides the container access to all available GPUs.<\/li>\n\n\n\n-v ollama:\/root\/.ollama<\/code>: Creates a Docker volume to persist Ollama’s data (models, etc.) outside the container. This means your downloaded models won’t be lost if you restart or remove the container.<\/li>\n\n\n\n-p 127.0.0.1:11434:11434<\/code>: Maps local port 11434 on your host machine to port 11434 inside the container.<\/li>\n\n\n\n--name ollama<\/code>: Assigns a convenient name to your container.<\/li>\n<\/ul>\n\n\n\n4. Pull a Mistral Model (e.g., Mistral 7B)<\/strong><\/h4>\n\n\n\n
Now that Ollama is running, you can interact with it to pull models. You’ll execute commands inside<\/em> the running Ollama container.<\/p>\n\n\n\n
To pull the popular Mistral 7B model:<\/p>\n\n\n\n
docker exec -it ollama ollama pull mistral:7b<\/code><\/pre>\n\n\n\nThe download progress will be shown in your terminal.<\/p>\n\n\n\n
5. Run the Mistral Model and Interact<\/strong><\/h4>\n\n\n\n
Once the model is downloaded, you can start an interactive session with it:<\/p>\n\n\n\n
docker exec -it ollama ollama run mistral:7b<\/code><\/pre>\n\n\n\nYou will now be in an interactive prompt where you can type your questions or prompts to the Mistral model. For example:<\/p>\n\n\n\n
>>> If UpCloud were an animal, what animal would it be?\n\nIf UpCloud were an animal, it might be a Gray Wolf (Canis lupus). Just like the wolf is known for its adaptability and pack mentality in the wild, UpCloud demonstrates adaptability in providing high-performance cloud services tailored to its clients' needs, while maintaining a strong community spirit within its user base. Additionally, both UpCloud and wolves are renowned for their intelligence and speed.<\/code><\/pre>\n\n\n\nTo exit the interactive session, type
\/bye<\/code> and press Enter.<\/p>\n\n\n\nConclusion<\/strong><\/h3>\n\n\n\n
You have successfully set up Ollama and run a Mistral model on your UpCloud GPU instance. This basic setup provides a powerful local inference environment for experimenting with large language models.<\/p>\n\n\n\n
You can now explore other models available on Ollama’s library<\/a> or integrate Ollama with applications via its API. For example, integrate it with Open WebUI<\/a>.<\/p>\n","protected":false},"author":69,"featured_media":0,"comment_status":"open","ping_status":"closed","template":"","community-category":[244,247],"class_list":["post-1849","tutorial","type-tutorial","status-publish","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/1849","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial"}],"about":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/types\/tutorial"}],"author":[{"embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/users\/69"}],"replies":[{"embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/comments?post=1849"}],"version-history":[{"count":1,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/1849\/revisions"}],"predecessor-version":[{"id":3988,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/1849\/revisions\/3988"}],"wp:attachment":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/media?parent=1849"}],"wp:term":[{"taxonomy":"community-category","embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/community-category?post=1849"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
- Cost-Efficiency:<\/strong> Running models on your own infrastructure can be more cost-effective for sustained use compared to API-based services.<\/li>\n\n\n\n
- Promote Transparency:<\/strong> Enable inspection and understanding of how models behave, contributing to explainable AI and responsible development.<\/li>\n\n\n\n