Amd on Ulis Notes https://wolf-u.li/en/tag/amd/ Recent content in Amd on Ulis Notes Hugo en Copyright 2006-CURRENTYEAR Uli Wolf - All rights reserved Sun, 04 Aug 2024 20:11:22 +0400 Using Ollama with AMD RX 6700 (gfx1031) or other older cards on Windows https://wolf-u.li/using-ollama-with-amd-rx-6700-gfx1031-or-other-older-cards-on-windows/ Sun, 04 Aug 2024 20:11:22 +0400 https://wolf-u.li/using-ollama-with-amd-rx-6700-gfx1031-or-other-older-cards-on-windows/ <p>Ollama, the open-source platform for running powerful AI models locally on your hardware, is gaining traction for its ease of use and accessibility. While it primarily leverages NVIDIA GPUs through CUDA, the world of open-source extends beyond NVIDIA. Today, I&rsquo;ll show you how to harness the power of an AMD RX 6700 GPU with ROCm to run Ollama, bringing powerful AI capabilities within reach of a wider range of users. <ins class="adsbygoogle" style="display:block; text-align:center;" data-ad-layout="in-article" data-ad-format="fluid" data-ad-client="ca-pub-2322978122735420" data-ad-slot="3161252284"></ins></p> Ollama, the open-source platform for running powerful AI models locally on your hardware, is gaining traction for its ease of use and accessibility. While it primarily leverages NVIDIA GPUs through CUDA, the world of open-source extends beyond NVIDIA. Today, I’ll show you how to harness the power of an AMD RX 6700 GPU with ROCm to run Ollama, bringing powerful AI capabilities within reach of a wider range of users.

What is ROCm?

ROCm (Radeon Open Compute Platform) is AMD’s open-source software platform for high-performance computing on Radeon GPUs. It provides the necessary drivers, libraries, and tools to unlock the full potential of AMD GPUs for various tasks, including machine learning.

What is the issue?

While the AMD graphic cards are a cost-effective option (Compared to some NVIDIA counterparts), the challenge comes in with the general AI toolset support. E.g. ROCm by default doesn’t feature the RX 6700 which i own. You can check the compatibility matrix here . Also Ollama recently announced  that it supports some of the AMD cards but also there the support (due to the limitation of ROCm) is limited to some of the cards.

Any way out?

Thanks to ROCm’s open nature allows for greater transparency and community collaboration, some very clever developers found a way to also compile the ROCm libraries, especially the HIP support for older cards.

Solution

The solution is quite easy actually. We’ll be using a repository which follows the official Ollama Releases but where the developer adds ROCm support for older devices.

Check Instructions

First check the instructions 

Install ROCm

Download ROCm (or respectively the HIP SDK) from here . As of the time of writing this article, the version 5.7.1 was the supported one by the Ollama repository.

Install modified Ollama

As mentioned above, we will be using a modified version of Ollama which you can download from here . Once installed it might ask you to update Ollama which you MUST NOT do.

Implement modified ROCblas

The next step is to visit this page  and to choose the correct file according to your graphics architecture. In my case, an RX 6700 (which is a gfx1031 architecture) i would download either rocm.gfx1031.for.hip.sdk.5.7.optimized.with.little.wu.logic.and.I8II.support.7z or rocBLAS-HIP5.7.1-win.rar. Now you need to unpack the rocblas.dll from the archive to the following directories:

  • C:\Program Files\AMD\ROCm\5.7\bin
  • %LocalAppData%\Programs\Ollama\lib\ollama

And the folder library from the archive to the following directories:

  • C:\Program Files\AMD\ROCm\5.7\bin\rocblas
  • %LocalAppData%\Programs\Ollama\lib\ollama\rocblas

Test

After completing these steps, open the Ollama and run your model.

Updates to this article

  • 2025-Jan-28: Tested ROCm 6.1 and failed miserably with ollama-for-amd v0.5.4. Went back to ROCm 5.7 with ollama-for-amd v0.5.4.
  • 2025-Jan-29: Added various clarifications and also more details for the unpacking of the modified rocm libraries.
]]>