Ollama, the open-source platform for running powerful AI models locally on your hardware, is gaining traction for its ease of use and accessibility. While it primarily leverages NVIDIA GPUs through CUDA, the world of open-source extends beyond NVIDIA. Today, I’ll show you how to harness the power of an AMD RX 6700 GPU with ROCm to run Ollama, bringing powerful AI capabilities within reach of a wider range of users.
What is ROCm?
ROCm (Radeon Open Compute Platform) is AMD’s open-source software platform for high-performance computing on Radeon GPUs. It provides the necessary drivers, libraries, and tools to unlock the full potential of AMD GPUs for various tasks, including machine learning.
What is the issue?
While the AMD graphic cards are a cost-effective option (Compared to some NVIDIA counterparts), the challenge comes in with the general AI toolset support. E.g. ROCm by default doesn’t feature the RX 6700 which i own. You can check the compatibility matrix here . Also Ollama recently announced that it supports some of the AMD cards but also there the support (due to the limitation of ROCm) is limited to some of the cards.
Any way out?
Thanks to ROCm’s open nature allows for greater transparency and community collaboration, some very clever developers found a way to also compile the ROCm libraries, especially the HIP support for older cards.
Solution
The solution is quite easy actually. We’ll be using a repository which follows the official Ollama Releases but where the developer adds ROCm support for older devices.
Check Instructions
First check the instructions
Install ROCm
Download ROCm (or respectively the HIP SDK) from here . As of the time of writing this article, the version 5.7.1 was the supported one by the Ollama repository.
Install modified Ollama
As mentioned above, we will be using a modified version of Ollama which you can download from here . Once installed it might ask you to update Ollama which you MUST NOT do.
Implement modified ROCblas
The next step is to visit this page and to choose the correct file according to your graphics architecture. In my case, an RX 6700 (which is a gfx1031 architecture) i would download either rocm.gfx1031.for.hip.sdk.5.7.optimized.with.little.wu.logic.and.I8II.support.7z or rocBLAS-HIP5.7.1-win.rar. Now you need to unpack this file to the following two directories:
C:\Program Files\AMD\ROCm\5.7\bin%LocalAppData%\Programs\Ollama\rocm
Test
After completing these steps, open the Ollama and run your model.