You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hey, it'd be really nice if we could run this model API on AMD cards, and it would provide a very nice speedup over CPU. Do you think you could add this when you have some free time? It shouldn't be too hard since afaik ONNX models can run with ROCM just fine.
We're refactoring a few things to more easily support different architectures, though I haven't worked with AMD cards yet. Will try adding them to the onnx providers but may need some community testing to get right
I've got it to work with rocm. The problem is that the docker image becomes huge: Around 22G when I use rocm/dev-ubuntu-22.04 as a base image. I might get it down with a slimmer base image, but I don't expect it to come even close to ghcr.io/remsky/kokoro-fastapi-gpu, which is about 5.7G.
I've got it to work with rocm. The problem is that the docker image becomes huge: Around 22G when I use rocm/dev-ubuntu-22.04 as a base image. I might get it down with a slimmer base image, but I don't expect it to come even close to ghcr.io/remsky/kokoro-fastapi-gpu, which is about 5.7G.
What version of rocm did you get it working with I have a 5700xt and that thing only supports rocm 5.2 unofficially. Any chance you could get it working on that? Or maybe get direct-ml working with the docker file.
What version of rocm did you get it working with I have a 5700xt and that thing only supports rocm 5.2 unofficially. Any chance you could get it working on that? Or maybe get direct-ml working with the docker file.
The Dockerfile is based on 6.3.1. Works on 7900XTX.
Okay I tried that and it gave me the following error
Error response from daemon: error gathering device information while adding cust
om device "/dev/kfd": no such file or directory
Taking a shot at this shortly, will post links if anyone is able to test it out
(I don't have an AMD card unfortunately, so can just skeleton it up)
My fork works on my linux machine. Didn't make a PR, though, because I personally didn't find it convincing enough to pursue. Please feel free to use my stuff if you see any value.
Would ONNX work on the intel Arc GPUs?
Theoretically yes (with the oneDNN executor it should work OpenVino not because of unsuported Ops)
But for Arc I already implemented a version with ipex (intel extention for pytorch) support that uses nearly the exact code as CUDA GPUs https://github.com/TheMrCodes/Kokoro-FastAPI
Taking a shot at this shortly, will post links if anyone is able to test it out
(I don't have an AMD card unfortunately, so can just skeleton it up)
I'm interested in setting this up and would be happy to help test things out. I'm running Ubuntu 24.04.1 with an RX 7900 XTX.
Thanks @bgs4free for your fork and rocm docker-compose file! Unfortunately I'm also seeing the issue where neither of the GPU devices /dev/kfd or /dev/dri are found in the container.
The devices show up on the host outside the container, and I can at least get the container running by either removing the device flags or setting the privileged flag, but in both cases the devices aren't visible inside. Inside the container, rocminfo
returns ROCk module is NOT loaded, possibly no GPU devices, while outside the container I get a large amount of GPU information.
I'm not entirely certain I have the right drivers installed, but ollama and alltalk_tts can both run and use my GPU.
Activity
remsky commentedon Jan 17, 2025
We're refactoring a few things to more easily support different architectures, though I haven't worked with AMD cards yet. Will try adding them to the onnx providers but may need some community testing to get right
Reed-Schimmel commentedon Jan 18, 2025
Would ONNX work on the intel Arc GPUs?
bgs4free commentedon Jan 18, 2025
I've got it to work with rocm. The problem is that the docker image becomes huge: Around 22G when I use
rocm/dev-ubuntu-22.04as a base image. I might get it down with a slimmer base image, but I don't expect it to come even close toghcr.io/remsky/kokoro-fastapi-gpu, which is about 5.7G.https://github.com/bgs4free/Kokoro-FastAPI/tree/add-rocm-support
bgs4free commentedon Jan 18, 2025
Theoretically, yes, but I had difficulties compiling it on linux.Edit: misread your question. Thought this was about AMD.
gitchat1 commentedon Jan 19, 2025
What version of rocm did you get it working with I have a 5700xt and that thing only supports rocm 5.2 unofficially. Any chance you could get it working on that? Or maybe get direct-ml working with the docker file.
bgs4free commentedon Jan 19, 2025
The
Dockerfileis based on 6.3.1. Works on 7900XTX.https://github.com/bgs4free/Kokoro-FastAPI/blob/b064066b9ef80ee09afefb8ff53e2ffbee1387de/docker/rocm/Dockerfile#L1
I'm using the pytorch index for 6.2, because I had issues with the 6.3 version (version mismatch seems to be no issue AFAIK).
https://github.com/bgs4free/Kokoro-FastAPI/blob/b064066b9ef80ee09afefb8ff53e2ffbee1387de/docker/rocm/pyproject.toml#L23
Change it to 5.2, try and see if it works. Make sure you have enough disk space.
gitchat1 commentedon Jan 20, 2025
Okay I tried that and it gave me the following error
Error response from daemon: error gathering device information while adding cust
om device "/dev/kfd": no such file or directory
Acording to this thread https://stackoverflow.com/questions/73229163/amd-rocm-with-pytorch-on-navi10-rx-5700-rx-5700-xt you have to Set the environment variable.HSA_OVERRIDE_GFX_VERSION=10.3.0 I just added the variable to the docker file under env. Any idea what else i could try?
bgs4free commentedon Jan 20, 2025
devices get passed through from the host. See: https://github.com/bgs4free/Kokoro-FastAPI/blob/add-rocm-support/docker/rocm/docker-compose.yml#L20
What is your host system? This was tested on Linux and I can't speak for any other OS.
gitchat1 commentedon Jan 20, 2025
Oh, okay its windows. I thought that wouldn't reaööy matter since the docker runs an Ubuntu image.
bgs4free commentedon Jan 20, 2025
https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/docker.html
Don't see anything about docker for windows here.
gitchat1 commentedon Jan 21, 2025
I checked my docker config it is running through wsl so that is technically Ubuntu so it should work.
remsky commentedon Jan 24, 2025
Taking a shot at this shortly, will post links if anyone is able to test it out
(I don't have an AMD card unfortunately, so can just skeleton it up)
bgs4free commentedon Jan 24, 2025
My fork works on my linux machine. Didn't make a PR, though, because I personally didn't find it convincing enough to pursue. Please feel free to use my stuff if you see any value.
TheMrCodes commentedon Jan 31, 2025
But for Arc I already implemented a version with ipex (intel extention for pytorch) support that uses nearly the exact code as CUDA GPUs
https://github.com/TheMrCodes/Kokoro-FastAPI
For more on Arc please use Issue#106
DavidSchmittTech commentedon Feb 8, 2025
I'm interested in setting this up and would be happy to help test things out. I'm running Ubuntu 24.04.1 with an RX 7900 XTX.
Thanks @bgs4free for your fork and rocm docker-compose file! Unfortunately I'm also seeing the issue where neither of the GPU devices
/dev/kfdor/dev/driare found in the container.The devices show up on the host outside the container, and I can at least get the container running by either removing the device flags or setting the privileged flag, but in both cases the devices aren't visible inside. Inside the container,
rocminforeturns
ROCk module is NOT loaded, possibly no GPU devices, while outside the container I get a large amount of GPU information.I'm not entirely certain I have the right drivers installed, but ollama and alltalk_tts can both run and use my GPU.