# Hardware acceleration

Source: https://frameleaf.app/docs/admin/hardware-acceleration/

A graphics card or NPU makes face recognition, smart search, text recognition and AI descriptions much faster, and takes the load off your CPU. Frameleaf can use one for two kinds of work:

- **Machine learning** in the machine learning container: smart search, faces, text recognition and AI descriptions. That's this page.
- **Video** in the server container: playback copies, seek-bar previews and Studio exports. See [Video transcoding](https://frameleaf.app/docs/admin/video-transcoding/).

Both are optional. CPU-only processing works, and for larger libraries and heavier models a GPU is strongly recommended. There's nothing to redo when you turn acceleration on: every job from then on uses the device.

## With Frameleaf Manager

Manager sets up the graphics card for you. When you install, update or restart Frameleaf, it looks at the server and:

- picks the machine learning build that matches: **NVIDIA** when Docker has the NVIDIA runtime, then **AMD**, then **Intel**. Without a supported card, or on an Arm server, it uses the processor build;
- passes the card to the machine learning container, and to the server container for video;
- starts **Faces and smart search** switched on during setup when it finds a card.

The **Machine learning** tab of [Containers](https://frameleaf.app/docs/admin/manager/#how-manager-runs-it) shows **Graphics card** when Manager found one. The same page suggests settings for it, such as **Intel graphics precision** FP16 on Intel graphics and **Run Studio masks on** `cuda` with an NVIDIA card. On a server with more than one card, add **Graphics cards to use**.

On an NVIDIA server, install the NVIDIA driver and the NVIDIA Container Toolkit on the server first, so Docker has the NVIDIA runtime. Manager only chooses the NVIDIA build when Docker reports that runtime. If Manager chose the wrong build, set `MANAGER_ML_ACCELERATION` to `cpu`, `cuda`, `rocm` or `openvino` on the Manager container and restart Frameleaf from **Services**.

Then use [Check your hardware](#check-your-hardware) to confirm the card is in use. The rest of this page, from [Before you start](#before-you-start), is for manual Docker Compose installations on Linux, or Windows through WSL2.

## Video and seek-bar previews

Turn the card on for video in **Settings**, then **Editing & playback**, then **Video playback copies**, with **Use the graphics card to convert videos faster**. From then on:

- playback copies are made with the card;
- the small pictures you see when you drag along a video's seek bar are read from the video with the card too, when it can read that video faithfully. If the card can't, Frameleaf reads that video with the processor instead, and the previews look the same either way.

If the card stops helping, the page says **The graphics card stopped helping** with the date, and copies are made with the processor until it's fixed. Every video still plays.

## Check your hardware

**Settings**, then **Computers & background work**, then **Graphics card check** answers first: **Your graphics card is working**, **Your graphics card isn't being used**, **No graphics card found** or **AI features are not running**. **Check again** reruns the check. Below that, **What this server can do** says, for each kind of heavy work, how well this server does it on its own.

Frameleaf runs as two separate programs, each in its own container, and each needs its own access to the GPU, so both are checked. Open **Set-up help**, then **Technical details** to see, for the server container (video and Studio export) and the machine learning container (AI features):

| Fact | What it means |
| --- | --- |
| **On the computer** | The host's PCI list, read from inside the container, shows a GPU. Under WSL2 or Docker Desktop the host's GPUs are hidden, so this shows **Not reported** unless the container sees one |
| **Visible to Frameleaf** | An NVIDIA device, a `/dev/dri` render node, `/dev/kfd` or `/dev/dxg` was passed in |
| **In use** | The test transcode ran on the GPU's video engine, or the analysis runtime (CUDA, ROCm or OpenVINO) is using the GPU |

**Other computers** lists the graphics cards your Studio and restoration helpers reported.

Under **How to fix it**, the check names the set-up problems it finds, each with the fix and, where Docker Compose can fix it, a ready-to-paste snippet with the values it found, such as the render group's number or the graphics card's PCI path. It detects, among others:

- the NVIDIA Container Toolkit missing, or an old driver
- the `video` capability missing for NVENC
- `/dev/dri` not passed in, or a render node owned by a group the container doesn't have
- the integrated GPU used instead of the graphics card, on a host with both
- an AMD RX 6000 or 7600 card that needs `HSA_OVERRIDE_GFX_VERSION` for ROCm
- WSL2, Unraid, Docker Desktop on a Mac, and the CPU image on a host with a GPU

A Turing card (compute capability 7.5, such as the GTX 16 series) gets a note: it has no bf16 or FlashAttention, so models that use them run slower.

### Benchmark

Under **Tune and measure**, **Measure this server's speed** (**Run the test**) takes from a few seconds to a few minutes: descriptions timed on the processor take longest. It times search embeddings and a test transcode, then records the speed of each kind of work the [model sliders](https://frameleaf.app/docs/admin/workers/#choose-how-heavy-a-model-to-use) estimate. Afterwards the speeds on the page come from your own server:

- **Descriptions and tags:** a few generated test photos are described with your description model on this server's machine learning container.
- **Restoration and Smooth motion:** the speeds the restoration worker measured on its GPU when it was qualified.
- **Upscale** has no local runner, and transcription workers report no speed; the benchmark tells you so.
- If a kind of work can't be timed here, for example because the description model can't run on this server, it reads **Couldn't be measured here**. The details are in the server's log.

## Supported hardware

| Backend | Hardware | Image suffix |
| --- | --- | --- |
| **CUDA** | NVIDIA GPUs with compute capability 5.2 or higher. The most reliable option | `-cuda` |
| **ROCm** | AMD GPUs | `-rocm` |
| **OpenVINO** | Intel GPUs such as Iris Xe and Arc | `-openvino` |
| **ARM NN** | Devices with Mali GPUs. Other Arm devices aren't supported | `-armnn` |
| **RKNN** | Rockchip RK3566, RK3568, RK3576 and RK3588 | `-rknn` |

Not every model runs on every backend. ARM NN speeds up smart search jobs, but not searching itself.

## Before you start

**CUDA**

- The official NVIDIA driver, version 545 or newer (it must support CUDA 12.3).
- On Linux (not WSL2), the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html).

**ROCm**

- On Linux, the AMDGPU driver module. With Secure Boot, the DKMS signing key must be enrolled in the UEFI firmware.
- A GPU supported by ROCm. If yours isn't officially supported, try `HSA_OVERRIDE_GFX_VERSION=<a supported version, such as 10.3.0>`, and if that doesn't work, also `HSA_USE_SVM=0`.
- At least 35 GiB of free disk space for the ROCm image. Later updates are usually only a few hundred megabytes.
- After running, the GPU can draw more power than usual until the service has been idle for 5 minutes (set by `MACHINE_LEARNING_MODEL_TTL`). The MIGraphX backend compiles models when they're first used, so the first few results are slow.

**OpenVINO**

- A kernel new enough to use the device.
- More RAM than CPU processing. Discrete GPUs are the smoothest choice; integrated GPUs on older processors or servers with little RAM are more likely to have problems.
- The `openvino` section of `hwaccel.ml.yml` passes in `/dev/dri`. The machine learning container runs as an ordinary user, not root, and joins the groups that own the graphics devices it was given by itself, so there are no `render` or `video` group numbers to add.
- On WSL2, use `openvino-wsl`, which passes in `/dev/dxg` and `/usr/lib/wsl` instead.
- `MACHINE_LEARNING_OPENVINO_PRECISION=FP16` is faster on Intel graphics and very slightly less exact. The default is `FP32`.

**ARM NN**

- The vendor's kernel driver, usually preinstalled on the vendor's Linux image.
- `/dev/mali0` on the host (check with `ls /dev`).
- The closed-source `libmali.so` firmware from your device vendor. `hwaccel.ml.yml` expects it at `/usr/lib/libmali.so`, plus `/lib/firmware/mali_csffw.bin`; change the paths if yours differ.
- Optionally, `MACHINE_LEARNING_ANN_FP16_TURBO=true` can make it much faster at the cost of very slightly lower accuracy.

**RKNN**

- The vendor's kernel driver, usually preinstalled.
- RKNPU driver 0.9.8 or later (check with `cat /sys/kernel/debug/rknpu/version`).
- Optionally, setting `MACHINE_LEARNING_RKNN_THREADS` to 2 or 3 can make RK3576 and RK3588 dramatically faster than the default of 1, but multiplies each model's RAM use by the same amount.

## Set it up

1. Download [`hwaccel.ml.yml`](https://github.com/Frameleaf/frameleaf-app/releases/latest/download/hwaccel.ml.yml) into the same folder as `docker-compose.yml`.
2. In the `frameleaf-machine-learning` service, add `-cuda`, `-rocm`, `-openvino`, `-armnn` or `-rknn` to the end of the `image` tag.
3. In the same service, uncomment the `extends` section and change `cpu` to your backend: `cuda`, `rocm`, `openvino`, `openvino-wsl`, `armnn` or `rknn`. Use the `-wsl` version on WSL2 where there is one.
4. Redeploy the machine learning container.

The result looks like this:

```yaml
frameleaf-machine-learning:
  container_name: frameleaf_machine_learning
  image: ghcr.io/frameleaf/frameleaf-machine-learning:${FRAMELEAF_VERSION:-latest}-cuda
  extends:
    file: hwaccel.ml.yml
    service: cuda
  volumes:
    - model-cache:/cache
    - studio-worker:/run/frameleaf/studio-worker:ro
  env_file:
    - path: .env.ml
      required: false
  restart: always
```

Only the `image` and `extends` lines change; leave the rest of the service as the release wrote it. Machine learning variables, such as `MACHINE_LEARNING_OPENVINO_PRECISION`, go in `.env.ml` next to `.env`, because the machine learning container doesn't read `.env`.

The worker that Studio uses in the browser runs in the same container and uses its graphics card when it has one.

### Confirm it's working

- Watch GPU use while a job runs, with `nvtop` (NVIDIA or Intel), `intel_gpu_top` (Intel) or `radeontop` (AMD).
- Or check the machine learning container's logs (`docker logs frameleaf_machine_learning`) when a smart search or face detection job starts, or when you search with text. You should see `Available ORT providers` listing your provider, such as `CUDAExecutionProvider`, or for ARM NN a `Loaded ANN model` line without errors.

## AI descriptions and Locked-content detection

Image enrichment uses two separate model paths:

| Task | CUDA | OpenVINO |
| --- | --- | --- |
| Locked-content (NSFW) detection | ONNX Runtime with `onnx-community/nsfw_image_detection-ONNX` | ONNX Runtime with OpenVINO |
| Descriptions and tags | Transformers and PyTorch in the CUDA image | OpenVINO GenAI |

The default description model is `Qwen/Qwen2.5-VL-3B-Instruct`. On Intel integrated graphics it's mapped to the OpenVINO-converted `llmware/qwen2.5-vl-3b-ov`. On NVIDIA it runs directly, and the same code path handles larger Qwen2.5-VL models (7B, 32B, 72B) and Qwen3-VL. Larger models need more video memory. The fallback model is `microsoft/Florence-2-base-ft`.

The CUDA image keeps NVIDIA acceleration for the other models too, and the service still prefers `CUDAExecutionProvider` when CUDA is installed. See [AI descriptions and smart albums](https://frameleaf.app/docs/using/ai-descriptions/).

### Model licences

Some models limit commercial use. If you use a model outside your own installation, check its licence first.

| Model | Licence |
| --- | --- |
| `Qwen/Qwen2.5-VL-3B-Instruct` and `llmware/qwen2.5-vl-3b-ov` | Qwen Research License Agreement (Alibaba Cloud) |
| `nllb-clip` search models (every variant) | CC-BY-NC-4.0 |
| MusicGen-small (`Xenova/musicgen-small`) | CC-BY-NC-4.0 |

## Unraid, Portainer and single Compose files

Some platforms can't use more than one Compose file. On Unraid with Manager, the card is set up for you; this section is for stacks you run yourself. Copy the relevant section of `hwaccel.ml.yml` into the service directly instead of using `extends`. For CUDA, that's:

```yaml
frameleaf-machine-learning:
  container_name: frameleaf_machine_learning
  # Note the -cuda at the end, and no extends section
  image: ghcr.io/frameleaf/frameleaf-machine-learning:${FRAMELEAF_VERSION:-latest}-cuda
  deploy:
    resources:
      reservations:
        devices:
          - driver: nvidia
            count: 1
            capabilities:
              - gpu
  volumes:
    - model-cache:/cache
    - studio-worker:/run/frameleaf/studio-worker:ro
  env_file:
    - path: .env.ml
      required: false
  restart: always
```

For Intel, use `-openvino` and, in place of the `deploy` section, the two entries from the `openvino` section of `hwaccel.ml.yml`:

```yaml
  device_cgroup_rules:
    - 'c 189:* rmw'
  devices:
    - /dev/dri:/dev/dri
```

Then redeploy the machine learning container.

For video transcoding on these platforms, such as Intel Quick Sync (pass in `/dev/dri`) or NVENC on Unraid, see [Video transcoding](https://frameleaf.app/docs/admin/video-transcoding/#single-compose-file).

## Several GPUs

For more than one NVIDIA or Intel GPU, set `MACHINE_LEARNING_DEVICE_IDS` to a comma-separated list of device IDs and `MACHINE_LEARNING_WORKERS` to how many you listed. Find the IDs with `nvidia-smi -L` or `glxinfo -B`.

```bash
MACHINE_LEARNING_DEVICE_IDS=0,1
MACHINE_LEARNING_WORKERS=2
```

The service then starts two workers, one per GPU, and each request goes to one or the other. To always use one particular GPU, list just that one, for example `MACHINE_LEARNING_DEVICE_IDS=1`.

Raise the job [concurrency](https://frameleaf.app/docs/admin/jobs/#concurrency) to keep several GPUs busy. Each GPU must be able to hold all the models on its own: a model can't be split across GPUs, and you can't send one model to one GPU.

## Tips

- If a model fails, try a different one to see whether the problem is specific to that model.
- Raising concurrency past the default improves GPU use, but also uses more video memory.
- Larger models benefit more from acceleration, if you have the video memory.
- On Rockchip boards, RKNN (the NPU) supports more models than ARM NN, including search, runs cooler and leaves the GPU free for transcoding. It always uses FP16, so it's very slightly less accurate than ARM NN's default FP32. With one thread it's slower than ARM NN for jobs but similar for searching; with three threads it's somewhat faster than ARM NN at FP32, at the cost of more RAM.
