Workers and where jobs run

Frameleaf can use computers other than the server itself for heavy work, such as a desktop with a good graphics card. You decide, for each kind of work, where it runs and how heavy a model it uses. Nothing ever moves somewhere else on its own.

Machine learning work goes to destinations: the machine learning container that ships with the server, and other machines on your network. Administration, then Processing destinations shows them in two parts:

Workers & endpoints, an inventory of every worker and its current state
Processing destinations, where you add workers and choose where each kind of work runs

For the jobs that run inside the server itself, see Jobs and queues.

Check your hardware first

Settings, then Compute & jobs, then Hardware & GPU checks whether each container can actually use your GPU, and suggests fixes. It can also run a short benchmark that times search, a transcode, descriptions and restoration. See Hardware acceleration.

Two kinds of worker

Library analysis and restoration never share a worker.

Work	Runs on	Image
Library analysis: faces, smart search and similarity, text recognition, descriptions	The machine learning container, or another one on your network	`frameleaf-machine-learning`
Restoration: restoring and upscaling photos and video (Faithful and Creative)	A restoration worker on this server or your network	Built from source; see below

Render workers handle Studio exports and previews.

The server enforces these rules:

A destination on this server or your network may do library analysis or restoration, not both. Ticking one kind in the form closes the other.
A restoration is refused, never moved, if its destination is still allowed both kinds of work from before this rule, or points at an address that library analysis is routed to.
Destinations saved before the rule that allow both kinds keep running library analysis, and are listed under Workers allowed both kinds of work until you limit them to one.

The inventory

Workers & endpoints lists every worker with:

Column	What it shows
State	See the table below
Capabilities, GPU memory and acceleration	As the worker reported them. A restoration worker reports each GPU’s memory; the machine learning container reports only its execution providers, so its GPU memory shows as unknown
Credentials	Only whether a token is stored. Tokens are never shown after they’re saved
Routed here and Admission	Which kinds of library work go to this worker, and what the server would answer for each, from the last check
Load	Jobs running and waiting for this worker

State	Meaning
Not checked	Frameleaf hasn’t checked it yet
Turned off	It’s switched off
Unreachable	The last check got no answer, or there’s no address
No models ready	It answered, but serves none of the work it’s allowed
CPU only	It does its work without an accelerator
Model ready	It’s ready to work

A configured address never counts as ready on its own.

Further down, Library analysis routes lists each kind of library work and the worker it goes to; a kind with no route is shown as refused. Render workers lists Studio render workers by when they last checked in (they call the server, so there’s no address to show), and Restoration runners on this server lists the server processes holding restoration jobs right now.

The inventory refreshes every 30 seconds while the page is open. If a refresh fails, the last snapshot stays on screen with its age. Check capabilities checks that one worker now and contacts nothing else.

Machine learning addresses

Machine-learning endpoints edits the list of machine learning addresses, the same list as in Administration, then Settings, then Machine Learning Settings.

Use each worker’s base address, over HTTP or HTTPS, without a password, query or fragment.
You can add up to 32, and at least one must remain. The default is http://immich-machine-learning:3003.
The server creates a destination for each address and sends any library work that has no route yet to the first one. The order sets nothing else.
When an address leaves the list, its destination is turned off: work routed there is refused, not moved, until you route it elsewhere. Add the address back and the destination is turned on again.
Removing an address doesn’t stop the worker itself, and existing restorations keep their destination.

To run the machine learning container on another computer, see Remote machine learning.

Choose how heavy a model to use

Where each job runs has a slider for each kind of work, from lighter to heavier models. The colour of each stop shows where it can run:

White: runs on the processor
Green: fits your GPU, as the last Hardware & GPU check found it
Blue: runs only on Cloud GPU

Without a hardware check every local stop is white; run the check to see which fit. Models that need more GPU memory than you have, or CUDA on a GPU without it, are crossed out with the reason. Models whose licence doesn’t allow commercial use never run in the cloud.

Choosing a white or green stop for descriptions changes the description model setting, saved with the settings bar. The machine learning container downloads a model it doesn’t have the first time a job uses it, which can take several minutes. Restoration workers bring their own models.

Local, cloud or both

For each kind of work, choose Local only, Both or Cloud only.

Always on your server: smart search, faces, text recognition and Studio exports. These can’t be sent to the cloud.
Never automatic: a job never moves to the cloud on its own. Cloud jobs always show an estimate and wait for someone to confirm.

A separate restoration container

From a source checkout, start it next to the regular services with the overlay file:

docker compose -f docker-compose.yml -f docker-compose.restoration.yml up -d

Then add a home network destination at http://frameleaf-restoration:3004, allowed restoration only, with the token from FRAMELEAF_RESTORATION_TOKEN if you set one.

Variable	What it sets	Default
`FRAMELEAF_RESTORATION_GPU`	The GPU the restoration worker uses	None; you must set it
`FRAMELEAF_RESTORATION_TOKEN`	A bearer token the worker requires. Store it on the destination	Empty (no token)
`FRAMELEAF_RESTORATION_CONFIG`	Folder with the model manifest and qualification record, read-only	`./restoration/config`
`FRAMELEAF_RESTORATION_WEIGHTS`	Folder with the model weights, read-only	`./restoration/weights`

Keep library analysis responsive

Separate GPUs: set FRAMELEAF_RESTORATION_GPU to a GPU the machine learning container doesn’t use, and pin that container to its own GPU with device_ids in hwaccel.ml.yml.
One GPU for both: tick This worker shares a GPU with library analysis on the restoration destination. Full restorations then wait while face, search, text and description jobs have work, and start when it’s done. Previews still run, because someone is waiting for them and they’re short. A paused library queue doesn’t hold restoration back.
On the server: each server process runs at most one restoration at a time on its own loop, so a restoration never takes a job queue’s slot.

Long jobs keep going

Studio renders, edits, restorations, bulk changes, description runs, Library Care scans, imports and physical deduplication are durable: they keep running if you close the browser and survive a server restart.

A failed attempt is retried once automatically after 30 seconds.
A failed attempt never replaces the previous good result.
Jobs belong to the person who started them. Administrators see how many there are, not what’s in them.

For scripts

GET /api/admin/workers returns the inventory to administrators. It reads stored state only: loads are counts, runners are process names, and no owner, item or credential is returned. Destinations, routes and checks use the /api/ml-destinations endpoints.