# Set up AI descriptions

Source: https://frameleaf.app/docs/using/ai-descriptions-setup/

Give everyone on your server searchable captions that name the people in their photos. This guide is for administrators: it shows you how to set up [AI descriptions](https://frameleaf.app/docs/using/ai-descriptions/), pick the right model, tune the wording and caption your existing library in one pass. The settings are in **Settings**, then **Search & intelligence**, then **Descriptions & tags**. Every control on that page is listed in [System settings](https://frameleaf.app/docs/admin/system-settings/#descriptions--tags).

:::caution[Read this before a big run]
Captioning a whole library takes hours or days. Follow the setup order below and you'll only need to do it once.
:::

## Before you start

- The computer that does this work is answering, ideally with a graphics card. See [Hardware acceleration](https://frameleaf.app/docs/admin/hardware-acceleration/#ai-descriptions-and-locked-content-detection).
- For names in captions, **Faces & identity** is on and you've named some people. See [People](https://frameleaf.app/docs/using/people/).
- Videos need nothing extra. Each video's moment frames are cut the first time it's described.

## Setup order

| Step | Do | Why now |
| --- | --- | --- |
| 1 | **Pick the caption model** | Florence-2 ignores most other choices on this page. Get the model right before you tune anything. |
| 2 | **Name your most-photographed people** | Captions only use names you've given. Doing this first means the first run already says "Kelly" instead of "Someone". |
| 3 | **Try it on a few photos** | Nothing is written, so you see the result before a library-wide run. |
| 4 | **Tune the wording** | Tuning before the big run means you don't caption everything twice. |
| 5 | **Update captions now** | One library-wide pass with everything set the way you want. |
| 6 | **Turn on smart albums** | The built-in smart albums use the tags captions add, and what visual search has read. |
| 7 | **Update albums now** once captions are done | A one-time catch-up for older photos. |

Already started in a different order? No problem. You can update captions at any time, and update smart albums one album at a time.

## Quick setup

1. Go to **Settings**, then **Search & intelligence**, then **Descriptions & tags**.
2. Make sure **Write captions for photos** is on. It is on a new server.
3. Under **Where and when captions are written**, pick the **Caption model**. The default, Qwen2.5-VL 3B, suits most servers.
4. Make sure **Mention people I have named** is on.
5. Choose **Try it** beside **Try it on a few photos**, and read the **Sample captions**.
6. Save your changes.
7. Choose **Update captions now** at the top of the page, check the estimate, and choose **Re-queue now**.
8. Turn on [smart albums](https://frameleaf.app/docs/using/smart-albums/) if you want them.

New photos are captioned soon after they arrive, using your current settings. Rewriting many captions waits for the [overnight schedule](https://frameleaf.app/docs/admin/jobs/#nightly-tasks).

## Choose a model

The **Caption model** picker shows the models that suit the computer doing the work, lightest first. A bigger model writes better captions and needs more graphics memory.

| Model | Graphics memory | Notes |
| --- | --- | --- |
| Florence-2 base | About 1 GB | Short captions, tags and text; not for long descriptions. The usual backup model. |
| Florence-2 large | About 3 GB | Short captions, tags and text; not for long descriptions |
| Qwen2.5-VL 3B (default) | About 6 GB | Natural descriptions on small graphics cards or the processor. On Intel it runs as `llmware/qwen2.5-vl-3b-ov`. |
| Qwen2.5-VL 7B | About 16 GB | More detailed descriptions and better text reading. On Intel it runs as `llmware/qwen2.5-vl-7b-ov`. |
| Qwen3-VL 30B-A3B | About 60 GB | Large-model quality at small-model speed. NVIDIA only. |
| Qwen2.5-VL 32B | About 64 GB | Very detailed descriptions. NVIDIA only. |
| Qwen2.5-VL 72B | About 144 GB | The most detailed descriptions; needs several graphics cards. NVIDIA only. |

A model that doesn't fit your hardware is shown as unavailable, with the reason, such as how much graphics memory it needs and how much your card has. When this server is linked to a Frameleaf account, the picker also shows the models Frameleaf Cloud offers, in blue, with the cost shown first.

To use a model that isn't in the picker, open **Advanced: caption model and hardware** and type its full name, like `organisation/model-name`, in **Hugging Face model**. It's downloaded the first time captions are written. Empty the box to go back to a published model.

:::caution[Florence-2 ignores your wording]
Florence-2 writes short captions, tags and text. It doesn't follow your extra instructions, word lists or the names of people. Use a Qwen model for captions that follow the choices on this page.
:::

:::caution[Model licences]
Each model comes with its own licence. Check the licence for the model you download. See [Model licences](https://frameleaf.app/docs/admin/hardware-acceleration/#model-licences).
:::

### Backup model

The **Backup model**, under **Advanced: caption model and hardware**, keeps captions flowing when the main model isn't available. Florence-2 base is the default because it's small enough to share a graphics card with other models. It must be installed too. Choose **Another model, by its Hugging Face name…** to type a different one.

### Hardware

**Kind of graphics card** is **Automatic** by default, and says what it found. Choose **NVIDIA (CUDA)** or **Intel (OpenVINO)** to set it yourself; that also picks the caption and private-photo models that suit the card. It doesn't install anything. Leave **Which graphics card** as `AUTO` unless the computer has more than one.

Models come from your configured model source. See [Where models come from](https://frameleaf.app/docs/admin/environment-variables/#where-models-come-from). Captions are written by your own server, or by a computer at home, unless you send that work to Frameleaf Cloud in **Where work runs**.

## Preview before you commit

See what you'll get before you run anything across the library.

**Try it on a few photos** writes **Sample captions** for three of your newest photos, with the choices on the page, saved or not. Each sample shows its caption and tags. Change something and choose **Try again with these choices**. Nothing in your library changes.

**More photos and choices…** opens **Try an enrichment change**, for a closer look:

1. **Choose sample.** Pick up to six of your own photos or videos, and the computer to run them on.
2. **Compare.** The model and wording you're editing run on each sample. The **Current** description and the **Candidate** appear side by side, with the tags, any names that were removed because they aren't confirmed on the photo and, for a video, how many frames it saw. Nothing is saved, and existing descriptions are unchanged.
3. **Scope.** Choose what a background plan should do to the samples you chose, such as descriptions and tags for photos, or frames, the moment search index and moment captions for videos. Moment captions add one model request per frame, and the dialog says how many.

A plan uses your **saved** model and wording, so save first to run the version you previewed. It runs in the background, shows in **Activity**, and continues after you close the page. You can **Pause**, **Resume** or **Cancel** it, and **Retry** runs the items that failed once more. Descriptions written by hand and people's own moments are kept.

## Prompt & Vocabulary

The everyday choices are at the top of the page:

| Setting | What it does | Default |
| --- | --- | --- |
| **Caption length** | **Short** (one sentence), **Medium** (two or three sentences) or **Detailed** (up to six sentences, with colours, weather and what people are doing). Longer captions are not always more accurate. | Medium |
| **Mention people I have named** | Captions say "Jamie at the lake". People you haven't named stay unnamed. | On |
| **Things to look out for** | Things captions mention when they can be seen in the photo | brands, signage, screens, documents, uniforms, tools, vehicles, animals, food, landmarks |
| **Never guess at** | Things captions leave out, even when the photo suggests them. Check a few captions after changing this list: it can't be guaranteed. | diagnoses, medication names, procedures, pregnancy, disability |

**Advanced: wording lists and extra instructions** has the rest:

| Setting | What it does | Default |
| --- | --- | --- |
| **Sentences in a caption** | From 1 to 6. Set by **Caption length**. The model sometimes goes one over. | 3 |
| **Writing style** | **Plain and brief**, **Balanced** or **Rich in detail**. Set by **Caption length**. | Balanced |
| **Preferred tag words** | One per line. Captions and tags use these words when they fit. | Empty |
| **Words allowed for medical things in view** | One per line. Visible things only, such as a cast or a wheelchair. Captions never name a condition. | A built-in list |
| **Words allowed in captions of Locked photos** | One per line. This doesn't change who can see those photos. | A built-in list |
| **Extra instructions** | Anything else captions should follow, in full sentences, up to 2,000 characters | Empty |

Change **Sentences in a caption** or **Writing style** by hand and **Caption length** shows **Custom**.

:::tip[Three lists that sound alike]
- **Things to look out for**: "pay attention to these things when you see them". Short entries.
- **Preferred tag words**: "when you tag one of these, spell it exactly like this". Short entries.
- **Extra instructions**: "follow these rules when you write". Full sentences.
:::

### Settings by library type

| Library | Caption length | Names | Add to Things to look out for | Add to Preferred tag words |
| --- | --- | --- | --- | --- |
| Family | Medium or Detailed | On | birthday cake, sparklers, candles, prom, graduation, recital, soccer ball, baseball glove | golden hour, overcast, dappled light, backyard, park, beach |
| Documents and receipts | Short | Off | total amount, store name, line items, payment method, invoice number, due date, expiry date, signature line | Keep it short |
| Travel | Medium | On | mountain range, beach, hotel lobby, train station, food market, street scene, landmark | The landmarks and cities you visit often |
| Pets | Short or Medium | Either | breed, leash, food bowl, harness, fetch, asleep, swimming, kennel | Your pets' breeds, such as golden retriever or tabby |

For a documents library, also add `account numbers verbatim, full credit card numbers, social security numbers` to **Never guess at**.

For a hobby, teach the model its words through **Preferred tag words**: `long exposure, leading lines, bokeh` for photography, `latte art, sourdough, plating` for cooking, or `gravel, derailleur, peloton` for cycling.

## Extra instructions

**Extra instructions**, under **Advanced: wording lists and extra instructions**, is the easiest way to change how captions are written without replacing the built-in instructions. Treat it like a short brief to a careful assistant: write full sentences, say what to do and when not to, and keep it to a couple of paragraphs. The text is added to the prompt before the output rules, so the model reads it first.

Some examples that work well:

**Vehicles**

```text
If you can clearly see a car, truck, or motorcycle, identify the make and model in
the description (e.g. "a red Tesla Model 3", not "a red car"). If you are uncertain
about either the make or the model, just say "a red car" rather than guessing.
```

**Sports**

```text
When people are clearly playing a sport, name the sport in the description (soccer,
baseball, basketball, tennis, etc.) and mention the visible equipment they're using.
Do not guess the sport from clothing alone, only from visible play, equipment, or
field markings.
```

**Travel**

```text
When the photo is clearly outdoors and looks like travel, identify the location type:
beach, mountain, forest, city street, market, train station, airport, museum, cathedral,
temple, ruins, hotel lobby. If a recognizable landmark is visible, name it. Do not
invent landmarks if you are not sure.
```

**Documents and receipts**

```text
When the photo is a document, receipt, or screenshot of text, transcribe the most
important visible identifiers: store name, total amount, date, invoice number, and
document type. Do not transcribe full account numbers, full credit card numbers,
social security numbers, or any other sensitive personal data even if visible; refer
to them generically (e.g. "account number redacted").
```

**Food**

```text
For food photos: identify the dish by name if recognizable, name the cuisine if you
can tell, and note the plating style (rustic, fine-dining, casual, street-food). Do
not name a dish you are not confident about; say "pasta dish" rather than guessing.
```

**Pets**

```text
For pets: identify the breed when you can recognize it. For dogs in particular, note
the activity (sleeping, playing fetch, swimming, on a leash, in a car). Do not guess
a breed from coat color alone.
```

**Tone**

```text
Keep descriptions plain and factual. Do not use poetic language ("a tapestry of
colors", "bathed in golden light"). Do not editorialize about the people or events
("a joyful family", "an intimate moment"). Stick to what is visible.
```

**A family library, all in one**

```text
Always name every recognized person and avoid generic group terms like "the family"
or "everyone".

If you see a car, name the make and model when recognizable.
If people are playing a sport, name the sport and any visible equipment.
For pets, identify the breed when recognizable.
For travel scenes, name the landmark if you are certain.

Keep descriptions factual and short. Do not use poetic language.
```

### What extra instructions can't do

- They can't override the safety rules. The handling of private and medical content, and the check that removes invented names, still apply.
- They can't add names. A name that isn't confirmed on the photo is still replaced with "Someone".
- They can't change the output format. Use **Write my own instructions** for that.
- They can't make the model remember anything between photos. Each one is described on its own.
- They're ignored while **Write my own instructions** is on.

## Names in captions

**Mention people I have named** passes the named people in each photo to the model. Two numbers under **Advanced: names, and writing your own instructions** tune it:

| Setting | What it does | Default |
| --- | --- | --- |
| **Most names in one caption** | How many people to name in one photo, from 1 to 20 | 5 |
| **How sure before naming someone** | From 0 to 1. Higher names fewer people. | 0.7 |

Lower **Most names in one caption** to 1 or 2 for crowd photos where only the main people matter. Raise it to 10 to 15 for teams, weddings and other big groups. If a photo has more named people than that, some are left out. Fixing a face updates its captions.

## Raw prompt editor

The choices above build the instructions for you, with safe defaults, and cover most needs. Reach for **Write my own instructions**, under **Advanced: names, and writing your own instructions**, when you want something they can't express, such as a different output format. It's for experts: it replaces the built-in caption instructions with your own text.

- Turn it on and **Your instructions** holds the text to edit.
- **Start again from the built-in text** replaces your text with the built-in instructions, discarding your edits.
- Turning it off goes back to the choices on the page.

These placeholders are filled in for each photo:

| Placeholder | Becomes |
| --- | --- |
| `{names}` | The recognised people. Empty when names are off or nobody is recognised. |
| `{schema}` | The output format the model must return. Keep it, or captions can't be read back. |
| `{vocabulary}` | Your preferred tag words |
| `{style_hint}` | The length and tone cue for the chosen caption length |

**If '{schema}' is missing** chooses what happens when you save without it: **Don't save**, the default, or **Warn me**.

**Extra instructions** aren't added to your own text; paste them where you want them. If your text is wrong, captions come out empty or broken. Turn **Write my own instructions** off to recover.

## Describe your library

The status at the top of the page says how captions stand: **Every photo has a caption**, or how many photos have one and how many are waiting.

**Update captions now** opens a window with:

| Field | Shows |
| --- | --- |
| **Total eligible assets** | Everything captions can be written for |
| **Already have a description** | Items with a description |
| **Missing a description** | Items without one |
| **Estimated seconds per asset** | The recent pace |
| **Estimated total time** | How long the whole run should take |
| **Active backend** and **Active model** | Where captions will be written, and with what |

- **Cancel** closes it.
- **Re-queue later** keeps a reminder, so you can make several changes and run once at the end. The page then shows **Caption choices changed since the captions were written**, with **Update captions now** to start.
- **Re-queue now** starts straight away. If a run is already going, Frameleaf tells you instead of starting a second one.

When captions are sent to Frameleaf Cloud, nothing is queued here: Frameleaf Cloud describes photos in batches, and you start them from Frameleaf Cloud processing, where the estimate comes first.

The first estimate is a starting point. Model size, hardware, other work and video length all change the speed, so try a sample first and plan the full library from what you measure.

### What happens to descriptions people wrote

**Descriptions written by hand are kept** says whether a run can replace someone's own words. It follows **Keep what I wrote myself**, in **Settings**, then **Library care**, then **Search and recognition progress**, which is on by default: writing captions again only replaces what Frameleaf wrote. With it off, a description written by hand can be replaced the next time captions are written.

### Describe moments in videos

**Describe moments in videos** is off by default. When it's on, each video also gets captions for a few moments in it, one extra model request per moment. It takes longer, and speech is not written down. For speech, see [Spoken words](https://frameleaf.app/docs/using/spoken-words/).

## Locked-content detection

If **Locked-content detection** is also on, Frameleaf checks each photo for private content first and passes the result to the caption model, so the caption stays factual. See [Private content](https://frameleaf.app/docs/using/private-content/).

## A worked example

Here's how to caption an existing library from start to finish. Let each stage finish before you start the next one.

1. **Prepare faces.** Turn on **Faces & identity** and let it finish. Name your 20 or so most-photographed people; the People page shows them first.
2. **Try captions.** Choose a model that runs on your hardware. Set **Caption length** to **Medium**, add a few things to **Things to look out for** and **Preferred tag words**, paste the family example into **Extra instructions**, and set **Most names in one caption** to 8. Try it on a few photos, then save.
3. **Caption the library.** Choose **Update captions now**. Turn on smart albums, add any extra words, and choose **Update albums now** when captions finish.
4. **Review results.** Open ten random photos and videos. Are names used? Are your instructions followed? Do smart albums make sense? Adjust, rewrite a few from their **Info** panels with **Rewrite with AI**, and repeat until you're happy.
5. **After that.** New photos are captioned automatically. New names appear in new captions straight away; rewrite older ones if you want them updated.

## Troubleshooting

### Captions are still generic with names turned on

1. Check the photo has at least one named face in its People list.
2. Check the model is a Qwen model, not Florence-2.
3. Check **Most names in one caption** is at least the number of named people in the photo.
4. Open the photo's **Info** and choose **Rewrite with AI**. Older captions keep the wording they were written with.

### Extra instructions aren't working

1. Check **Write my own instructions** is off.
2. Check you saved.
3. Rewrite some photos. Existing captions don't change by themselves.
4. Try plainer, more direct wording: "If you see a car, identify the make and model" works better than "try to be more specific about vehicles".
5. Check you're under 2,000 characters.

### A video gets no description

No frames could be cut. The video is very short, longer than three hours, or unreadable. Choose **Find moments** in its **Moments** section to try again. If that fails, check that video processing works for other files and that the file isn't damaged.

### The sample captions could not be written

Check that the computer doing this work is on, then choose **Try it** again. **Open Computers** on the **Search & intelligence** page shows whether it's answering.

### A real name was replaced with "Someone"

The person isn't named on that photo. Name them in its People list and rewrite the caption, or edit the description by hand. There's no separate list of allowed names.
