# How face recognition works

Source: https://frameleaf.app/docs/using/people-recognition/

Frameleaf finds the faces in your photos and groups them into people, all on your own server. This page shows what happens behind the [People](https://frameleaf.app/docs/using/people/) page, and how administrators can tune it for cleaner groups.

## Two steps: detection, then recognition

**Face detection** finds faces. Frameleaf reads each photo's unedited preview, so a face's box stays in the right place on the picture however you crop or rotate it later. The model is downloaded the first time it's needed. A detection model draws a box around each face and gives it a score. Each face is then cropped and passed to a recognition model, which turns it into a set of numbers describing what the face looks like. Those numbers are stored in your database and indexed so similar faces can be found quickly.

**Facial recognition** groups faces into people. It runs once detection has finished.

For videos, only the thumbnail is checked for faces.

## How faces are grouped

Think of every face as a point on a map, where similar faces sit close together. For each face, Frameleaf looks for other faces within a certain distance, set by **How alike two faces must be**.

1. If any of those nearby faces already belongs to a person, the face joins the person of the most similar face.
2. If none does, Frameleaf counts how many similar faces there are, including this one. If there are at least as many as **Photos needed before someone appears** (three by default), the face starts a new person. Faces processed later that are close to it join that person.
3. If there aren't enough, no person is created yet. The face waits until all the other faces in the batch have been processed. If one of its matches has gained a person by then, it joins that person. If not, it's treated as an outlier, such as a stranger in the background.

A face that has enough similar faces to start a person is called a *core point*. A face that doesn't can still join a person through a core point, but can't extend a person on its own. This stops loosely related faces from chaining two different people together.

### Why upload order matters

Grouping works on the faces waiting in the queue at the time, and keeps the people it already made. So the more faces in one batch, the better the groups. Recognition waits for detection and thumbnails to finish before it starts, for this reason.

If you uploaded in small batches, or the server kept up with your uploads, the groups may not be as good as they could be. Running recognition again for **all** faces regroups everything at once and gives the best result.

To keep up with new photos, Frameleaf only looks for a handful of the closest matches for each face, rather than every face within the distance. Each night, **Group new faces** (part of the **Empty Trash and tidy up** step of the [overnight schedule](https://frameleaf.app/docs/admin/jobs/#nightly-tasks)) then finds unassigned faces that now have matches. The result is very close to grouping everything at once.

:::tip
If you haven't put work into naming yet, group every face again for the best starting point. If you have, name or assign a selection of unassigned faces by hand, then choose **Catch up** on **Recognising people**. Frameleaf learns from your choices and assigns more faces automatically.
:::

## Jobs

Run these from **Settings**, then **Computers & background work**, then **Background work**:

| Job | Button | Does |
| --- | --- | --- |
| **Finding faces** | **Catch up** | Finds faces in photos that haven't been checked yet |
| **Finding faces** | **Do it all again** | Checks every photo again |
| **Recognising people** | **Catch up** | Groups faces that don't have a person yet |
| **Recognising people** | **Start again from scratch** | Groups every face again from scratch |

:::caution
**Start again from scratch** on **Recognising people** resets the groups Frameleaf made and works through the whole library again, so who is in which group can change. Read the warning it shows, and [back up your database](https://frameleaf.app/docs/admin/backup-and-restore/) first if you've put work into People.
:::

## Settings

Recognition settings are in **Settings**, then **Search & intelligence**, then **Faces & identity**.

| Setting | What it does | Default |
| --- | --- | --- |
| **Recognise people** | Groups photos of the same person so you can name them. Turned off, people you have named stay named, and new photos are not added to them. | On |
| **How carefully to group** | **Fewer mistakes** (someone appears after 5 clear photos; the same person may show up twice and need merging), **Balanced** (someone appears after 3 photos; a few look-alikes may need splitting) or **Group more photos** (someone appears after 2 photos, including blurry and side-on faces) | Balanced |
| **Recognition quality** | The face model, on a slider | `buffalo_l` |
| **Use people tags saved by other apps** | If Lightroom, digiKam or another app saved who is in a photo, shows those people here. It applies to photos added from then on; to include the photos you already have, choose **Do it all again** on **Reading photo details** in **Background work**. | Off |
| **Save people's names in XMP sidecars** | Saves names and face positions beside photos so other photo apps can read them. See [XMP sidecars](https://frameleaf.app/docs/using/xmp-sidecars/). | On |

**How carefully to group** sets three numbers, which you can also change by hand under **Advanced: face model and numbers**:

| Setting | What it does | Default |
| --- | --- | --- |
| **How alike two faces must be** | From 0.1 to 2. Lower is stricter and makes fewer wrong matches. | 0.5 |
| **Face clarity needed** | From 0.1 to 1. Higher ignores blurry or tiny faces in the background. | 0.7 |
| **Photos needed before someone appears** | 1 or more. Fewer shows people sooner, including some you may not know. | 3 |

Small changes work best. If you make one setting stricter, relax another a little to balance it, and the other way round.

### Your corrections always win

Every "same person" and "different person" choice you make in [People](https://frameleaf.app/docs/using/people/), and every name you give, is kept when faces are checked or grouped again.

### When a change takes effect

Change **How carefully to group**, or one of its numbers, and the page says **Saving this changes how faces are grouped**, with how many photos and videos are involved and about how long it takes. New photos follow the new choice straight away. Choose **Save and start now** to group the faces you already have again at once, or **Save and start tonight** to leave it for the overnight schedule. People pages update as it goes.

### Face model

The models are `buffalo_s` (small, fastest), `buffalo_m` (medium), `buffalo_l` (large) and `antelopev2` (largest). Larger models are slower and use more memory but give better results. The default is usually the best choice; pick a smaller one if your server struggles. The **Recognition quality** slider also shows, in blue, when Frameleaf Cloud can run your server's own face model.

Changing the face model means every photo and video is read again to find faces. The page says so before you save, with how long it takes, and **Save and start now** or **Save and start tonight** starts the work for you. Until it finishes, new photos may not match the people you've already named. Your corrections are kept.

### Face clarity needed

Setting this low finds more faces, but also things that aren't faces, which are hard to clean up and spoil the groups. Stay at 0.5 or above. Very high values like 0.9 miss real faces, because the default already leans towards being sure.

A change applies to faces found from then on. To apply it to your whole library, choose **Do it all again** on **Finding faces** in **Background work**.

### How alike two faces must be

The default suits most libraries. Lower it if you have twins or people who look very alike. A value that's too low only means merging duplicate people afterwards, but one that's too high can produce groups that can't be untangled. Stay between 0.3 and 0.7.

### Photos needed before someone appears

This setting does two things:

- It takes effect straight away: people with fewer faces than this are hidden from the list.
- It makes grouping more robust, because a person needs a certain number of similar faces before it exists.

Raise it if you loosen **How alike two faces must be** or lower **Face clarity needed**. Setting it to 1 turns off the core point idea entirely, which suits you if you'd rather do more by hand.

## Get better groups in a large library

This method suits a large library after you've imported most of it. It builds the first groups from only the most photographed faces, then works down.

:::caution
This starts the face groups again from scratch, so do it before you put work into naming people, and [back up your database](https://frameleaf.app/docs/admin/backup-and-restore/) before you start.
:::

1. Go to **Settings**, then **Search & intelligence**, then **Faces & identity**, then **Advanced: face model and numbers**.
2. Optionally, lower **How alike two faces must be**, for example to 0.4, if your library has people who look alike.
3. Set **Photos needed before someone appears** high: about 20 for a library of 100,000 photos or more, or about 10 for around 40,000. Only faces that appear that often can start a person, which gives clean first groups.
4. Go to **Settings**, then **Computers & background work**, then **Background work**, and choose **Start again from scratch** on **Recognising people**. Wait for it to finish.
5. Lower **Photos needed before someone appears**: to 10 for a large library, or to 5 for a medium one. Choose **Catch up** on **Recognising people**.
6. Lower **Photos needed before someone appears** to 3 and choose **Catch up** on **Recognising people** again.

:::tip[Try different values]
The best numbers depend on your library. Try a few before you start naming people, and keep what works best.
:::

## Names in AI descriptions

Named people can appear by name in [AI descriptions](https://frameleaf.app/docs/using/ai-descriptions/#names-in-descriptions). Only named people who aren't hidden are used, and correcting a face marks the descriptions that used the old name as out of date.

## Partner copies

A photo a partner shares with you is your own copy. Frameleaf shares the faces and search results already found on the original with that copy, so the same photo isn't read twice.

## Privacy

Faces are found from photo previews on your server, or on a computer at home that your administrator has added, and your server groups the results into people. An administrator can also choose Frameleaf Cloud for **Faces & identity**: the Cloud then finds faces in a metadata-stripped preview of each photo with the same face model your server uses, and sends them back. It doesn't identify anyone. Grouping faces into people and naming them happen only on your server. See [Search and faces on Frameleaf Cloud](https://frameleaf.app/docs/admin/workers/#search-and-faces-on-frameleaf-cloud). If you choose optional Cloud services such as Cloud descriptions or restoration, the photos you send can include faces. People who appear only in [Locked](https://frameleaf.app/docs/using/private-content/) photos are hidden until you unlock.
