Skip to content

How face recognition works

Face recognition: people, fixes and undo, on your own server Watch on YouTube

Frameleaf finds the faces in your photos and groups them into people, all on your own server. This page shows what happens behind the People page, and how administrators can tune it for cleaner groups.

Face detection finds faces. Frameleaf reads each photo’s unedited preview, so a face’s box stays in the right place on the picture however you crop or rotate it later. The model is downloaded the first time it’s needed. A detection model draws a box around each face and gives it a score. Each face is then cropped and passed to a recognition model, which turns it into a set of numbers describing what the face looks like. Those numbers are stored in your database and indexed so similar faces can be found quickly.

Facial recognition groups faces into people. It runs once detection has finished.

For videos, only the thumbnail is checked for faces.

Think of every face as a point on a map, where similar faces sit close together. For each face, Frameleaf looks for other faces within a certain distance, set by How alike two faces must be.

  1. If any of those nearby faces already belongs to a person, the face joins the person of the most similar face.
  2. If none does, Frameleaf counts how many similar faces there are, including this one. If there are at least as many as Photos needed before someone appears (three by default), the face starts a new person. Faces processed later that are close to it join that person.
  3. If there aren’t enough, no person is created yet. The face waits until all the other faces in the batch have been processed. If one of its matches has gained a person by then, it joins that person. If not, it’s treated as an outlier, such as a stranger in the background.

A face that has enough similar faces to start a person is called a core point. A face that doesn’t can still join a person through a core point, but can’t extend a person on its own. This stops loosely related faces from chaining two different people together.

Grouping works on the faces waiting in the queue at the time, and keeps the people it already made. So the more faces in one batch, the better the groups. Recognition waits for detection and thumbnails to finish before it starts, for this reason.

If you uploaded in small batches, or the server kept up with your uploads, the groups may not be as good as they could be. Running recognition again for all faces regroups everything at once and gives the best result.

To keep up with new photos, Frameleaf only looks for a handful of the closest matches for each face, rather than every face within the distance. Each night, Group new faces (part of the Empty Trash and tidy up step of the overnight schedule) then finds unassigned faces that now have matches. The result is very close to grouping everything at once.

Run these from Settings, then Computers & background work, then Background work:

Job Button Does
Finding faces Catch up Finds faces in photos that haven’t been checked yet
Finding faces Do it all again Checks every photo again
Recognising people Catch up Groups faces that don’t have a person yet
Recognising people Start again from scratch Groups every face again from scratch

Recognition settings are in Settings, then Search & intelligence, then Faces & identity.

Setting What it does Default
Recognise people Groups photos of the same person so you can name them. Turned off, people you have named stay named, and new photos are not added to them. On
How carefully to group Fewer mistakes (someone appears after 5 clear photos; the same person may show up twice and need merging), Balanced (someone appears after 3 photos; a few look-alikes may need splitting) or Group more photos (someone appears after 2 photos, including blurry and side-on faces) Balanced
Recognition quality The face model, on a slider buffalo_l
Use people tags saved by other apps If Lightroom, digiKam or another app saved who is in a photo, shows those people here. It applies to photos added from then on; to include the photos you already have, choose Do it all again on Reading photo details in Background work. Off
Save people’s names in XMP sidecars Saves names and face positions beside photos so other photo apps can read them. See XMP sidecars. On

How carefully to group sets three numbers, which you can also change by hand under Advanced: face model and numbers:

Setting What it does Default
How alike two faces must be From 0.1 to 2. Lower is stricter and makes fewer wrong matches. 0.5
Face clarity needed From 0.1 to 1. Higher ignores blurry or tiny faces in the background. 0.7
Photos needed before someone appears 1 or more. Fewer shows people sooner, including some you may not know. 3

Small changes work best. If you make one setting stricter, relax another a little to balance it, and the other way round.

Every “same person” and “different person” choice you make in People, and every name you give, is kept when faces are checked or grouped again.

Change How carefully to group, or one of its numbers, and the page says Saving this changes how faces are grouped, with how many photos and videos are involved and about how long it takes. New photos follow the new choice straight away. Choose Save and start now to group the faces you already have again at once, or Save and start tonight to leave it for the overnight schedule. People pages update as it goes.

The models are buffalo_s (small, fastest), buffalo_m (medium), buffalo_l (large) and antelopev2 (largest). Larger models are slower and use more memory but give better results. The default is usually the best choice; pick a smaller one if your server struggles. The Recognition quality slider also shows, in blue, when Frameleaf Cloud can run your server’s own face model.

Changing the face model means every photo and video is read again to find faces. The page says so before you save, with how long it takes, and Save and start now or Save and start tonight starts the work for you. Until it finishes, new photos may not match the people you’ve already named. Your corrections are kept.

Setting this low finds more faces, but also things that aren’t faces, which are hard to clean up and spoil the groups. Stay at 0.5 or above. Very high values like 0.9 miss real faces, because the default already leans towards being sure.

A change applies to faces found from then on. To apply it to your whole library, choose Do it all again on Finding faces in Background work.

The default suits most libraries. Lower it if you have twins or people who look very alike. A value that’s too low only means merging duplicate people afterwards, but one that’s too high can produce groups that can’t be untangled. Stay between 0.3 and 0.7.

This setting does two things:

  • It takes effect straight away: people with fewer faces than this are hidden from the list.
  • It makes grouping more robust, because a person needs a certain number of similar faces before it exists.

Raise it if you loosen How alike two faces must be or lower Face clarity needed. Setting it to 1 turns off the core point idea entirely, which suits you if you’d rather do more by hand.

This method suits a large library after you’ve imported most of it. It builds the first groups from only the most photographed faces, then works down.

  1. Go to Settings, then Search & intelligence, then Faces & identity, then Advanced: face model and numbers.
  2. Optionally, lower How alike two faces must be, for example to 0.4, if your library has people who look alike.
  3. Set Photos needed before someone appears high: about 20 for a library of 100,000 photos or more, or about 10 for around 40,000. Only faces that appear that often can start a person, which gives clean first groups.
  4. Go to Settings, then Computers & background work, then Background work, and choose Start again from scratch on Recognising people. Wait for it to finish.
  5. Lower Photos needed before someone appears: to 10 for a large library, or to 5 for a medium one. Choose Catch up on Recognising people.
  6. Lower Photos needed before someone appears to 3 and choose Catch up on Recognising people again.

Named people can appear by name in AI descriptions. Only named people who aren’t hidden are used, and correcting a face marks the descriptions that used the old name as out of date.

A photo a partner shares with you is your own copy. Frameleaf shares the faces and search results already found on the original with that copy, so the same photo isn’t read twice.

Faces are found from photo previews on your server, or on a computer at home that your administrator has added, and your server groups the results into people. An administrator can also choose Frameleaf Cloud for Faces & identity: the Cloud then finds faces in a metadata-stripped preview of each photo with the same face model your server uses, and sends them back. It doesn’t identify anyone. Grouping faces into people and naming them happen only on your server. See Search and faces on Frameleaf Cloud. If you choose optional Cloud services such as Cloud descriptions or restoration, the photos you send can include faces. People who appear only in Locked photos are hidden until you unlock.