Physical deduplication

In a family, the same photo often ends up in several people’s libraries: shared holiday pictures, copied phone backups, the same Google Photos export imported twice. Physical deduplication lets the server keep one stored copy of an identical file, while every account keeps its own photo in its own library.

Nothing about ownership changes. Each photo keeps its owner, albums, favourites, descriptions, faces, stacks, shared links, Locked state and permissions. Only the file on disk that a copy points at changes. It isn’t partner sharing, and it doesn’t merge libraries.

This is an administrator tool and it’s off by default. For reviewing duplicates within one library, see Duplicates.

Treat it like a storage migration

Applying a plan deletes redundant files after checking them. Make an independent, verified backup first. A shared original still needs protecting against disk loss.

How it works

You choose a master account, also called the retained account: the account whose originals are kept. Usually this is the household account whose folder layout you want shared files to follow.

• New uploads are handled automatically once the feature is on. After the server has worked out an upload’s checksum, it reuses the master account’s file only if the master account already has an exact copy with the same checksum and size. Otherwise the upload is stored as normal.
• Files already on the server are converted with a reviewed plan, described below.
• Uploads are still checked for duplicates inside the uploader’s own library, as before. Copies across accounts aren’t refused at upload.
• External library files are never moved, linked or deleted.

Turn it on

1. Choose a master account that’s active and won’t be deleted.
2. Make and verify a backup.
3. Open Settings, then Storage & originals, then Originals & folder structure.
4. Turn on Enable physical deduplication and choose the Master account.
5. Select Review changes, then Save changes.

New uploads now share files automatically. To convert your existing library, carry on below.

Convert an existing library

Open Settings, then Storage & originals, then Physical deduplication.

1. Prepare a preview

Choose the scope, all accounts or one account’s copies, and select Prepare preview plan. The preview runs in the background and changes no files. It lists every exact copy next to the retained original it matches, grouped by original, with:

• the checksum (SHA-1 for older uploads, SHA-256 for newer ones) and size of the copy and the original
• how many photos point at the original now, and after the plan
• the decision for each copy: share the retained original, or skip it with a reason, such as external library, no exact copy in the retained account, already shared, or size unavailable

Each plan is named after a fingerprint of its evidence, for example PD-1A2B3C4D. Preparing a new preview replaces the plan, with a new name.

A preview can use another account chosen on the page, but applying always uses the saved master account.

The list holds the first 500 copies. The totals cover everything, but only the listed copies are applied. Afterwards, prepare another plan for the rest; copies already shared are skipped.

Locked media

Another account’s Locked media is never listed. Its copies are counted (for example “2 copies are another account’s Locked media”), included in the plan and checked like any other, but their names, paths and thumbnails are only shown to their owner in an unlocked session.

2. Decide per group

Each group has Share this original in the plan. Clear it to leave that group alone: its copies keep their own files.

3. Mark the plan reviewed

Select Mark plan reviewed. The server checks the plan against the library again, and refuses (changing nothing) when:

• a newer preview replaced the plan, or the plan was already applied, fully or partly
• any copy or original in the plan was removed, trashed, moved to another account, or changed path, checksum or size
• a copy already points at its original
• the original gained or lost references since the preview

If it’s refused, the page shows the current state; prepare a new plan. Changing a group’s decision after the review asks for a new review.

4. Apply the reviewed plan

Select Apply reviewed plan. The confirmation shows the plan’s name, scope, master account, the number of copies to share, the estimated space saved and the groups left as they are. Type APPLY and the plan’s name exactly.

The server repeats every review check, and also refuses when:

• another plan is being applied, by anyone
• the saved master account changed, or the feature was turned off
• the decisions differ from the reviewed ones, or the review is no longer valid. Reviews don’t survive a server restart, so review again

Only one plan is applied at a time.

While a plan is applied

Applying is a background job. It’s listed in Activity and the notifications panel of the administrator who applied it, and on the Physical deduplication page for every administrator. The administrator who applied it can pause, resume or stop it. It survives closing the browser and restarting the server, and carries on from the last copy it finished.

Each time it starts or resumes, it checks that the person who applied it is still an administrator, that the feature is still on and that the saved master account is still the plan’s. Otherwise it stops.

For each copy, in order, the job:

1. reads the copy and its original again (owner, path, checksum, size and state), and leaves the copy alone if anything differs from the reviewed evidence
2. checks that the original’s file is on disk and still has the reviewed checksum and size. Without that, the copy is never removed
3. checks that the copy’s file still holds the reviewed bytes
4. moves the copy’s XMP sidecar to the copy’s own upload folder, points the copy at the original, and only then removes the copy’s old file, and only while nothing else uses it
5. points the copy’s thumbnail, preview, full-size image and playback video at the original’s, and removes its own once nothing uses them

Originals are never written to, and a retained original is never deleted. No photo records are removed or merged, so albums, faces, stacks, shared links and Locked records keep pointing at the same photos.

The job records what happened to every copy (shared, already shared, left as it was, failed) and the space actually freed. A failed copy is retried once after a short wait; a copy left alone because its evidence changed isn’t retried. A job that fails as a whole is retried once automatically, and running a copy again after an interruption finishes what was started without applying it twice.

As soon as the job changes a file, the plan reads Applied, or Partly applied if the job then stops or fails. Such a plan can’t be reviewed or applied again. To carry on, prepare a new plan, which skips copies already shared, then review and apply it.

Review history lists recent plans with who applied them and their counts.

Workers

Physical deduplication only runs on your server’s own workers. A render worker can only take Studio exports and previews, restorations and quick edits, never this job. See Workers and where jobs run.