Information estates
Why duplicate deletion is the wrong starting point for file rationalisation
Duplicate reports make deletion look like easy progress. Without ownership, context and retention wrapped around them, they are a list of ways to hurt yourself quickly.
Every estate scan produces a duplicate report, and every duplicate report produces the same reaction. The percentage looks like instant savings, the tooling offers to fix it, and someone proposes a deletion run as the obvious first win of the rationalisation.
Duplicates are a symptom of how the estate grew: teams copying what they could not share, permissions worked around, projects archived by copying everything one more time. Deleting the symptom first destroys the information you need to treat the cause, and it commonly does so in bulk.
What a hash match does not tell you
Content-identical files are the easy part to detect and the hard part to decide about. A hash match cannot tell you:
- Which copy is the record and which is the convenience.
- Who owns each location, and whether those owners agree.
- Why the copy exists: an offline habit, a permissions workaround, a team boundary.
- What retention applies in each location, which is a property of the place as much as the file.
- What links, shortcuts and processes point at each copy and would break with it.
Two identical files are not one file twice. They are two locations with two histories and, often, two owners.
“A duplicate report is a map of how the estate grew. Read it before you start cutting.”
Retention lives in places, not just files
Retention and governance are commonly properties of the location: this library keeps records for the regulated period, that team share does not. Two identical files in those two places are not equivalent, whatever the hash says. Deleting the copy in the governed location and keeping the convenient one preserves the bytes and destroys the compliance position.
This is why duplicate decisions cannot be made from a report alone. The decision needs the map of which locations carry which duties, and that map is a product of classification, not of scanning.
Why the copies exist
Duplication is rarely carelessness. In many estates the copies were rational answers to real friction:
- Sharing that was too awkward, so people copied instead of linking.
- Permissions that blocked access, worked around by duplicating into a reachable place.
- Offline habits from years when the network could not be trusted.
- Project snapshots taken as insurance at every milestone.
- Acquisition and migration imports that duplicated whole trees wholesale.
Each reason predicts the right outcome. A permissions workaround is fixed by fixing the permissions, then consolidating. A project snapshot is archive material. An acquisition import needs ownership before anything else. Delete first and every one of those signals is gone.
Context first, then the duplicate decision
Duplicate handling belongs inside classification and target mapping, where ownership and retention are already being established. Once the context exists, most duplicate cases resolve into one of a small number of shapes.
Duplicate cases and what to do with them
- Identical content, one clear owner, one governed location
- Consolidate to the governed copy and redirect the rest, with the owner's agreement recorded.
- Identical content, different owners or teams
- Not a deletion decision yet. Agree the record location between the owners, then consolidate.
- Same name, different content
- Not duplicates at all. Treat as versions and resolve through ownership, not through tooling.
- Duplicates inside archive or retention scope
- Archive under the retention rule. Disposal is a governance decision that leaves a record.
- Copies whose owner has left
- Route through discovery first. Unknown ownership is the risk; the duplication is incidental.
Where this sits in a rationalisation
The order that holds up is the same one that makes migrations safe.
Rationalisation in the right order
- 01Inventory read-onlyContent, age, ownership, access and duplication captured without changing anything.
- 02ClassifyCurrent, stale, duplicate, sensitive, unknown. Duplication is one signal among several, not a category of its own.
- 03Owners decideRetain, consolidate, archive or exclude, chosen by the people who own the content and the risk.
- 04Map into the targetConsolidations and redirects recorded in the source-to-target map, so nothing resolves into a gap.
- 05Execute with sign-offApproved actions only, with the record kept. Disposal happens on evidence, not on a report.
What a good consolidation record shows
Consolidation done properly leaves a record that survives scrutiny. For each duplicate group it should show:
AI changes the analysis, not the decision
AI-assisted classification makes duplicate and staleness analysis affordable at estate scale, which is exactly why the discipline matters more, not less. The analysis widens; the decision stays with owners. I have written about where AI genuinely helps in platform engineering, and read-only discovery with human approval is the operating model throughout.
The file migration readiness checklist treats duplicate handling as part of classification and mapping, which is where it belongs.
