Why Does a Dataset Contain Duplicate Features?

A dataset can contain duplicate-looking features for very different reasons: some are accidental copies, while others represent repeated observations, multiple records attached to one place, overlapping source versions or one-to-many joins.

Deleting duplicates safely therefore begins with a definition: duplicate in which fields, at which time, and with what geometric tolerance?

Exact duplicates are the simplest case

Two records are exact duplicates when the attributes and geometry are identical under the chosen comparison.

These often come from repeated imports, concatenating the same file twice or an upstream system emitting duplicate rows. Even here, check for a stable record identifier and timestamp before deleting, because two identical geometries might represent separate observations that happen to share the same current values.

Near-duplicates require a tolerance and a reason

Features may differ only slightly in coordinates or geometry.

Examples include:

two GPS observations of the same object;
the same building digitised from different imagery;
slightly different geocoding results for the same address;
boundaries from two dataset versions.

Those records are not exact duplicates, although a spatial tolerance can group them as possible matches. That tolerance must be justified by the positional accuracy and intended use.

A 5-metre threshold that makes sense for handheld GPS observations may be inappropriate for parcel boundaries.

Joins can create duplicates without duplicating the source

A one-to-many join repeats the geometry once for every matching table record.

Suppose one district polygon joins to twelve monthly observations. The output may contain twelve rows with the same polygon and different month values, intentionally repeating the geometry because the table relationship is one-to-many.

If you expected one row per district, the issue is not “duplicate polygons”; it is the data model. You may need aggregation, a separate related table or a composite key.

Before deduplicating, compare the source count with the post-join count and inspect the fields that differ between repeated geometries.

Time can make repetition legitimate

A dataset may contain one feature per event, survey visit or version. A school observed in 2024 and the same school observed in 2025 can therefore have identical geometry but represent different temporal records, and removing one because the coordinates match would destroy time-series information.

Include time fields in the duplicate definition when the dataset is longitudinal.

Multiple sources can overlap

Merged datasets often contain the same real-world feature from several providers.

The records may have different identifiers, names, attribute completeness and geometry quality, making their resolution a conflation task rather than a simple DISTINCT operation.

You may need rules for:

source priority;
attribute reconciliation;
spatial tolerance;
identifier crosswalks;
provenance preservation.

Keep the original source identifier even after choosing a canonical record so the merge remains traceable.

Diagnose duplicates with progressively stronger tests

Start with the least assumptive comparison:

duplicate stable identifier;
duplicate full attribute row;
identical geometry;
same identifier but different geometry;
spatially near features with similar names/attributes;
repeated geometry after a join.

Each test answers a different question.

Observation · Likely interpretation

Same ID, same geometry, same attributes — Accidental duplicate likely
Same ID, different date — Repeated observation/version
Same geometry, different joined fields — One-to-many relationship
Different IDs, near-identical geometry — Multi-source or matching problem
Same name, distant geometry — Name ambiguity rather than duplicate

Do not delete before understanding the key

The most dangerous deduplication rule is often “remove identical geometries”. Geometry alone may not define record identity.

State the intended entity first: one row per building? one row per observation? one row per administrative unit and year? Once that key is explicit, duplicate detection becomes testable.

After deduplication, verify that:

expected row counts changed by the intended amount;
unique identifiers remain unique where they should;
no time periods disappeared;
joins still return expected matches;
provenance of merged records is retained.

“Duplicate” describes several data conditions

Exact copies, repeated observations, join multiplication and multi-source overlap can all look like duplicates on a map, but they are not interchangeable errors.

The right fix follows from the dataset's unit of observation and stable key. Identify those first; delete records only when you can explain why two rows represent the same thing in the same state.