Heratio Help Center article. Category: AI & Automation.

Duplicate Detection

User Guide

Identify and manage duplicate archival records to maintain data quality and reduce redundancy in your archive.

Overview

What Gets Detected

Duplicate Detection Methods

How to Access

The Dashboard

When you open Duplicate Detection, you will see the main dashboard.

Statistics Cards

What Each Status Means

Reviewing Duplicates

Step 1: Open Browse View

Go to Admin > Duplicate Detection > Browse

Step 2: Filter the List

• FILTER OPTIONS
• Status: [Pending v]
• Method: [All Methods v]
• Minimum Score: [0.75 v]
• [Apply Filters]

Step 3: Review Results

You will see a table of detected duplicates:

Understanding Similarity Scores

Comparing Records

Step 1: Click Compare

From the browse list, click the Compare button for any duplicate pair.

Step 2: Review Side-by-Side

Fields that differ are highlighted in yellow for easy identification.

Step 3: Make a Decision

• Merge: Combine the records if they are true duplicates
• Dismiss: Mark as false positive if they are distinct records

Merging Duplicates

Step 1: Select Primary Record

Choose which record to keep as the primary:

Step 2: Review Merge Actions

The system will show what will happen:

• MERGE ACTIONS
• • Transfer 3 digital objects from Record B to Record A
• • Move 5 child records from Record B to Record A
• • Redirect slug 'minutes-85' to 'meeting-minutes-1985'
• • Create merge log entry for audit trail
• • Archive Record B (not deleted)

Step 3: Confirm and Merge

Check the confirmation box and click Merge Records.

Warning: Merging is permanent. The secondary record will be archived but cannot be easily restored.

Dismissing False Positives

If two records are not actually duplicates:

1. Click Dismiss from the browse list or compare view
2. Optionally add a note explaining why
3. The pair will be marked as "dismissed" and won't appear in pending

Running a New Scan

From the Web Interface

1. Go to Admin > Duplicate Detection > New Scan
2. Select a repository (or scan all)
3. Click Start Scan
4. The scan job is queued for background processing

From the Command Line

# Scan a specific repository
php artisan ahg:dedupe-scan --repository=1

# Scan entire system
php artisan ahg:dedupe-scan

# Limit records scanned
php artisan ahg:dedupe-scan --limit=1000

Scan Progress

• SCAN PROGRESS
• Status: Running
• Progress: 2,450 / 5,000 records (49%)
• Duplicates: 34 found so far
• Started: 10 Jan 2026, 14:30

Detection Rules

Viewing Rules

Go to Admin > Duplicate Detection > Rules

Default Rules

Creating a Custom Rule

1. Click Create Rule
2. Configure the rule:

• CREATE DETECTION RULE
• Name: [My Custom Rule ]
• Type: [Title Similarity v]
• Threshold: [0.80] (0.0 to 1.0)
• Repository: [All Repositories v]
• Priority: [100]
• Blocking (prevent save if match found)
• Enabled
• [Save Rule]

Blocking Rules

When a rule is set as "blocking":

• Users cannot save a record if a match is found
• Used for strict identifier uniqueness enforcement
• Shows warning during data entry

Real-Time Duplicate Checking

When creating new records, the system can check for duplicates in real-time:

Reports and Analytics

Accessing Reports

Go to Admin > Duplicate Detection > Reports

Available Reports

1. Monthly Statistics

   • Duplicates detected per month
   • Merged vs dismissed ratio
   • Detection method effectiveness

2. Top Duplicate Clusters

   • Records with most duplicate pairs
   • Helps identify systematic issues

3. Efficiency Metrics

   • Total detected vs reviewed
   • False positive rate
   • Average merge score

Exporting Reports

# Export to JSON (redirect to a file)
php artisan ahg:dedupe-report --format=json > duplicates.json

# Filter by scan status
php artisan ahg:dedupe-report --status=pending --format=json

# Show as a table (default), limit rows
php artisan ahg:dedupe-report --format=table --limit=20

Best Practices

CLI Commands Reference

Scanning

# Scan specific repository
php artisan ahg:dedupe-scan --repository=1

# Scan entire system
php artisan ahg:dedupe-scan

# Limit records
php artisan ahg:dedupe-scan --limit=5000

Merging

# Merge a duplicate pair (keep the winner, redirect the loser)
php artisan ahg:dedupe-merge --keep=456 --remove=123

# Preview the merge without writing
php artisan ahg:dedupe-merge --keep=456 --remove=123 --dry-run

# Merge every approved candidate from a scan run
php artisan ahg:dedupe-merge --scan-id=42

Reporting

# Show pending scan runs
php artisan ahg:dedupe-report --status=pending

# Limit rows returned
php artisan ahg:dedupe-report --limit=20

# Output as JSON (redirect to a file)
php artisan ahg:dedupe-report --format=json > report.json

Troubleshooting

No Duplicates Found After Scan

• Check if detection rules are enabled
• Verify threshold settings (may be too high)
• Ensure records have titles/identifiers to compare

Too Many False Positives

• Increase threshold values in rules
• Disable overly broad rules
• Use combined analysis instead of single-factor rules

Scan Taking Too Long

• Scan specific repositories instead of entire system
• Use --limit option for large archives
• Run scans during off-peak hours

Merge Failed

• Check if either record has been deleted
• Verify you have administrator permissions
• Check for database errors in logs

Workflow Integration

After Bulk Import

Regular Maintenance

  Weekly: Review pending duplicates
  Monthly: Run full system scan
  Quarterly: Review and tune detection rules

Need Help?

Contact your system administrator if you experience issues or need to modify detection rules.

Part of the Heratio AHG Framework