العودة إلى User Guides
المحتويات
User Guides
2,141 كلمة
Nov 13, 2026

Heratio Help Center article. Category: Import/Export.

Data Ingestion Manager

User Guide

OAIS-aligned multi-stage ingestion pipeline for batch import of archival records and digital objects. A 6-step wizard that sits between AtoM's basic CSV import and Archivematica's full OAIS pipeline: Configure, Upload, Map & Enrich, Validate, Preview, Commit.

Scanner / watched-folder capture uses this same engine through a different entry point. For continuous capture from scanning stations — where files arrive one at a time instead of in one big batch — see the Scanner / Capture user guide. The scanner reuses the ingest session, derivatives, virus scan, OCR, and SIP/AIP/DIP packaging you configure here — you do not need to re-learn anything.

Commit runner (delivered 2026-04-24)

The wizard's Commit step now runs a real commit: IngestCommitRunner walks every valid ingest_row, creates the information object (via InformationObjectService::create()), attaches any mapped digital object file (via IngestService::ingestFile()), and if the session has SIP/AIP/DIP flags set it invokes OaisPackagerService per IO. Progress is tracked in ingest_job and rendered on the commit page.

Invocation paths:

  • Web UI: the "Start Commit" button on the commit page posts to /ingest/{id}/commit which triggers the runner synchronously.
  • CLI: php artisan ahg:ingest-commit <session_id> — useful for long batches or scripted workflows.

For very large batches, the web commit path automatically dispatches to the Laravel queue worker when the session's valid-row count is at or above heratio.ingest.queue_threshold (default 500, configurable via HERATIO_INGEST_QUEUE_THRESHOLD). The commit view polls the seeded ingest_job row for progress — the UI behaves identically whether the run is sync or queued. If your deployment doesn't run a queue worker, set the threshold very high (or to 0 — which disables the queue path entirely) to force sync, and use php artisan ahg:ingest-commit <session_id> on the CLI for long-running batches.

Overview

INGESTIONMANAGERahgIngestPluginv1.0.0Step1Step2Step3Step4ConfigureUploadMap&ValidateEnrichStep6Step5CommitPreview&Report&ApproveSESSIONDASHBOARD(/ingest)ViewallsessionsRollbackDownloadmanifests

Key Features

Ingestion Capabilities

Item Description
[Wizard] 6-Step Pipeline Guided wizard from config
to commit with rollback
[Upload] Multi-Format CSV, ZIP (with DOs), EAD XML
[Map] Auto-Mapping Confidence-based field
matching with saved profiles
[Check] Validation Required fields, dates,
hierarchy, duplicates
[Tree] Preview Hierarchical tree view with
expand/collapse + approval
[Package] OAIS Packaging SIP, AIP, DIP generation
[Progress] Live Progress AJAX-polled progress bar
[Undo] Rollback Undo entire committed batch
[Hash] Checksums SHA-256 integrity hashing
[AI] AI Processing NER, OCR, virus scan,
summarize, spellcheck, more

How to Access

MainMenuvAdmin/ImportAtoMCSVImport(basic)vIngestionManager(/ingest)SessionDashboard(allsessions)[NewIngestion]6-StepWizardDownloadCSVTemplatesViewCompletedManifests

Direct URL: /ingest

Dependencies: ahgCorePlugin, ahgSecurityClearancePlugin

Required Role: Administrator or user with ingestion permissions

Session Dashboard

When you open the Ingestion Manager, you see all ingestion sessions:

INGESTION MANAGER - SESSION DASHBOARD
[+ New Ingestion] [Download CSV Templates v]
ID Title Sector Status Records Actions
ING-5 2026 Photo archive commit 1,240 [Manifest]
Collection -ted [Rollback]
ING-4 Museum museum in 380 [Resume]
Catalogue progr. [Cancel]
ING-3 Library library commit 2,500 [Manifest]
Batch Import -ted [Rollback]
ING-2 Gallery gallery rolled 150 [View]
Artworks back
ING-1 DAM Media dam cancel --- [View]
Upload -led

Dashboard Actions

Action Description
New Ingestion Start a new 6-step wizard session
Resume Continue an in-progress session at the last step
Cancel Abort an in-progress session
Manifest Download CSV with created AtoM record IDs
Rollback Undo all records created by a committed session
Download CSV Templates Get blank CSV templates per GLAM sector

Step 1: Configure

Set up the parameters for your ingestion session.

Step 1.1: Open a New Ingestion

Click [+ New Ingestion] on the dashboard or navigate to /ingest/configure

Step 1.2: Fill In Session Details

STEP1OF6:CONFIGURE[1Configure][2Upload][3Map][4Validate][5Preview][6Commit]SessionTitle2026PhotoCollectionBatchImportGLAMSector(o)Archive()Museum()Library()Gallery()DAM(DigitalAssetManagement)DescriptiveStandard[ISAD(G)v]Options:ISAD(G),DublinCore,CCO,CCO,RAD,DACSRepository[NationalArchives-NARSSAv]

Step 1.3: Configure Parent Placement

PARENTPLACEMENTWhereshouldimportedrecordsbeplaced?(o)Existinghierarchy-attachtoexistingrecordParentrecord:[Searchorbrowse...]()Newparent-createanewparentrecordParenttitle:[]()Top-level-createastop-levelfonds()CSVhierarchy-useparentId/legacyIdcolumnstobuildhierarchyfromCSVdata

Step 1.4: Configure Output Options

OUTPUTOPTIONSRecordCreation[X]CreateAtoMrecords[]GenerateSIPpackage(Submission)[]GenerateAIPpackage(Archival)[]GenerateDIPpackage(Dissemination)DerivativeOptions[X]Generatethumbnails[X]Generatereferenceimages[]NormalizetopreservationformatSecurityClassification(viaSecurityClearancePlugin)[Unclassifiedv]Options:Unclassified,Restricted,Confidential,Secret,TopSecret

Step 1.5: Configure AI Processing (Optional)

AIPROCESSING(requiresahgAIPlugin)[]NER-NamedEntityRecognition(persons,places)[]OCR-OpticalCharacterRecognition[]VirusScan-Checkuploadedfiles[]Summarize-Auto-generatescope&content[]Spellcheck-Checkspelling/grammar[]Translate-Translatemetadatafields[]FormatIdentification-Identifyfileformats[]FaceDetection-DetectfacesinimagesNote:AIprocessingrunsduringtheCommitstepandmayincreaseprocessingtimesignificantlyforlargebatches.[Cancel][Next:Upload]

Step 2: Upload

Upload your source data files.

Step 2.1: Choose Upload Method

STEP2OF6:UPLOAD[1Configure][2Upload][3Map][4Validate][5Preview][6Commit]DraganddropfileshereorclicktobrowseyourcomputerSupported:CSV,ZIP,EADXML[BrowseFiles]Orspecifyaserverdirectorypath(forlargebatches):/uploads/imports/2026-batch-photos/[ScanDirectory]

Step 2.2: File Auto-Detection

After upload, the system auto-detects file properties:

FILEANALYSISUploadedFile:photo_collection_2026.csvPropertyDetectedValueFileTypeCSV(Comma-Separated)Delimiter,(comma)EncodingUTF-8TotalRows1,240HeaderRowYes(row1)Columns18

Step 2.3: Preview Data (First 10 Rows)

DATAPREVIEW(first10of1,240rows)#identifiertitledatelevelscope1PHO-0001MarketDay1965ItemBlackPhotographand..2PHO-0002TownHall1966ItemColorOpeningpho..3PHO-0003School1967ItemPhot..Portrait.................

Step 2.4: ZIP File Extraction (if applicable)

When a ZIP file is uploaded, the system extracts and displays the file tree:

ZIPEXTRACTIONphoto_collection.zip(245MB)-Extractedsuccessfully[v]photo_collection/[v]metadata/records.csv(1,240rows)[v]digital_objects/[v]photographs/PHO-0001.tif(42MB)PHO-0002.tif(38MB)PHO-0003.jpg(12MB)...(1,237morefiles)Filesfound:1,241(1CSV1,240digitalobjects)[<<Back][Next:Map&Enrich]

Step 3: Map & Enrich

Map source columns to AtoM target fields and enrich with metadata.

Step 3.1: Two-Column Mapping Interface

STEP3OF6:MAP&ENRICH[1Configure][2Upload][3Map][4Validate][5Preview][6Commit]MappingProfile:[SelectSavedProfilev][LoadProfile][SaveCurrentMapping]SOURCECOLUMNCONFIDENCETARGETAtoMFIELDidentifier100%[identifierv][##]title100%[titlev][##]date_created85%[eventDatesv][#]scope_content100%[scopeAndContentv][##]creator_name70%[eventActorsv][#-]phys_desc60%[extentAndMediumv][#-]file_path0%[unmappedv][]ConfidenceLegend:[##]Green=Exactmatch[#-]Yellow=Fuzzymatch[]Red=Unmapped(requiresmanualselection)

Step 3.2: Default Value Assignment

DEFAULTVALUESAssigndefaultvaluesforunmappedoremptyfields:FieldDefaultValuelevelOfDescription[Itemv]publicationStatus[Draftv]language[Englishv]repository[NARSSAv]culture[en]

Step 3.3: Digital Object Matching Strategy

DIGITALOBJECTMATCHINGHowshoulddigitalobjectsbematchedtorecords?(o)Filenamematch-matchDOfilenametoidentifiere.g.,PHO-0001.tifmatchesidentifierPHO-0001()LegacyIDmatch-matchDOfilenametolegacyId()Titlematch-matchDOfilenametotitle()CSVcolumn-useaspecificCSVcolumnColumn:[digitalObjectPathv]

Step 3.4: Metadata Extraction Panel

EMBEDDEDMETADATAEXTRACTIONExtractmetadatafromdigitalobjects:[X]EXIFdata(camera,datetaken,GPS)[X]IPTCdata(caption,keywords,credit)[X]XMPdata(DublinCore,rights,description)MapextractedmetadatatoAtoMfields:EXIFDateTimeOriginal[eventDatesv]IPTCCaption[scopeAndContentv]IPTCKeywords[subjectAccessPointsv]XMPCreator[eventActorsv]XMPRights[accessConditionsv][<<Back][Next:Validate]

Saved Mapping Profiles

Mapping profiles can be loaded from the ahgDataMigrationPlugin (if installed) or saved during the current session for reuse in future ingestions.

Action Description
Load Profile Apply a previously saved column-to-field mapping
Save Current Mapping Store the current mapping for future sessions
Auto-Map Re-run auto-detection to reset confidence scores

Step 4: Validate

Validate all rows before committing to the database.

Step 4.1: Automatic Validation Runs

STEP4OF6:VALIDATE[1Configure][2Upload][3Map][4Validate][5Preview][6Commit]Validationinprogress...[============================================>]88%ChecksRunning:[X]RequiredfieldsperISAD(G)[X]Dateformatvalidation[X]Hierarchyintegrity(parentIdreferences)[X]Digitalobjectfileexistence[X]MIMEtypeverification[X]SHA-256checksumgeneration[>]Duplicatedetection...[]Summaryreport

Step 4.2: Validation Summary

VALIDATIONRESULTSTotalValidWarningsErrorsRows1,2401,195387(96.4%)(3.1%)(0.6%)

Step 4.3: Review Issues

VALIDATIONISSUESFilter:[All][ErrorsOnly][WarningsOnly]ERRORS(7)#RowFieldIssueFix142titleRequiredfieldisempty[E]2103eventDateInvalidformat:13/2026[E]3205parentIdRefID-999notfound[E]4340DOfilePHO-0340.tifnotinZIP[E]5512titleRequiredfieldisempty[E]6780eventDateInvalidformat:unknown[E]7901parentIdCircularreference[E]WARNINGS(38)#RowFieldIssueFix115checksumDuplicateofrow14[E]288titlePossibleduplicate:[E]"TownHall"matchesrow2..............

Step 4.4: Inline Fix or Exclude Rows

Click [E] (Edit) on any row to fix inline, or exclude it from the import:

  • EDIT ROW 42
  • identifier: [PHO-0042 ]
  • title: [Untitled Photograph ]
  • eventDates: [1968 ]
  • levelOfDesc: [Item v]
  • Include this row [ ] Exclude from import
  • [Cancel] [Save & Re-validate]

Duplicate Detection Methods

Method Description
Identifier Exact match on identifier field
Legacy ID Exact match on legacyId field
Title + Date Records with same title AND date
Checksum SHA-256 match on digital object file 
                        [<< Back]  [Next: Preview >>]

Step 5: Preview & Approve

Review the complete import as a hierarchical tree before committing.

Step 5.1: Hierarchical Tree Visualization

STEP5OF6:PREVIEW&APPROVE[1Configure][2Upload][3Map][4Validate][5Preview][6Commit]HIERARCHYTREERECORDDETAIL[v]PhotoCollectionTitle:MarketDayPhoto(Fonds)[green]Identifier:PHO-0001Date:1965[v]SeriesALevel:Item[green]Scope:BlackandwhitephotographoftheweeklyPHO-0001marketatChurchSquare[green]PHO-0002DigitalObject:[green]PHO-0003[amber]PHO-0001.tifimg42MB[v]SeriesB4000x3000px[green]PHO-0050[green]PHO-0051Status:Valid[red/excluded]...ColorLegend:[green]=Valid[amber]=Warning[red]=Excluded

Step 5.2: SIP/DIP Package Preview (if enabled)

PACKAGEPREVIEWPackageEstimatedSizeSIP(Submission)2.4GBRecords:1,233DigitalObjects:1,233MetadataXML:1DIP(Dissemination)890MBAccesscopies:1,233Thumbnails:1,233Findingaid:1

Step 5.3: Approval Actions

APPROVALSummary:-Recordstocreate:1,233(7excluded)-Digitalobjects:1,232(1missingfileexcluded)-Estimatedtime:~15minutes[ApproveAll]Importall1,233validrecords[ApprovewithExclusions]Importvalidonly,skip7excluded[Cancel]Returntodashboard[<<BacktoValidate]

Step 6: Commit & Report

Execute the import and monitor progress in real time.

Step 6.1: Live Progress Bar

STEP6OF6:COMMIT[1Configure][2Upload][3Map][4Validate][5Preview][6Commit]Ingestioninprogress...donotclosethispage.Stage:UploadingDigitalObjects[==============================>]62%765/1,233recordsElapsed:6m23sEstimatedremaining:3m50sStages:[X]CreatingRecords(1,233/1,233)[>]UploadingDigitalObjects(765/1,232)[]GeneratingDerivatives(0/1,232)[]BuildingSIPPackage[]BuildingAIPPackage[]BuildingDIPPackage[]IndexinginElasticsearchProgressupdatesevery2seconds(AJAXpolling)

Step 6.2: Completion Report

INGESTIONCOMPLETESession:2026PhotoCollectionBatchImportStatus:CommittedsuccessfullyDuration:14m52sMetricResultRecordsCreated1,233DigitalObjectsUploaded1,232ThumbnailsGenerated1,232ReferenceImages1,232SIPPackage1(2.4GB)DIPPackage1(890MB)ChecksumsComputed1,232Errors0RowsExcluded7Actions:[DownloadManifest]CSVwithAtoMIDsforallcreatedrecords[ViewRecords]BrowsecreatedrecordsinAtoM[Rollback]Undoallcreatedrecordsanddigitalobjects[NewIngestion]Startanothersession

Manifest CSV Format

The downloadable manifest contains:

row identifier atomId slug digitalObjectPath stat
1 PHO-0001 123456 pho-.. uploads/r/n/... ok
2 PHO-0002 123457 pho-.. uploads/r/n/... ok
3 PHO-0003 123458 pho-.. uploads/r/n/... ok
... ... ... ... ... ...

Rollback

Rollback undoes an entire committed session, removing all created records and digital objects.

How Rollback Works

ROLLBACKCONFIRMATION[!]WARNING:Thisactioncannotbeundone.YouareabouttorollbacksessionING-5:"2026PhotoCollectionBatchImport"Thiswillpermanentlydelete:-1,233informationobjectrecords-1,232digitalobjectfilesandderivatives-Associatedaccesspointsandevents-SIP/DIPpackagesType"ROLLBACK"toconfirm:ROLLBACK[Cancel][ExecuteRollback]

After rollback, the session status changes to "rolled back" and the records are permanently removed from AtoM.

CSV Templates

Download pre-built CSV templates for each GLAM sector from the dashboard:

Available Templates

Template Sector Standard Columns
Archive - ISAD(G) Archive ISAD(G) identifier, title, levelOfDescription, eventDates, eventTypes, eventActors, repository, scopeAndContent, extentAndMedium, ...
Archive - RAD Archive RAD identifier, title, levelOfDescription, dates, nameAccessPoints, placeAccessPoints, ...
Archive - DACS Archive DACS identifier, title, levelOfDescription, date, extent, scopeAndContent, ...
Museum Museum CCO objectNumber, objectName, title, briefDescription, numberOfObjects, ...
Library - Dublin Core Library Dublin Core identifier, title, creator, subject, description, publisher, date, type, format, ...
Gallery - CCO Gallery CCO identifier, title, creator, date, workType, measurements, materials, ...
DAM - Dublin Core DAM Dublin Core identifier, title, creator, date, format, rights, description, ...

Supported File Formats

Upload Formats

Format Extension Description
CSV .csv Comma/tab/semicolon delimited data
ZIP .zip Archive containing CSV + digital objects
EAD XML .xml Encoded Archival Description finding aid

Digital Object Formats (within ZIP)

Category Formats
Images TIFF, JPEG, PNG, GIF, BMP, SVG
Documents PDF, DOCX, DOC, ODT, TXT
Audio WAV, MP3, FLAC, OGG, AIFF
Video MP4, MOV, AVI, MKV, WEBM
3D OBJ, GLB, GLTF, STL

Quick Reference

Page URL Description
Dashboard /ingest All sessions overview
New Session /ingest/configure Start Step 1
Upload /ingest/upload Step 2 (within session)
Map & Enrich /ingest/map Step 3 (within session)
Validate /ingest/validate Step 4 (within session)
Preview /ingest/preview Step 5 (within session)
Commit /ingest/commit Step 6 (within session)

Session Statuses

Status Meaning
configuring Session created, Step 1 in progress
uploading Files being uploaded, Step 2
mapping Field mapping, Step 3
validating Validation running, Step 4
previewing Awaiting approval, Step 5
committing Import executing, Step 6
committed Successfully completed
rolled back Commit undone
cancelled Session aborted by user
failed Import failed with errors

Database Tables

Table Purpose
ingest_session Session metadata, status, configuration
ingest_file Uploaded files and extracted paths
ingest_mapping Column-to-field mapping definitions
ingest_validation Validation results per row
ingest_row Individual row data and status
ingest_job Background job tracking

Descriptive Standards by Sector

Sector Recommended Standard Alternatives
Archive ISAD(G) RAD, DACS
Museum Collections procedure CCO
Library Dublin Core -
Gallery CCO Dublin Core
DAM Dublin Core -

Troubleshooting

Upload Fails

[!] Troubleshooting

  • Problem: File upload fails or times out
  • Solution:
    1. Check file size against PHP upload_max_filesize
  • (default 2M - increase in php.ini)
    1. Check post_max_size in php.ini
    1. For large batches, use the server directory path
  • option instead of browser upload
    1. Verify file permissions on upload directory

Mapping Not Auto-Detecting

  • [!] TROUBLESHOOTING
  • Problem: All columns show as unmapped (red)
  • Solution:
    1. Ensure CSV headers match AtoM field names
  • (e.g., "title" not "Title" or "TITLE")
    1. Check the selected descriptive standard matches
  • your CSV structure
    1. Try loading a saved mapping profile
    1. Manually map columns using the dropdown selectors

Validation Errors on Hierarchy

  • [!] TROUBLESHOOTING
  • Problem: "parentId not found" or "circular reference"
  • Solution:
    1. Verify parentId values reference valid legacyId
  • values within the same CSV
    1. Check for typos in parentId column
    1. Ensure parent rows appear BEFORE child rows
    1. Use the inline edit [E] button to correct references
    1. Select "Top-level" placement if hierarchy is flat

Commit Takes Too Long

[!] Troubleshooting

  • Problem: Commit stage seems stuck or very slow
  • Solution:
    1. Large batches (>5,000 rows) are normal to take
  • 30+ minutes
    1. Digital object upload is the slowest stage -
  • consider smaller batches for large files
    1. AI processing (NER, OCR) adds significant time -
  • disable if not needed for this batch
    1. Check server disk space for derivative generation
    1. Do not close the browser tab - progress is polled
  • via AJAX every 2 seconds

Rollback Not Available

  • [!] TROUBLESHOOTING
  • Problem: Rollback button is disabled or missing
  • Solution:
    1. Only committed sessions can be rolled back
    1. Already rolled-back sessions cannot be rolled back
  • again
    1. Verify you have administrator privileges
    1. Sessions older than the configured retention period
  • may lose rollback capability

Digital Objects Not Matching

  • [!] TROUBLESHOOTING
  • Problem: Digital objects not linked to records
  • Solution:
    1. Check the matching strategy in Step 3
  • (filename, legacyId, title, CSV column)
    1. For filename matching, ensure the file name
  • (without extension) matches the identifier
    1. Verify digital objects are in the correct
  • directory within the ZIP
    1. Check for case sensitivity mismatches
  • (PHO-0001.tif vs pho-0001.tif)

Need Help?

Contact your system administrator or visit the AHG documentation at: https://github.com/ArchiveHeritageGroup/atom-extensions-catalog/tree/main/docs

Part of the AtoM AHG Framework