Source Archive Integrity Audit

Date: 2026-06-20

This document defines the LT-1 / Sentinel-1 source archive integrity audit flow. It is separate from normal source asset inventory scanning.

Boundary

Normal asset inventory scan remains lightweight:

Recursively discovers local SOURCE_PRODUCT_DIRS.
Extracts LT-1 XML or Sentinel-1 manifest.safe metadata from archives.
Builds metadata documents, preview caches, radar scene records, and orbit bindings.
Uses file_path + size_bytes + mtime_epoch + parser_version + parse_status to skip unchanged archives.

Archive integrity audit is an explicit background task:

Task type: AUDIT_SOURCE_ARCHIVE_INTEGRITY.
API: POST /assets/inventory/archive-integrity-audit.
Default formats: LT1_ARCHIVE, S1_ZIP.
Default families: caller supplied; UI uses LT-1 and Sentinel-1.
It can be started from the asset inventory panel or data monitor panel.

This prevents every metadata scan from fully reading multi-GB archives.

Validation Method

ZIP archives use Python zipfile.ZipFile.testzip() and safe member path validation.

TAR, TGZ, and TAR.GZ archives use Python tarfile.open(..., "r:*") and full member iteration. For gzip-compressed tar streams, reaching EOF through tarfile validates the gzip stream CRC/truncation state. TAR members are also checked for unsafe paths, links, devices, and other unsupported special member types.

The audit records:

archive_integrity_status: NOT_CHECKED, OK, FAILED, UNSUPPORTED
archive_integrity_method
archive_integrity_checked_at
archive_integrity_error
archive_integrity_version
archive_integrity_member_count

Incremental Semantics

An archive is skipped when all conditions hold:

force=false
size_bytes and mtime_epoch still match the filesystem
archive_integrity_version equals the current audit version
previous status is OK, FAILED, or UNSUPPORTED

If the archive changes or the audit version changes, the audit runs again. When normal metadata scanning reparses a changed source archive, it resets the integrity fields to NOT_CHECKED.

Issue Handling

Audit failures create an open asset_inventory_issues row:

inventory_type=source_product
asset_ref_id=<source_product_assets.id>
issue_code=source_archive_integrity_failed
severity=error

When a later audit passes or marks the archive unsupported, the previous open integrity issue for that source asset is resolved.

Operational Notes

The audit is intentionally I/O heavy. A 1.22 GB LT-1 tar.gz test archive with 8 members completed a full stream audit in about 9.8 seconds on the current server. Full-pool audits should be run manually, not on backend startup.

The original source archive remains the source of record and must not be deleted after materialization.

2.8 KiB Raw Blame History