insar-management-system-v2/docs/UNC_SOURCE_ARCHIVE_AND_MATERIALIZE_DESIGN_20260615.md

Local-Only Source Archive and Task_Pool Materialize Design

Current canonical operating contract: THREE_SENSOR_LOCAL_PRODUCTION_CONTRACT_20260616.md. This file remains the implementation detail for local archive metadata extraction and Task_Pool materialization.

Decision

As of 2026-06-15, UNC/SMB storage is removed from the active production path.

The switch throughput is too low for production data movement. Large source archives, precise orbit files, GF3 native result pools, task staging, and engine inputs must all live on local disks. A network share may exist outside this system for manual backup or manual transfer, but it must not be configured in runtime environment variables used by backend scans, inventory, materialization, or production.

Production engines consume local Task_Pool inputs. If a selected LT-1 or Sentinel-1 source product is archived, it is extracted from a local source archive into D:\Task_Pool or a task-specific subdirectory before engine execution.

The local source archive remains the source of record. LT-1 and Sentinel-1 materialization must never delete the original archive after extraction.

Current Implementation

• Source asset inventory recognizes archive assets:
  • S1_ZIP
  • LT1_ARCHIVE
  • GF3_ARCHIVE
• Sentinel-1 ZIP manifest parsing reads manifest.safe directly from the ZIP.
• LT-1 archive parsing reads *.meta.xml directly from .zip, .tar, .tar.gz, or .tgz and records contained TIFF members.
• SOURCE_PRODUCT_DIRS is the local LT-1/Sentinel-1 source archive inventory root.
• Source asset listing and inventory counts include archive assets instead of hiding S1_ZIP.
• A generic source materialize endpoint exists:
  • POST /api/assets/sources/{asset_id}/materialize
  • S1_ZIP uses the existing Sentinel-1 SAFE unpacker.
  • LT1_ARCHIVE and GF3_ARCHIVE extract to a local materialized directory.
  • Directory assets return DIRECTORY_READY.
• If no target_root is supplied, generic materialize defaults to TASK_POOL_ROOT\source_materialized\<sensor>.

Default source materialization is local and task-scoped. D-InSAR and SBAS callers should pass a Task_Pool target directory:

D:\Task_Pool\DInSAR\<task>\master
D:\Task_Pool\DInSAR\<task>\slave
D:\Task_Pool\SBAS\<stack>\sources\<YYYYMMDD>

The generic materialize endpoint still accepts target_root for ad hoc checks. Production callers should provide a task-specific Task_Pool destination.

Production Boundary

D-InSAR and SBAS store local source asset references in task/run manifests, then materialize selected inputs into the run directory before engine execution.

Required integration points:

• D-InSAR Task_Pool publishing:
  • store source_product_asset_id, archive_path, and source_format;
  • materialize master/slave archive assets into the task directory before engine dispatch.
• Gamma/PyINT:
  • consume local materialized paths because WSL conversion rejects or cannot reliably map network paths.
• LandSAR and ENVI/SARscape:
  • consume local materialized paths.
• SBAS:
  • stack discovery can use archive metadata;
  • selected scenes must be materialized into the SBAS RAW/input structure before Gamma commands such as par_LT1_SLC.

Production safety rule: if a run cannot materialize every selected source asset locally, the run must fail before invoking the engine.

GF3 Management

GF3 now has a separate operational rule:

• GF3 SARscape production is not run on this management machine.
• Already-produced SARscape _geo ENVI binary results are stored locally under GF3_SARSCAPE_NATIVE_DIRS.
• The system registers those local _geo native results and their .hdr/.sml sidecars.
• *_geo_ql.tif is retained only as an auxiliary quicklook file.
• WebP preview cache is generated locally from the _geo ENVI binary, not from *_geo_ql.tif.
• Standard GeoTIFF conversion remains a separate explicit path; the monitor button used for GF3 registration is native-result registration only.

Recommended Local Layout

D:\
  ├─ LuTan1_Image_Pool_Zip
  │  └─ LT1*.tar.gz / LT1*.tgz / LT1*.zip / LT1*.tar
  ├─ Sentinel1_Image_Pool_ZIP
  │  └─ S1*.zip
  ├─ LuTan1_Image_Pool
  │  └─ LT1 unpacked scene directories
  ├─ Sentinel1_Image_Pool
  │  └─ S1*.SAFE directories
  ├─ LT1_data_lsarorbit
  │  └─ LT1*_GpsData_*.txt
  ├─ Sentinel1_EOF_Pool
  │  └─ S1*.EOF
  ├─ production_results
  │  └─ gf3
  │     ├─ sarscape_native
  │     │  └─ YYYYMMDD_geo
  │     │     └─ GF3_*
  │     │        ├─ *_geo
  │     │        ├─ *_geo.hdr
  │     │        ├─ *_geo.sml
  │     │        └─ *_geo_ql.tif
  │     └─ standard_l2
  └─ Task_Pool
D:\Task_Pool
  ├─ DInSAR
  │  └─ <pair_task>
  │     ├─ task_manifest.json
  │     ├─ .dinsar_pair.json
  │     ├─ master
  │     ├─ slave
  │     ├─ orbit
  │     ├─ work
  │     └─ publish
  └─ SBAS
     └─ <stack_task>
        ├─ task_manifest.json
        ├─ sbas_stack_manifest.json
        ├─ sources
        ├─ orbits
        ├─ work
        └─ publish

Date folders are optional for the LT-1/Sentinel-1 scanners because inventory recurses through configured roots. GF3 native pools should keep the SARscape YYYYMMDD_geo/<scene> convention.

Current Local Configuration Example

The local .env should keep all runtime roots local:

SOURCE_PRODUCT_DIRS=D:\LuTan1_Image_Pool_Zip;D:\Sentinel1_Image_Pool_ZIP
SENTINEL1_STORAGE_DIRS=
ORBIT_SOURCE_DIRS=D:\LT1_data_lsarorbit;D:\Sentinel1_EOF_Pool
MONITOR_ORBIT_DIR=D:\LT1_data_lsarorbit
GF3_TASK_POOL_ROOT=D:\GaoFen3_Task_Pool
GF3_ARCHIVE_SOURCE_DIRS=D:\GaoFen3_Image_Pool\archives
GF3_SARSCAPE_NATIVE_DIRS=D:\GaoFen3_Image_Pool\sarscape_native
GF3_STORAGE_DIRS=D:\GaoFen3_Image_Pool\standard_l2
TASK_POOL_ROOT=D:\Task_Pool
DINSAR_TASK_POOL_ROOT=D:\Task_Pool\DInSAR
SBAS_TASK_POOL_ROOT=D:\Task_Pool\SBAS
GAMMA_SBAS_WORK_ROOT=D:\Task_Pool\SBAS

Do not configure UNC paths in these variables.

Orbit Pool Contract

There are two different orbit concepts:

• ORBIT_SOURCE_DIRS: local source inventory roots.
• ORBIT_POOL_ENVI / PYINT_ORBIT_POOL_TXT: local production orbit pools.

LT-1 local production orbit pool should support both flat and satellite-split layouts:

D:\orbit_pools\envi
  ├─ LT1A
  │  └─ LT1A_GpsData_GAS_C_YYYYMMDD.txt
  ├─ LT1B
  │  └─ LT1B_GpsData_GAS_C_YYYYMMDD.txt
  └─ converted
     └─ envi

The LT1A and LT1B names are satellite names, not product levels. ENVI/Gamma/PyINT/SBAS should use local orbit files copied or synchronized from ORBIT_SOURCE_DIRS.

Migration Phases

Phase 1: Local source archive inventory

Keep production source archives local:

• LT-1 compressed scenes in D:\LuTan1_Image_Pool_Zip.
• Sentinel-1 ZIP scenes in D:\Sentinel1_Image_Pool_ZIP.
• Existing unpacked local scene directories are not active management pools; they should only be task materialization outputs.

Phase 2: Local orbit deployment

Deploy orbit source files on this machine:

• LT-1 TXT files under D:\LT1_data_lsarorbit.
• Sentinel-1 EOF files under D:\Sentinel1_EOF_Pool.

Keep ORBIT_POOL_ENVI and PYINT_ORBIT_POOL_TXT local.

Phase 3: GF3 native-result registration

Copy completed SARscape _geo result folders to the local GF3 native pool:

D:\GaoFen3_Image_Pool\sarscape_native\YYYYMMDD_geo\<GF3 scene>\

Then run GF3 native-result registration and GF3 WebP generation. WebP generation reads the _geo ENVI binary and requires its .hdr sidecar.

Phase 4: Task_Pool production

After inventory scan verifies local assets:

1. D-InSAR Task_Pool stores source asset IDs and archive paths.
2. Task preparation materializes master/slave scenes and orbit files under D:\Task_Pool\DInSAR\<task>.
3. Engines run only against local Task_Pool paths.
4. Results register normally.
5. Local materialized inputs and intermediate products are eligible for cleanup after result registration.

Local Cleanup Design

After production results are registered as assets, Task_Pool materialized inputs and work folders can be treated as cleanup candidates. Local source archive roots are durable production inputs and must not be cleaned as cache.

Keep Classes

Cleanup must never delete:

• configured local source archive roots;
• configured local orbit source roots;
• registered D-InSAR result assets;
• registered SBAS result assets;
• registered GF3 SARscape native result assets;
• registered GF3 standardized L2 assets;
• SAR_ANALYSIS_READY_ROOT products and water extraction result assets;
• current pointers, manifests, previews, and catalog metadata needed to open results.

Cleanup Classes

Cleanup may delete only these local classes after verification:

• materialized source inputs under source_materialized;
• D-InSAR Task_Pool copied inputs after every required engine run is registered;
• D-InSAR engine intermediate folders not listed in the result manifest;
• Gamma/PyINT temporary project work directories after result registration;
• SBAS RAW, SLC, RSLC, MLI, DIFF, DIFF1, script logs, and temporary staging after SBAS product registration;
• GF3 SARscape runtime or temporary staging only after local native _geo registration and optional native-retention policy allows cleanup.

Safety Contract

Every cleanup operation should run in two phases:

1. preview: enumerate candidate paths, classify each path, show size, last modified time, owning task/run/product, and keep/delete reason.
2. execute: delete only candidates from a persisted preview token or exact candidate list.

Deletion must require:

• path is inside an approved local work root;
• path is not inside any configured source archive root;
• path is not inside a result publish root unless the exact file is classified as intermediate;
• associated result or standardized asset is registered;
• cleanup policy explicitly allows the class;
• operator confirmation is present.