Harmon/insar-management-system-v2
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor local InSAR asset and production workflows

Browse Source
This commit is contained in:
Harmon
2026-06-21 12:30:21 +08:00
parent 65a8cc4eac
commit 71c524967c
88 changed files with 11165 additions and 3017 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/UNC_SOURCE_ARCHIVE_AND_MATERIALIZE_DESIGN_20260615.md
+96 -159
View File
@@ -1,29 +1,35 @@
# UNC Source Archive and Local Materialize Design
# Local-Only Source Archive and Task_Pool Materialize Design
Current canonical operating contract: [THREE_SENSOR_LOCAL_PRODUCTION_CONTRACT_20260616.md](THREE_SENSOR_LOCAL_PRODUCTION_CONTRACT_20260616.md). This file remains the implementation detail for local archive metadata extraction and Task_Pool materialization.
## Decision
UNC/SMB storage is treated as the source archive pool. Production engines should not use UNC paths as their working input. D-InSAR, SBAS, Gamma/PyINT, LandSAR, and SARscape should consume local materialized task inputs.
As of 2026-06-15, UNC/SMB storage is removed from the active production path.
This keeps the 20 TB storage useful for long-term source management while protecting production from SMB disconnects, credential scope, WSL path conversion, and external engine UNC compatibility.
The switch throughput is too low for production data movement. Large source archives, precise orbit files, GF3 native result pools, task staging, and engine inputs must all live on local disks. A network share may exist outside this system for manual backup or manual transfer, but it must not be configured in runtime environment variables used by backend scans, inventory, materialization, or production.
Production engines consume local Task_Pool inputs. If a selected LT-1 or Sentinel-1 source product is archived, it is extracted from a local source archive into `D:\Task_Pool` or a task-specific subdirectory before engine execution.
The local source archive remains the source of record. LT-1 and Sentinel-1 materialization must never delete the original archive after extraction.
## Current Implementation
- Source asset inventory now recognizes archive assets:
- Source asset inventory recognizes archive assets:
- `S1_ZIP`
- `LT1_ARCHIVE`
- `GF3_ARCHIVE`
- Sentinel-1 ZIP manifest parsing already reads `manifest.safe` directly from the ZIP.
- Sentinel-1 ZIP manifest parsing reads `manifest.safe` directly from the ZIP.
- LT-1 archive parsing reads `*.meta.xml` directly from `.zip`, `.tar`, `.tar.gz`, or `.tgz` and records contained TIFF members.
- GF3 archive parsing reads the first XML member directly from `.zip`, `.tar`, `.tar.gz`, or `.tgz` and records quicklook-like members when present.
- `GF3_ARCHIVE_SOURCE_DIRS` roots are included in asset inventory scans as source pools.
- Source asset listing and inventory counts now include archive assets instead of hiding `S1_ZIP`.
- `SOURCE_PRODUCT_DIRS` is the local LT-1/Sentinel-1 source archive inventory root.
- Source asset listing and inventory counts include archive assets instead of hiding `S1_ZIP`.
- A generic source materialize endpoint exists:
- `POST /api/assets/sources/{asset_id}/materialize`
- `S1_ZIP` uses the existing Sentinel-1 SAFE unpacker.
- `LT1_ARCHIVE` and `GF3_ARCHIVE` extract to a local materialized directory.
- Directory assets return `DIRECTORY_READY`.
- If no `target_root` is supplied, generic materialize defaults to `TASK_POOL_ROOT\source_materialized\<sensor>`.
Default source materialization is task-scoped. D-InSAR and SBAS callers should pass a Task_Pool target directory:
Default source materialization is local and task-scoped. D-InSAR and SBAS callers should pass a Task_Pool target directory:
```text
D:\Task_Pool\DInSAR\<task>\master
@@ -31,84 +37,65 @@ D:\Task_Pool\DInSAR\<task>\slave
D:\Task_Pool\SBAS\<stack>\sources\<YYYYMMDD>
```
The generic materialize endpoint still accepts `target_root` for ad hoc checks. Production callers must provide a Task_Pool destination.
The generic materialize endpoint still accepts `target_root` for ad hoc checks. Production callers should provide a task-specific Task_Pool destination.
## Production Boundary
D-InSAR and SBAS should store source asset references in task/run manifests, then materialize selected inputs into the run directory before engine execution.
D-InSAR and SBAS store local source asset references in task/run manifests, then materialize selected inputs into the run directory before engine execution.
Required next integration points:
Required integration points:
- D-InSAR Task_Pool publishing:
- store `source_product_asset_id`, `archive_path`, `source_format`;
- store `source_product_asset_id`, `archive_path`, and `source_format`;
- materialize master/slave archive assets into the task directory before engine dispatch.
- Gamma/PyINT:
- always consume local materialized paths because WSL conversion rejects or cannot reliably map UNC paths.
- consume local materialized paths because WSL conversion rejects or cannot reliably map network paths.
- LandSAR and ENVI/SARscape:
- prefer local materialized paths even when Windows can see UNC, to avoid external engine path and credential issues.
- consume local materialized paths.
- SBAS:
- stack discovery can use archive metadata;
- selected scenes must be materialized into the SBAS `RAW`/input structure before Gamma commands such as `par_LT1_SLC`.
## GF3 Management
GF3 has two asset layers:
- `GF3_ARCHIVE`: original source archive, suitable for UNC source management and migration tracking.
- GF3 SARscape standardized L2: production result/analysis-ready layer, used for map footprint, preview, radar data management, and water extraction.
Do not replace standardized L2 management with raw archive management. Archive assets should link migration and production status; previews and water extraction should continue to consume standardized L2/analysis-ready products.
## Migration Guidance
1. Register UNC roots first and scan inventory.
2. Verify archive asset counts and parse status.
3. Keep existing local standardized results and D-InSAR/SBAS products in place.
4. Move source archives to UNC and update root configuration.
5. Only after inventory and materialize tests pass, switch D-InSAR/SBAS publishing to archive asset references.
Production safety rule: if a run cannot materialize every selected source asset locally, the run must fail before invoking the engine.
## Recommended UNC Layout
## GF3 Management
The current deployment uses two SMB shares:
GF3 now has a separate operational rule:
```text
\\DESKTOP-N16HJ84\InSAR_Storage_1
\\DESKTOP-N16HJ84\InSAR_Storage_2
```
- GF3 SARscape production is not run on this management machine.
- Already-produced SARscape `_geo` ENVI binary results are stored locally under `GF3_SARSCAPE_NATIVE_DIRS`.
- The system registers those local `_geo` native results and their `.hdr/.sml` sidecars.
- `*_geo_ql.tif` is retained only as an auxiliary quicklook file.
- WebP preview cache is generated locally from the `_geo` ENVI binary, not from `*_geo_ql.tif`.
- Standard GeoTIFF conversion remains a separate explicit path; the monitor button used for GF3 registration is native-result registration only.
Recommended source archive layout:
```text
\\DESKTOP-N16HJ84\InSAR_Storage_1
└─ GaoFen-3
├─ 20260513
│ └─ GF3_*.tar.gz
└─ 20260514
\\DESKTOP-N16HJ84\InSAR_Storage_2
├─ LuTan-1
│ └─ Archive
│ ├─ 20260513
│ │ └─ LT1*.tar.gz / LT1*.tgz / LT1*.zip / LT1*.tar
│ └─ 20260514
├─ Sentinel-1
│ └─ Archive
│ ├─ 20260513
│ │ └─ S1*.zip
│ └─ 20260514
└─ Orbit
├─ LuTan-1
│ ├─ LT1A_GpsData_GAS_C_YYYYMMDD.txt
│ └─ LT1B_GpsData_GAS_C_YYYYMMDD.txt
└─ Sentinel-1
└─ S1*.EOF
```
Recommended local Task_Pool layout:
## Recommended Local Layout
```text
D:\
├─ LuTan1_Image_Pool_Zip
│ └─ LT1*.tar.gz / LT1*.tgz / LT1*.zip / LT1*.tar
├─ Sentinel1_Image_Pool_ZIP
│ └─ S1*.zip
├─ LuTan1_Image_Pool
│ └─ LT1 unpacked scene directories
├─ Sentinel1_Image_Pool
│ └─ S1*.SAFE directories
├─ LT1_data_lsarorbit
│ └─ LT1*_GpsData_*.txt
├─ Sentinel1_EOF_Pool
│ └─ S1*.EOF
├─ production_results
│ └─ gf3
│ ├─ sarscape_native
│ │ └─ YYYYMMDD_geo
│ │ └─ GF3_*
│ │ ├─ *_geo
│ │ ├─ *_geo.hdr
│ │ ├─ *_geo.sml
│ │ └─ *_geo_ql.tif
│ └─ standard_l2
└─ Task_Pool
D:\Task_Pool
├─ DInSAR
│ └─ <pair_task>
@@ -129,30 +116,35 @@ D:\Task_Pool
└─ publish
```
Date folders are optional for the scanner because source and orbit inventory recurse through configured roots. They are recommended for operator readability and migration checks.
Date folders are optional for the LT-1/Sentinel-1 scanners because inventory recurses through configured roots. GF3 native pools should keep the SARscape `YYYYMMDD_geo/<scene>` convention.
## Current Local Configuration Example
The local `.env` should keep legacy local roots and UNC roots side by side during migration:
The local `.env` should keep all runtime roots local:
```text
SOURCE_PRODUCT_DIRS=D:\LuTan1_Image_Pool;D:\Sentinel1_Image_Pool_ZIP;\\DESKTOP-N16HJ84\InSAR_Storage_2\LuTan-1\Archive;\\DESKTOP-N16HJ84\InSAR_Storage_2\Sentinel-1\Archive
ORBIT_SOURCE_DIRS=D:\LT1_data_lsarorbit;D:\Sentinel1_EOF_Pool;\\DESKTOP-N16HJ84\InSAR_Storage_2\Orbit\LuTan-1;\\DESKTOP-N16HJ84\InSAR_Storage_2\Orbit\Sentinel-1
GF3_ARCHIVE_SOURCE_DIRS=\\DESKTOP-N16HJ84\InSAR_Storage_1\GaoFen-3
SOURCE_PRODUCT_DIRS=D:\LuTan1_Image_Pool_Zip;D:\Sentinel1_Image_Pool_ZIP
SENTINEL1_STORAGE_DIRS=
ORBIT_SOURCE_DIRS=D:\LT1_data_lsarorbit;D:\Sentinel1_EOF_Pool
MONITOR_ORBIT_DIR=D:\LT1_data_lsarorbit
GF3_TASK_POOL_ROOT=D:\GaoFen3_Task_Pool
GF3_ARCHIVE_SOURCE_DIRS=D:\GaoFen3_Image_Pool\archives
GF3_SARSCAPE_NATIVE_DIRS=D:\GaoFen3_Image_Pool\sarscape_native
GF3_STORAGE_DIRS=D:\GaoFen3_Image_Pool\standard_l2
TASK_POOL_ROOT=D:\Task_Pool
DINSAR_TASK_POOL_ROOT=D:\Task_Pool\DInSAR
SBAS_TASK_POOL_ROOT=D:\Task_Pool\SBAS
GAMMA_SBAS_WORK_ROOT=D:\Task_Pool\SBAS
```
Do not store SMB credentials in `.env`. Credentials should be stored in Windows Credential Manager for the account that runs the backend/worker service.
Do not configure UNC paths in these variables.
## Orbit Pool Contract
There are two different orbit concepts:
- `ORBIT_SOURCE_DIRS`: source inventory roots. These can be UNC and may be date-organized or flat.
- `ORBIT_POOL_ENVI` / `PYINT_ORBIT_POOL_TXT`: local production orbit pools. These should remain local disk paths.
- `ORBIT_SOURCE_DIRS`: local source inventory roots.
- `ORBIT_POOL_ENVI` / `PYINT_ORBIT_POOL_TXT`: local production orbit pools.
LT-1 local production orbit pool should support both flat and satellite-split layouts:
@@ -166,34 +158,40 @@ D:\orbit_pools\envi
└─ envi
```
The `LT1A` and `LT1B` names are satellite names, not product levels. ENVI/Gamma/PyINT/SBAS should use local orbit files copied or synchronized from `ORBIT_SOURCE_DIRS`; they should not be required to read UNC directly.
Sentinel-1 EOF files can be indexed from UNC. Gamma/PyINT/SBAS execution should stage required EOF files locally with the selected scenes.
The `LT1A` and `LT1B` names are satellite names, not product levels. ENVI/Gamma/PyINT/SBAS should use local orbit files copied or synchronized from `ORBIT_SOURCE_DIRS`.
## Migration Phases
### Phase 1: Source archive migration
### Phase 1: Local source archive inventory
Move or copy source archives only:
Keep production source archives local:
- LT-1 compressed scenes to `\\DESKTOP-N16HJ84\InSAR_Storage_2\LuTan-1\Archive\<YYYYMMDD>\`.
- Sentinel-1 ZIP scenes to `\\DESKTOP-N16HJ84\InSAR_Storage_2\Sentinel-1\Archive\<YYYYMMDD>\`.
- GF3 raw archives to `\\DESKTOP-N16HJ84\InSAR_Storage_1\GaoFen-3\<YYYYMMDD>\`.
- LT-1 compressed scenes in `D:\LuTan1_Image_Pool_Zip`.
- Sentinel-1 ZIP scenes in `D:\Sentinel1_Image_Pool_ZIP`.
- Existing unpacked local scene directories are not active management pools; they should only be task materialization outputs.
Keep current local unpacked scene directories in place until D-InSAR and SBAS archive materialization have been tested.
### Phase 2: Local orbit deployment
### Phase 2: Orbit source migration
Deploy orbit source files on this machine:
Copy orbit source files to UNC:
- LT-1 TXT files under `D:\LT1_data_lsarorbit`.
- Sentinel-1 EOF files under `D:\Sentinel1_EOF_Pool`.
- LT-1 TXT files to `\\DESKTOP-N16HJ84\InSAR_Storage_2\Orbit\LuTan-1\`.
- Sentinel-1 EOF files to `\\DESKTOP-N16HJ84\InSAR_Storage_2\Orbit\Sentinel-1\`.
Keep `ORBIT_POOL_ENVI` and `PYINT_ORBIT_POOL_TXT` local.
Keep `ORBIT_POOL_ENVI` and `PYINT_ORBIT_POOL_TXT` local. Add a later sync/materialize step to populate local orbit pools from the indexed UNC source assets.
### Phase 3: GF3 native-result registration
### Phase 3: Production cutover
Copy completed SARscape `_geo` result folders to the local GF3 native pool:
After inventory scan verifies UNC assets:
```text
D:\GaoFen3_Image_Pool\sarscape_native\YYYYMMDD_geo\<GF3 scene>\
```
Then run GF3 native-result registration and GF3 WebP generation. WebP generation reads the `_geo` ENVI binary and requires its `.hdr` sidecar.
### Phase 4: Task_Pool production
After inventory scan verifies local assets:
1. D-InSAR Task_Pool stores source asset IDs and archive paths.
2. Task preparation materializes master/slave scenes and orbit files under `D:\Task_Pool\DInSAR\<task>`.
@@ -201,26 +199,19 @@ After inventory scan verifies UNC assets:
4. Results register normally.
5. Local materialized inputs and intermediate products are eligible for cleanup after result registration.
### Phase 4: Retire old local source pools
Only after repeated D-InSAR/SBAS runs succeed from archive materialization:
- remove old local source roots from `SOURCE_PRODUCT_DIRS`;
- keep local work/result roots;
- keep standardized GF3 L2 products unless explicitly migrated and revalidated.
## Local Cleanup Design
After source archives are managed on UNC and production results are registered as assets, local disk can be treated as a cache/work area. Cleanup should be explicit and asset-aware.
After production results are registered as assets, Task_Pool materialized inputs and work folders can be treated as cleanup candidates. Local source archive roots are durable production inputs and must not be cleaned as cache.
### Keep Classes
Cleanup must never delete:
- configured UNC source archive roots;
- local or UNC orbit source roots;
- configured local source archive roots;
- configured local orbit source roots;
- registered D-InSAR result assets;
- registered SBAS result assets;
- registered GF3 SARscape native result assets;
- registered GF3 standardized L2 assets;
- `SAR_ANALYSIS_READY_ROOT` products and water extraction result assets;
- current pointers, manifests, previews, and catalog metadata needed to open results.
@@ -234,7 +225,7 @@ Cleanup may delete only these local classes after verification:
- D-InSAR engine intermediate folders not listed in the result manifest;
- Gamma/PyINT temporary project work directories after result registration;
- SBAS `RAW`, `SLC`, `RSLC`, `MLI`, `DIFF`, `DIFF1`, script logs, and temporary staging after SBAS product registration;
- GF3 SARscape native intermediates only after standardized L2 registration and optional native-retention policy allows cleanup.
- GF3 SARscape runtime or temporary staging only after local native `_geo` registration and optional native-retention policy allows cleanup.
### Safety Contract
@@ -249,59 +240,5 @@ Deletion must require:
- path is not inside any configured source archive root;
- path is not inside a result publish root unless the exact file is classified as intermediate;
- associated result or standardized asset is registered;
- candidate is older than a configurable minimum age;
- no active task references the path.
### Proposed API
```text
POST /api/maintenance/cleanup/preview
POST /api/maintenance/cleanup/execute
```
Preview request fields:
```json
{
"scope": "dinsar|sbas|gf3|materialized|all",
"root_ids": [],
"older_than_hours": 24,
"require_registered_result": true,
"include_task_pool_inputs": false
}
```
Preview response should include:
```json
{
"preview_id": "...",
"total_bytes": 0,
"candidates": [
{
"path": "D:\\production_runtime\\...",
"class": "materialized_source",
"owner": "task/run/product id",
"size_bytes": 0,
"eligible": true,
"reason": "registered_result_exists"
}
],
"blocked": []
}
```
### Recommended Defaults
- `materialized`: delete after 24 hours if no active task references it.
- `dinsar`: delete engine intermediates after result registration; keep Task_Pool inputs until all selected engines are complete or user opts in.
- `sbas`: delete heavy Gamma working directories after SBAS catalog registration and product assets exist.
- `gf3`: keep standardized L2; clean SARscape native only when `GF3_SARSCAPE_CLEAN_AFTER_SUCCESS=true` and standardized registration is confirmed.
### Implementation Order
1. Add read-only cleanup preview service.
2. Add path classification and approved-root checks.
3. Add execute endpoint with preview token.
4. Add frontend maintenance panel.
5. Wire D-InSAR/SBAS/GF3 run pages to show cleanup eligibility after successful registration.
- cleanup policy explicitly allows the class;
- operator confirmation is present.