Harmon/insar-management-system-v2
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor local InSAR asset and production workflows

Browse Source
This commit is contained in:
Harmon
2026-06-21 12:30:21 +08:00
parent 65a8cc4eac
commit 71c524967c
88 changed files with 11165 additions and 3017 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/DEPLOYMENT.md
+11 -8
View File
@@ -97,14 +97,16 @@ NGINX_HEALTH_URL=http://127.0.0.1/
UNPACK_SOURCE_DIRS=D:\Archives
INSAR_STORAGE_DIRS=D:\LuTan1_Image_Pool
MONITOR_RADAR_DIRS=D:\LuTan1_Image_Pool
MONITOR_DINSAR_DIRS=D:\DInSARResult
MONITOR_DINSAR_DIRS=D:\production_results\dinsar
MONITOR_ORBIT_DIR=D:\LT1_data_lsarorbit
ORBIT_POOL_ENVI=D:\orbit_pools\envi
ORBIT_POOL_ISCE2=D:\orbit_pools\isce2
ORBIT_POOL_ISCE2=
ORBIT_POOL_LANDSAR=
```
`ORBIT_SOURCE_DIRS` is the source asset layer. LT-1 orbit scans also synchronize the production TXT pool under `ORBIT_POOL_ENVI\LT1A|LT1B`. `PYINT_ORBIT_POOL_TXT` and `GAMMA_SBAS_ORBIT_ROOTS` should point to that same local TXT pool unless a separate Gamma pool is deliberately maintained. `ORBIT_POOL_ISCE2` is legacy and remains empty while `ISCE2_ENABLED=false`.
### 3.4 ENVI / SARscape
```env
@@ -120,24 +122,25 @@ ENVI_TASK_TIMEOUT_SECONDS=21600
GF3 当前主线是“ENVI/SARscape 生产原生 `_geo` 证据层，平台标准化成 GeoTIFF 后入库和供洪涝/水体算法消费”。旧 Python/GDAL L1A 预处理链路默认关闭。
```env
GF3_ARCHIVE_SOURCE_DIRS=D:\production_inputs\gf3\archives
GF3_TASK_POOL_ROOT=D:\GaoFen3_Pool\task_pool
GF3_ARCHIVE_SOURCE_DIRS=D:\GaoFen3_Pool\archives
GF3_LEGACY_GDAL_ENABLED=false
GF3_SOURCE_DIRS=
GF3_SARSCAPE_NATIVE_DIRS=D:\production_results\gf3\sarscape_native
GF3_STORAGE_DIRS=D:\production_results\gf3\standard_l2
GF3_SARSCAPE_RUNTIME_DIR=D:\production_runtime\gf3\sarscape_runtime
GF3_SARSCAPE_NATIVE_DIRS=D:\GaoFen3_Pool\native_geo
GF3_STORAGE_DIRS=D:\GaoFen3_Pool\catalog
GF3_SARSCAPE_RUNTIME_DIR=D:\GaoFen3_Pool\task_pool\sarscape_runtime
GF3_SARSCAPE_WRAPPER_EXE=D:\Code\Insar_management_system_v2\third_party\GF3_L1A_To_L2_pipeline\dist\windows\gf3wrapper.exe
GF3_SARSCAPE_IDLRT_PATH=C:\Program Files\Harris\ENVI56\IDL88\bin\bin.x86_64\idlrt.exe
GF3_SARSCAPE_DEM_PATH=D:\DEM\COPDEM_GLO30_China_4326_DEM
GF3_SARSCAPE_POLARIZATIONS=HH,HV
GF3_SARSCAPE_AUTO_STANDARDIZE=true
GF3_SARSCAPE_AUTO_STANDARDIZE=false
GF3_SARSCAPE_CLEAN_AFTER_SUCCESS=true
```
说明：
- `GF3_SARSCAPE_NATIVE_DIRS` 长期保留 `_geo`、`.hdr`、`.sml`、快视、KML、日志和 manifest。
- `GF3_STORAGE_DIRS` 是标准 L2 GeoTIFF 池，后续入库、预览、洪涝/水体分析优先消费这里和 `SAR_ANALYSIS_READY_ROOT`。
- `GF3_STORAGE_DIRS` 是标准目录池，后续入库、预览、洪涝/水体分析优先消费这里和 `SAR_ANALYSIS_READY_ROOT`。
- `GF3_SARSCAPE_RUNTIME_DIR` 只放 wrapper 配置和运行时临时文件，不应混入原生结果池。
- 如确需恢复旧 L1A 解包/预处理，需要同时设置 `GF3_LEGACY_GDAL_ENABLED=true` 和 `GF3_SOURCE_DIRS`。
docs/DINSAR_TASK_POOL_THREE_ENGINE_REFACTOR_20260614.md
+1 -1
View File
@@ -222,7 +222,7 @@ Current direction:
- directory sources are copied into `master/` and `slave/`;
- `S1_ZIP`, `LT1_ARCHIVE`, and other supported archives are extracted into `master/` and `slave/`;
- staged orbit files go into `orbit/`.
- Engines must consume local Task_Pool paths, not UNC source archive paths.
- Engines must consume local Task_Pool paths. LT-1 and Sentinel-1 source archives are also local; UNC is not an active production source pool.
- `.dinsar_pair.json` records `source_materialization` so cleanup can distinguish copied directories, extracted archives, and staged files.
Cleanup implication:
docs/FLOOD_WATER_ALGORITHM_ENGINEERING_HANDOFF_20260602.md
+1 -1
View File
@@ -54,7 +54,7 @@
| `pixel_size_m` | 近似像元大小 |
| `status` | `PENDING/RUNNING/DONE/FAILED` |
当前 GF3 SARscape 链路会先产出原生 `_geo` ENVI 二进制，再由平台转换为 `D:\production_results\gf3\standard_l2` 下的 GeoTIFF，并注册到这里。
当前 GF3 SARscape 链路会先产出原生 `_geo` ENVI 二进制，再由平台转换为 `D:\GaoFen3_Image_Pool\standard_l2` 下的 GeoTIFF，并注册到这里。
### 3.2 WaterExtractionORM
docs/FRONTEND_NAVIGATION_ARCHITECTURE.md
+76 -163
View File
@@ -1,28 +1,14 @@
# Frontend Navigation Architecture
## 1. Purpose
Last updated: 2026-06-20
This document is the source of truth for the left-side navigation structure and the production workspace view model in the frontend.
This document is the frontend navigation source of truth. It reflects the current product decision: the system manages data for LT-1, Sentinel-1, and GF-3, but production is organized only around D-InSAR and SBAS-InSAR. PS-InSAR and legacy time-series entries are compatibility code, not visible primary workflows.
It explains:
- the first-level menu groups
- the sectioned groups and their leaf tabs
- the special-case production workspace entry
- the reserved entries and legacy route aliases
- the files that must be updated when navigation changes
The goal is to keep module boundaries stable as the system expands across D-InSAR, time-series InSAR, AI analysis, and operations workflows.
## 2. Current First-Level Groups
The current first-level menu groups are:
## Current First-Level Groups
- `data`: 数据管理
- `production_planning`: 生产规划
- `production_management`: 生产管理
- `insar_analysis`: InSAR形变分析
- `ai_analysis`: AI分析
- `flood_analysis`: 洪涝灾害分析
- `ops`: 运行维护
@@ -30,179 +16,106 @@ Definition files:
- `frontend/src/config/appConstants.js`
- `frontend/src/utils/appUiHelpers.js`
- `frontend/src/components/app/AppSidePanel.jsx`
- `frontend/src/ProductionWorkspace.jsx`
## 3. Navigation Model
The current frontend uses two navigation patterns:
1. Sectioned navigation:
first-level group -> second-level section -> leaf tab
2. Workspace navigation:
first-level group -> single leaf tab -> internal workspace view switcher
### 3.1 生产规划
This is a sectioned group.
## Data Management
```text
生产规划
├─ 规划编组
│ ├─ 配对规划 (`pairing`)
│ ├─ 任务规划 (`pairs`)
│ ├─ 时序候选栈 (`ps_results`)
│ └─ 任务批次 (`batches`)
└─ 数据分发
└─ 数据分发 (`copier`)
数据管理
├─ 入库监控 (`ingest`)
├─ 资产库存 (`asset_inventory`)
├─ 数据列表 (`data`)
└─ 灾害点 (`hazard`)
```
Notes:
Boundary:
- `ps_results` here means planning-stage candidate stacks, not analysis-facing result pages.
- This group no longer hosts D-InSAR production or product pages.
- LT-1 and Sentinel-1 source data are managed as local compressed archives.
- Metadata, footprint, and preview are extracted from archives without full unpacking.
- Full materialization happens only when production preparation creates a task under the local Task_Pool.
- GF-3 registers copied native `_geo` production results and generates local WebP previews.
### 3.2 生产管理
## Production Management
This is a workspace group, not a multi-tab planning tree.
`production_management` is a workspace group. The left navigation exposes one entry; the workspace owns the internal production views.
```text
生产管理
└─ 生产管理 (`production_management`)
├─ D-InSAR运行 (`dinsar_runs`)
├─ SBAS-InSAR Production (`sbas_insar_production`)
├─ SBAS-InSAR结果 (`sbas_insar_products`)
└─ D-InSAR产物 (`dinsar_products`)
生产管理 (`production_management`)
├─ D-InSAR配对规划 (`dinsar_pairing`)
├─ D-InSAR任务规划 (`dinsar_pairs`)
├─ D-InSAR任务批次 (`dinsar_batches`)
├─ D-InSAR生产准备/分发 (`dinsar_prepare`)
├─ D-InSAR运行 (`dinsar_runs`)
├─ D-InSAR产物 (`dinsar_products`)
├─ SBAS-InSAR Production (`sbas_insar_production`)
├─ SBAS-InSAR结果 (`sbas_insar_products`)
├─ 陆探生产占位 (`lt1_production`)
├─ 哨兵生产占位 (`sentinel1_production`)
└─ 高分三结果登记 (`gf3_native_registration`)
```
Notes:
Boundary:
- The left navigation contains only one tab for this group: `production_management`.
- Internal workspace views are controlled by `PRODUCTION_WORKSPACE_VIEWS`.
- Route alias mapping is controlled by `PRODUCTION_WORKSPACE_ENTRY_TO_VIEW`.
- Legacy route tabs such as `dinsar_production`, `ps_production`, and `ps_products` map into this workspace and should not be treated as standalone left-nav entries.
- The old ISCE2/MintPy `timeseries_runs` and `timeseries_products` workspace views are deprecated and hidden; SBAS production is handled by the Gamma `sbas_insar_production` view.
- D-InSAR uses the sequence: pair planning -> selected pairs -> D-InSAR batch -> production preparation -> run -> product catalog.
- D-InSAR production preparation materializes archive sources into `DINSAR_TASK_POOL_ROOT` and must not use UNC paths.
- Data distribution is a separate D-InSAR mode that exports source archive bundles under `DATA_DISTRIBUTION_ROOT`; it is not the production runtime path.
- SBAS-InSAR uses the dedicated Gamma/LandSAR SBAS production page. It does not depend on the old D-InSAR pair list or PS candidate-stack page.
- GF-3 is not produced on this server. The server registers native `_geo` results copied into the configured GF-3 pool and builds WebP from the produced binary raster, not from quicklook TIFFs.
### 3.3 InSAR形变分析
Compatibility route aliases:
This is a sectioned group.
- `pairing` -> `dinsar_pairing`
- `pairs` -> `dinsar_pairs`
- `ps_results` -> `sbas_insar_production`
- `batches` -> `dinsar_batches`
- `copier` -> `dinsar_prepare`
- `dinsar_production` -> `dinsar_runs`
- `dinsar_products` -> `dinsar_products`
- `ps_production` -> `sbas_insar_production`
- `ps_products` -> `sbas_insar_products`
These aliases exist so existing code paths can redirect into the workspace. They are not standalone left-navigation entries.
## InSAR Analysis
```text
InSAR形变分析
├─ D-InSAR
│ ├─ D-InSAR结果 (`dinsar_results`)
│ └─ D-InSAR分析 (`dinsar_analysis`)
└─ 时序InSAR
├─ 时序InSAR结果 (`psinsar_results`)
└─ 时序InSAR分析 (`psinsar_analysis`)
│ ├─ AI质量评估
│ └─ D-InSAR诊断
└─ SBAS
└─ SBAS-InSAR分析 (`psinsar_analysis`)
```
Notes:
Boundary:
- This group is for business-facing result browsing and interpretation.
- AI diagnosis does not belong here.
- `dinsar_analysis`, `psinsar_results`, and `psinsar_analysis` are currently reserved placeholders.
- Analysis pages consume registered result catalogs.
- They should not submit production jobs or materialize source archives.
- The standalone `AI分析` first-level page has been removed. D-InSAR quality assessment and D-InSAR diagnosis are owned by `dinsar_analysis`.
- D-InSAR diagnosis uses the `AI_DIAGNOSIS` task type and persists reports in the `ai_diagnosis` table. The older `AI_ANALYZE` endpoint/task is compatibility code only.
- `psinsar_analysis` remains the historical route key, but its user-facing meaning is SBAS-InSAR analysis.
### 3.4 AI分析
## Deprecated Visible Workflows
This is a sectioned group.
The following workflows must not be shown as primary UI entries:
```text
AI分析
├─ 形变智能分析
│ ├─ AI质量评估 (`ai_quality`)
│ └─ D-InSAR诊断 (`ai_diagnosis`)
└─ 遥感视觉分析
├─ 滑坡语义分割 (`landslide_segmentation`)
└─ 无人机影像分析 (`uav_image_analysis`)
```
- PS-InSAR production
- PS candidate-stack distribution
- legacy ISCE2/MintPy time-series production
- source-folder distribution for unpacked LT-1 or Sentinel-1 folders
- standalone AI analysis first-level navigation
- remote-sensing vision AI placeholder pages
Notes:
Backend compatibility code may remain until historical data models and catalog names are migrated.
- `ai_diagnosis` is the actual tab key; its display label is `D-InSAR诊断`.
- `landslide_segmentation` and `uav_image_analysis` remain reserved placeholders.
## Navigation Update Rules
### 3.5 无二级分组的一级入口
The following groups do not define second-level sections:
- `data`
leaf tabs: `ingest`, `data`, `hazard`
- `flood_analysis`
leaf tabs: `flood_analysis`
- `ops`
leaf tabs: `health`, `users`, `audit`
## 4. Source-Of-Truth Rules
The navigation follows these rules:
- `LEFT_GROUP_LABELS` defines the first-level group vocabulary.
- `LEFT_GROUP_SECTIONS` defines second-level sections where they exist.
- `LEFT_GROUP_TABS` defines which leaf tabs belong to each group.
- `LEFT_TAB_GROUP` and `LEFT_TAB_SECTION` are derived maps and should not be edited manually.
- `leftPanelTab` remains the route/state source of truth for the selected leaf entry.
- `production_management` is a special case: one left-nav tab owns multiple internal workspace views.
- New features should be added under an existing group whenever possible.
- A new first-level group should be introduced only for a durable, independent capability area.
## 5. Naming Rules
To avoid future ambiguity, use these naming constraints:
- Use `结果` for browsing, querying, and result-facing visualization pages.
- Use `产物` for extraction, publishing, packaging, and catalog-management pages.
- Use `运行` for task submission, engine selection, execution control, and runtime monitoring views.
- Use `分析` for interpretation, statistics, and analyst-facing thematic workflows.
- Use `诊断` for model-assisted fault analysis or AI-driven reasoning pages.
- Use `时序候选栈` only for planning-stage candidate stacks under `production_planning`.
- Use `时序InSAR结果` for analysis-facing result pages under `insar_analysis`.
## 6. Files To Update When Navigation Changes
When adding or moving a tab, update these files together:
- `frontend/src/config/appConstants.js`
Defines first-level groups, sections, tab ownership, workspace view mappings, and admin-only visibility.
- `frontend/src/utils/appUiHelpers.js`
Defines display labels for leaf tabs.
- `frontend/src/components/app/AppSidePanel.jsx`
Renders the side-panel navigation and group/section switching behavior.
- `frontend/src/App.jsx`
Connects route state with panel rendering.
- `frontend/src/ProductionWorkspace.jsx`
Owns the internal production workspace view switcher.
- `frontend/src/App.css`
Styles the navigation hierarchy and workspace entry state.
If the new tab is a real page instead of a placeholder, also add or update the corresponding panel component.
## 7. Reserved Entries And Legacy Route Aliases
Reserved leaf tabs:
- `dinsar_analysis`
- `psinsar_results`
- `psinsar_analysis`
- `landslide_segmentation`
- `uav_image_analysis`
Legacy route aliases mapped into `production_management`:
- `dinsar_production`
- `dinsar_products`
- `ps_production`
- `ps_products`
These aliases exist for compatibility, but they are not first-class left-nav entries anymore.
## 8. Future Extension Guidance
Recommended future additions:
- Use `flood_analysis` as the combined first-level group for water extraction, flood detection, overlay analysis, and flood results. The legacy `water` route may remain in code for compatibility, but it is no longer a first-class left-nav entry.
- Put new production execution or product-governance capability under `production_management` as an internal workspace view unless a separate first-level domain is clearly required.
- Put planning, batching, pairing, and dispatch preparation capability under `production_planning`.
- Put result browsing and analyst-facing deformation interpretation under `insar_analysis`.
- Put intelligent interpretation, diagnosis, segmentation, and computer-vision modules under `ai_analysis`.
If a new feature belongs to intelligent interpretation or computer vision, prefer `AI分析`.
If a new feature belongs to result browsing or deformation business analysis, prefer `InSAR形变分析`.
- Add production execution, preparation, product registration, and product catalog features inside `ProductionWorkspace`.
- Add source ingestion, archive scanning, orbit scanning, and storage inventory under `data`.
- Add result interpretation and map analysis under `insar_analysis`.
- Keep `production_management` as the only production first-level group.
- Do not reintroduce a separate `production_planning` first-level group.
- When changing navigation, update `appConstants.js`, `appUiHelpers.js`, `AppSidePanel.jsx`, `ProductionWorkspace.jsx`, and this document together.
docs/GF3_SARSCAPE_NATIVE_TO_GEOTIFF_DESIGN_20260530.md
+25 -34
View File
@@ -22,12 +22,13 @@ GF3 原始压缩包池
推荐把输入、生产结果和运行时目录分开，避免把系统工作文件混入业务结果池。
```env
GF3_ARCHIVE_SOURCE_DIRS=D:\production_inputs\gf3\archives
GF3_TASK_POOL_ROOT=D:\GaoFen3_Task_Pool
GF3_ARCHIVE_SOURCE_DIRS=D:\GaoFen3_Image_Pool\archives
GF3_LEGACY_GDAL_ENABLED=false
GF3_SOURCE_DIRS=
GF3_SARSCAPE_NATIVE_DIRS=D:\production_results\gf3\sarscape_native
GF3_STORAGE_DIRS=D:\production_results\gf3\standard_l2
GF3_SARSCAPE_RUNTIME_DIR=D:\production_runtime\gf3\sarscape_runtime
GF3_SARSCAPE_NATIVE_DIRS=D:\GaoFen3_Image_Pool\sarscape_native
GF3_STORAGE_DIRS=D:\GaoFen3_Image_Pool\standard_l2
GF3_SARSCAPE_RUNTIME_DIR=D:\GaoFen3_Task_Pool\sarscape_runtime
SAR_ANALYSIS_READY_ROOT=D:\production_results\sar_analysis_ready
```
@@ -49,7 +50,7 @@ SAR_ANALYSIS_READY_ROOT=D:\production_results\sar_analysis_ready
原生池以批次日期或人工批次号分组。单景目录名尽量保持 GF3 原始产品名。
```text
D:\production_results\gf3\sarscape_native
D:\GaoFen3_Image_Pool\sarscape_native
20260514
GF3_MH1_FSII_051377_E132.3_N48.2_20260514_L1A_HHHV_L10007356478
GF3_MH1_FSII_..._hh_geo
@@ -115,7 +116,7 @@ SLC 中间产物
系统从原生池转换后写入 `GF3_STORAGE_DIRS`。
```text
D:\production_results\gf3\standard_l2
D:\GaoFen3_Image_Pool\standard_l2
20260514
GF3_MH1_FSII_051377_E132.3_N48.2_20260514_L1A_HHHV_L10007356478
HH_L2.tif
@@ -456,7 +457,7 @@ GF3_SARSCAPE_IDLRT_PATH=C:\Program Files\Harris\ENVI56\IDL88\bin\bin.x86_64\idlr
GF3_SARSCAPE_DEM_PATH=D:\DEM\GMTED2010.jp2
GF3_SARSCAPE_POLARIZATIONS=HH,HV
GF3_SARSCAPE_KEEP_EXTRACTED=true
GF3_SARSCAPE_AUTO_STANDARDIZE=true
GF3_SARSCAPE_AUTO_STANDARDIZE=false
GF3_SARSCAPE_CLEAN_AFTER_SUCCESS=true
GF3_SARSCAPE_PRODUCE_TIMEOUT_SECONDS=0
```
@@ -465,8 +466,9 @@ GF3_SARSCAPE_PRODUCE_TIMEOUT_SECONDS=0
| 任务 | 接口 | 用途 |
| --- | --- | --- |
| `GF3_SARSCAPE_PRODUCE` | `POST /api/monitor/gf3-sarscape-produce` | 从原始 `.tar.gz/.tgz` 触发 SARscape 生产，随后自动标准化、入库、清理 |
| `GF3_SARSCAPE_SYNC` | `POST /api/monitor/gf3-sarscape-sync` | 仅扫描已有 `_geo` 原生结果并转 GeoTIFF 入库 |
| `GF3_SARSCAPE_PRODUCE` | `POST /api/monitor/gf3-sarscape-produce` | 停用；本机不触发 SARscape wrapper 生产 |
| `GF3_SARSCAPE_SYNC` | `POST /api/monitor/gf3-sarscape-sync` | 登记本机已有 `_geo` 原生结果；监控面板使用原生结果登记模式 |
| `GF3_QUICKLOOK_WEBP` | `POST /api/monitor/gf3-quicklook-webp` | 从已登记 `_geo` ENVI 二进制生成本机 WebP 预览缓存 |
| `GF3_SARSCAPE_CLEAN` | `POST /api/monitor/gf3-sarscape-clean` | 手动清理原生池中间数据 |
清理策略：
@@ -478,34 +480,23 @@ GF3_SARSCAPE_PRODUCE_TIMEOUT_SECONDS=0
- 每景写 `gf3_cleanup_manifest.json`，记录删除条目和释放字节数。
- 不删除 `GF3_ARCHIVE_SOURCE_DIRS` 中的原始压缩包，也不删除 `GF3_STORAGE_DIRS` 中的标准 GeoTIFF。
这样 `D:\production_results\gf3\sarscape_native` 只长期保存可追溯的最终 `_geo` 原生结果组，中间过程文件在标准化完成后自动释放空间；wrapper 配置和临时运行文件放在 `D:\production_runtime\gf3\sarscape_runtime`。
这样 `D:\GaoFen3_Image_Pool\sarscape_native` 只长期保存可追溯的最终 `_geo` 原生结果组，中间过程文件在标准化完成后自动释放空间；wrapper 配置和临时运行文件放在 `D:\GaoFen3_Task_Pool\sarscape_runtime`。
## 2026-06-15 Production Preflight Rule
## 2026-06-15 Local Production Stop Rule
GF3 SARscape production must check existing results before invoking the external wrapper.
GF3 SARscape production on this management machine is disabled.
Skip production when either condition is true:
- `POST /api/monitor/gf3-sarscape-produce` returns 409.
- `GET /api/monitor/gf3-sarscape-dates` returns 409 because date-scoped production selection is no longer used here.
- Existing queued `GF3_SARSCAPE_PRODUCE` jobs fail immediately and do not call `gf3wrapper.exe`.
- The durable result layer is the registered local `_geo` native result plus optional standardized L2 assets.
- SARscape native output is already complete in `GF3_SARSCAPE_NATIVE_DIRS` for every requested polarization.
- Standardized L2 output already exists in `GF3_STORAGE_DIRS/<imaging_date>/<scene_name>` with `gf3_standard_manifest.json` status `DONE`/`PARTIAL` and every requested polarization has a valid `*_L2.tif`.
## 2026-06-15 Local Native Registration Rule
This prevents reprocessing when native intermediates were cleaned but registered/standardized results still exist. The standardized L2 and registered assets are the durable result layer; source archives on UNC should not be reprocessed unless the operator explicitly removes or invalidates the existing result.
GF3 SARscape production is not run on this management machine.
## 2026-06-15 Date-Scoped Production
GF3 SARscape production now supports an optional scene-date filter.
- `GET /api/monitor/gf3-sarscape-dates` scans `GF3_ARCHIVE_SOURCE_DIRS`, groups wrapper-supported raw archives by the `YYYYMMDD` date embedded in the GF3 scene name, and returns scene counts per date.
- `POST /api/monitor/gf3-sarscape-produce` accepts `selected_dates: ["YYYYMMDD"]`. When omitted or empty, production keeps the previous all-date behavior.
- The frontend monitor panel exposes a date selector in the GF3 SARscape production section. Operators can select one image date before starting production, or leave it as all dates.
- The existing duplicate-result preflight still runs after date filtering. A selected date will not reprocess scenes whose standardized L2 result or complete native `_geo` outputs already exist.
## 2026-06-15 Local Task_Pool Staging
GF3 SARscape production no longer passes UNC archives directly to `gf3wrapper.exe`.
- Source archives may remain on UNC storage for source management.
- Before each wrapper run, the selected archive is copied to `GF3_TASK_POOL_ROOT\SARscape\<task_id>\<scene>\source\`.
- The wrapper receives the local staged archive path as `-input`.
- Native SARscape output still goes to `GF3_SARSCAPE_NATIVE_DIRS`, then standardization writes durable L2 GeoTIFFs to `GF3_STORAGE_DIRS`.
- This avoids network extraction stalls and makes source staging part of the local Task_Pool cleanup domain.
- Completed SARscape `_geo` result folders are copied to local `GF3_SARSCAPE_NATIVE_DIRS`.
- The monitor registration action scans real `_geo` ENVI binaries plus `.hdr/.sml` sidecars.
- `*_geo_ql.tif` remains an auxiliary quicklook only and is not used as the WebP source.
- WebP preview cache is generated locally from the `_geo` ENVI binary with rasterio.
- Full GeoTIFF standardization remains a separate explicit path and is not triggered by native-result registration.
docs/INDEX.md
+14 -2
View File
@@ -1,6 +1,6 @@
# 文档索引
最后更新：2026-06-14
最后更新：2026-06-16
本页是当前有效文档入口。没有列在本页的历史设计、实验记录和过程文档不再作为当前系统事实依据。
@@ -23,15 +23,25 @@
- [GLOBAL_TASK_STATUS_LOCK_REDESIGN_20260612.md](GLOBAL_TASK_STATUS_LOCK_REDESIGN_20260612.md)
全局界面锁降级为任务状态中心、功能级任务面板和后端资源锁的重构设计。
- [OLLAMA_DINSAR_DIAGNOSIS_DEPLOYMENT_20260620.md](OLLAMA_DINSAR_DIAGNOSIS_DEPLOYMENT_20260620.md)
D-InSAR 分析中 Ollama 本机 VLM 诊断的部署配置、模型选择和运行约定。
## 生产与结果
- [THREE_SENSOR_LOCAL_PRODUCTION_CONTRACT_20260616.md](THREE_SENSOR_LOCAL_PRODUCTION_CONTRACT_20260616.md)
当前陆探一号、Sentinel-1、高分三本机生产、按需解包、GF3 外部生产登记、结果管理和 UNC 退出约定。
- [PRODUCTION_RESULTS_MULTI_ENGINE_DESIGN_20260423.md](PRODUCTION_RESULTS_MULTI_ENGINE_DESIGN_20260423.md)
统一结果目录、标准产品包、catalog 与多引擎结果共存约定。D-InSAR 当前引擎集合以 2026-06-14 三引擎 Task_Pool 重构设计为准。
- [DINSAR_TASK_POOL_THREE_ENGINE_REFACTOR_20260614.md](DINSAR_TASK_POOL_THREE_ENGINE_REFACTOR_20260614.md)
D-InSAR 保留 ENVI/SARscape、LandSAR、Gamma/PyINT 三引擎，退出 ISCE2，统一 Task_Pool、结果聚合和中间文件清理的当前设计。
- [LANDSAR_DEM_PREPARATION_CONTRACT_20260618.md](LANDSAR_DEM_PREPARATION_CONTRACT_20260618.md)
LandSAR D-InSAR/SBAS 的全球 DEM 一次性 Int16 标准化、区域裁剪 tif、生产配置和 guardrail 约定。
- [UNC_SOURCE_ARCHIVE_AND_MATERIALIZE_DESIGN_20260615.md](UNC_SOURCE_ARCHIVE_AND_MATERIALIZE_DESIGN_20260615.md)
UNC/SMB 源压缩包管理、包内 XML/manifest 资产化、本地 materialize 和 D-InSAR/SBAS 生产边界。
LT-1/Sentinel-1 本地源压缩包管理、包内 XML/manifest 资产化、本地 Task_Pool materialize，以及 UNC 退出运行链路后的本机部署边界。
- [SOURCE_ARCHIVE_INTEGRITY_AUDIT_20260620.md](SOURCE_ARCHIVE_INTEGRITY_AUDIT_20260620.md)
LT-1/Sentinel-1 源压缩包完整性审计的独立任务、增量语义、数据库字段和问题登记规则。
- [DINSAR_PRODUCTION_CORES_OVERVIEW.md](DINSAR_PRODUCTION_CORES_OVERVIEW.md)
旧版 ENVI/SARscape、ISCE2、Gamma/PyINT D-InSAR 生产核心说明。ISCE2 相关内容仅作历史背景。
@@ -57,6 +67,8 @@
- [SENTINEL1_SOURCE_ORBIT_ASSET_DESIGN_20260512.md](SENTINEL1_SOURCE_ORBIT_ASSET_DESIGN_20260512.md)
Sentinel-1 / LT-1 源数据与精密轨道资产层设计。
- [PRECISE_ORBIT_PRODUCTION_CONTRACT_20260617.md](PRECISE_ORBIT_PRODUCTION_CONTRACT_20260617.md)
Current LT-1/Sentinel-1 precise-orbit source assets, LT-1 production TXT pools, Gamma/PyINT, LandSAR, and retired ISCE2 orbit boundaries.
- [FLOOD_GEOTIFF_GAMMA_PREPROCESS_DESIGN_20260515.md](FLOOD_GEOTIFF_GAMMA_PREPROCESS_DESIGN_20260515.md)
洪涝模块 GeoTIFF 化与 Gamma 前处理方向。
docs/LANDSAR_DEM_PREPARATION_CONTRACT_20260618.md
+63
View File
@@ -0,0 +1,63 @@
# LandSAR DEM Preparation Contract
Updated: 2026-06-18
## Decision
LandSAR D-InSAR and LandSAR SBAS must not run directly against the full global DEM.
The maintained flow is:
1. Convert the global DEM once into an uncompressed Int16 GeoTIFF.
2. Configure LandSAR with that global Int16 GeoTIFF as the DEM source.
3. Before each LandSAR task executes, crop a task-level DEM from the global Int16 source.
4. Write the task-level crop path into `200014.txt` or `280039.txt`.
The global Int16 file is a reusable source DEM. The actual LandSAR console input is the small crop stored under the run work directory.
## Paths
Prepared global DEM source:
```text
D:\DEM\SRTMDEM_RSP_SARscape_global_int16.tif
```
Runtime task crop location:
```text
<run work root>\...\dem_crop\<task>_<bbox-hash>_dem.tif
```
Production configuration:
```text
LANDSAR_DEM_PATH=D:\DEM\SRTMDEM_RSP_SARscape_global_int16.tif
LANDSAR_SBAS_DEM_PATH=D:\DEM\SRTMDEM_RSP_SARscape_global_int16.tif
```
## Commands
One-time global conversion:
```powershell
Set-Location D:\Code\Insar_management_system_v2; & C:\ProgramData\anaconda3\envs\InSAR\python.exe scripts\prepare_landsar_dem_int16.py --source D:\DEM\SRTMDEM_RSP_SARscape.wgs84 --target D:\DEM\SRTMDEM_RSP_SARscape_global_int16.tif --block-size 1024 --overwrite
```
Optional manual crop test from the prepared global GeoTIFF:
```powershell
Set-Location D:\Code\Insar_management_system_v2; & C:\ProgramData\anaconda3\envs\InSAR\python.exe scripts\prepare_landsar_dem_int16.py --source D:\DEM\SRTMDEM_RSP_SARscape_global_int16.tif --crop-only --bbox 120,42,136,54 --target D:\DEM\landsar_prepared\SRTMDEM_RSP_SARscape_ne_china_int16.tif --block-size 2048 --overwrite
```
The manual crop command is for verification or emergency operation. Normal D-InSAR/SBAS production performs task-level cropping automatically.
## Guardrails
- `LANDSAR_DEM_PATH` and `LANDSAR_SBAS_DEM_PATH` point to the global prepared Int16 GeoTIFF source.
- D-InSAR derives the crop bbox from master/slave LT-1 XML corner coordinates.
- LandSAR SBAS derives the crop bbox from all selected `Input_Data` LT-1 XML corner coordinates.
- The crop bbox is expanded by a margin before writing the task DEM.
- The source DEM must be Int16; Float/ENVI sources are rejected at runtime.
- Each crop writes a JSON manifest next to the crop tif.
- Run metadata records both `dem_source_path` and the actual task `dem_path`.
docs/OLLAMA_DINSAR_DIAGNOSIS_DEPLOYMENT_20260620.md
+97
View File
@@ -0,0 +1,97 @@
# Ollama D-InSAR Diagnosis Deployment
Last updated: 2026-06-20
This document is the current deployment contract for local Ollama integration in the D-InSAR analysis workflow.
## Scope
- Ollama is used only for D-InSAR diagnosis and map/image interpretation tasks.
- The visible UI entry is `InSAR形变分析 / D-InSAR / D-InSAR分析 / D-InSAR诊断`.
- The standalone `AI分析` first-level page is retired.
- Quality model training and batch quality prediction remain local backend tasks. They do not call Ollama.
## Configuration
Set these values in the backend environment:
```env
OLLAMA_BASE_URL=http://127.0.0.1:11434
OLLAMA_API_URL=http://127.0.0.1:11434/api/generate
DEFAULT_VLM_MODEL=qwen3-vl:30b
```
The backend reads them through `backend/app/config.py`.
`OLLAMA_BASE_URL` must point to the server-local Ollama service. Do not point it to UNC or a workstation share. `OLLAMA_API_URL` should normally be `${OLLAMA_BASE_URL}/api/generate`.
## Model Selection
`GET /ai/status` checks `${OLLAMA_BASE_URL}/api/tags` and returns:
- `ollama_online`
- `ollama_models`
- `ollama_vlm_models`
- `ollama_base_url`
- `default_vlm_model`
The D-InSAR diagnosis panel uses `ollama_vlm_models` for its model dropdown. `ollama_models` is the raw installed-model list and may include pure text models. If `DEFAULT_VLM_MODEL` is installed and classified as a vision model, it is selected. Otherwise the first detected vision model is selected.
The backend still has a fallback detector for compatibility. Preference order is:
1. User-selected model, if present in Ollama.
2. Model name containing `qwen3-vl`.
3. Model name containing `qwen2-vl`.
4. Model name containing `minicpm-v`.
5. Model name containing `llama3.2-vision` or `llava`.
6. Any model name containing `vl`, `vision`, or `llava`.
7. `DEFAULT_VLM_MODEL`.
## Runtime Flow
1. User opens `D-InSAR分析`.
2. User selects `D-InSAR诊断`.
3. Frontend calls `POST /ai/diagnosis`.
4. Backend creates an `AI_DIAGNOSIS` task and job.
5. Job handler reads the registered D-InSAR preview image, injects spatial context and quality context into the prompt, then calls Ollama `/api/generate`.
6. The diagnosis report is written to `ai_diagnosis`.
7. The panel lists diagnosis records from `GET /ai/diagnosis`.
The older `POST /ai/analyze-result/{result_id}` and `AI_ANALYZE` task remain compatibility code. New UI should use `POST /ai/diagnosis` and `AI_DIAGNOSIS`.
## Deployment Check
Run these checks on the server:
```powershell
ollama list
curl http://127.0.0.1:11434/api/tags
```
Then check the system endpoint:
```powershell
curl http://127.0.0.1:8000/api/ai/status
```
Expected result:
- `ollama_online` is `true`.
- `ollama_vlm_models` contains at least one vision-capable model.
Recommended model families for this project:
- `qwen3-vl`
- `qwen2-vl`
- `minicpm-v`
- `llama3.2-vision`
- `llava`
Avoid pure text models such as `qwen2`, `llama3`, `mistral`, or `gemma` for D-InSAR diagnosis.
## Failure Handling
- If the panel shows Ollama offline, verify `ollama serve` is running and the configured port matches `.env`.
- If diagnosis stays queued or fails quickly, inspect the task log for `AI_DIAGNOSIS`.
- If the model dropdown is empty, `/api/tags` is unreachable or Ollama has no models installed.
- If a selected model fails at generation time, confirm the model supports image input.
docs/PRECISE_ORBIT_PRODUCTION_CONTRACT_20260617.md
+109
View File
@@ -0,0 +1,109 @@
# Precise Orbit Production Contract 2026-06-17
This document is the current contract for LT-1 and Sentinel-1 precise orbit management. UNC paths are not allowed in active source, orbit, task, or production paths.
## Two Layers
There are two different orbit layers.
`ORBIT_SOURCE_DIRS` is the source asset layer. It is scanned into `orbit_assets` and used for scene-orbit binding:
```env
ORBIT_SOURCE_DIRS=D:\LT1_data_lsarorbit;D:\Sentinel1_EOF_Pool
```
`ORBIT_POOL_ENVI` / `PYINT_ORBIT_POOL_TXT` / `GAMMA_SBAS_ORBIT_ROOTS` are LT-1 production orbit pools. They are local TXT pools consumed by ENVI/SARscape and Gamma/PyINT:
```env
ORBIT_POOL_ENVI=D:\orbit_pools\envi
PYINT_ORBIT_POOL_TXT=D:\orbit_pools\envi
GAMMA_SBAS_ORBIT_ROOTS=D:\orbit_pools\envi
```
Expected LT-1 production layout:
```text
D:\orbit_pools\envi
LT1A\
LT1A_GpsData_GAS_C_YYYYMMDD.txt
LT1B\
LT1B_GpsData_GAS_C_YYYYMMDD.txt
```
`LT1A` and `LT1B` are satellite names, not product levels.
## Scan Semantics
The active scan entry is `/assets/inventory/scan`.
- `inventory_types=["orbit_asset"], families=["LT1"]` scans LT-1 orbit TXT files.
- `inventory_types=["orbit_asset"], families=["S1"]` scans Sentinel-1 EOF files.
- `inventory_types=["orbit_asset"], families=["LT1","S1"]` scans both.
LT-1 scanning recognizes `LT1A_GpsData_GAS_C_YYYYMMDD.txt` and `LT1B_GpsData_GAS_C_YYYYMMDD.txt`.
Sentinel-1 scanning recognizes `S1*_OPER_AUX_*.EOF` and matches EOF validity windows to scene acquisition windows.
After an LT-1 orbit scan, the scanner also synchronizes the LT-1 production TXT pool under `ORBIT_POOL_ENVI`. This keeps Gamma/PyINT and Gamma SBAS able to find the same TXT files without relying on the old monitor scan.
Orbit asset scans are incremental. For already indexed TXT/EOF files, the scanner skips metadata parsing and database upsert when the file path, size, mtime, parser version, active flag, and `parse_status=OK` still match the database record. Missing files are still marked inactive by comparing the scanned `seen_paths` set with existing assets under the same managed root.
## Engine Consumers
ENVI/SARscape D-InSAR:
- Uses local LT-1 data prepared for the SARscape workflow.
- The LT-1 TXT production pool is `ORBIT_POOL_ENVI`.
- The pool must be split by satellite because ENVI-side tools expect stable satellite folders.
Gamma/PyINT D-InSAR:
- Reads LT-1 TXT orbit files from `PYINT_ORBIT_POOL_TXT`.
- Current default keeps `PYINT_ORBIT_POOL_TXT=ORBIT_POOL_ENVI`.
- For LT-1, input assets may stage TXT orbits into the task input manifest when the precise-orbit bridge is enabled.
- For Sentinel-1, EOF paths come from source/orbit asset binding or task `orbit` staging.
Gamma SBAS:
- For LT-1, reads TXT orbit roots from `GAMMA_SBAS_ORBIT_ROOTS`.
- For Sentinel-1, planning uses `ORBIT_SOURCE_DIRS` EOF roots; S1 SBAS execution is not enabled.
- LT-1 Gamma SBAS scripts use the orbit path recorded in scene discovery.
LandSAR:
- Current D-InSAR integration does not independently scan an orbit pool.
- It consumes already prepared LT-1 task input.
- `ORBIT_POOL_LANDSAR` is not an active synchronization target in current code.
ISCE2:
- ISCE2 is retired from the active D-InSAR production path.
- `ORBIT_POOL_ISCE2` is legacy only and should be empty unless `ISCE2_ENABLED=true`.
- When `ISCE2_ENABLED=false`, health and orbit status must not treat missing ISCE2 XML as a production error.
## Database State
`orbit_assets` records original orbit files from `ORBIT_SOURCE_DIRS`.
`scene_orbit_bindings` records candidate and selected scene-orbit matches.
`radar_data.selected_orbit_asset_id`, `radar_data.orbit_binding_status`, `radar_data.has_orbit_data`, and `radar_data.orbit_file_path` are compatibility fields for production and search.
`orbit_asset_derivatives` records production-pool derivatives. For current LT-1 TXT production, derivative records use:
```text
engine_code=lt1_txt_pool
derivative_format=LT1_TXT
derivative_role=production_orbit_txt
pool_path=D:\orbit_pools\envi\LT1A|LT1B\*.txt
```
## Current Defaults
```env
ISCE2_ENABLED=false
ORBIT_POOL_ISCE2=
ORBIT_POOL_LANDSAR=
```
The old `/monitor/run-now?target=orbit` path is legacy. It may still synchronize the LT-1 production pool for compatibility, but new UI should use asset inventory orbit scans.
docs/SBAS_INSAR_CURRENT_WORKFLOW.md
+1 -1
View File
@@ -276,4 +276,4 @@ D:\Task_Pool\SBAS\<stack_task>
└─ publish
```
Source archives remain on UNC. Selected scenes and orbit files are materialized under the SBAS task directory before Gamma/LandSAR execution. Cleanup may remove `sources`, `orbits`, and `work` after result registration, but must preserve manifests, `publish`, previews, and catalog assets.
LT-1 and Sentinel-1 source archives remain local. Selected scenes and orbit files are materialized under the SBAS task directory before Gamma/LandSAR execution. Cleanup may remove `sources`, `orbits`, and `work` after result registration, but must preserve manifests, `publish`, previews, and catalog assets.
docs/SENTINEL1_SOURCE_ORBIT_ASSET_DESIGN_20260512.md
+1 -1
View File
@@ -345,7 +345,7 @@ ORBIT_SOURCE_DIRS=
对用户当前样本，推荐后续配置形态是：
```text
SOURCE_PRODUCT_DIRS=D:\LuTan1_Image_Pool;D:\Sentinel1_Image_Pool_ZIP
SOURCE_PRODUCT_DIRS=D:\LuTan1_Image_Pool_Zip;D:\Sentinel1_Image_Pool_ZIP
ORBIT_SOURCE_DIRS=D:\LT1_data_lsarorbit;D:\Sentinel1_EOF_Pool
```
docs/SOURCE_ARCHIVE_INTEGRITY_AUDIT_20260620.md
+67
View File
@@ -0,0 +1,67 @@
# Source Archive Integrity Audit
Date: 2026-06-20
This document defines the LT-1 / Sentinel-1 source archive integrity audit flow. It is separate from normal source asset inventory scanning.
## Boundary
Normal asset inventory scan remains lightweight:
- Recursively discovers local `SOURCE_PRODUCT_DIRS`.
- Extracts LT-1 XML or Sentinel-1 `manifest.safe` metadata from archives.
- Builds metadata documents, preview caches, radar scene records, and orbit bindings.
- Uses `file_path + size_bytes + mtime_epoch + parser_version + parse_status` to skip unchanged archives.
Archive integrity audit is an explicit background task:
- Task type: `AUDIT_SOURCE_ARCHIVE_INTEGRITY`.
- API: `POST /assets/inventory/archive-integrity-audit`.
- Default formats: `LT1_ARCHIVE`, `S1_ZIP`.
- Default families: caller supplied; UI uses LT-1 and Sentinel-1.
- It can be started from the asset inventory panel or data monitor panel.
This prevents every metadata scan from fully reading multi-GB archives.
## Validation Method
ZIP archives use Python `zipfile.ZipFile.testzip()` and safe member path validation.
TAR, TGZ, and TAR.GZ archives use Python `tarfile.open(..., "r:*")` and full member iteration. For gzip-compressed tar streams, reaching EOF through `tarfile` validates the gzip stream CRC/truncation state. TAR members are also checked for unsafe paths, links, devices, and other unsupported special member types.
The audit records:
- `archive_integrity_status`: `NOT_CHECKED`, `OK`, `FAILED`, `UNSUPPORTED`
- `archive_integrity_method`
- `archive_integrity_checked_at`
- `archive_integrity_error`
- `archive_integrity_version`
- `archive_integrity_member_count`
## Incremental Semantics
An archive is skipped when all conditions hold:
- `force=false`
- `size_bytes` and `mtime_epoch` still match the filesystem
- `archive_integrity_version` equals the current audit version
- previous status is `OK`, `FAILED`, or `UNSUPPORTED`
If the archive changes or the audit version changes, the audit runs again. When normal metadata scanning reparses a changed source archive, it resets the integrity fields to `NOT_CHECKED`.
## Issue Handling
Audit failures create an open `asset_inventory_issues` row:
- `inventory_type=source_product`
- `asset_ref_id=<source_product_assets.id>`
- `issue_code=source_archive_integrity_failed`
- `severity=error`
When a later audit passes or marks the archive unsupported, the previous open integrity issue for that source asset is resolved.
## Operational Notes
The audit is intentionally I/O heavy. A 1.22 GB LT-1 `tar.gz` test archive with 8 members completed a full stream audit in about 9.8 seconds on the current server. Full-pool audits should be run manually, not on backend startup.
The original source archive remains the source of record and must not be deleted after materialization.
docs/THREE_SENSOR_LOCAL_PRODUCTION_CONTRACT_20260616.md
+121
View File
@@ -0,0 +1,121 @@
# 三数据本机生产与结果管理约定
最后更新：2026-06-17
本文件是陆探一号、Sentinel-1、高分三在当前系统中的运行边界。2026-06-16 起，UNC 不再作为活动生产链路、源数据池、精轨池或 Task_Pool。网络共享只可作为人工搬运的外部介质，不进入后台生产任务。
## 1. 总原则
1. 源数据和精轨都放在本机路径，UNC 不进入活动生产链路。
2. LT-1 和 Sentinel-1 当前管理对象是本机压缩包源池，不管理旧解包目录。
3. 管理阶段只从压缩包中抽取 XML/manifest、元数据和预览图，用于资产索引、日期/轨道绑定和检索。
4. 生产任务需要真实文件树时，才按任务解包或复制到本机 `Task_Pool`，不从 UNC 拉取。
5. D-InSAR 和 SBAS-InSAR 的运行材料、工作目录、结果发布目录必须是本机路径。
6. 生产结果进入数据管理或结果 catalog，InSAR 形变分析只读取已登记结果，不直接扫描临时工作目录。
7. 洪水检测模块本轮冻结，不纳入本次审计和改造。
## 2. 三类数据边界
| 数据 | 源数据位置 | 精轨位置 | 生产模块 | 结果管理 |
| --- | --- | --- | --- | --- |
| 陆探一号 LT-1 | `D:\LuTan1_Image_Pool_Zip`，本机 LT-1 压缩包源池 | `D:\LT1_data_lsarorbit` | 生产管理保留占位；D-InSAR 走 LandSAR、ENVI+SARscape、Gamma/PyINT；SBAS 走 Gamma/PyINT 主线 | D-InSAR/SBAS 结果 catalog |
| Sentinel-1 | `D:\Sentinel1_Image_Pool_ZIP`，本机 Sentinel-1 ZIP/SAFE 压缩包源池 | `D:\Sentinel1_EOF_Pool` | 生产管理保留占位；D-InSAR 当前只走 Gamma/PyINT；SBAS 只做发现和规划，执行未启用 | D-InSAR 结果 catalog；SBAS 暂无执行产物 |
| 高分三 GF3 | 本机只管理已生产的 `_geo` 成品；原始归档仅追踪 | 无精轨链路 | 外部 SARscape 服务器生产 `_geo`；本机不启动 SARscape wrapper | 复制到 `D:\GaoFen3_Pool\native_geo` 后登记，WebP 从 `_geo` 主二进制生成 |
高分三外部结果命名约定：
```text
D:\GaoFen3_Pool\native_geo\20260609_geo\
GF3_MDJ_FSI_051759_E130.2_N43.5_20260609_L1A_HHHV_L10007375467\
GF3_MDJ_FSI_051759_E130.2_N43.5_20260609_L1A_HHHV_L10007375467_hh_geo
GF3_MDJ_FSI_051759_E130.2_N43.5_20260609_L1A_HHHV_L10007375467_hh_geo.hdr
GF3_MDJ_FSI_051759_E130.2_N43.5_20260609_L1A_HHHV_L10007375467_hh_geo.sml
```
`*_geo_ql.tif` 可作为辅助材料，但不作为正式 WebP 预览源。正式预览从 SARscape `_geo` ENVI 二进制读取生成。
### GF3 成品池扫描入库流程
1. 外部 SARscape 服务器完成生产后，把整景目录复制到本机 `D:\GaoFen3_Pool\native_geo\YYYYMMDD_geo\<GF3_SCENE>`。
2. 每个场景目录至少包含一个完整极化的 `*_geo`、`*_geo.hdr`、`*_geo.sml`；`*_geo_ql.tif` 可以同时提供，作为范围读取兜底和人工核验材料。
3. 在系统中点击 `登记 _geo 结果`，后端递归扫描所有 `*_geo` 场景目录，写入 `gf3_native_manifest.json` 和 `gf3_native_preview_manifest.json`，并登记 `source_product_assets` 与 `radar_data`。
4. 登记时优先从 `*_geo` ENVI 主数据读取 CRS、范围和中心点；如果主数据无法读取范围，再尝试对应的 `*_geo_ql.tif`。
5. 点击 `生成 WebP` 后，系统从已登记的 `*_geo` 主数据生成本机 WebP 缓存，缓存路径仍使用 `backend\image_cache\radar_raw` / `radar_geo` 体系。
6. `D:\GaoFen3_Pool\catalog` 只保存平台登记 manifest 和后续可选派生物；默认不复制完整影像、不把 `_geo` 转成全量 GeoTIFF。
## 3. 本机路径与按需解包
当前核心路径：
```text
UNPACK_SOURCE_DIRS=
SOURCE_PRODUCT_DIRS=D:\LuTan1_Image_Pool_Zip;D:\Sentinel1_Image_Pool_ZIP
SENTINEL1_STORAGE_DIRS=
INSAR_STORAGE_DIRS=
MONITOR_RADAR_DIRS=
ORBIT_SOURCE_DIRS=D:\LT1_data_lsarorbit;D:\Sentinel1_EOF_Pool
ORBIT_POOL_ENVI=D:\orbit_pools\envi
PYINT_ORBIT_POOL_TXT=D:\orbit_pools\envi
GAMMA_SBAS_ORBIT_ROOTS=D:\orbit_pools\envi
ISCE2_ENABLED=false
ORBIT_POOL_ISCE2=
TASK_POOL_ROOT=D:\Task_Pool
DINSAR_TASK_POOL_ROOT=D:\Task_Pool\DInSAR
SBAS_TASK_POOL_ROOT=D:\Task_Pool\SBAS
DATA_DISTRIBUTION_ROOT=D:\Task_Pool\Data_Distribution
GF3_TASK_POOL_ROOT=D:\GaoFen3_Pool\task_pool
GF3_ARCHIVE_SOURCE_DIRS=D:\GaoFen3_Pool\archives
GF3_SARSCAPE_NATIVE_DIRS=D:\GaoFen3_Pool\native_geo
GF3_STORAGE_DIRS=D:\GaoFen3_Pool\catalog
GF3_SARSCAPE_RUNTIME_DIR=D:\GaoFen3_Pool\task_pool\sarscape_runtime
```
压缩包资产索引只建立资产索引、预览缓存和轨道绑定，不提供“全量入库/全量解包”按钮。当前 LT-1/Sentinel-1 主流程从本机压缩包源池扫描，只抽取 XML/manifest、元数据和预览图；参与计算时才按任务解包或复制到 `Task_Pool` 或具体任务目录。
生产准备和数据分发是两个归口。前端不让用户输入服务器绝对路径，只填写任务名；后端按 `.env` 中固定根目录创建子目录，避免远程浏览器把用户本机路径传给服务器。`生产数据准备` 以批次配对为输入，从 LT-1/Sentinel-1 源压缩包按需 materialize 到 `DINSAR_TASK_POOL_ROOT\<任务名>`，在其下生成 `Task_YYYYMMDD_YYYYMMDD\master`、`slave`、`orbit` 和 `pair_metadata.json`，供 D-InSAR 引擎直接运行；这是生产工作区，不是源池。`数据分发` 只用于外发或跨目录搬运源压缩包，导出到 `DATA_DISTRIBUTION_ROOT\<任务名>`，目录结构为 `data/`、`orbit/`、`pairs.json`、`manifest.json`。其中 `data/` 只能保存 LT-1/Sentinel-1 源压缩包文件，不能保存旧解包目录。旧批次如果仍指向 `D:\LuTan1_Image_Pool` 或 `D:\Sentinel1_Image_Pool` 解包目录，应重新从压缩包资产重建批次后再执行生产准备或分发。
源池扫描采用增量解析语义：系统仍递归列出 `SOURCE_PRODUCT_DIRS` 下的候选压缩包文件名，用于发现新增和删除；数据库中已有且 `file_path`、`size_bytes`、`mtime_epoch`、`parser_version`、`parse_status` 均满足未变化条件的 `S1_ZIP` / `LT1_ARCHIVE` 资产会跳过包内 XML/manifest 读取，不再每次全量重读压缩包内容。新增、修改、曾经失效或解析器版本变化的压缩包会重新解析并更新资产索引。
LT-1 解析字段必须区分：规范文件名中的 `SLC/SSC` 是 `product_type` 和干涉源类型判断依据；XML 中 `imageDataInfo/imageDataType=COMPLEX` 只能写入 `image_data_type`，不能覆盖 `product_type`。前端“范围/可用性”依赖 `coverage_polygon + imaging_date + imaging_mode + polarization + complex token`，因此解析器变更必须同时验证 LT-1 `coverage_polygon` 和 `insar_source_ready` 计数。
LT-1/GF3 XML 的四个 `sceneCornerCoord` 不得直接按 XML 出现顺序连线。解析器必须把角点重排为非自交闭合四边形后再写入 `coverage_polygon` 和 PostGIS `geom`；验收时需检查 Shapely/PostGIS polygon valid，避免前端显示成沙漏形。
资产扫描入口支持三种语义：`families=["LT1"]` 只扫陆探源包/精轨，`families=["S1"]` 只扫哨兵源包/精轨，空 `families` 表示合扫。精密轨道扫描同样按 `inventory_types=["orbit_asset"] + families` 区分 LT-1、S1 或全部，前端不得再通过固定路径顺序猜测 root id。
精轨生产池按 [PRECISE_ORBIT_PRODUCTION_CONTRACT_20260617.md](PRECISE_ORBIT_PRODUCTION_CONTRACT_20260617.md) 执行：`ORBIT_SOURCE_DIRS` 是原生资产层；LT-1 扫描后同步到 `ORBIT_POOL_ENVI\LT1A|LT1B`，供 ENVI/SARscape、Gamma/PyINT D-InSAR 和 Gamma SBAS 使用；Sentinel-1 使用 EOF 原生资产绑定；ISCE2 精轨 XML 池为 legacy，默认不启用。
`UNPACK_SOURCE_DIRS` 为空是当前设计状态；通用全量解包入口不进入前端主流程。压缩包源池由 `SOURCE_PRODUCT_DIRS` 管理。
已执行的代码约束：
- `validate_runtime_config()` 对源数据、精轨、Task_Pool、D-InSAR/SBAS 工作根、结果根、GF3 `_geo` 根执行 UNC 校验。
- Sentinel-1 单个/批量解包和通用 materialize 拒绝 UNC 源路径与 UNC 目标路径。
- `/monitor/status` 返回按实际磁盘卷汇总的 `storage_roots`，例如多个 `D:\...` 路径只展示一个 `D:\` 容量项，同时报告配置路径数量和缺失路径数量。
## 4. 生产管理界面
生产管理工作台现在承担三类数据的生产边界展示：
- `陆探生产占位`：说明 LT-1 本机压缩包源池、精轨、按需解包、D-InSAR/SBAS 本机 Task_Pool。
- `哨兵生产占位`：说明 Sentinel-1 本机压缩包源池、EOF 精轨、按需解包、D-InSAR Gamma/PyINT、SBAS 规划态。
- `高分三结果登记`：说明外部 SARscape 生产、本机 `_geo` 登记和 WebP 生成。
- `D-InSAR 运行`、`D-InSAR 产物`、`SBAS-InSAR Production`、`SBAS-InSAR 结果` 保留现有生产和结果 catalog 功能。
这些占位不是最终生产向导，但先把三类数据放进同一生产管理域，避免继续把数据扫描、生产运行、结果登记混在数据监控按钮里。
## 5. InSAR 形变分析审计
当前边界：
- D-InSAR 结果由 D-InSAR product catalog 管理，分析页不应直接读取 Task_Pool 临时目录。
- SBAS 结果由 SBAS product catalog 管理，速率图、质量指标和监测点曲线从发布包读取。
- Sentinel-1 SBAS 目前只是规划态，不能在形变分析中伪装成可执行生产链路。
- GF3 `_geo` 登记到雷达资产和 WebP 缓存后，可作为数据管理资产；它不是 D-InSAR/SBAS 形变分析的生产输入。
后续如果要把分析页做成正式工作台，应先统一读取结果 catalog，再补地图叠加、剖面、时间序列和质量过滤，不应回退到扫描任意目录。
## 6. 本轮不改内容
- 洪水检测、GF3 水体检测和洪水 GeoTIFF 预处理暂不调整。
- D-InSAR 三引擎内部执行细节不在本轮重写，只强化本机路径边界。
- Sentinel-1 SBAS 不启用执行，只保留发现和规划。
docs/UNC_SOURCE_ARCHIVE_AND_MATERIALIZE_DESIGN_20260615.md
+96 -159
View File
@@ -1,29 +1,35 @@
# UNC Source Archive and Local Materialize Design
# Local-Only Source Archive and Task_Pool Materialize Design
Current canonical operating contract: [THREE_SENSOR_LOCAL_PRODUCTION_CONTRACT_20260616.md](THREE_SENSOR_LOCAL_PRODUCTION_CONTRACT_20260616.md). This file remains the implementation detail for local archive metadata extraction and Task_Pool materialization.
## Decision
UNC/SMB storage is treated as the source archive pool. Production engines should not use UNC paths as their working input. D-InSAR, SBAS, Gamma/PyINT, LandSAR, and SARscape should consume local materialized task inputs.
As of 2026-06-15, UNC/SMB storage is removed from the active production path.
This keeps the 20 TB storage useful for long-term source management while protecting production from SMB disconnects, credential scope, WSL path conversion, and external engine UNC compatibility.
The switch throughput is too low for production data movement. Large source archives, precise orbit files, GF3 native result pools, task staging, and engine inputs must all live on local disks. A network share may exist outside this system for manual backup or manual transfer, but it must not be configured in runtime environment variables used by backend scans, inventory, materialization, or production.
Production engines consume local Task_Pool inputs. If a selected LT-1 or Sentinel-1 source product is archived, it is extracted from a local source archive into `D:\Task_Pool` or a task-specific subdirectory before engine execution.
The local source archive remains the source of record. LT-1 and Sentinel-1 materialization must never delete the original archive after extraction.
## Current Implementation
- Source asset inventory now recognizes archive assets:
- Source asset inventory recognizes archive assets:
- `S1_ZIP`
- `LT1_ARCHIVE`
- `GF3_ARCHIVE`
- Sentinel-1 ZIP manifest parsing already reads `manifest.safe` directly from the ZIP.
- Sentinel-1 ZIP manifest parsing reads `manifest.safe` directly from the ZIP.
- LT-1 archive parsing reads `*.meta.xml` directly from `.zip`, `.tar`, `.tar.gz`, or `.tgz` and records contained TIFF members.
- GF3 archive parsing reads the first XML member directly from `.zip`, `.tar`, `.tar.gz`, or `.tgz` and records quicklook-like members when present.
- `GF3_ARCHIVE_SOURCE_DIRS` roots are included in asset inventory scans as source pools.
- Source asset listing and inventory counts now include archive assets instead of hiding `S1_ZIP`.
- `SOURCE_PRODUCT_DIRS` is the local LT-1/Sentinel-1 source archive inventory root.
- Source asset listing and inventory counts include archive assets instead of hiding `S1_ZIP`.
- A generic source materialize endpoint exists:
- `POST /api/assets/sources/{asset_id}/materialize`
- `S1_ZIP` uses the existing Sentinel-1 SAFE unpacker.
- `LT1_ARCHIVE` and `GF3_ARCHIVE` extract to a local materialized directory.
- Directory assets return `DIRECTORY_READY`.
- If no `target_root` is supplied, generic materialize defaults to `TASK_POOL_ROOT\source_materialized\<sensor>`.
Default source materialization is task-scoped. D-InSAR and SBAS callers should pass a Task_Pool target directory:
Default source materialization is local and task-scoped. D-InSAR and SBAS callers should pass a Task_Pool target directory:
```text
D:\Task_Pool\DInSAR\<task>\master
@@ -31,84 +37,65 @@ D:\Task_Pool\DInSAR\<task>\slave
D:\Task_Pool\SBAS\<stack>\sources\<YYYYMMDD>
```
The generic materialize endpoint still accepts `target_root` for ad hoc checks. Production callers must provide a Task_Pool destination.
The generic materialize endpoint still accepts `target_root` for ad hoc checks. Production callers should provide a task-specific Task_Pool destination.
## Production Boundary
D-InSAR and SBAS should store source asset references in task/run manifests, then materialize selected inputs into the run directory before engine execution.
D-InSAR and SBAS store local source asset references in task/run manifests, then materialize selected inputs into the run directory before engine execution.
Required next integration points:
Required integration points:
- D-InSAR Task_Pool publishing:
- store `source_product_asset_id`, `archive_path`, `source_format`;
- store `source_product_asset_id`, `archive_path`, and `source_format`;
- materialize master/slave archive assets into the task directory before engine dispatch.
- Gamma/PyINT:
- always consume local materialized paths because WSL conversion rejects or cannot reliably map UNC paths.
- consume local materialized paths because WSL conversion rejects or cannot reliably map network paths.
- LandSAR and ENVI/SARscape:
- prefer local materialized paths even when Windows can see UNC, to avoid external engine path and credential issues.
- consume local materialized paths.
- SBAS:
- stack discovery can use archive metadata;
- selected scenes must be materialized into the SBAS `RAW`/input structure before Gamma commands such as `par_LT1_SLC`.
## GF3 Management
GF3 has two asset layers:
- `GF3_ARCHIVE`: original source archive, suitable for UNC source management and migration tracking.
- GF3 SARscape standardized L2: production result/analysis-ready layer, used for map footprint, preview, radar data management, and water extraction.
Do not replace standardized L2 management with raw archive management. Archive assets should link migration and production status; previews and water extraction should continue to consume standardized L2/analysis-ready products.
## Migration Guidance
1. Register UNC roots first and scan inventory.
2. Verify archive asset counts and parse status.
3. Keep existing local standardized results and D-InSAR/SBAS products in place.
4. Move source archives to UNC and update root configuration.
5. Only after inventory and materialize tests pass, switch D-InSAR/SBAS publishing to archive asset references.
Production safety rule: if a run cannot materialize every selected source asset locally, the run must fail before invoking the engine.
## Recommended UNC Layout
## GF3 Management
The current deployment uses two SMB shares:
GF3 now has a separate operational rule:
```text
\\DESKTOP-N16HJ84\InSAR_Storage_1
\\DESKTOP-N16HJ84\InSAR_Storage_2
```
- GF3 SARscape production is not run on this management machine.
- Already-produced SARscape `_geo` ENVI binary results are stored locally under `GF3_SARSCAPE_NATIVE_DIRS`.
- The system registers those local `_geo` native results and their `.hdr/.sml` sidecars.
- `*_geo_ql.tif` is retained only as an auxiliary quicklook file.
- WebP preview cache is generated locally from the `_geo` ENVI binary, not from `*_geo_ql.tif`.
- Standard GeoTIFF conversion remains a separate explicit path; the monitor button used for GF3 registration is native-result registration only.
Recommended source archive layout:
```text
\\DESKTOP-N16HJ84\InSAR_Storage_1
└─ GaoFen-3
├─ 20260513
│ └─ GF3_*.tar.gz
└─ 20260514
\\DESKTOP-N16HJ84\InSAR_Storage_2
├─ LuTan-1
│ └─ Archive
│ ├─ 20260513
│ │ └─ LT1*.tar.gz / LT1*.tgz / LT1*.zip / LT1*.tar
│ └─ 20260514
├─ Sentinel-1
│ └─ Archive
│ ├─ 20260513
│ │ └─ S1*.zip
│ └─ 20260514
└─ Orbit
├─ LuTan-1
│ ├─ LT1A_GpsData_GAS_C_YYYYMMDD.txt
│ └─ LT1B_GpsData_GAS_C_YYYYMMDD.txt
└─ Sentinel-1
└─ S1*.EOF
```
Recommended local Task_Pool layout:
## Recommended Local Layout
```text
D:\
├─ LuTan1_Image_Pool_Zip
│ └─ LT1*.tar.gz / LT1*.tgz / LT1*.zip / LT1*.tar
├─ Sentinel1_Image_Pool_ZIP
│ └─ S1*.zip
├─ LuTan1_Image_Pool
│ └─ LT1 unpacked scene directories
├─ Sentinel1_Image_Pool
│ └─ S1*.SAFE directories
├─ LT1_data_lsarorbit
│ └─ LT1*_GpsData_*.txt
├─ Sentinel1_EOF_Pool
│ └─ S1*.EOF
├─ production_results
│ └─ gf3
│ ├─ sarscape_native
│ │ └─ YYYYMMDD_geo
│ │ └─ GF3_*
│ │ ├─ *_geo
│ │ ├─ *_geo.hdr
│ │ ├─ *_geo.sml
│ │ └─ *_geo_ql.tif
│ └─ standard_l2
└─ Task_Pool
D:\Task_Pool
├─ DInSAR
│ └─ <pair_task>
@@ -129,30 +116,35 @@ D:\Task_Pool
└─ publish
```
Date folders are optional for the scanner because source and orbit inventory recurse through configured roots. They are recommended for operator readability and migration checks.
Date folders are optional for the LT-1/Sentinel-1 scanners because inventory recurses through configured roots. GF3 native pools should keep the SARscape `YYYYMMDD_geo/<scene>` convention.
## Current Local Configuration Example
The local `.env` should keep legacy local roots and UNC roots side by side during migration:
The local `.env` should keep all runtime roots local:
```text
SOURCE_PRODUCT_DIRS=D:\LuTan1_Image_Pool;D:\Sentinel1_Image_Pool_ZIP;\\DESKTOP-N16HJ84\InSAR_Storage_2\LuTan-1\Archive;\\DESKTOP-N16HJ84\InSAR_Storage_2\Sentinel-1\Archive
ORBIT_SOURCE_DIRS=D:\LT1_data_lsarorbit;D:\Sentinel1_EOF_Pool;\\DESKTOP-N16HJ84\InSAR_Storage_2\Orbit\LuTan-1;\\DESKTOP-N16HJ84\InSAR_Storage_2\Orbit\Sentinel-1
GF3_ARCHIVE_SOURCE_DIRS=\\DESKTOP-N16HJ84\InSAR_Storage_1\GaoFen-3
SOURCE_PRODUCT_DIRS=D:\LuTan1_Image_Pool_Zip;D:\Sentinel1_Image_Pool_ZIP
SENTINEL1_STORAGE_DIRS=
ORBIT_SOURCE_DIRS=D:\LT1_data_lsarorbit;D:\Sentinel1_EOF_Pool
MONITOR_ORBIT_DIR=D:\LT1_data_lsarorbit
GF3_TASK_POOL_ROOT=D:\GaoFen3_Task_Pool
GF3_ARCHIVE_SOURCE_DIRS=D:\GaoFen3_Image_Pool\archives
GF3_SARSCAPE_NATIVE_DIRS=D:\GaoFen3_Image_Pool\sarscape_native
GF3_STORAGE_DIRS=D:\GaoFen3_Image_Pool\standard_l2
TASK_POOL_ROOT=D:\Task_Pool
DINSAR_TASK_POOL_ROOT=D:\Task_Pool\DInSAR
SBAS_TASK_POOL_ROOT=D:\Task_Pool\SBAS
GAMMA_SBAS_WORK_ROOT=D:\Task_Pool\SBAS
```
Do not store SMB credentials in `.env`. Credentials should be stored in Windows Credential Manager for the account that runs the backend/worker service.
Do not configure UNC paths in these variables.
## Orbit Pool Contract
There are two different orbit concepts:
- `ORBIT_SOURCE_DIRS`: source inventory roots. These can be UNC and may be date-organized or flat.
- `ORBIT_POOL_ENVI` / `PYINT_ORBIT_POOL_TXT`: local production orbit pools. These should remain local disk paths.
- `ORBIT_SOURCE_DIRS`: local source inventory roots.
- `ORBIT_POOL_ENVI` / `PYINT_ORBIT_POOL_TXT`: local production orbit pools.
LT-1 local production orbit pool should support both flat and satellite-split layouts:
@@ -166,34 +158,40 @@ D:\orbit_pools\envi
└─ envi
```
The `LT1A` and `LT1B` names are satellite names, not product levels. ENVI/Gamma/PyINT/SBAS should use local orbit files copied or synchronized from `ORBIT_SOURCE_DIRS`; they should not be required to read UNC directly.
Sentinel-1 EOF files can be indexed from UNC. Gamma/PyINT/SBAS execution should stage required EOF files locally with the selected scenes.
The `LT1A` and `LT1B` names are satellite names, not product levels. ENVI/Gamma/PyINT/SBAS should use local orbit files copied or synchronized from `ORBIT_SOURCE_DIRS`.
## Migration Phases
### Phase 1: Source archive migration
### Phase 1: Local source archive inventory
Move or copy source archives only:
Keep production source archives local:
- LT-1 compressed scenes to `\\DESKTOP-N16HJ84\InSAR_Storage_2\LuTan-1\Archive\<YYYYMMDD>\`.
- Sentinel-1 ZIP scenes to `\\DESKTOP-N16HJ84\InSAR_Storage_2\Sentinel-1\Archive\<YYYYMMDD>\`.
- GF3 raw archives to `\\DESKTOP-N16HJ84\InSAR_Storage_1\GaoFen-3\<YYYYMMDD>\`.
- LT-1 compressed scenes in `D:\LuTan1_Image_Pool_Zip`.
- Sentinel-1 ZIP scenes in `D:\Sentinel1_Image_Pool_ZIP`.
- Existing unpacked local scene directories are not active management pools; they should only be task materialization outputs.
Keep current local unpacked scene directories in place until D-InSAR and SBAS archive materialization have been tested.
### Phase 2: Local orbit deployment
### Phase 2: Orbit source migration
Deploy orbit source files on this machine:
Copy orbit source files to UNC:
- LT-1 TXT files under `D:\LT1_data_lsarorbit`.
- Sentinel-1 EOF files under `D:\Sentinel1_EOF_Pool`.
- LT-1 TXT files to `\\DESKTOP-N16HJ84\InSAR_Storage_2\Orbit\LuTan-1\`.
- Sentinel-1 EOF files to `\\DESKTOP-N16HJ84\InSAR_Storage_2\Orbit\Sentinel-1\`.
Keep `ORBIT_POOL_ENVI` and `PYINT_ORBIT_POOL_TXT` local.
Keep `ORBIT_POOL_ENVI` and `PYINT_ORBIT_POOL_TXT` local. Add a later sync/materialize step to populate local orbit pools from the indexed UNC source assets.
### Phase 3: GF3 native-result registration
### Phase 3: Production cutover
Copy completed SARscape `_geo` result folders to the local GF3 native pool:
After inventory scan verifies UNC assets:
```text
D:\GaoFen3_Image_Pool\sarscape_native\YYYYMMDD_geo\<GF3 scene>\
```
Then run GF3 native-result registration and GF3 WebP generation. WebP generation reads the `_geo` ENVI binary and requires its `.hdr` sidecar.
### Phase 4: Task_Pool production
After inventory scan verifies local assets:
1. D-InSAR Task_Pool stores source asset IDs and archive paths.
2. Task preparation materializes master/slave scenes and orbit files under `D:\Task_Pool\DInSAR\<task>`.
@@ -201,26 +199,19 @@ After inventory scan verifies UNC assets:
4. Results register normally.
5. Local materialized inputs and intermediate products are eligible for cleanup after result registration.
### Phase 4: Retire old local source pools
Only after repeated D-InSAR/SBAS runs succeed from archive materialization:
- remove old local source roots from `SOURCE_PRODUCT_DIRS`;
- keep local work/result roots;
- keep standardized GF3 L2 products unless explicitly migrated and revalidated.
## Local Cleanup Design
After source archives are managed on UNC and production results are registered as assets, local disk can be treated as a cache/work area. Cleanup should be explicit and asset-aware.
After production results are registered as assets, Task_Pool materialized inputs and work folders can be treated as cleanup candidates. Local source archive roots are durable production inputs and must not be cleaned as cache.
### Keep Classes
Cleanup must never delete:
- configured UNC source archive roots;
- local or UNC orbit source roots;
- configured local source archive roots;
- configured local orbit source roots;
- registered D-InSAR result assets;
- registered SBAS result assets;
- registered GF3 SARscape native result assets;
- registered GF3 standardized L2 assets;
- `SAR_ANALYSIS_READY_ROOT` products and water extraction result assets;
- current pointers, manifests, previews, and catalog metadata needed to open results.
@@ -234,7 +225,7 @@ Cleanup may delete only these local classes after verification:
- D-InSAR engine intermediate folders not listed in the result manifest;
- Gamma/PyINT temporary project work directories after result registration;
- SBAS `RAW`, `SLC`, `RSLC`, `MLI`, `DIFF`, `DIFF1`, script logs, and temporary staging after SBAS product registration;
- GF3 SARscape native intermediates only after standardized L2 registration and optional native-retention policy allows cleanup.
- GF3 SARscape runtime or temporary staging only after local native `_geo` registration and optional native-retention policy allows cleanup.
### Safety Contract
@@ -249,59 +240,5 @@ Deletion must require:
- path is not inside any configured source archive root;
- path is not inside a result publish root unless the exact file is classified as intermediate;
- associated result or standardized asset is registered;
- candidate is older than a configurable minimum age;
- no active task references the path.
### Proposed API
```text
POST /api/maintenance/cleanup/preview
POST /api/maintenance/cleanup/execute
```
Preview request fields:
```json
{
"scope": "dinsar|sbas|gf3|materialized|all",
"root_ids": [],
"older_than_hours": 24,
"require_registered_result": true,
"include_task_pool_inputs": false
}
```
Preview response should include:
```json
{
"preview_id": "...",
"total_bytes": 0,
"candidates": [
{
"path": "D:\\production_runtime\\...",
"class": "materialized_source",
"owner": "task/run/product id",
"size_bytes": 0,
"eligible": true,
"reason": "registered_result_exists"
}
],
"blocked": []
}
```
### Recommended Defaults
- `materialized`: delete after 24 hours if no active task references it.
- `dinsar`: delete engine intermediates after result registration; keep Task_Pool inputs until all selected engines are complete or user opts in.
- `sbas`: delete heavy Gamma working directories after SBAS catalog registration and product assets exist.
- `gf3`: keep standardized L2; clean SARscape native only when `GF3_SARSCAPE_CLEAN_AFTER_SUCCESS=true` and standardized registration is confirmed.
### Implementation Order
1. Add read-only cleanup preview service.
2. Add path classification and approved-root checks.
3. Add execute endpoint with preview token.
4. Add frontend maintenance panel.
5. Wire D-InSAR/SBAS/GF3 run pages to show cleanup eligibility after successful registration.
- cleanup policy explicitly allows the class;
- operator confirmation is present.