feat: engineer SBAS timeseries production workflow
This commit is contained in:
+141
-2
@@ -205,7 +205,7 @@ python run_worker.py
|
||||
这套链路要求:
|
||||
|
||||
- `.env` 中的根目录配置要真实可访问。
|
||||
- `backend/migrations/001` 到 `006` 必须保持幂等。
|
||||
- `backend/migrations/001` 到 `007` 必须保持幂等。
|
||||
- 启动自维护默认是保守模式,不会在 schema 不匹配时自动破坏性重建。
|
||||
|
||||
## 6. 数据库自维护边界
|
||||
@@ -216,7 +216,7 @@ python run_worker.py
|
||||
- 自动创建 `postgis` 扩展
|
||||
- 自动创建缺失表
|
||||
- 自动补齐缺失列
|
||||
- 自动执行 `001` 到 `006` 号 SQL 文件
|
||||
- 自动执行 `001` 到 `007` 号 SQL 文件
|
||||
- 自动引导管理员账号与灾害点初始化
|
||||
|
||||
- 不支持:
|
||||
@@ -304,3 +304,142 @@ VITE_TILE_SERVER_TOKEN=change_me
|
||||
- [DATABASE_SELF_MAINTENANCE_AUDIT_20260425.md](DATABASE_SELF_MAINTENANCE_AUDIT_20260425.md)
|
||||
- [PRODUCTION_RESULTS_MULTI_ENGINE_DESIGN_20260423.md](PRODUCTION_RESULTS_MULTI_ENGINE_DESIGN_20260423.md)
|
||||
- [WSL_RUNTIME_REFACTOR_DESIGN_20260422.md](WSL_RUNTIME_REFACTOR_DESIGN_20260422.md)
|
||||
|
||||
## DEM Sidecar Migration Warning
|
||||
|
||||
If you copy or move a prepared ISCE DEM bundle to a new directory or machine, do not assume
|
||||
that the sidecar XML files are portable as-is.
|
||||
|
||||
Affected files typically include:
|
||||
|
||||
- `<dem>.xml`
|
||||
- `<dem>.vrt`
|
||||
- `<dem>.wgs84`
|
||||
- `<dem>.wgs84.xml`
|
||||
- `<dem>.wgs84.vrt`
|
||||
|
||||
The ISCE XML sidecars may still contain absolute historical paths in:
|
||||
|
||||
- `file_name`
|
||||
- `metadata_location`
|
||||
- `extra_file_name`
|
||||
|
||||
Typical failure symptom:
|
||||
|
||||
- `verifyDEM` succeeds, but `topo` fails with `FileNotFoundError` pointing at an old
|
||||
`/mnt/...` path from the previous machine or previous directory layout.
|
||||
|
||||
Recommended post-migration repair:
|
||||
|
||||
```powershell
|
||||
C:\ProgramData\anaconda3\envs\InSAR\python.exe `
|
||||
backend\app\isce2_pipeline\repair_dem_sidecars.py `
|
||||
--root D:\DEM `
|
||||
--repair
|
||||
```
|
||||
|
||||
The managed ISCE2 pipeline now repairs the selected DEM sidecar paths before running, but
|
||||
the directory-level repair is still recommended after deployment or storage migration so the
|
||||
whole DEM bundle remains internally consistent.
|
||||
|
||||
## ISCE2 Rubbersheeting Runtime Dependency
|
||||
|
||||
The managed `ISCE2` `lt1_stripmap` profile enables dense offsets plus range and azimuth
|
||||
rubbersheeting by default. This is an ISCE2 native workflow step, not an export-time
|
||||
correction.
|
||||
|
||||
The range rubbersheeting implementation imports `astropy.convolution`, so every deployed
|
||||
or migrated WSL runtime must include `astropy` in `insar_wsl_v1`.
|
||||
|
||||
Check:
|
||||
|
||||
```bash
|
||||
/home/administrator/miniconda3/envs/insar_wsl_v1/bin/python -c "from astropy.convolution import convolve; print('astropy_ok')"
|
||||
```
|
||||
|
||||
Repair:
|
||||
|
||||
```bash
|
||||
conda install -n insar_wsl_v1 -c conda-forge astropy
|
||||
```
|
||||
|
||||
If this package is missing, production now fails during preflight instead of after the
|
||||
long dense-offset stage.
|
||||
|
||||
## ISCE2 Ionosphere Runtime Dependency
|
||||
|
||||
The managed `ISCE2` `lt1_stripmap` profile now keeps the standard stripmap
|
||||
`split-spectrum -> low/high-band unwrap -> ionosphere -> geocode` path enabled.
|
||||
This is part of the native workflow and replaces the older fake `PICKLE/ionosphere`
|
||||
resume shortcut.
|
||||
|
||||
Operational consequences:
|
||||
|
||||
- `resume_from=unwrap` now resumes the full stage-2 chain up to `ionosphere`
|
||||
- `resume_from=geocode` now starts from real `ionosphere` state when available
|
||||
- the export step prefers geocoded `ionosphere/nondispersive.bil.unwCor.filt`
|
||||
when that product exists
|
||||
|
||||
The ionosphere implementation in ISCE2 imports `cv2` and `scipy`, so every
|
||||
deployed or migrated WSL runtime must include both packages.
|
||||
|
||||
Check:
|
||||
|
||||
```bash
|
||||
/home/administrator/miniconda3/envs/insar_wsl_v1/bin/python -c "import cv2, scipy; print('ionosphere_ok')"
|
||||
```
|
||||
|
||||
Repair:
|
||||
|
||||
```bash
|
||||
conda install -n insar_wsl_v1 -c conda-forge opencv scipy
|
||||
```
|
||||
|
||||
If these packages are missing, production now fails during preflight instead of
|
||||
after the long stripmap filtering / unwrap stage.
|
||||
|
||||
## Git Clone Bootstrap
|
||||
|
||||
Goal: a clean `git clone` on a new Windows host should already contain the
|
||||
deployment entrypoints required to install dependencies, validate `.env`, and
|
||||
start the system.
|
||||
|
||||
Recommended path:
|
||||
|
||||
```powershell
|
||||
git clone <repo-url>
|
||||
cd Insar_management_system_v2
|
||||
Copy-Item .env.example .env
|
||||
notepad .env
|
||||
powershell -ExecutionPolicy Bypass -File .\scripts\bootstrap_clone.ps1 -InitFrontend -BuildFrontend
|
||||
start_system.bat
|
||||
```
|
||||
|
||||
Optional runtime bootstrap:
|
||||
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File .\scripts\bootstrap_clone.ps1 -InitWindowsConda
|
||||
powershell -ExecutionPolicy Bypass -File .\scripts\bootstrap_clone.ps1 -InitWslConda
|
||||
powershell -ExecutionPolicy Bypass -File .\scripts\bootstrap_clone.ps1 -All
|
||||
```
|
||||
|
||||
`scripts/bootstrap_clone.ps1` is intentionally conservative:
|
||||
|
||||
- It copies `.env.example` to `.env` only when `.env` is missing.
|
||||
- It can create or update the Windows conda env from `environment.yml`.
|
||||
- It can create or update the shared WSL conda env from
|
||||
`deploy/wsl/conda/insar_wsl_v1.environment.yml`.
|
||||
- It can run `npm ci` and `npm run build` in `frontend/`.
|
||||
- It runs `scripts/check_runtime_config.py` unless `-SkipChecks` is specified.
|
||||
- `start_system.bat` now fails early when `frontend/dist/index.html` is missing.
|
||||
- It does not auto-start backend, worker, or Nginx.
|
||||
- It does not bypass the existing database self-maintenance safeguards.
|
||||
|
||||
Typical deployment sequence:
|
||||
|
||||
1. Edit `.env` to match the new server.
|
||||
2. Run `bootstrap_clone.ps1` with the switches required by that server.
|
||||
3. Run `start_system.bat`.
|
||||
4. Check `GET /api/health`.
|
||||
5. Trigger a small real production task and confirm the result is registered in
|
||||
the catalog.
|
||||
|
||||
@@ -49,6 +49,8 @@
|
||||
|
||||
- [ISCE2_SBAS_TIMESERIES_DESIGN.md](ISCE2_SBAS_TIMESERIES_DESIGN.md)
|
||||
- [ISCE2_SBAS_PRODUCT_SPEC.md](ISCE2_SBAS_PRODUCT_SPEC.md)
|
||||
- [ISCE2_SBAS_ENGINEERING_DESIGN_20260428.md](ISCE2_SBAS_ENGINEERING_DESIGN_20260428.md)
|
||||
Current-phase engineering design for the managed ISCE2 + MintPy SBAS route.
|
||||
|
||||
说明:
|
||||
|
||||
@@ -83,6 +85,13 @@
|
||||
|
||||
- [SECURITY_AUDIT_2026-03-12.md](SECURITY_AUDIT_2026-03-12.md)
|
||||
|
||||
## Clone Bootstrap
|
||||
|
||||
- [../scripts/bootstrap_clone.ps1](../scripts/bootstrap_clone.ps1)
|
||||
Fresh-server bootstrap entry for clone-based deployment. It keeps `.env`
|
||||
initialization, frontend dependency install/build, runtime bootstrap, and
|
||||
deployment validation in one place without changing the main startup chain.
|
||||
|
||||
## 7. 工作笔记
|
||||
|
||||
- [../INIT.md](../INIT.md)
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
# ISCE2 LT-1 Enhancement Alignment 2026-04-27
|
||||
|
||||
## Purpose
|
||||
|
||||
This note records why the managed `ISCE2` `lt1_stripmap` production profile now
|
||||
enables the built-in stripmap enhancement steps by default, and how that choice
|
||||
relates to the existing SARscape `custom6` production chain.
|
||||
|
||||
## Background
|
||||
|
||||
The SARscape `custom6` chain already goes beyond a bare minimum D-InSAR run.
|
||||
Its production semantics include:
|
||||
|
||||
1. Interferogram generation
|
||||
2. Filtering and coherence
|
||||
3. Orbital trend / residual phase frequency removal
|
||||
4. Phase unwrapping
|
||||
5. GCP-based refinement and reflattening
|
||||
6. Phase to displacement and geocoding
|
||||
|
||||
This means the current LT-1 production baseline in the system is not a
|
||||
scientifically "raw" interferometric export. It is an operationally enhanced
|
||||
delivery chain.
|
||||
|
||||
## ISCE2 Mapping
|
||||
|
||||
ISCE2 stripmap does not expose the exact same SARscape modules, but it does
|
||||
provide native enhancement steps that address the same operational risk class:
|
||||
residual misregistration and geometry-driven long-wavelength artifacts.
|
||||
|
||||
The relevant built-in ISCE2 controls are:
|
||||
|
||||
- `doDenseOffsets`
|
||||
- `doRubbersheetingRange`
|
||||
- `doRubbersheetingAzimuth`
|
||||
- `do split spectrum`
|
||||
- `do dispersive`
|
||||
- `rubberSheetSNRThreshold`
|
||||
- `rubberSheetFilterSize`
|
||||
|
||||
When enabled, the stripmap workflow:
|
||||
|
||||
- estimates dense offsets from cross-correlation
|
||||
- filters / masks those offsets
|
||||
- updates the geometry offset fields
|
||||
- performs a fine resampling pass using the corrected offsets
|
||||
- unwraps low/high-band interferograms and estimates a dispersive ionosphere term
|
||||
- geocodes the ionosphere-corrected nondispersive phase for delivery
|
||||
|
||||
This is not identical to SARscape's `RemoveResidualPhaseFrequency` plus
|
||||
`RefinementAndReflattening`, but it is the closest native ISCE2 enhancement
|
||||
path inside the standard stripmap application.
|
||||
|
||||
## Production Decision
|
||||
|
||||
The managed `ISCE2` `lt1_stripmap` profile now treats these steps as part of the
|
||||
default LT-1 production workflow:
|
||||
|
||||
- split-spectrum ionosphere correction: enabled
|
||||
- dense offsets: enabled
|
||||
- range rubbersheeting: enabled
|
||||
- azimuth rubbersheeting: enabled
|
||||
|
||||
Default numeric parameters:
|
||||
|
||||
- `rubberSheetSNRThreshold = 5.0`
|
||||
- `rubberSheetFilterSize = 9`
|
||||
- `denseWindowWidth = 64`
|
||||
- `denseWindowHeight = 64`
|
||||
- `denseSearchWidth = 20`
|
||||
- `denseSearchHeight = 20`
|
||||
- `denseSkipWidth = 32`
|
||||
- `denseSkipHeight = 32`
|
||||
|
||||
These defaults are stored as profile semantics in code, not as loose `.env`
|
||||
feature toggles.
|
||||
|
||||
## Runtime Dependency
|
||||
|
||||
The stripmap ionosphere implementation imports `cv2` and `scipy`.
|
||||
|
||||
Deployment check:
|
||||
|
||||
```bash
|
||||
/home/administrator/miniconda3/envs/insar_wsl_v1/bin/python -c "import cv2, scipy; print('ionosphere_ok')"
|
||||
```
|
||||
|
||||
Repair command for an existing runtime:
|
||||
|
||||
```bash
|
||||
conda install -n insar_wsl_v1 -c conda-forge opencv scipy
|
||||
```
|
||||
|
||||
The range rubbersheeting implementation in ISCE2 imports
|
||||
`astropy.convolution` from `runRubbersheetRange.py`. The shared WSL conda
|
||||
runtime therefore must include `astropy`.
|
||||
|
||||
Deployment check:
|
||||
|
||||
```bash
|
||||
/home/administrator/miniconda3/envs/insar_wsl_v1/bin/python -c "from astropy.convolution import convolve; print('astropy_ok')"
|
||||
```
|
||||
|
||||
Repair command for an existing runtime:
|
||||
|
||||
```bash
|
||||
conda install -n insar_wsl_v1 -c conda-forge astropy
|
||||
```
|
||||
|
||||
## Boundary
|
||||
|
||||
This change does **not** mean that every long-wavelength ramp problem is solved.
|
||||
|
||||
It only means the default managed ISCE2 LT-1 profile now includes the native
|
||||
registration-enhancement path that was previously omitted.
|
||||
|
||||
If a run still shows a strong residual scene-wide ramp after rubbersheeting,
|
||||
that should be treated as a separate quality / post-processing issue and should
|
||||
be diagnosed explicitly rather than silently hidden inside export logic.
|
||||
|
||||
## Operational Implication
|
||||
|
||||
When comparing current SARscape and ISCE2 LT-1 products:
|
||||
|
||||
- SARscape `custom6` remains the more explicitly refined chain
|
||||
- ISCE2 `lt1_stripmap` is no longer a bare stripmap baseline
|
||||
- both engines now include standard enhancement intent in default production
|
||||
|
||||
This makes cross-engine behavior more defensible for LT-1 operational delivery.
|
||||
|
||||
## Operator Controls
|
||||
|
||||
As of 2026-04-29, the production UI no longer hides these choices behind code
|
||||
defaults only.
|
||||
|
||||
The ISCE2 production panel now exposes the managed LT-1 profile parameters in
|
||||
three groups:
|
||||
|
||||
- `Execution`
|
||||
- `Delivery`
|
||||
- `Enhancement`
|
||||
|
||||
The user-visible controls now cover:
|
||||
|
||||
- split-spectrum ionosphere correction on/off
|
||||
- dense offsets on/off
|
||||
- range rubbersheeting on/off
|
||||
- azimuth rubbersheeting on/off
|
||||
- reference normalization mode (`coh_median` or `none`)
|
||||
- deramp mode (`plane` or `none`)
|
||||
|
||||
This keeps the default managed behavior unchanged, while allowing operators to
|
||||
fall back toward a more conservative stripmap delivery path when a specific
|
||||
scene looks worse after enhancement.
|
||||
@@ -0,0 +1,663 @@
|
||||
# ISCE2 + MintPy SBAS Engineering Design
|
||||
|
||||
Updated: 2026-04-28
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
This document defines the engineering expansion plan for the current stack-based time-series InSAR route:
|
||||
|
||||
- `LT-1 stack batch -> ISCE2 stripmapStack -> MintPy SBAS -> publish bundle -> psinsar catalog`
|
||||
|
||||
The repository already has a working phase-1 skeleton. The goal of this document is not to restart the design from zero, but to align the next implementation round with the code that already exists in:
|
||||
|
||||
- `backend/app/services/timeseries_service.py`
|
||||
- `backend/app/routers/timeseries_production.py`
|
||||
- `backend/app/services/psinsar_catalog_service.py`
|
||||
- `frontend/src/TimeseriesProductionPanel.jsx`
|
||||
- `frontend/src/components/PsinsarCatalogPanel.jsx`
|
||||
|
||||
This document supersedes the "missing pieces" parts of `docs/ISCE2_SBAS_TIMESERIES_DESIGN.md` for the current implementation phase.
|
||||
|
||||
## 2. Decisions
|
||||
|
||||
### 2.1 Primary processing route
|
||||
|
||||
Keep the current scientific split:
|
||||
|
||||
- ISCE2 is responsible for LT-1 stack preparation, stack geometry, co-registration, baseline generation, interferogram generation, and unwrap inputs.
|
||||
- MintPy is responsible for SBAS inversion and time-series products.
|
||||
- The system registers only publish-grade bundles, not raw MintPy work directories.
|
||||
|
||||
This means the production claim for the current phase is:
|
||||
|
||||
- `SBAS time-series production on top of ISCE2 + MintPy`
|
||||
|
||||
It is not:
|
||||
|
||||
- full PS-InSAR
|
||||
- full StaMPS integration
|
||||
- full commercial-grade atmospheric/error-correction stack
|
||||
|
||||
### 2.2 Keep the current business model
|
||||
|
||||
Use the current model already implemented in code:
|
||||
|
||||
- planning-layer stack snapshot:
|
||||
- `PsTaskBatchORM`
|
||||
- `PsTaskItemORM`
|
||||
- business-facing production run:
|
||||
- `PsTimeseriesRunORM`
|
||||
- step orchestration:
|
||||
- `WorkflowRunORM`
|
||||
- `WorkflowStepORM`
|
||||
- publish/catalog registration:
|
||||
- `ResultProductORM`
|
||||
- `ResultAssetORM`
|
||||
- `ResultIssueORM`
|
||||
|
||||
Do not redesign the run model into a new engine abstraction in this round.
|
||||
|
||||
### 2.3 Keep naming stable for now
|
||||
|
||||
Current naming in the repository is mixed:
|
||||
|
||||
- product family shown to users: `timeseries`
|
||||
- processing mode: `sbas`
|
||||
- catalog namespace and package schema legacy: `psinsar`
|
||||
|
||||
For this round:
|
||||
|
||||
- keep `product_family = timeseries`
|
||||
- keep `mode = sbas`
|
||||
- keep `catalog_name = psinsar` for compatibility
|
||||
- keep `psinsar.publish.v1` ingestion support working
|
||||
|
||||
Do not do a DB/API namespace rename and a pipeline hardening round at the same time.
|
||||
|
||||
## 2.4 Current implementation scope (2026-04-28)
|
||||
|
||||
This round is intentionally constrained to avoid impact on other production business:
|
||||
|
||||
- no DB schema migration
|
||||
- no catalog namespace rename
|
||||
- no workflow framework refactor
|
||||
- no change to the system self-maintenance / self-check contract
|
||||
|
||||
The implementation landed in this round focuses on pipeline hardening around the existing `PsTimeseriesRunORM` path:
|
||||
|
||||
- preflight gating before SBAS run creation
|
||||
- stronger runtime self-check visibility
|
||||
- publish-bundle validation before catalog registration
|
||||
- frontend visibility for preflight, runtime checks, and publish validation
|
||||
|
||||
This means the current engineering target is:
|
||||
|
||||
- make the existing ISCE2 + MintPy SBAS route operationally safer
|
||||
|
||||
not:
|
||||
|
||||
- redesign the overall architecture
|
||||
- replace the existing result registration model
|
||||
- introduce a second persistence path for timeseries products
|
||||
|
||||
### 2.5 Phase-2 scope (2026-04-28)
|
||||
|
||||
The second implementation round keeps the same production chain but upgrades the planning trace from ad-hoc JSON to first-class additive schema objects.
|
||||
|
||||
Additive schema only:
|
||||
|
||||
- new planning tables:
|
||||
- `TimeseriesStackPlanORM`
|
||||
- `TimeseriesStackPlanItemORM`
|
||||
- nullable trace columns on existing objects:
|
||||
- `PsTaskBatchORM.plan_id`
|
||||
- `PsTaskBatchORM.plan_strategy`
|
||||
- `PsTaskItemORM.plan_item_ref_id`
|
||||
- `PsTimeseriesRunORM.plan_id`
|
||||
- `PsTimeseriesRunORM.plan_strategy`
|
||||
|
||||
Operational rules for phase 2:
|
||||
|
||||
- do not introduce a separate migration framework
|
||||
- rely on the existing database self-maintenance path:
|
||||
- `Base.metadata.create_all()`
|
||||
- missing-column auto-add in `backend/app/db_maintenance.py`
|
||||
- keep phase-1 `planning_context` / `remark` compatibility for old batches
|
||||
|
||||
The engineering target of phase 2 is:
|
||||
|
||||
- formalize `plan -> batch -> run -> publish bundle -> catalog product` traceability
|
||||
- expose `plan_id` in frontend production and result views
|
||||
- keep old batches runnable without backfilling or hard migration
|
||||
- expose `GET /timeseries-plans/{plan_id}` for plan audit/detail lookup
|
||||
|
||||
## 3. Current Baseline In Code
|
||||
|
||||
The current code already implements the core production skeleton.
|
||||
|
||||
### 3.1 Run record and workflow
|
||||
|
||||
`backend/app/models/orm.py`
|
||||
|
||||
- `PsTimeseriesRunORM` already stores:
|
||||
- run identity
|
||||
- batch binding
|
||||
- processor/runtime metadata
|
||||
- work and publish roots
|
||||
- input/orbit/quality summaries
|
||||
- failure state
|
||||
- `WorkflowRunORM` and `WorkflowStepORM` already support DAG execution and retry.
|
||||
- `SystemJobORM` already supports queued worker execution per workflow step.
|
||||
|
||||
### 3.2 Current workflow steps
|
||||
|
||||
`backend/app/services/timeseries_service.py`
|
||||
|
||||
Current step chain is already eight steps:
|
||||
|
||||
1. `prepare`
|
||||
2. `stack_prep_initial`
|
||||
3. `materialize`
|
||||
4. `stack_prep_refresh`
|
||||
5. `run_isce2_stack`
|
||||
6. `run_mintpy_sbas`
|
||||
7. `export_publish_bundle`
|
||||
8. `register_psinsar_product`
|
||||
|
||||
This is already the correct backbone for the managed SBAS route.
|
||||
|
||||
### 3.3 Current scientific execution boundary
|
||||
|
||||
The scientific boundary is still script-based, and that is acceptable for now:
|
||||
|
||||
- `experiments/isce2_sbas_timeseries/scripts/build_lt1_stack_prep.py`
|
||||
- `experiments/isce2_sbas_timeseries/scripts/materialize_lt1_stack_scenes.py`
|
||||
- `experiments/isce2_sbas_timeseries/scripts/prepare_lt1_stack_dem.py`
|
||||
- `experiments/isce2_sbas_timeseries/scripts/run_generated_stack_runfile_ubuntu2404.sh`
|
||||
- `experiments/isce2_sbas_timeseries/scripts/run_mintpy_sbas_unified_env_smoketest_ubuntu2404.sh`
|
||||
- `experiments/isce2_sbas_timeseries/scripts/export_mintpy_publish_products_ubuntu2404.sh`
|
||||
|
||||
The current implementation should continue to wrap these scripts instead of rewriting the scientific logic prematurely.
|
||||
|
||||
### 3.4 Current frontend and ops surface
|
||||
|
||||
Already present:
|
||||
|
||||
- run submission and run detail:
|
||||
- `frontend/src/TimeseriesProductionPanel.jsx`
|
||||
- product catalog panel:
|
||||
- `frontend/src/components/PsinsarCatalogPanel.jsx`
|
||||
- health-check visibility:
|
||||
- `frontend/src/HealthCheckPanel.jsx`
|
||||
- catalog rebuild API:
|
||||
- `backend/app/routers/ps_products.py`
|
||||
|
||||
So the next round is a hardening and extension round, not an empty scaffold round.
|
||||
|
||||
## 4. Main Gaps
|
||||
|
||||
The next engineering work should focus on the following gaps.
|
||||
|
||||
### 4.1 Self-check is present but still shallow
|
||||
|
||||
Current runtime check already validates:
|
||||
|
||||
- WSL distro
|
||||
- Python path
|
||||
- stack script path
|
||||
- configured helper scripts
|
||||
- MintPy import
|
||||
- DEM/orbit/output root presence
|
||||
|
||||
What is still missing:
|
||||
|
||||
- write permission checks for work and publish roots
|
||||
- DEM sidecar consistency checks
|
||||
- runtime dependency checks for scientific imports used by ISCE2/MintPy
|
||||
- batch-level readiness checks before a run is queued
|
||||
- publish-bundle structural validation before catalog registration
|
||||
|
||||
### 4.2 Quality summary exists, but quality gating is weak
|
||||
|
||||
Current code validates:
|
||||
|
||||
- stack prep readiness
|
||||
- required run files
|
||||
- required ISCE2 output directories
|
||||
- required MintPy outputs
|
||||
- publish manifest existence
|
||||
|
||||
But it still does not promote enough scientific quality indicators into release gates, for example:
|
||||
|
||||
- interferogram count versus expected network count
|
||||
- non-empty unwrap/correlation outputs
|
||||
- valid-pixel ratio after `maskAllValid`
|
||||
- temporal coherence thresholds
|
||||
- reference point presence and stability summary
|
||||
|
||||
### 4.3 Frontend is functional but still operationally thin
|
||||
|
||||
Current frontend can:
|
||||
|
||||
- submit a run
|
||||
- run WSL check
|
||||
- list runs
|
||||
- show workflow steps
|
||||
- retry failed workflow steps
|
||||
- browse catalog entries
|
||||
|
||||
Still missing:
|
||||
|
||||
- structured preflight diagnostics for the selected batch
|
||||
- clearer phase summaries per run
|
||||
- direct visibility into quality summaries and key artifacts
|
||||
- better linkage between run detail and published product detail
|
||||
- a richer product detail view closer to the D-InSAR catalog panel depth
|
||||
|
||||
### 4.4 Result management needs stricter contract enforcement
|
||||
|
||||
The catalog path is correct, but the following rules should be made explicit and enforced:
|
||||
|
||||
- `manifest.json` is the only registration entrypoint
|
||||
- every published run must have a stable `publish_dir`
|
||||
- required assets must exist before registration
|
||||
- missing assets should generate catalog issues and possibly quarantine status
|
||||
- every product should carry:
|
||||
- processor code
|
||||
- runtime id
|
||||
- native output trace
|
||||
- stack identity
|
||||
|
||||
## 5. Target Pipeline
|
||||
|
||||
### 5.1 Input contract
|
||||
|
||||
The run input must remain stack-based, not pair-based.
|
||||
|
||||
Source objects:
|
||||
|
||||
- one `ps_task_batch`
|
||||
- many `ps_task_items`
|
||||
- one selected stack manifest:
|
||||
- `input/selected_stack_manifest.json`
|
||||
- one generated stack manifest:
|
||||
- `input/stack_input_manifest.json`
|
||||
|
||||
The selected manifest is the planning snapshot.
|
||||
|
||||
The generated stack manifest is the execution snapshot and must include:
|
||||
|
||||
- stack dates
|
||||
- reference date
|
||||
- stack key
|
||||
- group key
|
||||
- resolved DEM and orbit dependencies
|
||||
- readiness flags
|
||||
- blocking reasons
|
||||
- generated ISCE2 command arguments
|
||||
|
||||
### 5.2 Runtime directory model
|
||||
|
||||
Keep the current directory split:
|
||||
|
||||
- work root:
|
||||
- `backend/runtime/timeseries_work/<run_id>/...`
|
||||
- publish root:
|
||||
- `TIMESERIES_PRODUCT_DIR/<stack_key>/runs/<run_id>/...`
|
||||
|
||||
Recommended internal layout under the work root:
|
||||
|
||||
- `input/`
|
||||
- `inputs/dem/`
|
||||
- `stack_work/`
|
||||
- `logs/`
|
||||
- `mintpy/`
|
||||
|
||||
Recommended publish layout:
|
||||
|
||||
- `manifest.json`
|
||||
- `assets/`
|
||||
- `preview/`
|
||||
- `metadata/`
|
||||
|
||||
### 5.3 Managed workflow
|
||||
|
||||
The current eight-step chain is the correct managed workflow and should be kept:
|
||||
|
||||
1. `prepare`
|
||||
- validate batch
|
||||
- resolve stack identity
|
||||
- choose reference date
|
||||
- write `selected_stack_manifest.json`
|
||||
2. `stack_prep_initial`
|
||||
- generate execution-layer stack manifest
|
||||
- resolve DEM/orbits
|
||||
- tell the system whether materialization is the only blocker
|
||||
3. `materialize`
|
||||
- materialize LT-1 scenes into stack input layout
|
||||
- materialize orbit XML and local dependencies
|
||||
4. `stack_prep_refresh`
|
||||
- re-run readiness check after materialization
|
||||
- must reach ready state
|
||||
5. `run_isce2_stack`
|
||||
- prepare local DEM sidecars
|
||||
- generate run files
|
||||
- run `run_01` to `run_08`
|
||||
- validate `geom_reference`, `baselines`, and `Igrams`
|
||||
6. `run_mintpy_sbas`
|
||||
- write MintPy config
|
||||
- run controlled `smallbaselineApp`
|
||||
- validate core MintPy outputs
|
||||
7. `export_publish_bundle`
|
||||
- geocode MintPy outputs
|
||||
- export GeoTIFF browse layers
|
||||
- generate preview and manifest
|
||||
- augment the manifest with canonical metadata
|
||||
8. `register_psinsar_product`
|
||||
- register the publish bundle into catalog
|
||||
- mark the run as published
|
||||
|
||||
### 5.4 Current publish contract
|
||||
|
||||
Keep `docs/ISCE2_SBAS_PRODUCT_SPEC.md` as the publish contract source of truth.
|
||||
|
||||
Required publish assets for the managed SBAS route:
|
||||
|
||||
- `assets/geo_timeseries.h5`
|
||||
- `assets/geo_velocity.h5`
|
||||
- `assets/velocity.tif`
|
||||
- `assets/geo_temporalCoherence.h5`
|
||||
- `assets/geo_maskTempCoh.h5`
|
||||
- `preview/velocity_preview.png`
|
||||
- `metadata/smallbaselineApp.cfg`
|
||||
- `manifest.json`
|
||||
|
||||
Optional but recommended:
|
||||
|
||||
- `preview/numTriNonzeroIntAmbiguity.png`
|
||||
- extra quality JSON files
|
||||
|
||||
## 6. Self-Check Design
|
||||
|
||||
Self-check should exist at four levels.
|
||||
|
||||
### 6.1 Runtime preflight
|
||||
|
||||
Primary entry:
|
||||
|
||||
- `POST /timeseries-production/wsl-check`
|
||||
|
||||
Current checks should be kept and extended with:
|
||||
|
||||
- WSL distro reachable
|
||||
- configured Python reachable
|
||||
- ISCE2 stack script import/help check
|
||||
- MintPy import check
|
||||
- helper script existence checks
|
||||
- DEM root existence
|
||||
- orbit pool existence
|
||||
- publish root existence
|
||||
- work root existence
|
||||
- write-test for work root
|
||||
- write-test for publish root
|
||||
- DEM sidecar consistency check
|
||||
- optional import checks for:
|
||||
- `cv2`
|
||||
- `scipy`
|
||||
- `astropy`
|
||||
|
||||
Return structure should remain machine-readable so the frontend can render a diagnostic card instead of a plain message string.
|
||||
|
||||
### 6.2 Batch preflight
|
||||
|
||||
Add a run-specific preflight before or during `create_run`.
|
||||
|
||||
Minimum checks:
|
||||
|
||||
- scene count meets SBAS minimum
|
||||
- all scene dates are valid
|
||||
- scene dates are unique
|
||||
- direction is consistent
|
||||
- source files exist and are readable
|
||||
- orbit coverage is complete or explicitly degraded
|
||||
- `group_key` and `stack_key` are derivable
|
||||
- publish path does not collide with another active run
|
||||
|
||||
Recommended surface:
|
||||
|
||||
- a new backend helper in `timeseries_service.py`
|
||||
- frontend summary block in `TimeseriesProductionPanel.jsx`
|
||||
|
||||
### 6.3 In-run gates
|
||||
|
||||
Each workflow step should continue to fail fast when hard requirements are not met.
|
||||
|
||||
Required gates:
|
||||
|
||||
- `stack_prep_refresh` must report `ready_for_stackStripMap_nofocus = true`
|
||||
- all expected run files must exist before stack execution
|
||||
- `run_08_igram` output directories must exist
|
||||
- MintPy required outputs must exist and be non-empty
|
||||
- export must generate a manifest plus required assets
|
||||
- registration must succeed against the catalog service
|
||||
|
||||
### 6.4 Post-publish health
|
||||
|
||||
Health is not only "the run finished".
|
||||
|
||||
The catalog and package checks must continue to validate:
|
||||
|
||||
- manifest exists
|
||||
- publish dir exists
|
||||
- processor code present
|
||||
- runtime id present for WSL-native engines
|
||||
- native output dir present
|
||||
- canonical package schema valid
|
||||
- manifest count versus DB count consistency
|
||||
|
||||
## 7. Frontend Design
|
||||
|
||||
### 7.1 Timeseries production panel
|
||||
|
||||
Keep `frontend/src/TimeseriesProductionPanel.jsx` as the main run workspace.
|
||||
|
||||
Planned enhancements:
|
||||
|
||||
- show structured runtime preflight results
|
||||
- show batch preflight results before submission
|
||||
- show phase-oriented run summary:
|
||||
- input prepared
|
||||
- stack ready
|
||||
- ISCE2 complete
|
||||
- MintPy complete
|
||||
- exported
|
||||
- published
|
||||
- show key paths and quality summary blocks without forcing the operator to inspect raw JSON
|
||||
- keep failed-step retry
|
||||
- add clearer linkage to the published product once available
|
||||
|
||||
### 7.2 Product catalog panel
|
||||
|
||||
Keep `frontend/src/components/PsinsarCatalogPanel.jsx` as the catalog entry.
|
||||
|
||||
Planned enhancements:
|
||||
|
||||
- retain catalog status and rebuild actions
|
||||
- enrich product detail with:
|
||||
- stack identity
|
||||
- processor/runtime identity
|
||||
- preview and primary assets
|
||||
- quality summary
|
||||
- asset list
|
||||
- issue list
|
||||
- coverage summary
|
||||
- keep the publish bundle as the fact source
|
||||
|
||||
### 7.3 Health panel
|
||||
|
||||
Keep health visibility in `frontend/src/HealthCheckPanel.jsx`.
|
||||
|
||||
The timeseries section should continue to show:
|
||||
|
||||
- catalog status
|
||||
- rebuild need
|
||||
- manifest vs DB counts
|
||||
- issue count
|
||||
|
||||
It should remain aligned with:
|
||||
|
||||
- `timeseries_result_catalog`
|
||||
- `product_packages`
|
||||
- `wsl_runtime`
|
||||
|
||||
## 8. Result Management and Registration
|
||||
|
||||
### 8.1 Registration rule
|
||||
|
||||
Register only from:
|
||||
|
||||
- `<publish_dir>/manifest.json`
|
||||
|
||||
Do not register from:
|
||||
|
||||
- MintPy work directories
|
||||
- ISCE2 runtime directories
|
||||
- ad hoc copied assets
|
||||
|
||||
### 8.2 Catalog model
|
||||
|
||||
Keep:
|
||||
|
||||
- `catalog_name = psinsar`
|
||||
- `product_family = timeseries`
|
||||
|
||||
Current catalog service already derives:
|
||||
|
||||
- product id
|
||||
- display name
|
||||
- stack identity
|
||||
- runtime and processor metadata
|
||||
- bbox from asset summaries
|
||||
- preview and primary asset paths
|
||||
|
||||
The next round should strengthen issue generation for missing assets and invalid package states.
|
||||
|
||||
### 8.3 Quarantine policy
|
||||
|
||||
Do not auto-delete broken publish packages.
|
||||
|
||||
If a rebuild finds broken packages, the preferred behavior is:
|
||||
|
||||
- keep package on disk
|
||||
- create `ResultIssueORM` records
|
||||
- downgrade `health_status`
|
||||
- use quarantine status only when the package is structurally unusable
|
||||
|
||||
This keeps auditability intact.
|
||||
|
||||
## 9. Implementation Strategy
|
||||
|
||||
### 9.1 Do not over-refactor first
|
||||
|
||||
The old design expected many new modules. That is no longer necessary because the codebase already has the main modules.
|
||||
|
||||
For the next round:
|
||||
|
||||
- keep `timeseries_service.py` as the orchestration center
|
||||
- keep `job_handlers.py` as worker entrypoints
|
||||
- keep `psinsar_catalog_service.py` as catalog authority
|
||||
- extract helper modules only when a block becomes independently reusable or too large
|
||||
|
||||
### 9.2 Recommended implementation phases
|
||||
|
||||
#### Phase A: contract and self-check hardening
|
||||
|
||||
Files likely involved:
|
||||
|
||||
- `backend/app/services/timeseries_service.py`
|
||||
- `backend/app/services/health_service.py`
|
||||
- `frontend/src/TimeseriesProductionPanel.jsx`
|
||||
|
||||
Target:
|
||||
|
||||
- stronger runtime report
|
||||
- batch preflight
|
||||
- clearer failure reasons
|
||||
|
||||
#### Phase B: quality summary and gating
|
||||
|
||||
Files likely involved:
|
||||
|
||||
- `backend/app/services/timeseries_service.py`
|
||||
- `experiments/isce2_sbas_timeseries/scripts/build_mintpy_publish_bundle.py`
|
||||
- `backend/app/services/psinsar_catalog_service.py`
|
||||
|
||||
Target:
|
||||
|
||||
- richer `quality_summary_json`
|
||||
- richer manifest quality block
|
||||
- stronger publish/register gates
|
||||
|
||||
#### Phase C: frontend run and catalog UX
|
||||
|
||||
Files likely involved:
|
||||
|
||||
- `frontend/src/TimeseriesProductionPanel.jsx`
|
||||
- `frontend/src/components/PsinsarCatalogPanel.jsx`
|
||||
- `frontend/src/api/timeseriesProduction.js`
|
||||
- `frontend/src/api/psinsarProducts.js`
|
||||
|
||||
Target:
|
||||
|
||||
- better preflight display
|
||||
- better run summary
|
||||
- richer product detail
|
||||
|
||||
#### Phase D: validation and operator closure
|
||||
|
||||
Target:
|
||||
|
||||
- one small LT-1 AOI end-to-end validation
|
||||
- one rerun-from-failure validation
|
||||
- catalog rebuild validation
|
||||
- deployment/ops notes update
|
||||
|
||||
## 10. Non-Goals For This Round
|
||||
|
||||
Do not include the following in the same implementation round:
|
||||
|
||||
- Gamma/PyINT timeseries integration
|
||||
- full PS-InSAR or StaMPS
|
||||
- atmospheric correction productization
|
||||
- topographic residual correction productization
|
||||
- large database namespace migration from `psinsar` to `timeseries`
|
||||
|
||||
These are valid future directions, but they should not be mixed into the current SBAS production hardening round.
|
||||
|
||||
## 11. Acceptance Criteria
|
||||
|
||||
The engineering expansion can be treated as complete for this round when all of the following are true:
|
||||
|
||||
1. An operator can run runtime preflight and understand failures before queuing a run.
|
||||
2. A stored PS stack batch can be submitted as one managed SBAS run.
|
||||
3. The run can execute through all eight workflow steps in the managed path.
|
||||
4. Failure at any step produces a clear error and supports controlled retry.
|
||||
5. The publish bundle is complete and canonical.
|
||||
6. The product is registered into the `psinsar` catalog from `manifest.json`.
|
||||
7. The frontend can show:
|
||||
- run state
|
||||
- workflow step state
|
||||
- published product linkage
|
||||
- product assets and quality summary
|
||||
8. `GET /api/health` remains healthy for:
|
||||
- `timeseries_result_catalog`
|
||||
- `product_packages`
|
||||
- `wsl_runtime`
|
||||
|
||||
## 12. Related Documents
|
||||
|
||||
- `docs/ISCE2_SBAS_TIMESERIES_DESIGN.md`
|
||||
- `docs/ISCE2_SBAS_PRODUCT_SPEC.md`
|
||||
- `docs/DEPLOYMENT.md`
|
||||
- `docs/CURRENT_STATUS_20260425.md`
|
||||
- `experiments/isce2_sbas_timeseries/README.md`
|
||||
@@ -1,6 +1,6 @@
|
||||
# ISCE2 SBAS Time-Series Production Design
|
||||
|
||||
Updated: 2026-04-06
|
||||
Updated: 2026-04-29
|
||||
|
||||
## 1. Goal
|
||||
|
||||
@@ -126,6 +126,33 @@ Current LT-1 stack experiment status already de-risks the processing side of pha
|
||||
- the original pair-oriented D-InSAR public entry was not removed
|
||||
- `backend/app/isce2_pipeline/run_lt1_dinsar_pipeline.py` still owns the existing workflow entry and now delegates shared input preparation to the helper
|
||||
|
||||
### 2.5 Stack Planning Thresholds
|
||||
|
||||
The planning entry that historically used the old PS preparation label is now treated as time-series stack preparation.
|
||||
The internal API path remains `/find-ps-timeseries` for compatibility.
|
||||
|
||||
Current threshold meaning:
|
||||
|
||||
- `initial_overlap_threshold`
|
||||
- single-scene AOI coverage gate
|
||||
- formula: `area(scene footprint ∩ AOI) / area(AOI)`
|
||||
- default: `0.30`
|
||||
- purpose: remove scenes that barely intersect the study area
|
||||
- `final_overlap_threshold`
|
||||
- final stack footprint consistency gate
|
||||
- formula: `area(common footprint of selected stack ∩ AOI) / min(area(each selected scene footprint ∩ AOI))`
|
||||
- default: `0.95`
|
||||
- purpose: ensure the retained stack has a stable common processing area without requiring each strip to cover the whole AOI
|
||||
|
||||
Planning algorithm rule:
|
||||
|
||||
- first group candidates by orbit direction, satellite family, imaging mode, and polarization
|
||||
- `LT1A` and `LT1B` are treated as the same `LT1` satellite family for stack planning
|
||||
- then search each compatible group for the largest stack whose footprint consistency satisfies `final_overlap_threshold`
|
||||
- when one or more outlier scenes break the common area, the planner may drop scenes until a valid stack is found
|
||||
- if no all-scene common-overlap stack exists, the planner may return a connected pairwise SBAS network when each retained network edge satisfies `final_overlap_threshold`
|
||||
- at least 3 scenes are required before a stack can be persisted as a `TimeseriesStackPlan`
|
||||
|
||||
## 3. Recommendation
|
||||
|
||||
### 3.1 Deliver SBAS first
|
||||
|
||||
@@ -81,3 +81,104 @@ Delivered adjustments:
|
||||
operators do not treat post-processing heuristics as part of the standard workflow.
|
||||
- Revalidated the modified pipeline, engine, and WSL runner modules with `python3 -m py_compile`
|
||||
inside the target WSL runtime environment.
|
||||
|
||||
## Additional Update: DEM Sidecar Path Repair
|
||||
|
||||
After switching the managed DEM bundle to a copied `SRTMDEM_RSP_SARscape` dataset under
|
||||
`D:\DEM`, an ISCE2 run failed in `topo` even though the outer pipeline XML pointed at the
|
||||
new location. The root cause was that the copied DEM sidecar XML files still contained old
|
||||
absolute `/mnt/...` paths in `file_name`, `metadata_location`, and `extra_file_name`.
|
||||
|
||||
Delivered adjustments:
|
||||
|
||||
- Added `backend/app/isce2_pipeline/repair_dem_sidecars.py` to audit and repair moved DEM
|
||||
sidecar XML files at directory scope.
|
||||
- Added sidecar self-repair for the selected ISCE2 DEM during pipeline resolution so the
|
||||
managed run no longer depends on manually editing copied `.xml` files first.
|
||||
- Documented the migration risk and the recommended repair command in `docs/DEPLOYMENT.md`.
|
||||
|
||||
## Additional Update: LT-1 Enhancement Alignment
|
||||
|
||||
The managed `ISCE2` `lt1_stripmap` production profile now enables the built-in
|
||||
stripmap enhancement path by default:
|
||||
|
||||
- dense offsets
|
||||
- range rubbersheeting
|
||||
- azimuth rubbersheeting
|
||||
|
||||
This was done to bring the default ISCE2 LT-1 production semantics closer to the
|
||||
existing SARscape `custom6` chain, which already includes non-trivial refinement
|
||||
steps rather than shipping a bare minimum interferometric result.
|
||||
|
||||
The implementation now passes these parameters end-to-end through:
|
||||
|
||||
- `backend/app/dinsar_engines/isce2_engine.py`
|
||||
- `deploy/wsl/runners/isce2_runner.py`
|
||||
- `backend/app/isce2_pipeline/run_lt1_dinsar_pipeline.py`
|
||||
|
||||
The rationale and the SARscape / ISCE2 mapping are documented in:
|
||||
|
||||
- `docs/ISCE2_LT1_ENHANCEMENT_ALIGNMENT_20260427.md`
|
||||
|
||||
## Additional Update: Rubbersheet Runtime Dependency
|
||||
|
||||
The first enhanced LT-1 run reached ISCE2 `dense_offsets` successfully and then
|
||||
failed at `rubber_sheet_range` with:
|
||||
|
||||
```text
|
||||
ModuleNotFoundError: No module named 'astropy'
|
||||
```
|
||||
|
||||
Root cause:
|
||||
|
||||
- ISCE2's `runRubbersheetRange.py` imports `astropy.convolution`.
|
||||
- The shared WSL runtime `insar_wsl_v1` had ISCE2 and SciPy installed, but did
|
||||
not include `astropy`.
|
||||
|
||||
Delivered adjustments:
|
||||
|
||||
- Added `astropy` to `deploy/wsl/conda/insar_wsl_v1.environment.yml`.
|
||||
- Added a runtime dependency preflight in the WSL runner and LT-1 pipeline so
|
||||
rubbersheeting fails immediately with a clear message instead of after the
|
||||
dense-offset stage has already run.
|
||||
- Added `astropy.convolution` to the ISCE2 WSL availability check.
|
||||
|
||||
Required deployment action:
|
||||
|
||||
```bash
|
||||
conda install -n insar_wsl_v1 -c conda-forge astropy
|
||||
```
|
||||
|
||||
## Additional Update: Real Ionosphere Stage Integration
|
||||
|
||||
The managed `ISCE2` LT-1 stripmap workflow now runs the native stripmap
|
||||
dispersive correction path instead of faking `PICKLE/ionosphere` state during
|
||||
resume.
|
||||
|
||||
Delivered adjustments:
|
||||
|
||||
- Enabled `do split spectrum = True` and `do dispersive = True` in the generated
|
||||
`stripmapApp` XML.
|
||||
- Changed stage-2 execution from a narrow `unwrap -> unwrap` run to a real
|
||||
`filter_low_band/unwrap/ionosphere` resume path.
|
||||
- Changed stage-3 execution to resume from real `ionosphere` state when present,
|
||||
or from the low/high-band unwrap state when only stage-2 products exist.
|
||||
- Extended the reduced geocode export list with:
|
||||
- `ionosphere/dispersive.bil.unwCor.filt`
|
||||
- `ionosphere/nondispersive.bil.unwCor.filt`
|
||||
- `ionosphere/mask.bil`
|
||||
- Updated the export step to prefer geocoded
|
||||
`ionosphere/nondispersive.bil.unwCor.filt` when available.
|
||||
|
||||
Operational effect:
|
||||
|
||||
- `resume_from=unwrap` now resumes the complete stage-2 chain up to
|
||||
`ionosphere`
|
||||
- `resume_from=geocode` now performs real `ionosphere -> geocode` continuation
|
||||
instead of relying on copied pickle files
|
||||
|
||||
Deployment note:
|
||||
|
||||
- The native ionosphere implementation imports `cv2` and `scipy`
|
||||
- The WSL runner, pipeline preflight, and health check now verify those modules
|
||||
before production starts
|
||||
|
||||
Reference in New Issue
Block a user