feat: engineer SBAS timeseries production workflow

This commit is contained in:
2026-04-29 14:43:31 +08:00
parent dace8b20f6
commit 4c0d1f2c2b
54 changed files with 5843 additions and 201 deletions
+141 -2
View File
@@ -205,7 +205,7 @@ python run_worker.py
这套链路要求:
- `.env` 中的根目录配置要真实可访问。
- `backend/migrations/001``006` 必须保持幂等。
- `backend/migrations/001``007` 必须保持幂等。
- 启动自维护默认是保守模式,不会在 schema 不匹配时自动破坏性重建。
## 6. 数据库自维护边界
@@ -216,7 +216,7 @@ python run_worker.py
- 自动创建 `postgis` 扩展
- 自动创建缺失表
- 自动补齐缺失列
- 自动执行 `001``006` 号 SQL 文件
- 自动执行 `001``007` 号 SQL 文件
- 自动引导管理员账号与灾害点初始化
- 不支持:
@@ -304,3 +304,142 @@ VITE_TILE_SERVER_TOKEN=change_me
- [DATABASE_SELF_MAINTENANCE_AUDIT_20260425.md](DATABASE_SELF_MAINTENANCE_AUDIT_20260425.md)
- [PRODUCTION_RESULTS_MULTI_ENGINE_DESIGN_20260423.md](PRODUCTION_RESULTS_MULTI_ENGINE_DESIGN_20260423.md)
- [WSL_RUNTIME_REFACTOR_DESIGN_20260422.md](WSL_RUNTIME_REFACTOR_DESIGN_20260422.md)
## DEM Sidecar Migration Warning
If you copy or move a prepared ISCE DEM bundle to a new directory or machine, do not assume
that the sidecar XML files are portable as-is.
Affected files typically include:
- `<dem>.xml`
- `<dem>.vrt`
- `<dem>.wgs84`
- `<dem>.wgs84.xml`
- `<dem>.wgs84.vrt`
The ISCE XML sidecars may still contain absolute historical paths in:
- `file_name`
- `metadata_location`
- `extra_file_name`
Typical failure symptom:
- `verifyDEM` succeeds, but `topo` fails with `FileNotFoundError` pointing at an old
`/mnt/...` path from the previous machine or previous directory layout.
Recommended post-migration repair:
```powershell
C:\ProgramData\anaconda3\envs\InSAR\python.exe `
backend\app\isce2_pipeline\repair_dem_sidecars.py `
--root D:\DEM `
--repair
```
The managed ISCE2 pipeline now repairs the selected DEM sidecar paths before running, but
the directory-level repair is still recommended after deployment or storage migration so the
whole DEM bundle remains internally consistent.
## ISCE2 Rubbersheeting Runtime Dependency
The managed `ISCE2` `lt1_stripmap` profile enables dense offsets plus range and azimuth
rubbersheeting by default. This is an ISCE2 native workflow step, not an export-time
correction.
The range rubbersheeting implementation imports `astropy.convolution`, so every deployed
or migrated WSL runtime must include `astropy` in `insar_wsl_v1`.
Check:
```bash
/home/administrator/miniconda3/envs/insar_wsl_v1/bin/python -c "from astropy.convolution import convolve; print('astropy_ok')"
```
Repair:
```bash
conda install -n insar_wsl_v1 -c conda-forge astropy
```
If this package is missing, production now fails during preflight instead of after the
long dense-offset stage.
## ISCE2 Ionosphere Runtime Dependency
The managed `ISCE2` `lt1_stripmap` profile now keeps the standard stripmap
`split-spectrum -> low/high-band unwrap -> ionosphere -> geocode` path enabled.
This is part of the native workflow and replaces the older fake `PICKLE/ionosphere`
resume shortcut.
Operational consequences:
- `resume_from=unwrap` now resumes the full stage-2 chain up to `ionosphere`
- `resume_from=geocode` now starts from real `ionosphere` state when available
- the export step prefers geocoded `ionosphere/nondispersive.bil.unwCor.filt`
when that product exists
The ionosphere implementation in ISCE2 imports `cv2` and `scipy`, so every
deployed or migrated WSL runtime must include both packages.
Check:
```bash
/home/administrator/miniconda3/envs/insar_wsl_v1/bin/python -c "import cv2, scipy; print('ionosphere_ok')"
```
Repair:
```bash
conda install -n insar_wsl_v1 -c conda-forge opencv scipy
```
If these packages are missing, production now fails during preflight instead of
after the long stripmap filtering / unwrap stage.
## Git Clone Bootstrap
Goal: a clean `git clone` on a new Windows host should already contain the
deployment entrypoints required to install dependencies, validate `.env`, and
start the system.
Recommended path:
```powershell
git clone <repo-url>
cd Insar_management_system_v2
Copy-Item .env.example .env
notepad .env
powershell -ExecutionPolicy Bypass -File .\scripts\bootstrap_clone.ps1 -InitFrontend -BuildFrontend
start_system.bat
```
Optional runtime bootstrap:
```powershell
powershell -ExecutionPolicy Bypass -File .\scripts\bootstrap_clone.ps1 -InitWindowsConda
powershell -ExecutionPolicy Bypass -File .\scripts\bootstrap_clone.ps1 -InitWslConda
powershell -ExecutionPolicy Bypass -File .\scripts\bootstrap_clone.ps1 -All
```
`scripts/bootstrap_clone.ps1` is intentionally conservative:
- It copies `.env.example` to `.env` only when `.env` is missing.
- It can create or update the Windows conda env from `environment.yml`.
- It can create or update the shared WSL conda env from
`deploy/wsl/conda/insar_wsl_v1.environment.yml`.
- It can run `npm ci` and `npm run build` in `frontend/`.
- It runs `scripts/check_runtime_config.py` unless `-SkipChecks` is specified.
- `start_system.bat` now fails early when `frontend/dist/index.html` is missing.
- It does not auto-start backend, worker, or Nginx.
- It does not bypass the existing database self-maintenance safeguards.
Typical deployment sequence:
1. Edit `.env` to match the new server.
2. Run `bootstrap_clone.ps1` with the switches required by that server.
3. Run `start_system.bat`.
4. Check `GET /api/health`.
5. Trigger a small real production task and confirm the result is registered in
the catalog.
+9
View File
@@ -49,6 +49,8 @@
- [ISCE2_SBAS_TIMESERIES_DESIGN.md](ISCE2_SBAS_TIMESERIES_DESIGN.md)
- [ISCE2_SBAS_PRODUCT_SPEC.md](ISCE2_SBAS_PRODUCT_SPEC.md)
- [ISCE2_SBAS_ENGINEERING_DESIGN_20260428.md](ISCE2_SBAS_ENGINEERING_DESIGN_20260428.md)
Current-phase engineering design for the managed ISCE2 + MintPy SBAS route.
说明:
@@ -83,6 +85,13 @@
- [SECURITY_AUDIT_2026-03-12.md](SECURITY_AUDIT_2026-03-12.md)
## Clone Bootstrap
- [../scripts/bootstrap_clone.ps1](../scripts/bootstrap_clone.ps1)
Fresh-server bootstrap entry for clone-based deployment. It keeps `.env`
initialization, frontend dependency install/build, runtime bootstrap, and
deployment validation in one place without changing the main startup chain.
## 7. 工作笔记
- [../INIT.md](../INIT.md)
@@ -0,0 +1,154 @@
# ISCE2 LT-1 Enhancement Alignment 2026-04-27
## Purpose
This note records why the managed `ISCE2` `lt1_stripmap` production profile now
enables the built-in stripmap enhancement steps by default, and how that choice
relates to the existing SARscape `custom6` production chain.
## Background
The SARscape `custom6` chain already goes beyond a bare minimum D-InSAR run.
Its production semantics include:
1. Interferogram generation
2. Filtering and coherence
3. Orbital trend / residual phase frequency removal
4. Phase unwrapping
5. GCP-based refinement and reflattening
6. Phase to displacement and geocoding
This means the current LT-1 production baseline in the system is not a
scientifically "raw" interferometric export. It is an operationally enhanced
delivery chain.
## ISCE2 Mapping
ISCE2 stripmap does not expose the exact same SARscape modules, but it does
provide native enhancement steps that address the same operational risk class:
residual misregistration and geometry-driven long-wavelength artifacts.
The relevant built-in ISCE2 controls are:
- `doDenseOffsets`
- `doRubbersheetingRange`
- `doRubbersheetingAzimuth`
- `do split spectrum`
- `do dispersive`
- `rubberSheetSNRThreshold`
- `rubberSheetFilterSize`
When enabled, the stripmap workflow:
- estimates dense offsets from cross-correlation
- filters / masks those offsets
- updates the geometry offset fields
- performs a fine resampling pass using the corrected offsets
- unwraps low/high-band interferograms and estimates a dispersive ionosphere term
- geocodes the ionosphere-corrected nondispersive phase for delivery
This is not identical to SARscape's `RemoveResidualPhaseFrequency` plus
`RefinementAndReflattening`, but it is the closest native ISCE2 enhancement
path inside the standard stripmap application.
## Production Decision
The managed `ISCE2` `lt1_stripmap` profile now treats these steps as part of the
default LT-1 production workflow:
- split-spectrum ionosphere correction: enabled
- dense offsets: enabled
- range rubbersheeting: enabled
- azimuth rubbersheeting: enabled
Default numeric parameters:
- `rubberSheetSNRThreshold = 5.0`
- `rubberSheetFilterSize = 9`
- `denseWindowWidth = 64`
- `denseWindowHeight = 64`
- `denseSearchWidth = 20`
- `denseSearchHeight = 20`
- `denseSkipWidth = 32`
- `denseSkipHeight = 32`
These defaults are stored as profile semantics in code, not as loose `.env`
feature toggles.
## Runtime Dependency
The stripmap ionosphere implementation imports `cv2` and `scipy`.
Deployment check:
```bash
/home/administrator/miniconda3/envs/insar_wsl_v1/bin/python -c "import cv2, scipy; print('ionosphere_ok')"
```
Repair command for an existing runtime:
```bash
conda install -n insar_wsl_v1 -c conda-forge opencv scipy
```
The range rubbersheeting implementation in ISCE2 imports
`astropy.convolution` from `runRubbersheetRange.py`. The shared WSL conda
runtime therefore must include `astropy`.
Deployment check:
```bash
/home/administrator/miniconda3/envs/insar_wsl_v1/bin/python -c "from astropy.convolution import convolve; print('astropy_ok')"
```
Repair command for an existing runtime:
```bash
conda install -n insar_wsl_v1 -c conda-forge astropy
```
## Boundary
This change does **not** mean that every long-wavelength ramp problem is solved.
It only means the default managed ISCE2 LT-1 profile now includes the native
registration-enhancement path that was previously omitted.
If a run still shows a strong residual scene-wide ramp after rubbersheeting,
that should be treated as a separate quality / post-processing issue and should
be diagnosed explicitly rather than silently hidden inside export logic.
## Operational Implication
When comparing current SARscape and ISCE2 LT-1 products:
- SARscape `custom6` remains the more explicitly refined chain
- ISCE2 `lt1_stripmap` is no longer a bare stripmap baseline
- both engines now include standard enhancement intent in default production
This makes cross-engine behavior more defensible for LT-1 operational delivery.
## Operator Controls
As of 2026-04-29, the production UI no longer hides these choices behind code
defaults only.
The ISCE2 production panel now exposes the managed LT-1 profile parameters in
three groups:
- `Execution`
- `Delivery`
- `Enhancement`
The user-visible controls now cover:
- split-spectrum ionosphere correction on/off
- dense offsets on/off
- range rubbersheeting on/off
- azimuth rubbersheeting on/off
- reference normalization mode (`coh_median` or `none`)
- deramp mode (`plane` or `none`)
This keeps the default managed behavior unchanged, while allowing operators to
fall back toward a more conservative stripmap delivery path when a specific
scene looks worse after enhancement.
@@ -0,0 +1,663 @@
# ISCE2 + MintPy SBAS Engineering Design
Updated: 2026-04-28
## 1. Purpose
This document defines the engineering expansion plan for the current stack-based time-series InSAR route:
- `LT-1 stack batch -> ISCE2 stripmapStack -> MintPy SBAS -> publish bundle -> psinsar catalog`
The repository already has a working phase-1 skeleton. The goal of this document is not to restart the design from zero, but to align the next implementation round with the code that already exists in:
- `backend/app/services/timeseries_service.py`
- `backend/app/routers/timeseries_production.py`
- `backend/app/services/psinsar_catalog_service.py`
- `frontend/src/TimeseriesProductionPanel.jsx`
- `frontend/src/components/PsinsarCatalogPanel.jsx`
This document supersedes the "missing pieces" parts of `docs/ISCE2_SBAS_TIMESERIES_DESIGN.md` for the current implementation phase.
## 2. Decisions
### 2.1 Primary processing route
Keep the current scientific split:
- ISCE2 is responsible for LT-1 stack preparation, stack geometry, co-registration, baseline generation, interferogram generation, and unwrap inputs.
- MintPy is responsible for SBAS inversion and time-series products.
- The system registers only publish-grade bundles, not raw MintPy work directories.
This means the production claim for the current phase is:
- `SBAS time-series production on top of ISCE2 + MintPy`
It is not:
- full PS-InSAR
- full StaMPS integration
- full commercial-grade atmospheric/error-correction stack
### 2.2 Keep the current business model
Use the current model already implemented in code:
- planning-layer stack snapshot:
- `PsTaskBatchORM`
- `PsTaskItemORM`
- business-facing production run:
- `PsTimeseriesRunORM`
- step orchestration:
- `WorkflowRunORM`
- `WorkflowStepORM`
- publish/catalog registration:
- `ResultProductORM`
- `ResultAssetORM`
- `ResultIssueORM`
Do not redesign the run model into a new engine abstraction in this round.
### 2.3 Keep naming stable for now
Current naming in the repository is mixed:
- product family shown to users: `timeseries`
- processing mode: `sbas`
- catalog namespace and package schema legacy: `psinsar`
For this round:
- keep `product_family = timeseries`
- keep `mode = sbas`
- keep `catalog_name = psinsar` for compatibility
- keep `psinsar.publish.v1` ingestion support working
Do not do a DB/API namespace rename and a pipeline hardening round at the same time.
## 2.4 Current implementation scope (2026-04-28)
This round is intentionally constrained to avoid impact on other production business:
- no DB schema migration
- no catalog namespace rename
- no workflow framework refactor
- no change to the system self-maintenance / self-check contract
The implementation landed in this round focuses on pipeline hardening around the existing `PsTimeseriesRunORM` path:
- preflight gating before SBAS run creation
- stronger runtime self-check visibility
- publish-bundle validation before catalog registration
- frontend visibility for preflight, runtime checks, and publish validation
This means the current engineering target is:
- make the existing ISCE2 + MintPy SBAS route operationally safer
not:
- redesign the overall architecture
- replace the existing result registration model
- introduce a second persistence path for timeseries products
### 2.5 Phase-2 scope (2026-04-28)
The second implementation round keeps the same production chain but upgrades the planning trace from ad-hoc JSON to first-class additive schema objects.
Additive schema only:
- new planning tables:
- `TimeseriesStackPlanORM`
- `TimeseriesStackPlanItemORM`
- nullable trace columns on existing objects:
- `PsTaskBatchORM.plan_id`
- `PsTaskBatchORM.plan_strategy`
- `PsTaskItemORM.plan_item_ref_id`
- `PsTimeseriesRunORM.plan_id`
- `PsTimeseriesRunORM.plan_strategy`
Operational rules for phase 2:
- do not introduce a separate migration framework
- rely on the existing database self-maintenance path:
- `Base.metadata.create_all()`
- missing-column auto-add in `backend/app/db_maintenance.py`
- keep phase-1 `planning_context` / `remark` compatibility for old batches
The engineering target of phase 2 is:
- formalize `plan -> batch -> run -> publish bundle -> catalog product` traceability
- expose `plan_id` in frontend production and result views
- keep old batches runnable without backfilling or hard migration
- expose `GET /timeseries-plans/{plan_id}` for plan audit/detail lookup
## 3. Current Baseline In Code
The current code already implements the core production skeleton.
### 3.1 Run record and workflow
`backend/app/models/orm.py`
- `PsTimeseriesRunORM` already stores:
- run identity
- batch binding
- processor/runtime metadata
- work and publish roots
- input/orbit/quality summaries
- failure state
- `WorkflowRunORM` and `WorkflowStepORM` already support DAG execution and retry.
- `SystemJobORM` already supports queued worker execution per workflow step.
### 3.2 Current workflow steps
`backend/app/services/timeseries_service.py`
Current step chain is already eight steps:
1. `prepare`
2. `stack_prep_initial`
3. `materialize`
4. `stack_prep_refresh`
5. `run_isce2_stack`
6. `run_mintpy_sbas`
7. `export_publish_bundle`
8. `register_psinsar_product`
This is already the correct backbone for the managed SBAS route.
### 3.3 Current scientific execution boundary
The scientific boundary is still script-based, and that is acceptable for now:
- `experiments/isce2_sbas_timeseries/scripts/build_lt1_stack_prep.py`
- `experiments/isce2_sbas_timeseries/scripts/materialize_lt1_stack_scenes.py`
- `experiments/isce2_sbas_timeseries/scripts/prepare_lt1_stack_dem.py`
- `experiments/isce2_sbas_timeseries/scripts/run_generated_stack_runfile_ubuntu2404.sh`
- `experiments/isce2_sbas_timeseries/scripts/run_mintpy_sbas_unified_env_smoketest_ubuntu2404.sh`
- `experiments/isce2_sbas_timeseries/scripts/export_mintpy_publish_products_ubuntu2404.sh`
The current implementation should continue to wrap these scripts instead of rewriting the scientific logic prematurely.
### 3.4 Current frontend and ops surface
Already present:
- run submission and run detail:
- `frontend/src/TimeseriesProductionPanel.jsx`
- product catalog panel:
- `frontend/src/components/PsinsarCatalogPanel.jsx`
- health-check visibility:
- `frontend/src/HealthCheckPanel.jsx`
- catalog rebuild API:
- `backend/app/routers/ps_products.py`
So the next round is a hardening and extension round, not an empty scaffold round.
## 4. Main Gaps
The next engineering work should focus on the following gaps.
### 4.1 Self-check is present but still shallow
Current runtime check already validates:
- WSL distro
- Python path
- stack script path
- configured helper scripts
- MintPy import
- DEM/orbit/output root presence
What is still missing:
- write permission checks for work and publish roots
- DEM sidecar consistency checks
- runtime dependency checks for scientific imports used by ISCE2/MintPy
- batch-level readiness checks before a run is queued
- publish-bundle structural validation before catalog registration
### 4.2 Quality summary exists, but quality gating is weak
Current code validates:
- stack prep readiness
- required run files
- required ISCE2 output directories
- required MintPy outputs
- publish manifest existence
But it still does not promote enough scientific quality indicators into release gates, for example:
- interferogram count versus expected network count
- non-empty unwrap/correlation outputs
- valid-pixel ratio after `maskAllValid`
- temporal coherence thresholds
- reference point presence and stability summary
### 4.3 Frontend is functional but still operationally thin
Current frontend can:
- submit a run
- run WSL check
- list runs
- show workflow steps
- retry failed workflow steps
- browse catalog entries
Still missing:
- structured preflight diagnostics for the selected batch
- clearer phase summaries per run
- direct visibility into quality summaries and key artifacts
- better linkage between run detail and published product detail
- a richer product detail view closer to the D-InSAR catalog panel depth
### 4.4 Result management needs stricter contract enforcement
The catalog path is correct, but the following rules should be made explicit and enforced:
- `manifest.json` is the only registration entrypoint
- every published run must have a stable `publish_dir`
- required assets must exist before registration
- missing assets should generate catalog issues and possibly quarantine status
- every product should carry:
- processor code
- runtime id
- native output trace
- stack identity
## 5. Target Pipeline
### 5.1 Input contract
The run input must remain stack-based, not pair-based.
Source objects:
- one `ps_task_batch`
- many `ps_task_items`
- one selected stack manifest:
- `input/selected_stack_manifest.json`
- one generated stack manifest:
- `input/stack_input_manifest.json`
The selected manifest is the planning snapshot.
The generated stack manifest is the execution snapshot and must include:
- stack dates
- reference date
- stack key
- group key
- resolved DEM and orbit dependencies
- readiness flags
- blocking reasons
- generated ISCE2 command arguments
### 5.2 Runtime directory model
Keep the current directory split:
- work root:
- `backend/runtime/timeseries_work/<run_id>/...`
- publish root:
- `TIMESERIES_PRODUCT_DIR/<stack_key>/runs/<run_id>/...`
Recommended internal layout under the work root:
- `input/`
- `inputs/dem/`
- `stack_work/`
- `logs/`
- `mintpy/`
Recommended publish layout:
- `manifest.json`
- `assets/`
- `preview/`
- `metadata/`
### 5.3 Managed workflow
The current eight-step chain is the correct managed workflow and should be kept:
1. `prepare`
- validate batch
- resolve stack identity
- choose reference date
- write `selected_stack_manifest.json`
2. `stack_prep_initial`
- generate execution-layer stack manifest
- resolve DEM/orbits
- tell the system whether materialization is the only blocker
3. `materialize`
- materialize LT-1 scenes into stack input layout
- materialize orbit XML and local dependencies
4. `stack_prep_refresh`
- re-run readiness check after materialization
- must reach ready state
5. `run_isce2_stack`
- prepare local DEM sidecars
- generate run files
- run `run_01` to `run_08`
- validate `geom_reference`, `baselines`, and `Igrams`
6. `run_mintpy_sbas`
- write MintPy config
- run controlled `smallbaselineApp`
- validate core MintPy outputs
7. `export_publish_bundle`
- geocode MintPy outputs
- export GeoTIFF browse layers
- generate preview and manifest
- augment the manifest with canonical metadata
8. `register_psinsar_product`
- register the publish bundle into catalog
- mark the run as published
### 5.4 Current publish contract
Keep `docs/ISCE2_SBAS_PRODUCT_SPEC.md` as the publish contract source of truth.
Required publish assets for the managed SBAS route:
- `assets/geo_timeseries.h5`
- `assets/geo_velocity.h5`
- `assets/velocity.tif`
- `assets/geo_temporalCoherence.h5`
- `assets/geo_maskTempCoh.h5`
- `preview/velocity_preview.png`
- `metadata/smallbaselineApp.cfg`
- `manifest.json`
Optional but recommended:
- `preview/numTriNonzeroIntAmbiguity.png`
- extra quality JSON files
## 6. Self-Check Design
Self-check should exist at four levels.
### 6.1 Runtime preflight
Primary entry:
- `POST /timeseries-production/wsl-check`
Current checks should be kept and extended with:
- WSL distro reachable
- configured Python reachable
- ISCE2 stack script import/help check
- MintPy import check
- helper script existence checks
- DEM root existence
- orbit pool existence
- publish root existence
- work root existence
- write-test for work root
- write-test for publish root
- DEM sidecar consistency check
- optional import checks for:
- `cv2`
- `scipy`
- `astropy`
Return structure should remain machine-readable so the frontend can render a diagnostic card instead of a plain message string.
### 6.2 Batch preflight
Add a run-specific preflight before or during `create_run`.
Minimum checks:
- scene count meets SBAS minimum
- all scene dates are valid
- scene dates are unique
- direction is consistent
- source files exist and are readable
- orbit coverage is complete or explicitly degraded
- `group_key` and `stack_key` are derivable
- publish path does not collide with another active run
Recommended surface:
- a new backend helper in `timeseries_service.py`
- frontend summary block in `TimeseriesProductionPanel.jsx`
### 6.3 In-run gates
Each workflow step should continue to fail fast when hard requirements are not met.
Required gates:
- `stack_prep_refresh` must report `ready_for_stackStripMap_nofocus = true`
- all expected run files must exist before stack execution
- `run_08_igram` output directories must exist
- MintPy required outputs must exist and be non-empty
- export must generate a manifest plus required assets
- registration must succeed against the catalog service
### 6.4 Post-publish health
Health is not only "the run finished".
The catalog and package checks must continue to validate:
- manifest exists
- publish dir exists
- processor code present
- runtime id present for WSL-native engines
- native output dir present
- canonical package schema valid
- manifest count versus DB count consistency
## 7. Frontend Design
### 7.1 Timeseries production panel
Keep `frontend/src/TimeseriesProductionPanel.jsx` as the main run workspace.
Planned enhancements:
- show structured runtime preflight results
- show batch preflight results before submission
- show phase-oriented run summary:
- input prepared
- stack ready
- ISCE2 complete
- MintPy complete
- exported
- published
- show key paths and quality summary blocks without forcing the operator to inspect raw JSON
- keep failed-step retry
- add clearer linkage to the published product once available
### 7.2 Product catalog panel
Keep `frontend/src/components/PsinsarCatalogPanel.jsx` as the catalog entry.
Planned enhancements:
- retain catalog status and rebuild actions
- enrich product detail with:
- stack identity
- processor/runtime identity
- preview and primary assets
- quality summary
- asset list
- issue list
- coverage summary
- keep the publish bundle as the fact source
### 7.3 Health panel
Keep health visibility in `frontend/src/HealthCheckPanel.jsx`.
The timeseries section should continue to show:
- catalog status
- rebuild need
- manifest vs DB counts
- issue count
It should remain aligned with:
- `timeseries_result_catalog`
- `product_packages`
- `wsl_runtime`
## 8. Result Management and Registration
### 8.1 Registration rule
Register only from:
- `<publish_dir>/manifest.json`
Do not register from:
- MintPy work directories
- ISCE2 runtime directories
- ad hoc copied assets
### 8.2 Catalog model
Keep:
- `catalog_name = psinsar`
- `product_family = timeseries`
Current catalog service already derives:
- product id
- display name
- stack identity
- runtime and processor metadata
- bbox from asset summaries
- preview and primary asset paths
The next round should strengthen issue generation for missing assets and invalid package states.
### 8.3 Quarantine policy
Do not auto-delete broken publish packages.
If a rebuild finds broken packages, the preferred behavior is:
- keep package on disk
- create `ResultIssueORM` records
- downgrade `health_status`
- use quarantine status only when the package is structurally unusable
This keeps auditability intact.
## 9. Implementation Strategy
### 9.1 Do not over-refactor first
The old design expected many new modules. That is no longer necessary because the codebase already has the main modules.
For the next round:
- keep `timeseries_service.py` as the orchestration center
- keep `job_handlers.py` as worker entrypoints
- keep `psinsar_catalog_service.py` as catalog authority
- extract helper modules only when a block becomes independently reusable or too large
### 9.2 Recommended implementation phases
#### Phase A: contract and self-check hardening
Files likely involved:
- `backend/app/services/timeseries_service.py`
- `backend/app/services/health_service.py`
- `frontend/src/TimeseriesProductionPanel.jsx`
Target:
- stronger runtime report
- batch preflight
- clearer failure reasons
#### Phase B: quality summary and gating
Files likely involved:
- `backend/app/services/timeseries_service.py`
- `experiments/isce2_sbas_timeseries/scripts/build_mintpy_publish_bundle.py`
- `backend/app/services/psinsar_catalog_service.py`
Target:
- richer `quality_summary_json`
- richer manifest quality block
- stronger publish/register gates
#### Phase C: frontend run and catalog UX
Files likely involved:
- `frontend/src/TimeseriesProductionPanel.jsx`
- `frontend/src/components/PsinsarCatalogPanel.jsx`
- `frontend/src/api/timeseriesProduction.js`
- `frontend/src/api/psinsarProducts.js`
Target:
- better preflight display
- better run summary
- richer product detail
#### Phase D: validation and operator closure
Target:
- one small LT-1 AOI end-to-end validation
- one rerun-from-failure validation
- catalog rebuild validation
- deployment/ops notes update
## 10. Non-Goals For This Round
Do not include the following in the same implementation round:
- Gamma/PyINT timeseries integration
- full PS-InSAR or StaMPS
- atmospheric correction productization
- topographic residual correction productization
- large database namespace migration from `psinsar` to `timeseries`
These are valid future directions, but they should not be mixed into the current SBAS production hardening round.
## 11. Acceptance Criteria
The engineering expansion can be treated as complete for this round when all of the following are true:
1. An operator can run runtime preflight and understand failures before queuing a run.
2. A stored PS stack batch can be submitted as one managed SBAS run.
3. The run can execute through all eight workflow steps in the managed path.
4. Failure at any step produces a clear error and supports controlled retry.
5. The publish bundle is complete and canonical.
6. The product is registered into the `psinsar` catalog from `manifest.json`.
7. The frontend can show:
- run state
- workflow step state
- published product linkage
- product assets and quality summary
8. `GET /api/health` remains healthy for:
- `timeseries_result_catalog`
- `product_packages`
- `wsl_runtime`
## 12. Related Documents
- `docs/ISCE2_SBAS_TIMESERIES_DESIGN.md`
- `docs/ISCE2_SBAS_PRODUCT_SPEC.md`
- `docs/DEPLOYMENT.md`
- `docs/CURRENT_STATUS_20260425.md`
- `experiments/isce2_sbas_timeseries/README.md`
+28 -1
View File
@@ -1,6 +1,6 @@
# ISCE2 SBAS Time-Series Production Design
Updated: 2026-04-06
Updated: 2026-04-29
## 1. Goal
@@ -126,6 +126,33 @@ Current LT-1 stack experiment status already de-risks the processing side of pha
- the original pair-oriented D-InSAR public entry was not removed
- `backend/app/isce2_pipeline/run_lt1_dinsar_pipeline.py` still owns the existing workflow entry and now delegates shared input preparation to the helper
### 2.5 Stack Planning Thresholds
The planning entry that historically used the old PS preparation label is now treated as time-series stack preparation.
The internal API path remains `/find-ps-timeseries` for compatibility.
Current threshold meaning:
- `initial_overlap_threshold`
- single-scene AOI coverage gate
- formula: `area(scene footprint ∩ AOI) / area(AOI)`
- default: `0.30`
- purpose: remove scenes that barely intersect the study area
- `final_overlap_threshold`
- final stack footprint consistency gate
- formula: `area(common footprint of selected stack ∩ AOI) / min(area(each selected scene footprint ∩ AOI))`
- default: `0.95`
- purpose: ensure the retained stack has a stable common processing area without requiring each strip to cover the whole AOI
Planning algorithm rule:
- first group candidates by orbit direction, satellite family, imaging mode, and polarization
- `LT1A` and `LT1B` are treated as the same `LT1` satellite family for stack planning
- then search each compatible group for the largest stack whose footprint consistency satisfies `final_overlap_threshold`
- when one or more outlier scenes break the common area, the planner may drop scenes until a valid stack is found
- if no all-scene common-overlap stack exists, the planner may return a connected pairwise SBAS network when each retained network edge satisfies `final_overlap_threshold`
- at least 3 scenes are required before a stack can be persisted as a `TimeseriesStackPlan`
## 3. Recommendation
### 3.1 Deliver SBAS first
@@ -81,3 +81,104 @@ Delivered adjustments:
operators do not treat post-processing heuristics as part of the standard workflow.
- Revalidated the modified pipeline, engine, and WSL runner modules with `python3 -m py_compile`
inside the target WSL runtime environment.
## Additional Update: DEM Sidecar Path Repair
After switching the managed DEM bundle to a copied `SRTMDEM_RSP_SARscape` dataset under
`D:\DEM`, an ISCE2 run failed in `topo` even though the outer pipeline XML pointed at the
new location. The root cause was that the copied DEM sidecar XML files still contained old
absolute `/mnt/...` paths in `file_name`, `metadata_location`, and `extra_file_name`.
Delivered adjustments:
- Added `backend/app/isce2_pipeline/repair_dem_sidecars.py` to audit and repair moved DEM
sidecar XML files at directory scope.
- Added sidecar self-repair for the selected ISCE2 DEM during pipeline resolution so the
managed run no longer depends on manually editing copied `.xml` files first.
- Documented the migration risk and the recommended repair command in `docs/DEPLOYMENT.md`.
## Additional Update: LT-1 Enhancement Alignment
The managed `ISCE2` `lt1_stripmap` production profile now enables the built-in
stripmap enhancement path by default:
- dense offsets
- range rubbersheeting
- azimuth rubbersheeting
This was done to bring the default ISCE2 LT-1 production semantics closer to the
existing SARscape `custom6` chain, which already includes non-trivial refinement
steps rather than shipping a bare minimum interferometric result.
The implementation now passes these parameters end-to-end through:
- `backend/app/dinsar_engines/isce2_engine.py`
- `deploy/wsl/runners/isce2_runner.py`
- `backend/app/isce2_pipeline/run_lt1_dinsar_pipeline.py`
The rationale and the SARscape / ISCE2 mapping are documented in:
- `docs/ISCE2_LT1_ENHANCEMENT_ALIGNMENT_20260427.md`
## Additional Update: Rubbersheet Runtime Dependency
The first enhanced LT-1 run reached ISCE2 `dense_offsets` successfully and then
failed at `rubber_sheet_range` with:
```text
ModuleNotFoundError: No module named 'astropy'
```
Root cause:
- ISCE2's `runRubbersheetRange.py` imports `astropy.convolution`.
- The shared WSL runtime `insar_wsl_v1` had ISCE2 and SciPy installed, but did
not include `astropy`.
Delivered adjustments:
- Added `astropy` to `deploy/wsl/conda/insar_wsl_v1.environment.yml`.
- Added a runtime dependency preflight in the WSL runner and LT-1 pipeline so
rubbersheeting fails immediately with a clear message instead of after the
dense-offset stage has already run.
- Added `astropy.convolution` to the ISCE2 WSL availability check.
Required deployment action:
```bash
conda install -n insar_wsl_v1 -c conda-forge astropy
```
## Additional Update: Real Ionosphere Stage Integration
The managed `ISCE2` LT-1 stripmap workflow now runs the native stripmap
dispersive correction path instead of faking `PICKLE/ionosphere` state during
resume.
Delivered adjustments:
- Enabled `do split spectrum = True` and `do dispersive = True` in the generated
`stripmapApp` XML.
- Changed stage-2 execution from a narrow `unwrap -> unwrap` run to a real
`filter_low_band/unwrap/ionosphere` resume path.
- Changed stage-3 execution to resume from real `ionosphere` state when present,
or from the low/high-band unwrap state when only stage-2 products exist.
- Extended the reduced geocode export list with:
- `ionosphere/dispersive.bil.unwCor.filt`
- `ionosphere/nondispersive.bil.unwCor.filt`
- `ionosphere/mask.bil`
- Updated the export step to prefer geocoded
`ionosphere/nondispersive.bil.unwCor.filt` when available.
Operational effect:
- `resume_from=unwrap` now resumes the complete stage-2 chain up to
`ionosphere`
- `resume_from=geocode` now performs real `ionosphere -> geocode` continuation
instead of relying on copied pickle files
Deployment note:
- The native ionosphere implementation imports `cv2` and `scipy`
- The WSL runner, pipeline preflight, and health check now verify those modules
before production starts