Harmon/insar-management-system-v2

Fork 0

Files

T

Harmon 4c0d1f2c2b feat: engineer SBAS timeseries production workflow

2026-04-29 14:43:31 +08:00

21 KiB

Raw Blame History

ISCE2 SBAS Time-Series Production Design

Updated: 2026-04-29

1. Goal

Add stack-based time-series production under the current ISCE2-oriented architecture.

The current repository already has:

image discovery and stack selection
PS/time-series stack batch persistence
single-pair D-InSAR production

The current repository does not yet have:

a stack-level production entry
time-series workflow orchestration
SBAS inversion output registration
PS-InSAR result/product pages beyond placeholders

This design focuses on the first deliverable:

implement SBAS time-series production first
keep true PS-InSAR or StaMPS as a later phase

2. Current State

2.1 What already exists

backend/app/routers/pairing.py
- /find-ps-timeseries can search image stacks for time-series use.
backend/app/routers/task_batches.py
- /task-batches/ps persists a selected stack into PsTaskBatchORM and PsTaskItemORM.
backend/app/models/orm.py
- ps_task_batches and ps_task_items already represent the planning-layer stack snapshot.
backend/app/models/orm.py
- workflow_runs, workflow_steps, and workflow_artifacts already exist and are suitable for multi-step orchestration.
backend/app/models/orm.py
- result_products already supports multiple catalogs through catalog_name.
frontend/src/App.jsx
- ps_production, ps_products, psinsar_results, and psinsar_analysis are reserved placeholders.

2.2 What is missing

backend/app/dinsar_engines/isce2_engine.py is pair-oriented.
- It runs a custom LT-1 stripmapApp.py flow for one pair or a pair-root directory.
- It does not model stack/network/time-series execution.
Current SBAS in pairing-related code and docs means a pairing strategy.
- It does not mean completed SBAS inversion, velocity estimation, or time-series products.
The current PS batch is not a production run.
- It is only a stored stack selection.

2.3 Architectural implication

Time-series production is not a small extension of the current pair-based D-InSAR engine.

The processing object changes from:

pair -> one run -> one main displacement output

to:

stack/network -> multi-step run -> multiple intermediate and final products

Because of that, time-series production should not be forced into DinsarEngine as-is.

Still, the LT-1 input-preparation layer should be shared.

Recommended reuse point:

keep DEM path resolution, orbit-pool resolution, and LT-1 precise-orbit XML generation in a shared helper
current implementation anchor:
- backend/app/isce2_pipeline/lt1_input_resolver.py

2.4 Experiment Status

Current LT-1 stack experiment status already de-risks the processing side of phase 1:

one offline LT-1 sample stack has completed the generated run_01 to run_07 chain under Ubuntu-24.04
the same LT-1 sample stack has also completed run_08_igram, producing filtered and unwrapped pair products under Igrams/
the same LT-1 sample stack has now also completed the first MintPy SBAS smoke test through radar-coordinate timeseries.h5 and velocity.h5
current successful MintPy work directory:
- /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/mintpy_sbas_v5
the same LT-1 sample stack has now also completed an experiment-layer geocode plus publish-bundle export
current successful publish-style directory:
- /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/publish/mintpy_sbas_v5
local SAR scenes, local orbit data, and a local DEM were sufficient for this stage
Earthdata SWBD was not treated as a hard dependency during experiments because run_01_reference can recover with a synthetic all-land waterMask
current runtime split decision:
- keep ISCE2 stack processing in the existing WSL isce2 env
- validate MintPy in a separate WSL mintpy env to avoid mutating the working processing env on the development machine
phase-4 unified-env update:
- a recreated WSL env isce2_mintpy_v1 has now also completed the same LT-1 SBAS smoke test and publish export without the isce bridge
- current successful unified SBAS work directory:
  - /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/mintpy_sbas_unified_v1
- current successful unified publish directory:
  - /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/publish/mintpy_sbas_unified_v1
- current judgment:
  - unified env is viable at experiment layer and is now the preferred SBAS experiment runtime
  - existing WSL isce2 should remain untouched for current D-InSAR production safety
  - bridge env remains the fallback baseline until a fuller comparison is written
current MintPy bridge decision:
- keep mintpy isolated
- bridge only the top-level isce package from the isce2 env into the mintpy env, because MintPy's ISCE stripmap metadata path imports isce
- do not bridge the full isce2 site-packages, because that polluted the MintPy runtime with conflicting h5py
current MintPy runtime workaround decision:
- use a strict maskAllValid.h5 before inversion to suppress unstable partial-network pixels
- use a repo-local patched launcher for the current MintPy 1.6.2 single-pixel partial-network inversion bug
- current implementation anchors:
  - experiments/isce2_sbas_timeseries/scripts/create_mintpy_all_ifgram_mask.py
  - experiments/isce2_sbas_timeseries/scripts/run_smallbaselineApp_patched.py
  - experiments/isce2_sbas_timeseries/scripts/run_mintpy_sbas_smoketest_ubuntu2404.sh
current experiment-layer publish export decision:
- keep geocode/export as a separate post-MintPy stage
- current implementation anchors:
  - experiments/isce2_sbas_timeseries/scripts/export_mintpy_publish_products_ubuntu2404.sh
  - experiments/isce2_sbas_timeseries/scripts/build_mintpy_publish_bundle.py
the shared helper refactor is already in place:
- backend/app/isce2_pipeline/lt1_input_resolver.py
- compatibility rule:
  - the original pair-oriented D-InSAR public entry was not removed
  - backend/app/isce2_pipeline/run_lt1_dinsar_pipeline.py still owns the existing workflow entry and now delegates shared input preparation to the helper

2.5 Stack Planning Thresholds

The planning entry that historically used the old PS preparation label is now treated as time-series stack preparation. The internal API path remains /find-ps-timeseries for compatibility.

Current threshold meaning:

initial_overlap_threshold
- single-scene AOI coverage gate
- formula: area(scene footprint ∩ AOI) / area(AOI)
- default: 0.30
- purpose: remove scenes that barely intersect the study area
final_overlap_threshold
- final stack footprint consistency gate
- formula: area(common footprint of selected stack ∩ AOI) / min(area(each selected scene footprint ∩ AOI))
- default: 0.95
- purpose: ensure the retained stack has a stable common processing area without requiring each strip to cover the whole AOI

Planning algorithm rule:

first group candidates by orbit direction, satellite family, imaging mode, and polarization
LT1A and LT1B are treated as the same LT1 satellite family for stack planning
then search each compatible group for the largest stack whose footprint consistency satisfies final_overlap_threshold
when one or more outlier scenes break the common area, the planner may drop scenes until a valid stack is found
if no all-scene common-overlap stack exists, the planner may return a connected pairwise SBAS network when each retained network edge satisfies final_overlap_threshold
at least 3 scenes are required before a stack can be persisted as a TimeseriesStackPlan

3. Recommendation

3.1 Deliver SBAS first

Recommended first path:

stack planning -> stack prep -> ISCE2 stack processing -> MintPy smallbaselineApp -> psinsar product publish -> result browsing

Reasons:

the repository already has stack-selection concepts
SBAS outputs are easier to standardize into cataloged products
MintPy is a practical downstream tool for small-baseline inversion over ISCE outputs
the UI and backend risk are both lower than starting from full PS-InSAR

3.2 Do not start with StaMPS / true PS route

True PS-InSAR should be treated as phase 2 or later because it adds extra uncertainty in:

candidate selection and amplitude stability logic
stack export conventions
result structure standardization
point-level visualization and analysis UX

4. Recommended Technical Chain

Phase-1 recommended chain:

User creates or reuses a PS stack batch.
System creates a stack-level production run bound to that batch.
Workflow prepares a working directory and stack metadata.
Workflow runs ISCE2 stack processing in WSL.
Workflow runs MintPy SBAS inversion in a dedicated WSL MintPy runtime.
Workflow exports standardized products and manifest files.
Workflow publishes the outputs into a psinsar result catalog.
Frontend reads the catalog and workflow state for products and results.

Product interpretation rule:

manifest.json is the publish entrypoint
assets/geo_timeseries.h5 is the canonical time-series data product
assets/velocity.tif is the default browse layer, not the only final deliverable

5. Scope

5.1 In scope for phase 1

SBAS time-series production entry
stack-level workflow orchestration
WSL runtime checks for stack and MintPy steps
product extraction and catalog registration
basic ps_production and ps_products pages
basic psinsar_results browsing capability

5.2 Out of scope for phase 1

full PS-InSAR / StaMPS route
advanced point-based analysis UI
cluster scheduling
multi-sensor stack fusion
automatic reference-point intelligence

6. Target Architecture

6.1 Domain split

Keep the system split into three layers:

Planning layer
- existing find-ps-timeseries
- existing ps_task_batches
Production layer
- new stack-level time-series run submission and monitoring
Result layer
- psinsar products and business-facing browsing

This keeps planning data and production runs decoupled.

6.2 Orchestration model

Reuse the existing generic workflow subsystem:

workflow_runs
workflow_steps
workflow_artifacts

Recommended rule:

do not extend DinsarEngine for time-series phase 1
use workflow as the orchestration primitive
add a thin time-series metadata table that binds:
- one ps_task_batch
- one workflow run
- one processing mode
- one output root

6.3 Suggested new metadata table

Add a new business table such as ps_timeseries_runs.

Suggested fields:

run_id
batch_id
workflow_run_id
mode
- sbas
engine_code
- isce2
processor_code
- isce2_stack_mintpy
direction
status
work_dir
output_dir
publish_dir
reference_date
reference_strategy
params_json
summary_json
error_message
created_by
created_at
updated_at
started_at
ended_at

Notes:

PsTaskBatchORM remains the planning snapshot.
WorkflowRunORM remains the step-level orchestration record.
ps_timeseries_runs becomes the business-facing production record.

6.4 Suggested workflow

Recommended workflow name:

isce2_sbas_mintpy_v1

Recommended steps:

prepare_stack_input
- validate stack size and metadata
- build stack working directory
- resolve DEM/orbit/output paths
run_isce2_stack
- run stack pre-processing in WSL
- collect stack intermediate artifacts
run_mintpy_sbas
- run repo-controlled MintPy smallbaselineApp
- generate strict inversion mask if required by the pinned MintPy runtime
- generate time-series and velocity products
export_standard_products
- copy or convert outputs into stable publish structure
- generate manifest and preview assets
publish_psinsar_products
- register catalog entries
- update product status and coverage

6.5 Suggested job types

7. Backend Design

7.1 New modules

Recommended backend additions:

backend/app/routers/timeseries_production.py
backend/app/routers/ps_products.py
backend/app/services/timeseries_service.py
backend/app/services/timeseries_workflow_factory.py
backend/app/services/isce2_stack_service.py
backend/app/services/mintpy_service.py
backend/app/services/psinsar_catalog_service.py
backend/app/services/timeseries_paths.py

7.2 API sketch

7.3 Reuse of current tables and services

Recommended reuse:

ps_task_batches / ps_task_items
- keep as the source stack definition
workflow_runs / workflow_steps / workflow_artifacts
- keep as the run and step state model
result_products
- extend with catalog_name = psinsar
current job queue / worker / task log chain
- reuse for execution and monitoring

7.4 Product catalog strategy

Do not build a separate product table just for time-series phase 1.

Use result_products with:

catalog_name = psinsar
engine_code = isce2
product_type values specific to time-series outputs

Recommended product_type values:

velocity_map
timeseries_cube
temporal_coherence
ifgram_network
point_series_csv
preview_png
summary_report

Use the existing generic asset and issue models where possible.

7.5 Current Phase-1 Artifact Contract

The current validated LT-1 experiment supports the following publish contract:

timeseries_cube
- source file:
  - assets/geo_timeseries.h5
velocity_map
- source file:
  - assets/geo_velocity.h5
velocity_geotiff
- source file:
  - assets/velocity.tif
temporal_coherence
- source file:
  - assets/geo_temporalCoherence.h5
quality_mask
- source file:
  - assets/geo_maskTempCoh.h5
temporal_coherence_geotiff
- source file:
  - assets/temporalCoherence.tif
quality_mask_geotiff
- source file:
  - assets/maskTempCoh.tif
ifgram_network
- source file:
  - runtime diagnostic source:
    - numTriNonzeroIntAmbiguity.h5
preview_png
- source file:
  - preview/velocity_preview.png
diagnostic_png
- source file:
  - preview/numTriNonzeroIntAmbiguity.png
retained runtime-only artifacts:
- maskAllValid.h5
- avgSpatialCoh.h5
- smallbaselineApp.cfg

Current sample manifest draft:

experiments/isce2_sbas_timeseries/configs/sample_psinsar_manifest_lt1_e123p3_n46p1.json

Current successful experiment publish bundle:

experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/publish/mintpy_sbas_v5/manifest.json

8. Directory Conventions

Recommended Windows roots:

PS_TIMESERIES_WORK_ROOT
PS_TIMESERIES_OUTPUT_ROOT
PSINSAR_PRODUCT_DIR

Recommended run layout:

<PS_TIMESERIES_WORK_ROOT>/<batch_id>/<run_id>/
  stack_input/
  isce2_stack/
  mintpy/
  export/
  logs/

Recommended publish layout:

<PSINSAR_PRODUCT_DIR>/<product_id>/
  manifest.json
  preview/
  assets/
  metadata/

Rules:

production should always have a stable publish directory
intermediate work directory and final publish directory should stay separate
raw imagery must not be copied into publish output unnecessarily

9. Frontend Design

9.1 `ps_production`

Phase-1 page content:

select an existing PS stack batch
choose processing mode
- phase 1 only exposes SBAS
show runtime environment status
- WSL
- ISCE2 stack runtime
- MintPy runtime
submit workflow run
show step status, logs, and artifacts

9.2 `ps_products`

Phase-1 page content:

list psinsar products
filter by status, run, batch, direction, date
publish and rebuild operations
basic manifest view

9.3 `psinsar_results`

Phase-1 page content:

basic product query
map preview of velocity or deformation raster
metadata drawer
quick link to workflow run and source batch

9.4 `psinsar_analysis`

Keep this reserved in phase 1.

Later it can take:

point time-series browsing
rate classification
hotspot statistics
thematic reporting

10. Configuration

11. Key Risk: LT-1 Stack Compatibility

This is the main technical uncertainty.

The current repository already uses a custom LT-1 single-pair stripmapApp.py flow. That does not prove official ISCE2 stack tooling will work directly for LT-1/LUTAN1.

Therefore phase 0 must verify:

whether official ISCE2 stack tooling can ingest LT-1 metadata directly
whether orbit formatting is already sufficient for stack mode
whether extra stack-input conversion is required

If official stack tooling is not directly compatible:

keep the overall architecture unchanged
implement a custom stack builder or adapter layer before MintPy

Do not hard-code the design around official stack scripts until this experiment is confirmed.

12. Option Comparison

Option	Fit for current repo	Technical risk	UI/ops complexity	Phase-1 recommendation
ISCE2 stack + MintPy SBAS	High	Medium	Medium	Yes
ISCE2 StackToStaMPS / PS route	Medium-Low	High	High	No

Why SBAS wins first:

closer to the current stack-planning model
easier to standardize outputs
easier to explain and operate
lower uncertainty in first delivery

13. TODO List

Phase 0. Validation

Verify whether LT-1/LUTAN1 can run through official ISCE2 stack tooling.
Decide whether MintPy shares the current ISCE2 environment or uses a separate env.
Finalize work/output/publish directory rules.
Decide the minimal product set for first release.

Phase 1. Backend scaffold

Add ps_timeseries_runs ORM and schema.
Add timeseries_production router.
Add timeseries_workflow_factory for isce2_sbas_mintpy_v1.
Add new job handlers for stack prep, stack run, MintPy run, export, and publish.
Bind workflow runs to ps_task_batches.
Persist workflow artifacts for major intermediate and final outputs.

Phase 2. Catalog and products

Extend catalog publishing to support catalog_name = psinsar.
Define manifest schema for SBAS products.
Add coverage extraction and preview generation.
Add rebuild and health-check operations for psinsar products.

Phase 3. Frontend

Replace the ps_production placeholder with a real production panel.
Replace the ps_products placeholder with a product management panel.
Add a basic psinsar_results result page.
Keep psinsar_analysis reserved until point-series UX is clear.

Phase 4. Validation and rollout

Run one small AOI stack end to end in the experimental workflow.
Verify rerun, resume, and failure-recovery behavior.
Verify published products can be queried and rendered.
Add operator documentation and troubleshooting notes.

14. Experimental Environment Recommendation

Yes. A dedicated experiment folder is recommended.

Purpose:

validate LT-1 stack compatibility without polluting production code
collect command templates, notes, and sample manifests
separate exploratory scripts from backend services

Recommended location:

experiments/isce2_sbas_timeseries/

Rules:

store only scripts, notes, configs, and tiny mock artifacts in git
do not commit raw SAR scenes, DEM rasters, or large intermediate outputs
once an experiment stabilizes, move the conclusion back into docs/ and production code

21 KiB Raw Blame History

ISCE2 SBAS Time-Series Production Design

1. Goal

2. Current State

2.1 What already exists

2.2 What is missing

2.3 Architectural implication

2.4 Experiment Status

2.5 Stack Planning Thresholds

3. Recommendation

3.1 Deliver SBAS first

3.2 Do not start with StaMPS / true PS route

4. Recommended Technical Chain

5. Scope

5.1 In scope for phase 1

5.2 Out of scope for phase 1

6. Target Architecture

6.1 Domain split

6.2 Orchestration model

6.3 Suggested new metadata table

6.4 Suggested workflow

6.5 Suggested job types

7. Backend Design

7.1 New modules

7.2 API sketch

7.3 Reuse of current tables and services

7.4 Product catalog strategy

7.5 Current Phase-1 Artifact Contract

8. Directory Conventions

9. Frontend Design

9.1 ps_production

9.2 ps_products

9.3 psinsar_results

9.4 psinsar_analysis

10. Configuration

11. Key Risk: LT-1 Stack Compatibility

12. Option Comparison

13. TODO List

Phase 0. Validation

Phase 1. Backend scaffold

Phase 2. Catalog and products

Phase 3. Frontend

Phase 4. Validation and rollout

14. Experimental Environment Recommendation

21 KiB

Raw Blame History

9.1 `ps_production`

9.2 `ps_products`

9.3 `psinsar_results`

9.4 `psinsar_analysis`