Harmon/insar-management-system-v2
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ecc72ec9cde6284c5e6eab39e60a979e4c03123a
insar-management-system-v2/docs/ISCE2_SBAS_TIMESERIES_DESIGN.md
T

19 KiB
Raw Blame History

ISCE2 SBAS Time-Series Production Design

Updated: 2026-04-06

1. Goal

Add stack-based time-series production under the current ISCE2-oriented architecture.

The current repository already has:

  • image discovery and stack selection
  • PS/time-series stack batch persistence
  • single-pair D-InSAR production

The current repository does not yet have:

  • a stack-level production entry
  • time-series workflow orchestration
  • SBAS inversion output registration
  • PS-InSAR result/product pages beyond placeholders

This design focuses on the first deliverable:

  • implement SBAS time-series production first
  • keep true PS-InSAR or StaMPS as a later phase

Related product contract:

  • docs/ISCE2_SBAS_PRODUCT_SPEC.md

2. Current State

2.1 What already exists

  • backend/app/routers/pairing.py
    • /find-ps-timeseries can search image stacks for time-series use.
  • backend/app/routers/task_batches.py
    • /task-batches/ps persists a selected stack into PsTaskBatchORM and PsTaskItemORM.
  • backend/app/models/orm.py
    • ps_task_batches and ps_task_items already represent the planning-layer stack snapshot.
  • backend/app/models/orm.py
    • workflow_runs, workflow_steps, and workflow_artifacts already exist and are suitable for multi-step orchestration.
  • backend/app/models/orm.py
    • result_products already supports multiple catalogs through catalog_name.
  • frontend/src/App.jsx
    • ps_production, ps_products, psinsar_results, and psinsar_analysis are reserved placeholders.

2.2 What is missing

  • backend/app/dinsar_engines/isce2_engine.py is pair-oriented.
    • It runs a custom LT-1 stripmapApp.py flow for one pair or a pair-root directory.
    • It does not model stack/network/time-series execution.
  • Current SBAS in pairing-related code and docs means a pairing strategy.
    • It does not mean completed SBAS inversion, velocity estimation, or time-series products.
  • The current PS batch is not a production run.
    • It is only a stored stack selection.

2.3 Architectural implication

Time-series production is not a small extension of the current pair-based D-InSAR engine.

The processing object changes from:

  • pair -> one run -> one main displacement output

to:

  • stack/network -> multi-step run -> multiple intermediate and final products

Because of that, time-series production should not be forced into DinsarEngine as-is.

Still, the LT-1 input-preparation layer should be shared.

Recommended reuse point:

  • keep DEM path resolution, orbit-pool resolution, and LT-1 precise-orbit XML generation in a shared helper
  • current implementation anchor:
    • backend/app/isce2_pipeline/lt1_input_resolver.py

2.4 Experiment Status

Current LT-1 stack experiment status already de-risks the processing side of phase 1:

  • one offline LT-1 sample stack has completed the generated run_01 to run_07 chain under Ubuntu-24.04
  • the same LT-1 sample stack has also completed run_08_igram, producing filtered and unwrapped pair products under Igrams/
  • the same LT-1 sample stack has now also completed the first MintPy SBAS smoke test through radar-coordinate timeseries.h5 and velocity.h5
  • current successful MintPy work directory:
    • /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/mintpy_sbas_v5
  • the same LT-1 sample stack has now also completed an experiment-layer geocode plus publish-bundle export
  • current successful publish-style directory:
    • /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/publish/mintpy_sbas_v5
  • local SAR scenes, local orbit data, and a local DEM were sufficient for this stage
  • Earthdata SWBD was not treated as a hard dependency during experiments because run_01_reference can recover with a synthetic all-land waterMask
  • current runtime split decision:
    • keep ISCE2 stack processing in the existing WSL isce2 env
    • validate MintPy in a separate WSL mintpy env to avoid mutating the working processing env on the development machine
  • phase-4 unified-env update:
    • a recreated WSL env isce2_mintpy_v1 has now also completed the same LT-1 SBAS smoke test and publish export without the isce bridge
    • current successful unified SBAS work directory:
      • /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/mintpy_sbas_unified_v1
    • current successful unified publish directory:
      • /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/publish/mintpy_sbas_unified_v1
    • current judgment:
      • unified env is viable at experiment layer and is now the preferred SBAS experiment runtime
      • existing WSL isce2 should remain untouched for current D-InSAR production safety
      • bridge env remains the fallback baseline until a fuller comparison is written
  • current MintPy bridge decision:
    • keep mintpy isolated
    • bridge only the top-level isce package from the isce2 env into the mintpy env, because MintPy's ISCE stripmap metadata path imports isce
    • do not bridge the full isce2 site-packages, because that polluted the MintPy runtime with conflicting h5py
  • current MintPy runtime workaround decision:
    • use a strict maskAllValid.h5 before inversion to suppress unstable partial-network pixels
    • use a repo-local patched launcher for the current MintPy 1.6.2 single-pixel partial-network inversion bug
    • current implementation anchors:
      • experiments/isce2_sbas_timeseries/scripts/create_mintpy_all_ifgram_mask.py
      • experiments/isce2_sbas_timeseries/scripts/run_smallbaselineApp_patched.py
      • experiments/isce2_sbas_timeseries/scripts/run_mintpy_sbas_smoketest_ubuntu2404.sh
  • current experiment-layer publish export decision:
    • keep geocode/export as a separate post-MintPy stage
    • current implementation anchors:
      • experiments/isce2_sbas_timeseries/scripts/export_mintpy_publish_products_ubuntu2404.sh
      • experiments/isce2_sbas_timeseries/scripts/build_mintpy_publish_bundle.py
  • the shared helper refactor is already in place:
    • backend/app/isce2_pipeline/lt1_input_resolver.py
    • compatibility rule:
      • the original pair-oriented D-InSAR public entry was not removed
      • backend/app/isce2_pipeline/run_lt1_dinsar_pipeline.py still owns the existing workflow entry and now delegates shared input preparation to the helper

3. Recommendation

3.1 Deliver SBAS first

Recommended first path:

  • stack planning -> stack prep -> ISCE2 stack processing -> MintPy smallbaselineApp -> psinsar product publish -> result browsing

Reasons:

  • the repository already has stack-selection concepts
  • SBAS outputs are easier to standardize into cataloged products
  • MintPy is a practical downstream tool for small-baseline inversion over ISCE outputs
  • the UI and backend risk are both lower than starting from full PS-InSAR

3.2 Do not start with StaMPS / true PS route

True PS-InSAR should be treated as phase 2 or later because it adds extra uncertainty in:

  • candidate selection and amplitude stability logic
  • stack export conventions
  • result structure standardization
  • point-level visualization and analysis UX

4. Recommended Technical Chain

Phase-1 recommended chain:

  1. User creates or reuses a PS stack batch.
  2. System creates a stack-level production run bound to that batch.
  3. Workflow prepares a working directory and stack metadata.
  4. Workflow runs ISCE2 stack processing in WSL.
  5. Workflow runs MintPy SBAS inversion in a dedicated WSL MintPy runtime.
  6. Workflow exports standardized products and manifest files.
  7. Workflow publishes the outputs into a psinsar result catalog.
  8. Frontend reads the catalog and workflow state for products and results.

Product interpretation rule:

  • manifest.json is the publish entrypoint
  • assets/geo_timeseries.h5 is the canonical time-series data product
  • assets/velocity.tif is the default browse layer, not the only final deliverable

5. Scope

5.1 In scope for phase 1

  • SBAS time-series production entry
  • stack-level workflow orchestration
  • WSL runtime checks for stack and MintPy steps
  • product extraction and catalog registration
  • basic ps_production and ps_products pages
  • basic psinsar_results browsing capability

5.2 Out of scope for phase 1

  • full PS-InSAR / StaMPS route
  • advanced point-based analysis UI
  • cluster scheduling
  • multi-sensor stack fusion
  • automatic reference-point intelligence

6. Target Architecture

6.1 Domain split

Keep the system split into three layers:

  • Planning layer
    • existing find-ps-timeseries
    • existing ps_task_batches
  • Production layer
    • new stack-level time-series run submission and monitoring
  • Result layer
    • psinsar products and business-facing browsing

This keeps planning data and production runs decoupled.

6.2 Orchestration model

Reuse the existing generic workflow subsystem:

  • workflow_runs
  • workflow_steps
  • workflow_artifacts

Recommended rule:

  • do not extend DinsarEngine for time-series phase 1
  • use workflow as the orchestration primitive
  • add a thin time-series metadata table that binds:
    • one ps_task_batch
    • one workflow run
    • one processing mode
    • one output root

6.3 Suggested new metadata table

Add a new business table such as ps_timeseries_runs.

Suggested fields:

  • run_id
  • batch_id
  • workflow_run_id
  • mode
    • sbas
  • engine_code
    • isce2
  • processor_code
    • isce2_stack_mintpy
  • direction
  • status
  • work_dir
  • output_dir
  • publish_dir
  • reference_date
  • reference_strategy
  • params_json
  • summary_json
  • error_message
  • created_by
  • created_at
  • updated_at
  • started_at
  • ended_at

Notes:

  • PsTaskBatchORM remains the planning snapshot.
  • WorkflowRunORM remains the step-level orchestration record.
  • ps_timeseries_runs becomes the business-facing production record.

6.4 Suggested workflow

Recommended workflow name:

  • isce2_sbas_mintpy_v1

Recommended steps:

  1. prepare_stack_input
    • validate stack size and metadata
    • build stack working directory
    • resolve DEM/orbit/output paths
  2. run_isce2_stack
    • run stack pre-processing in WSL
    • collect stack intermediate artifacts
  3. run_mintpy_sbas
    • run repo-controlled MintPy smallbaselineApp
    • generate strict inversion mask if required by the pinned MintPy runtime
    • generate time-series and velocity products
  4. export_standard_products
    • copy or convert outputs into stable publish structure
    • generate manifest and preview assets
  5. publish_psinsar_products
    • register catalog entries
    • update product status and coverage

6.5 Suggested job types

Recommended new job types:

  • TIMESERIES_PREP_STACK
  • ISCE2_STACK_RUN
  • MINTPY_SBAS_RUN
  • EXPORT_PSINSAR_PRODUCTS
  • PUBLISH_PSINSAR_PRODUCTS

These fit naturally under the existing job queue and worker model.

7. Backend Design

7.1 New modules

Recommended backend additions:

  • backend/app/routers/timeseries_production.py
  • backend/app/routers/ps_products.py
  • backend/app/services/timeseries_service.py
  • backend/app/services/timeseries_workflow_factory.py
  • backend/app/services/isce2_stack_service.py
  • backend/app/services/mintpy_service.py
  • backend/app/services/psinsar_catalog_service.py
  • backend/app/services/timeseries_paths.py

7.2 API sketch

Recommended new APIs:

  • POST /timeseries-production/runs
    • submit one SBAS run for one stored PS stack batch
  • GET /timeseries-production/runs
    • list recent time-series runs
  • GET /timeseries-production/runs/{run_id}
    • get business metadata + workflow state + artifacts
  • POST /timeseries-production/runs/{run_id}/retry-step
    • retry failed step when allowed
  • POST /timeseries-production/wsl-check
    • check stack + MintPy runtime environment
  • GET /ps-products
    • list registered psinsar products
  • POST /ps-products/rebuild-catalog
    • rebuild psinsar product catalog

7.3 Reuse of current tables and services

Recommended reuse:

  • ps_task_batches / ps_task_items
    • keep as the source stack definition
  • workflow_runs / workflow_steps / workflow_artifacts
    • keep as the run and step state model
  • result_products
    • extend with catalog_name = psinsar
  • current job queue / worker / task log chain
    • reuse for execution and monitoring

7.4 Product catalog strategy

Do not build a separate product table just for time-series phase 1.

Use result_products with:

  • catalog_name = psinsar
  • engine_code = isce2
  • product_type values specific to time-series outputs

Recommended product_type values:

  • velocity_map
  • timeseries_cube
  • temporal_coherence
  • ifgram_network
  • point_series_csv
  • preview_png
  • summary_report

Use the existing generic asset and issue models where possible.

7.5 Current Phase-1 Artifact Contract

The current validated LT-1 experiment supports the following publish contract:

  • timeseries_cube
    • source file:
      • assets/geo_timeseries.h5
  • velocity_map
    • source file:
      • assets/geo_velocity.h5
  • velocity_geotiff
    • source file:
      • assets/velocity.tif
  • temporal_coherence
    • source file:
      • assets/geo_temporalCoherence.h5
  • quality_mask
    • source file:
      • assets/geo_maskTempCoh.h5
  • temporal_coherence_geotiff
    • source file:
      • assets/temporalCoherence.tif
  • quality_mask_geotiff
    • source file:
      • assets/maskTempCoh.tif
  • ifgram_network
    • source file:
      • runtime diagnostic source:
        • numTriNonzeroIntAmbiguity.h5
  • preview_png
    • source file:
      • preview/velocity_preview.png
  • diagnostic_png
    • source file:
      • preview/numTriNonzeroIntAmbiguity.png
  • retained runtime-only artifacts:
    • maskAllValid.h5
    • avgSpatialCoh.h5
    • smallbaselineApp.cfg

Current sample manifest draft:

  • experiments/isce2_sbas_timeseries/configs/sample_psinsar_manifest_lt1_e123p3_n46p1.json

Current successful experiment publish bundle:

  • experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/publish/mintpy_sbas_v5/manifest.json

8. Directory Conventions

Recommended Windows roots:

  • PS_TIMESERIES_WORK_ROOT
  • PS_TIMESERIES_OUTPUT_ROOT
  • PSINSAR_PRODUCT_DIR

Recommended run layout:

<PS_TIMESERIES_WORK_ROOT>/<batch_id>/<run_id>/
  stack_input/
  isce2_stack/
  mintpy/
  export/
  logs/

Recommended publish layout:

<PSINSAR_PRODUCT_DIR>/<product_id>/
  manifest.json
  preview/
  assets/
  metadata/

Rules:

  • production should always have a stable publish directory
  • intermediate work directory and final publish directory should stay separate
  • raw imagery must not be copied into publish output unnecessarily

9. Frontend Design

9.1 ps_production

Phase-1 page content:

  • select an existing PS stack batch
  • choose processing mode
    • phase 1 only exposes SBAS
  • show runtime environment status
    • WSL
    • ISCE2 stack runtime
    • MintPy runtime
  • submit workflow run
  • show step status, logs, and artifacts

9.2 ps_products

Phase-1 page content:

  • list psinsar products
  • filter by status, run, batch, direction, date
  • publish and rebuild operations
  • basic manifest view

9.3 psinsar_results

Phase-1 page content:

  • basic product query
  • map preview of velocity or deformation raster
  • metadata drawer
  • quick link to workflow run and source batch

9.4 psinsar_analysis

Keep this reserved in phase 1.

Later it can take:

  • point time-series browsing
  • rate classification
  • hotspot statistics
  • thematic reporting

10. Configuration

Recommended new or clarified settings:

  • PS_TIMESERIES_ENABLED
  • PS_TIMESERIES_WORK_ROOT
  • PS_TIMESERIES_OUTPUT_ROOT
  • PSINSAR_PRODUCT_DIR
  • MINTPY_PYTHON
  • MINTPY_SMALLBASELINE_APP
  • MINTPY_RUNNER
  • MINTPY_PATCHED_SMALLBASELINE_APP
  • MINTPY_TEMPLATE_DIR
  • PS_TIMESERIES_AUTO_PUBLISH

Where practical:

  • reuse ISCE2_WSL_DISTRO
  • reuse ISCE2_PYTHON if MintPy is installed in the same environment
  • reuse current DEM and orbit path conventions

11. Key Risk: LT-1 Stack Compatibility

This is the main technical uncertainty.

The current repository already uses a custom LT-1 single-pair stripmapApp.py flow. That does not prove official ISCE2 stack tooling will work directly for LT-1/LUTAN1.

Therefore phase 0 must verify:

  • whether official ISCE2 stack tooling can ingest LT-1 metadata directly
  • whether orbit formatting is already sufficient for stack mode
  • whether extra stack-input conversion is required

If official stack tooling is not directly compatible:

  • keep the overall architecture unchanged
  • implement a custom stack builder or adapter layer before MintPy

Do not hard-code the design around official stack scripts until this experiment is confirmed.

12. Option Comparison

Option Fit for current repo Technical risk UI/ops complexity Phase-1 recommendation
ISCE2 stack + MintPy SBAS High Medium Medium Yes
ISCE2 StackToStaMPS / PS route Medium-Low High High No

Why SBAS wins first:

  • closer to the current stack-planning model
  • easier to standardize outputs
  • easier to explain and operate
  • lower uncertainty in first delivery

13. TODO List

Phase 0. Validation

  • Verify whether LT-1/LUTAN1 can run through official ISCE2 stack tooling.
  • Decide whether MintPy shares the current ISCE2 environment or uses a separate env.
  • Finalize work/output/publish directory rules.
  • Decide the minimal product set for first release.

Phase 1. Backend scaffold

  • Add ps_timeseries_runs ORM and schema.
  • Add timeseries_production router.
  • Add timeseries_workflow_factory for isce2_sbas_mintpy_v1.
  • Add new job handlers for stack prep, stack run, MintPy run, export, and publish.
  • Bind workflow runs to ps_task_batches.
  • Persist workflow artifacts for major intermediate and final outputs.

Phase 2. Catalog and products

  • Extend catalog publishing to support catalog_name = psinsar.
  • Define manifest schema for SBAS products.
  • Add coverage extraction and preview generation.
  • Add rebuild and health-check operations for psinsar products.

Phase 3. Frontend

  • Replace the ps_production placeholder with a real production panel.
  • Replace the ps_products placeholder with a product management panel.
  • Add a basic psinsar_results result page.
  • Keep psinsar_analysis reserved until point-series UX is clear.

Phase 4. Validation and rollout

  • Run one small AOI stack end to end in the experimental workflow.
  • Verify rerun, resume, and failure-recovery behavior.
  • Verify published products can be queried and rendered.
  • Add operator documentation and troubleshooting notes.

14. Experimental Environment Recommendation

Yes. A dedicated experiment folder is recommended.

Purpose:

  • validate LT-1 stack compatibility without polluting production code
  • collect command templates, notes, and sample manifests
  • separate exploratory scripts from backend services

Recommended location:

  • experiments/isce2_sbas_timeseries/

Rules:

  • store only scripts, notes, configs, and tiny mock artifacts in git
  • do not commit raw SAR scenes, DEM rasters, or large intermediate outputs
  • once an experiment stabilizes, move the conclusion back into docs/ and production code
Reference in New Issue View Git Blame Copy Permalink