insar-management-system-v2/docs/ISCE2_SBAS_ENGINEERING_DESIGN_20260428.md

ISCE2 + MintPy SBAS Engineering Design

Updated: 2026-04-28

1. Purpose

This document defines the engineering expansion plan for the current stack-based time-series InSAR route:

• LT-1 stack batch -> ISCE2 stripmapStack -> MintPy SBAS -> publish bundle -> psinsar catalog

The repository already has a working phase-1 skeleton. The goal of this document is not to restart the design from zero, but to align the next implementation round with the code that already exists in:

• backend/app/services/timeseries_service.py
• backend/app/routers/timeseries_production.py
• backend/app/services/psinsar_catalog_service.py
• frontend/src/TimeseriesProductionPanel.jsx
• frontend/src/components/PsinsarCatalogPanel.jsx

This document supersedes the "missing pieces" parts of docs/ISCE2_SBAS_TIMESERIES_DESIGN.md for the current implementation phase.

2. Decisions

2.1 Primary processing route

Keep the current scientific split:

• ISCE2 is responsible for LT-1 stack preparation, stack geometry, co-registration, baseline generation, interferogram generation, and unwrap inputs.
• MintPy is responsible for SBAS inversion and time-series products.
• The system registers only publish-grade bundles, not raw MintPy work directories.

This means the production claim for the current phase is:

• SBAS time-series production on top of ISCE2 + MintPy

It is not:

• full PS-InSAR
• full StaMPS integration
• full commercial-grade atmospheric/error-correction stack

2.2 Keep the current business model

Use the current model already implemented in code:

• planning-layer stack snapshot:
  • PsTaskBatchORM
  • PsTaskItemORM
• business-facing production run:
  • PsTimeseriesRunORM
• step orchestration:
  • WorkflowRunORM
  • WorkflowStepORM
• publish/catalog registration:
  • ResultProductORM
  • ResultAssetORM
  • ResultIssueORM

Do not redesign the run model into a new engine abstraction in this round.

2.3 Keep naming stable for now

Current naming in the repository is mixed:

• product family shown to users: timeseries
• processing mode: sbas
• catalog namespace and package schema legacy: psinsar

For this round:

• keep product_family = timeseries
• keep mode = sbas
• keep catalog_name = psinsar for compatibility
• keep psinsar.publish.v1 ingestion support working

Do not do a DB/API namespace rename and a pipeline hardening round at the same time.

2.4 Current implementation scope (2026-04-28)

This round is intentionally constrained to avoid impact on other production business:

• no DB schema migration
• no catalog namespace rename
• no workflow framework refactor
• no change to the system self-maintenance / self-check contract

The implementation landed in this round focuses on pipeline hardening around the existing PsTimeseriesRunORM path:

• preflight gating before SBAS run creation
• stronger runtime self-check visibility
• publish-bundle validation before catalog registration
• frontend visibility for preflight, runtime checks, and publish validation

This means the current engineering target is:

• make the existing ISCE2 + MintPy SBAS route operationally safer

not:

• redesign the overall architecture
• replace the existing result registration model
• introduce a second persistence path for timeseries products

2.5 Phase-2 scope (2026-04-28)

The second implementation round keeps the same production chain but upgrades the planning trace from ad-hoc JSON to first-class additive schema objects.

Additive schema only:

• new planning tables:
  • TimeseriesStackPlanORM
  • TimeseriesStackPlanItemORM
• nullable trace columns on existing objects:
  • PsTaskBatchORM.plan_id
  • PsTaskBatchORM.plan_strategy
  • PsTaskItemORM.plan_item_ref_id
  • PsTimeseriesRunORM.plan_id
  • PsTimeseriesRunORM.plan_strategy

Operational rules for phase 2:

• do not introduce a separate migration framework
• rely on the existing database self-maintenance path:
  • Base.metadata.create_all()
  • missing-column auto-add in backend/app/db_maintenance.py
• keep phase-1 planning_context / remark compatibility for old batches

The engineering target of phase 2 is:

• formalize plan -> batch -> run -> publish bundle -> catalog product traceability
• expose plan_id in frontend production and result views
• keep old batches runnable without backfilling or hard migration
• expose GET /timeseries-plans/{plan_id} for plan audit/detail lookup

3. Current Baseline In Code

The current code already implements the core production skeleton.

3.1 Run record and workflow

backend/app/models/orm.py

• PsTimeseriesRunORM already stores:
  • run identity
  • batch binding
  • processor/runtime metadata
  • work and publish roots
  • input/orbit/quality summaries
  • failure state
• WorkflowRunORM and WorkflowStepORM already support DAG execution and retry.
• SystemJobORM already supports queued worker execution per workflow step.

3.2 Current workflow steps

backend/app/services/timeseries_service.py

Current step chain is already eight steps:

1. prepare
2. stack_prep_initial
3. materialize
4. stack_prep_refresh
5. run_isce2_stack
6. run_mintpy_sbas
7. export_publish_bundle
8. register_psinsar_product

This is already the correct backbone for the managed SBAS route.

3.3 Current scientific execution boundary

The scientific boundary is still script-based, and that is acceptable for now:

• experiments/isce2_sbas_timeseries/scripts/build_lt1_stack_prep.py
• experiments/isce2_sbas_timeseries/scripts/materialize_lt1_stack_scenes.py
• experiments/isce2_sbas_timeseries/scripts/prepare_lt1_stack_dem.py
• experiments/isce2_sbas_timeseries/scripts/run_generated_stack_runfile_ubuntu2404.sh
• experiments/isce2_sbas_timeseries/scripts/run_mintpy_sbas_unified_env_smoketest_ubuntu2404.sh
• experiments/isce2_sbas_timeseries/scripts/export_mintpy_publish_products_ubuntu2404.sh

The current implementation should continue to wrap these scripts instead of rewriting the scientific logic prematurely.

3.4 Current frontend and ops surface

Already present:

• run submission and run detail:
  • frontend/src/TimeseriesProductionPanel.jsx
• product catalog panel:
  • frontend/src/components/PsinsarCatalogPanel.jsx
• health-check visibility:
  • frontend/src/HealthCheckPanel.jsx
• catalog rebuild API:
  • backend/app/routers/ps_products.py

So the next round is a hardening and extension round, not an empty scaffold round.

4. Main Gaps

The next engineering work should focus on the following gaps.

4.1 Self-check is present but still shallow

Current runtime check already validates:

• WSL distro
• Python path
• stack script path
• configured helper scripts
• MintPy import
• DEM/orbit/output root presence

What is still missing:

• write permission checks for work and publish roots
• DEM sidecar consistency checks
• runtime dependency checks for scientific imports used by ISCE2/MintPy
• batch-level readiness checks before a run is queued
• publish-bundle structural validation before catalog registration

4.2 Quality summary exists, but quality gating is weak

Current code validates:

• stack prep readiness
• required run files
• required ISCE2 output directories
• required MintPy outputs
• publish manifest existence

But it still does not promote enough scientific quality indicators into release gates, for example:

• interferogram count versus expected network count
• non-empty unwrap/correlation outputs
• valid-pixel ratio after maskAllValid
• temporal coherence thresholds
• reference point presence and stability summary

4.3 Frontend is functional but still operationally thin

Current frontend can:

• submit a run
• run WSL check
• list runs
• show workflow steps
• retry failed workflow steps
• browse catalog entries

Still missing:

• structured preflight diagnostics for the selected batch
• clearer phase summaries per run
• direct visibility into quality summaries and key artifacts
• better linkage between run detail and published product detail
• a richer product detail view closer to the D-InSAR catalog panel depth

4.4 Result management needs stricter contract enforcement

The catalog path is correct, but the following rules should be made explicit and enforced:

• manifest.json is the only registration entrypoint
• every published run must have a stable publish_dir
• required assets must exist before registration
• missing assets should generate catalog issues and possibly quarantine status
• every product should carry:
  • processor code
  • runtime id
  • native output trace
  • stack identity

5. Target Pipeline

5.1 Input contract

The run input must remain stack-based, not pair-based.

Source objects:

• one ps_task_batch
• many ps_task_items
• one selected stack manifest:
  • input/selected_stack_manifest.json
• one generated stack manifest:
  • input/stack_input_manifest.json

The selected manifest is the planning snapshot.

The generated stack manifest is the execution snapshot and must include:

• stack dates
• reference date
• stack key
• group key
• resolved DEM and orbit dependencies
• readiness flags
• blocking reasons
• generated ISCE2 command arguments

5.2 Runtime directory model

Keep the current directory split:

• work root:
  • backend/runtime/timeseries_work/<run_id>/...
• publish root:
  • TIMESERIES_PRODUCT_DIR/<stack_key>/runs/<run_id>/...

Recommended internal layout under the work root:

• input/
• inputs/dem/
• stack_work/
• logs/
• mintpy/

Recommended publish layout:

• manifest.json
• assets/
• preview/
• metadata/

5.3 Managed workflow

The current eight-step chain is the correct managed workflow and should be kept:

1. prepare
   • validate batch
   • resolve stack identity
   • choose reference date
   • write selected_stack_manifest.json
2. stack_prep_initial
   • generate execution-layer stack manifest
   • resolve DEM/orbits
   • tell the system whether materialization is the only blocker
3. materialize
   • materialize LT-1 scenes into stack input layout
   • materialize orbit XML and local dependencies
4. stack_prep_refresh
   • re-run readiness check after materialization
   • must reach ready state
5. run_isce2_stack
   • prepare local DEM sidecars
   • generate run files
   • run run_01 to run_08
   • validate geom_reference, baselines, and Igrams
6. run_mintpy_sbas
   • write MintPy config
   • run controlled smallbaselineApp
   • validate core MintPy outputs
7. export_publish_bundle
   • geocode MintPy outputs
   • export GeoTIFF browse layers
   • generate preview and manifest
   • augment the manifest with canonical metadata
8. register_psinsar_product
   • register the publish bundle into catalog
   • mark the run as published

5.4 Current publish contract

Keep docs/ISCE2_SBAS_PRODUCT_SPEC.md as the publish contract source of truth.

Required publish assets for the managed SBAS route:

• assets/geo_timeseries.h5
• assets/geo_velocity.h5
• assets/velocity.tif
• assets/geo_temporalCoherence.h5
• assets/geo_maskTempCoh.h5
• preview/velocity_preview.png
• metadata/smallbaselineApp.cfg
• manifest.json

Optional but recommended:

• preview/numTriNonzeroIntAmbiguity.png
• extra quality JSON files

6. Self-Check Design

Self-check should exist at four levels.

6.1 Runtime preflight

Primary entry:

• POST /timeseries-production/wsl-check

Current checks should be kept and extended with:

• WSL distro reachable
• configured Python reachable
• ISCE2 stack script import/help check
• MintPy import check
• helper script existence checks
• DEM root existence
• orbit pool existence
• publish root existence
• work root existence
• write-test for work root
• write-test for publish root
• DEM sidecar consistency check
• optional import checks for:
  • cv2
  • scipy
  • astropy

Return structure should remain machine-readable so the frontend can render a diagnostic card instead of a plain message string.

6.2 Batch preflight

Add a run-specific preflight before or during create_run.

Minimum checks:

• scene count meets SBAS minimum
• all scene dates are valid
• scene dates are unique
• direction is consistent
• source files exist and are readable
• orbit coverage is complete or explicitly degraded
• group_key and stack_key are derivable
• publish path does not collide with another active run

Recommended surface:

• a new backend helper in timeseries_service.py
• frontend summary block in TimeseriesProductionPanel.jsx

6.3 In-run gates

Each workflow step should continue to fail fast when hard requirements are not met.

Required gates:

• stack_prep_refresh must report ready_for_stackStripMap_nofocus = true
• all expected run files must exist before stack execution
• run_08_igram output directories must exist
• MintPy required outputs must exist and be non-empty
• export must generate a manifest plus required assets
• registration must succeed against the catalog service

6.4 Post-publish health

Health is not only "the run finished".

The catalog and package checks must continue to validate:

• manifest exists
• publish dir exists
• processor code present
• runtime id present for WSL-native engines
• native output dir present
• canonical package schema valid
• manifest count versus DB count consistency

7. Frontend Design

7.1 Timeseries production panel

Keep frontend/src/TimeseriesProductionPanel.jsx as the main run workspace.

Planned enhancements:

• show structured runtime preflight results
• show batch preflight results before submission
• show phase-oriented run summary:
  • input prepared
  • stack ready
  • ISCE2 complete
  • MintPy complete
  • exported
  • published
• show key paths and quality summary blocks without forcing the operator to inspect raw JSON
• keep failed-step retry
• add clearer linkage to the published product once available

7.2 Product catalog panel

Keep frontend/src/components/PsinsarCatalogPanel.jsx as the catalog entry.

Planned enhancements:

• retain catalog status and rebuild actions
• enrich product detail with:
  • stack identity
  • processor/runtime identity
  • preview and primary assets
  • quality summary
  • asset list
  • issue list
  • coverage summary
• keep the publish bundle as the fact source

7.3 Health panel

Keep health visibility in frontend/src/HealthCheckPanel.jsx.

The timeseries section should continue to show:

• catalog status
• rebuild need
• manifest vs DB counts
• issue count

It should remain aligned with:

• timeseries_result_catalog
• product_packages
• wsl_runtime

8. Result Management and Registration

8.1 Registration rule

Register only from:

• <publish_dir>/manifest.json

Do not register from:

• MintPy work directories
• ISCE2 runtime directories
• ad hoc copied assets

8.2 Catalog model

Keep:

• catalog_name = psinsar
• product_family = timeseries

Current catalog service already derives:

• product id
• display name
• stack identity
• runtime and processor metadata
• bbox from asset summaries
• preview and primary asset paths

The next round should strengthen issue generation for missing assets and invalid package states.

8.3 Quarantine policy

Do not auto-delete broken publish packages.

If a rebuild finds broken packages, the preferred behavior is:

• keep package on disk
• create ResultIssueORM records
• downgrade health_status
• use quarantine status only when the package is structurally unusable

This keeps auditability intact.

9. Implementation Strategy

9.1 Do not over-refactor first

The old design expected many new modules. That is no longer necessary because the codebase already has the main modules.

For the next round:

• keep timeseries_service.py as the orchestration center
• keep job_handlers.py as worker entrypoints
• keep psinsar_catalog_service.py as catalog authority
• extract helper modules only when a block becomes independently reusable or too large

9.2 Recommended implementation phases

Phase A: contract and self-check hardening

Files likely involved:

• backend/app/services/timeseries_service.py
• backend/app/services/health_service.py
• frontend/src/TimeseriesProductionPanel.jsx

Target:

• stronger runtime report
• batch preflight
• clearer failure reasons

Phase B: quality summary and gating

Files likely involved:

• backend/app/services/timeseries_service.py
• experiments/isce2_sbas_timeseries/scripts/build_mintpy_publish_bundle.py
• backend/app/services/psinsar_catalog_service.py

Target:

• richer quality_summary_json
• richer manifest quality block
• stronger publish/register gates

Phase C: frontend run and catalog UX

Files likely involved:

• frontend/src/TimeseriesProductionPanel.jsx
• frontend/src/components/PsinsarCatalogPanel.jsx
• frontend/src/api/timeseriesProduction.js
• frontend/src/api/psinsarProducts.js

Target:

• better preflight display
• better run summary
• richer product detail

Phase D: validation and operator closure

Target:

• one small LT-1 AOI end-to-end validation
• one rerun-from-failure validation
• catalog rebuild validation
• deployment/ops notes update

10. Non-Goals For This Round

Do not include the following in the same implementation round:

• Gamma/PyINT timeseries integration
• full PS-InSAR or StaMPS
• atmospheric correction productization
• topographic residual correction productization
• large database namespace migration from psinsar to timeseries

These are valid future directions, but they should not be mixed into the current SBAS production hardening round.

11. Acceptance Criteria

The engineering expansion can be treated as complete for this round when all of the following are true:

1. An operator can run runtime preflight and understand failures before queuing a run.
2. A stored PS stack batch can be submitted as one managed SBAS run.
3. The run can execute through all eight workflow steps in the managed path.
4. Failure at any step produces a clear error and supports controlled retry.
5. The publish bundle is complete and canonical.
6. The product is registered into the psinsar catalog from manifest.json.
7. The frontend can show:
   • run state
   • workflow step state
   • published product linkage
   • product assets and quality summary
8. GET /api/health remains healthy for:
   • timeseries_result_catalog
   • product_packages
   • wsl_runtime

12. Related Documents

• docs/ISCE2_SBAS_TIMESERIES_DESIGN.md
• docs/ISCE2_SBAS_PRODUCT_SPEC.md
• docs/DEPLOYMENT.md
• docs/CURRENT_STATUS_20260425.md
• experiments/isce2_sbas_timeseries/README.md