Files
insar-management-system-v2/experiments/isce2_sbas_timeseries

ISCE2 SBAS Time-Series Experiments

This folder is the isolated sandbox for validating the SBAS/time-series route before wiring it into production code.

Purpose

  • verify LT-1 stack compatibility with ISCE2 stack tooling
  • validate MintPy input and output expectations
  • record runnable command templates
  • collect conclusions that should later be promoted into docs/ or backend services
  • maintain the current SBAS product contract before backend embedding

Structure

  • notes/
    • experiment notes, pitfalls, conclusions
  • configs/
    • sample templates, parameter files, manifest drafts
  • scripts/
    • throwaway or semi-stable experiment scripts
  • scratch/
    • local temporary workspace placeholder only

Rules

  • do not commit raw SAR scenes
  • do not commit large DEM or orbit datasets
  • do not commit large intermediate outputs
  • keep production code changes out of this folder unless the goal is to prototype file layout or commands
  • when an experiment becomes stable, move the result back into the formal backend or docs/

Suggested first experiments

  1. Check whether LT-1/LUTAN1 scenes can be ingested by official ISCE2 stack tooling.
  2. Determine whether MintPy can consume the generated stack layout without extra conversion.
  3. Record the minimal command chain needed for one small AOI smoke test.
  4. Draft the first psinsar manifest and product directory convention.

Current scripts

  • backend/app/isce2_pipeline/lt1_input_resolver.py
    • shared LT-1 input helper reused by D-InSAR and stack experiments
    • centralizes DEM resolution, orbit-pool resolution, and LT-1 precise-orbit XML generation
  • scripts/scan_lt1_stack_candidates.py
    • scan LT-1 single-scene folders and build a stack candidate manifest
  • scripts/build_lt1_stack_prep.py
    • consume a selected stack manifest
    • resolve orbit pool and DEM
    • generate a dry-run scratch/... workspace for stripmapStack --nofocus
    • write the current adapter contract and a preflight run script
  • scripts/materialize_lt1_stack_scenes.py
    • consume scratch/.../stack_input_manifest.json
    • materialize one or more LT-1 acquisitions into SLC/YYYYMMDD/
    • write YYYYMMDD.slc, YYYYMMDD.slc.xml, and data
  • scripts/install_isce2_stack_runtime_ubuntu2404.sh
    • install known WSL isce2 runtime dependencies
    • default pip mirror is Tsinghua
  • scripts/install_mintpy_runtime_ubuntu2404.sh
    • create or update a dedicated WSL mintpy conda environment
    • default conda channels use Tsinghua mirror URLs
  • scripts/install_mintpy_into_cloned_isce2_env_ubuntu2404.sh
    • clone the working WSL isce2 env into a dedicated unified-env target such as isce2_mintpy
    • install MintPy into that clone with Tsinghua mirror channels
  • scripts/run_mintpy_unified_env_ubuntu2404.sh
    • run MintPy commands directly inside the cloned unified env
  • scripts/run_mintpy_sbas_unified_env_smoketest_ubuntu2404.sh
    • run the current LT-1 SBAS smoke test in the cloned unified env
    • reuses the same strict-mask and patched-launcher helpers as the bridge route
  • scripts/run_mintpy_with_isce_ubuntu2404.sh
    • run MintPy commands in the dedicated mintpy env
    • bridge only the top-level WSL isce package into the mintpy env
    • avoids pulling conflicting h5py / numeric packages from the isce2 env
  • scripts/create_mintpy_all_ifgram_mask.py
    • build a strict maskAllValid.h5 from inputs/ifgramStack.h5
    • keep only pixels valid in all interferograms before SBAS inversion
  • scripts/run_smallbaselineApp_patched.py
    • repo-local launcher for MintPy smallbaselineApp
    • applies a local workaround for the MintPy 1.6.2 single-pixel partial-network inversion bug
  • scripts/run_mintpy_sbas_smoketest_ubuntu2404.sh
    • run the current LT-1 SBAS smoke test in three steps:
      • load_data
      • strict-mask generation
      • modify_network -> velocity
  • scripts/export_mintpy_publish_products_ubuntu2404.sh
    • geocode MintPy outputs into latitude/longitude grids
    • convert selected outputs into GeoTIFF
    • build a publish-style bundle with manifest.json, assets/, preview/, and metadata/
    • defaults to the bridge runner but now also supports MINTPY_RUNNER=... override
  • scripts/export_mintpy_publish_products_unified_env_ubuntu2404.sh
    • run the same publish export logic through the cloned unified env runner
  • scripts/export_conda_env_snapshot_ubuntu2404.sh
    • export one WSL conda environment into reproducible snapshot files
    • writes no_builds.yml, explicit.txt, conda_list.txt, and runtime_versions.txt
  • scripts/export_phase4_env_snapshots_ubuntu2404.sh
    • export both isce2 and isce2_mintpy_v1 snapshots for the current phase-4 record
  • scripts/build_mintpy_publish_bundle.py
    • generate preview/velocity_preview.png
    • summarize quality masks
    • write the publish-style manifest.json
  • scripts/prepare_lt1_stack_dem.py
    • clip a stack-local DEM window from the source DEM
    • store it under scratch/.../inputs/dem/
    • avoid global-DEM bbox problems during createWaterMask
  • scripts/run_generated_stack_runfile_ubuntu2404.sh
    • execute one generated run_XX_* file under Ubuntu-24.04
    • standardize PATH, PYTHONPATH, and log output
  • scripts/create_synthetic_watermask.py
    • create a local all-land geom_reference/waterMask.rdr
    • used only when run_01_reference cannot download SWBD from Earthdata

Reproducible Flow

  1. Generate or refresh the sample stack workspace with build_lt1_stack_prep.py.
  2. Materialize the LT-1 acquisitions with materialize_lt1_stack_scenes.py.
  3. Clip a stack-local DEM window with prepare_lt1_stack_dem.py.
  4. In Ubuntu-24.04, run scripts/install_isce2_stack_runtime_ubuntu2404.sh.
  5. Regenerate the stack workspace so it picks the local DEM.
  6. Run the generated wrapper through the validated chain:
    • scripts/run_generated_stack_runfile_ubuntu2404.sh <scratch_root_wsl> run_01_reference
    • scripts/run_generated_stack_runfile_ubuntu2404.sh <scratch_root_wsl> run_02_focus_split
    • scripts/run_generated_stack_runfile_ubuntu2404.sh <scratch_root_wsl> run_03_geo2rdr_coarseResamp
    • scripts/run_generated_stack_runfile_ubuntu2404.sh <scratch_root_wsl> run_04_refineSecondaryTiming
    • scripts/run_generated_stack_runfile_ubuntu2404.sh <scratch_root_wsl> run_05_invertMisreg
    • scripts/run_generated_stack_runfile_ubuntu2404.sh <scratch_root_wsl> run_06_fineResamp
    • scripts/run_generated_stack_runfile_ubuntu2404.sh <scratch_root_wsl> run_07_grid_baseline
    • if Earthdata credentials are missing, the wrapper now auto-generates a synthetic all-land waterMask.rdr from shadowMask.rdr and treats run_01_reference as recovered
  7. In Ubuntu-24.04, run scripts/install_mintpy_runtime_ubuntu2404.sh before the first MintPy validation.
  8. In Ubuntu-24.04, run scripts/run_mintpy_with_isce_ubuntu2404.sh prep_isce.py ... for the first MintPy metadata preparation on stripmapStack outputs.
  9. In Ubuntu-24.04, run:
    • scripts/run_mintpy_sbas_smoketest_ubuntu2404.sh <cfg_wsl> <mintpy_work_dir_wsl>
  10. Review:
  • notes/PHASE2_MINTPY_SBAS_SMOKETEST.md
  1. In Ubuntu-24.04, run:
  • scripts/export_mintpy_publish_products_ubuntu2404.sh <mintpy_work_dir_wsl> <publish_dir_wsl>
  1. Review:
  • notes/PHASE3_PUBLISH_EXPORT_SMOKETEST.md
  1. Record findings under notes/ before promoting anything into backend code.

Product Contract

Formal product guidance now lives in:

  • docs/ISCE2_SBAS_PRODUCT_SPEC.md

Current practical rule:

  • runtime success is proven by radar-coordinate timeseries.h5 and velocity.h5
  • publish success is proven by a geocoded bundle under publish/.../
  • system embedding should treat manifest.json as the publish entrypoint
  • true time-series capability should be judged against assets/geo_timeseries.h5, not only assets/velocity.tif

Unified-Environment Track

The bridge route remains the validated baseline.

A separate unified-environment experiment is now staged in:

  • notes/PHASE4_UNIFIED_ENV_EXPERIMENT.md
  • notes/PHASE4_UNIFIED_ENV_DECISION.md

Current practical rule:

  • the unified env has now completed the same SBAS smoke test chain and publish export in experiment scope
  • the unified env is now the preferred SBAS experiment runtime
  • do not replace or mutate the current isce2 env used by D-InSAR production
  • do not replace the bridge route as the default fallback until the comparison note is fully written
  • current successful unified env:
    • /home/administrator/miniconda3/envs/isce2_mintpy_v1
  • current successful unified SBAS work dir:
    • scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/mintpy_sbas_unified_v1
  • current successful unified publish dir:
    • scratch/lt1a_strip1_hh_descending_e123p3_n46p1/publish/mintpy_sbas_unified_v1

Current Offline Assumption

  • the LT-1 sample experiment already has local SAR scenes, local orbit XML, and a local DEM
  • the remaining optional online dependency is the SWBD water mask normally fetched by createWaterMask.py
  • for this experiment track, do not download SWBD
  • continue with the local synthetic all-land waterMask.rdr fallback until the stack route is otherwise stable
  • current validated MintPy boundary is radar-coordinate timeseries.h5 plus velocity.h5
  • current validated publish boundary is a geocoded experiment bundle under publish/.../