chore: initialize insar management system v2

This commit is contained in:
2026-04-14 13:16:01 +08:00
commit ecc72ec9cd
361 changed files with 2142522 additions and 0 deletions
@@ -0,0 +1 @@
@@ -0,0 +1,62 @@
# Phase 0 Environment Check
Updated: 2026-04-03
## Confirmed
- Project workspace:
- Windows repo root: `Z:\Code\Insar_management_system_v2`
- WSL mount path: `/mnt/z/Code/Insar_management_system_v2`
- Windows project Python env:
- `C:\Users\Administrator\.conda\envs\InSAR`
- WSL experiment distro:
- `Ubuntu-24.04`
- WSL project access:
- `/mnt/z/Code/Insar_management_system_v2`
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries`
- WSL system Python:
- `Python 3.12.3`
- WSL conda root:
- `/home/administrator/miniconda3`
- WSL conda envs found:
- `base`
- `isce2`
- ISCE2 env package:
- `isce2 2.6.4`
- ISCE2 Python import path:
- `/home/administrator/miniconda3/envs/isce2/lib/python3.11/site-packages/isce/__init__.py`
- Lutan1 sensor module:
- `/home/administrator/miniconda3/envs/isce2/lib/python3.11/site-packages/isce/components/isceobj/Sensor/Lutan1.py`
- Official stack directories present:
- `/home/administrator/miniconda3/envs/isce2/share/isce2/stripmapStack`
- `/home/administrator/miniconda3/envs/isce2/share/isce2/topsStack`
## Confirmed Gaps
- `conda` is not currently on the default shell `PATH` inside `Ubuntu-24.04`.
- Use `/home/administrator/miniconda3/bin/conda` directly in scripts.
- `MintPy` is not installed in the `isce2` env yet.
- `conda list -n isce2 mintpy` returned no match.
- Calling `conda list -n isce2 ...` from a WSL bash script triggered a segmentation fault once.
- For experiment scripts, prefer `conda run -n isce2 python ...` checks over `conda list`.
## Implication
Phase 0 can start immediately for:
- LT-1 / ISCE2 stack compatibility checks
- workspace and path validation
- command-chain drafting
But the full SBAS chain cannot run end to end until one of these is true:
- `MintPy` is installed into `isce2`, or
- a separate WSL env with `MintPy` is prepared
## Next Checks
1. Verify ISCE2 stack scripts actually exist in `Ubuntu-24.04`.
2. Read `stripmapStack/README.md` and `stackStripMap.py` to identify required stack inputs.
3. Decide whether `MintPy` should share the `isce2` env or live in a separate env.
4. Verify LT-1 / LUTAN1 support is usable for stack-mode inputs, not only single-pair mode.
5. Draft a minimal stack experiment command chain under this folder.
@@ -0,0 +1,123 @@
# Phase 0 Repo Reader Findings
Updated: 2026-04-03
## Existing repo reader path
The current repository already has a stable LT-1 single-scene metadata ingestion path.
Main code path:
- `backend/app/services/data_service.py`
- `scan_radar_data()`
- `backend/app/utils.py`
- `parse_lt1_radar_filename()`
- `find_xml_file()`
- `parse_xml_metadata()`
## What the existing reader does
### 1. Folder-name parsing
`parse_lt1_radar_filename()` extracts from the directory name:
- `satellite`
- `satellite_mode`
- `receiving_station`
- `imaging_mode`
- `orbit_circle`
- `scene_center_lon`
- `scene_center_lat`
- `imaging_date`
- `acquisition_time_utc`
- `product_type`
- `polarization`
- `product_level`
- `product_unique_id`
Example supported name:
```text
LT1B_MONO_SYC_STRIP1_018153_E135.4_N48.3_20250701_SLC_HH_S2A_0000790171
```
### 2. XML discovery
`find_xml_file()` prefers:
- `*.meta.xml`
and falls back to:
- the only XML file in the directory, if there is just one
### 3. XML parsing
`parse_xml_metadata()` extracts:
- `orbit_direction`
- `imaging_mode`
- `polarization`
- `receiving_station`
- `satellite_mode`
- `orbit_circle` from `absOrbit`
- `scene_center_lon`
- `scene_center_lat`
- `acquisition_time_utc`
- `product_type`
- `product_level`
- `product_unique_id`
- `look_direction`
- corner coordinates and coverage polygon
### 4. Merge rule
`scan_radar_data()` merges:
- folder-name metadata
- XML metadata
with XML preferred for most fields, except `product_unique_id` where the folder-name value is preserved if present.
## Why this matters for SBAS experiments
This means the SBAS experiment should not invent a separate metadata interpretation unless absolutely necessary.
Recommended rule:
- reuse the same field semantics already used by `RadarDataORM`
- reuse the same `.meta.xml` discovery logic
- treat `scan_radar_data()` output as the canonical single-scene asset layer
## Data layout check against `F:\Insar_data_pool_1`
Sample scene directories under `F:\Insar_data_pool_1` are compatible with the current single-scene reader:
- one folder per scene
- directory name matches LT-1 parser expectations
- contains `*.meta.xml`
- contains `*.tiff`
- contains preview and auxiliary files
Example sample directory:
```text
F:\Insar_data_pool_1\LT1A_MONO_KSC_STRIP1_017030_E123.3_N46.1_20250315_SLC_HH_S2A_0000678238
```
Example files inside:
- `...meta.xml`
- `...tiff`
- `...rpc`
- `...browse.jpg`
- `...thumb.jpg`
## Practical implication
For phase 1 design and experiments:
- the current repo already knows how to ingest these LT-1 scene folders as `RadarData`
- stack preparation should build on top of this asset layer
- the real unknown is not scene metadata parsing
- the real unknown is how to transform these scene folders into a stack layout acceptable to `stripmapStack`
@@ -0,0 +1,92 @@
# Phase 0 Sample Stack Selection
Updated: 2026-04-03
## Selected baseline sample
Current baseline sample stack:
- group key:
- `LT1A|STRIP1|HH|DESCENDING|E123.3_N46.1`
- manifest:
- `experiments/isce2_sbas_timeseries/configs/sample_stack_e123p3_n46p1.json`
## Sample summary
- satellite:
- `LT1A`
- mode:
- `STRIP1`
- polarization:
- `HH`
- orbit direction:
- `DESCENDING`
- scene count:
- `5`
- dates:
- `20250118`
- `20250315`
- `20250510`
- `20250705`
- `20250830`
- recommended reference date:
- `20250510`
- receiving stations observed:
- `SYC`
- `KSC`
## Why this sample is useful
- It already satisfies a minimal SBAS smoke-test stack size.
- All scenes share the same:
- satellite
- imaging mode
- polarization
- orbit direction
- tile key
- The dates are evenly spaced enough to act as a first time-series experiment set.
## Important adjacent-tile signal
This sample is not isolated.
The same date sequence also appears in multiple neighboring descending tiles, including:
- `E123.5_N46.6`
- `E123.6_N47.0`
- `E123.8_N47.5`
- `E123.9_N48.0`
- `E124.5_N49.9`
- `E124.7_N50.3`
- `E124.8_N50.8`
- `E125.0_N51.3`
- `E125.1_N51.8`
- `E125.3_N52.2`
- `E125.5_N52.7`
These neighboring tiles share the same 5 acquisition dates:
- `20250118`
- `20250315`
- `20250510`
- `20250705`
- `20250830`
## Implication
This strongly suggests the data pool contains a larger repeated strip-family, not just isolated scenes.
Recommended experiment order:
1. Start with one tile-level stack smoke test using `E123.3_N46.1`.
2. If stack-prep works, expand to multiple adjacent tiles with the same date family.
3. Only after that, test a wider strip or mosaic strategy.
## Current risk judgment
The main remaining uncertainty is still not scene selection.
The main uncertainty is:
- how to convert LT-1 per-scene `tiff + meta.xml` folders
- into a stack layout and sensor input form acceptable to the ISCE2 stripmap stack workflow
@@ -0,0 +1,140 @@
# Phase 0 Stack Findings
Updated: 2026-04-03
## Confirmed
- Official stack tooling exists in the `isce2` env:
- `/home/administrator/miniconda3/envs/isce2/share/isce2/stripmapStack`
- `/home/administrator/miniconda3/envs/isce2/share/isce2/topsStack`
- `stackStripMap.py` exists and is the stripmap stack entry point.
- `prepStripmap4timeseries.py` exists in the official `stripmapStack` toolset.
- `stackStripMap.py` expects:
- an SLC root directory via `-s/--slc_directory`
- a DEM via `-d/--dem`
- an optional reference date via `-m/--reference_date`
- temporal and baseline thresholds
- The script scans date subdirectories under the SLC root.
- Default behavior looks for `<date>.raw` inside each acquisition directory.
- With `--nofocus`, it instead looks for `<date>.slc`.
- Deeper code inspection confirms:
- `topo.py` opens `<date>/data`
- `geo2rdr.py` opens each secondary `<date>/data`
- `refineSecondaryTiming` uses both `<date>.slc` and the acquisition directory as metadata roots
## Important implication
Official stack processing expects a stack-style input layout such as:
```text
SLC/
YYYYMMDD/
YYYYMMDD.raw
```
or, when data are already focused:
```text
SLC/
YYYYMMDD/
YYYYMMDD.slc
YYYYMMDD.slc.xml
data
```
This is different from the current repository's custom LT-1 single-pair production flow.
## Time-series bridge signal
- `prepStripmap4timeseries.py` takes:
- pair/interferogram directories
- baseline directory
- geometry directory
- shelve metadata directory
- The script writes `.rsc` sidecars and explicitly references `pysar`-style downstream usage.
This is useful because it confirms the official stripmap stack toolset already contains a bridge from stack outputs toward time-series preparation.
The weak point is still the LT-1 stack input/preparation stage, not the existence of a downstream time-series bridge.
## Existing repo bridge signal
The repository's current LT-1 single-pair pipeline already proves one important thing:
- `stripmapApp.py` can be driven with:
- `sensor name = LUTAN1`
- direct `tiff` input path
- direct `orbitFile` XML path
See:
- `backend/app/isce2_pipeline/run_lt1_dinsar_pipeline.py`
The generated XML writes:
- `Reference -> tiff`
- `Reference -> orbitFile`
- `Secondary -> tiff`
- `Secondary -> orbitFile`
This suggests a promising adapter direction:
- do not try to pretend LT-1 is ALOS or another officially prepared raw sensor
- instead, explore generating LT-1-aware stack configs directly from:
- scene `tiff`
- scene `meta.xml`
- converted orbit XML
That does not prove the official stack driver will accept this without modification.
But it is the strongest current indication for how a custom LT-1 `stack-prep` layer should be shaped.
## LT-1 / LUTAN1 signal so far
- ISCE core does include a `Lutan1.py` sensor module.
- But no direct `lutan` match was found in the `stripmapStack` helper scripts.
- The `stripmapStack` README examples and preparation hints mention:
- `prepRawALOS.py`
- `prepRawSensor.py`
- The README explicitly states automatic raw-data preparation support is currently oriented to:
- ALOS
- CSK
- `prepRawSensors.py` automatic raw detection currently covers:
- Envisat
- ERS CEOS
- ERS ENV
- ALOS1
- CSK
- `prepSlcSensors.py` automatic SLC detection currently covers:
- Envisat
- ALOS1
- CSK
- RSAT2
- TSX/TDX
- No LT-1 or LUTAN1 hook was found in these official stack preparation scripts.
## Interim conclusion
Current evidence suggests:
- ISCE2 core can parse LT-1/LUTAN1 at the sensor level.
- Official `stripmapStack` tooling is present.
- But the official stack preparation helpers do not currently advertise LT-1/LUTAN1 support.
- There is no direct evidence yet that LT-1 can be fed into the official stack helpers without an adapter step.
- `--nofocus` does not remove the need for acquisition metadata preparation.
- It still needs a per-date `data` shelve and an ISCE-style `.slc` image.
This means the working assumption should be:
- `LT-1 stack via official stripmapStack` is possible but unproven
- an LT-1-specific stack preparation or conversion layer is required unless an existing hidden tool can materialize `data` + `.slc` directly for LUTAN1
## Next checks
1. Implement a dry-run LT-1 stack-prep workspace generator against the selected sample manifest.
2. Design a scene materializer that transforms one LT-1 scene into:
- `YYYYMMDD.slc`
- `YYYYMMDD.slc.xml`
- `data`
3. Decide whether that materializer should:
- call ISCE/LUTAN1 directly, or
- reuse parts of the existing pair pipeline
4. After materialization is proven, run `stackStripMap.py --nofocus` on one tile-level stack
@@ -0,0 +1,285 @@
# Phase 1 Stack Generation Smoke Test
Updated: 2026-04-05
Follow-up note:
- MintPy SBAS continuation is now recorded separately in:
- `notes/PHASE2_MINTPY_SBAS_SMOKETEST.md`
## Goal
Validate that one LT-1 stack can be transformed from:
- per-scene `tiff + meta.xml + orbit.xml`
into:
- `stripmapStack --nofocus` compatible acquisition directories
- a generated stripmap stack work plan
without changing production code yet.
## Sample
- group key:
- `LT1A|STRIP1|HH|DESCENDING|E123.3_N46.1`
- dates:
- `20250118`
- `20250315`
- `20250510`
- `20250705`
- `20250830`
- reference date:
- `20250510`
## Commands Used
1. Build dry-run stack workspace:
```text
C:\Users\Administrator\.conda\envs\InSAR\python.exe experiments\isce2_sbas_timeseries\scripts\build_lt1_stack_prep.py --manifest-path experiments\isce2_sbas_timeseries\configs\sample_stack_e123p3_n46p1.json
```
2. Materialize LT-1 acquisitions inside `Ubuntu-24.04`:
```text
wsl -d Ubuntu-24.04 /home/administrator/miniconda3/bin/conda run -n isce2 python /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/materialize_lt1_stack_scenes.py --stack-manifest /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_input_manifest.json
```
3. Run generated wrapper:
```text
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/run_stripmap_stack_dryrun.sh
```
4. Execute the frozen stack step chain inside `Ubuntu-24.04`:
```text
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_generated_stack_runfile_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1 run_01_reference
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_generated_stack_runfile_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1 run_02_focus_split
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_generated_stack_runfile_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1 run_03_geo2rdr_coarseResamp
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_generated_stack_runfile_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1 run_04_refineSecondaryTiming
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_generated_stack_runfile_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1 run_05_invertMisreg
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_generated_stack_runfile_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1 run_06_fineResamp
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_generated_stack_runfile_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1 run_07_grid_baseline
```
5. Regenerate the stack in `interferogram` workflow mode and execute the new pair-processing stage:
```text
C:\Users\Administrator\.conda\envs\InSAR\python.exe experiments\isce2_sbas_timeseries\scripts\build_lt1_stack_prep.py --manifest-path experiments\isce2_sbas_timeseries\configs\sample_stack_e123p3_n46p1.json --workflow interferogram
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/run_stripmap_stack_dryrun.sh
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_generated_stack_runfile_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1 run_08_igram
```
6. Prepare MintPy metadata in the dedicated `mintpy` env while bridging the working `isce2` Python package:
```text
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/install_mintpy_runtime_ubuntu2404.sh
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_mintpy_with_isce_ubuntu2404.sh prep_isce.py -f "/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/Igrams/*/filt_*.unw" -m /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/Igrams/20250118_20250315/referenceShelve/data.dat -b /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/baselines -g /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/geom_reference
```
## Confirmed Results
- All 5 acquisitions were materialized into:
- `scratch/lt1a_strip1_hh_descending_e123p3_n46p1/SLC/YYYYMMDD/`
- Each acquisition directory now contains:
- `YYYYMMDD.slc`
- `YYYYMMDD.slc.xml`
- `YYYYMMDD.slc.vrt`
- `data.dat/.dir/.bak`
- Example materialized output:
- `20250510.slc`
- size: `3262658784` bytes
- `stackStripMap.py --nofocus` ran successfully far enough to:
- discover all 5 acquisitions
- estimate stack baselines
- select interferometric pairs
- generate stack config files
- generate run files
## Baseline Snapshot
Relative to reference date `20250510`, the generated stack reported:
- `20250118`
- `-199.6873591838194`
- `20250315`
- `-89.0330513325973`
- `20250705`
- `-561.9056433404087`
- `20250830`
- `215.98283570831154`
The generated network reported:
- minimum connection degree:
- `4.0`
- number of pairs:
- `10`
## Generated Stack Work Products
Under `scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/`:
- `baselines/`
- `configs/`
- `run_files/`
- `pairs.pdf`
Generated run files:
- `run_01_reference`
- `run_02_focus_split`
- `run_03_geo2rdr_coarseResamp`
- `run_04_refineSecondaryTiming`
- `run_05_invertMisreg`
- `run_06_fineResamp`
- `run_07_grid_baseline`
## Important Runtime Fixes
- `matplotlib` was missing from the `isce2` env.
- fixed by installing it with `pip` inside the WSL `isce2` environment
- `stackStripMap.py` could not import `stripmapStack.Stack` by default.
- fixed by exporting:
- `PYTHONPATH=/home/administrator/miniconda3/envs/isce2/share/isce2`
- `PATH=/home/administrator/miniconda3/envs/isce2/share/isce2/stripmapStack:$PATH`
- the generated run files now include these prefixes automatically
- `run_01_reference` reached `topo` successfully but `createWaterMask.py` failed without Earthdata credentials.
- root cause:
- `SWBD` download requires `~/.netrc` for `urs.earthdata.nasa.gov`
- current judgment:
- local DEM is already sufficient for this experiment
- the only missing optional online input is the water-body mask download
- experimental fallback:
- `scripts/run_generated_stack_runfile_ubuntu2404.sh` now auto-generates a synthetic all-land `geom_reference/waterMask.rdr`
- the helper script is `scripts/create_synthetic_watermask.py`
- limitation:
- this fallback preserves stack execution but does not provide a true coastline mask
- LT-1 input preparation now has a shared helper:
- `backend/app/isce2_pipeline/lt1_input_resolver.py`
- purpose:
- centralize DEM resolution
- centralize orbit-pool resolution
- centralize LT-1 precise-orbit XML reuse or generation
- compatibility rule:
- this is a refactor of shared input-prep logic
- the original D-InSAR execution path was not removed
- `run_lt1_dinsar_pipeline.py` still keeps its original public workflow entry and now calls the helper internally
## Latest Execution Status
- `run_01_reference`
- `topo` completed successfully in `Ubuntu-24.04`
- local `geom_reference/waterMask.rdr` was synthesized from `shadowMask.rdr`
- `run_02_focus_split`
- completed successfully
- generated configs were effectively no-op under the current `--nofocus` contract
- `run_03_geo2rdr_coarseResamp`
- completed successfully in `Ubuntu-24.04`
- generated `offsets/<date>/range.off` and `azimuth.off` for:
- `20250118`
- `20250315`
- `20250705`
- `20250830`
- generated `coregSLC/Coarse/<date>/YYYYMMDD.slc` products for:
- `20250118`
- `20250315`
- `20250705`
- `20250830`
- runtime observation:
- this stage is long-running and mostly silent in the log file
- progress is easier to confirm from product directories than from stdout
- `run_04_refineSecondaryTiming`
- completed successfully in `Ubuntu-24.04`
- generated pair-level `refineSecondaryTiming/pairs/<pair>/misreg.*` for all 10 pairs
- log observation:
- `Bad match at level 1` and `correlation error` appeared in the log
- despite that noise, the stage exited `0` and downstream inversion succeeded
- `run_05_invertMisreg`
- completed successfully in `Ubuntu-24.04`
- generated date-level `refineSecondaryTiming/dates/<date>/misreg.*` for:
- `20250118`
- `20250315`
- `20250510`
- `20250705`
- `20250830`
- inversion observation:
- design matrix was reported as full rank
- RMSE in azimuth was `0.002341399255443996` pixels
- RMSE in range was `0.0027480408593210303` pixels
- `run_06_fineResamp`
- completed successfully in `Ubuntu-24.04`
- generated fine coregistered `merged/SLC/<date>/YYYYMMDD.slc` for all 5 dates
- each merged date directory now also includes:
- `referenceShelve/`
- `secondaryShelve/`
- `run_07_grid_baseline`
- completed successfully in `Ubuntu-24.04`
- generated `merged/baselines/<date>/` baseline grids for all 5 dates
- each date-level baseline directory now includes:
- raw baseline raster
- `.xml`
- `.vrt`
- `.full.vrt`
- `build_lt1_stack_prep.py --workflow interferogram`
- now regenerates the official stripmapStack command in `interferogram` mode instead of hard-coding `slc`
- regenerated run files now include:
- `run_08_igram`
- `run_08_igram`
- completed successfully in `Ubuntu-24.04`
- generated 10 pair directories under `stack_work/Igrams/`
- each pair now includes:
- wrapped interferogram `.int`
- amplitude `.amp`
- filtered interferogram `filt_*.int`
- coherence `filt_*.cor`
- unwrapped phase `filt_*_snaphu.unw`
- connected components `*.unw.conncomp`
- `referenceShelve/data.*`
- MintPy runtime bootstrap
- `scripts/install_mintpy_runtime_ubuntu2404.sh` created a dedicated WSL env:
- `mintpy`
- verified commands:
- `smallbaselineApp.py`
- `prep_isce.py`
- installed MintPy version:
- `1.6.2`
- `prep_isce.py`
- first run in the clean `mintpy` env failed because `mintpy.utils.isce_utils` imports `isce`
- experimental resolution:
- `scripts/run_mintpy_with_isce_ubuntu2404.sh` now bridges only the top-level `isce` package from the WSL `isce2` env into the `mintpy` env
- result:
- `prep_isce.py` completed successfully over the LT-1 `stripmapStack` outputs
- geometry `.rsc` files were written under `stack_work/geom_reference/`
- observation `.rsc` files were written for all 10 unwrapped interferograms under `stack_work/Igrams/*/`
## Current Boundary
This smoke test confirms:
- LT-1 scenes can be materialized into `stripmapStack` acquisition directories
- the official stripmap stack driver can build the stack work plan over those LT-1 products
- the generated `run_01` to `run_07` chain can complete offline in `Ubuntu-24.04` over the sample LT-1 stack
- local DEM plus local orbit data are sufficient for this stack-preparation stage
- Earthdata credentials are not a hard blocker for this experiment track because the wrapper can recover `run_01_reference` with a synthetic all-land `waterMask`
- the same LT-1 stack can be regenerated in `interferogram` workflow mode to produce 10 filtered and unwrapped pair products
- MintPy metadata preparation is now viable through the dedicated `mintpy` env plus the explicit ISCE bridge wrapper
This smoke test does not yet confirm:
- `smallbaselineApp.py` execution beyond `prep_isce.py`
- final time-series or velocity products
Follow-up status:
- both of the above were later confirmed in:
- `notes/PHASE2_MINTPY_SBAS_SMOKETEST.md`
## Next Tasks
1. Keep this note focused on stack-generation findings only.
2. Use `notes/PHASE2_MINTPY_SBAS_SMOKETEST.md` for MintPy SBAS runtime conclusions.
3. Promote the stable runtime and artifact contract into backend workflow code and `docs/`.
@@ -0,0 +1,157 @@
# Phase 2 Bridge Smoketest
Date: 2026-04-06
## Scope
Validate the current SBAS bridge chain with the previously verified LT-1 sample stack:
1. `build_lt1_stack_prep.py`
2. `materialize_lt1_stack_scenes.py`
3. `build_lt1_stack_prep.py` refresh
This run validates the current bridge boundary only:
- raw LT-1 scenes
- local precise orbit pool
- local prepared DEM
- fresh scratch workspace
It does not run:
- `stripmapStack`
- MintPy
- geocode/export/publish
## Inputs
- sample manifest:
- `experiments/isce2_sbas_timeseries/configs/sample_stack_e123p3_n46p1.json`
- orbit pool:
- `/mnt/d/orbit_pools/isce2`
- DEM:
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/inputs/dem/stack_dem_window.wgs84`
## Workspace
- scratch root:
- `experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406`
## Result
- final readiness: `True`
- scene count: `5`
- reference date: `20250510`
- all orbits resolved: `True`
- all `.slc/.slc.xml` present: `True`
- all `data` shelves present: `True`
## Materialization Summary
- dates:
- `20250118`
- `20250315`
- `20250510`
- `20250705`
- `20250830`
- status counts:
- `materialized: 5`
- total bytes written:
- `16313641344`
## Artifacts
- selected manifest used for this WSL run:
- `experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406/selected_stack_manifest_wsl.json`
- generated stack manifest:
- `experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406/stack_input_manifest.json`
- materialization summary:
- `experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406/materialization_summary.json`
- generated stack dry-run wrapper:
- `experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406/run_stripmap_stack_dryrun.sh`
- synthetic water-mask recovery report:
- `experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406/stack_work/logs/run_01_reference.synthetic_watermask.json`
## Finding
`build_lt1_stack_prep.py` currently reads `scene["tiff_path"]` and `scene["meta_path"]` directly.
When the script is run inside WSL against the sample manifest, the original `F:\...` Windows paths are not readable as Linux paths.
For this smoketest, a temporary WSL-path manifest copy was generated and used:
- `selected_stack_manifest_wsl.json`
This is an experiment-side workaround only.
No production/system logic was changed for this run.
## Update 2026-04-07
The same fresh workspace was then continued through the stripmap stack run files.
### Additional Result
- `run_01_reference`
- reached `createWaterMask`
- failed on remote `SWBD` retrieval from:
- `https://e4ftl01.cr.usgs.gov/MEASURES/SRTMSWBD.003/...`
- recovered with a local synthetic all-land `waterMask.rdr`
- `run_02_focus_split`
- completed
- `run_03_geo2rdr_coarseResamp`
- completed
- `run_04_refineSecondaryTiming`
- completed
- `run_05_invertMisreg`
- completed
- `run_06_fineResamp`
- completed
- `run_07_grid_baseline`
- completed
- `run_08_igram`
- completed through interferogram generation, filtering, coherence, and `snaphu` unwrapping
### Interferogram Snapshot
- pair count on disk:
- `10`
- verified pair folders:
- `20250118_20250315`
- `20250118_20250510`
- `20250118_20250705`
- `20250118_20250830`
- `20250315_20250510`
- `20250315_20250705`
- `20250315_20250830`
- `20250510_20250705`
- `20250510_20250830`
- `20250705_20250830`
- verified key files in every pair directory:
- `filt_<date12>.int`
- `filt_<date12>.cor`
- `filt_<date12>_snaphu.unw`
- `filt_<date12>_snaphu.unw.conncomp`
### Finding Update
The original `run_generated_stack_runfile_ubuntu2404.sh` fallback only matched the `.netrc` credential failure text.
This workspace showed a second offline failure mode:
- `createWaterMask` started normally
- ISCE2 `DataRetriever` failed during `SWBD` file retrieval
- the wrapper therefore did not auto-recover on the first attempt
The experiment helper has now been widened to recognize both:
- missing Earthdata credential text
- direct `SWBD` retrieval failure text from `createWaterMask`
No production/system runtime was changed by this fix.
## Recommended Next Step
Use this fresh workspace to continue with:
1. unified-env MintPy smoketest using:
- `configs/phase2_bridge_smoketest_20260406_smallbaseline.cfg`
2. publish-style geocode/export if MintPy succeeds
3. compare this fresh replay with the earlier baseline workspace
@@ -0,0 +1,200 @@
# Phase 2 MintPy SBAS Smoke Test
Updated: 2026-04-05
Follow-up note:
- publish-style geocode/export continuation is now recorded separately in:
- `notes/PHASE3_PUBLISH_EXPORT_SMOKETEST.md`
## Goal
Validate that the LT-1 sample stack can continue from `stripmapStack` interferogram products into MintPy SBAS outputs under `Ubuntu-24.04`.
## Sample
- group key:
- `LT1A|STRIP1|HH|DESCENDING|E123.3_N46.1`
- dates:
- `20250118`
- `20250315`
- `20250510`
- `20250705`
- `20250830`
- pair count:
- `10`
- reference date:
- `20250510`
- reference point:
- `y/x = 1994,52`
## Successful Run
Successful SBAS smoke-test work directory:
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/mintpy_sbas_v5`
Successful WSL command:
```text
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_mintpy_sbas_smoketest_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/configs/sample_smallbaseline_lt1_e123p3_n46p1.cfg /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/mintpy_sbas_v5
```
This run completed through:
- `load_data`
- `modify_network`
- `reference_point`
- `quick_overview`
- `invert_network`
- `reference_date`
- `velocity`
Disabled for this first offline smoke test:
- unwrap-error correction
- solid-earth-tide correction
- ionosphere correction
- troposphere correction
- deramp
- topographic residual correction
- geocode
## Output Snapshot
Generated under `mintpy_sbas_v5/`:
- `timeseries.h5`
- size: `82576752` bytes
- `velocity.h5`
- size: `83086408` bytes
- `temporalCoherence.h5`
- size: `16632424` bytes
- `maskTempCoh.h5`
- size: `4136792` bytes
- `avgSpatialCoh.h5`
- size: `16631784` bytes
- `numTriNonzeroIntAmbiguity.h5`
- size: `16632000` bytes
- `numTriNonzeroIntAmbiguity.png`
- size: `232878` bytes
Quality summary from the successful run:
- strict valid pixels for inversion:
- `1219001 / 4076199`
- `29.91%`
- reliable pixels in `maskTempCoh.h5` with threshold `0.7`:
- `62987`
## Required Runtime Decisions
### 1. Keep MintPy separate from ISCE2
- keep stack processing in WSL conda env:
- `isce2`
- keep MintPy in WSL conda env:
- `mintpy`
Reason:
- avoids mutating the already working ISCE2 processing env on the development machine
### 2. Bridge only the top-level `isce` package
Required helper:
- `scripts/run_mintpy_with_isce_ubuntu2404.sh`
Current rule:
- do not add the entire `isce2` `site-packages` into `PYTHONPATH`
- only bridge the top-level `isce` package into a cache directory
Reason:
- adding the whole `site-packages` caused MintPy to import `h5py` from the wrong env and fail during `load_data`
### 3. Do not load `wrapPhase` for this LT-1 smoke test
Current config rule:
- `mintpy.load.intFile = None`
Reason:
- loading `filt_*.int` into MintPy `wrapPhase` caused HDF5 type-conversion failure during `load_data`
### 4. Build a strict `maskAllValid.h5` before inversion
Required helper:
- `scripts/create_mintpy_all_ifgram_mask.py`
Current rule:
- keep only pixels that are finite and non-zero in all unwrapped interferograms
- also require non-zero connected components in all interferograms
Reason:
- this reduces unstable partial-network pixels before SBAS inversion
### 5. Use the repo-local patched launcher
Required helper:
- `scripts/run_smallbaselineApp_patched.py`
Current workaround:
- patch `mintpy.ifgram_inversion.estimate_timeseries()` at runtime
- coerce shape-`(1,)` inversion-quality output into a scalar for the single-pixel partial-network branch
Reason:
- MintPy `1.6.2` hit a `ValueError: setting an array element with a sequence`
- failure point:
- `mintpy/ifgram_inversion.py`
- partial-network pixel branch inside `run_ifgram_inversion_patch()`
Current judgment:
- this is a MintPy runtime issue in the current environment
- it is better to keep the workaround in repo-local launcher code than silently editing the third-party env
## Current Boundary
This smoke test now confirms:
- LT-1 `stripmapStack` outputs can be loaded by MintPy in the dedicated `mintpy` env
- the LT-1 sample stack can be inverted into radar-coordinate `timeseries.h5`
- the LT-1 sample stack can generate radar-coordinate `velocity.h5`
- the current repo-local workaround chain is reproducible in `Ubuntu-24.04`
This smoke test does not yet confirm:
- geocoded SBAS exports
- atmospheric or DEM-residual correction quality
- product publishing into backend `psinsar` catalog
- frontend rendering of published SBAS products
## System Embedding Implications
The current experiment suggests the future backend runtime contract should be:
1. Run ISCE2 stack workflow in WSL `isce2`.
2. Run MintPy through the repo-controlled bridge runner instead of calling upstream `smallbaselineApp.py` directly.
3. Generate a strict inversion mask after `load_data`.
4. Persist the following files as first-class workflow artifacts:
- `timeseries.h5`
- `velocity.h5`
- `temporalCoherence.h5`
- `maskTempCoh.h5`
- `numTriNonzeroIntAmbiguity.h5`
- `numTriNonzeroIntAmbiguity.png`
5. Convert these into a stable publish manifest before catalog registration.
Current sample publish-manifest draft:
- `configs/sample_psinsar_manifest_lt1_e123p3_n46p1.json`
@@ -0,0 +1,142 @@
# Phase 3 Publish Export Smoke Test
Updated: 2026-04-06
## Goal
Validate that the successful MintPy SBAS experiment can be converted into a publish-style artifact bundle without touching the main system.
## Inputs
Source MintPy work directory:
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/mintpy_sbas_v5`
Source stack:
- group key:
- `LT1A|STRIP1|HH|DESCENDING|E123.3_N46.1`
- dates:
- `20250118`
- `20250315`
- `20250510`
- `20250705`
- `20250830`
## Successful Command
```text
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/export_mintpy_publish_products_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/mintpy_sbas_v5 /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/publish/mintpy_sbas_v5
```
## Current Export Scripts
- `scripts/export_mintpy_publish_products_ubuntu2404.sh`
- geocode core MintPy outputs
- convert selected outputs to GeoTIFF
- copy preview and metadata files
- build root `manifest.json`
- `scripts/build_mintpy_publish_bundle.py`
- generate `preview/velocity_preview.png`
- generate `metadata/source_quality_summary.json`
- generate publish-style root `manifest.json`
## Geocode Contract
Current experiment settings:
- lookup source:
- `inputs/geometryRadar.h5`
- output pixel size:
- latitude step:
- `-0.000185185`
- longitude step:
- `0.000185185`
- interpolation:
- `nearest`
Observed geocoded grid:
- extent:
- south:
- `45.80391`
- north:
- `46.42206`
- west:
- `122.930244`
- east:
- `123.76654`
- shape:
- rows:
- `3338`
- columns:
- `4516`
## Generated Publish Bundle
Successful publish-style directory:
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/publish/mintpy_sbas_v5`
Main outputs:
- `manifest.json`
- `assets/geo_timeseries.h5`
- `assets/geo_velocity.h5`
- `assets/geo_temporalCoherence.h5`
- `assets/geo_maskTempCoh.h5`
- `assets/velocity.tif`
- `assets/temporalCoherence.tif`
- `assets/maskTempCoh.tif`
- `preview/velocity_preview.png`
- `preview/numTriNonzeroIntAmbiguity.png`
- `metadata/smallbaselineApp.cfg`
- `metadata/source_quality_summary.json`
## Current Interpretation
This confirms:
- the experiment now supports radar-coordinate MintPy inversion
- the experiment also supports geocoded HDF5 exports
- the experiment can produce publish-style GeoTIFF outputs
- the experiment can build a stable manifest-driven bundle outside the MintPy work directory
This does not yet confirm:
- direct backend catalog registration
- frontend rendering against the real system APIs
- whether `EPSG:4326` should remain the final publish CRS decision
## Important Notes
### 1. Current CRS assumption
`save_gdal.py` warned that no explicit `EPSG` metadata was found and assumed:
- `EPSG:4326`
Current judgment:
- acceptable for this experiment because the geocoded outputs are in latitude/longitude grids
- should still be checked when formalizing the production publish contract
### 2. Group key in the generated manifest
Because PowerShell treats `|` specially, passing group keys on the command line is awkward from Windows.
Current practical rule:
- the sample manifest stored in git remains the clean reference:
- `configs/sample_psinsar_manifest_lt1_e123p3_n46p1.json`
- the generated publish bundle manifest can be post-filled or generated from backend metadata later
## System Embedding Implication
At this point the experiment-layer chain is split cleanly into three stages:
1. `stripmapStack` preprocessing
2. MintPy SBAS inversion
3. geocode + publish-bundle export
That means the future system workflow can wire them as separate workflow steps without changing the validated experiment logic first.
@@ -0,0 +1,120 @@
# Phase 4 Unified-Environment Decision
Updated: 2026-04-06
## Decision
Current experiment preference:
- prefer the unified WSL environment for SBAS experiment work
Current production safety rule:
- do not replace or mutate the existing D-InSAR production environment
- keep pair-oriented D-InSAR on the existing WSL `isce2` env
- keep the current backend/public D-InSAR entry unchanged
Current environment split:
- D-InSAR production baseline:
- `isce2`
- SBAS experiment preferred runtime:
- `isce2_mintpy_v1`
## Why This Decision Is Reasonable
The unified env has now completed the full current experiment chain:
- MintPy command invocation without the `isce` bridge
- `load_data`
- strict-mask generation
- `modify_network -> velocity`
- publish export to geocoded HDF5, GeoTIFF, preview, and `manifest.json`
This makes the unified env a valid experiment baseline.
At the same time, the existing pair-oriented D-InSAR route is already working and should not be destabilized just to simplify the SBAS experiment runtime.
The safest rule is therefore:
- let SBAS experiments move forward in a separate unified env
- do not touch the current `isce2` production env used by D-InSAR
## Evidence
Successful unified env:
- `/home/administrator/miniconda3/envs/isce2_mintpy_v1`
Successful unified SBAS work directory:
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/mintpy_sbas_unified_v1`
Successful unified publish directory:
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/publish/mintpy_sbas_unified_v1`
Matched quality indicators:
- `maskAllValid`:
- `1219001 / 4076199`
- `29.91%`
- `maskTempCoh`:
- `62987 / 4076199`
- `1.55%`
Key package difference:
- current `isce2` env does not provide the MintPy-side package set needed for this SBAS route
- current `isce2_mintpy_v1` env includes:
- `mintpy`
- `cartopy`
- `pyaps3`
- `pykml`
- `cvxopt`
## What This Decision Does Not Mean
It does not mean:
- the backend should immediately switch to unified-env execution
- the bridge route must be deleted now
- the current D-InSAR runtime should be modified
It only means:
- for the next experiment steps, unified env is the preferred path
- for current production safety, `isce2` remains untouched
## Required Guardrails
For the next phase, keep these rules:
- do not install MintPy into the existing `isce2` env
- do not redirect current D-InSAR scripts to `isce2_mintpy_v1`
- do not remove the bridge-based helpers yet
- keep all SBAS work in experiment scripts, notes, and scratch directories
## Reproducibility
Environment snapshots should be exported and kept with the experiment record.
Current snapshot command:
```text
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/export_phase4_env_snapshots_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/configs/env_snapshots/20260406
```
## Next Stable Experiment Steps
Before any system integration work:
1. Keep the unified env as the default SBAS experiment runtime.
2. Preserve environment snapshots for both `isce2` and `isce2_mintpy_v1`.
3. Write one comparison note focused on:
- runtime simplicity
- reproducibility
- remaining workarounds
- risk to D-InSAR production
4. Optionally repeat the chain on one more LT-1 sample stack.
5. Only after the experiment is stable, design the system integration boundary.
@@ -0,0 +1,221 @@
# Phase 4 Unified-Environment Experiment
Updated: 2026-04-06
## Goal
Validate whether the current WSL `isce2` runtime can be cloned and extended with MintPy so that the SBAS experiment can run without the temporary `isce` bridge helper.
This is still an experiment-layer task.
Do not change the production backend yet.
## Why run this phase
The bridge-based route is already validated, but a unified environment may be cleaner because:
- many ISCE2 + MintPy users operate in one environment
- command invocation becomes simpler
- future worker deployment may be easier if one runtime is stable
The bridge-based route still remains the fallback baseline until this phase is verified.
## Current Known Starting Point
WSL distro:
- `Ubuntu-24.04`
Current environments observed on 2026-04-06:
- `isce2`
- `mintpy`
Observed package state:
- `isce2` env:
- `isce2 2.6.4`
- `h5py 3.15.1`
- `mintpy` not installed
- dedicated `mintpy` env:
- `mintpy 1.6.3`
## Initial Hypothesis
Expected best-case outcome:
- clone `isce2` into `isce2_mintpy`
- install `mintpy` directly into the clone
- reuse the same repo-local strict-mask and patched-launcher helpers
- run the same LT-1 smoke test without the `isce` bridge wrapper
Main risk areas:
- package solver may replace or downgrade key ISCE2-side numeric dependencies
- MintPy may still require the same repo-local runtime workaround even in a unified env
- GDAL / h5py / pyaps3 dependency changes may alter the known-good stack behavior
## Reproducible Commands
### 1. Bootstrap the unified env
```text
wsl -d Ubuntu-24.04 env TARGET_ENV=isce2_mintpy_v1 BOOTSTRAP_MODE=recreate USE_TUNA_MIRROR=1 MINTPY_SPEC=mintpy=1.6.3 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/install_mintpy_into_cloned_isce2_env_ubuntu2404.sh
```
Why `BOOTSTRAP_MODE=recreate`:
- direct `conda create --clone` was not stable enough for this machine
- it still followed source-package URLs and hit channel/TOS friction
- the successful path was:
- export the current `isce2` dependency list
- recreate the env through Tsinghua mirror channels
- reinstall exported pip packages
- install MintPy into the recreated env
Optional environment override:
```text
TARGET_ENV=isce2_mintpy_v1 MINTPY_SPEC='mintpy=1.6.3'
```
### 2. Run MintPy commands directly inside the unified env
```text
wsl -d Ubuntu-24.04 env MINTPY_ENV=isce2_mintpy_v1 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_mintpy_unified_env_ubuntu2404.sh prep_isce.py -h
```
### 3. Re-run the current LT-1 SBAS smoke test in the unified env
```text
wsl -d Ubuntu-24.04 env MINTPY_ENV=isce2_mintpy_v1 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_mintpy_sbas_unified_env_smoketest_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/configs/sample_smallbaseline_lt1_e123p3_n46p1.cfg /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/mintpy_sbas_unified_v1
```
### 4. Export publish bundle in the unified env
```text
wsl -d Ubuntu-24.04 env MINTPY_ENV=isce2_mintpy_v1 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/export_mintpy_publish_products_unified_env_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/mintpy_sbas_unified_v1 /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/publish/mintpy_sbas_unified_v1
```
## Comparison Checklist
When this phase is executed, compare it against the bridge route on:
- package versions after install
- whether `prep_isce.py` imports cleanly without bridge
- whether `load_data` succeeds
- whether the strict-mask step is still required
- whether the patched launcher is still required
- whether output files match the existing bridge-based artifact set
- whether geocode/export still succeeds from the unified env
## Current Status
Successful environment:
- `/home/administrator/miniconda3/envs/isce2_mintpy_v1`
Observed package/version state in the successful unified env:
- `conda list` shows:
- `mintpy 1.6.3`
- runtime `mintpy.__version__` reports:
- `1.6.2`
- `isce` import path:
- `/home/administrator/miniconda3/envs/isce2_mintpy_v1/lib/python3.11/site-packages/isce/__init__.py`
- `mintpy` import path:
- `/home/administrator/miniconda3/envs/isce2_mintpy_v1/lib/python3.11/site-packages/mintpy/__init__.py`
- `h5py`:
- `3.15.1`
Successful unified-env SBAS work directory:
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/mintpy_sbas_unified_v1`
Successful unified-env publish directory:
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/publish/mintpy_sbas_unified_v1`
Validated in the unified env:
- `prep_isce.py -h` works without the `isce` bridge
- `load_data` succeeded
- strict-mask generation succeeded
- `modify_network -> velocity` succeeded
- publish export succeeded through geocoded HDF5, GeoTIFF, preview, and `manifest.json`
Observed unified-env output set under `mintpy_sbas_unified_v1/`:
- `timeseries.h5`
- `velocity.h5`
- `temporalCoherence.h5`
- `maskTempCoh.h5`
- `maskAllValid.h5`
- `avgSpatialCoh.h5`
- `numTriNonzeroIntAmbiguity.h5`
- `numTriNonzeroIntAmbiguity.png`
Observed publish bundle under `publish/mintpy_sbas_unified_v1/`:
- `manifest.json`
- `assets/geo_timeseries.h5`
- `assets/geo_velocity.h5`
- `assets/geo_temporalCoherence.h5`
- `assets/geo_maskTempCoh.h5`
- `assets/velocity.tif`
- `assets/temporalCoherence.tif`
- `assets/maskTempCoh.tif`
- `preview/velocity_preview.png`
- `preview/numTriNonzeroIntAmbiguity.png`
- `metadata/smallbaselineApp.cfg`
- `metadata/source_quality_summary.json`
Quality summary matched the bridge-based route:
- `maskAllValid`:
- `1219001 / 4076199`
- `29.91%`
- `maskTempCoh`:
- `62987 / 4076199`
- `1.55%`
Still required in the unified env:
- strict `maskAllValid.h5` before inversion
- repo-local patched `smallbaselineApp` launcher
Prepared:
- unified-env bootstrap script
- unified-env MintPy runner
- unified-env SBAS smoke-test runner
- unified-env publish-export wrapper
Completed:
- actual clone + install execution
- actual smoke-test result capture
- actual publish-export capture
Pending:
- deeper comparison of output metadata against the bridge route
- decide whether unified env or bridge env should be the default production candidate
- decide whether to pin the runtime to the conda package label `1.6.3` or the internal MintPy version string `1.6.2`
## Current Judgment
At experiment level, the unified environment is now viable.
This phase confirms:
- the current LT-1 SBAS route does not fundamentally require the `isce` bridge
- a recreated `isce2 + mintpy` WSL env can complete:
- MintPy load/inversion
- geocode/export
- publish-bundle generation
Current recommendation:
- keep the bridge route as the already-known baseline until a fuller diff is written
- but treat the unified env as a valid candidate for the future production runtime
@@ -0,0 +1,193 @@
# Phase 4 Unified-Env Replay On Fresh Phase-2 Workspace
Updated: 2026-04-08
## Goal
Re-run the already validated unified-env SBAS route on the fresh workspace:
- `scratch/phase2_bridge_smoketest_20260406`
This checks that the current experiment does not rely on the older sample workspace only.
## Inputs
- WSL distro:
- `Ubuntu-24.04`
- unified env:
- `/home/administrator/miniconda3/envs/isce2_mintpy_v1`
- stack config:
- `configs/phase2_bridge_smoketest_20260406_smallbaseline.cfg`
- stack workspace:
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406`
- MintPy work dir:
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406/stack_work/mintpy_sbas_unified_phase2_20260407`
- publish dir:
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406/publish/mintpy_sbas_unified_phase2_20260407`
## Successful Commands
Unified-env MintPy smoke test:
```text
wsl -d Ubuntu-24.04 env MINTPY_ENV=isce2_mintpy_v1 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_mintpy_sbas_unified_env_smoketest_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/configs/phase2_bridge_smoketest_20260406_smallbaseline.cfg /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406/stack_work/mintpy_sbas_unified_phase2_20260407
```
Unified-env publish export:
```text
wsl -d Ubuntu-24.04 env MINTPY_ENV=isce2_mintpy_v1 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/export_mintpy_publish_products_unified_env_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406/stack_work/mintpy_sbas_unified_phase2_20260407 /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406/publish/mintpy_sbas_unified_phase2_20260407
```
## Result
The fresh workspace replay succeeded through:
- `load_data`
- strict `maskAllValid.h5` generation
- `modify_network`
- `reference_point`
- `quick_overview`
- `invert_network`
- `reference_date`
- `velocity`
- geocode/export
- publish-bundle generation
This confirms the current fresh workspace now reaches the same experiment boundary as the earlier baseline sample:
- radar-coordinate MintPy runtime products
- geocoded publish bundle
## Runtime Output Snapshot
Generated under `stack_work/mintpy_sbas_unified_phase2_20260407/`:
- `timeseries.h5`
- `82579888` bytes
- `velocity.h5`
- `83086408` bytes
- `temporalCoherence.h5`
- `16632424` bytes
- `maskTempCoh.h5`
- `4136848` bytes
- `maskAllValid.h5`
- `4095215` bytes
- `avgSpatialCoh.h5`
- `16631784` bytes
- `numTriNonzeroIntAmbiguity.h5`
- `16632000` bytes
- `numTriNonzeroIntAmbiguity.png`
- `232923` bytes
## Publish Bundle Snapshot
Generated under `publish/mintpy_sbas_unified_phase2_20260407/`:
- `manifest.json`
- `32442` bytes
- `assets/geo_timeseries.h5`
- `304443360` bytes
- `assets/geo_velocity.h5`
- `305627744` bytes
- `assets/geo_temporalCoherence.h5`
- `61141152` bytes
- `assets/geo_maskTempCoh.h5`
- `15260536` bytes
- `assets/velocity.tif`
- `60318026` bytes
- `assets/temporalCoherence.tif`
- `60318026` bytes
- `assets/maskTempCoh.tif`
- `15094802` bytes
- `preview/velocity_preview.png`
- `813928` bytes
- `preview/numTriNonzeroIntAmbiguity.png`
- `232923` bytes
- `metadata/smallbaselineApp.cfg`
- `26419` bytes
- `metadata/source_quality_summary.json`
- `403` bytes
## Quality Summary
From `metadata/source_quality_summary.json`:
- `maskAllValid`
- `1219067 / 4076199`
- `29.91%`
- `maskTempCoh`
- `63092 / 4076199`
- `1.55%`
- preview stretch:
- `vmin = -0.2251889556646347`
- `vmax = 0.2251889556646347`
## Findings
### 1. The fresh workspace replay is reproducible
The new workspace produced:
- the full 10-pair interferogram stack
- MintPy `timeseries.h5`
- MintPy `velocity.h5`
- publish-layer geocoded HDF5 / GeoTIFF / preview / `manifest.json`
This is the current strongest experiment proof that the SBAS route is not tied to the earlier historical scratch directory.
### 2. The same two MintPy experiment helpers are still required
The unified env still depends on:
- `create_mintpy_all_ifgram_mask.py`
- `run_smallbaselineApp_patched.py`
Current interpretation:
- unified env removes the temporary `isce` bridge
- it does not remove the current strict-mask or patched-launcher workarounds
### 3. Offline water-mask strategy remains valid
This replay consumed the synthetic all-land `waterMask.rdr` created earlier in the fresh workspace.
No Earthdata / `SWBD` download was needed for the MintPy or publish stages.
### 4. Geocode/export completed with a tolerable warning
`save_gdal.py` warned that no EPSG / UTM metadata was found and assumed:
- `EPSG:4326`
For this experiment chain, that is acceptable because the export was already driven by latitude / longitude lookup plus explicit `--lalo` sampling.
This warning should still be recorded for later production hardening.
### 5. `group_key` should come from system metadata
This host-side replay exported a valid publish bundle, and the bundle should carry:
- `"group_key": "LT1A|STRIP1|HH|DESCENDING|E123.3_N46.1"`
Reason:
- the bundle builder accepts `group_key` as a CLI argument
- the LT-1 group key contains pipe characters:
- `LT1A|STRIP1|HH|DESCENDING|E123.3_N46.1`
- when invoked naively from Windows PowerShell into WSL, the value may be truncated or split by the host shell
Current judgment:
- this is not a blocker for the SBAS scientific chain
- future system embedding should write `group_key` from task/run metadata inside backend code instead of relying on manual shell arguments
## Current Judgment
The current fresh workspace now confirms the full experiment chain:
- `raw LT-1 -> stripmapStack -> MintPy SBAS -> geocode/export -> publish bundle`
At experiment level, the unified env remains the preferred SBAS runtime.
At production-safety level, the existing D-InSAR `isce2` environment should still remain untouched.