chore: initialize insar management system v2
This commit is contained in:
@@ -0,0 +1 @@
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
# Phase 0 Environment Check
|
||||
|
||||
Updated: 2026-04-03
|
||||
|
||||
## Confirmed
|
||||
|
||||
- Project workspace:
|
||||
- Windows repo root: `Z:\Code\Insar_management_system_v2`
|
||||
- WSL mount path: `/mnt/z/Code/Insar_management_system_v2`
|
||||
- Windows project Python env:
|
||||
- `C:\Users\Administrator\.conda\envs\InSAR`
|
||||
- WSL experiment distro:
|
||||
- `Ubuntu-24.04`
|
||||
- WSL project access:
|
||||
- `/mnt/z/Code/Insar_management_system_v2`
|
||||
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries`
|
||||
- WSL system Python:
|
||||
- `Python 3.12.3`
|
||||
- WSL conda root:
|
||||
- `/home/administrator/miniconda3`
|
||||
- WSL conda envs found:
|
||||
- `base`
|
||||
- `isce2`
|
||||
- ISCE2 env package:
|
||||
- `isce2 2.6.4`
|
||||
- ISCE2 Python import path:
|
||||
- `/home/administrator/miniconda3/envs/isce2/lib/python3.11/site-packages/isce/__init__.py`
|
||||
- Lutan1 sensor module:
|
||||
- `/home/administrator/miniconda3/envs/isce2/lib/python3.11/site-packages/isce/components/isceobj/Sensor/Lutan1.py`
|
||||
- Official stack directories present:
|
||||
- `/home/administrator/miniconda3/envs/isce2/share/isce2/stripmapStack`
|
||||
- `/home/administrator/miniconda3/envs/isce2/share/isce2/topsStack`
|
||||
|
||||
## Confirmed Gaps
|
||||
|
||||
- `conda` is not currently on the default shell `PATH` inside `Ubuntu-24.04`.
|
||||
- Use `/home/administrator/miniconda3/bin/conda` directly in scripts.
|
||||
- `MintPy` is not installed in the `isce2` env yet.
|
||||
- `conda list -n isce2 mintpy` returned no match.
|
||||
- Calling `conda list -n isce2 ...` from a WSL bash script triggered a segmentation fault once.
|
||||
- For experiment scripts, prefer `conda run -n isce2 python ...` checks over `conda list`.
|
||||
|
||||
## Implication
|
||||
|
||||
Phase 0 can start immediately for:
|
||||
|
||||
- LT-1 / ISCE2 stack compatibility checks
|
||||
- workspace and path validation
|
||||
- command-chain drafting
|
||||
|
||||
But the full SBAS chain cannot run end to end until one of these is true:
|
||||
|
||||
- `MintPy` is installed into `isce2`, or
|
||||
- a separate WSL env with `MintPy` is prepared
|
||||
|
||||
## Next Checks
|
||||
|
||||
1. Verify ISCE2 stack scripts actually exist in `Ubuntu-24.04`.
|
||||
2. Read `stripmapStack/README.md` and `stackStripMap.py` to identify required stack inputs.
|
||||
3. Decide whether `MintPy` should share the `isce2` env or live in a separate env.
|
||||
4. Verify LT-1 / LUTAN1 support is usable for stack-mode inputs, not only single-pair mode.
|
||||
5. Draft a minimal stack experiment command chain under this folder.
|
||||
@@ -0,0 +1,123 @@
|
||||
# Phase 0 Repo Reader Findings
|
||||
|
||||
Updated: 2026-04-03
|
||||
|
||||
## Existing repo reader path
|
||||
|
||||
The current repository already has a stable LT-1 single-scene metadata ingestion path.
|
||||
|
||||
Main code path:
|
||||
|
||||
- `backend/app/services/data_service.py`
|
||||
- `scan_radar_data()`
|
||||
- `backend/app/utils.py`
|
||||
- `parse_lt1_radar_filename()`
|
||||
- `find_xml_file()`
|
||||
- `parse_xml_metadata()`
|
||||
|
||||
## What the existing reader does
|
||||
|
||||
### 1. Folder-name parsing
|
||||
|
||||
`parse_lt1_radar_filename()` extracts from the directory name:
|
||||
|
||||
- `satellite`
|
||||
- `satellite_mode`
|
||||
- `receiving_station`
|
||||
- `imaging_mode`
|
||||
- `orbit_circle`
|
||||
- `scene_center_lon`
|
||||
- `scene_center_lat`
|
||||
- `imaging_date`
|
||||
- `acquisition_time_utc`
|
||||
- `product_type`
|
||||
- `polarization`
|
||||
- `product_level`
|
||||
- `product_unique_id`
|
||||
|
||||
Example supported name:
|
||||
|
||||
```text
|
||||
LT1B_MONO_SYC_STRIP1_018153_E135.4_N48.3_20250701_SLC_HH_S2A_0000790171
|
||||
```
|
||||
|
||||
### 2. XML discovery
|
||||
|
||||
`find_xml_file()` prefers:
|
||||
|
||||
- `*.meta.xml`
|
||||
|
||||
and falls back to:
|
||||
|
||||
- the only XML file in the directory, if there is just one
|
||||
|
||||
### 3. XML parsing
|
||||
|
||||
`parse_xml_metadata()` extracts:
|
||||
|
||||
- `orbit_direction`
|
||||
- `imaging_mode`
|
||||
- `polarization`
|
||||
- `receiving_station`
|
||||
- `satellite_mode`
|
||||
- `orbit_circle` from `absOrbit`
|
||||
- `scene_center_lon`
|
||||
- `scene_center_lat`
|
||||
- `acquisition_time_utc`
|
||||
- `product_type`
|
||||
- `product_level`
|
||||
- `product_unique_id`
|
||||
- `look_direction`
|
||||
- corner coordinates and coverage polygon
|
||||
|
||||
### 4. Merge rule
|
||||
|
||||
`scan_radar_data()` merges:
|
||||
|
||||
- folder-name metadata
|
||||
- XML metadata
|
||||
|
||||
with XML preferred for most fields, except `product_unique_id` where the folder-name value is preserved if present.
|
||||
|
||||
## Why this matters for SBAS experiments
|
||||
|
||||
This means the SBAS experiment should not invent a separate metadata interpretation unless absolutely necessary.
|
||||
|
||||
Recommended rule:
|
||||
|
||||
- reuse the same field semantics already used by `RadarDataORM`
|
||||
- reuse the same `.meta.xml` discovery logic
|
||||
- treat `scan_radar_data()` output as the canonical single-scene asset layer
|
||||
|
||||
## Data layout check against `F:\Insar_data_pool_1`
|
||||
|
||||
Sample scene directories under `F:\Insar_data_pool_1` are compatible with the current single-scene reader:
|
||||
|
||||
- one folder per scene
|
||||
- directory name matches LT-1 parser expectations
|
||||
- contains `*.meta.xml`
|
||||
- contains `*.tiff`
|
||||
- contains preview and auxiliary files
|
||||
|
||||
Example sample directory:
|
||||
|
||||
```text
|
||||
F:\Insar_data_pool_1\LT1A_MONO_KSC_STRIP1_017030_E123.3_N46.1_20250315_SLC_HH_S2A_0000678238
|
||||
```
|
||||
|
||||
Example files inside:
|
||||
|
||||
- `...meta.xml`
|
||||
- `...tiff`
|
||||
- `...rpc`
|
||||
- `...browse.jpg`
|
||||
- `...thumb.jpg`
|
||||
|
||||
## Practical implication
|
||||
|
||||
For phase 1 design and experiments:
|
||||
|
||||
- the current repo already knows how to ingest these LT-1 scene folders as `RadarData`
|
||||
- stack preparation should build on top of this asset layer
|
||||
- the real unknown is not scene metadata parsing
|
||||
- the real unknown is how to transform these scene folders into a stack layout acceptable to `stripmapStack`
|
||||
@@ -0,0 +1,92 @@
|
||||
# Phase 0 Sample Stack Selection
|
||||
|
||||
Updated: 2026-04-03
|
||||
|
||||
## Selected baseline sample
|
||||
|
||||
Current baseline sample stack:
|
||||
|
||||
- group key:
|
||||
- `LT1A|STRIP1|HH|DESCENDING|E123.3_N46.1`
|
||||
- manifest:
|
||||
- `experiments/isce2_sbas_timeseries/configs/sample_stack_e123p3_n46p1.json`
|
||||
|
||||
## Sample summary
|
||||
|
||||
- satellite:
|
||||
- `LT1A`
|
||||
- mode:
|
||||
- `STRIP1`
|
||||
- polarization:
|
||||
- `HH`
|
||||
- orbit direction:
|
||||
- `DESCENDING`
|
||||
- scene count:
|
||||
- `5`
|
||||
- dates:
|
||||
- `20250118`
|
||||
- `20250315`
|
||||
- `20250510`
|
||||
- `20250705`
|
||||
- `20250830`
|
||||
- recommended reference date:
|
||||
- `20250510`
|
||||
- receiving stations observed:
|
||||
- `SYC`
|
||||
- `KSC`
|
||||
|
||||
## Why this sample is useful
|
||||
|
||||
- It already satisfies a minimal SBAS smoke-test stack size.
|
||||
- All scenes share the same:
|
||||
- satellite
|
||||
- imaging mode
|
||||
- polarization
|
||||
- orbit direction
|
||||
- tile key
|
||||
- The dates are evenly spaced enough to act as a first time-series experiment set.
|
||||
|
||||
## Important adjacent-tile signal
|
||||
|
||||
This sample is not isolated.
|
||||
|
||||
The same date sequence also appears in multiple neighboring descending tiles, including:
|
||||
|
||||
- `E123.5_N46.6`
|
||||
- `E123.6_N47.0`
|
||||
- `E123.8_N47.5`
|
||||
- `E123.9_N48.0`
|
||||
- `E124.5_N49.9`
|
||||
- `E124.7_N50.3`
|
||||
- `E124.8_N50.8`
|
||||
- `E125.0_N51.3`
|
||||
- `E125.1_N51.8`
|
||||
- `E125.3_N52.2`
|
||||
- `E125.5_N52.7`
|
||||
|
||||
These neighboring tiles share the same 5 acquisition dates:
|
||||
|
||||
- `20250118`
|
||||
- `20250315`
|
||||
- `20250510`
|
||||
- `20250705`
|
||||
- `20250830`
|
||||
|
||||
## Implication
|
||||
|
||||
This strongly suggests the data pool contains a larger repeated strip-family, not just isolated scenes.
|
||||
|
||||
Recommended experiment order:
|
||||
|
||||
1. Start with one tile-level stack smoke test using `E123.3_N46.1`.
|
||||
2. If stack-prep works, expand to multiple adjacent tiles with the same date family.
|
||||
3. Only after that, test a wider strip or mosaic strategy.
|
||||
|
||||
## Current risk judgment
|
||||
|
||||
The main remaining uncertainty is still not scene selection.
|
||||
|
||||
The main uncertainty is:
|
||||
|
||||
- how to convert LT-1 per-scene `tiff + meta.xml` folders
|
||||
- into a stack layout and sensor input form acceptable to the ISCE2 stripmap stack workflow
|
||||
@@ -0,0 +1,140 @@
|
||||
# Phase 0 Stack Findings
|
||||
|
||||
Updated: 2026-04-03
|
||||
|
||||
## Confirmed
|
||||
|
||||
- Official stack tooling exists in the `isce2` env:
|
||||
- `/home/administrator/miniconda3/envs/isce2/share/isce2/stripmapStack`
|
||||
- `/home/administrator/miniconda3/envs/isce2/share/isce2/topsStack`
|
||||
- `stackStripMap.py` exists and is the stripmap stack entry point.
|
||||
- `prepStripmap4timeseries.py` exists in the official `stripmapStack` toolset.
|
||||
- `stackStripMap.py` expects:
|
||||
- an SLC root directory via `-s/--slc_directory`
|
||||
- a DEM via `-d/--dem`
|
||||
- an optional reference date via `-m/--reference_date`
|
||||
- temporal and baseline thresholds
|
||||
- The script scans date subdirectories under the SLC root.
|
||||
- Default behavior looks for `<date>.raw` inside each acquisition directory.
|
||||
- With `--nofocus`, it instead looks for `<date>.slc`.
|
||||
- Deeper code inspection confirms:
|
||||
- `topo.py` opens `<date>/data`
|
||||
- `geo2rdr.py` opens each secondary `<date>/data`
|
||||
- `refineSecondaryTiming` uses both `<date>.slc` and the acquisition directory as metadata roots
|
||||
|
||||
## Important implication
|
||||
|
||||
Official stack processing expects a stack-style input layout such as:
|
||||
|
||||
```text
|
||||
SLC/
|
||||
YYYYMMDD/
|
||||
YYYYMMDD.raw
|
||||
```
|
||||
|
||||
or, when data are already focused:
|
||||
|
||||
```text
|
||||
SLC/
|
||||
YYYYMMDD/
|
||||
YYYYMMDD.slc
|
||||
YYYYMMDD.slc.xml
|
||||
data
|
||||
```
|
||||
|
||||
This is different from the current repository's custom LT-1 single-pair production flow.
|
||||
|
||||
## Time-series bridge signal
|
||||
|
||||
- `prepStripmap4timeseries.py` takes:
|
||||
- pair/interferogram directories
|
||||
- baseline directory
|
||||
- geometry directory
|
||||
- shelve metadata directory
|
||||
- The script writes `.rsc` sidecars and explicitly references `pysar`-style downstream usage.
|
||||
|
||||
This is useful because it confirms the official stripmap stack toolset already contains a bridge from stack outputs toward time-series preparation.
|
||||
The weak point is still the LT-1 stack input/preparation stage, not the existence of a downstream time-series bridge.
|
||||
|
||||
## Existing repo bridge signal
|
||||
|
||||
The repository's current LT-1 single-pair pipeline already proves one important thing:
|
||||
|
||||
- `stripmapApp.py` can be driven with:
|
||||
- `sensor name = LUTAN1`
|
||||
- direct `tiff` input path
|
||||
- direct `orbitFile` XML path
|
||||
|
||||
See:
|
||||
|
||||
- `backend/app/isce2_pipeline/run_lt1_dinsar_pipeline.py`
|
||||
|
||||
The generated XML writes:
|
||||
|
||||
- `Reference -> tiff`
|
||||
- `Reference -> orbitFile`
|
||||
- `Secondary -> tiff`
|
||||
- `Secondary -> orbitFile`
|
||||
|
||||
This suggests a promising adapter direction:
|
||||
|
||||
- do not try to pretend LT-1 is ALOS or another officially prepared raw sensor
|
||||
- instead, explore generating LT-1-aware stack configs directly from:
|
||||
- scene `tiff`
|
||||
- scene `meta.xml`
|
||||
- converted orbit XML
|
||||
|
||||
That does not prove the official stack driver will accept this without modification.
|
||||
But it is the strongest current indication for how a custom LT-1 `stack-prep` layer should be shaped.
|
||||
|
||||
## LT-1 / LUTAN1 signal so far
|
||||
|
||||
- ISCE core does include a `Lutan1.py` sensor module.
|
||||
- But no direct `lutan` match was found in the `stripmapStack` helper scripts.
|
||||
- The `stripmapStack` README examples and preparation hints mention:
|
||||
- `prepRawALOS.py`
|
||||
- `prepRawSensor.py`
|
||||
- The README explicitly states automatic raw-data preparation support is currently oriented to:
|
||||
- ALOS
|
||||
- CSK
|
||||
- `prepRawSensors.py` automatic raw detection currently covers:
|
||||
- Envisat
|
||||
- ERS CEOS
|
||||
- ERS ENV
|
||||
- ALOS1
|
||||
- CSK
|
||||
- `prepSlcSensors.py` automatic SLC detection currently covers:
|
||||
- Envisat
|
||||
- ALOS1
|
||||
- CSK
|
||||
- RSAT2
|
||||
- TSX/TDX
|
||||
- No LT-1 or LUTAN1 hook was found in these official stack preparation scripts.
|
||||
|
||||
## Interim conclusion
|
||||
|
||||
Current evidence suggests:
|
||||
|
||||
- ISCE2 core can parse LT-1/LUTAN1 at the sensor level.
|
||||
- Official `stripmapStack` tooling is present.
|
||||
- But the official stack preparation helpers do not currently advertise LT-1/LUTAN1 support.
|
||||
- There is no direct evidence yet that LT-1 can be fed into the official stack helpers without an adapter step.
|
||||
- `--nofocus` does not remove the need for acquisition metadata preparation.
|
||||
- It still needs a per-date `data` shelve and an ISCE-style `.slc` image.
|
||||
|
||||
This means the working assumption should be:
|
||||
|
||||
- `LT-1 stack via official stripmapStack` is possible but unproven
|
||||
- an LT-1-specific stack preparation or conversion layer is required unless an existing hidden tool can materialize `data` + `.slc` directly for LUTAN1
|
||||
|
||||
## Next checks
|
||||
|
||||
1. Implement a dry-run LT-1 stack-prep workspace generator against the selected sample manifest.
|
||||
2. Design a scene materializer that transforms one LT-1 scene into:
|
||||
- `YYYYMMDD.slc`
|
||||
- `YYYYMMDD.slc.xml`
|
||||
- `data`
|
||||
3. Decide whether that materializer should:
|
||||
- call ISCE/LUTAN1 directly, or
|
||||
- reuse parts of the existing pair pipeline
|
||||
4. After materialization is proven, run `stackStripMap.py --nofocus` on one tile-level stack
|
||||
@@ -0,0 +1,285 @@
|
||||
# Phase 1 Stack Generation Smoke Test
|
||||
|
||||
Updated: 2026-04-05
|
||||
|
||||
Follow-up note:
|
||||
|
||||
- MintPy SBAS continuation is now recorded separately in:
|
||||
- `notes/PHASE2_MINTPY_SBAS_SMOKETEST.md`
|
||||
|
||||
## Goal
|
||||
|
||||
Validate that one LT-1 stack can be transformed from:
|
||||
|
||||
- per-scene `tiff + meta.xml + orbit.xml`
|
||||
|
||||
into:
|
||||
|
||||
- `stripmapStack --nofocus` compatible acquisition directories
|
||||
- a generated stripmap stack work plan
|
||||
|
||||
without changing production code yet.
|
||||
|
||||
## Sample
|
||||
|
||||
- group key:
|
||||
- `LT1A|STRIP1|HH|DESCENDING|E123.3_N46.1`
|
||||
- dates:
|
||||
- `20250118`
|
||||
- `20250315`
|
||||
- `20250510`
|
||||
- `20250705`
|
||||
- `20250830`
|
||||
- reference date:
|
||||
- `20250510`
|
||||
|
||||
## Commands Used
|
||||
|
||||
1. Build dry-run stack workspace:
|
||||
|
||||
```text
|
||||
C:\Users\Administrator\.conda\envs\InSAR\python.exe experiments\isce2_sbas_timeseries\scripts\build_lt1_stack_prep.py --manifest-path experiments\isce2_sbas_timeseries\configs\sample_stack_e123p3_n46p1.json
|
||||
```
|
||||
|
||||
2. Materialize LT-1 acquisitions inside `Ubuntu-24.04`:
|
||||
|
||||
```text
|
||||
wsl -d Ubuntu-24.04 /home/administrator/miniconda3/bin/conda run -n isce2 python /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/materialize_lt1_stack_scenes.py --stack-manifest /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_input_manifest.json
|
||||
```
|
||||
|
||||
3. Run generated wrapper:
|
||||
|
||||
```text
|
||||
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/run_stripmap_stack_dryrun.sh
|
||||
```
|
||||
|
||||
4. Execute the frozen stack step chain inside `Ubuntu-24.04`:
|
||||
|
||||
```text
|
||||
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_generated_stack_runfile_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1 run_01_reference
|
||||
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_generated_stack_runfile_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1 run_02_focus_split
|
||||
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_generated_stack_runfile_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1 run_03_geo2rdr_coarseResamp
|
||||
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_generated_stack_runfile_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1 run_04_refineSecondaryTiming
|
||||
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_generated_stack_runfile_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1 run_05_invertMisreg
|
||||
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_generated_stack_runfile_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1 run_06_fineResamp
|
||||
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_generated_stack_runfile_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1 run_07_grid_baseline
|
||||
```
|
||||
|
||||
5. Regenerate the stack in `interferogram` workflow mode and execute the new pair-processing stage:
|
||||
|
||||
```text
|
||||
C:\Users\Administrator\.conda\envs\InSAR\python.exe experiments\isce2_sbas_timeseries\scripts\build_lt1_stack_prep.py --manifest-path experiments\isce2_sbas_timeseries\configs\sample_stack_e123p3_n46p1.json --workflow interferogram
|
||||
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/run_stripmap_stack_dryrun.sh
|
||||
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_generated_stack_runfile_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1 run_08_igram
|
||||
```
|
||||
|
||||
6. Prepare MintPy metadata in the dedicated `mintpy` env while bridging the working `isce2` Python package:
|
||||
|
||||
```text
|
||||
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/install_mintpy_runtime_ubuntu2404.sh
|
||||
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_mintpy_with_isce_ubuntu2404.sh prep_isce.py -f "/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/Igrams/*/filt_*.unw" -m /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/Igrams/20250118_20250315/referenceShelve/data.dat -b /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/baselines -g /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/geom_reference
|
||||
```
|
||||
|
||||
## Confirmed Results
|
||||
|
||||
- All 5 acquisitions were materialized into:
|
||||
- `scratch/lt1a_strip1_hh_descending_e123p3_n46p1/SLC/YYYYMMDD/`
|
||||
- Each acquisition directory now contains:
|
||||
- `YYYYMMDD.slc`
|
||||
- `YYYYMMDD.slc.xml`
|
||||
- `YYYYMMDD.slc.vrt`
|
||||
- `data.dat/.dir/.bak`
|
||||
- Example materialized output:
|
||||
- `20250510.slc`
|
||||
- size: `3262658784` bytes
|
||||
- `stackStripMap.py --nofocus` ran successfully far enough to:
|
||||
- discover all 5 acquisitions
|
||||
- estimate stack baselines
|
||||
- select interferometric pairs
|
||||
- generate stack config files
|
||||
- generate run files
|
||||
|
||||
## Baseline Snapshot
|
||||
|
||||
Relative to reference date `20250510`, the generated stack reported:
|
||||
|
||||
- `20250118`
|
||||
- `-199.6873591838194`
|
||||
- `20250315`
|
||||
- `-89.0330513325973`
|
||||
- `20250705`
|
||||
- `-561.9056433404087`
|
||||
- `20250830`
|
||||
- `215.98283570831154`
|
||||
|
||||
The generated network reported:
|
||||
|
||||
- minimum connection degree:
|
||||
- `4.0`
|
||||
- number of pairs:
|
||||
- `10`
|
||||
|
||||
## Generated Stack Work Products
|
||||
|
||||
Under `scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/`:
|
||||
|
||||
- `baselines/`
|
||||
- `configs/`
|
||||
- `run_files/`
|
||||
- `pairs.pdf`
|
||||
|
||||
Generated run files:
|
||||
|
||||
- `run_01_reference`
|
||||
- `run_02_focus_split`
|
||||
- `run_03_geo2rdr_coarseResamp`
|
||||
- `run_04_refineSecondaryTiming`
|
||||
- `run_05_invertMisreg`
|
||||
- `run_06_fineResamp`
|
||||
- `run_07_grid_baseline`
|
||||
|
||||
## Important Runtime Fixes
|
||||
|
||||
- `matplotlib` was missing from the `isce2` env.
|
||||
- fixed by installing it with `pip` inside the WSL `isce2` environment
|
||||
- `stackStripMap.py` could not import `stripmapStack.Stack` by default.
|
||||
- fixed by exporting:
|
||||
- `PYTHONPATH=/home/administrator/miniconda3/envs/isce2/share/isce2`
|
||||
- `PATH=/home/administrator/miniconda3/envs/isce2/share/isce2/stripmapStack:$PATH`
|
||||
- the generated run files now include these prefixes automatically
|
||||
- `run_01_reference` reached `topo` successfully but `createWaterMask.py` failed without Earthdata credentials.
|
||||
- root cause:
|
||||
- `SWBD` download requires `~/.netrc` for `urs.earthdata.nasa.gov`
|
||||
- current judgment:
|
||||
- local DEM is already sufficient for this experiment
|
||||
- the only missing optional online input is the water-body mask download
|
||||
- experimental fallback:
|
||||
- `scripts/run_generated_stack_runfile_ubuntu2404.sh` now auto-generates a synthetic all-land `geom_reference/waterMask.rdr`
|
||||
- the helper script is `scripts/create_synthetic_watermask.py`
|
||||
- limitation:
|
||||
- this fallback preserves stack execution but does not provide a true coastline mask
|
||||
- LT-1 input preparation now has a shared helper:
|
||||
- `backend/app/isce2_pipeline/lt1_input_resolver.py`
|
||||
- purpose:
|
||||
- centralize DEM resolution
|
||||
- centralize orbit-pool resolution
|
||||
- centralize LT-1 precise-orbit XML reuse or generation
|
||||
- compatibility rule:
|
||||
- this is a refactor of shared input-prep logic
|
||||
- the original D-InSAR execution path was not removed
|
||||
- `run_lt1_dinsar_pipeline.py` still keeps its original public workflow entry and now calls the helper internally
|
||||
|
||||
## Latest Execution Status
|
||||
|
||||
- `run_01_reference`
|
||||
- `topo` completed successfully in `Ubuntu-24.04`
|
||||
- local `geom_reference/waterMask.rdr` was synthesized from `shadowMask.rdr`
|
||||
- `run_02_focus_split`
|
||||
- completed successfully
|
||||
- generated configs were effectively no-op under the current `--nofocus` contract
|
||||
- `run_03_geo2rdr_coarseResamp`
|
||||
- completed successfully in `Ubuntu-24.04`
|
||||
- generated `offsets/<date>/range.off` and `azimuth.off` for:
|
||||
- `20250118`
|
||||
- `20250315`
|
||||
- `20250705`
|
||||
- `20250830`
|
||||
- generated `coregSLC/Coarse/<date>/YYYYMMDD.slc` products for:
|
||||
- `20250118`
|
||||
- `20250315`
|
||||
- `20250705`
|
||||
- `20250830`
|
||||
- runtime observation:
|
||||
- this stage is long-running and mostly silent in the log file
|
||||
- progress is easier to confirm from product directories than from stdout
|
||||
- `run_04_refineSecondaryTiming`
|
||||
- completed successfully in `Ubuntu-24.04`
|
||||
- generated pair-level `refineSecondaryTiming/pairs/<pair>/misreg.*` for all 10 pairs
|
||||
- log observation:
|
||||
- `Bad match at level 1` and `correlation error` appeared in the log
|
||||
- despite that noise, the stage exited `0` and downstream inversion succeeded
|
||||
- `run_05_invertMisreg`
|
||||
- completed successfully in `Ubuntu-24.04`
|
||||
- generated date-level `refineSecondaryTiming/dates/<date>/misreg.*` for:
|
||||
- `20250118`
|
||||
- `20250315`
|
||||
- `20250510`
|
||||
- `20250705`
|
||||
- `20250830`
|
||||
- inversion observation:
|
||||
- design matrix was reported as full rank
|
||||
- RMSE in azimuth was `0.002341399255443996` pixels
|
||||
- RMSE in range was `0.0027480408593210303` pixels
|
||||
- `run_06_fineResamp`
|
||||
- completed successfully in `Ubuntu-24.04`
|
||||
- generated fine coregistered `merged/SLC/<date>/YYYYMMDD.slc` for all 5 dates
|
||||
- each merged date directory now also includes:
|
||||
- `referenceShelve/`
|
||||
- `secondaryShelve/`
|
||||
- `run_07_grid_baseline`
|
||||
- completed successfully in `Ubuntu-24.04`
|
||||
- generated `merged/baselines/<date>/` baseline grids for all 5 dates
|
||||
- each date-level baseline directory now includes:
|
||||
- raw baseline raster
|
||||
- `.xml`
|
||||
- `.vrt`
|
||||
- `.full.vrt`
|
||||
- `build_lt1_stack_prep.py --workflow interferogram`
|
||||
- now regenerates the official stripmapStack command in `interferogram` mode instead of hard-coding `slc`
|
||||
- regenerated run files now include:
|
||||
- `run_08_igram`
|
||||
- `run_08_igram`
|
||||
- completed successfully in `Ubuntu-24.04`
|
||||
- generated 10 pair directories under `stack_work/Igrams/`
|
||||
- each pair now includes:
|
||||
- wrapped interferogram `.int`
|
||||
- amplitude `.amp`
|
||||
- filtered interferogram `filt_*.int`
|
||||
- coherence `filt_*.cor`
|
||||
- unwrapped phase `filt_*_snaphu.unw`
|
||||
- connected components `*.unw.conncomp`
|
||||
- `referenceShelve/data.*`
|
||||
- MintPy runtime bootstrap
|
||||
- `scripts/install_mintpy_runtime_ubuntu2404.sh` created a dedicated WSL env:
|
||||
- `mintpy`
|
||||
- verified commands:
|
||||
- `smallbaselineApp.py`
|
||||
- `prep_isce.py`
|
||||
- installed MintPy version:
|
||||
- `1.6.2`
|
||||
- `prep_isce.py`
|
||||
- first run in the clean `mintpy` env failed because `mintpy.utils.isce_utils` imports `isce`
|
||||
- experimental resolution:
|
||||
- `scripts/run_mintpy_with_isce_ubuntu2404.sh` now bridges only the top-level `isce` package from the WSL `isce2` env into the `mintpy` env
|
||||
- result:
|
||||
- `prep_isce.py` completed successfully over the LT-1 `stripmapStack` outputs
|
||||
- geometry `.rsc` files were written under `stack_work/geom_reference/`
|
||||
- observation `.rsc` files were written for all 10 unwrapped interferograms under `stack_work/Igrams/*/`
|
||||
|
||||
## Current Boundary
|
||||
|
||||
This smoke test confirms:
|
||||
|
||||
- LT-1 scenes can be materialized into `stripmapStack` acquisition directories
|
||||
- the official stripmap stack driver can build the stack work plan over those LT-1 products
|
||||
- the generated `run_01` to `run_07` chain can complete offline in `Ubuntu-24.04` over the sample LT-1 stack
|
||||
- local DEM plus local orbit data are sufficient for this stack-preparation stage
|
||||
- Earthdata credentials are not a hard blocker for this experiment track because the wrapper can recover `run_01_reference` with a synthetic all-land `waterMask`
|
||||
- the same LT-1 stack can be regenerated in `interferogram` workflow mode to produce 10 filtered and unwrapped pair products
|
||||
- MintPy metadata preparation is now viable through the dedicated `mintpy` env plus the explicit ISCE bridge wrapper
|
||||
|
||||
This smoke test does not yet confirm:
|
||||
|
||||
- `smallbaselineApp.py` execution beyond `prep_isce.py`
|
||||
- final time-series or velocity products
|
||||
|
||||
Follow-up status:
|
||||
|
||||
- both of the above were later confirmed in:
|
||||
- `notes/PHASE2_MINTPY_SBAS_SMOKETEST.md`
|
||||
|
||||
## Next Tasks
|
||||
|
||||
1. Keep this note focused on stack-generation findings only.
|
||||
2. Use `notes/PHASE2_MINTPY_SBAS_SMOKETEST.md` for MintPy SBAS runtime conclusions.
|
||||
3. Promote the stable runtime and artifact contract into backend workflow code and `docs/`.
|
||||
@@ -0,0 +1,157 @@
|
||||
# Phase 2 Bridge Smoketest
|
||||
|
||||
Date: 2026-04-06
|
||||
|
||||
## Scope
|
||||
|
||||
Validate the current SBAS bridge chain with the previously verified LT-1 sample stack:
|
||||
|
||||
1. `build_lt1_stack_prep.py`
|
||||
2. `materialize_lt1_stack_scenes.py`
|
||||
3. `build_lt1_stack_prep.py` refresh
|
||||
|
||||
This run validates the current bridge boundary only:
|
||||
|
||||
- raw LT-1 scenes
|
||||
- local precise orbit pool
|
||||
- local prepared DEM
|
||||
- fresh scratch workspace
|
||||
|
||||
It does not run:
|
||||
|
||||
- `stripmapStack`
|
||||
- MintPy
|
||||
- geocode/export/publish
|
||||
|
||||
## Inputs
|
||||
|
||||
- sample manifest:
|
||||
- `experiments/isce2_sbas_timeseries/configs/sample_stack_e123p3_n46p1.json`
|
||||
- orbit pool:
|
||||
- `/mnt/d/orbit_pools/isce2`
|
||||
- DEM:
|
||||
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/inputs/dem/stack_dem_window.wgs84`
|
||||
|
||||
## Workspace
|
||||
|
||||
- scratch root:
|
||||
- `experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406`
|
||||
|
||||
## Result
|
||||
|
||||
- final readiness: `True`
|
||||
- scene count: `5`
|
||||
- reference date: `20250510`
|
||||
- all orbits resolved: `True`
|
||||
- all `.slc/.slc.xml` present: `True`
|
||||
- all `data` shelves present: `True`
|
||||
|
||||
## Materialization Summary
|
||||
|
||||
- dates:
|
||||
- `20250118`
|
||||
- `20250315`
|
||||
- `20250510`
|
||||
- `20250705`
|
||||
- `20250830`
|
||||
- status counts:
|
||||
- `materialized: 5`
|
||||
- total bytes written:
|
||||
- `16313641344`
|
||||
|
||||
## Artifacts
|
||||
|
||||
- selected manifest used for this WSL run:
|
||||
- `experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406/selected_stack_manifest_wsl.json`
|
||||
- generated stack manifest:
|
||||
- `experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406/stack_input_manifest.json`
|
||||
- materialization summary:
|
||||
- `experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406/materialization_summary.json`
|
||||
- generated stack dry-run wrapper:
|
||||
- `experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406/run_stripmap_stack_dryrun.sh`
|
||||
- synthetic water-mask recovery report:
|
||||
- `experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406/stack_work/logs/run_01_reference.synthetic_watermask.json`
|
||||
|
||||
## Finding
|
||||
|
||||
`build_lt1_stack_prep.py` currently reads `scene["tiff_path"]` and `scene["meta_path"]` directly.
|
||||
When the script is run inside WSL against the sample manifest, the original `F:\...` Windows paths are not readable as Linux paths.
|
||||
|
||||
For this smoketest, a temporary WSL-path manifest copy was generated and used:
|
||||
|
||||
- `selected_stack_manifest_wsl.json`
|
||||
|
||||
This is an experiment-side workaround only.
|
||||
No production/system logic was changed for this run.
|
||||
|
||||
## Update 2026-04-07
|
||||
|
||||
The same fresh workspace was then continued through the stripmap stack run files.
|
||||
|
||||
### Additional Result
|
||||
|
||||
- `run_01_reference`
|
||||
- reached `createWaterMask`
|
||||
- failed on remote `SWBD` retrieval from:
|
||||
- `https://e4ftl01.cr.usgs.gov/MEASURES/SRTMSWBD.003/...`
|
||||
- recovered with a local synthetic all-land `waterMask.rdr`
|
||||
- `run_02_focus_split`
|
||||
- completed
|
||||
- `run_03_geo2rdr_coarseResamp`
|
||||
- completed
|
||||
- `run_04_refineSecondaryTiming`
|
||||
- completed
|
||||
- `run_05_invertMisreg`
|
||||
- completed
|
||||
- `run_06_fineResamp`
|
||||
- completed
|
||||
- `run_07_grid_baseline`
|
||||
- completed
|
||||
- `run_08_igram`
|
||||
- completed through interferogram generation, filtering, coherence, and `snaphu` unwrapping
|
||||
|
||||
### Interferogram Snapshot
|
||||
|
||||
- pair count on disk:
|
||||
- `10`
|
||||
- verified pair folders:
|
||||
- `20250118_20250315`
|
||||
- `20250118_20250510`
|
||||
- `20250118_20250705`
|
||||
- `20250118_20250830`
|
||||
- `20250315_20250510`
|
||||
- `20250315_20250705`
|
||||
- `20250315_20250830`
|
||||
- `20250510_20250705`
|
||||
- `20250510_20250830`
|
||||
- `20250705_20250830`
|
||||
- verified key files in every pair directory:
|
||||
- `filt_<date12>.int`
|
||||
- `filt_<date12>.cor`
|
||||
- `filt_<date12>_snaphu.unw`
|
||||
- `filt_<date12>_snaphu.unw.conncomp`
|
||||
|
||||
### Finding Update
|
||||
|
||||
The original `run_generated_stack_runfile_ubuntu2404.sh` fallback only matched the `.netrc` credential failure text.
|
||||
This workspace showed a second offline failure mode:
|
||||
|
||||
- `createWaterMask` started normally
|
||||
- ISCE2 `DataRetriever` failed during `SWBD` file retrieval
|
||||
- the wrapper therefore did not auto-recover on the first attempt
|
||||
|
||||
The experiment helper has now been widened to recognize both:
|
||||
|
||||
- missing Earthdata credential text
|
||||
- direct `SWBD` retrieval failure text from `createWaterMask`
|
||||
|
||||
No production/system runtime was changed by this fix.
|
||||
|
||||
## Recommended Next Step
|
||||
|
||||
Use this fresh workspace to continue with:
|
||||
|
||||
1. unified-env MintPy smoketest using:
|
||||
- `configs/phase2_bridge_smoketest_20260406_smallbaseline.cfg`
|
||||
2. publish-style geocode/export if MintPy succeeds
|
||||
3. compare this fresh replay with the earlier baseline workspace
|
||||
@@ -0,0 +1,200 @@
|
||||
# Phase 2 MintPy SBAS Smoke Test
|
||||
|
||||
Updated: 2026-04-05
|
||||
|
||||
Follow-up note:
|
||||
|
||||
- publish-style geocode/export continuation is now recorded separately in:
|
||||
- `notes/PHASE3_PUBLISH_EXPORT_SMOKETEST.md`
|
||||
|
||||
## Goal
|
||||
|
||||
Validate that the LT-1 sample stack can continue from `stripmapStack` interferogram products into MintPy SBAS outputs under `Ubuntu-24.04`.
|
||||
|
||||
## Sample
|
||||
|
||||
- group key:
|
||||
- `LT1A|STRIP1|HH|DESCENDING|E123.3_N46.1`
|
||||
- dates:
|
||||
- `20250118`
|
||||
- `20250315`
|
||||
- `20250510`
|
||||
- `20250705`
|
||||
- `20250830`
|
||||
- pair count:
|
||||
- `10`
|
||||
- reference date:
|
||||
- `20250510`
|
||||
- reference point:
|
||||
- `y/x = 1994,52`
|
||||
|
||||
## Successful Run
|
||||
|
||||
Successful SBAS smoke-test work directory:
|
||||
|
||||
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/mintpy_sbas_v5`
|
||||
|
||||
Successful WSL command:
|
||||
|
||||
```text
|
||||
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_mintpy_sbas_smoketest_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/configs/sample_smallbaseline_lt1_e123p3_n46p1.cfg /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/mintpy_sbas_v5
|
||||
```
|
||||
|
||||
This run completed through:
|
||||
|
||||
- `load_data`
|
||||
- `modify_network`
|
||||
- `reference_point`
|
||||
- `quick_overview`
|
||||
- `invert_network`
|
||||
- `reference_date`
|
||||
- `velocity`
|
||||
|
||||
Disabled for this first offline smoke test:
|
||||
|
||||
- unwrap-error correction
|
||||
- solid-earth-tide correction
|
||||
- ionosphere correction
|
||||
- troposphere correction
|
||||
- deramp
|
||||
- topographic residual correction
|
||||
- geocode
|
||||
|
||||
## Output Snapshot
|
||||
|
||||
Generated under `mintpy_sbas_v5/`:
|
||||
|
||||
- `timeseries.h5`
|
||||
- size: `82576752` bytes
|
||||
- `velocity.h5`
|
||||
- size: `83086408` bytes
|
||||
- `temporalCoherence.h5`
|
||||
- size: `16632424` bytes
|
||||
- `maskTempCoh.h5`
|
||||
- size: `4136792` bytes
|
||||
- `avgSpatialCoh.h5`
|
||||
- size: `16631784` bytes
|
||||
- `numTriNonzeroIntAmbiguity.h5`
|
||||
- size: `16632000` bytes
|
||||
- `numTriNonzeroIntAmbiguity.png`
|
||||
- size: `232878` bytes
|
||||
|
||||
Quality summary from the successful run:
|
||||
|
||||
- strict valid pixels for inversion:
|
||||
- `1219001 / 4076199`
|
||||
- `29.91%`
|
||||
- reliable pixels in `maskTempCoh.h5` with threshold `0.7`:
|
||||
- `62987`
|
||||
|
||||
## Required Runtime Decisions
|
||||
|
||||
### 1. Keep MintPy separate from ISCE2
|
||||
|
||||
- keep stack processing in WSL conda env:
|
||||
- `isce2`
|
||||
- keep MintPy in WSL conda env:
|
||||
- `mintpy`
|
||||
|
||||
Reason:
|
||||
|
||||
- avoids mutating the already working ISCE2 processing env on the development machine
|
||||
|
||||
### 2. Bridge only the top-level `isce` package
|
||||
|
||||
Required helper:
|
||||
|
||||
- `scripts/run_mintpy_with_isce_ubuntu2404.sh`
|
||||
|
||||
Current rule:
|
||||
|
||||
- do not add the entire `isce2` `site-packages` into `PYTHONPATH`
|
||||
- only bridge the top-level `isce` package into a cache directory
|
||||
|
||||
Reason:
|
||||
|
||||
- adding the whole `site-packages` caused MintPy to import `h5py` from the wrong env and fail during `load_data`
|
||||
|
||||
### 3. Do not load `wrapPhase` for this LT-1 smoke test
|
||||
|
||||
Current config rule:
|
||||
|
||||
- `mintpy.load.intFile = None`
|
||||
|
||||
Reason:
|
||||
|
||||
- loading `filt_*.int` into MintPy `wrapPhase` caused HDF5 type-conversion failure during `load_data`
|
||||
|
||||
### 4. Build a strict `maskAllValid.h5` before inversion
|
||||
|
||||
Required helper:
|
||||
|
||||
- `scripts/create_mintpy_all_ifgram_mask.py`
|
||||
|
||||
Current rule:
|
||||
|
||||
- keep only pixels that are finite and non-zero in all unwrapped interferograms
|
||||
- also require non-zero connected components in all interferograms
|
||||
|
||||
Reason:
|
||||
|
||||
- this reduces unstable partial-network pixels before SBAS inversion
|
||||
|
||||
### 5. Use the repo-local patched launcher
|
||||
|
||||
Required helper:
|
||||
|
||||
- `scripts/run_smallbaselineApp_patched.py`
|
||||
|
||||
Current workaround:
|
||||
|
||||
- patch `mintpy.ifgram_inversion.estimate_timeseries()` at runtime
|
||||
- coerce shape-`(1,)` inversion-quality output into a scalar for the single-pixel partial-network branch
|
||||
|
||||
Reason:
|
||||
|
||||
- MintPy `1.6.2` hit a `ValueError: setting an array element with a sequence`
|
||||
- failure point:
|
||||
- `mintpy/ifgram_inversion.py`
|
||||
- partial-network pixel branch inside `run_ifgram_inversion_patch()`
|
||||
|
||||
Current judgment:
|
||||
|
||||
- this is a MintPy runtime issue in the current environment
|
||||
- it is better to keep the workaround in repo-local launcher code than silently editing the third-party env
|
||||
|
||||
## Current Boundary
|
||||
|
||||
This smoke test now confirms:
|
||||
|
||||
- LT-1 `stripmapStack` outputs can be loaded by MintPy in the dedicated `mintpy` env
|
||||
- the LT-1 sample stack can be inverted into radar-coordinate `timeseries.h5`
|
||||
- the LT-1 sample stack can generate radar-coordinate `velocity.h5`
|
||||
- the current repo-local workaround chain is reproducible in `Ubuntu-24.04`
|
||||
|
||||
This smoke test does not yet confirm:
|
||||
|
||||
- geocoded SBAS exports
|
||||
- atmospheric or DEM-residual correction quality
|
||||
- product publishing into backend `psinsar` catalog
|
||||
- frontend rendering of published SBAS products
|
||||
|
||||
## System Embedding Implications
|
||||
|
||||
The current experiment suggests the future backend runtime contract should be:
|
||||
|
||||
1. Run ISCE2 stack workflow in WSL `isce2`.
|
||||
2. Run MintPy through the repo-controlled bridge runner instead of calling upstream `smallbaselineApp.py` directly.
|
||||
3. Generate a strict inversion mask after `load_data`.
|
||||
4. Persist the following files as first-class workflow artifacts:
|
||||
- `timeseries.h5`
|
||||
- `velocity.h5`
|
||||
- `temporalCoherence.h5`
|
||||
- `maskTempCoh.h5`
|
||||
- `numTriNonzeroIntAmbiguity.h5`
|
||||
- `numTriNonzeroIntAmbiguity.png`
|
||||
5. Convert these into a stable publish manifest before catalog registration.
|
||||
|
||||
Current sample publish-manifest draft:
|
||||
|
||||
- `configs/sample_psinsar_manifest_lt1_e123p3_n46p1.json`
|
||||
@@ -0,0 +1,142 @@
|
||||
# Phase 3 Publish Export Smoke Test
|
||||
|
||||
Updated: 2026-04-06
|
||||
|
||||
## Goal
|
||||
|
||||
Validate that the successful MintPy SBAS experiment can be converted into a publish-style artifact bundle without touching the main system.
|
||||
|
||||
## Inputs
|
||||
|
||||
Source MintPy work directory:
|
||||
|
||||
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/mintpy_sbas_v5`
|
||||
|
||||
Source stack:
|
||||
|
||||
- group key:
|
||||
- `LT1A|STRIP1|HH|DESCENDING|E123.3_N46.1`
|
||||
- dates:
|
||||
- `20250118`
|
||||
- `20250315`
|
||||
- `20250510`
|
||||
- `20250705`
|
||||
- `20250830`
|
||||
|
||||
## Successful Command
|
||||
|
||||
```text
|
||||
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/export_mintpy_publish_products_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/mintpy_sbas_v5 /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/publish/mintpy_sbas_v5
|
||||
```
|
||||
|
||||
## Current Export Scripts
|
||||
|
||||
- `scripts/export_mintpy_publish_products_ubuntu2404.sh`
|
||||
- geocode core MintPy outputs
|
||||
- convert selected outputs to GeoTIFF
|
||||
- copy preview and metadata files
|
||||
- build root `manifest.json`
|
||||
- `scripts/build_mintpy_publish_bundle.py`
|
||||
- generate `preview/velocity_preview.png`
|
||||
- generate `metadata/source_quality_summary.json`
|
||||
- generate publish-style root `manifest.json`
|
||||
|
||||
## Geocode Contract
|
||||
|
||||
Current experiment settings:
|
||||
|
||||
- lookup source:
|
||||
- `inputs/geometryRadar.h5`
|
||||
- output pixel size:
|
||||
- latitude step:
|
||||
- `-0.000185185`
|
||||
- longitude step:
|
||||
- `0.000185185`
|
||||
- interpolation:
|
||||
- `nearest`
|
||||
|
||||
Observed geocoded grid:
|
||||
|
||||
- extent:
|
||||
- south:
|
||||
- `45.80391`
|
||||
- north:
|
||||
- `46.42206`
|
||||
- west:
|
||||
- `122.930244`
|
||||
- east:
|
||||
- `123.76654`
|
||||
- shape:
|
||||
- rows:
|
||||
- `3338`
|
||||
- columns:
|
||||
- `4516`
|
||||
|
||||
## Generated Publish Bundle
|
||||
|
||||
Successful publish-style directory:
|
||||
|
||||
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/publish/mintpy_sbas_v5`
|
||||
|
||||
Main outputs:
|
||||
|
||||
- `manifest.json`
|
||||
- `assets/geo_timeseries.h5`
|
||||
- `assets/geo_velocity.h5`
|
||||
- `assets/geo_temporalCoherence.h5`
|
||||
- `assets/geo_maskTempCoh.h5`
|
||||
- `assets/velocity.tif`
|
||||
- `assets/temporalCoherence.tif`
|
||||
- `assets/maskTempCoh.tif`
|
||||
- `preview/velocity_preview.png`
|
||||
- `preview/numTriNonzeroIntAmbiguity.png`
|
||||
- `metadata/smallbaselineApp.cfg`
|
||||
- `metadata/source_quality_summary.json`
|
||||
|
||||
## Current Interpretation
|
||||
|
||||
This confirms:
|
||||
|
||||
- the experiment now supports radar-coordinate MintPy inversion
|
||||
- the experiment also supports geocoded HDF5 exports
|
||||
- the experiment can produce publish-style GeoTIFF outputs
|
||||
- the experiment can build a stable manifest-driven bundle outside the MintPy work directory
|
||||
|
||||
This does not yet confirm:
|
||||
|
||||
- direct backend catalog registration
|
||||
- frontend rendering against the real system APIs
|
||||
- whether `EPSG:4326` should remain the final publish CRS decision
|
||||
|
||||
## Important Notes
|
||||
|
||||
### 1. Current CRS assumption
|
||||
|
||||
`save_gdal.py` warned that no explicit `EPSG` metadata was found and assumed:
|
||||
|
||||
- `EPSG:4326`
|
||||
|
||||
Current judgment:
|
||||
|
||||
- acceptable for this experiment because the geocoded outputs are in latitude/longitude grids
|
||||
- should still be checked when formalizing the production publish contract
|
||||
|
||||
### 2. Group key in the generated manifest
|
||||
|
||||
Because PowerShell treats `|` specially, passing group keys on the command line is awkward from Windows.
|
||||
|
||||
Current practical rule:
|
||||
|
||||
- the sample manifest stored in git remains the clean reference:
|
||||
- `configs/sample_psinsar_manifest_lt1_e123p3_n46p1.json`
|
||||
- the generated publish bundle manifest can be post-filled or generated from backend metadata later
|
||||
|
||||
## System Embedding Implication
|
||||
|
||||
At this point the experiment-layer chain is split cleanly into three stages:
|
||||
|
||||
1. `stripmapStack` preprocessing
|
||||
2. MintPy SBAS inversion
|
||||
3. geocode + publish-bundle export
|
||||
|
||||
That means the future system workflow can wire them as separate workflow steps without changing the validated experiment logic first.
|
||||
@@ -0,0 +1,120 @@
|
||||
# Phase 4 Unified-Environment Decision
|
||||
|
||||
Updated: 2026-04-06
|
||||
|
||||
## Decision
|
||||
|
||||
Current experiment preference:
|
||||
|
||||
- prefer the unified WSL environment for SBAS experiment work
|
||||
|
||||
Current production safety rule:
|
||||
|
||||
- do not replace or mutate the existing D-InSAR production environment
|
||||
- keep pair-oriented D-InSAR on the existing WSL `isce2` env
|
||||
- keep the current backend/public D-InSAR entry unchanged
|
||||
|
||||
Current environment split:
|
||||
|
||||
- D-InSAR production baseline:
|
||||
- `isce2`
|
||||
- SBAS experiment preferred runtime:
|
||||
- `isce2_mintpy_v1`
|
||||
|
||||
## Why This Decision Is Reasonable
|
||||
|
||||
The unified env has now completed the full current experiment chain:
|
||||
|
||||
- MintPy command invocation without the `isce` bridge
|
||||
- `load_data`
|
||||
- strict-mask generation
|
||||
- `modify_network -> velocity`
|
||||
- publish export to geocoded HDF5, GeoTIFF, preview, and `manifest.json`
|
||||
|
||||
This makes the unified env a valid experiment baseline.
|
||||
|
||||
At the same time, the existing pair-oriented D-InSAR route is already working and should not be destabilized just to simplify the SBAS experiment runtime.
|
||||
|
||||
The safest rule is therefore:
|
||||
|
||||
- let SBAS experiments move forward in a separate unified env
|
||||
- do not touch the current `isce2` production env used by D-InSAR
|
||||
|
||||
## Evidence
|
||||
|
||||
Successful unified env:
|
||||
|
||||
- `/home/administrator/miniconda3/envs/isce2_mintpy_v1`
|
||||
|
||||
Successful unified SBAS work directory:
|
||||
|
||||
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/mintpy_sbas_unified_v1`
|
||||
|
||||
Successful unified publish directory:
|
||||
|
||||
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/publish/mintpy_sbas_unified_v1`
|
||||
|
||||
Matched quality indicators:
|
||||
|
||||
- `maskAllValid`:
|
||||
- `1219001 / 4076199`
|
||||
- `29.91%`
|
||||
- `maskTempCoh`:
|
||||
- `62987 / 4076199`
|
||||
- `1.55%`
|
||||
|
||||
Key package difference:
|
||||
|
||||
- current `isce2` env does not provide the MintPy-side package set needed for this SBAS route
|
||||
- current `isce2_mintpy_v1` env includes:
|
||||
- `mintpy`
|
||||
- `cartopy`
|
||||
- `pyaps3`
|
||||
- `pykml`
|
||||
- `cvxopt`
|
||||
|
||||
## What This Decision Does Not Mean
|
||||
|
||||
It does not mean:
|
||||
|
||||
- the backend should immediately switch to unified-env execution
|
||||
- the bridge route must be deleted now
|
||||
- the current D-InSAR runtime should be modified
|
||||
|
||||
It only means:
|
||||
|
||||
- for the next experiment steps, unified env is the preferred path
|
||||
- for current production safety, `isce2` remains untouched
|
||||
|
||||
## Required Guardrails
|
||||
|
||||
For the next phase, keep these rules:
|
||||
|
||||
- do not install MintPy into the existing `isce2` env
|
||||
- do not redirect current D-InSAR scripts to `isce2_mintpy_v1`
|
||||
- do not remove the bridge-based helpers yet
|
||||
- keep all SBAS work in experiment scripts, notes, and scratch directories
|
||||
|
||||
## Reproducibility
|
||||
|
||||
Environment snapshots should be exported and kept with the experiment record.
|
||||
|
||||
Current snapshot command:
|
||||
|
||||
```text
|
||||
wsl -d Ubuntu-24.04 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/export_phase4_env_snapshots_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/configs/env_snapshots/20260406
|
||||
```
|
||||
|
||||
## Next Stable Experiment Steps
|
||||
|
||||
Before any system integration work:
|
||||
|
||||
1. Keep the unified env as the default SBAS experiment runtime.
|
||||
2. Preserve environment snapshots for both `isce2` and `isce2_mintpy_v1`.
|
||||
3. Write one comparison note focused on:
|
||||
- runtime simplicity
|
||||
- reproducibility
|
||||
- remaining workarounds
|
||||
- risk to D-InSAR production
|
||||
4. Optionally repeat the chain on one more LT-1 sample stack.
|
||||
5. Only after the experiment is stable, design the system integration boundary.
|
||||
@@ -0,0 +1,221 @@
|
||||
# Phase 4 Unified-Environment Experiment
|
||||
|
||||
Updated: 2026-04-06
|
||||
|
||||
## Goal
|
||||
|
||||
Validate whether the current WSL `isce2` runtime can be cloned and extended with MintPy so that the SBAS experiment can run without the temporary `isce` bridge helper.
|
||||
|
||||
This is still an experiment-layer task.
|
||||
|
||||
Do not change the production backend yet.
|
||||
|
||||
## Why run this phase
|
||||
|
||||
The bridge-based route is already validated, but a unified environment may be cleaner because:
|
||||
|
||||
- many ISCE2 + MintPy users operate in one environment
|
||||
- command invocation becomes simpler
|
||||
- future worker deployment may be easier if one runtime is stable
|
||||
|
||||
The bridge-based route still remains the fallback baseline until this phase is verified.
|
||||
|
||||
## Current Known Starting Point
|
||||
|
||||
WSL distro:
|
||||
|
||||
- `Ubuntu-24.04`
|
||||
|
||||
Current environments observed on 2026-04-06:
|
||||
|
||||
- `isce2`
|
||||
- `mintpy`
|
||||
|
||||
Observed package state:
|
||||
|
||||
- `isce2` env:
|
||||
- `isce2 2.6.4`
|
||||
- `h5py 3.15.1`
|
||||
- `mintpy` not installed
|
||||
- dedicated `mintpy` env:
|
||||
- `mintpy 1.6.3`
|
||||
|
||||
## Initial Hypothesis
|
||||
|
||||
Expected best-case outcome:
|
||||
|
||||
- clone `isce2` into `isce2_mintpy`
|
||||
- install `mintpy` directly into the clone
|
||||
- reuse the same repo-local strict-mask and patched-launcher helpers
|
||||
- run the same LT-1 smoke test without the `isce` bridge wrapper
|
||||
|
||||
Main risk areas:
|
||||
|
||||
- package solver may replace or downgrade key ISCE2-side numeric dependencies
|
||||
- MintPy may still require the same repo-local runtime workaround even in a unified env
|
||||
- GDAL / h5py / pyaps3 dependency changes may alter the known-good stack behavior
|
||||
|
||||
## Reproducible Commands
|
||||
|
||||
### 1. Bootstrap the unified env
|
||||
|
||||
```text
|
||||
wsl -d Ubuntu-24.04 env TARGET_ENV=isce2_mintpy_v1 BOOTSTRAP_MODE=recreate USE_TUNA_MIRROR=1 MINTPY_SPEC=mintpy=1.6.3 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/install_mintpy_into_cloned_isce2_env_ubuntu2404.sh
|
||||
```
|
||||
|
||||
Why `BOOTSTRAP_MODE=recreate`:
|
||||
|
||||
- direct `conda create --clone` was not stable enough for this machine
|
||||
- it still followed source-package URLs and hit channel/TOS friction
|
||||
- the successful path was:
|
||||
- export the current `isce2` dependency list
|
||||
- recreate the env through Tsinghua mirror channels
|
||||
- reinstall exported pip packages
|
||||
- install MintPy into the recreated env
|
||||
|
||||
Optional environment override:
|
||||
|
||||
```text
|
||||
TARGET_ENV=isce2_mintpy_v1 MINTPY_SPEC='mintpy=1.6.3'
|
||||
```
|
||||
|
||||
### 2. Run MintPy commands directly inside the unified env
|
||||
|
||||
```text
|
||||
wsl -d Ubuntu-24.04 env MINTPY_ENV=isce2_mintpy_v1 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_mintpy_unified_env_ubuntu2404.sh prep_isce.py -h
|
||||
```
|
||||
|
||||
### 3. Re-run the current LT-1 SBAS smoke test in the unified env
|
||||
|
||||
```text
|
||||
wsl -d Ubuntu-24.04 env MINTPY_ENV=isce2_mintpy_v1 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_mintpy_sbas_unified_env_smoketest_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/configs/sample_smallbaseline_lt1_e123p3_n46p1.cfg /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/mintpy_sbas_unified_v1
|
||||
```
|
||||
|
||||
### 4. Export publish bundle in the unified env
|
||||
|
||||
```text
|
||||
wsl -d Ubuntu-24.04 env MINTPY_ENV=isce2_mintpy_v1 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/export_mintpy_publish_products_unified_env_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/mintpy_sbas_unified_v1 /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/publish/mintpy_sbas_unified_v1
|
||||
```
|
||||
|
||||
## Comparison Checklist
|
||||
|
||||
When this phase is executed, compare it against the bridge route on:
|
||||
|
||||
- package versions after install
|
||||
- whether `prep_isce.py` imports cleanly without bridge
|
||||
- whether `load_data` succeeds
|
||||
- whether the strict-mask step is still required
|
||||
- whether the patched launcher is still required
|
||||
- whether output files match the existing bridge-based artifact set
|
||||
- whether geocode/export still succeeds from the unified env
|
||||
|
||||
## Current Status
|
||||
|
||||
Successful environment:
|
||||
|
||||
- `/home/administrator/miniconda3/envs/isce2_mintpy_v1`
|
||||
|
||||
Observed package/version state in the successful unified env:
|
||||
|
||||
- `conda list` shows:
|
||||
- `mintpy 1.6.3`
|
||||
- runtime `mintpy.__version__` reports:
|
||||
- `1.6.2`
|
||||
- `isce` import path:
|
||||
- `/home/administrator/miniconda3/envs/isce2_mintpy_v1/lib/python3.11/site-packages/isce/__init__.py`
|
||||
- `mintpy` import path:
|
||||
- `/home/administrator/miniconda3/envs/isce2_mintpy_v1/lib/python3.11/site-packages/mintpy/__init__.py`
|
||||
- `h5py`:
|
||||
- `3.15.1`
|
||||
|
||||
Successful unified-env SBAS work directory:
|
||||
|
||||
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/stack_work/mintpy_sbas_unified_v1`
|
||||
|
||||
Successful unified-env publish directory:
|
||||
|
||||
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/lt1a_strip1_hh_descending_e123p3_n46p1/publish/mintpy_sbas_unified_v1`
|
||||
|
||||
Validated in the unified env:
|
||||
|
||||
- `prep_isce.py -h` works without the `isce` bridge
|
||||
- `load_data` succeeded
|
||||
- strict-mask generation succeeded
|
||||
- `modify_network -> velocity` succeeded
|
||||
- publish export succeeded through geocoded HDF5, GeoTIFF, preview, and `manifest.json`
|
||||
|
||||
Observed unified-env output set under `mintpy_sbas_unified_v1/`:
|
||||
|
||||
- `timeseries.h5`
|
||||
- `velocity.h5`
|
||||
- `temporalCoherence.h5`
|
||||
- `maskTempCoh.h5`
|
||||
- `maskAllValid.h5`
|
||||
- `avgSpatialCoh.h5`
|
||||
- `numTriNonzeroIntAmbiguity.h5`
|
||||
- `numTriNonzeroIntAmbiguity.png`
|
||||
|
||||
Observed publish bundle under `publish/mintpy_sbas_unified_v1/`:
|
||||
|
||||
- `manifest.json`
|
||||
- `assets/geo_timeseries.h5`
|
||||
- `assets/geo_velocity.h5`
|
||||
- `assets/geo_temporalCoherence.h5`
|
||||
- `assets/geo_maskTempCoh.h5`
|
||||
- `assets/velocity.tif`
|
||||
- `assets/temporalCoherence.tif`
|
||||
- `assets/maskTempCoh.tif`
|
||||
- `preview/velocity_preview.png`
|
||||
- `preview/numTriNonzeroIntAmbiguity.png`
|
||||
- `metadata/smallbaselineApp.cfg`
|
||||
- `metadata/source_quality_summary.json`
|
||||
|
||||
Quality summary matched the bridge-based route:
|
||||
|
||||
- `maskAllValid`:
|
||||
- `1219001 / 4076199`
|
||||
- `29.91%`
|
||||
- `maskTempCoh`:
|
||||
- `62987 / 4076199`
|
||||
- `1.55%`
|
||||
|
||||
Still required in the unified env:
|
||||
|
||||
- strict `maskAllValid.h5` before inversion
|
||||
- repo-local patched `smallbaselineApp` launcher
|
||||
|
||||
Prepared:
|
||||
|
||||
- unified-env bootstrap script
|
||||
- unified-env MintPy runner
|
||||
- unified-env SBAS smoke-test runner
|
||||
- unified-env publish-export wrapper
|
||||
|
||||
Completed:
|
||||
|
||||
- actual clone + install execution
|
||||
- actual smoke-test result capture
|
||||
- actual publish-export capture
|
||||
|
||||
Pending:
|
||||
|
||||
- deeper comparison of output metadata against the bridge route
|
||||
- decide whether unified env or bridge env should be the default production candidate
|
||||
- decide whether to pin the runtime to the conda package label `1.6.3` or the internal MintPy version string `1.6.2`
|
||||
|
||||
## Current Judgment
|
||||
|
||||
At experiment level, the unified environment is now viable.
|
||||
|
||||
This phase confirms:
|
||||
|
||||
- the current LT-1 SBAS route does not fundamentally require the `isce` bridge
|
||||
- a recreated `isce2 + mintpy` WSL env can complete:
|
||||
- MintPy load/inversion
|
||||
- geocode/export
|
||||
- publish-bundle generation
|
||||
|
||||
Current recommendation:
|
||||
|
||||
- keep the bridge route as the already-known baseline until a fuller diff is written
|
||||
- but treat the unified env as a valid candidate for the future production runtime
|
||||
@@ -0,0 +1,193 @@
|
||||
# Phase 4 Unified-Env Replay On Fresh Phase-2 Workspace
|
||||
|
||||
Updated: 2026-04-08
|
||||
|
||||
## Goal
|
||||
|
||||
Re-run the already validated unified-env SBAS route on the fresh workspace:
|
||||
|
||||
- `scratch/phase2_bridge_smoketest_20260406`
|
||||
|
||||
This checks that the current experiment does not rely on the older sample workspace only.
|
||||
|
||||
## Inputs
|
||||
|
||||
- WSL distro:
|
||||
- `Ubuntu-24.04`
|
||||
- unified env:
|
||||
- `/home/administrator/miniconda3/envs/isce2_mintpy_v1`
|
||||
- stack config:
|
||||
- `configs/phase2_bridge_smoketest_20260406_smallbaseline.cfg`
|
||||
- stack workspace:
|
||||
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406`
|
||||
- MintPy work dir:
|
||||
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406/stack_work/mintpy_sbas_unified_phase2_20260407`
|
||||
- publish dir:
|
||||
- `/mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406/publish/mintpy_sbas_unified_phase2_20260407`
|
||||
|
||||
## Successful Commands
|
||||
|
||||
Unified-env MintPy smoke test:
|
||||
|
||||
```text
|
||||
wsl -d Ubuntu-24.04 env MINTPY_ENV=isce2_mintpy_v1 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/run_mintpy_sbas_unified_env_smoketest_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/configs/phase2_bridge_smoketest_20260406_smallbaseline.cfg /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406/stack_work/mintpy_sbas_unified_phase2_20260407
|
||||
```
|
||||
|
||||
Unified-env publish export:
|
||||
|
||||
```text
|
||||
wsl -d Ubuntu-24.04 env MINTPY_ENV=isce2_mintpy_v1 bash /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scripts/export_mintpy_publish_products_unified_env_ubuntu2404.sh /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406/stack_work/mintpy_sbas_unified_phase2_20260407 /mnt/z/Code/Insar_management_system_v2/experiments/isce2_sbas_timeseries/scratch/phase2_bridge_smoketest_20260406/publish/mintpy_sbas_unified_phase2_20260407
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
The fresh workspace replay succeeded through:
|
||||
|
||||
- `load_data`
|
||||
- strict `maskAllValid.h5` generation
|
||||
- `modify_network`
|
||||
- `reference_point`
|
||||
- `quick_overview`
|
||||
- `invert_network`
|
||||
- `reference_date`
|
||||
- `velocity`
|
||||
- geocode/export
|
||||
- publish-bundle generation
|
||||
|
||||
This confirms the current fresh workspace now reaches the same experiment boundary as the earlier baseline sample:
|
||||
|
||||
- radar-coordinate MintPy runtime products
|
||||
- geocoded publish bundle
|
||||
|
||||
## Runtime Output Snapshot
|
||||
|
||||
Generated under `stack_work/mintpy_sbas_unified_phase2_20260407/`:
|
||||
|
||||
- `timeseries.h5`
|
||||
- `82579888` bytes
|
||||
- `velocity.h5`
|
||||
- `83086408` bytes
|
||||
- `temporalCoherence.h5`
|
||||
- `16632424` bytes
|
||||
- `maskTempCoh.h5`
|
||||
- `4136848` bytes
|
||||
- `maskAllValid.h5`
|
||||
- `4095215` bytes
|
||||
- `avgSpatialCoh.h5`
|
||||
- `16631784` bytes
|
||||
- `numTriNonzeroIntAmbiguity.h5`
|
||||
- `16632000` bytes
|
||||
- `numTriNonzeroIntAmbiguity.png`
|
||||
- `232923` bytes
|
||||
|
||||
## Publish Bundle Snapshot
|
||||
|
||||
Generated under `publish/mintpy_sbas_unified_phase2_20260407/`:
|
||||
|
||||
- `manifest.json`
|
||||
- `32442` bytes
|
||||
- `assets/geo_timeseries.h5`
|
||||
- `304443360` bytes
|
||||
- `assets/geo_velocity.h5`
|
||||
- `305627744` bytes
|
||||
- `assets/geo_temporalCoherence.h5`
|
||||
- `61141152` bytes
|
||||
- `assets/geo_maskTempCoh.h5`
|
||||
- `15260536` bytes
|
||||
- `assets/velocity.tif`
|
||||
- `60318026` bytes
|
||||
- `assets/temporalCoherence.tif`
|
||||
- `60318026` bytes
|
||||
- `assets/maskTempCoh.tif`
|
||||
- `15094802` bytes
|
||||
- `preview/velocity_preview.png`
|
||||
- `813928` bytes
|
||||
- `preview/numTriNonzeroIntAmbiguity.png`
|
||||
- `232923` bytes
|
||||
- `metadata/smallbaselineApp.cfg`
|
||||
- `26419` bytes
|
||||
- `metadata/source_quality_summary.json`
|
||||
- `403` bytes
|
||||
|
||||
## Quality Summary
|
||||
|
||||
From `metadata/source_quality_summary.json`:
|
||||
|
||||
- `maskAllValid`
|
||||
- `1219067 / 4076199`
|
||||
- `29.91%`
|
||||
- `maskTempCoh`
|
||||
- `63092 / 4076199`
|
||||
- `1.55%`
|
||||
- preview stretch:
|
||||
- `vmin = -0.2251889556646347`
|
||||
- `vmax = 0.2251889556646347`
|
||||
|
||||
## Findings
|
||||
|
||||
### 1. The fresh workspace replay is reproducible
|
||||
|
||||
The new workspace produced:
|
||||
|
||||
- the full 10-pair interferogram stack
|
||||
- MintPy `timeseries.h5`
|
||||
- MintPy `velocity.h5`
|
||||
- publish-layer geocoded HDF5 / GeoTIFF / preview / `manifest.json`
|
||||
|
||||
This is the current strongest experiment proof that the SBAS route is not tied to the earlier historical scratch directory.
|
||||
|
||||
### 2. The same two MintPy experiment helpers are still required
|
||||
|
||||
The unified env still depends on:
|
||||
|
||||
- `create_mintpy_all_ifgram_mask.py`
|
||||
- `run_smallbaselineApp_patched.py`
|
||||
|
||||
Current interpretation:
|
||||
|
||||
- unified env removes the temporary `isce` bridge
|
||||
- it does not remove the current strict-mask or patched-launcher workarounds
|
||||
|
||||
### 3. Offline water-mask strategy remains valid
|
||||
|
||||
This replay consumed the synthetic all-land `waterMask.rdr` created earlier in the fresh workspace.
|
||||
|
||||
No Earthdata / `SWBD` download was needed for the MintPy or publish stages.
|
||||
|
||||
### 4. Geocode/export completed with a tolerable warning
|
||||
|
||||
`save_gdal.py` warned that no EPSG / UTM metadata was found and assumed:
|
||||
|
||||
- `EPSG:4326`
|
||||
|
||||
For this experiment chain, that is acceptable because the export was already driven by latitude / longitude lookup plus explicit `--lalo` sampling.
|
||||
|
||||
This warning should still be recorded for later production hardening.
|
||||
|
||||
### 5. `group_key` should come from system metadata
|
||||
|
||||
This host-side replay exported a valid publish bundle, and the bundle should carry:
|
||||
|
||||
- `"group_key": "LT1A|STRIP1|HH|DESCENDING|E123.3_N46.1"`
|
||||
|
||||
Reason:
|
||||
|
||||
- the bundle builder accepts `group_key` as a CLI argument
|
||||
- the LT-1 group key contains pipe characters:
|
||||
- `LT1A|STRIP1|HH|DESCENDING|E123.3_N46.1`
|
||||
- when invoked naively from Windows PowerShell into WSL, the value may be truncated or split by the host shell
|
||||
|
||||
Current judgment:
|
||||
|
||||
- this is not a blocker for the SBAS scientific chain
|
||||
- future system embedding should write `group_key` from task/run metadata inside backend code instead of relying on manual shell arguments
|
||||
|
||||
## Current Judgment
|
||||
|
||||
The current fresh workspace now confirms the full experiment chain:
|
||||
|
||||
- `raw LT-1 -> stripmapStack -> MintPy SBAS -> geocode/export -> publish bundle`
|
||||
|
||||
At experiment level, the unified env remains the preferred SBAS runtime.
|
||||
|
||||
At production-safety level, the existing D-InSAR `isce2` environment should still remain untouched.
|
||||
Reference in New Issue
Block a user