185 lines
7.1 KiB
Markdown
185 lines
7.1 KiB
Markdown
# ISCE2 Stabilization Update Log
|
|
|
|
Date: `2026-04-27`
|
|
|
|
## Scope
|
|
|
|
This update hardens the managed `ISCE2` LT-1 stripmap D-InSAR production path against
|
|
large raw DEM reuse, long geocode stalls, and incomplete recovered-run metadata.
|
|
|
|
## Delivered Changes
|
|
|
|
### 1. Managed ISCE2 pipeline hardening
|
|
|
|
- Added `--resume-from unwrap|geocode|export` support to the LT-1 pipeline.
|
|
- Added `--full-geocode` support. The default path now geocodes only the export-critical
|
|
products instead of ISCE2's full default list.
|
|
- Added stage-aware logging with stdout flush to improve long-run observability.
|
|
- Added geocode DEM subset preparation so resumed geocode/export runs do not reprocess the
|
|
full base DEM.
|
|
- Added a guard that blocks fresh runs when the selected DEM resolves to a very large raw
|
|
base raster without a prepared `.wgs84` sibling.
|
|
|
|
### 2. Engine and runtime configuration fixes
|
|
|
|
- `ISCE2` now prefers an existing prepared `.wgs84` DEM over the raw base path.
|
|
- `PyINT` DEM resolution now follows the same prepared-first preference.
|
|
- The WSL ISCE2 runner now passes `resume_from` and `full_geocode` through to the pipeline.
|
|
- The WSL runner now sets both `PROJ_DATA` and `PROJ_LIB`.
|
|
|
|
### 3. Recovery and catalog self-healing
|
|
|
|
- Added a completion-file repair helper for managed ISCE2 runs.
|
|
- In-place ISCE2 publish/rebuild now repairs missing:
|
|
- `execution_manifest.json`
|
|
- `current/isce2__<profile>.json`
|
|
- This allows recovered runs to be reintroduced into the managed result catalog without
|
|
hand-editing completion markers.
|
|
|
|
### 4. One-time DEM preparation tooling
|
|
|
|
- Added `backend/app/isce2_pipeline/prepare_isce2_base_dem.py`.
|
|
- The script resolves DEM paths from `.env`, validates existing prepared outputs, and can
|
|
generate a reusable `WGS84` `.wgs84` DEM from an `EGM96` base DEM.
|
|
- The script also auto-configures `PROJ` paths for standalone execution.
|
|
|
|
### 5. Configuration guidance
|
|
|
|
- Updated `.env.example` comments to distinguish:
|
|
- raw SARscape/ENVI DEM source path
|
|
- prepared ISCE2/PyINT `.wgs84` path
|
|
|
|
## Validation
|
|
|
|
- Python syntax validation was run for the modified ISCE2 pipeline, engine, runtime, and
|
|
result-catalog modules.
|
|
- The one-time full DEM preparation completed successfully and produced:
|
|
- `.wgs84`
|
|
- `.wgs84.xml`
|
|
- `.wgs84.vrt`
|
|
- The prepared DEM metadata was verified to report `WGS84`.
|
|
|
|
## Required Local Follow-Up
|
|
|
|
These operational steps are intentionally not committed:
|
|
|
|
- Point local `.env` `ISCE2_DEM_PATH` to the prepared `.wgs84` file.
|
|
- Point local `.env` `PYINT_PREPARED_DEM_PATH` to the same prepared `.wgs84` file.
|
|
- Restart the backend so the running process reloads the updated `.env`.
|
|
|
|
## Additional Update: Strict Production Workflow Boundary
|
|
|
|
After reviewing the ISCE2 production semantics, the export path was tightened so the
|
|
default managed `ISCE2` D-InSAR product remains a strict pipeline result rather than an
|
|
implicitly corrected interpretation layer.
|
|
|
|
Delivered adjustments:
|
|
|
|
- Kept the reference-normalization helper only as an optional debug/export capability.
|
|
- Restored the default export behavior to `reference_mode=none`.
|
|
- Removed reference-normalization controls from the regular managed production profile so
|
|
operators do not treat post-processing heuristics as part of the standard workflow.
|
|
- Revalidated the modified pipeline, engine, and WSL runner modules with `python3 -m py_compile`
|
|
inside the target WSL runtime environment.
|
|
|
|
## Additional Update: DEM Sidecar Path Repair
|
|
|
|
After switching the managed DEM bundle to a copied `SRTMDEM_RSP_SARscape` dataset under
|
|
`D:\DEM`, an ISCE2 run failed in `topo` even though the outer pipeline XML pointed at the
|
|
new location. The root cause was that the copied DEM sidecar XML files still contained old
|
|
absolute `/mnt/...` paths in `file_name`, `metadata_location`, and `extra_file_name`.
|
|
|
|
Delivered adjustments:
|
|
|
|
- Added `backend/app/isce2_pipeline/repair_dem_sidecars.py` to audit and repair moved DEM
|
|
sidecar XML files at directory scope.
|
|
- Added sidecar self-repair for the selected ISCE2 DEM during pipeline resolution so the
|
|
managed run no longer depends on manually editing copied `.xml` files first.
|
|
- Documented the migration risk and the recommended repair command in `docs/DEPLOYMENT.md`.
|
|
|
|
## Additional Update: LT-1 Enhancement Alignment
|
|
|
|
The managed `ISCE2` `lt1_stripmap` production profile now enables the built-in
|
|
stripmap enhancement path by default:
|
|
|
|
- dense offsets
|
|
- range rubbersheeting
|
|
- azimuth rubbersheeting
|
|
|
|
This was done to bring the default ISCE2 LT-1 production semantics closer to the
|
|
existing SARscape `custom6` chain, which already includes non-trivial refinement
|
|
steps rather than shipping a bare minimum interferometric result.
|
|
|
|
The implementation now passes these parameters end-to-end through:
|
|
|
|
- `backend/app/dinsar_engines/isce2_engine.py`
|
|
- `deploy/wsl/runners/isce2_runner.py`
|
|
- `backend/app/isce2_pipeline/run_lt1_dinsar_pipeline.py`
|
|
|
|
The rationale and the SARscape / ISCE2 mapping are documented in:
|
|
|
|
- `docs/ISCE2_LT1_ENHANCEMENT_ALIGNMENT_20260427.md`
|
|
|
|
## Additional Update: Rubbersheet Runtime Dependency
|
|
|
|
The first enhanced LT-1 run reached ISCE2 `dense_offsets` successfully and then
|
|
failed at `rubber_sheet_range` with:
|
|
|
|
```text
|
|
ModuleNotFoundError: No module named 'astropy'
|
|
```
|
|
|
|
Root cause:
|
|
|
|
- ISCE2's `runRubbersheetRange.py` imports `astropy.convolution`.
|
|
- The shared WSL runtime `insar_wsl_v1` had ISCE2 and SciPy installed, but did
|
|
not include `astropy`.
|
|
|
|
Delivered adjustments:
|
|
|
|
- Added `astropy` to `deploy/wsl/conda/insar_wsl_v1.environment.yml`.
|
|
- Added a runtime dependency preflight in the WSL runner and LT-1 pipeline so
|
|
rubbersheeting fails immediately with a clear message instead of after the
|
|
dense-offset stage has already run.
|
|
- Added `astropy.convolution` to the ISCE2 WSL availability check.
|
|
|
|
Required deployment action:
|
|
|
|
```bash
|
|
conda install -n insar_wsl_v1 -c conda-forge astropy
|
|
```
|
|
|
|
## Additional Update: Real Ionosphere Stage Integration
|
|
|
|
The managed `ISCE2` LT-1 stripmap workflow now runs the native stripmap
|
|
dispersive correction path instead of faking `PICKLE/ionosphere` state during
|
|
resume.
|
|
|
|
Delivered adjustments:
|
|
|
|
- Enabled `do split spectrum = True` and `do dispersive = True` in the generated
|
|
`stripmapApp` XML.
|
|
- Changed stage-2 execution from a narrow `unwrap -> unwrap` run to a real
|
|
`filter_low_band/unwrap/ionosphere` resume path.
|
|
- Changed stage-3 execution to resume from real `ionosphere` state when present,
|
|
or from the low/high-band unwrap state when only stage-2 products exist.
|
|
- Extended the reduced geocode export list with:
|
|
- `ionosphere/dispersive.bil.unwCor.filt`
|
|
- `ionosphere/nondispersive.bil.unwCor.filt`
|
|
- `ionosphere/mask.bil`
|
|
- Updated the export step to prefer geocoded
|
|
`ionosphere/nondispersive.bil.unwCor.filt` when available.
|
|
|
|
Operational effect:
|
|
|
|
- `resume_from=unwrap` now resumes the complete stage-2 chain up to
|
|
`ionosphere`
|
|
- `resume_from=geocode` now performs real `ionosphere -> geocode` continuation
|
|
instead of relying on copied pickle files
|
|
|
|
Deployment note:
|
|
|
|
- The native ionosphere implementation imports `cv2` and `scipy`
|
|
- The WSL runner, pipeline preflight, and health check now verify those modules
|
|
before production starts
|