Compare commits

..
11 Commits
Author SHA1 Message Date
Harmon 72747071d0 docs: 添加 README 索引、CONVENTIONS 协定与 CI
CI / python (s1-orbit-downloader) (push) Waiting to run
CI / python (sar-gf3-l1a2l2) (push) Waiting to run
2026-10-16 18:59:55 +08:00
Harmon 7740f341a7 chore: 合并 GF3_L1A_To_L2_pipeline 历史 2026-10-16 18:59:39 +08:00
Harmon 19ba4d2a65 chore: 合并 sentinel-orbit-downloader 历史 2026-10-16 18:59:38 +08:00
Harmon 42b15d9cf0 chore: 合并 lutan1-envi-idl-dinsar-pipeline 历史 2026-10-16 18:59:38 +08:00
Harmon d4ae7dbffb chore: 初始化 rs-toolkit 仓库 2026-10-16 18:59:38 +08:00
GF3 Pipeline Bot 28d8bab6b9 Skip completed GF3 scenes on rerun 2026-05-30 02:02:26 +08:00
GF3 Pipeline Bot 363408f8d2 Handle corrupt GF3 archives in batch runs 2026-05-30 01:46:04 +08:00
GF3 Pipeline Bot 258a4c6329 Add SARscape Go wrapper workflow 2026-05-29 19:52:32 +08:00
Harmon d9d80d7f17 Refactor GF3 L1A to L2 pipeline 2026-05-03 15:41:08 +08:00
Harmon 3d2a34a4b3 Fix Sentinel UI shutdown flow 2026-04-29 00:28:49 +08:00
Harmon 753a84dc59 Initial commit 2026-04-14 01:39:51 +08:00
54 changed files with 4742 additions and 37 deletions
+30
View File
@@ -0,0 +1,30 @@
name: CI
on:
push:
branches: [main]
pull_request:
jobs:
python:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
tool: [sar-gf3-l1a2l2, s1-orbit-downloader]
defaults:
run:
working-directory: python/${{ matrix.tool }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: 编译检查
run: python -m compileall -q src
- name: Ruff(仅报错类规则)
run: pip install ruff && ruff check --select E9,F63,F7,F82 src
- name: 安装
run: pip install .
- name: 测试
run: |
if [ -d tests ]; then pip install pytest && pytest -q; else echo "无测试"; fi
+5
View File
@@ -0,0 +1,5 @@
__pycache__/
*.py[cod]
.venv/
*.egg-info/
/build/
+57
View File
@@ -0,0 +1,57 @@
# Harmon 代码协定
适用于 git.rsharmon.cn/Harmon 下的所有仓库(GitHub HarmonRS 为镜像)。新代码必须遵守,旧代码在修改时逐步对齐。
## 1. 仓库命名
- 全小写,短横线连接:`领域-数据源-功能`,如 `sar-lt1-dinsar`、`sar-gf3-l1a2l2`、`s1-orbit-downloader`。
- 常用前缀:`sar`(雷达通用)、`lt1`(陆探1号)、`gf3`(高分3号)、`s1`(Sentinel-1)、`opt`(光学)。
- 禁止下划线、大写、`-v2`/`-new`/`-final` 之类的版本后缀(版本靠 tag)。
- 主力项目(系统/服务)独立成仓库;小工具、脚本统一放进 `rs-toolkit`。
## 2. rs-toolkit 目录结构
```
rs-toolkit/
idl/<工具名>/ # IDL/ENVI 脚本
python/<工具名>/ # 每个工具是独立的 Python 包
go/<工具名>/ # 如有
README.md # 工具索引:一行说明 + 安装/运行方式
CONVENTIONS.md
```
- 工具目录名同样遵守第 1 条命名规则。
- 工具之间不互相 import;共用代码再考虑抽成 `python/rs-common/`。
- 新增工具时同步更新根 README 索引。
## 3. 分支与提交
- 默认分支 `main`,始终保持可用;功能在 `feat/xxx`、`fix/xxx` 分支上开发,通过 PR 合并。
- 提交信息采用 Conventional Commits:`<type>(<scope>): <说明>`
- type:`feat` `fix` `refactor` `docs` `test` `chore` `ci` `perf`
- scope:rs-toolkit 中写工具名,如 `fix(sar-gf3-l1a2l2): 跳过损坏的压缩包`
- 一个提交只做一件事;不提交生成物、数据、密钥。
- 发布打 tag:主项目 `vX.Y.Z`;toolkit 内工具 `<工具名>-vX.Y.Z`。
## 4. Python
- 一律 `pyproject.toml` + `src/` 布局:`src/<包名>/`,包名为目录名的下划线形式。
- 声明 `requires-python`、依赖下限,命令行入口写在 `[project.scripts]`。
- 安装方式(无需拷脚本):
`pip install "git+https://git.rsharmon.cn/Harmon/rs-toolkit.git#subdirectory=python/<工具名>"`
- 代码用 `ruff` 检查与格式化;测试放 `tests/`,用 `pytest`。
- 路径、账号等配置不写死在代码里,走命令行参数/配置文件/环境变量。
## 5. IDL / ENVI
- 文件名小写下划线(IDL 惯例),一个 `.pro` 一个主过程,过程名与文件名一致。
- 文件头注释写明:用途、输入输出、依赖的 ENVI/SARscape 版本、作者与日期。
- 输入输出路径作为参数传入,不写死;批处理入口统一命名 `batch_*.pro`。
- `.sav` 等编译产物如必须入库,标记为 binary(.gitattributes)。
## 6. README 要求
每个仓库/工具目录必须有 README,至少包含:
1. 一句话说明做什么;2. 环境与依赖;3. 安装;4. 最小运行示例;5. 输入/输出说明;6. 已知限制。
## 7. CI
- 每个仓库在 `.gitea/workflows/` 下配置 CI,PR 和推送 main 时触发。
- Python:ruff + 编译检查 + pytest(有测试时);Rust:`cargo fmt --check`、`cargo clippy -D warnings`、`cargo test`;Go:`go vet`、`go test`。
- `main` 开分支保护:CI 通过后才能合并。AI 生成的代码同样必须过 CI。
## 8. 其他
- 文本统一 UTF-8、LF(Windows `.cmd/.bat` 除外)。
- 大于 10MB 的二进制/数据不入库,放 release 或 NAS。
+20
View File
@@ -0,0 +1,20 @@
# rs-toolkit
遥感(SAR/InSAR)常用小工具与脚本合集。规范见 [CONVENTIONS.md](CONVENTIONS.md)。
| 工具 | 语言 | 说明 | 原仓库 |
|---|---|---|---|
| [idl/sar-lt1-dinsar](idl/sar-lt1-dinsar) | IDL | 陆探1号 ENVI/SARscape D-InSAR 批处理流水线 | lutan1-envi-idl-dinsar-pipeline |
| [python/sar-gf3-l1a2l2](python/sar-gf3-l1a2l2) | Python | 高分3号 L1A → L2 处理流水线(含 SARscape Go 封装) | GF3_L1A_To_L2_pipeline |
| [python/s1-orbit-downloader](python/s1-orbit-downloader) | Python | Sentinel-1 影像与精密轨道查询下载(CLI + Web UI) | sentinel-orbit-downloader |
## 安装 Python 工具
```bash
pip install "git+https://git.rsharmon.cn/Harmon/rs-toolkit.git#subdirectory=python/sar-gf3-l1a2l2"
pip install "git+https://git.rsharmon.cn/Harmon/rs-toolkit.git#subdirectory=python/s1-orbit-downloader"
```
IDL 工具直接在 ENVI/IDL 中打开对应目录下的 `.pro`,用法见各目录 README。
各工具的提交历史均已保留(`git log -- <目录>`)。
+373
View File
@@ -0,0 +1,373 @@
;+
; PURPOSE:
; This script automates a full D-InSAR workflow, from interferogram
; generation to geocoding (Steps 1-6).
;
; This is the final, complete, and integrated version resulting from
; our collaborative development and debugging.
;-
;================================================================================
; ### HELPER FUNCTION: READ SML PARAMETER ###
;================================================================================
FUNCTION Read_SML_Parameter, sml_file, param_name_to_find
IF N_ELEMENTS(sml_file) EQ 0 OR sml_file EQ '' THEN BEGIN
PRINT, 'Error: An empty or undefined filename was passed to Read_SML_Parameter.'
RETURN, ''
ENDIF
IF NOT FILE_TEST(sml_file) THEN BEGIN
PRINT, 'Error: SML file not found -> ' + sml_file
RETURN, ''
ENDIF
lun = -1
ON_ERROR, 2
OPENR, lun, sml_file, /GET_LUN, ERROR=err
IF (err NE 0) THEN BEGIN
PRINT, 'Error: Could not open SML file for reading -> ' + sml_file
IF lun NE -1 THEN FREE_LUN, lun
RETURN, ''
ENDIF
found_value = ''
line = ''
start_tag = '<' + STRUPCASE(param_name_to_find) + '>'
end_tag = '</' + STRUPCASE(param_name_to_find) + '>'
WHILE NOT EOF(lun) DO BEGIN
READF, lun, line
clean_line = STRTRIM(line, 2)
line_upper = STRUPCASE(clean_line)
start_pos = STRPOS(line_upper, start_tag)
IF start_pos NE -1 THEN BEGIN
end_pos = STRPOS(line_upper, end_tag, start_pos)
IF end_pos NE -1 THEN BEGIN
start_extract = start_pos + STRLEN(start_tag)
value_length = end_pos - start_extract
found_value = STRMID(clean_line, start_extract, value_length)
BREAK
ENDIF
ENDIF
ENDWHILE
IF lun NE -1 THEN FREE_LUN, lun
RETURN, found_value
END
;================================================================================
; ### HELPER FUNCTION: CREATE GCPs FROM COHERENCE ###
;================================================================================
FUNCTION CREATE_GCPS_FROM_COHERENCE, coherence_file, output_shp_file, COH_THRESHOLD=coh_threshold, NUM_POINTS=num_points
ON_ERROR, 2
IF FILE_TEST(output_shp_file) THEN FILE_DELETE, output_shp_file, /QUIET
IF N_ELEMENTS(coh_threshold) EQ 0 THEN coh_threshold = 0.7
IF N_ELEMENTS(num_points) EQ 0 THEN num_points = 100
e = ENVI(/HEADLESS)
PRINT, 'Opening coherence file: ' + coherence_file
oRaster = e.OpenRaster(coherence_file)
ns = oRaster.NCOLUMNS
nl = oRaster.NROWS
data = oRaster.GetData(BANDS=[0])
PRINT, 'Searching for GCPs with coherence > ' + STRTRIM(coh_threshold, 2)
grid_dim = CEIL(SQRT(num_points))
x_step = FLOOR(ns / grid_dim)
y_step = FLOOR(nl / grid_dim)
pixel_points_x = [] & pixel_points_y = []
FOR j=0L, grid_dim-1 DO BEGIN
FOR i=0L, grid_dim-1 DO BEGIN
x_start = i * x_step & y_start = j * y_step
x_end = (i+1) * x_step -1 < (ns-1)
y_end = (j+1) * y_step -1 < (nl-1)
IF (x_start GE ns) OR (y_start GE nl) THEN CONTINUE
cell_data = data[x_start:x_end, y_start:y_end]
max_val = MAX(cell_data, max_idx)
IF max_val GE coh_threshold THEN BEGIN
max_y_cell = max_idx / (x_end - x_start + 1)
max_x_cell = max_idx MOD (x_end - x_start + 1)
pixel_points_x = [pixel_points_x, x_start + max_x_cell]
pixel_points_y = [pixel_points_y, y_start + max_y_cell]
ENDIF
ENDFOR
ENDFOR
point_count = N_ELEMENTS(pixel_points_x)
IF point_count EQ 0 THEN BEGIN
PRINT, 'Error: No GCPs found.' & OBJ_DESTROY, oRaster & RETURN, 0
ENDIF
PRINT, 'Found ' + STRTRIM(point_count, 2) + ' suitable GCPs.'
PRINT, 'Creating shapefile using final authoritative methodology...'
oShape = OBJ_NEW('IDLffShape', output_shp_file, ENTITY_TYPE=1, /UPDATE)
oShape->AddAttribute, 'SHP_ID', 3, 10
oShape->AddAttribute, 'GCP_LABEL', 7, 20
oShape->AddAttribute, 'GCP_TYPE', 7, 20
oShape->AddAttribute, 'GCP_COLUMN', 5, 24, PRECISION=6
oShape->AddAttribute, 'GCP_ROW', 5, 24, PRECISION=6
oShape->AddAttribute, 'GCP_OTHER_', 7, 10 ; User-verified field name
attr_template = oShape->GetAttributes(/ATTRIBUTE_STRUCTURE)
entNew = {IDL_SHAPE_ENTITY}
entNew.SHAPE_TYPE = 1
FOR i=0, point_count-1 DO BEGIN
x_float = FLOAT(pixel_points_x[i])
y_float = FLOAT(pixel_points_y[i])
entNew.ISHAPE = i
entNew.BOUNDS = [x_float, y_float, 0.0, 0.0, x_float, y_float, 0.0, 0.0]
oShape->PutEntity, entNew
attr = attr_template
attr.ATTRIBUTE_0 = i
attr.ATTRIBUTE_1 = 'GCP_' + STRTRIM(i+1, 2)
attr.ATTRIBUTE_2 = 'undefined'
attr.ATTRIBUTE_3 = x_float
attr.ATTRIBUTE_4 = y_float
attr.ATTRIBUTE_5 = ''
oShape->SetAttributes, i, attr
ENDFOR
oShape->Close
OBJ_DESTROY, [oShape, oRaster]
PRINT, 'Successfully created GCP shapefile: ' + output_shp_file
RETURN, 1
END
;================================================================================
; ### MAIN WORKFLOW PROCEDURE ###
;================================================================================
PRO DinsarWorkflow_ALL
; --- Initialize SARscape Environment ---
temporary_directory = 'D:\Sarscape_IDL_Area'
SARscape_Batch_Init,Temp_Directory=temporary_directory
e = ENVI(/HEADLESS)
PRINT, 'ENVI 和 SARscape 环境初始化成功!'
PRINT, ''
; ===================================================================
; ### GLOBAL CONFIGURATION ###
; ===================================================================
master_base_file = 'G:/Code/InSAR/IDLTools/Task_20250504_20250601_1/LuTan-1_LT1A_39_20250504_091105615_STRIP1_A_HH_slc'
slave_base_file = 'G:/Code/InSAR/IDLTools/Task_20250504_20250601_1/LuTan-1_LT1B_47_20250601_091101714_STRIP1_A_HH_slc'
dem_base_file = 'D:/SRTM30m/SRTMDEM_RSP_SARscape'
workflow_root_name = 'G:/Code/InSAR/IDLTools/test/test' ; Use a consistent root name
target_ground_resolution_m = 10.0
filter_method = 'GOLDSTEIN'
unwrapping_coh_threshold = 0.05
gcp_coh_threshold = 0.7
gcp_number = 100
geocoding_coh_threshold = 0.0
geocoding_pixel_size_m = 10.0
; ===================================================================
; ### STEP 1: INTERFEROGRAM GENERATION ###
; ===================================================================
print, '======================================================'
print, '### STEP 1: INTERFEROGRAM GENERATION ###'
print, '======================================================'
master_sml = master_base_file + '.sml' & slave_sml = slave_base_file + '.sml'
m_rg_sp_str = Read_SML_Parameter(master_sml, 'PixelSpacingRg')
m_az_sp_str = Read_SML_Parameter(master_sml, 'PixelSpacingAz')
m_inc_ang_str = Read_SML_Parameter(master_sml, 'IncidenceAngle')
s_rg_sp_str = Read_SML_Parameter(slave_sml, 'PixelSpacingRg')
s_az_sp_str = Read_SML_Parameter(slave_sml, 'PixelSpacingAz')
s_inc_ang_str = Read_SML_Parameter(slave_sml, 'IncidenceAngle')
IF m_rg_sp_str EQ '' OR s_rg_sp_str EQ '' THEN GOTO, WORKFLOW_EXIT_FAILURE
m_rg_sp = FLOAT(m_rg_sp_str) & m_az_sp = FLOAT(m_az_sp_str) & m_inc_ang = FLOAT(m_inc_ang_str)
s_rg_sp = FLOAT(s_rg_sp_str) & s_az_sp = FLOAT(s_az_sp_str) & s_inc_ang = FLOAT(s_inc_ang_str)
avg_az_spacing = (m_az_sp + s_az_sp) / 2.0 & azimuth_looks = (long(target_ground_resolution_m / avg_az_spacing)) > 1
m_ground_rg_sp = m_rg_sp / SIN(m_inc_ang * !DPI / 180.0) & s_ground_rg_sp = s_rg_sp / SIN(s_inc_ang * !DPI / 180.0)
avg_ground_rg_sp = (m_ground_rg_sp + s_ground_rg_sp) / 2.0 & range_looks = (long(target_ground_resolution_m / avg_ground_rg_sp)) > 1
task1_obj = obj_new('SARscapeBatch', Module='InSARInterferogramGeneration')
prefix_1 = 'MAIN_INSAR_INTERFEROGRAM_GENERATION_CMD.'
task1_obj->SetParam, prefix_1 + 'INPUT_REFERENCE_FILE_NAME' , master_base_file
task1_obj->SetParam, prefix_1 + 'INPUT_SECONDARY_FILE_NAME' , slave_base_file
task1_obj->SetParam, prefix_1 + 'DEM_FILE_NAME' , dem_base_file
task1_obj->SetParam, prefix_1 + 'OUTPUT_ROOT_FILE_NAME' , workflow_root_name
task1_obj->SetParam, prefix_1 + 'RG_LOOKS_NBR' , string(range_looks)
task1_obj->SetParam, prefix_1 + 'AZ_LOOKS_NBR' , string(azimuth_looks)
task1_obj->SetParam, 'COREGISTRATION_CMD.COREGISTRATION_WITH_DEM_FLAG' , 'OK'
task1_obj->SetParam, 'FLAT_CMD.MAKE_FLATTENING_FLAG', 'OK'
ok_step1 = task1_obj->Execute()
obj_destroy, task1_obj
IF NOT ok_step1 THEN GOTO, WORKFLOW_EXIT_FAILURE
print, '### STEP 1: SUCCESS!'
; ===================================================================
; ### STEP 2: FILTERING AND COHERENCE GENERATION ###
; ===================================================================
print, '======================================================'
print, '### STEP 2: FILTERING AND COHERENCE GENERATION ###'
print, '======================================================'
task2_obj = obj_new('SARscapeBatch', Module='InSARFilterAndCoherence')
prefix_main2 = 'MAIN_INSAR_FILTER_COHERENCE_CMD.'
prefix_filt2 = 'FILTERING_CMD.'
task2_obj->SetParam, prefix_main2 + 'INPUT_INTERF_FILE_NAME', workflow_root_name + '_dint'
task2_obj->SetParam, prefix_main2 + 'INPUT_REFERENCE_FILE_NAME', workflow_root_name + '_reference_pwr'
task2_obj->SetParam, prefix_main2 + 'INPUT_SECONDARY_FILE_NAME', workflow_root_name + '_secondary_pwr'
task2_obj->SetParam, prefix_main2 + 'OUT_ROOT_FILE', workflow_root_name
task2_obj->SetParam, prefix_filt2 + 'FILTERING_METHOD', filter_method
task2_obj->SetParam, prefix_main2 + 'COHERENCE_FLAG', 'OK'
task2_obj->SetParam, prefix_main2 + 'INTERF_FILT_FLAG', 'OK'
ok_step2 = task2_obj->Execute()
obj_destroy, task2_obj
IF NOT ok_step2 THEN GOTO, WORKFLOW_EXIT_FAILURE
print, '### STEP 2: SUCCESS!'
; ===================================================================
; ### STEP 3: ORBITAL TREND REMOVAL ###
; ===================================================================
print, '======================================================'
print, '### STEP 3: ORBITAL TREND REMOVAL ###'
print, '======================================================'
reflat_output_root = workflow_root_name + '_rrpf'
task3_obj = obj_new('SARscapeBatch', Module='InSARRemoveResidualPhaseFrequency')
prefix_3 = 'MAIN_INSAR_REMOVE_RESIDUAL_PHASE_FREQUENCY.'
task3_obj->SetParam, prefix_3 + 'INTERF_FILE_NAME', workflow_root_name + '_fint'
task3_obj->SetParam, prefix_3 + 'COHERENCE_FILE_NAME', workflow_root_name + '_cc'
task3_obj->SetParam, prefix_3 + 'OUT_ROOT_FILE_NAME', reflat_output_root
ok_step3 = task3_obj->Execute()
obj_destroy, task3_obj
IF NOT ok_step3 THEN GOTO, WORKFLOW_EXIT_FAILURE
print, '### STEP 3: SUCCESS!'
; ===================================================================
; ### STEP 4: PHASE UNWRAPPING ###
; ===================================================================
print, '======================================================'
print, '### STEP 4: PHASE UNWRAPPING ###'
print, '======================================================'
reflat_upha_output_file = workflow_root_name + '_upha'
task4_obj = obj_new('SARscapeBatch', Module='InSARPhaseUnwrapping')
prefix_main4 = 'MAIN_INSAR_PHASE_UNWRAPPING_CMD.'
prefix_upha4 = 'UPHA_CMD.'
task4_obj->SetParam, prefix_main4 + 'INFILE_NAME', reflat_output_root + '_fint'
task4_obj->SetParam, prefix_main4 + 'COHERENCEFILE_NAME', workflow_root_name + '_cc'
task4_obj->SetParam, prefix_upha4 + 'UPHA_COH_THRESHOLD', string(unwrapping_coh_threshold)
task4_obj->SetParam, prefix_main4 + 'OUTFILE_NAME', reflat_upha_output_file
ok_step4 = task4_obj->Execute()
obj_destroy, task4_obj
IF NOT ok_step4 THEN GOTO, WORKFLOW_EXIT_FAILURE
print, '### STEP 4: SUCCESS!'
; ===================================================================
; ### STEP 5: REFINEMENT AND REFLATTENING ###
; ===================================================================
; --- STEP 5A: Automatic GCP Generation ---
print, '======================================================'
print, '### STEP 5A: 正在自动生成GCP... ###'
print, '======================================================'
auto_gcp_shp = workflow_root_name + '_auto_gcp.shp'
gcp_success = CREATE_GCPS_FROM_COHERENCE(workflow_root_name + '_cc', auto_gcp_shp, COH_THRESHOLD=gcp_coh_threshold, NUM_POINTS=gcp_number)
IF gcp_success EQ 0 THEN BEGIN
PRINT, '错误: 自动生成GCP shapefile失败。流程终止。'
GOTO, WORKFLOW_EXIT_FAILURE
ENDIF
PRINT, '### STEP 5A: SUCCESS!'
; --- STEP 5B: Refinement and Reflattening Execution ---
print, '======================================================'
print, '### STEP 5B: 正在执行 REFINEMENT AND REFLATTENING ###'
print, '======================================================'
ref_output_root = workflow_root_name + '_reflat'
task_refine = obj_new('SARscapeBatch', Module='InSARRefinementAndReflattening')
prefix_refine = 'MAIN_INSAR_REFINEMENT_AND_REFLATTENING_CMD.'
task_refine->SetParam, prefix_refine + 'INPUT_UPHA_FILE_NAME', reflat_upha_output_file
task_refine->SetParam, prefix_refine + 'INPUT_REFERENCE_FILE_NAME', workflow_root_name + '_reference_pwr'
task_refine->SetParam, prefix_refine + 'INPUT_SECONDARY_FILE_NAME', workflow_root_name + '_secondary_pwr'
task_refine->SetParam, prefix_refine + 'SLANT_RANGE_DEM_FILE_NAME', workflow_root_name + '_srdem'
task_refine->SetParam, prefix_refine + 'SYNTHETIC_FILE_NAME', workflow_root_name + '_sint'
task_refine->SetParam, prefix_refine + 'COHERENCE_FILE_NAME', workflow_root_name + '_cc'
task_refine->SetParam, prefix_refine + 'OUTPUT_ROOT_NAME', ref_output_root
task_refine->SetParam, prefix_refine + 'DEM_FILE_NAME', dem_base_file
task_refine->SetParam, 'REFINEMENT_CMD.REFINEMENT_GCP_FILE_NAME', auto_gcp_shp
ok_step5 = task_refine->Execute()
obj_destroy, task_refine
IF NOT ok_step5 THEN GOTO, WORKFLOW_EXIT_FAILURE
print, '### STEP 5B: SUCCESS!'
; ===================================================================
; ### STEP 6: PHASE TO DISPLACEMENT AND GEOCODING ###
; ===================================================================
print, '======================================================'
print, '### STEP 6: PHASE TO DISPLACEMENT AND GEOCODING ###'
print, '======================================================'
final_upha_file = ref_output_root + '_upha'
geocoded_output_root = workflow_root_name + '_geo'
task6_obj = obj_new('SARscapeBatch', Module='InSARPhaseToDisplacement')
IF (~OBJ_VALID(task6_obj)) THEN BEGIN
PRINT, '错误: 创建 SARscapeBatch 对象失败: InSARPhaseToDisplacement'
GOTO, WORKFLOW_EXIT_FAILURE
ENDIF
; --- Set Main Parameters ---
prefix_main6 = 'MAIN_INSAR_PHASE_TO_DISPLACEMENT_CMD.'
task6_obj->SetParam, prefix_main6 + 'INPUT_FILE_NAME', final_upha_file
task6_obj->SetParam, prefix_main6 + 'OUTPUT_FILE', geocoded_output_root
task6_obj->SetParam, prefix_main6 + 'COHERENCE_FILE', workflow_root_name + '_cc'
task6_obj->SetParam, prefix_main6 + 'DEM_FILE_NAME', dem_base_file
task6_obj->SetParam, prefix_main6 + 'COHERENCE_THRESHOLD', STRING(geocoding_coh_threshold)
; --- Set Other Required Parameters ---
task6_obj->SetParam, prefix_main6 + 'INTERPOL_TYPE' , '4th_order_cc'
task6_obj->SetParam, prefix_main6 + 'ALLOW_SKIP_REFINEMENT' , 'NotOK'
task6_obj->SetParam, prefix_main6 + 'SARSCAPEENVIRONMENT' , 'IDL_ENVI_ENV'
; --- Set Geocoding Parameters ---
prefix_geocode6 = 'GEOCODE_CMD.'
task6_obj->SetParam, prefix_geocode6 + 'GEOCODE_RG_GRID_SIZE', STRING(geocoding_pixel_size_m)
task6_obj->SetParam, prefix_geocode6 + 'GEOCODE_AZ_GRID_SIZE', STRING(geocoding_pixel_size_m)
task6_obj->SetParam, prefix_geocode6 + 'GEOCODE_INTERPOL_BOX_SIZE', '7'
; --- Set Displacement Product Generation Parameters ---
prefix_disp6 = 'DISPLACEMENT_PROJECTION_CMD.'
task6_obj->SetParam, prefix_disp6 + 'GENERATE_LOS_FLAG', 'OK'
task6_obj->SetParam, prefix_disp6 + 'GENERATE_VERTICAL_FLAG', 'NotOK'
task6_obj->SetParam, prefix_disp6 + 'GENERATE_MAX_SLOPE_FLAG', 'NotOK'
print, '正在执行地理编码任务...'
ok_step6 = task6_obj->Execute()
obj_destroy, task6_obj
IF NOT ok_step6 THEN GOTO, WORKFLOW_EXIT_FAILURE
print, '### STEP 6: SUCCESS!'
; --- Final Success Message ---
print, '======================================================='
print, ' FULL D-InSAR WORKFLOW (STEPS 1-6) COMPLETED SUCCESSFULLY!'
print, '======================================================='
GOTO, WORKFLOW_EXIT_NORMAL
WORKFLOW_EXIT_FAILURE:
print, 'Workflow terminated due to a failure in one of the steps.'
WORKFLOW_EXIT_NORMAL:
SARscape_Batch_Exit
end
+316
View File
@@ -0,0 +1,316 @@
# ENVI + IDL D-InSAR 批处理流水线
本仓库是一套基于 ENVI / IDL / SARscape 的 D-InSAR 自动化处理脚本,面向陆探一号等 SAR 数据的批量导入与差分干涉处理。代码包含两个主要阶段:
1. 批量导入原始数据,将 `*.meta.xml` 导入为 SARscape 可继续处理的产品。
2. 对已经完成导入的主辅影像执行完整 D-InSAR 流程,直到位移反演与地理编码输出。
仓库当前核心脚本只有 3 个:
- `Run_Batch_Task_Import.pro`:批量导入 `Task_*` 目录中的原始数据。
- `batch_dinsarworkflow_all.pro`:批量执行 D-InSAR 流程。
- `DinsarWorkflow_ALL.pro`:单景测试版 / 调试版完整流程。
## 1. 依赖环境
运行本项目需要以下软件环境已经正确安装并可在 IDL 中调用:
- ENVI
- IDL
- SARscape
- 对应 SARscape Batch 模块与 ENVI Task 环境
脚本中直接使用了以下对象 / 任务:
- `ENVI(/HEADLESS)`
- `SARscape_Batch_Init`
- `SARscapeBatch`
- `ENVITask('SARsImportLuTan1')`
- `IDLffShape`
因此该仓库不是纯 IDL 通用脚本,必须在装有 ENVI + SARscape 的环境中运行。
## 2. 仓库结构
```text
ENVI_IDL_DInSAR/
├─ Run_Batch_Task_Import.pro
├─ batch_dinsarworkflow_all.pro
├─ DinsarWorkflow_ALL.pro
└─ README.md
```
各脚本职责如下。
### 2.1 `Run_Batch_Task_Import.pro`
用途:批量扫描根目录下的 `Task_*` 文件夹,并分别处理其中的 `master` 与 `slave` 子目录。
处理逻辑:
- 在 `master` / `slave` 目录中查找 `*.meta.xml`
- 使用 `ENVITask('SARsImportLuTan1')` 执行导入
- 输出直接写回当前 `master` 或 `slave` 目录
- 如果目录中已经存在 `*.sml`,则认为该目录已导入,自动跳过
适合放在整个流程的第一步,用来把原始数据准备成后续 D-InSAR 可识别的输入。
### 2.2 `batch_dinsarworkflow_all.pro`
用途:批量执行完整 D-InSAR 流程。
处理逻辑:
- 扫描根目录下所有 `Task_*` 文件夹
- 检查每个任务目录下是否存在 `master` / `slave`
- 在 `master`、`slave` 中各取第一个 `*.sml`
- 去掉 `.sml` 后缀,得到 SARscape 输入基名
- 在任务目录内创建 `dinsar_results` 输出目录
- 调用 `Execute_Single_Dinsar_Workflow` 执行完整处理
这是生产使用时最重要的脚本。
### 2.3 `DinsarWorkflow_ALL.pro`
用途:单组主辅影像的完整 D-InSAR 处理脚本。
特点:
- 直接在脚本内手工指定 `master_base_file`、`slave_base_file`、`dem_base_file`
- 更适合单景调试、流程验证、参数试验
- 逻辑上与批处理版本一致,但不是批量入口
## 3. 建议的数据组织方式
根据代码,批处理脚本假定数据目录组织如下:
```text
<rootDir>/
├─ Task_20250504_20250601_1/
│ ├─ master/
│ │ ├─ *.meta.xml
│ │ ├─ ... 导入后生成的 *.sml、影像及配套文件
│ ├─ slave/
│ │ ├─ *.meta.xml
│ │ ├─ ... 导入后生成的 *.sml、影像及配套文件
│ └─ dinsar_results/
│ └─ workflow*
├─ Task_xxx/
└─ ...
```
处理顺序通常是:
1. 先运行 `Run_Batch_Task_Import.pro`
2. 确认每个 `master` / `slave` 目录中已经生成 `*.sml`
3. 再运行 `batch_dinsarworkflow_all.pro`
## 4. D-InSAR 主流程说明
`batch_dinsarworkflow_all.pro` 中的 `Execute_Single_Dinsar_Workflow` 实现了完整 6 步流程:
### Step 1. 干涉图生成
- 从主影像和辅影像的 `.sml` 中读取:
- `PixelSpacingRg`
- `PixelSpacingAz`
- `IncidenceAngle`
- 根据目标地面分辨率自动计算多视数:
- `RG_LOOKS_NBR`
- `AZ_LOOKS_NBR`
- 调用 `InSARInterferogramGeneration`
### Step 2. 滤波与相干图生成
- 调用 `InSARFilterAndCoherence`
- 使用设定的滤波方法,例如 `GOLDSTEIN`
- 生成滤波干涉图与相干图
### Step 3. 轨道残余相位频率去除
- 调用 `InSARRemoveResidualPhaseFrequency`
- 输入为上一步生成的滤波干涉图和相干图
### Step 4. 相位解缠
- 调用 `InSARPhaseUnwrapping`
- 使用相干阈值 `UPHA_COH_THRESHOLD`
### Step 5. 精化与再平地
分为两步:
1. 自动 GCP 生成
2. Refinement and Reflattening
自动 GCP 生成的逻辑为:
- 打开相干图
- 按网格均匀划分影像
- 在每个网格中选取相干系数最大的像元
- 当该像元相干值高于阈值时,将其写入 shapefile
- 最终生成 `*_auto_gcp.shp`
随后调用 `InSARRefinementAndReflattening`,并将自动生成的 GCP 文件作为输入。
### Step 6. 相位转位移与地理编码
- 调用 `InSARPhaseToDisplacement`
- 使用最终解缠结果生成 LOS 位移产品
- 执行地理编码
- 当前脚本仅开启 LOS 输出:
- `GENERATE_LOS_FLAG = OK`
- `GENERATE_VERTICAL_FLAG = NotOK`
- `GENERATE_MAX_SLOPE_FLAG = NotOK`
## 5. 关键输入参数
以下参数需要在脚本中按实际工程修改。
### 5.1 导入脚本参数
文件:`Run_Batch_Task_Import.pro`
- `temporary_directory`:SARscape 临时目录
- `rootDir`:待扫描的任务根目录
- `numToProcess`:最多处理多少个任务,`0` 表示全部处理
### 5.2 批处理 D-InSAR 参数
文件:`batch_dinsarworkflow_all.pro`
- `temporary_directory`:SARscape 临时目录
- `rootDir`:包含多个 `Task_*` 目录的根目录
- `dem_base_file`:DEM 基础文件路径
- `target_ground_resolution_m`:目标地面分辨率,控制自动多视
- `filter_method`:滤波方法,默认 `GOLDSTEIN`
- `unwrapping_coh_threshold`:解缠相干阈值
- `gcp_coh_threshold`:自动 GCP 相干阈值
- `gcp_number`:目标 GCP 数量
- `geocoding_coh_threshold`:地理编码时使用的相干阈值
- `geocoding_pixel_size_m`:地理编码输出像元大小
- `numToProcess`:最多处理多少个任务,`0` 表示全部处理
### 5.3 单景调试参数
文件:`DinsarWorkflow_ALL.pro`
- `master_base_file`
- `slave_base_file`
- `dem_base_file`
- `workflow_root_name`
- 其余处理参数与批处理版本基本一致
注意:`master_base_file` 和 `slave_base_file` 填写的是“基名”,不是 `.sml` 文件全名。脚本内部会自动拼接 `.sml` 去读取参数。
## 6. 输出结果说明
批处理脚本默认在每个任务目录下创建:
```text
Task_xxx/
└─ dinsar_results/
├─ workflow_dint
├─ workflow_reference_pwr
├─ workflow_secondary_pwr
├─ workflow_fint
├─ workflow_cc
├─ workflow_rrpf_fint
├─ workflow_upha
├─ workflow_auto_gcp.shp
├─ workflow_reflat_upha
├─ workflow_geo*
└─ ...
```
其中常见关键中间结果包括:
- `*_dint`:初始干涉图
- `*_fint`:滤波后干涉图
- `*_cc`:相干图
- `*_upha`:解缠结果
- `*_auto_gcp.shp`:自动生成的 GCP 矢量文件
- `*_geo*`:地理编码后的位移产品
实际生成的附属文件数量和后缀会受 SARscape 版本及模块设置影响。
## 7. 典型使用方式
### 7.1 第一步:批量导入
在 `Run_Batch_Task_Import.pro` 中修改:
- `temporary_directory`
- `rootDir`
- `numToProcess`
然后在 IDL 命令行或脚本环境中运行:
```idl
.compile Run_Batch_Task_Import.pro
Run_Batch_Task_Import
```
### 7.2 第二步:批量执行 D-InSAR
在 `batch_dinsarworkflow_all.pro` 中修改:
- `temporary_directory`
- `rootDir`
- `dem_base_file`
- 各处理参数
然后运行:
```idl
.compile batch_dinsarworkflow_all.pro
Batch_DinsarWorkflow_ALL
```
### 7.3 单景测试
在 `DinsarWorkflow_ALL.pro` 中直接修改主影像、辅影像、DEM 和输出路径,然后运行:
```idl
.compile DinsarWorkflow_ALL.pro
DinsarWorkflow_ALL
```
## 8. 代码实现要点
### 8.1 自动读取 SML 参数
`Read_SML_Parameter` 会直接从 `.sml` 文件中读取指定 XML 标签内容,用于自动计算多视参数。这意味着:
- 输入数据必须已经正确导入并生成 `.sml`
- `.sml` 内部需要包含 `PixelSpacingRg`、`PixelSpacingAz`、`IncidenceAngle` 等字段
### 8.2 自动 GCP 生成策略
`CREATE_GCPS_FROM_COHERENCE` 不是随机取点,而是“规则网格 + 每格取最大相干值”的策略。这样做的优点是:
- GCP 空间分布更均匀
- 能避免点位过度集中
- 能通过相干阈值控制稳定区域
### 8.3 批处理脚本的选择策略
当前批处理版本在每个 `master` / `slave` 目录中只取“第一个找到的 `.sml`”作为输入,因此建议:
- 每个 `master` 目录内只保留一组待处理主影像
- 每个 `slave` 目录内只保留一组待处理辅影像
- 若目录中存在多份 `.sml`,需要先确认 `FILE_SEARCH` 返回顺序是否符合预期
## 9. 注意事项
1. 路径需要按本机环境手动修改,仓库中的路径示例均为作者本地路径。
2. 脚本默认使用 `ENVI(/HEADLESS)`,适合批处理环境。
3. `temporary_directory` 需要保证有足够磁盘空间。
4. `gcp_coh_threshold` 过高时,可能导致无法生成足够 GCP,流程会在 Step 5A 失败。
5. `unwrapping_coh_threshold`、`geocoding_coh_threshold` 需要结合研究区相干性调整。
6. 本仓库目前没有统一的日志输出、配置文件和异常恢复机制,失败后通常需要根据控制台信息排查。
@@ -0,0 +1,160 @@
PRO Run_Batch_Task_Import
;===========================================================================
; --- 用户配置区域 ---
; 1. **数据根目录 (rootDir)**
; 指定包含所有Task_开头文件夹的根目录路径。
; !!! 重要: 路径末尾不要带斜杠,脚本会自动添加。
; - 正确示例: 'C:\MyData\LuTan1' 或 '\\SERVER\Share\Data'
; 2. **处理数量 (numToProcess)**
; 指定要处理的Task文件夹的最大数量。
; - 如果设置为 0 或一个负数, 程序将处理所有找到的文件夹。
;===========================================================================
temporary_directory = 'D:/Sarscape_IDL_Area'
SARscape_Batch_Init,Temp_Directory=temporary_directory
rootDir = 'K:/T2'
numToProcess = 0
;===========================================================================
; --- 脚本执行区域 ---
;===========================================================================
pathSep = ''
IF (!VERSION.OS_FAMILY EQ 'Windows') THEN pathSep = '\' ELSE pathSep = '/'
e = ENVI(/HEADLESS)
IF rootDir EQ '' OR FILE_TEST(rootDir, /DIRECTORY) EQ 0 THEN BEGIN
PRINT, '错误: "rootDir" 参数未设置或指定的目录不存在: ' + rootDir
RETURN
ENDIF
PRINT, '=============================================='
PRINT, '开始批量导入陆探一号数据...'
PRINT, '扫描目录: ' + rootDir
PRINT, '=============================================='
; 查找所有Task_开头的文件夹 - 使用兼容的方法
PRINT, '正在扫描Task_开头的文件夹...'
; 兼容旧版本IDL的方法:先搜索所有匹配项,然后筛选出文件夹
allItems = FILE_SEARCH(rootDir + pathSep + 'Task_*', COUNT=nTotal)
taskFolders = []
nTasks = 0
IF nTotal GT 0 THEN BEGIN
FOR i = 0, nTotal - 1 DO BEGIN
IF FILE_TEST(allItems[i], /DIRECTORY) THEN BEGIN
taskFolders = [taskFolders, allItems[i]]
nTasks = nTasks + 1
ENDIF
ENDFOR
ENDIF
IF nTasks EQ 0 THEN BEGIN
PRINT, '在指定目录下没有找到任何Task_开头的文件夹。'
RETURN
ENDIF
PRINT, '共找到 ' + STRING(nTasks) + ' 个Task文件夹,准备开始处理...'
PRINT, ''
processed_count = 0
; 使用标准的FOR循环替代FOREACH以确保兼容性
FOR taskIndex = 0, nTasks - 1 DO BEGIN
taskFolder = taskFolders[taskIndex]
IF (numToProcess GT 0) AND (processed_count GE numToProcess) THEN BEGIN
PRINT, ''
PRINT, '已达到指定的处理数量 (' + STRING(numToProcess) + '),程序终止。'
BREAK
ENDIF
taskName = FILE_BASENAME(taskFolder)
PRINT, '=== 正在处理任务文件夹: ' + taskName + ' ==='
; 检查master和slave子文件夹
masterDir = taskFolder + pathSep + 'master'
slaveDir = taskFolder + pathSep + 'slave'
; 处理master文件夹
IF FILE_TEST(masterDir, /DIRECTORY) THEN BEGIN
PRINT, '--- 处理master文件夹 ---'
processed_count = processed_count + Process_LuTan_Folder(masterDir, pathSep)
ENDIF ELSE BEGIN
PRINT, '警告: ' + taskName + ' 中未找到master文件夹'
ENDELSE
; 处理slave文件夹
IF FILE_TEST(slaveDir, /DIRECTORY) THEN BEGIN
PRINT, '--- 处理slave文件夹 ---'
processed_count = processed_count + Process_LuTan_Folder(slaveDir, pathSep)
ENDIF ELSE BEGIN
PRINT, '警告: ' + taskName + ' 中未找到slave文件夹'
ENDELSE
PRINT, '完成处理任务文件夹: ' + taskName
PRINT, ''
ENDFOR
PRINT, '=============================================='
PRINT, '所有任务已完成!'
PRINT, '总共成功处理了 ' + STRING(processed_count) + ' 个数据文件夹。'
PRINT, '=============================================='
END
; 处理单个陆探一号数据文件夹的子函数
FUNCTION Process_LuTan_Folder, folderPath, pathSep
COMPILE_OPT STRICTARR
baseName = FILE_BASENAME(folderPath)
PRINT, '-> 正在检查文件夹: ' + baseName
; 直接使用master/slave文件夹作为输出目录,不再创建envi_import子目录
outputDir = folderPath
; 检查是否已经处理过(通过检查是否存在.sml文件)
check_sml_file = FILE_SEARCH(outputDir + pathSep + '*.sml', COUNT=sml_count)
IF sml_count GT 0 THEN BEGIN
PRINT, '-> 跳过: ' + baseName + ' 文件夹中已存在处理结果。'
RETURN, 0
ENDIF
; 查找元数据文件 - 查找文件夹中所有的.meta.xml文件
metaFiles = FILE_SEARCH(folderPath + pathSep + '*.meta.xml', COUNT=nMetaFiles)
IF nMetaFiles EQ 0 THEN BEGIN
PRINT, '-> 警告: 在文件夹 ' + baseName + ' 中未找到元数据文件(*.meta.xml),跳过。'
RETURN, 0
ENDIF
; 处理找到的元数据文件
success_count = 0
FOR i = 0, nMetaFiles - 1 DO BEGIN
inputFile = metaFiles[i]
metaFileName = FILE_BASENAME(inputFile)
PRINT, '-> 准备导入: ' + metaFileName
; 执行导入任务
oSB = obj_new('SARscapeBatch')
task = ENVITask('SARsImportLuTan1')
task.INPUT_FILE_LIST = [inputFile]
task.ROOT_URI_FOR_OUTPUT = outputDir ; 直接输出到master/slave文件夹
CATCH, error_status
IF error_status NE 0 THEN BEGIN
PRINT, '-> 错误: 在处理 ' + metaFileName + ' 时发生严重错误。'
PRINT, !ERROR_STATE.MSG
CATCH, /CANCEL
CONTINUE
ENDIF
task.Execute
CATCH, /CANCEL
PRINT, '-> 成功: 数据已导入到 ' + baseName + ' 文件夹中。'
success_count = success_count + 1
ENDFOR
RETURN, success_count
END
@@ -0,0 +1,488 @@
;================================================================================
; ### HELPER FUNCTION: READ SML PARAMETER ###
;================================================================================
FUNCTION Read_SML_Parameter, sml_file, param_name_to_find
IF N_ELEMENTS(sml_file) EQ 0 OR sml_file EQ '' THEN BEGIN
PRINT, 'Error: An empty or undefined filename was passed to Read_SML_Parameter.'
RETURN, ''
ENDIF
IF NOT FILE_TEST(sml_file) THEN BEGIN
PRINT, 'Error: SML file not found -> ' + sml_file
RETURN, ''
ENDIF
lun = -1
ON_ERROR, 2
OPENR, lun, sml_file, /GET_LUN, ERROR=err
IF (err NE 0) THEN BEGIN
PRINT, 'Error: Could not open SML file for reading -> ' + sml_file
IF lun NE -1 THEN FREE_LUN, lun
RETURN, ''
ENDIF
found_value = ''
line = ''
start_tag = '<' + STRUPCASE(param_name_to_find) + '>'
end_tag = '</' + STRUPCASE(param_name_to_find) + '>'
WHILE NOT EOF(lun) DO BEGIN
READF, lun, line
clean_line = STRTRIM(line, 2)
line_upper = STRUPCASE(clean_line)
start_pos = STRPOS(line_upper, start_tag)
IF start_pos NE -1 THEN BEGIN
end_pos = STRPOS(line_upper, end_tag, start_pos)
IF end_pos NE -1 THEN BEGIN
start_extract = start_pos + STRLEN(start_tag)
value_length = end_pos - start_extract
found_value = STRMID(clean_line, start_extract, value_length)
BREAK
ENDIF
ENDIF
ENDWHILE
IF lun NE -1 THEN FREE_LUN, lun
RETURN, found_value
END
;================================================================================
; ### HELPER FUNCTION: CREATE GCPs FROM COHERENCE ###
;================================================================================
FUNCTION CREATE_GCPS_FROM_COHERENCE, coherence_file, output_shp_file, COH_THRESHOLD=coh_threshold, NUM_POINTS=num_points
ON_ERROR, 2
IF FILE_TEST(output_shp_file) THEN FILE_DELETE, output_shp_file, /QUIET
IF N_ELEMENTS(coh_threshold) EQ 0 THEN coh_threshold = 0.7
IF N_ELEMENTS(num_points) EQ 0 THEN num_points = 100
e = ENVI(/HEADLESS)
PRINT, 'Opening coherence file: ' + coherence_file
oRaster = e.OpenRaster(coherence_file)
ns = oRaster.NCOLUMNS
nl = oRaster.NROWS
data = oRaster.GetData(BANDS=[0])
PRINT, 'Searching for GCPs with coherence > ' + STRTRIM(coh_threshold, 2)
grid_dim = CEIL(SQRT(num_points))
x_step = FLOOR(ns / grid_dim)
y_step = FLOOR(nl / grid_dim)
pixel_points_x = [] & pixel_points_y = []
FOR j=0L, grid_dim-1 DO BEGIN
FOR i=0L, grid_dim-1 DO BEGIN
x_start = i * x_step & y_start = j * y_step
x_end = (i+1) * x_step -1 < (ns-1)
y_end = (j+1) * y_step -1 < (nl-1)
IF (x_start GE ns) OR (y_start GE nl) THEN CONTINUE
cell_data = data[x_start:x_end, y_start:y_end]
max_val = MAX(cell_data, max_idx)
IF max_val GE coh_threshold THEN BEGIN
max_y_cell = max_idx / (x_end - x_start + 1)
max_x_cell = max_idx MOD (x_end - x_start + 1)
pixel_points_x = [pixel_points_x, x_start + max_x_cell]
pixel_points_y = [pixel_points_y, y_start + max_y_cell]
ENDIF
ENDFOR
ENDFOR
point_count = N_ELEMENTS(pixel_points_x)
IF point_count EQ 0 THEN BEGIN
PRINT, 'Error: No GCPs found.' & OBJ_DESTROY, oRaster & RETURN, 0
ENDIF
PRINT, 'Found ' + STRTRIM(point_count, 2) + ' suitable GCPs.'
PRINT, 'Creating shapefile using final authoritative methodology...'
oShape = OBJ_NEW('IDLffShape', output_shp_file, ENTITY_TYPE=1, /UPDATE)
oShape->AddAttribute, 'SHP_ID', 3, 10
oShape->AddAttribute, 'GCP_LABEL', 7, 20
oShape->AddAttribute, 'GCP_TYPE', 7, 20
oShape->AddAttribute, 'GCP_COLUMN', 5, 24, PRECISION=6
oShape->AddAttribute, 'GCP_ROW', 5, 24, PRECISION=6
oShape->AddAttribute, 'GCP_OTHER_', 7, 10 ; User-verified field name
attr_template = oShape->GetAttributes(/ATTRIBUTE_STRUCTURE)
entNew = {IDL_SHAPE_ENTITY}
entNew.SHAPE_TYPE = 1
FOR i=0, point_count-1 DO BEGIN
x_float = FLOAT(pixel_points_x[i])
y_float = FLOAT(pixel_points_y[i])
entNew.ISHAPE = i
entNew.BOUNDS = [x_float, y_float, 0.0, 0.0, x_float, y_float, 0.0, 0.0]
oShape->PutEntity, entNew
attr = attr_template
attr.ATTRIBUTE_0 = i
attr.ATTRIBUTE_1 = 'GCP_' + STRTRIM(i+1, 2)
attr.ATTRIBUTE_2 = 'undefined'
attr.ATTRIBUTE_3 = x_float
attr.ATTRIBUTE_4 = y_float
attr.ATTRIBUTE_5 = ''
oShape->SetAttributes, i, attr
ENDFOR
oShape->Close
OBJ_DESTROY, [oShape, oRaster]
PRINT, 'Successfully created GCP shapefile: ' + output_shp_file
RETURN, 1
END
;================================================================================
; ### 执行单个D-InSAR工作流的函数 ###
;================================================================================
FUNCTION Execute_Single_Dinsar_Workflow, master_base_file, slave_base_file, dem_base_file, workflow_root_name, $
target_ground_resolution_m, filter_method, unwrapping_coh_threshold, $
gcp_coh_threshold, gcp_number, geocoding_coh_threshold, geocoding_pixel_size_m
; ===================================================================
; ### STEP 1: INTERFEROGRAM GENERATION ###
; ===================================================================
PRINT, '======================================================'
PRINT, '### STEP 1: INTERFEROGRAM GENERATION ###'
PRINT, '======================================================'
master_sml = master_base_file + '.sml'
slave_sml = slave_base_file + '.sml'
m_rg_sp_str = Read_SML_Parameter(master_sml, 'PixelSpacingRg')
m_az_sp_str = Read_SML_Parameter(master_sml, 'PixelSpacingAz')
m_inc_ang_str = Read_SML_Parameter(master_sml, 'IncidenceAngle')
s_rg_sp_str = Read_SML_Parameter(slave_sml, 'PixelSpacingRg')
s_az_sp_str = Read_SML_Parameter(slave_sml, 'PixelSpacingAz')
s_inc_ang_str = Read_SML_Parameter(slave_sml, 'IncidenceAngle')
IF m_rg_sp_str EQ '' OR s_rg_sp_str EQ '' THEN RETURN, 0
m_rg_sp = FLOAT(m_rg_sp_str) & m_az_sp = FLOAT(m_az_sp_str) & m_inc_ang = FLOAT(m_inc_ang_str)
s_rg_sp = FLOAT(s_rg_sp_str) & s_az_sp = FLOAT(s_az_sp_str) & s_inc_ang = FLOAT(s_inc_ang_str)
avg_az_spacing = (m_az_sp + s_az_sp) / 2.0
azimuth_looks = (LONG(target_ground_resolution_m / avg_az_spacing)) > 1
m_ground_rg_sp = m_rg_sp / SIN(m_inc_ang * !DPI / 180.0)
s_ground_rg_sp = s_rg_sp / SIN(s_inc_ang * !DPI / 180.0)
avg_ground_rg_sp = (m_ground_rg_sp + s_ground_rg_sp) / 2.0
range_looks = (LONG(target_ground_resolution_m / avg_ground_rg_sp)) > 1
task1_obj = OBJ_NEW('SARscapeBatch', Module='InSARInterferogramGeneration')
prefix_1 = 'MAIN_INSAR_INTERFEROGRAM_GENERATION_CMD.'
task1_obj->SetParam, prefix_1 + 'INPUT_REFERENCE_FILE_NAME', master_base_file
task1_obj->SetParam, prefix_1 + 'INPUT_SECONDARY_FILE_NAME', slave_base_file
task1_obj->SetParam, prefix_1 + 'DEM_FILE_NAME', dem_base_file
task1_obj->SetParam, prefix_1 + 'OUTPUT_ROOT_FILE_NAME', workflow_root_name
task1_obj->SetParam, prefix_1 + 'RG_LOOKS_NBR', STRING(range_looks)
task1_obj->SetParam, prefix_1 + 'AZ_LOOKS_NBR', STRING(azimuth_looks)
task1_obj->SetParam, 'COREGISTRATION_CMD.COREGISTRATION_WITH_DEM_FLAG', 'OK'
task1_obj->SetParam, 'FLAT_CMD.MAKE_FLATTENING_FLAG', 'OK'
ok_step1 = task1_obj->Execute()
OBJ_DESTROY, task1_obj
IF NOT ok_step1 THEN RETURN, 0
PRINT, '### STEP 1: SUCCESS!'
; ===================================================================
; ### STEP 2: FILTERING AND COHERENCE GENERATION ###
; ===================================================================
PRINT, '======================================================'
PRINT, '### STEP 2: FILTERING AND COHERENCE GENERATION ###'
PRINT, '======================================================'
task2_obj = OBJ_NEW('SARscapeBatch', Module='InSARFilterAndCoherence')
prefix_main2 = 'MAIN_INSAR_FILTER_COHERENCE_CMD.'
prefix_filt2 = 'FILTERING_CMD.'
task2_obj->SetParam, prefix_main2 + 'INPUT_INTERF_FILE_NAME', workflow_root_name + '_dint'
task2_obj->SetParam, prefix_main2 + 'INPUT_REFERENCE_FILE_NAME', workflow_root_name + '_reference_pwr'
task2_obj->SetParam, prefix_main2 + 'INPUT_SECONDARY_FILE_NAME', workflow_root_name + '_secondary_pwr'
task2_obj->SetParam, prefix_main2 + 'OUT_ROOT_FILE', workflow_root_name
task2_obj->SetParam, prefix_filt2 + 'FILTERING_METHOD', filter_method
task2_obj->SetParam, prefix_main2 + 'COHERENCE_FLAG', 'OK'
task2_obj->SetParam, prefix_main2 + 'INTERF_FILT_FLAG', 'OK'
ok_step2 = task2_obj->Execute()
OBJ_DESTROY, task2_obj
IF NOT ok_step2 THEN RETURN, 0
PRINT, '### STEP 2: SUCCESS!'
; ===================================================================
; ### STEP 3: ORBITAL TREND REMOVAL ###
; ===================================================================
PRINT, '======================================================'
PRINT, '### STEP 3: ORBITAL TREND REMOVAL ###'
PRINT, '======================================================'
reflat_output_root = workflow_root_name + '_rrpf'
task3_obj = OBJ_NEW('SARscapeBatch', Module='InSARRemoveResidualPhaseFrequency')
prefix_3 = 'MAIN_INSAR_REMOVE_RESIDUAL_PHASE_FREQUENCY.'
task3_obj->SetParam, prefix_3 + 'INTERF_FILE_NAME', workflow_root_name + '_fint'
task3_obj->SetParam, prefix_3 + 'COHERENCE_FILE_NAME', workflow_root_name + '_cc'
task3_obj->SetParam, prefix_3 + 'OUT_ROOT_FILE_NAME', reflat_output_root
ok_step3 = task3_obj->Execute()
OBJ_DESTROY, task3_obj
IF NOT ok_step3 THEN RETURN, 0
PRINT, '### STEP 3: SUCCESS!'
; ===================================================================
; ### STEP 4: PHASE UNWRAPPING ###
; ===================================================================
PRINT, '======================================================'
PRINT, '### STEP 4: PHASE UNWRAPPING ###'
PRINT, '======================================================'
reflat_upha_output_file = workflow_root_name + '_upha'
task4_obj = OBJ_NEW('SARscapeBatch', Module='InSARPhaseUnwrapping')
prefix_main4 = 'MAIN_INSAR_PHASE_UNWRAPPING_CMD.'
prefix_upha4 = 'UPHA_CMD.'
task4_obj->SetParam, prefix_main4 + 'INFILE_NAME', reflat_output_root + '_fint'
task4_obj->SetParam, prefix_main4 + 'COHERENCEFILE_NAME', workflow_root_name + '_cc'
task4_obj->SetParam, prefix_upha4 + 'UPHA_COH_THRESHOLD', STRING(unwrapping_coh_threshold)
task4_obj->SetParam, prefix_main4 + 'OUTFILE_NAME', reflat_upha_output_file
ok_step4 = task4_obj->Execute()
OBJ_DESTROY, task4_obj
IF NOT ok_step4 THEN RETURN, 0
PRINT, '### STEP 4: SUCCESS!'
; ===================================================================
; ### STEP 5: REFINEMENT AND REFLATTENING ###
; ===================================================================
; --- STEP 5A: Automatic GCP Generation ---
PRINT, '======================================================'
PRINT, '### STEP 5A: 正在自动生成GCP... ###'
PRINT, '======================================================'
auto_gcp_shp = workflow_root_name + '_auto_gcp.shp'
gcp_success = CREATE_GCPS_FROM_COHERENCE(workflow_root_name + '_cc', auto_gcp_shp, COH_THRESHOLD=gcp_coh_threshold, NUM_POINTS=gcp_number)
IF gcp_success EQ 0 THEN BEGIN
PRINT, '错误: 自动生成GCP shapefile失败。流程终止。'
RETURN, 0
ENDIF
PRINT, '### STEP 5A: SUCCESS!'
; --- STEP 5B: Refinement and Reflattening Execution ---
PRINT, '======================================================'
PRINT, '### STEP 5B: 正在执行 REFINEMENT AND REFLATTENING ###'
PRINT, '======================================================'
ref_output_root = workflow_root_name + '_reflat'
task_refine = OBJ_NEW('SARscapeBatch', Module='InSARRefinementAndReflattening')
prefix_refine = 'MAIN_INSAR_REFINEMENT_AND_REFLATTENING_CMD.'
task_refine->SetParam, prefix_refine + 'INPUT_UPHA_FILE_NAME', reflat_upha_output_file
task_refine->SetParam, prefix_refine + 'INPUT_REFERENCE_FILE_NAME', workflow_root_name + '_reference_pwr'
task_refine->SetParam, prefix_refine + 'INPUT_SECONDARY_FILE_NAME', workflow_root_name + '_secondary_pwr'
task_refine->SetParam, prefix_refine + 'SLANT_RANGE_DEM_FILE_NAME', workflow_root_name + '_srdem'
task_refine->SetParam, prefix_refine + 'SYNTHETIC_FILE_NAME', workflow_root_name + '_sint'
task_refine->SetParam, prefix_refine + 'COHERENCE_FILE_NAME', workflow_root_name + '_cc'
task_refine->SetParam, prefix_refine + 'OUTPUT_ROOT_NAME', ref_output_root
task_refine->SetParam, prefix_refine + 'DEM_FILE_NAME', dem_base_file
task_refine->SetParam, 'REFINEMENT_CMD.REFINEMENT_GCP_FILE_NAME', auto_gcp_shp
ok_step5 = task_refine->Execute()
OBJ_DESTROY, task_refine
IF NOT ok_step5 THEN RETURN, 0
PRINT, '### STEP 5B: SUCCESS!'
; ===================================================================
; ### STEP 6: PHASE TO DISPLACEMENT AND GEOCODING ###
; ===================================================================
PRINT, '======================================================'
PRINT, '### STEP 6: PHASE TO DISPLACEMENT AND GEOCODING ###'
PRINT, '======================================================'
final_upha_file = ref_output_root + '_upha'
geocoded_output_root = workflow_root_name + '_geo'
task6_obj = OBJ_NEW('SARscapeBatch', Module='InSARPhaseToDisplacement')
IF (~OBJ_VALID(task6_obj)) THEN BEGIN
PRINT, '错误: 创建 SARscapeBatch 对象失败: InSARPhaseToDisplacement'
RETURN, 0
ENDIF
; --- Set Main Parameters ---
prefix_main6 = 'MAIN_INSAR_PHASE_TO_DISPLACEMENT_CMD.'
task6_obj->SetParam, prefix_main6 + 'INPUT_FILE_NAME', final_upha_file
task6_obj->SetParam, prefix_main6 + 'OUTPUT_FILE', geocoded_output_root
task6_obj->SetParam, prefix_main6 + 'COHERENCE_FILE', workflow_root_name + '_cc'
task6_obj->SetParam, prefix_main6 + 'DEM_FILE_NAME', dem_base_file
task6_obj->SetParam, prefix_main6 + 'COHERENCE_THRESHOLD', STRING(geocoding_coh_threshold)
; --- Set Other Required Parameters ---
task6_obj->SetParam, prefix_main6 + 'INTERPOL_TYPE' , '4th_order_cc'
task6_obj->SetParam, prefix_main6 + 'ALLOW_SKIP_REFINEMENT' , 'NotOK'
task6_obj->SetParam, prefix_main6 + 'SARSCAPEENVIRONMENT' , 'IDL_ENVI_ENV'
; --- Set Geocoding Parameters ---
prefix_geocode6 = 'GEOCODE_CMD.'
task6_obj->SetParam, prefix_geocode6 + 'GEOCODE_RG_GRID_SIZE', STRING(geocoding_pixel_size_m)
task6_obj->SetParam, prefix_geocode6 + 'GEOCODE_AZ_GRID_SIZE', STRING(geocoding_pixel_size_m)
task6_obj->SetParam, prefix_geocode6 + 'GEOCODE_INTERPOL_BOX_SIZE', '7'
; --- Set Displacement Product Generation Parameters ---
prefix_disp6 = 'DISPLACEMENT_PROJECTION_CMD.'
task6_obj->SetParam, prefix_disp6 + 'GENERATE_LOS_FLAG', 'OK'
task6_obj->SetParam, prefix_disp6 + 'GENERATE_VERTICAL_FLAG', 'NotOK'
task6_obj->SetParam, prefix_disp6 + 'GENERATE_MAX_SLOPE_FLAG', 'NotOK'
PRINT, '正在执行地理编码任务...'
ok_step6 = task6_obj->Execute()
OBJ_DESTROY, task6_obj
IF NOT ok_step6 THEN RETURN, 0
PRINT, '### STEP 6: SUCCESS!'
; --- Final Success Message ---
PRINT, '======================================================='
PRINT, ' FULL D-InSAR WORKFLOW (STEPS 1-6) COMPLETED SUCCESSFULLY!'
PRINT, '======================================================='
RETURN, 1
END
;================================================================================
; ### 批量D-InSAR工作流主程序 ###
;================================================================================
PRO Batch_DinsarWorkflow_ALL
; --- 初始化SARscape环境 ---
temporary_directory = 'D:\Sarscape_IDL_Area'
SARscape_Batch_Init, Temp_Directory=temporary_directory
e = ENVI(/HEADLESS)
PRINT, 'ENVI 和 SARscape 环境初始化成功!'
PRINT, ''
; ===================================================================
; ### 全局配置 ###
; ===================================================================
rootDir = 'G:/Province_Result/new' ; 修改为您的根目录
dem_base_file = 'F:/Data/SRTM30m/SRTMDEM_RSP_SARscape' ; DEM文件路径
; 处理参数配置
target_ground_resolution_m = 10.0
filter_method = 'GOLDSTEIN'
unwrapping_coh_threshold = 0.05
gcp_coh_threshold = 0.7
gcp_number = 100
geocoding_coh_threshold = 0.0
geocoding_pixel_size_m = 10.0
; 最大处理数量 (0表示处理所有)
numToProcess = 0
; ===================================================================
; ### 查找并处理所有Task文件夹 ###
; ===================================================================
pathSep = ''
IF (!VERSION.OS_FAMILY EQ 'Windows') THEN pathSep = '\' ELSE pathSep = '/'
IF rootDir EQ '' OR FILE_TEST(rootDir, /DIRECTORY) EQ 0 THEN BEGIN
PRINT, '错误: "rootDir" 参数未设置或指定的目录不存在: ' + rootDir
RETURN
ENDIF
PRINT, '=============================================='
PRINT, '开始批量D-InSAR处理...'
PRINT, '扫描目录: ' + rootDir
PRINT, '=============================================='
; 查找所有Task_开头的文件夹
PRINT, '正在扫描Task_开头的文件夹...'
taskFolders = FILE_SEARCH(rootDir + pathSep + 'Task_*', /FOLDERS, COUNT=nTasks)
IF nTasks EQ 0 THEN BEGIN
PRINT, '在指定目录下没有找到任何Task_开头的文件夹。'
RETURN
ENDIF
PRINT, '共找到 ' + STRING(nTasks) + ' 个Task文件夹,准备开始处理...'
PRINT, ''
processed_count = 0
; 循环处理每个Task文件夹
FOREACH taskFolder, taskFolders DO BEGIN
IF (numToProcess GT 0) AND (processed_count GE numToProcess) THEN BEGIN
PRINT, ''
PRINT, '已达到指定的处理数量 (' + STRING(numToProcess) + '),程序终止。'
BREAK
ENDIF
taskName = FILE_BASENAME(taskFolder)
PRINT, '=== 正在处理任务文件夹: ' + taskName + ' ==='
; 检查master和slave子文件夹
masterDir = taskFolder + pathSep + 'master'
slaveDir = taskFolder + pathSep + 'slave'
IF (NOT FILE_TEST(masterDir, /DIRECTORY)) OR (NOT FILE_TEST(slaveDir, /DIRECTORY)) THEN BEGIN
PRINT, '警告: ' + taskName + ' 中缺少master或slave文件夹,跳过。'
PRINT, ''
CONTINUE
ENDIF
; 查找master和slave文件夹中的SML文件
masterSmlFiles = FILE_SEARCH(masterDir + pathSep + '*.sml', COUNT=nMaster)
slaveSmlFiles = FILE_SEARCH(slaveDir + pathSep + '*.sml', COUNT=nSlave)
IF (nMaster EQ 0) OR (nSlave EQ 0) THEN BEGIN
PRINT, '警告: ' + taskName + ' 中未找到足够的SML文件 (master: ' + STRING(nMaster) + ', slave: ' + STRING(nSlave) + '),跳过。'
PRINT, ''
CONTINUE
ENDIF
; 取第一个找到的SML文件作为主从影像
master_base_file = masterSmlFiles[0]
slave_base_file = slaveSmlFiles[0]
; 移除.sml扩展名,得到基础文件名
master_base_file = STRMID(master_base_file, 0, STRLEN(master_base_file) - 4)
slave_base_file = STRMID(slave_base_file, 0, STRLEN(slave_base_file) - 4)
; 设置工作流输出根名称(在Task文件夹内创建结果)
workflow_root_name = taskFolder + pathSep + 'dinsar_results' + pathSep + 'workflow'
; 创建输出目录
outputDir = FILE_DIRNAME(workflow_root_name)
IF NOT FILE_TEST(outputDir, /DIRECTORY) THEN BEGIN
FILE_MKDIR, outputDir
PRINT, '-> 创建输出目录: ' + outputDir
ENDIF
PRINT, '-> Master文件: ' + FILE_BASENAME(master_base_file)
PRINT, '-> Slave文件: ' + FILE_BASENAME(slave_base_file)
PRINT, '-> 输出路径: ' + workflow_root_name
PRINT, ''
; 执行单个D-InSAR工作流
success = Execute_Single_Dinsar_Workflow(master_base_file, slave_base_file, dem_base_file, workflow_root_name, $
target_ground_resolution_m, filter_method, unwrapping_coh_threshold, $
gcp_coh_threshold, gcp_number, geocoding_coh_threshold, geocoding_pixel_size_m)
IF success THEN BEGIN
PRINT, '### ' + taskName + ': D-InSAR工作流完成! ###'
processed_count = processed_count + 1
ENDIF ELSE BEGIN
PRINT, '### ' + taskName + ': D-InSAR工作流失败! ###'
ENDELSE
PRINT, ''
ENDFOREACH
PRINT, '=============================================='
PRINT, '批量D-InSAR处理完成!'
PRINT, '总共成功处理了 ' + STRING(processed_count) + ' 个Task文件夹。'
PRINT, '=============================================='
SARscape_Batch_Exit
END
+85 -22
View File
@@ -94,6 +94,51 @@ function Get-ListeningProcessId {
return 0 return 0
} }
function Invoke-UiShutdown {
param(
[string]$Url,
[string]$ShutdownToken
)
if ([string]::IsNullOrWhiteSpace($Url) -or [string]::IsNullOrWhiteSpace($ShutdownToken)) {
return $false
}
try {
$headers = @{
"X-S1DL-Shutdown-Token" = $ShutdownToken
}
$shutdownUrl = $Url.TrimEnd("/") + "/api/shutdown"
$null = Invoke-RestMethod -Uri $shutdownUrl -Method Post -Headers $headers -TimeoutSec 5 -ErrorAction Stop
return $true
} catch {
return $false
}
}
function Wait-ForUiShutdown {
param(
[int]$ProcessId,
[int]$Port,
[int]$TimeoutSeconds = 10
)
$deadline = (Get-Date).AddSeconds($TimeoutSeconds)
while ((Get-Date) -lt $deadline) {
$running = Get-RunningProcess -Id $ProcessId
$listenerProcessId = 0
if ($Port -gt 0) {
$listenerProcessId = Get-ListeningProcessId -Port $Port
}
if (-not $running -and $listenerProcessId -le 0) {
return $true
}
Start-Sleep -Milliseconds 500
}
return $false
}
function Load-State { function Load-State {
param([string]$Path) param([string]$Path)
@@ -113,14 +158,16 @@ function Save-State {
[string]$Path, [string]$Path,
[int]$ProcessId, [int]$ProcessId,
[int]$Port, [int]$Port,
[string]$Url [string]$Url,
[string]$ShutdownToken
) )
$payload = [ordered]@{ $payload = [ordered]@{
pid = $ProcessId pid = $ProcessId
port = $Port port = $Port
url = $Url url = $Url
started_at = (Get-Date).ToString("s") shutdown_token = $ShutdownToken
started_at = (Get-Date).ToString("s")
} }
$payload | ConvertTo-Json | Set-Content -LiteralPath $Path -Encoding UTF8 $payload | ConvertTo-Json | Set-Content -LiteralPath $Path -Encoding UTF8
@@ -157,11 +204,21 @@ if ($state -and -not $ForceRestart) {
} }
if ($ForceRestart -and $state) { if ($ForceRestart -and $state) {
$oldProcess = Get-RunningProcess -Id ([int]$state.pid) $existingStopped = $false
if ($oldProcess) { if ($state.url -and $state.shutdown_token) {
Write-Info "Stopping previous tracked process $($oldProcess.Id)" Write-Info "Requesting shutdown for previous tracked UI"
Stop-Process -Id $oldProcess.Id -Force if (Invoke-UiShutdown -Url $state.url -ShutdownToken $state.shutdown_token) {
Start-Sleep -Seconds 1 $existingStopped = Wait-ForUiShutdown -ProcessId ([int]$state.pid) -Port ([int]$state.port)
}
}
if (-not $existingStopped) {
$oldProcess = Get-RunningProcess -Id ([int]$state.pid)
if ($oldProcess) {
Write-Info "Stopping previous tracked process $($oldProcess.Id)"
Stop-Process -Id $oldProcess.Id -Force
Start-Sleep -Seconds 1
}
} }
Remove-State -Path $statePath Remove-State -Path $statePath
} }
@@ -183,7 +240,7 @@ foreach ($port in $PreferredPorts) {
if (Test-Health -Port $port) { if (Test-Health -Port $port) {
$url = "http://127.0.0.1:$port/" $url = "http://127.0.0.1:$port/"
Write-Info "Found an existing running UI at $url" Write-Info "Found an existing running UI at $url"
Save-State -Path $statePath -ProcessId 0 -Port $port -Url $url Save-State -Path $statePath -ProcessId 0 -Port $port -Url $url -ShutdownToken ""
if (-not $NoBrowser) { if (-not $NoBrowser) {
Start-Process $url | Out-Null Start-Process $url | Out-Null
} }
@@ -206,6 +263,7 @@ if (-not (Test-Path -LiteralPath $pythonPath)) {
$portToUse = Get-FreePort -Candidates $PreferredPorts $portToUse = Get-FreePort -Candidates $PreferredPorts
$url = "http://127.0.0.1:$portToUse/" $url = "http://127.0.0.1:$portToUse/"
$shutdownToken = [guid]::NewGuid().ToString("N")
if (Test-Path -LiteralPath $stdoutPath) { if (Test-Path -LiteralPath $stdoutPath) {
Remove-Item -LiteralPath $stdoutPath -Force Remove-Item -LiteralPath $stdoutPath -Force
@@ -225,14 +283,19 @@ $arguments = @(
) )
Write-Info "Starting UI at $url" Write-Info "Starting UI at $url"
$process = Start-Process ` try {
-FilePath $pythonPath ` $env:S1DL_UI_SHUTDOWN_TOKEN = $shutdownToken
-ArgumentList $arguments ` $process = Start-Process `
-WorkingDirectory $projectRoot ` -FilePath $pythonPath `
-WindowStyle Hidden ` -ArgumentList $arguments `
-RedirectStandardOutput $stdoutPath ` -WorkingDirectory $projectRoot `
-RedirectStandardError $stderrPath ` -WindowStyle Hidden `
-PassThru -RedirectStandardOutput $stdoutPath `
-RedirectStandardError $stderrPath `
-PassThru
} finally {
Remove-Item Env:S1DL_UI_SHUTDOWN_TOKEN -ErrorAction SilentlyContinue
}
$started = $false $started = $false
for ($i = 0; $i -lt 30; $i++) { for ($i = 0; $i -lt 30; $i++) {
@@ -261,12 +324,12 @@ if (-not $started) {
throw "UI failed to start." throw "UI failed to start."
} }
Save-State -Path $statePath -ProcessId $process.Id -Port $portToUse -Url $url Save-State -Path $statePath -ProcessId $process.Id -Port $portToUse -Url $url -ShutdownToken $shutdownToken
$listenerProcessId = Get-ListeningProcessId -Port $portToUse $listenerProcessId = Get-ListeningProcessId -Port $portToUse
if ($listenerProcessId -gt 0) { if ($listenerProcessId -gt 0) {
Save-State -Path $statePath -ProcessId $listenerProcessId -Port $portToUse -Url $url Save-State -Path $statePath -ProcessId $listenerProcessId -Port $portToUse -Url $url -ShutdownToken $shutdownToken
} else { } else {
Save-State -Path $statePath -ProcessId $process.Id -Port $portToUse -Url $url Save-State -Path $statePath -ProcessId $process.Id -Port $portToUse -Url $url -ShutdownToken $shutdownToken
} }
Write-Info "UI is ready: $url" Write-Info "UI is ready: $url"
+83 -12
View File
@@ -44,6 +44,51 @@ function Get-ListeningProcessId {
return 0 return 0
} }
function Invoke-UiShutdown {
param(
[string]$Url,
[string]$ShutdownToken
)
if ([string]::IsNullOrWhiteSpace($Url) -or [string]::IsNullOrWhiteSpace($ShutdownToken)) {
return $false
}
try {
$headers = @{
"X-S1DL-Shutdown-Token" = $ShutdownToken
}
$shutdownUrl = $Url.TrimEnd("/") + "/api/shutdown"
$null = Invoke-RestMethod -Uri $shutdownUrl -Method Post -Headers $headers -TimeoutSec 5 -ErrorAction Stop
return $true
} catch {
return $false
}
}
function Wait-ForUiShutdown {
param(
[int]$ProcessId,
[int]$Port,
[int]$TimeoutSeconds = 10
)
$deadline = (Get-Date).AddSeconds($TimeoutSeconds)
while ((Get-Date) -lt $deadline) {
$running = Get-RunningProcess -Id $ProcessId
$listenerProcessId = 0
if ($Port -gt 0) {
$listenerProcessId = Get-ListeningProcessId -Port $Port
}
if (-not $running -and $listenerProcessId -le 0) {
return $true
}
Start-Sleep -Milliseconds 500
}
return $false
}
$projectRoot = Split-Path -Parent $PSCommandPath $projectRoot = Split-Path -Parent $PSCommandPath
$statePath = Join-Path $projectRoot ".s1dl\web-ui-state.json" $statePath = Join-Path $projectRoot ".s1dl\web-ui-state.json"
@@ -53,23 +98,49 @@ if (-not (Test-Path -LiteralPath $statePath)) {
} }
$state = Get-Content -LiteralPath $statePath -Raw | ConvertFrom-Json $state = Get-Content -LiteralPath $statePath -Raw | ConvertFrom-Json
$process = Get-RunningProcess -Id ([int]$state.pid) $stopped = $false
if ($process) { if ($state.url -and $state.shutdown_token) {
Write-Info "Stopping UI process $($process.Id)" Write-Info "Requesting UI shutdown via API"
Stop-Process -Id $process.Id -Force if (Invoke-UiShutdown -Url $state.url -ShutdownToken $state.shutdown_token) {
} else { $stopped = Wait-ForUiShutdown -ProcessId ([int]$state.pid) -Port ([int]$state.port)
$listenerProcessId = 0
if ($state.port) {
$listenerProcessId = Get-ListeningProcessId -Port ([int]$state.port)
} }
if ($listenerProcessId -gt 0) { }
Write-Info "Stopping listener process $listenerProcessId"
Stop-Process -Id $listenerProcessId -Force if (-not $stopped) {
$process = Get-RunningProcess -Id ([int]$state.pid)
if ($process) {
Write-Info "Stopping UI process $($process.Id)"
try {
Stop-Process -Id $process.Id -Force -ErrorAction Stop
$stopped = $true
} catch {
throw "Failed to stop UI process $($process.Id). It may have been started from an elevated shell. Start or stop it with the same privilege level, or restart it with the updated launcher once so it can use API shutdown. $($_.Exception.Message)"
}
} else { } else {
Write-Info "Tracked process is not running." $listenerProcessId = 0
if ($state.port) {
$listenerProcessId = Get-ListeningProcessId -Port ([int]$state.port)
}
if ($listenerProcessId -gt 0) {
Write-Info "Stopping listener process $listenerProcessId"
try {
Stop-Process -Id $listenerProcessId -Force -ErrorAction Stop
$stopped = $true
} catch {
throw "Failed to stop UI listener process $listenerProcessId. It may have been started from an elevated shell. Start or stop it with the same privilege level, or restart it with the updated launcher once so it can use API shutdown. $($_.Exception.Message)"
}
} else {
Write-Info "Tracked process is not running."
$stopped = $true
}
} }
} }
if (-not $stopped) {
throw "UI shutdown did not complete."
}
Remove-Item -LiteralPath $statePath -Force Remove-Item -LiteralPath $statePath -Force
Write-Info "Done." Write-Info "Done."
@@ -14,7 +14,11 @@ from sentinel_orbit_downloader.api.routes_projects import router as projects_rou
from sentinel_orbit_downloader.api.routes_regions import router as regions_router from sentinel_orbit_downloader.api.routes_regions import router as regions_router
from sentinel_orbit_downloader.api.routes_search import router as search_router from sentinel_orbit_downloader.api.routes_search import router as search_router
from sentinel_orbit_downloader.api.routes_settings import router as settings_router from sentinel_orbit_downloader.api.routes_settings import router as settings_router
from sentinel_orbit_downloader.core.config import configured_api_host, configured_api_port from sentinel_orbit_downloader.core.config import (
configured_api_host,
configured_api_port,
configured_ui_shutdown_token,
)
def create_app() -> FastAPI: def create_app() -> FastAPI:
@@ -23,6 +27,7 @@ def create_app() -> FastAPI:
version="0.1.0", version="0.1.0",
description="Local API for Sentinel-1 search, orbit matching, and task management.", description="Local API for Sentinel-1 search, orbit matching, and task management.",
) )
app.state.shutdown_token = configured_ui_shutdown_token()
web_root = Path(__file__).resolve().parents[1] / "web" web_root = Path(__file__).resolve().parents[1] / "web"
app.mount("/static", StaticFiles(directory=web_root), name="static") app.mount("/static", StaticFiles(directory=web_root), name="static")
@@ -1,14 +1,42 @@
from __future__ import annotations from __future__ import annotations
from fastapi import APIRouter, Depends import os
import signal
import time
from fastapi import APIRouter, BackgroundTasks, Depends, Header, HTTPException, Request, status
from sentinel_orbit_downloader.api.deps import get_db_path from sentinel_orbit_downloader.api.deps import get_db_path
from sentinel_orbit_downloader.api.schemas import HealthResponse from sentinel_orbit_downloader.api.schemas import HealthResponse, MessageResponse
router = APIRouter(tags=["health"]) router = APIRouter(tags=["health"])
def _terminate_current_process() -> None:
time.sleep(0.2)
try:
os.kill(os.getpid(), signal.SIGTERM)
except Exception:
os._exit(0)
@router.get("/health", response_model=HealthResponse) @router.get("/health", response_model=HealthResponse)
def health(db_path=Depends(get_db_path)) -> HealthResponse: def health(db_path=Depends(get_db_path)) -> HealthResponse:
return HealthResponse(status="ok", db_path=str(db_path)) return HealthResponse(status="ok", db_path=str(db_path))
@router.post("/shutdown", response_model=MessageResponse, include_in_schema=False)
def shutdown(
background_tasks: BackgroundTasks,
request: Request,
x_s1dl_shutdown_token: str | None = Header(default=None, alias="X-S1DL-Shutdown-Token"),
) -> MessageResponse:
expected_token = getattr(request.app.state, "shutdown_token", None)
if not expected_token:
raise HTTPException(status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail="Shutdown is disabled.")
if x_s1dl_shutdown_token != expected_token:
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Invalid shutdown token.")
background_tasks.add_task(_terminate_current_process)
return MessageResponse(message="Shutdown requested.")
@@ -25,3 +25,10 @@ def configured_api_port() -> int:
if not raw: if not raw:
return DEFAULT_API_PORT return DEFAULT_API_PORT
return int(raw) return int(raw)
def configured_ui_shutdown_token() -> str | None:
raw = os.getenv("S1DL_UI_SHUTDOWN_TOKEN")
if not raw:
return None
return raw
+16
View File
@@ -0,0 +1,16 @@
* text=auto
.gitattributes text eol=lf
.gitignore text eol=lf
*.md text eol=lf
*.py text eol=lf
*.go text eol=lf
*.mod text eol=lf
*.toml text eol=lf
*.yml text eol=lf
*.idl text eol=lf
*.pro text eol=lf
*.cmd text eol=crlf
*.exe binary
*.sav binary
+27
View File
@@ -0,0 +1,27 @@
.venv/
__pycache__/
*.py[cod]
.pytest_cache/
.mypy_cache/
.ruff_cache/
/.tmp_tests/
logs/
GMTED2010.jp2
# Local wrapper state
gf3wrapper.json
.gf3_extract/
.gf3_runtime/
# Production data and generated outputs
/data/
/outputs/
*.tar.gz
*.tgz
*.zip
*.tif
*.tiff
*.img
*.dat
*.sml
*.hdr
+30
View File
@@ -0,0 +1,30 @@
"""Compatibility wrapper for archive extraction."""
from __future__ import annotations
import argparse
import sys
from pathlib import Path
_SRC = Path(__file__).resolve().parent / "src"
if _SRC.is_dir():
sys.path.insert(0, str(_SRC))
from gf3_l1a2l2.decompress import extract_compressed_files
def main() -> int:
parser = argparse.ArgumentParser(description="Extract GF3 archives")
parser.add_argument("input", type=Path, help="folder containing .tar.gz or .zip archives")
parser.add_argument("--overwrite", action="store_true", help="overwrite existing extracted folders")
args = parser.parse_args()
results = extract_compressed_files(args.input, overwrite=args.overwrite)
failed = [item for item in results if item.status == "failed"]
for item in results:
print(f"{item.status}\t{item.archive}\t{item.output_dir}\t{item.message}")
return 1 if failed else 0
if __name__ == "__main__":
raise SystemExit(main())
+103
View File
@@ -0,0 +1,103 @@
# GF3 L1A to L2 Pipeline
This repository contains two GF-3 L1A to L2 processing paths:
- **SARscape Go wrapper**: Windows-oriented production wrapper around ENVI/IDL Runtime and SARscape. This is the recommended path when SARscape is available.
- **Python pipeline**: GDAL/Rasterio/NumPy implementation kept as a pure Python package and CLI.
The SARscape wrapper currently has the more complete operational workflow for GF-3 scene import, multilook/filter/geocode processing, batch directory discovery, saved defaults, and progress reporting.
## Repository Layout
```text
dist/windows/ Prebuilt Windows executable
docs/ Operational documentation
src/gf3_l1a2l2/ Python package
tests/ Python unit tests and small fixtures
tools/gf3_sarscape_wrapper/ Go wrapper source and embedded SAV asset
tools/gf3_sarscape_wrapper/idl/ IDL/SARscape source scripts
```
Large GF-3 archives, extracted scenes, SARscape outputs, and local runtime files are intentionally not tracked in Git.
## Recommended: SARscape Go Wrapper
Use the prebuilt executable:
```powershell
.\dist\windows\gf3wrapper.exe `
-input "D:\GF3\L1A_BATCH" `
-output "E:\GF3\L2_SARscape" `
-dem "D:\DEM\COPDEM_GLO30_China_4326_DEM" `
-pol "HH,HV"
```
Supported input forms:
- A single `*.tar.gz` GF-3 L1A package.
- A single `*.meta.xml` scene metadata file.
- A directory containing multiple `*.tar.gz` or `*.meta.xml` scenes.
The wrapper writes one output subdirectory per scene. On successful runs it saves defaults next to the executable in `gf3wrapper.json`, so double-click interactive mode can reuse the previous input/output/DEM/polarization settings.
Batch directory runs continue after per-scene failures. If one archive is truncated, invalid, missing metadata, or fails during SARscape processing, the wrapper prints the failed scene and reason, skips that scene, and continues with the remaining scenes. At the end it prints a batch summary and returns a non-zero exit code when any scene failed.
Before processing each scene, the wrapper checks the scene output directory for completed geocoded products for the requested polarizations. Already completed scenes are skipped, so rerunning the same input/output pair only processes missing or incomplete scenes.
More details: [docs/sarscape_go_wrapper.md](docs/sarscape_go_wrapper.md)
## Python Pipeline
Use an environment that already has working GDAL/OSGeo Python bindings. On Windows, Conda/Miniforge is usually the least fragile route.
```powershell
conda env create -f environment.yml
conda activate gf3-l1a2l2
pip install -e . --no-deps
```
Run:
```powershell
gf3-l1a2l2 run --input D:\GF3\L1A_BATCH --output E:\GF3\L2_Image --dem .\GMTED2010.jp2
```
Common options:
```powershell
gf3-l1a2l2 run `
--input D:\GF3\L1A_BATCH `
--output E:\GF3\L2_Image `
--dem .\GMTED2010.jp2 `
--workers 1 `
--x-res 0.0002 `
--y-res 0.0002 `
--dst-srs EPSG:4326
```
Legacy wrappers remain available:
```powershell
python gf3_L1A_To_L2.py run --input D:\GF3\L1A_BATCH --output E:\GF3\L2_Image --dem .\GMTED2010.jp2
python Decompression.py D:\GF3\L1A_BATCH
```
## Development Checks
Python:
```powershell
python -m pytest
```
Go wrapper:
```powershell
cd .\tools\gf3_sarscape_wrapper
$env:GOTELEMETRY = "off"
go build -o ..\..\dist\windows\gf3wrapper.exe .
```
## Data Policy
Do not commit production GF-3 archives or generated products. Keep large data on NAS/storage and document the path in run notes. Only small sanitized fixtures should be added under `tests/fixtures`.
+21
View File
@@ -0,0 +1,21 @@
# Windows Distribution
`gf3wrapper.exe` is the prebuilt SARscape Go wrapper.
Run from the repository root:
```powershell
.\dist\windows\gf3wrapper.exe `
-input "D:\GF3\L1A_BATCH" `
-output "E:\GF3\L2_SARscape" `
-dem "D:\DEM\COPDEM_GLO30_China_4326_DEM" `
-pol "HH,HV"
```
The executable requires ENVI/IDL Runtime and SARscape on the target machine.
In directory mode, failed scenes are skipped and listed in a final batch summary. The wrapper continues with remaining scenes, but exits non-zero if any scene failed.
Rerunning the same input/output pair skips scenes with complete geocoded outputs for the requested polarizations and processes only missing or incomplete scenes.
After successful runs, `gf3wrapper.json` is created in this directory to store defaults. That file is local runtime state and is ignored by Git.
Binary file not shown.
@@ -0,0 +1,45 @@
# Repository Layout
```text
.
|-- dist/
| `-- windows/
| `-- gf3wrapper.exe
|-- docs/
| |-- repository_layout.md
| `-- sarscape_go_wrapper.md
|-- src/
| `-- gf3_l1a2l2/
|-- tests/
| `-- fixtures/
|-- tools/
| `-- gf3_sarscape_wrapper/
| |-- assets/
| |-- idl/
| |-- go.mod
| `-- main.go
|-- README.md
|-- environment.yml
`-- pyproject.toml
```
## What Belongs In Git
- Python package source.
- Go wrapper source.
- IDL/SARscape source scripts.
- Small embedded runtime assets needed to build the wrapper.
- Small sanitized tests and fixtures.
- Documentation and reproducible build instructions.
- Small convenience binaries when the team explicitly wants a ready-to-run Windows artifact.
## What Does Not Belong In Git
- Raw GF-3 production archives.
- Extracted scenes.
- SARscape output directories.
- Local wrapper config files such as `gf3wrapper.json`.
- Runtime folders such as `.gf3_extract` and `.gf3_runtime`.
- DEM files and other large environment-specific dependencies.
Use NAS or object storage for production data and record exact paths in run logs or issue notes.
@@ -0,0 +1,142 @@
# SARscape Go Wrapper
`gf3wrapper.exe` is a Windows command-line wrapper for the SARscape/IDL GF-3 processing workflow. It embeds `gf3_sarscape_cli.sav`, starts `idlrt.exe`, and passes scene metadata, output directory, DEM, and polarization arguments to SARscape.
This path is intended for operational runs where ENVI/IDL Runtime and SARscape are installed and licensed.
## Requirements
- Windows host.
- ENVI/IDL Runtime, default path:
`C:\Program Files\Harris\ENVI56\IDL88\bin\bin.x86_64\idlrt.exe`
- SARscape available to the IDL Runtime environment.
- DEM file, for example:
`D:\DEM\COPDEM_GLO30_China_4326_DEM`
- GF-3 L1A input as `*.tar.gz`, `*.meta.xml`, or a directory containing multiple scenes.
The IDL Runtime path can be overridden with `-idlrt` or the `IDLRT_PATH` environment variable.
## Quick Start
Use the prebuilt executable:
```powershell
.\dist\windows\gf3wrapper.exe `
-input "D:\GF3\L1A_BATCH" `
-output "E:\GF3\L2_SARscape" `
-dem "D:\DEM\COPDEM_GLO30_China_4326_DEM" `
-pol "HH,HV"
```
Double-click mode is also supported. When launched without arguments, the wrapper prompts for input, output, DEM, polarizations, and IDL Runtime path. Press Enter to reuse saved defaults.
## Inputs
The wrapper supports:
- A single GF-3 archive: `GF3_....tar.gz`
- A single extracted scene metadata file: `GF3_....meta.xml`
- A directory containing many archives and/or extracted scenes
Directory mode recursively scans for `*.tar.gz`, `*.tgz`, and `*.meta.xml`. If both archive and metadata exist for the same scene name, the archive is preferred. Internal `.gf3_extract` folders are skipped.
## Batch Failure Handling
Directory runs are fault-tolerant at the scene level. If a single scene cannot be extracted or processed, the wrapper records that scene as failed, prints the input path and error reason, and continues with the next scene.
Archive read errors include the archive path, archive size when available, and the archive member being read when the failure happened. For example, a truncated `.tar.gz` reports that the archive appears incomplete and should be re-copied or re-downloaded.
At the end of the run, the wrapper prints a summary:
```text
Batch summary: 3 succeeded, 1 failed, 4 total
Failed scenes:
- GF3_... (<archive path>): <failure reason>
```
If any scene failed, the process exits with a non-zero status after the summary. This lets scheduled jobs detect that the batch needs attention while still preserving successful outputs from other scenes.
## Resume Behavior
Before extracting or processing a scene, the wrapper checks the scene output directory for completed geocoded SARscape products for every requested polarization. A product is considered complete when both the final `*_geo.sml` metadata file and its matching data file exist and are non-empty.
Already completed scenes are skipped:
```text
SKIPPED GF3_...
Input D:\GF3\L1A_BATCH\GF3_....tar.gz
Output already complete E:\GF3\L2_SARscape\GF3_...
```
This is intended for interrupted or partially failed batch runs. For example, after replacing a truncated archive in the original input directory, rerun the same command with the same output directory; scenes that already have complete HH/HV geocoded outputs are skipped, and only missing or incomplete scenes are processed.
## Outputs
Each scene is processed into its own output directory:
```text
E:\GF3\L2_SARscape\
GF3_SCENE_NAME\
.gf3_extract\ Extracted archive, when input was .tar.gz
*_slc.sml
*_ml.sml
*_filt.sml
*_geo.sml
```
The embedded SAV is materialized under:
```text
<output>\.gf3_runtime\gf3_sarscape_cli.sav
```
## Saved Defaults
After a successful run, the wrapper writes `gf3wrapper.json` next to the executable:
```json
{
"idlrt_path": "C:\\Program Files\\Harris\\ENVI56\\IDL88\\bin\\bin.x86_64\\idlrt.exe",
"dem_file": "D:\\DEM\\COPDEM_GLO30_China_4326_DEM",
"polarizations": "HH,HV",
"last_input": "D:\\GF3\\L1A_BATCH",
"last_output": "E:\\GF3\\L2_SARscape",
"keep_extracted": true
}
```
Command-line flags override saved defaults. If `-input`, `-output`, `-dem`, or `-pol` are omitted, saved values are reused when available.
## Build
From the repository root:
```powershell
cd .\tools\gf3_sarscape_wrapper
$env:GOTELEMETRY = "off"
go build -o ..\..\dist\windows\gf3wrapper.exe .
```
If Go cannot write to the default user cache directory, set local cache paths:
```powershell
$env:GOCACHE = "C:\Temp\go-build-cache"
$env:APPDATA = "C:\Temp\go-appdata"
```
## Source Files
```text
tools/gf3_sarscape_wrapper/main.go
tools/gf3_sarscape_wrapper/assets/gf3_sarscape_cli.sav
tools/gf3_sarscape_wrapper/idl/gf3_sarscape_cli.pro
tools/gf3_sarscape_wrapper/idl/gf3_sarscape_pipeline.pro
```
`assets/gf3_sarscape_cli.sav` is embedded into the Go executable at build time.
## Notes
- Keep production archives and generated products outside Git.
- Keep only small sanitized fixtures in `tests/fixtures`.
- The prebuilt executable in `dist/windows` is a convenience artifact for Windows users. Rebuild it after changing `main.go` or the embedded `.sav`.
+10
View File
@@ -0,0 +1,10 @@
name: gf3-l1a2l2
channels:
- conda-forge
dependencies:
- python=3.11
- gdal
- rasterio
- numpy
- tqdm
- pip
+16
View File
@@ -0,0 +1,16 @@
"""Compatibility wrapper for the packaged GF3 L1A to L2 pipeline."""
from __future__ import annotations
import sys
from pathlib import Path
_SRC = Path(__file__).resolve().parent / "src"
if _SRC.is_dir():
sys.path.insert(0, str(_SRC))
from gf3_l1a2l2.cli import main
if __name__ == "__main__":
raise SystemExit(main())
+24
View File
@@ -0,0 +1,24 @@
[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "gf3-l1a2l2-pipeline"
version = "0.1.0"
description = "GF3 L1A to L2 processing pipeline"
readme = "README.md"
requires-python = ">=3.10"
dependencies = [
"numpy>=1.22",
"rasterio>=1.3",
"tqdm>=4.64",
]
[project.scripts]
gf3-l1a2l2 = "gf3_l1a2l2.cli:main"
[tool.setuptools]
package-dir = {"" = "src"}
[tool.setuptools.packages.find]
where = ["src"]
@@ -0,0 +1,16 @@
"""GF3 L1A to L2 processing pipeline."""
__all__ = ["PipelineConfig", "run_pipeline"]
__version__ = "0.1.0"
def __getattr__(name: str):
if name == "PipelineConfig":
from gf3_l1a2l2.config import PipelineConfig
return PipelineConfig
if name == "run_pipeline":
from gf3_l1a2l2.pipeline import run_pipeline
return run_pipeline
raise AttributeError(name)
@@ -0,0 +1,5 @@
from gf3_l1a2l2.cli import main
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,88 @@
from __future__ import annotations
import logging
import math
from pathlib import Path
from typing import Iterator, Optional
import numpy as np
import rasterio
from rasterio.windows import Window
from gf3_l1a2l2.errors import ProcessingError
from gf3_l1a2l2.models import CalibrationValue
from gf3_l1a2l2.paths import output_name
def radiometric_calibrate_l1a_to_l1b(
image_path: Path | str,
output_dir: Path | str,
calibration: CalibrationValue,
*,
block_size: int,
overwrite: bool = False,
logger: Optional[logging.Logger] = None,
) -> Path:
source = Path(image_path)
out_dir = Path(output_dir)
out_dir.mkdir(parents=True, exist_ok=True)
l1b_path = out_dir / output_name(source, "L1A", "L1B")
if l1b_path.exists() and not overwrite:
if logger:
logger.info("Skip existing L1B: %s", l1b_path)
return l1b_path
if not math.isfinite(calibration.qualify_value):
raise ProcessingError(f"Invalid QualifyValue for {source.name}: {calibration.qualify_value}")
if not math.isfinite(calibration.calibration_const):
raise ProcessingError(
f"Invalid CalibrationConst for {source.name}: {calibration.calibration_const}"
)
if logger:
logger.info("Calibrating %s -> %s", source, l1b_path)
try:
with rasterio.open(source) as src:
if src.count < 2:
raise ProcessingError(f"Expected at least 2 bands in L1A image: {source}")
profile = src.profile.copy()
profile.update(
driver="GTiff",
count=1,
dtype="float32",
compress="lzw",
BIGTIFF="YES",
tiled=True,
)
with rasterio.open(l1b_path, "w", **profile) as dst:
for window in _iter_windows(src.width, src.height, block_size):
real = src.read(1, window=window, out_dtype="float32")
imag = src.read(2, window=window, out_dtype="float32")
db = _calibrate_window(real, imag, calibration)
dst.write(db, 1, window=window)
except ProcessingError:
raise
except Exception as exc:
raise ProcessingError(f"Failed to calibrate {source}") from exc
return l1b_path
def _calibrate_window(real: np.ndarray, imag: np.ndarray, calibration: CalibrationValue) -> np.ndarray:
amplitude = np.hypot(real, imag)
scaled = (amplitude / np.float32(32767.0)) * np.float32(calibration.qualify_value)
with np.errstate(divide="ignore", invalid="ignore"):
db = np.float32(20.0) * np.log10(scaled) - np.float32(calibration.calibration_const)
return db.astype("float32", copy=False)
def _iter_windows(width: int, height: int, block_size: int) -> Iterator[Window]:
for row_off in range(0, height, block_size):
window_height = min(block_size, height - row_off)
for col_off in range(0, width, block_size):
window_width = min(block_size, width - col_off)
yield Window(col_off, row_off, window_width, window_height)
+107
View File
@@ -0,0 +1,107 @@
from __future__ import annotations
import argparse
import sys
from pathlib import Path
from gf3_l1a2l2.config import PipelineConfig
from gf3_l1a2l2.decompress import extract_compressed_files
from gf3_l1a2l2.discovery import discover_scenes
from gf3_l1a2l2.errors import GF3PipelineError
def main(argv: list[str] | None = None) -> int:
parser = build_parser()
args = parser.parse_args(argv)
if not hasattr(args, "handler"):
parser.print_help()
return 2
try:
return args.handler(args)
except GF3PipelineError as exc:
print(f"ERROR: {exc}", file=sys.stderr)
return 1
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(prog="gf3-l1a2l2", description="GF3 L1A to L2 pipeline")
subparsers = parser.add_subparsers(dest="command")
run_parser = subparsers.add_parser("run", help="extract and process GF3 L1A products")
run_parser.add_argument("--input", required=True, type=Path, help="input folder containing archives or scenes")
run_parser.add_argument("--output", required=True, type=Path, help="output folder")
run_parser.add_argument("--dem", type=Path, default=None, help="DEM file for RPC orthorectification")
run_parser.add_argument("--x-res", type=float, default=0.0002, help="output x resolution")
run_parser.add_argument("--y-res", type=float, default=0.0002, help="output y resolution")
run_parser.add_argument("--dst-srs", default="EPSG:4326", help="destination spatial reference")
run_parser.add_argument("--block-size", type=int, default=2048, help="calibration block size in pixels")
run_parser.add_argument("--workers", type=int, default=1, help="number of scenes to process concurrently")
run_parser.add_argument("--skip-decompress", action="store_true", help="do not extract archives before processing")
run_parser.add_argument("--overwrite", action="store_true", help="overwrite existing extraction and output files")
run_parser.add_argument("--flat-output", action="store_true", help="write all outputs directly to output folder")
run_parser.add_argument("--fail-fast", action="store_true", help="stop on the first failed scene or polarization")
run_parser.add_argument("--log-level", default="INFO", choices=["DEBUG", "INFO", "WARNING", "ERROR"])
run_parser.set_defaults(handler=_run)
decompress_parser = subparsers.add_parser("decompress", help="extract .tar.gz and .zip archives")
decompress_parser.add_argument("input", type=Path, help="folder containing archives")
decompress_parser.add_argument("--overwrite", action="store_true", help="overwrite existing extracted folders")
decompress_parser.set_defaults(handler=_decompress)
discover_parser = subparsers.add_parser("discover", help="list discovered GF3 scenes")
discover_parser.add_argument("input", type=Path, help="input folder")
discover_parser.set_defaults(handler=_discover)
return parser
def _run(args: argparse.Namespace) -> int:
from gf3_l1a2l2.pipeline import run_pipeline
dem_path = args.dem or _default_dem_path()
config = PipelineConfig(
input_dir=args.input,
output_dir=args.output,
dem_path=dem_path,
x_res=args.x_res,
y_res=args.y_res,
dst_srs=args.dst_srs,
block_size=args.block_size,
workers=args.workers,
skip_decompress=args.skip_decompress,
overwrite=args.overwrite,
flat_output=args.flat_output,
fail_fast=args.fail_fast,
log_level=args.log_level,
)
summary = run_pipeline(config)
return 1 if summary.failed else 0
def _decompress(args: argparse.Namespace) -> int:
results = extract_compressed_files(args.input, overwrite=args.overwrite)
failed = [item for item in results if item.status == "failed"]
for item in results:
print(f"{item.status}\t{item.archive}\t{item.output_dir}\t{item.message}")
return 1 if failed else 0
def _discover(args: argparse.Namespace) -> int:
scenes = discover_scenes(args.input)
for scene in scenes:
polarizations = ",".join(scene.available_polarizations)
print(f"{scene.root}\tmetadata={scene.metadata_xml.name}\tpol={polarizations}")
return 0
def _default_dem_path() -> Path | None:
candidates = [
Path.cwd() / "GMTED2010.jp2",
Path(__file__).resolve().parents[2] / "GMTED2010.jp2",
]
for candidate in candidates:
if candidate.is_file():
return candidate
return None
@@ -0,0 +1,41 @@
from __future__ import annotations
from dataclasses import dataclass, replace
from pathlib import Path
from typing import Optional
from gf3_l1a2l2.constants import (
DEFAULT_BLOCK_SIZE,
DEFAULT_DST_SRS,
DEFAULT_X_RES,
DEFAULT_Y_RES,
)
@dataclass(frozen=True)
class PipelineConfig:
input_dir: Path
output_dir: Path
dem_path: Optional[Path] = None
x_res: float = DEFAULT_X_RES
y_res: float = DEFAULT_Y_RES
dst_srs: str = DEFAULT_DST_SRS
block_size: int = DEFAULT_BLOCK_SIZE
workers: int = 1
skip_decompress: bool = False
overwrite: bool = False
flat_output: bool = False
keep_l1b: bool = True
fail_fast: bool = False
log_level: str = "INFO"
def resolved(self) -> "PipelineConfig":
dem_path = self.dem_path.resolve() if self.dem_path else None
return replace(
self,
input_dir=self.input_dir.resolve(),
output_dir=self.output_dir.resolve(),
dem_path=dem_path,
workers=max(1, int(self.workers)),
block_size=max(128, int(self.block_size)),
)
@@ -0,0 +1,6 @@
POLARIZATIONS = ("HH", "HV", "VH", "VV")
DEFAULT_X_RES = 0.0002
DEFAULT_Y_RES = 0.0002
DEFAULT_DST_SRS = "EPSG:4326"
DEFAULT_BLOCK_SIZE = 2048
@@ -0,0 +1,112 @@
from __future__ import annotations
import logging
import shutil
import tarfile
import zipfile
from dataclasses import dataclass
from pathlib import Path
from typing import Iterable, Optional
try:
from tqdm import tqdm
except Exception: # pragma: no cover - only used when tqdm is unavailable
def tqdm(iterable: Iterable, **_: object) -> Iterable:
return iterable
from gf3_l1a2l2.errors import ProcessingError
@dataclass(frozen=True)
class ExtractionResult:
archive: Path
output_dir: Path
status: str
message: str = ""
def extract_compressed_files(
folder_path: Path | str,
*,
overwrite: bool = False,
logger: Optional[logging.Logger] = None,
) -> list[ExtractionResult]:
folder = Path(folder_path).resolve()
if not folder.is_dir():
raise ProcessingError(f"Input folder does not exist: {folder}")
archives = sorted(
path
for path in folder.iterdir()
if path.is_file()
and (path.name.lower().endswith(".tar.gz") or path.suffix.lower() == ".zip")
)
results: list[ExtractionResult] = []
if not archives:
if logger:
logger.info("No .tar.gz or .zip archives found in %s", folder)
return results
for archive in tqdm(archives, desc="Extract archives", unit="file"):
output_dir = _archive_output_dir(archive)
try:
if output_dir.exists():
if not overwrite:
results.append(ExtractionResult(archive, output_dir, "skipped", "output exists"))
if logger:
logger.info("Skip existing extraction: %s", output_dir)
continue
shutil.rmtree(output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
if archive.name.lower().endswith(".tar.gz"):
_extract_tar_gz(archive, output_dir)
elif archive.suffix.lower() == ".zip":
_extract_zip(archive, output_dir)
else:
continue
results.append(ExtractionResult(archive, output_dir, "done"))
if logger:
logger.info("Extracted %s -> %s", archive, output_dir)
except Exception as exc:
results.append(ExtractionResult(archive, output_dir, "failed", str(exc)))
if logger:
logger.exception("Failed to extract %s", archive)
return results
def _archive_output_dir(archive: Path) -> Path:
if archive.name.lower().endswith(".tar.gz"):
return archive.with_name(archive.name[:-7])
return archive.with_suffix("")
def _extract_tar_gz(archive: Path, output_dir: Path) -> None:
base = output_dir.resolve()
with tarfile.open(archive, "r:gz") as tar:
members = tar.getmembers()
for member in tqdm(members, desc=f"Extract {archive.name}", leave=False, unit="file"):
if member.issym() or member.islnk():
raise ProcessingError(f"Refusing to extract link from archive: {member.name}")
_ensure_safe_target(base, member.name)
tar.extract(member, path=base)
def _extract_zip(archive: Path, output_dir: Path) -> None:
base = output_dir.resolve()
with zipfile.ZipFile(archive, "r") as zip_ref:
names = zip_ref.namelist()
for name in tqdm(names, desc=f"Extract {archive.name}", leave=False, unit="file"):
_ensure_safe_target(base, name)
zip_ref.extract(name, base)
def _ensure_safe_target(base: Path, member_name: str) -> None:
target = (base / member_name).resolve()
if target == base:
return
if base not in target.parents:
raise ProcessingError(f"Unsafe archive path: {member_name}")
@@ -0,0 +1,90 @@
from __future__ import annotations
import logging
from pathlib import Path
from typing import Iterable, Optional
from gf3_l1a2l2.errors import DiscoveryError
from gf3_l1a2l2.models import SceneFiles
from gf3_l1a2l2.paths import detect_polarization
SKIP_DIR_NAMES = {".git", ".venv", ".tmp_tests", "__pycache__", ".pytest_cache", "logs"}
def discover_scenes(input_dir: Path | str, logger: Optional[logging.Logger] = None) -> list[SceneFiles]:
root_dir = Path(input_dir).resolve()
if not root_dir.is_dir():
raise DiscoveryError(f"Input directory does not exist: {root_dir}")
scenes: list[SceneFiles] = []
for directory in _walk_directories(root_dir):
try:
files = [path for path in directory.iterdir() if path.is_file()]
except OSError as exc:
if logger:
logger.warning("Skipping unreadable directory %s: %s", directory, exc)
continue
metadata_files = sorted(path for path in files if path.name.lower().endswith(".meta.xml"))
image_files = sorted(
path
for path in files
if path.suffix.lower() in (".tif", ".tiff") and detect_polarization(path)
)
if not metadata_files or not image_files:
continue
metadata_xml = metadata_files[0]
if len(metadata_files) > 1 and logger:
logger.warning("Multiple metadata files in %s; using %s", directory, metadata_xml.name)
images = _collect_by_polarization(image_files, logger)
rpcs = _collect_by_polarization(
sorted(path for path in files if path.suffix.lower() in (".rpc", ".rpb")),
logger,
)
scenes.append(SceneFiles(root=directory, metadata_xml=metadata_xml, images=images, rpcs=rpcs))
scenes.sort(key=lambda scene: str(scene.root).lower())
if logger:
logger.info("Discovered %s scene(s) under %s", len(scenes), root_dir)
return scenes
def _walk_directories(root_dir: Path) -> Iterable[Path]:
stack = [root_dir]
while stack:
directory = stack.pop()
if directory.name in SKIP_DIR_NAMES:
continue
yield directory
try:
children = list(directory.iterdir())
except OSError:
continue
stack.extend(path for path in children if path.is_dir() and path.name not in SKIP_DIR_NAMES)
def _collect_by_polarization(
files: Iterable[Path],
logger: Optional[logging.Logger] = None,
) -> dict[str, Path]:
collected: dict[str, Path] = {}
for path in files:
polarization = detect_polarization(path)
if not polarization:
continue
if polarization in collected:
if logger:
logger.warning(
"Duplicate %s file in %s; using %s and ignoring %s",
polarization,
path.parent,
collected[polarization].name,
path.name,
)
continue
collected[polarization] = path
return collected
@@ -0,0 +1,18 @@
class GF3PipelineError(Exception):
"""Base error for pipeline failures."""
class DiscoveryError(GF3PipelineError):
"""Raised when input products cannot be discovered."""
class MetadataError(GF3PipelineError):
"""Raised when GF3 metadata cannot be parsed."""
class RpcError(GF3PipelineError):
"""Raised when RPC metadata cannot be parsed."""
class ProcessingError(GF3PipelineError):
"""Raised when image processing fails."""
@@ -0,0 +1,78 @@
from __future__ import annotations
import logging
from pathlib import Path
from typing import Optional
from gf3_l1a2l2.errors import ProcessingError
from gf3_l1a2l2.paths import output_name
from gf3_l1a2l2.rpc import read_rpc_metadata
def geometric_correction(
l1b_path: Path | str,
rpc_path: Path | str,
output_dir: Path | str,
*,
dem_path: Optional[Path],
x_res: float,
y_res: float,
dst_srs: str,
overwrite: bool = False,
logger: Optional[logging.Logger] = None,
) -> Path:
source = Path(l1b_path)
out_dir = Path(output_dir)
out_dir.mkdir(parents=True, exist_ok=True)
l2_path = out_dir / output_name(source, "L1B", "L2")
if l2_path.exists() and not overwrite:
if logger:
logger.info("Skip existing L2: %s", l2_path)
return l2_path
if dem_path and not Path(dem_path).is_file():
raise ProcessingError(f"DEM file does not exist: {dem_path}")
rpc_metadata = read_rpc_metadata(rpc_path)
try:
from osgeo import gdal
except Exception as exc:
raise ProcessingError("Cannot import osgeo.gdal. Install GDAL Python bindings.") from exc
if logger:
logger.info("Geocorrecting %s -> %s", source, l2_path)
dataset = gdal.Open(str(source), gdal.GA_Update)
if dataset is None:
raise ProcessingError(f"GDAL cannot open L1B image: {source}")
try:
dataset.SetMetadata(rpc_metadata, "RPC")
transformer_options = []
if dem_path:
transformer_options.append(f"RPC_DEM={Path(dem_path)}")
warp_options = gdal.WarpOptions(
dstSRS=dst_srs,
xRes=x_res,
yRes=y_res,
rpc=True,
multithread=True,
transformerOptions=transformer_options,
creationOptions=["COMPRESS=LZW", "TILED=YES", "BIGTIFF=YES"],
)
result = gdal.Warp(str(l2_path), dataset, options=warp_options)
if result is None:
raise ProcessingError(f"GDAL Warp failed for {source}")
result.FlushCache()
result = None
except ProcessingError:
raise
except Exception as exc:
raise ProcessingError(f"Failed to geocorrect {source}") from exc
finally:
dataset = None
return l2_path
@@ -0,0 +1,34 @@
from __future__ import annotations
import logging
from datetime import datetime
from pathlib import Path
def configure_logging(output_dir: Path, level: str = "INFO") -> logging.Logger:
output_dir.mkdir(parents=True, exist_ok=True)
log_dir = output_dir / "logs"
log_dir.mkdir(parents=True, exist_ok=True)
logger = logging.getLogger("gf3_l1a2l2")
logger.setLevel(getattr(logging, level.upper(), logging.INFO))
logger.propagate = False
logger.handlers.clear()
formatter = logging.Formatter(
fmt="%(asctime)s %(levelname)s %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
)
console = logging.StreamHandler()
console.setFormatter(formatter)
console.setLevel(getattr(logging, level.upper(), logging.INFO))
logger.addHandler(console)
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
file_handler = logging.FileHandler(log_dir / f"gf3_l1a2l2_{timestamp}.log", encoding="utf-8")
file_handler.setFormatter(formatter)
file_handler.setLevel(logging.DEBUG)
logger.addHandler(file_handler)
return logger
@@ -0,0 +1,145 @@
from __future__ import annotations
import math
import xml.etree.ElementTree as ET
from pathlib import Path
from typing import Iterable, Optional
from gf3_l1a2l2.constants import POLARIZATIONS
from gf3_l1a2l2.errors import MetadataError
from gf3_l1a2l2.models import CalibrationMetadata, CalibrationValue
from gf3_l1a2l2.paths import detect_polarization
QUALIFY_KEYWORDS = ("qualifyvalue", "qualityvalue")
CALIBRATION_KEYWORDS = ("calibrationconst", "calibrationconstant")
def read_calibration_metadata(xml_path: Path | str) -> CalibrationMetadata:
path = Path(xml_path)
try:
root = ET.parse(path).getroot()
except ET.ParseError as exc:
raise MetadataError(f"Invalid metadata XML: {path}") from exc
except OSError as exc:
raise MetadataError(f"Cannot read metadata XML: {path}") from exc
qualify_values = _find_values(root, QUALIFY_KEYWORDS) or _legacy_values(root, (17, 13))
calibration_values = _find_values(root, CALIBRATION_KEYWORDS) or _legacy_values(root, (18, 3))
if not qualify_values:
raise MetadataError(f"Cannot find QualifyValue values in {path}")
if not calibration_values:
raise MetadataError(f"Cannot find CalibrationConst values in {path}")
values: dict[str, CalibrationValue] = {}
for polarization in POLARIZATIONS:
if polarization not in qualify_values or polarization not in calibration_values:
continue
values[polarization] = CalibrationValue(
qualify_value=qualify_values[polarization],
calibration_const=calibration_values[polarization],
)
if not values:
raise MetadataError(f"No usable calibration values found in {path}")
return CalibrationMetadata(values=values)
def _find_values(root: ET.Element, keywords: Iterable[str]) -> Optional[dict[str, float]]:
lowered = tuple(keyword.lower() for keyword in keywords)
for element in root.iter():
name = _local_name(element.tag).lower()
if not any(keyword in name for keyword in lowered):
continue
values = _values_by_child_polarization(element)
if values:
return values
values = _values_by_child_order(element)
if values:
return values
values = _values_from_text(element.text)
if values:
return values
return None
def _legacy_values(root: ET.Element, indexes: tuple[int, int]) -> Optional[dict[str, float]]:
try:
node = root
for index in indexes:
node = list(node)[index]
values = [_parse_float(child.text) for child in list(node)[:4]]
except (IndexError, TypeError, ValueError):
return None
if len(values) < 4:
return None
return dict(zip(POLARIZATIONS, values))
def _values_by_child_polarization(element: ET.Element) -> Optional[dict[str, float]]:
values: dict[str, float] = {}
for child in list(element):
polarization = _element_polarization(child)
if not polarization:
continue
try:
values[polarization] = _parse_float(child.text)
except ValueError:
continue
return values if values else None
def _values_by_child_order(element: ET.Element) -> Optional[dict[str, float]]:
numeric_values: list[float] = []
for child in list(element):
try:
numeric_values.append(_parse_float(child.text))
except ValueError:
continue
if len(numeric_values) < 4:
return None
return dict(zip(POLARIZATIONS, numeric_values[:4]))
def _values_from_text(text: Optional[str]) -> Optional[dict[str, float]]:
if not text:
return None
normalized = text.replace(",", " ").replace(";", " ")
parts = [part for part in normalized.split() if part]
if len(parts) < 4:
return None
try:
values = [_parse_float(part) for part in parts[:4]]
except ValueError:
return None
return dict(zip(POLARIZATIONS, values))
def _element_polarization(element: ET.Element) -> Optional[str]:
for value in element.attrib.values():
polarization = detect_polarization(value)
if polarization:
return polarization
return detect_polarization(_local_name(element.tag))
def _parse_float(text: Optional[str]) -> float:
if text is None:
raise ValueError("missing value")
value = text.strip().strip("=")
if not value or value.upper() == "NULL":
return math.nan
return float(value)
def _local_name(tag: str) -> str:
if "}" in tag:
return tag.rsplit("}", 1)[1]
return tag
@@ -0,0 +1,66 @@
from __future__ import annotations
from dataclasses import dataclass, field
from pathlib import Path
from typing import Mapping, Optional
@dataclass(frozen=True)
class CalibrationValue:
qualify_value: float
calibration_const: float
@dataclass(frozen=True)
class CalibrationMetadata:
values: Mapping[str, CalibrationValue]
def for_polarization(self, polarization: str) -> CalibrationValue:
return self.values[polarization]
@dataclass(frozen=True)
class SceneFiles:
root: Path
metadata_xml: Path
images: Mapping[str, Path]
rpcs: Mapping[str, Path]
@property
def name(self) -> str:
return self.root.name
@property
def available_polarizations(self) -> tuple[str, ...]:
return tuple(sorted(self.images))
@dataclass(frozen=True)
class TaskResult:
scene: str
polarization: str
status: str
l1b_path: Optional[Path] = None
l2_path: Optional[Path] = None
message: str = ""
@dataclass
class PipelineSummary:
results: list[TaskResult] = field(default_factory=list)
@property
def succeeded(self) -> int:
return sum(1 for item in self.results if item.status == "done")
@property
def skipped(self) -> int:
return sum(1 for item in self.results if item.status == "skipped")
@property
def failed(self) -> int:
return sum(1 for item in self.results if item.status == "failed")
@property
def total(self) -> int:
return len(self.results)
@@ -0,0 +1,43 @@
from __future__ import annotations
from pathlib import Path
from typing import Optional
from gf3_l1a2l2.constants import POLARIZATIONS
def detect_polarization(path_or_name: Path | str) -> Optional[str]:
name = Path(path_or_name).name.upper()
for polarization in POLARIZATIONS:
if polarization in name:
return polarization
return None
def output_name(source: Path, source_level: str, target_level: str) -> str:
name = source.name
index = name.lower().find(source_level.lower())
if index >= 0:
return name[:index] + target_level + name[index + len(source_level) :]
stem = source.stem
suffix = "".join(source.suffixes)
if suffix and stem.endswith(suffix):
stem = stem[: -len(suffix)]
return f"{stem}_{target_level}{source.suffix}"
def scene_output_dir(input_dir: Path, output_dir: Path, scene_dir: Path, flat: bool) -> Path:
if flat:
return output_dir
try:
relative = scene_dir.relative_to(input_dir)
except ValueError:
relative = Path(scene_dir.name)
if str(relative) in ("", "."):
relative = Path(scene_dir.name)
safe_name = "_".join(part for part in relative.parts if part not in ("", "."))
return output_dir / safe_name
@@ -0,0 +1,182 @@
from __future__ import annotations
import concurrent.futures
import logging
from pathlib import Path
from typing import Iterable
from gf3_l1a2l2.calibration import radiometric_calibrate_l1a_to_l1b
from gf3_l1a2l2.config import PipelineConfig
from gf3_l1a2l2.constants import POLARIZATIONS
from gf3_l1a2l2.decompress import extract_compressed_files
from gf3_l1a2l2.discovery import discover_scenes
from gf3_l1a2l2.errors import GF3PipelineError
from gf3_l1a2l2.geocorrection import geometric_correction
from gf3_l1a2l2.log import configure_logging
from gf3_l1a2l2.metadata import read_calibration_metadata
from gf3_l1a2l2.models import PipelineSummary, SceneFiles, TaskResult
from gf3_l1a2l2.paths import scene_output_dir
def run_pipeline(config: PipelineConfig) -> PipelineSummary:
config = config.resolved()
logger = configure_logging(config.output_dir, config.log_level)
_validate_config(config)
logger.info("Input: %s", config.input_dir)
logger.info("Output: %s", config.output_dir)
if config.dem_path:
logger.info("DEM: %s", config.dem_path)
if not config.skip_decompress:
extract_compressed_files(config.input_dir, overwrite=config.overwrite, logger=logger)
scenes = discover_scenes(config.input_dir, logger=logger)
if not scenes:
logger.warning("No GF3 scenes found under %s", config.input_dir)
return PipelineSummary()
summary = PipelineSummary()
for results in _process_scenes(scenes, config, logger):
summary.results.extend(results)
logger.info(
"Pipeline finished: total=%s done=%s skipped=%s failed=%s",
summary.total,
summary.succeeded,
summary.skipped,
summary.failed,
)
return summary
def _validate_config(config: PipelineConfig) -> None:
if not config.input_dir.is_dir():
raise GF3PipelineError(f"Input directory does not exist: {config.input_dir}")
if config.dem_path and not config.dem_path.is_file():
raise GF3PipelineError(f"DEM file does not exist: {config.dem_path}")
if config.x_res <= 0 or config.y_res <= 0:
raise GF3PipelineError("x_res and y_res must be positive")
def _process_scenes(
scenes: list[SceneFiles],
config: PipelineConfig,
logger: logging.Logger,
) -> Iterable[list[TaskResult]]:
if config.workers == 1:
for scene in scenes:
yield _process_scene(scene, config, logger)
return
with concurrent.futures.ThreadPoolExecutor(max_workers=config.workers) as executor:
future_map = {
executor.submit(_process_scene, scene, config, logger): scene
for scene in scenes
}
for future in concurrent.futures.as_completed(future_map):
scene = future_map[future]
try:
yield future.result()
except Exception as exc:
logger.exception("Scene failed: %s", scene.root)
if config.fail_fast:
raise
yield [
TaskResult(
scene=scene.name,
polarization=polarization,
status="failed",
message=str(exc),
)
for polarization in scene.available_polarizations
]
def _process_scene(
scene: SceneFiles,
config: PipelineConfig,
logger: logging.Logger,
) -> list[TaskResult]:
logger.info("Processing scene: %s", scene.root)
out_dir = scene_output_dir(config.input_dir, config.output_dir, scene.root, config.flat_output)
out_dir.mkdir(parents=True, exist_ok=True)
try:
metadata = read_calibration_metadata(scene.metadata_xml)
except Exception as exc:
logger.exception("Failed to parse metadata for %s", scene.root)
if config.fail_fast:
raise
return [
TaskResult(scene=scene.name, polarization=pol, status="failed", message=str(exc))
for pol in scene.available_polarizations
]
results: list[TaskResult] = []
for polarization in POLARIZATIONS:
image_path = scene.images.get(polarization)
if not image_path:
continue
rpc_path = scene.rpcs.get(polarization)
if not rpc_path:
message = f"Missing RPC/RPB file for {polarization}"
logger.error("%s in %s", message, scene.root)
results.append(TaskResult(scene=scene.name, polarization=polarization, status="failed", message=message))
if config.fail_fast:
raise GF3PipelineError(message)
continue
if polarization not in metadata.values:
message = f"Missing calibration metadata for {polarization}"
logger.error("%s in %s", message, scene.metadata_xml)
results.append(TaskResult(scene=scene.name, polarization=polarization, status="failed", message=message))
if config.fail_fast:
raise GF3PipelineError(message)
continue
try:
calibration = metadata.for_polarization(polarization)
l1b_path = radiometric_calibrate_l1a_to_l1b(
image_path,
out_dir,
calibration,
block_size=config.block_size,
overwrite=config.overwrite,
logger=logger,
)
l2_path = geometric_correction(
l1b_path,
rpc_path,
out_dir,
dem_path=config.dem_path,
x_res=config.x_res,
y_res=config.y_res,
dst_srs=config.dst_srs,
overwrite=config.overwrite,
logger=logger,
)
results.append(
TaskResult(
scene=scene.name,
polarization=polarization,
status="done",
l1b_path=l1b_path,
l2_path=l2_path,
)
)
except Exception as exc:
logger.exception("Failed %s %s", scene.name, polarization)
results.append(
TaskResult(
scene=scene.name,
polarization=polarization,
status="failed",
message=str(exc),
)
)
if config.fail_fast:
raise
return results
@@ -0,0 +1,70 @@
from __future__ import annotations
import re
from pathlib import Path
from gf3_l1a2l2.errors import RpcError
RPC_KEY_MAP = {
"errBias": "ERR_BIAS",
"errRand": "ERR_RAND",
"lineOffset": "LINE_OFF",
"sampOffset": "SAMP_OFF",
"latOffset": "LAT_OFF",
"longOffset": "LONG_OFF",
"heightOffset": "HEIGHT_OFF",
"lineScale": "LINE_SCALE",
"sampScale": "SAMP_SCALE",
"latScale": "LAT_SCALE",
"longScale": "LONG_SCALE",
"heightScale": "HEIGHT_SCALE",
"lineNumCoef": "LINE_NUM_COEFF",
"lineDenCoef": "LINE_DEN_COEFF",
"sampNumCoef": "SAMP_NUM_COEFF",
"sampDenCoef": "SAMP_DEN_COEFF",
}
COEFFICIENT_KEYS = {
"lineNumCoef",
"lineDenCoef",
"sampNumCoef",
"sampDenCoef",
}
def read_rpc_metadata(rpc_path: Path | str) -> dict[str, str]:
path = Path(rpc_path)
try:
text = path.read_text(encoding="utf-8", errors="ignore")
except OSError as exc:
raise RpcError(f"Cannot read RPC file: {path}") from exc
metadata: dict[str, str] = {}
missing: list[str] = []
for source_key, target_key in RPC_KEY_MAP.items():
value = _find_rpc_value(text, source_key)
if value is None:
missing.append(source_key)
continue
metadata[target_key] = _normalize_rpc_value(source_key, value)
if missing:
raise RpcError(f"RPC file {path} is missing fields: {', '.join(missing)}")
return metadata
def _find_rpc_value(text: str, key: str) -> str | None:
pattern = re.compile(rf"\b{re.escape(key)}\b\s*[:=]?\s*(.*?);", re.IGNORECASE | re.DOTALL)
match = pattern.search(text)
if not match:
return None
return match.group(1)
def _normalize_rpc_value(key: str, raw_value: str) -> str:
value = raw_value.strip().strip("=")
if key in COEFFICIENT_KEYS:
value = value.replace("(", " ").replace(")", " ").replace(",", " ")
return " ".join(value.split())
+8
View File
@@ -0,0 +1,8 @@
<root>
<QualifyValue>
<HH>1.0</HH><HV>2.0</HV><VH>NULL</VH><VV>4.0</VV>
</QualifyValue>
<CalibrationConst>
<HH>10.0</HH><HV>20.0</HV><VH>30.0</VH><VV>40.0</VV>
</CalibrationConst>
</root>
+16
View File
@@ -0,0 +1,16 @@
errBias = -1;
errRand = -1;
lineOffset = 10;
sampOffset = 20;
latOffset = 30;
longOffset = 40;
heightOffset = 50;
lineScale = 60;
sampScale = 70;
latScale = 80;
longScale = 90;
heightScale = 100;
lineNumCoef = (1, 2, 3);
lineDenCoef = (4, 5, 6);
sampNumCoef = (7, 8, 9);
sampDenCoef = (10, 11, 12);
@@ -0,0 +1,19 @@
import math
import unittest
from pathlib import Path
from gf3_l1a2l2.metadata import read_calibration_metadata
class MetadataTests(unittest.TestCase):
def test_reads_named_calibration_values(self):
path = Path(__file__).parent / "fixtures" / "sample.meta.xml"
metadata = read_calibration_metadata(path)
self.assertEqual(metadata.for_polarization("HH").qualify_value, 1.0)
self.assertEqual(metadata.for_polarization("HV").calibration_const, 20.0)
self.assertTrue(math.isnan(metadata.for_polarization("VH").qualify_value))
if __name__ == "__main__":
unittest.main()
+18
View File
@@ -0,0 +1,18 @@
import unittest
from pathlib import Path
from gf3_l1a2l2.rpc import read_rpc_metadata
class RpcTests(unittest.TestCase):
def test_reads_rpc_metadata(self):
path = Path(__file__).parent / "fixtures" / "sample.rpb"
metadata = read_rpc_metadata(path)
self.assertEqual(metadata["ERR_BIAS"], "-1")
self.assertEqual(metadata["LINE_OFF"], "10")
self.assertEqual(metadata["LINE_NUM_COEFF"], "1 2 3")
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,51 @@
# GF-3 SARscape Go Wrapper
This wrapper embeds `assets/gf3_sarscape_cli.sav` and calls the local ENVI/IDL Runtime:
```text
idlrt.exe gf3_sarscape_cli.sav -args input_meta_xml out_dir dem_file HH,HV
```
It is the operational Windows path for SARscape-based GF-3 processing.
## Build
From this directory:
```powershell
$env:GOTELEMETRY = "off"
go build -o ..\..\dist\windows\gf3wrapper.exe .
```
## Run
Single archive:
```powershell
..\..\dist\windows\gf3wrapper.exe `
-input "D:\GF3\GF3_SCENE.tar.gz" `
-output "E:\GF3\L2_SARscape" `
-dem "D:\DEM\COPDEM_GLO30_China_4326_DEM" `
-pol "HH,HV"
```
Batch directory:
```powershell
..\..\dist\windows\gf3wrapper.exe `
-input "D:\GF3\L1A_BATCH" `
-output "E:\GF3\L2_SARscape"
```
If `gf3wrapper.json` exists next to the executable, omitted `-input`, `-output`, `-dem`, and `-pol` values are loaded from that file.
In batch directory mode, a failed scene does not stop the whole run. Invalid or truncated archives, missing metadata, and SARscape failures are reported with the scene input path and reason, then the wrapper continues with the next scene. A final summary lists succeeded and failed scenes, and the process exits non-zero when any scene failed.
Reruns with the same output directory skip scenes that already have complete `*_geo.sml` outputs and matching data files for the requested polarizations. This lets a repaired archive be placed back into the original input folder without reprocessing scenes that already finished.
## Notes
- The server still needs ENVI, IDL Runtime, and SARscape installed and licensed.
- `.tar.gz` inputs are extracted under each scene output directory in `.gf3_extract`.
- The embedded SAV is extracted to `<output>\.gf3_runtime\gf3_sarscape_cli.sav`.
- More operational details are in `docs/sarscape_go_wrapper.md`.
@@ -0,0 +1,3 @@
module gf3wrapper
go 1.22
@@ -0,0 +1,5 @@
.compile 'C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\gf3_sarscape_pipeline.pro'
.compile 'C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\gf3_sarscape_cli.pro'
resolve_all, skip_routines='envi', /continue_on_error
save, /routines, filename='C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\gf3_sarscape_cli.sav'
exit
@@ -0,0 +1,5 @@
.f
.compile 'C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\gf3_sarscape_pipeline.pro'
.compile 'C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\gf3_sarscape_cli.pro'
save, /routines, filename='C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\gf3_sarscape_cli.sav'
exit
@@ -0,0 +1,79 @@
function gf3_cli_split_csv, value
compile_opt idl2
text = strtrim(value, 2)
if text eq '' then return, ['HH', 'HV']
parts = ['']
first = 1
while 1 do begin
pos = strpos(text, ',')
if pos lt 0 then begin
token = strtrim(text, 2)
if first then parts = [token] else parts = [parts, token]
break
endif
token = strtrim(strmid(text, 0, pos), 2)
if first then begin
parts = [token]
first = 0
endif else begin
parts = [parts, token]
endelse
text = strtrim(strmid(text, pos + 1), 2)
endwhile
return, parts
end
pro gf3_sarscape_cli
compile_opt idl2
args = command_line_args(count=argc)
if argc lt 3 then begin
print, 'Usage: idlrt.exe gf3_sarscape_cli.sav -args input_meta_xml out_dir dem_file [polarizations]'
print, 'Example polarizations: HH,HV or HH or HV'
return
endif
input_meta_xml = args[0]
out_dir = args[1]
dem_file = args[2]
file_mkdir, out_dir
log_file = out_dir + '\gf3_sarscape_cli.log'
openw, lun, log_file, /get_lun
catch, err
if err ne 0 then begin
printf, lun, 'ERROR: ', !error_state.msg
free_lun, lun
catch, /cancel
return
endif
printf, lun, 'CLI start'
printf, lun, 'input_meta_xml=', input_meta_xml
printf, lun, 'out_dir=', out_dir
printf, lun, 'dem_file=', dem_file
if argc ge 4 then begin
polarizations = gf3_cli_split_csv(args[3])
endif else begin
polarizations = ['HH', 'HV']
endelse
printf, lun, 'polarizations=', strjoin(polarizations, ',')
e = envi(/headless)
printf, lun, 'ENVI headless initialized'
gf3_sarscape_run, $
input_meta_xml=input_meta_xml, $
out_dir=out_dir, $
dem_file=dem_file, $
polarizations=polarizations
printf, lun, 'CLI end'
free_lun, lun
end
@@ -0,0 +1,506 @@
function gf3_normalize_errmsg, value
compile_opt idl2
if n_elements(value) eq 0 then return, ''
sz = size(value, /tname)
if sz eq 'STRING' then begin
if n_elements(value) eq 1 then return, value
return, strjoin(value, ' ')
endif
if sz eq 'BYTE' then begin
if n_elements(value) eq 0 then return, ''
return, string(value)
endif
return, string(value)
end
function gf3_float_str, value
compile_opt idl2
return, string(double(value), format='(F0.1)')
end
function gf3_base_root, polarization
compile_opt idl2
common gf3_pipeline_cfg, cfg
pol = strupcase(strtrim(polarization, 2))
case pol of
'HH': return, cfg.imported_hh_root
'HV': return, cfg.imported_hv_root
else: message, 'Unsupported polarization: ' + pol
endcase
end
function gf3_stage_root, polarization, stage_name
compile_opt idl2
common gf3_pipeline_cfg, cfg
pol = strlowcase(strtrim(polarization, 2))
prefix = cfg.output_prefix
if prefix eq '' then prefix = 'gf3'
case strlowcase(strtrim(stage_name, 2)) of
'multilook': return, cfg.out_dir + '\' + prefix + '_' + pol + '_ml'
'filter': return, cfg.out_dir + '\' + prefix + '_' + pol + '_filt'
'geocode': return, cfg.out_dir + '\' + prefix + '_' + pol + '_geo'
else: message, 'Unsupported stage: ' + stage_name
endcase
end
function gf3_scene_token, meta_xml
compile_opt idl2
base = file_basename(meta_xml)
pos = strpos(strlowcase(base), '.meta.xml')
if pos ge 0 then base = strmid(base, 0, pos)
return, base
end
function gf3_imported_root_for, polarization
compile_opt idl2
common gf3_pipeline_cfg, cfg
pol = strupcase(strtrim(polarization, 2))
pattern = cfg.out_dir + '\*_' + pol + '_slc.sml'
matches = file_search(pattern, count=count)
if count eq 0 then return, ''
newest = matches[0]
newest_time = (file_info(newest)).mtime
for i = 1, count - 1 do begin
info = file_info(matches[i])
if info.mtime gt newest_time then begin
newest = matches[i]
newest_time = info.mtime
endif
endfor
return, strmid(newest, 0, strlen(newest) - 4)
end
pro gf3_refresh_import_roots
compile_opt idl2
common gf3_pipeline_cfg, cfg
hh = gf3_imported_root_for('HH')
hv = gf3_imported_root_for('HV')
if hh ne '' then cfg.imported_hh_root = hh
if hv ne '' then cfg.imported_hv_root = hv
end
pro gf3_run_module, module_name, set_params_proc, errmsg=errmsg, working_directory=working_directory
compile_opt idl2
errmsg = ''
ob = obj_new('SARscapeBatch', Module=module_name)
if ~obj_valid(ob) then begin
errmsg = 'Create object failed: ' + module_name
print, errmsg
return
endif
ob->SetParam, 'GENERAL_PARAMETERS_CMD.AVAILABLE_MEMORY_SIZE_GB', '8.0000000'
ob->SetParam, 'GENERAL_PARAMETERS_CMD.LOAD_IMAGES', 'OK'
ob->SetParam, 'GENERAL_PARAMETERS_CMD.DELETE_TEMPORARY_FILES', 'OK'
ob->SetParam, 'GENERAL_PARAMETERS_CMD.MAKE_TIFF', 'OK'
ob->SetParam, 'GENERAL_PARAMETERS_CMD.QUICK_LOOK_FORMAT', 'ql_tiff'
ob->SetParam, 'GENERAL_PARAMETERS_CMD.SARSCAPE_TRACE_LEVEL', '3.0000000'
ob->SetParam, 'GENERAL_PARAMETERS_CMD.RENAME_THE_FILE_USING_PARAMETERS_FLAG', 'OK'
ob->SetParam, 'GENERAL_PARAMETERS_CMD.INSERT_GEO_POINTS_FLAG', 'NotOK'
ob->SetParam, 'GENERAL_PARAMETERS_CMD.VERBOSE_TRACE', 'OK'
ob->SetParam, 'GENERAL_PARAMETERS_CMD.VERBOSE_STEP', 'OK'
ob->SetParam, 'GENERAL_PARAMETERS_CMD.VERBOSE_BAR', 'OK'
ob->SetParam, 'GENERAL_PARAMETERS_CMD.APPLY_CALIBRATION_CONSTANT_FLAG', 'OK'
ob->SetParam, 'GENERAL_PARAMETERS_CMD.FILL_DUMMY_DURING_IMPORT', 'OK'
call_procedure, set_params_proc, ob
ok = ob->VerifyParams(Silent=0)
if ~ok then begin
errmsg = 'VerifyParams failed: ' + module_name
print, errmsg
return
endif
if n_elements(working_directory) eq 0 then begin
ok = ob->Execute()
endif else begin
exec_errmsg = ''
ok = ob->Execute(WORKING_DIRECTORY=working_directory, ERRMSG=exec_errmsg)
errmsg = gf3_normalize_errmsg(exec_errmsg)
endelse
if ok then begin
print, 'Success: ' + module_name
endif else begin
if errmsg eq '' then begin
err_code = ''
err_text = get_SARscape_error_string('OK', ERROR_CODE=err_code)
errmsg = 'Execution failed [' + module_name + '] EC [' + err_code + ']: ' + err_text
endif
print, errmsg
endelse
end
pro gf3_pipeline_set_import_params, ob
compile_opt idl2
common gf3_pipeline_cfg, cfg
ob->SetParam, 'MAIN_BASIC_IMPORT_GAOFEN3_CMD_CMD.SARSCAPEENVIRONMENT', 'IDL_ENVI_ENV'
ob->SetParam, 'MAIN_BASIC_IMPORT_GAOFEN3_CMD_CMD.INPUT_FILE_LIST', [cfg.input_meta_xml]
ob->SetParam, 'MAIN_BASIC_IMPORT_GAOFEN3_CMD_CMD.OUTPUT_FILE_LIST', [cfg.import_root]
ob->SetParam, 'MAIN_BASIC_IMPORT_GAOFEN3_CMD_CMD.CROSS_COPOLARIZATION_FLAG', cfg.import_polarization_mode
ob->SetParam, 'MAIN_BASIC_IMPORT_GAOFEN3_CMD_CMD.DOPP_ESTIMATION_FROM_RASTER_FLAG', 'OK'
ob->SetParam, 'MAIN_BASIC_IMPORT_GAOFEN3_CMD_CMD.KEEP_ALL_LINES_FLAG', 'NotOK'
end
pro gf3_pipeline_set_multilook_params, ob
compile_opt idl2
common gf3_pipeline_cfg, cfg
common gf3_pipeline_state, state
ob->SetParam, 'MAIN_BASIC_MULTILOOKING.SARSCAPEENVIRONMENT', 'IDL_ENVI_ENV'
ob->SetParam, 'MAIN_BASIC_MULTILOOKING.INPUT_FILE_LIST', [state.current_input_root]
ob->SetParam, 'MAIN_BASIC_MULTILOOKING.OUTPUT_FILE_LIST', [state.current_output_root]
ob->SetParam, 'MAIN_BASIC_MULTILOOKING.MULTILOOK_METHOD', cfg.multilook_method
ob->SetParam, 'MAIN_BASIC_MULTILOOKING.RANGE_MULTILOOK', gf3_float_str(cfg.range_looks)
ob->SetParam, 'MAIN_BASIC_MULTILOOKING.AZIMUTH_MULTILOOK', gf3_float_str(cfg.azimuth_looks)
ob->SetParam, 'MAIN_BASIC_MULTILOOKING.GRID_SIZE_FOR_SUGGESTED_LOOKS', gf3_float_str(cfg.grid_size_for_suggested_looks)
ob->SetParam, 'MAIN_BASIC_MULTILOOKING.FILL_DUMMY_FLAG', 'OK'
ob->SetParam, 'MAIN_BASIC_MULTILOOKING.FILL_DUMMY_METHOD', 'min_image_value'
ob->SetParam, 'MAIN_BASIC_MULTILOOKING.ROWS_WINDOW_NUMBER', '3.0000000'
ob->SetParam, 'MAIN_BASIC_MULTILOOKING.COLS_WINDOW_NUMBER', '3.0000000'
end
pro gf3_pipeline_set_filter_params, ob
compile_opt idl2
common gf3_pipeline_cfg, cfg
common gf3_pipeline_state, state
ob->SetParam, 'MAIN_BASIC_DESPECKLE_CONVENTIONAL_CMD.SARSCAPEENVIRONMENT', 'IDL_ENVI_ENV'
ob->SetParam, 'MAIN_BASIC_DESPECKLE_CONVENTIONAL_CMD.INPUT_FILE_LIST', [state.current_input_root]
ob->SetParam, 'MAIN_BASIC_DESPECKLE_CONVENTIONAL_CMD.OUTPUT_FILE_LIST', [state.current_output_root]
ob->SetParam, 'MAIN_BASIC_DESPECKLE_CONVENTIONAL_CMD.FILT_TYPE', cfg.filter_type
ob->SetParam, 'MAIN_BASIC_DESPECKLE_CONVENTIONAL_CMD.EQUIVALENT_LOOKS', '-1.0000000'
ob->SetParam, 'PARAMETERS_GENERIC_FILTERS.WIN_SIZE', gf3_float_str(cfg.filter_window)
ob->SetParam, 'PARAMETERS_GENERIC_FILTERS.ROWS_WINDOW_NUMBER', gf3_float_str(cfg.filter_window)
ob->SetParam, 'PARAMETERS_GENERIC_FILTERS.COLS_WINDOW_NUMBER', gf3_float_str(cfg.filter_window)
ob->SetParam, 'PARAMETERS_GENERIC_FILTERS.WIN_MODE_SIZE', gf3_float_str(cfg.filter_window)
end
pro gf3_pipeline_set_geocode_params, ob
compile_opt idl2
common gf3_pipeline_cfg, cfg
common gf3_pipeline_state, state
ob->SetParam, 'OUT_CARTOGRAPHIC_SYSTEM.OCS_CARTOGRAPHIC_SYSTEM_SPECIFICATION', ''
ob->SetParam, 'OUT_CARTOGRAPHIC_SYSTEM.OCS_STATE', cfg.ocs_state
ob->SetParam, 'OUT_CARTOGRAPHIC_SYSTEM.OCS_HEMISPHERE', cfg.ocs_hemisphere
ob->SetParam, 'OUT_CARTOGRAPHIC_SYSTEM.OCS_PROJECTION', cfg.ocs_projection
ob->SetParam, 'OUT_CARTOGRAPHIC_SYSTEM.OCS_ELLIPSOID', 'WGS84'
ob->SetParam, 'OUT_CARTOGRAPHIC_SYSTEM.OCS_REFERENCE_HEIGHT', '0.0'
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.SARSCAPEENVIRONMENT', 'IDL_ENVI_ENV'
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.INPUT_FILE_LIST', [state.current_input_root]
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.OUTPUT_FILE_LIST', [state.current_output_root]
if cfg.dem_file ne '' then ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.DEM_FILE_NAME', cfg.dem_file
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GEOCODE_GRID_SIZE_X', gf3_float_str(cfg.geocode_grid_x)
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GEOCODE_GRID_SIZE_Y', gf3_float_str(cfg.geocode_grid_y)
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GEOCODE_RESAMPLING_TYPE', cfg.geocode_resampling
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.CALIBRATION_FLAG', cfg.calibration_flag
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GEO_SCATTERING_AREA_METHOD', 'sine_area_estimation'
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.TRUE_AREA_EXPONENT', '1.0000000'
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.MAX_VALUE_IN_CALIBRATION', '100.00000'
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.RAD_NORMALIZATION_FLAG', cfg.rad_normalization_flag
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GEOCODE_RAD_NORM_DEG', '2.0000000'
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GEOCODE_RAD_NORM_ANG', '-1.0000000'
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GEOCODE_SIGMA_FLAG', cfg.output_sigma_flag
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GEOCODE_GAMMA_FLAG', cfg.output_gamma_flag
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GEOCODE_BETA_FLAG', cfg.output_beta_flag
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GEOCODE_COREG_IMAGES_FLAG', 'NotOK'
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GENERATE_LIA_FLAG', 'NotOK'
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.OUTPUT_TYPE', cfg.output_type
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.DUMMY_REMOVAL_FLAG', 'NotOK'
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GEOCODE_BLOCK_SIZE', '30000.000'
end
pro gf3_init_default_cfg
compile_opt idl2
common gf3_pipeline_cfg, cfg
cfg = { $
input_meta_xml: 'C:\Users\Administrator\Desktop\GF3_Test\GF3_scene_20260513\GF3_SYC_FSI_051370_E124.6_N51.7_20260513_L1A_HHHV_L10007355955.meta.xml', $
work_dir: 'C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\work', $
temp_dir: 'C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\temp', $
out_dir: 'C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\out', $
output_prefix: '', $
dem_file: 'D:\DEM\COPDEM_GLO30_China_4326_DEM', $
import_root: 'C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\out\gf3_import', $
import_polarization_mode: 'ALL_POL', $
imported_hh_root: 'C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\out\gaofer3_20260513_214340051_D_HH_slc', $
imported_hv_root: 'C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\out\gaofer3_20260513_214340051_D_HV_slc', $
multilook_method: 'time_domain', $
range_looks: 2.0, $
azimuth_looks: 2.0, $
grid_size_for_suggested_looks: 15.0, $
filter_type: 'lee', $
filter_window: 5.0, $
geocode_grid_x: 15.0, $
geocode_grid_y: 15.0, $
geocode_resampling: '4th_order_cc', $
calibration_flag: 'OK', $
rad_normalization_flag: 'OK', $
output_sigma_flag: 'OK', $
output_gamma_flag: 'NotOK', $
output_beta_flag: 'NotOK', $
output_type: 'output_type_linear', $
ocs_state: 'GEO-GLOBAL', $
ocs_hemisphere: 'NONE', $
ocs_projection: 'GEO' $
}
end
pro gf3_configure, input_meta_xml=input_meta_xml, out_dir=out_dir, dem_file=dem_file, $
work_dir=work_dir, temp_dir=temp_dir, range_looks=range_looks, $
azimuth_looks=azimuth_looks, filter_type=filter_type, filter_window=filter_window, $
geocode_grid_x=geocode_grid_x, geocode_grid_y=geocode_grid_y, $
imported_hh_root=imported_hh_root, imported_hv_root=imported_hv_root
compile_opt idl2
common gf3_pipeline_cfg, cfg
gf3_init_default_cfg
if n_elements(input_meta_xml) gt 0 then cfg.input_meta_xml = input_meta_xml
if n_elements(out_dir) gt 0 then cfg.out_dir = out_dir
if n_elements(dem_file) gt 0 then cfg.dem_file = dem_file
if n_elements(work_dir) gt 0 then cfg.work_dir = work_dir
if n_elements(temp_dir) gt 0 then cfg.temp_dir = temp_dir
if n_elements(range_looks) gt 0 then cfg.range_looks = double(range_looks)
if n_elements(azimuth_looks) gt 0 then cfg.azimuth_looks = double(azimuth_looks)
if n_elements(filter_type) gt 0 then cfg.filter_type = filter_type
if n_elements(filter_window) gt 0 then cfg.filter_window = double(filter_window)
if n_elements(geocode_grid_x) gt 0 then cfg.geocode_grid_x = double(geocode_grid_x)
if n_elements(geocode_grid_y) gt 0 then cfg.geocode_grid_y = double(geocode_grid_y)
if n_elements(imported_hh_root) gt 0 then cfg.imported_hh_root = imported_hh_root
if n_elements(imported_hv_root) gt 0 then cfg.imported_hv_root = imported_hv_root
cfg.import_root = cfg.out_dir + '\gf3_import'
cfg.output_prefix = gf3_scene_token(cfg.input_meta_xml)
if n_elements(work_dir) eq 0 then cfg.work_dir = cfg.out_dir + '\work'
if n_elements(temp_dir) eq 0 then cfg.temp_dir = cfg.out_dir + '\temp'
if n_elements(imported_hh_root) eq 0 then cfg.imported_hh_root = ''
if n_elements(imported_hv_root) eq 0 then cfg.imported_hv_root = ''
end
pro gf3_prepare_dirs
compile_opt idl2
common gf3_pipeline_cfg, cfg
file_mkdir, cfg.work_dir
file_mkdir, cfg.temp_dir
file_mkdir, cfg.out_dir
end
pro gf3_start_batch
compile_opt idl2
common gf3_pipeline_cfg, cfg
SARscape_Batch_Init, Temp_Directory=cfg.temp_dir
end
pro gf3_stop_batch
compile_opt idl2
SARscape_Batch_Exit
end
pro gf3_run_import, errmsg=errmsg
compile_opt idl2
common gf3_pipeline_cfg, cfg
errmsg = ''
gf3_run_module, 'ImportGaofen3', 'gf3_pipeline_set_import_params', errmsg=errmsg, working_directory=cfg.work_dir
end
pro gf3_run_multilook, polarization, errmsg=errmsg
compile_opt idl2
common gf3_pipeline_cfg, cfg
common gf3_pipeline_state, state
state = {current_input_root: gf3_base_root(polarization), current_output_root: gf3_stage_root(polarization, 'multilook')}
errmsg = ''
gf3_run_module, 'BaseMultilooking', 'gf3_pipeline_set_multilook_params', errmsg=errmsg, working_directory=cfg.work_dir
end
pro gf3_run_filter, polarization, errmsg=errmsg
compile_opt idl2
common gf3_pipeline_cfg, cfg
common gf3_pipeline_state, state
state = {current_input_root: gf3_stage_root(polarization, 'multilook'), current_output_root: gf3_stage_root(polarization, 'filter')}
errmsg = ''
gf3_run_module, 'DespeckleConventionalSingle', 'gf3_pipeline_set_filter_params', errmsg=errmsg, working_directory=cfg.work_dir
end
pro gf3_run_geocode, polarization, errmsg=errmsg
compile_opt idl2
common gf3_pipeline_cfg, cfg
common gf3_pipeline_state, state
state = {current_input_root: gf3_stage_root(polarization, 'filter'), current_output_root: gf3_stage_root(polarization, 'geocode')}
errmsg = ''
gf3_run_module, 'BasicGeocoding', 'gf3_pipeline_set_geocode_params', errmsg=errmsg, working_directory=cfg.work_dir
end
pro gf3_run_single_pol_pipeline, polarization, do_import=do_import, errmsg=errmsg
compile_opt idl2
errmsg = ''
if keyword_set(do_import) then begin
gf3_run_import, errmsg=errmsg
if errmsg ne '' then return
gf3_refresh_import_roots
endif
gf3_run_multilook, polarization, errmsg=errmsg
if errmsg ne '' then return
gf3_run_filter, polarization, errmsg=errmsg
if errmsg ne '' then return
gf3_run_geocode, polarization, errmsg=errmsg
end
function gf3_should_run_pol, polarizations, polarization
compile_opt idl2
if n_elements(polarizations) eq 0 then return, 1
wanted = strupcase(strtrim(polarizations, 2))
pol = strupcase(strtrim(polarization, 2))
for i = 0, n_elements(wanted) - 1 do begin
if wanted[i] eq 'ALL' then return, 1
if wanted[i] eq pol then return, 1
endfor
return, 0
end
pro gf3_sarscape_run, input_meta_xml=input_meta_xml, out_dir=out_dir, dem_file=dem_file, $
polarizations=polarizations, skip_import=skip_import, work_dir=work_dir, temp_dir=temp_dir, $
range_looks=range_looks, azimuth_looks=azimuth_looks, filter_type=filter_type, $
filter_window=filter_window, geocode_grid_x=geocode_grid_x, geocode_grid_y=geocode_grid_y
compile_opt idl2
gf3_configure, input_meta_xml=input_meta_xml, out_dir=out_dir, dem_file=dem_file, $
work_dir=work_dir, temp_dir=temp_dir, range_looks=range_looks, $
azimuth_looks=azimuth_looks, filter_type=filter_type, filter_window=filter_window, $
geocode_grid_x=geocode_grid_x, geocode_grid_y=geocode_grid_y
gf3_prepare_dirs
gf3_start_batch
errmsg = ''
if ~keyword_set(skip_import) then begin
gf3_run_import, errmsg=errmsg
if errmsg eq '' then gf3_refresh_import_roots
endif else begin
gf3_refresh_import_roots
endelse
if errmsg eq '' && gf3_should_run_pol(polarizations, 'HH') then gf3_run_single_pol_pipeline, 'HH', errmsg=errmsg
if errmsg eq '' && gf3_should_run_pol(polarizations, 'HV') then gf3_run_single_pol_pipeline, 'HV', errmsg=errmsg
gf3_stop_batch
if errmsg ne '' then print, 'Pipeline stopped: ' + errmsg
end
pro gf3_sarscape_pipeline
compile_opt idl2
common gf3_pipeline_cfg, cfg
gf3_init_default_cfg
gf3_prepare_dirs
gf3_start_batch
errmsg = ''
gf3_run_import, errmsg=errmsg
if errmsg eq '' then gf3_refresh_import_roots
if errmsg eq '' then gf3_run_single_pol_pipeline, 'HH', errmsg=errmsg
if errmsg eq '' then gf3_run_single_pol_pipeline, 'HV', errmsg=errmsg
gf3_stop_batch
if errmsg ne '' then print, 'Pipeline stopped: ' + errmsg
end
pro gf3_sarscape_import_only
compile_opt idl2
gf3_init_default_cfg
gf3_prepare_dirs
gf3_start_batch
errmsg = ''
gf3_run_import, errmsg=errmsg
gf3_stop_batch
if errmsg ne '' then print, 'Import stopped: ' + errmsg
end
pro gf3_sarscape_hh_only
compile_opt idl2
gf3_init_default_cfg
gf3_prepare_dirs
gf3_start_batch
errmsg = ''
gf3_run_single_pol_pipeline, 'HH', do_import=1, errmsg=errmsg
gf3_stop_batch
if errmsg ne '' then print, 'HH pipeline stopped: ' + errmsg
end
pro gf3_sarscape_hv_only
compile_opt idl2
gf3_init_default_cfg
gf3_prepare_dirs
gf3_start_batch
errmsg = ''
gf3_run_single_pol_pipeline, 'HV', do_import=1, errmsg=errmsg
gf3_stop_batch
if errmsg ne '' then print, 'HV pipeline stopped: ' + errmsg
end
@@ -0,0 +1,8 @@
@echo off
set IDLRT=C:\Program Files\Harris\ENVI56\IDL88\bin\bin.x86_64\idlrt.exe
set SAV=C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\gf3_sarscape_cli.sav
set META=C:\Users\Administrator\Desktop\GF3_Test\GF3_scene_20260513\GF3_SYC_FSI_051370_E124.6_N51.7_20260513_L1A_HHHV_L10007355955.meta.xml
set OUT=C:\Users\Administrator\Desktop\GF3_Test\GF3_cli_test_out_20260513
set DEM=D:\DEM\COPDEM_GLO30_China_4326_DEM
"%IDLRT%" "%SAV%" -args "%META%" "%OUT%" "%DEM%" "HH,HV"
@@ -0,0 +1,768 @@
package main
import (
"archive/tar"
"bufio"
"compress/gzip"
"embed"
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"io/fs"
"os"
"os/exec"
"path/filepath"
"sort"
"strings"
"time"
)
//go:embed assets/gf3_sarscape_cli.sav
var embedded embed.FS
type config struct {
configPath string `json:"-"`
input string `json:"input"`
output string `json:"output"`
dem string `json:"dem"`
polarizations string `json:"polarizations"`
idlrt string `json:"idlrt"`
keepExtracted bool `json:"keep_extracted"`
}
type scene struct {
metaPath string
archivePath string
name string
}
type sceneFailure struct {
name string
input string
err error
}
type batchError struct {
failures []sceneFailure
}
func (e batchError) Error() string {
return fmt.Sprintf("batch completed with %d failed scene(s)", len(e.failures))
}
type completedProduct struct {
polarization string
path string
}
func main() {
cfg := parseFlags()
if len(os.Args) == 1 {
var err error
cfg, err = promptConfig(cfg)
if err != nil {
fmt.Printf("ERROR: %v\n", err)
waitForEnter()
os.Exit(1)
}
}
if err := run(cfg); err != nil {
fmt.Printf("ERROR: %v\n", err)
if len(os.Args) == 1 {
waitForEnter()
}
os.Exit(1)
}
if err := saveConfig(cfg.configPath, cfg); err != nil {
fmt.Printf("WARNING: save config: %v\n", err)
}
if len(os.Args) == 1 {
fmt.Println("Done.")
waitForEnter()
}
}
func parseFlags() config {
cfg := defaultConfig()
configPath := preScanConfigPath()
cfg.configPath = configPath
_ = loadConfig(configPath, &cfg)
flag.StringVar(&cfg.configPath, "config", configPath, "Config file path")
flag.StringVar(&cfg.input, "input", cfg.input, "GF-3 input: .meta.xml, .tar.gz, or directory")
flag.StringVar(&cfg.output, "output", cfg.output, "Output directory")
flag.StringVar(&cfg.dem, "dem", cfg.dem, "SARscape DEM file")
flag.StringVar(&cfg.polarizations, "pol", cfg.polarizations, "Polarizations: HH, HV, or HH,HV")
flag.StringVar(&cfg.idlrt, "idlrt", cfg.idlrt, "Path to idlrt.exe")
flag.BoolVar(&cfg.keepExtracted, "keep-extracted", cfg.keepExtracted, "Keep extracted archives under output/.gf3_extract")
flag.Parse()
return cfg
}
func defaultConfig() config {
return config{
dem: `D:\DEM\COPDEM_GLO30_China_4326_DEM`,
polarizations: "HH,HV",
idlrt: defaultIDLRT(),
keepExtracted: true,
}
}
func preScanConfigPath() string {
defaultPath := defaultConfigPath()
for i := 1; i < len(os.Args)-1; i++ {
if os.Args[i] == "-config" {
return os.Args[i+1]
}
}
return defaultPath
}
func defaultConfigPath() string {
exe, err := os.Executable()
if err != nil {
return "gf3wrapper.json"
}
return filepath.Join(filepath.Dir(exe), "gf3wrapper.json")
}
func loadConfig(path string, cfg *config) error {
data, err := os.ReadFile(path)
if err != nil {
return err
}
var stored struct {
IDLRTPath string `json:"idlrt_path"`
DemFile string `json:"dem_file"`
Polarizations string `json:"polarizations"`
LastInput string `json:"last_input"`
LastOutput string `json:"last_output"`
IDLRT string `json:"idlrt"`
Dem string `json:"dem"`
Input string `json:"input"`
Output string `json:"output"`
KeepExtracted *bool `json:"keep_extracted"`
}
if err := json.Unmarshal(data, &stored); err != nil {
return err
}
if stored.IDLRTPath != "" {
cfg.idlrt = stored.IDLRTPath
} else if stored.IDLRT != "" {
cfg.idlrt = stored.IDLRT
}
if stored.DemFile != "" {
cfg.dem = stored.DemFile
} else if stored.Dem != "" {
cfg.dem = stored.Dem
}
if stored.Polarizations != "" {
cfg.polarizations = stored.Polarizations
}
if stored.LastInput != "" {
cfg.input = stored.LastInput
} else if stored.Input != "" {
cfg.input = stored.Input
}
if stored.LastOutput != "" {
cfg.output = stored.LastOutput
} else if stored.Output != "" {
cfg.output = stored.Output
}
if stored.KeepExtracted != nil {
cfg.keepExtracted = *stored.KeepExtracted
}
cfg.configPath = path
return nil
}
func saveConfig(path string, cfg config) error {
if path == "" {
path = defaultConfigPath()
}
stored := struct {
IDLRTPath string `json:"idlrt_path"`
DemFile string `json:"dem_file"`
Polarizations string `json:"polarizations"`
LastInput string `json:"last_input"`
LastOutput string `json:"last_output"`
KeepExtracted bool `json:"keep_extracted"`
}{
IDLRTPath: cfg.idlrt,
DemFile: cfg.dem,
Polarizations: cfg.polarizations,
LastInput: cfg.input,
LastOutput: cfg.output,
KeepExtracted: cfg.keepExtracted,
}
data, err := json.MarshalIndent(stored, "", " ")
if err != nil {
return err
}
return os.WriteFile(path, data, 0o644)
}
func promptConfig(defaults config) (config, error) {
reader := bufio.NewReader(os.Stdin)
cfg := defaults
fmt.Println("GF-3 SARscape Wrapper")
fmt.Println()
cfg.input = prompt(reader, "Input .tar.gz / .meta.xml / folder", cfg.input)
cfg.output = prompt(reader, "Output folder", cfg.output)
cfg.dem = prompt(reader, "DEM file", cfg.dem)
cfg.polarizations = prompt(reader, "Polarizations", cfg.polarizations)
cfg.idlrt = prompt(reader, "IDL Runtime", cfg.idlrt)
if cfg.input == "" || cfg.output == "" || cfg.dem == "" {
return cfg, errors.New("input, output, and DEM are required")
}
return cfg, nil
}
func prompt(reader *bufio.Reader, label, defaultValue string) string {
if defaultValue != "" {
fmt.Printf("%s [%s]: ", label, defaultValue)
} else {
fmt.Printf("%s: ", label)
}
text, _ := reader.ReadString('\n')
text = strings.TrimSpace(text)
if text == "" {
return defaultValue
}
return text
}
func waitForEnter() {
fmt.Println()
fmt.Print("Press Enter to exit...")
_, _ = bufio.NewReader(os.Stdin).ReadString('\n')
}
func run(cfg config) error {
if cfg.input == "" || cfg.output == "" || cfg.dem == "" {
return errors.New("required flags: -input, -output, -dem")
}
if _, err := os.Stat(cfg.idlrt); err != nil {
return fmt.Errorf("idlrt not found: %s: %w", cfg.idlrt, err)
}
if _, err := os.Stat(cfg.dem); err != nil {
return fmt.Errorf("DEM not found: %s: %w", cfg.dem, err)
}
if err := os.MkdirAll(cfg.output, 0o755); err != nil {
return err
}
scenes, err := discoverScenes(cfg)
if err != nil {
return err
}
if len(scenes) == 0 {
return fmt.Errorf("no *.meta.xml scenes found in %s", cfg.input)
}
failures := make([]sceneFailure, 0)
processed := 0
skipped := 0
savPath := ""
for _, sc := range scenes {
sceneOut := filepath.Join(cfg.output, sc.name)
if products, ok := completedSceneOutputs(sceneOut, sc.name, cfg.polarizations); ok {
recordSceneSkip(sc, sceneOut, products)
skipped++
continue
}
if err := os.MkdirAll(sceneOut, 0o755); err != nil {
recordSceneFailure(&failures, sc, err)
continue
}
if sc.archivePath != "" {
extractDir := filepath.Join(sceneOut, ".gf3_extract")
fmt.Printf("Extracting %s\n", sc.archivePath)
if err := extractTarGz(sc.archivePath, extractDir); err != nil {
recordSceneFailure(&failures, sc, fmt.Errorf("failed to extract archive before processing scene %s: %w", sc.archivePath, err))
continue
}
extractedScenes, err := findMetaXML(extractDir)
if err != nil {
recordSceneFailure(&failures, sc, err)
continue
}
if len(extractedScenes) == 0 {
recordSceneFailure(&failures, sc, fmt.Errorf("no *.meta.xml found in archive: %s", sc.archivePath))
continue
}
sc.metaPath = extractedScenes[0].metaPath
sc.name = sceneName(sc.metaPath)
}
if savPath == "" {
var err error
savPath, err = materializeSAV(cfg.output)
if err != nil {
recordSceneFailure(&failures, sc, err)
continue
}
}
if err := runScene(cfg, savPath, sc, sceneOut); err != nil {
recordSceneFailure(&failures, sc, err)
continue
}
processed++
}
printBatchSummary(len(scenes), processed, skipped, failures)
if len(failures) > 0 {
return batchError{failures: failures}
}
return nil
}
func sceneInput(sc scene) string {
if sc.archivePath != "" {
return sc.archivePath
}
if sc.metaPath != "" {
return sc.metaPath
}
return sc.name
}
func recordSceneFailure(failures *[]sceneFailure, sc scene, err error) {
input := sceneInput(sc)
failure := sceneFailure{name: sc.name, input: input, err: err}
*failures = append(*failures, failure)
fmt.Printf("FAILED %s\n", sc.name)
fmt.Printf("Input %s\n", input)
fmt.Printf("Reason: %v\n", err)
fmt.Println("Continuing with next scene.")
}
func recordSceneSkip(sc scene, sceneOut string, products []completedProduct) {
parts := make([]string, 0, len(products))
for _, product := range products {
parts = append(parts, product.polarization+"="+product.path)
}
fmt.Printf("SKIPPED %s\n", sc.name)
fmt.Printf("Input %s\n", sceneInput(sc))
fmt.Printf("Output already complete %s\n", sceneOut)
fmt.Printf("Products %s\n", strings.Join(parts, "; "))
}
func printBatchSummary(total, processed, skipped int, failures []sceneFailure) {
fmt.Println()
fmt.Printf("Batch summary: %d processed, %d skipped, %d failed, %d total\n", processed, skipped, len(failures), total)
if len(failures) == 0 {
return
}
fmt.Println("Failed scenes:")
for _, failure := range failures {
fmt.Printf("- %s (%s): %v\n", failure.name, failure.input, failure.err)
}
}
func materializeSAV(outputDir string) (string, error) {
data, err := embedded.ReadFile("assets/gf3_sarscape_cli.sav")
if err != nil {
return "", err
}
runtimeDir := filepath.Join(outputDir, ".gf3_runtime")
if err := os.MkdirAll(runtimeDir, 0o755); err != nil {
return "", err
}
savPath := filepath.Join(runtimeDir, "gf3_sarscape_cli.sav")
if err := os.WriteFile(savPath, data, 0o644); err != nil {
return "", err
}
return savPath, nil
}
func discoverScenes(cfg config) ([]scene, error) {
info, err := os.Stat(cfg.input)
if err != nil {
return nil, err
}
if info.IsDir() {
return discoverScenesInDir(cfg.input, cfg.output)
}
lower := strings.ToLower(cfg.input)
if strings.HasSuffix(lower, ".meta.xml") {
return []scene{{metaPath: cfg.input, name: sceneName(cfg.input)}}, nil
}
if strings.HasSuffix(lower, ".tar.gz") || strings.HasSuffix(lower, ".tgz") {
return []scene{{archivePath: cfg.input, name: safeName(trimArchiveExt(filepath.Base(cfg.input)))}}, nil
}
return nil, fmt.Errorf("unsupported input type: %s", cfg.input)
}
func discoverScenesInDir(inputDir, outputDir string) ([]scene, error) {
_ = outputDir
scenesByName := make(map[string]scene)
err := filepath.WalkDir(inputDir, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() {
if d.Name() == ".gf3_extract" {
return filepath.SkipDir
}
return nil
}
lower := strings.ToLower(d.Name())
switch {
case strings.HasSuffix(lower, ".meta.xml"):
sc := scene{metaPath: path, name: sceneName(path)}
key := strings.ToLower(sc.name)
if _, ok := scenesByName[key]; !ok {
scenesByName[key] = sc
}
case strings.HasSuffix(lower, ".tar.gz") || strings.HasSuffix(lower, ".tgz"):
sc := scene{archivePath: path, name: safeName(trimArchiveExt(filepath.Base(path)))}
scenesByName[strings.ToLower(sc.name)] = sc
}
return nil
})
if err != nil {
return nil, err
}
scenes := make([]scene, 0, len(scenesByName))
for _, sc := range scenesByName {
scenes = append(scenes, sc)
}
sort.Slice(scenes, func(i, j int) bool {
return strings.ToLower(scenes[i].name) < strings.ToLower(scenes[j].name)
})
return scenes, nil
}
func findMetaXML(root string) ([]scene, error) {
var scenes []scene
err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() {
return nil
}
if strings.HasSuffix(strings.ToLower(d.Name()), ".meta.xml") {
scenes = append(scenes, scene{metaPath: path, name: sceneName(path)})
}
return nil
})
return scenes, err
}
func uniqueScenes(scenes []scene) []scene {
seen := make(map[string]bool, len(scenes))
out := make([]scene, 0, len(scenes))
for _, sc := range scenes {
key := filepath.Clean(sc.metaPath)
if seen[key] {
continue
}
seen[key] = true
out = append(out, sc)
}
return out
}
func runScene(cfg config, savPath string, sc scene, sceneOut string) error {
fmt.Printf("Processing %s\n", sc.metaPath)
fmt.Printf("Output %s\n", sceneOut)
cmd := exec.Command(cfg.idlrt, savPath, "-args", sc.metaPath, sceneOut, cfg.dem, cfg.polarizations)
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
cmd.Dir = sceneOut
if err := cmd.Start(); err != nil {
return err
}
done := make(chan error, 1)
go func() {
done <- cmd.Wait()
}()
ticker := time.NewTicker(2 * time.Second)
defer ticker.Stop()
for {
select {
case err := <-done:
fmt.Printf("\r%-100s\r", "")
if err != nil {
return fmt.Errorf("scene failed %s: %w", sc.metaPath, err)
}
fmt.Println("Completed", sc.name)
return nil
case <-ticker.C:
fmt.Printf("\r%s", progressLine(sceneOut, sc.name, cfg.polarizations))
}
}
}
func progressLine(sceneOut, sceneNameValue, polarizations string) string {
stage, done, total := progressStage(sceneOut, sceneNameValue, polarizations)
width := 28
filled := 0
if total > 0 {
filled = done * width / total
}
if filled > width {
filled = width
}
bar := strings.Repeat("#", filled) + strings.Repeat("-", width-filled)
return fmt.Sprintf("[%s] %2d/%2d %s", bar, done, total, stage)
}
func progressStage(sceneOut, sceneNameValue, polarizations string) (string, int, int) {
pols := requestedPolarizations(polarizations)
checks := make([]struct {
label string
path string
}, 0, len(pols)*4)
for _, pol := range pols {
checks = append(checks, struct {
label string
path string
}{"import " + pol, filepath.Join(sceneOut, "*_"+pol+"_slc.sml")})
}
for _, pol := range pols {
lower := strings.ToLower(pol)
prefix := sceneNameValue + "_" + lower
checks = append(checks,
struct {
label string
path string
}{"multilook " + pol, filepath.Join(sceneOut, prefix+"_ml.sml")},
struct {
label string
path string
}{"filter " + pol, filepath.Join(sceneOut, prefix+"_filt.sml")},
struct {
label string
path string
}{"geocode " + pol, filepath.Join(sceneOut, prefix+"_geo.sml")},
)
}
done := 0
for _, check := range checks {
if matched(check.path) {
done++
continue
}
return check.label, done, len(checks)
}
return "finishing", done, len(checks)
}
func completedSceneOutputs(sceneOut, sceneNameValue, polarizations string) ([]completedProduct, bool) {
info, err := os.Stat(sceneOut)
if err != nil || !info.IsDir() {
return nil, false
}
pols := requestedPolarizations(polarizations)
products := make([]completedProduct, 0, len(pols))
for _, pol := range pols {
path, ok := completedGeoProduct(sceneOut, sceneNameValue, pol)
if !ok {
return nil, false
}
products = append(products, completedProduct{polarization: pol, path: path})
}
return products, true
}
func completedGeoProduct(sceneOut, sceneNameValue, polarization string) (string, bool) {
lower := strings.ToLower(polarization)
exact := filepath.Join(sceneOut, sceneNameValue+"_"+lower+"_geo.sml")
if completeGeoSML(exact) {
return exact, true
}
matches, err := filepath.Glob(filepath.Join(sceneOut, "*_"+lower+"_geo.sml"))
if err != nil {
return "", false
}
sort.Strings(matches)
for _, match := range matches {
if completeGeoSML(match) {
return match, true
}
}
return "", false
}
func completeGeoSML(path string) bool {
info, err := os.Stat(path)
if err != nil || info.IsDir() || info.Size() == 0 {
return false
}
dataPath := strings.TrimSuffix(path, ".sml")
dataInfo, err := os.Stat(dataPath)
return err == nil && !dataInfo.IsDir() && dataInfo.Size() > 0
}
func requestedPolarizations(value string) []string {
parts := strings.Split(value, ",")
out := make([]string, 0, len(parts))
for _, part := range parts {
pol := strings.ToUpper(strings.TrimSpace(part))
if pol == "" {
continue
}
if pol == "ALL" {
return []string{"HH", "HV"}
}
out = append(out, pol)
}
if len(out) == 0 {
return []string{"HH", "HV"}
}
return out
}
func matched(pattern string) bool {
matches, err := filepath.Glob(pattern)
if err != nil {
return false
}
return len(matches) > 0
}
func extractTarGz(src, dst string) error {
dst = filepath.Clean(dst)
tmpDst := dst + ".tmp"
if err := os.RemoveAll(tmpDst); err != nil {
return fmt.Errorf("remove stale temporary extraction directory %s: %w", tmpDst, err)
}
if err := os.MkdirAll(tmpDst, 0o755); err != nil {
return err
}
if err := extractTarGzToDir(src, tmpDst); err != nil {
_ = os.RemoveAll(tmpDst)
return err
}
if err := os.RemoveAll(dst); err != nil {
_ = os.RemoveAll(tmpDst)
return fmt.Errorf("replace extraction directory %s: %w", dst, err)
}
if err := os.Rename(tmpDst, dst); err != nil {
_ = os.RemoveAll(tmpDst)
return fmt.Errorf("move completed extraction into %s: %w", dst, err)
}
return nil
}
func extractTarGzToDir(src, dst string) error {
f, err := os.Open(src)
if err != nil {
return err
}
defer f.Close()
gz, err := gzip.NewReader(f)
if err != nil {
return archiveReadError(src, "", err)
}
defer gz.Close()
tr := tar.NewReader(gz)
var currentEntry string
for {
hdr, err := tr.Next()
if errors.Is(err, io.EOF) {
break
}
if err != nil {
return archiveReadError(src, currentEntry, err)
}
currentEntry = hdr.Name
target := filepath.Join(dst, filepath.Clean(hdr.Name))
if !strings.HasPrefix(target, filepath.Clean(dst)+string(os.PathSeparator)) && filepath.Clean(target) != filepath.Clean(dst) {
return fmt.Errorf("archive entry escapes destination: %s", hdr.Name)
}
switch hdr.Typeflag {
case tar.TypeDir:
if err := os.MkdirAll(target, 0o755); err != nil {
return err
}
case tar.TypeReg:
if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
return err
}
out, err := os.OpenFile(target, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o644)
if err != nil {
return err
}
_, copyErr := io.Copy(out, tr)
closeErr := out.Close()
if copyErr != nil {
return archiveReadError(src, hdr.Name, copyErr)
}
if closeErr != nil {
return closeErr
}
}
}
return nil
}
func archiveReadError(src, entry string, err error) error {
detail := src
if info, statErr := os.Stat(src); statErr == nil {
detail = fmt.Sprintf("%s (%d bytes)", src, info.Size())
}
if entry != "" {
detail = fmt.Sprintf("%s while reading %s", detail, entry)
}
if errors.Is(err, io.ErrUnexpectedEOF) || strings.Contains(strings.ToLower(err.Error()), "unexpected eof") {
return fmt.Errorf("archive appears truncated or incomplete: %s: %w; re-copy or re-download the .tar.gz", detail, err)
}
return fmt.Errorf("read archive %s: %w", detail, err)
}
func sceneName(metaPath string) string {
base := filepath.Base(metaPath)
base = strings.TrimSuffix(base, ".meta.xml")
base = strings.TrimSuffix(base, ".META.XML")
return safeName(base)
}
func safeName(s string) string {
replacer := strings.NewReplacer(" ", "_", ":", "_", "/", "_", "\\", "_")
return replacer.Replace(s)
}
func trimArchiveExt(name string) string {
lower := strings.ToLower(name)
switch {
case strings.HasSuffix(lower, ".tar.gz"):
return name[:len(name)-7]
case strings.HasSuffix(lower, ".tgz"):
return name[:len(name)-4]
default:
return strings.TrimSuffix(name, filepath.Ext(name))
}
}
func defaultIDLRT() string {
if v := os.Getenv("IDLRT_PATH"); v != "" {
return v
}
return `C:\Program Files\Harris\ENVI56\IDL88\bin\bin.x86_64\idlrt.exe`
}
@@ -0,0 +1,61 @@
package main
import (
"os"
"path/filepath"
"testing"
)
func TestCompletedSceneOutputsRequiresGeoSMLAndDataForAllPolarizations(t *testing.T) {
dir := t.TempDir()
sceneName := "GF3_TEST_SCENE"
writeFile(t, filepath.Join(dir, sceneName+"_hh_geo.sml"), "hh sml")
writeFile(t, filepath.Join(dir, sceneName+"_hh_geo"), "hh data")
writeFile(t, filepath.Join(dir, sceneName+"_hv_geo.sml"), "hv sml")
writeFile(t, filepath.Join(dir, sceneName+"_hv_geo"), "hv data")
products, ok := completedSceneOutputs(dir, sceneName, "HH,HV")
if !ok {
t.Fatal("expected completed scene outputs")
}
if len(products) != 2 {
t.Fatalf("expected 2 completed products, got %d", len(products))
}
}
func TestCompletedSceneOutputsRejectsMissingDataFile(t *testing.T) {
dir := t.TempDir()
sceneName := "GF3_TEST_SCENE"
writeFile(t, filepath.Join(dir, sceneName+"_hh_geo.sml"), "hh sml")
writeFile(t, filepath.Join(dir, sceneName+"_hv_geo.sml"), "hv sml")
writeFile(t, filepath.Join(dir, sceneName+"_hv_geo"), "hv data")
if _, ok := completedSceneOutputs(dir, sceneName, "HH,HV"); ok {
t.Fatal("expected incomplete scene when one geocoded data file is missing")
}
}
func TestCompletedSceneOutputsRequiresRequestedPolarizations(t *testing.T) {
dir := t.TempDir()
sceneName := "GF3_TEST_SCENE"
writeFile(t, filepath.Join(dir, sceneName+"_hh_geo.sml"), "hh sml")
writeFile(t, filepath.Join(dir, sceneName+"_hh_geo"), "hh data")
if _, ok := completedSceneOutputs(dir, sceneName, "HH,HV"); ok {
t.Fatal("expected incomplete scene when HV output is missing")
}
if _, ok := completedSceneOutputs(dir, sceneName, "HH"); !ok {
t.Fatal("expected completed scene for HH-only request")
}
}
func writeFile(t *testing.T, path, value string) {
t.Helper()
if err := os.WriteFile(path, []byte(value), 0o644); err != nil {
t.Fatal(err)
}
}