Compare commits
11
Commits
838ae9caa1
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
72747071d0 | ||
|
|
7740f341a7 | ||
|
|
19ba4d2a65 | ||
|
|
42b15d9cf0 | ||
|
|
d4ae7dbffb | ||
|
|
28d8bab6b9 | ||
|
|
363408f8d2 | ||
|
|
258a4c6329 | ||
|
|
d9d80d7f17 | ||
|
|
3d2a34a4b3 | ||
|
|
753a84dc59 |
@@ -0,0 +1,30 @@
|
||||
name: CI
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
|
||||
jobs:
|
||||
python:
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
tool: [sar-gf3-l1a2l2, s1-orbit-downloader]
|
||||
defaults:
|
||||
run:
|
||||
working-directory: python/${{ matrix.tool }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
- name: 编译检查
|
||||
run: python -m compileall -q src
|
||||
- name: Ruff(仅报错类规则)
|
||||
run: pip install ruff && ruff check --select E9,F63,F7,F82 src
|
||||
- name: 安装
|
||||
run: pip install .
|
||||
- name: 测试
|
||||
run: |
|
||||
if [ -d tests ]; then pip install pytest && pytest -q; else echo "无测试"; fi
|
||||
@@ -0,0 +1,5 @@
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
.venv/
|
||||
*.egg-info/
|
||||
/build/
|
||||
@@ -0,0 +1,57 @@
|
||||
# Harmon 代码协定
|
||||
|
||||
适用于 git.rsharmon.cn/Harmon 下的所有仓库(GitHub HarmonRS 为镜像)。新代码必须遵守,旧代码在修改时逐步对齐。
|
||||
|
||||
## 1. 仓库命名
|
||||
- 全小写,短横线连接:`领域-数据源-功能`,如 `sar-lt1-dinsar`、`sar-gf3-l1a2l2`、`s1-orbit-downloader`。
|
||||
- 常用前缀:`sar`(雷达通用)、`lt1`(陆探1号)、`gf3`(高分3号)、`s1`(Sentinel-1)、`opt`(光学)。
|
||||
- 禁止下划线、大写、`-v2`/`-new`/`-final` 之类的版本后缀(版本靠 tag)。
|
||||
- 主力项目(系统/服务)独立成仓库;小工具、脚本统一放进 `rs-toolkit`。
|
||||
|
||||
## 2. rs-toolkit 目录结构
|
||||
```
|
||||
rs-toolkit/
|
||||
idl/<工具名>/ # IDL/ENVI 脚本
|
||||
python/<工具名>/ # 每个工具是独立的 Python 包
|
||||
go/<工具名>/ # 如有
|
||||
README.md # 工具索引:一行说明 + 安装/运行方式
|
||||
CONVENTIONS.md
|
||||
```
|
||||
- 工具目录名同样遵守第 1 条命名规则。
|
||||
- 工具之间不互相 import;共用代码再考虑抽成 `python/rs-common/`。
|
||||
- 新增工具时同步更新根 README 索引。
|
||||
|
||||
## 3. 分支与提交
|
||||
- 默认分支 `main`,始终保持可用;功能在 `feat/xxx`、`fix/xxx` 分支上开发,通过 PR 合并。
|
||||
- 提交信息采用 Conventional Commits:`<type>(<scope>): <说明>`
|
||||
- type:`feat` `fix` `refactor` `docs` `test` `chore` `ci` `perf`
|
||||
- scope:rs-toolkit 中写工具名,如 `fix(sar-gf3-l1a2l2): 跳过损坏的压缩包`
|
||||
- 一个提交只做一件事;不提交生成物、数据、密钥。
|
||||
- 发布打 tag:主项目 `vX.Y.Z`;toolkit 内工具 `<工具名>-vX.Y.Z`。
|
||||
|
||||
## 4. Python
|
||||
- 一律 `pyproject.toml` + `src/` 布局:`src/<包名>/`,包名为目录名的下划线形式。
|
||||
- 声明 `requires-python`、依赖下限,命令行入口写在 `[project.scripts]`。
|
||||
- 安装方式(无需拷脚本):
|
||||
`pip install "git+https://git.rsharmon.cn/Harmon/rs-toolkit.git#subdirectory=python/<工具名>"`
|
||||
- 代码用 `ruff` 检查与格式化;测试放 `tests/`,用 `pytest`。
|
||||
- 路径、账号等配置不写死在代码里,走命令行参数/配置文件/环境变量。
|
||||
|
||||
## 5. IDL / ENVI
|
||||
- 文件名小写下划线(IDL 惯例),一个 `.pro` 一个主过程,过程名与文件名一致。
|
||||
- 文件头注释写明:用途、输入输出、依赖的 ENVI/SARscape 版本、作者与日期。
|
||||
- 输入输出路径作为参数传入,不写死;批处理入口统一命名 `batch_*.pro`。
|
||||
- `.sav` 等编译产物如必须入库,标记为 binary(.gitattributes)。
|
||||
|
||||
## 6. README 要求
|
||||
每个仓库/工具目录必须有 README,至少包含:
|
||||
1. 一句话说明做什么;2. 环境与依赖;3. 安装;4. 最小运行示例;5. 输入/输出说明;6. 已知限制。
|
||||
|
||||
## 7. CI
|
||||
- 每个仓库在 `.gitea/workflows/` 下配置 CI,PR 和推送 main 时触发。
|
||||
- Python:ruff + 编译检查 + pytest(有测试时);Rust:`cargo fmt --check`、`cargo clippy -D warnings`、`cargo test`;Go:`go vet`、`go test`。
|
||||
- `main` 开分支保护:CI 通过后才能合并。AI 生成的代码同样必须过 CI。
|
||||
|
||||
## 8. 其他
|
||||
- 文本统一 UTF-8、LF(Windows `.cmd/.bat` 除外)。
|
||||
- 大于 10MB 的二进制/数据不入库,放 release 或 NAS。
|
||||
@@ -0,0 +1,20 @@
|
||||
# rs-toolkit
|
||||
|
||||
遥感(SAR/InSAR)常用小工具与脚本合集。规范见 [CONVENTIONS.md](CONVENTIONS.md)。
|
||||
|
||||
| 工具 | 语言 | 说明 | 原仓库 |
|
||||
|---|---|---|---|
|
||||
| [idl/sar-lt1-dinsar](idl/sar-lt1-dinsar) | IDL | 陆探1号 ENVI/SARscape D-InSAR 批处理流水线 | lutan1-envi-idl-dinsar-pipeline |
|
||||
| [python/sar-gf3-l1a2l2](python/sar-gf3-l1a2l2) | Python | 高分3号 L1A → L2 处理流水线(含 SARscape Go 封装) | GF3_L1A_To_L2_pipeline |
|
||||
| [python/s1-orbit-downloader](python/s1-orbit-downloader) | Python | Sentinel-1 影像与精密轨道查询下载(CLI + Web UI) | sentinel-orbit-downloader |
|
||||
|
||||
## 安装 Python 工具
|
||||
|
||||
```bash
|
||||
pip install "git+https://git.rsharmon.cn/Harmon/rs-toolkit.git#subdirectory=python/sar-gf3-l1a2l2"
|
||||
pip install "git+https://git.rsharmon.cn/Harmon/rs-toolkit.git#subdirectory=python/s1-orbit-downloader"
|
||||
```
|
||||
|
||||
IDL 工具直接在 ENVI/IDL 中打开对应目录下的 `.pro`,用法见各目录 README。
|
||||
|
||||
各工具的提交历史均已保留(`git log -- <目录>`)。
|
||||
@@ -0,0 +1,373 @@
|
||||
;+
|
||||
; PURPOSE:
|
||||
; This script automates a full D-InSAR workflow, from interferogram
|
||||
; generation to geocoding (Steps 1-6).
|
||||
;
|
||||
; This is the final, complete, and integrated version resulting from
|
||||
; our collaborative development and debugging.
|
||||
;-
|
||||
|
||||
;================================================================================
|
||||
; ### HELPER FUNCTION: READ SML PARAMETER ###
|
||||
;================================================================================
|
||||
FUNCTION Read_SML_Parameter, sml_file, param_name_to_find
|
||||
IF N_ELEMENTS(sml_file) EQ 0 OR sml_file EQ '' THEN BEGIN
|
||||
PRINT, 'Error: An empty or undefined filename was passed to Read_SML_Parameter.'
|
||||
RETURN, ''
|
||||
ENDIF
|
||||
IF NOT FILE_TEST(sml_file) THEN BEGIN
|
||||
PRINT, 'Error: SML file not found -> ' + sml_file
|
||||
RETURN, ''
|
||||
ENDIF
|
||||
lun = -1
|
||||
ON_ERROR, 2
|
||||
OPENR, lun, sml_file, /GET_LUN, ERROR=err
|
||||
IF (err NE 0) THEN BEGIN
|
||||
PRINT, 'Error: Could not open SML file for reading -> ' + sml_file
|
||||
IF lun NE -1 THEN FREE_LUN, lun
|
||||
RETURN, ''
|
||||
ENDIF
|
||||
found_value = ''
|
||||
line = ''
|
||||
start_tag = '<' + STRUPCASE(param_name_to_find) + '>'
|
||||
end_tag = '</' + STRUPCASE(param_name_to_find) + '>'
|
||||
WHILE NOT EOF(lun) DO BEGIN
|
||||
READF, lun, line
|
||||
clean_line = STRTRIM(line, 2)
|
||||
line_upper = STRUPCASE(clean_line)
|
||||
start_pos = STRPOS(line_upper, start_tag)
|
||||
IF start_pos NE -1 THEN BEGIN
|
||||
end_pos = STRPOS(line_upper, end_tag, start_pos)
|
||||
IF end_pos NE -1 THEN BEGIN
|
||||
start_extract = start_pos + STRLEN(start_tag)
|
||||
value_length = end_pos - start_extract
|
||||
found_value = STRMID(clean_line, start_extract, value_length)
|
||||
BREAK
|
||||
ENDIF
|
||||
ENDIF
|
||||
ENDWHILE
|
||||
IF lun NE -1 THEN FREE_LUN, lun
|
||||
RETURN, found_value
|
||||
END
|
||||
|
||||
;================================================================================
|
||||
; ### HELPER FUNCTION: CREATE GCPs FROM COHERENCE ###
|
||||
;================================================================================
|
||||
FUNCTION CREATE_GCPS_FROM_COHERENCE, coherence_file, output_shp_file, COH_THRESHOLD=coh_threshold, NUM_POINTS=num_points
|
||||
|
||||
ON_ERROR, 2
|
||||
IF FILE_TEST(output_shp_file) THEN FILE_DELETE, output_shp_file, /QUIET
|
||||
|
||||
IF N_ELEMENTS(coh_threshold) EQ 0 THEN coh_threshold = 0.7
|
||||
IF N_ELEMENTS(num_points) EQ 0 THEN num_points = 100
|
||||
|
||||
e = ENVI(/HEADLESS)
|
||||
|
||||
PRINT, 'Opening coherence file: ' + coherence_file
|
||||
oRaster = e.OpenRaster(coherence_file)
|
||||
|
||||
ns = oRaster.NCOLUMNS
|
||||
nl = oRaster.NROWS
|
||||
data = oRaster.GetData(BANDS=[0])
|
||||
|
||||
PRINT, 'Searching for GCPs with coherence > ' + STRTRIM(coh_threshold, 2)
|
||||
|
||||
grid_dim = CEIL(SQRT(num_points))
|
||||
x_step = FLOOR(ns / grid_dim)
|
||||
y_step = FLOOR(nl / grid_dim)
|
||||
|
||||
pixel_points_x = [] & pixel_points_y = []
|
||||
|
||||
FOR j=0L, grid_dim-1 DO BEGIN
|
||||
FOR i=0L, grid_dim-1 DO BEGIN
|
||||
x_start = i * x_step & y_start = j * y_step
|
||||
x_end = (i+1) * x_step -1 < (ns-1)
|
||||
y_end = (j+1) * y_step -1 < (nl-1)
|
||||
IF (x_start GE ns) OR (y_start GE nl) THEN CONTINUE
|
||||
cell_data = data[x_start:x_end, y_start:y_end]
|
||||
max_val = MAX(cell_data, max_idx)
|
||||
IF max_val GE coh_threshold THEN BEGIN
|
||||
max_y_cell = max_idx / (x_end - x_start + 1)
|
||||
max_x_cell = max_idx MOD (x_end - x_start + 1)
|
||||
pixel_points_x = [pixel_points_x, x_start + max_x_cell]
|
||||
pixel_points_y = [pixel_points_y, y_start + max_y_cell]
|
||||
ENDIF
|
||||
ENDFOR
|
||||
ENDFOR
|
||||
|
||||
point_count = N_ELEMENTS(pixel_points_x)
|
||||
IF point_count EQ 0 THEN BEGIN
|
||||
PRINT, 'Error: No GCPs found.' & OBJ_DESTROY, oRaster & RETURN, 0
|
||||
ENDIF
|
||||
|
||||
PRINT, 'Found ' + STRTRIM(point_count, 2) + ' suitable GCPs.'
|
||||
PRINT, 'Creating shapefile using final authoritative methodology...'
|
||||
|
||||
oShape = OBJ_NEW('IDLffShape', output_shp_file, ENTITY_TYPE=1, /UPDATE)
|
||||
|
||||
oShape->AddAttribute, 'SHP_ID', 3, 10
|
||||
oShape->AddAttribute, 'GCP_LABEL', 7, 20
|
||||
oShape->AddAttribute, 'GCP_TYPE', 7, 20
|
||||
oShape->AddAttribute, 'GCP_COLUMN', 5, 24, PRECISION=6
|
||||
oShape->AddAttribute, 'GCP_ROW', 5, 24, PRECISION=6
|
||||
oShape->AddAttribute, 'GCP_OTHER_', 7, 10 ; User-verified field name
|
||||
|
||||
attr_template = oShape->GetAttributes(/ATTRIBUTE_STRUCTURE)
|
||||
|
||||
entNew = {IDL_SHAPE_ENTITY}
|
||||
entNew.SHAPE_TYPE = 1
|
||||
|
||||
FOR i=0, point_count-1 DO BEGIN
|
||||
x_float = FLOAT(pixel_points_x[i])
|
||||
y_float = FLOAT(pixel_points_y[i])
|
||||
|
||||
entNew.ISHAPE = i
|
||||
entNew.BOUNDS = [x_float, y_float, 0.0, 0.0, x_float, y_float, 0.0, 0.0]
|
||||
oShape->PutEntity, entNew
|
||||
|
||||
attr = attr_template
|
||||
attr.ATTRIBUTE_0 = i
|
||||
attr.ATTRIBUTE_1 = 'GCP_' + STRTRIM(i+1, 2)
|
||||
attr.ATTRIBUTE_2 = 'undefined'
|
||||
attr.ATTRIBUTE_3 = x_float
|
||||
attr.ATTRIBUTE_4 = y_float
|
||||
attr.ATTRIBUTE_5 = ''
|
||||
oShape->SetAttributes, i, attr
|
||||
ENDFOR
|
||||
|
||||
oShape->Close
|
||||
OBJ_DESTROY, [oShape, oRaster]
|
||||
|
||||
PRINT, 'Successfully created GCP shapefile: ' + output_shp_file
|
||||
RETURN, 1
|
||||
END
|
||||
|
||||
;================================================================================
|
||||
; ### MAIN WORKFLOW PROCEDURE ###
|
||||
;================================================================================
|
||||
PRO DinsarWorkflow_ALL
|
||||
|
||||
; --- Initialize SARscape Environment ---
|
||||
temporary_directory = 'D:\Sarscape_IDL_Area'
|
||||
SARscape_Batch_Init,Temp_Directory=temporary_directory
|
||||
e = ENVI(/HEADLESS)
|
||||
PRINT, 'ENVI 和 SARscape 环境初始化成功!'
|
||||
PRINT, ''
|
||||
|
||||
; ===================================================================
|
||||
; ### GLOBAL CONFIGURATION ###
|
||||
; ===================================================================
|
||||
|
||||
master_base_file = 'G:/Code/InSAR/IDLTools/Task_20250504_20250601_1/LuTan-1_LT1A_39_20250504_091105615_STRIP1_A_HH_slc'
|
||||
slave_base_file = 'G:/Code/InSAR/IDLTools/Task_20250504_20250601_1/LuTan-1_LT1B_47_20250601_091101714_STRIP1_A_HH_slc'
|
||||
dem_base_file = 'D:/SRTM30m/SRTMDEM_RSP_SARscape'
|
||||
workflow_root_name = 'G:/Code/InSAR/IDLTools/test/test' ; Use a consistent root name
|
||||
|
||||
target_ground_resolution_m = 10.0
|
||||
filter_method = 'GOLDSTEIN'
|
||||
unwrapping_coh_threshold = 0.05
|
||||
gcp_coh_threshold = 0.7
|
||||
gcp_number = 100
|
||||
geocoding_coh_threshold = 0.0
|
||||
geocoding_pixel_size_m = 10.0
|
||||
|
||||
; ===================================================================
|
||||
; ### STEP 1: INTERFEROGRAM GENERATION ###
|
||||
; ===================================================================
|
||||
|
||||
print, '======================================================'
|
||||
print, '### STEP 1: INTERFEROGRAM GENERATION ###'
|
||||
print, '======================================================'
|
||||
|
||||
master_sml = master_base_file + '.sml' & slave_sml = slave_base_file + '.sml'
|
||||
m_rg_sp_str = Read_SML_Parameter(master_sml, 'PixelSpacingRg')
|
||||
m_az_sp_str = Read_SML_Parameter(master_sml, 'PixelSpacingAz')
|
||||
m_inc_ang_str = Read_SML_Parameter(master_sml, 'IncidenceAngle')
|
||||
s_rg_sp_str = Read_SML_Parameter(slave_sml, 'PixelSpacingRg')
|
||||
s_az_sp_str = Read_SML_Parameter(slave_sml, 'PixelSpacingAz')
|
||||
s_inc_ang_str = Read_SML_Parameter(slave_sml, 'IncidenceAngle')
|
||||
|
||||
IF m_rg_sp_str EQ '' OR s_rg_sp_str EQ '' THEN GOTO, WORKFLOW_EXIT_FAILURE
|
||||
|
||||
m_rg_sp = FLOAT(m_rg_sp_str) & m_az_sp = FLOAT(m_az_sp_str) & m_inc_ang = FLOAT(m_inc_ang_str)
|
||||
s_rg_sp = FLOAT(s_rg_sp_str) & s_az_sp = FLOAT(s_az_sp_str) & s_inc_ang = FLOAT(s_inc_ang_str)
|
||||
|
||||
avg_az_spacing = (m_az_sp + s_az_sp) / 2.0 & azimuth_looks = (long(target_ground_resolution_m / avg_az_spacing)) > 1
|
||||
m_ground_rg_sp = m_rg_sp / SIN(m_inc_ang * !DPI / 180.0) & s_ground_rg_sp = s_rg_sp / SIN(s_inc_ang * !DPI / 180.0)
|
||||
avg_ground_rg_sp = (m_ground_rg_sp + s_ground_rg_sp) / 2.0 & range_looks = (long(target_ground_resolution_m / avg_ground_rg_sp)) > 1
|
||||
|
||||
task1_obj = obj_new('SARscapeBatch', Module='InSARInterferogramGeneration')
|
||||
prefix_1 = 'MAIN_INSAR_INTERFEROGRAM_GENERATION_CMD.'
|
||||
task1_obj->SetParam, prefix_1 + 'INPUT_REFERENCE_FILE_NAME' , master_base_file
|
||||
task1_obj->SetParam, prefix_1 + 'INPUT_SECONDARY_FILE_NAME' , slave_base_file
|
||||
task1_obj->SetParam, prefix_1 + 'DEM_FILE_NAME' , dem_base_file
|
||||
task1_obj->SetParam, prefix_1 + 'OUTPUT_ROOT_FILE_NAME' , workflow_root_name
|
||||
task1_obj->SetParam, prefix_1 + 'RG_LOOKS_NBR' , string(range_looks)
|
||||
task1_obj->SetParam, prefix_1 + 'AZ_LOOKS_NBR' , string(azimuth_looks)
|
||||
task1_obj->SetParam, 'COREGISTRATION_CMD.COREGISTRATION_WITH_DEM_FLAG' , 'OK'
|
||||
task1_obj->SetParam, 'FLAT_CMD.MAKE_FLATTENING_FLAG', 'OK'
|
||||
ok_step1 = task1_obj->Execute()
|
||||
obj_destroy, task1_obj
|
||||
IF NOT ok_step1 THEN GOTO, WORKFLOW_EXIT_FAILURE
|
||||
print, '### STEP 1: SUCCESS!'
|
||||
|
||||
; ===================================================================
|
||||
; ### STEP 2: FILTERING AND COHERENCE GENERATION ###
|
||||
; ===================================================================
|
||||
|
||||
print, '======================================================'
|
||||
print, '### STEP 2: FILTERING AND COHERENCE GENERATION ###'
|
||||
print, '======================================================'
|
||||
task2_obj = obj_new('SARscapeBatch', Module='InSARFilterAndCoherence')
|
||||
prefix_main2 = 'MAIN_INSAR_FILTER_COHERENCE_CMD.'
|
||||
prefix_filt2 = 'FILTERING_CMD.'
|
||||
task2_obj->SetParam, prefix_main2 + 'INPUT_INTERF_FILE_NAME', workflow_root_name + '_dint'
|
||||
task2_obj->SetParam, prefix_main2 + 'INPUT_REFERENCE_FILE_NAME', workflow_root_name + '_reference_pwr'
|
||||
task2_obj->SetParam, prefix_main2 + 'INPUT_SECONDARY_FILE_NAME', workflow_root_name + '_secondary_pwr'
|
||||
task2_obj->SetParam, prefix_main2 + 'OUT_ROOT_FILE', workflow_root_name
|
||||
task2_obj->SetParam, prefix_filt2 + 'FILTERING_METHOD', filter_method
|
||||
task2_obj->SetParam, prefix_main2 + 'COHERENCE_FLAG', 'OK'
|
||||
task2_obj->SetParam, prefix_main2 + 'INTERF_FILT_FLAG', 'OK'
|
||||
ok_step2 = task2_obj->Execute()
|
||||
obj_destroy, task2_obj
|
||||
IF NOT ok_step2 THEN GOTO, WORKFLOW_EXIT_FAILURE
|
||||
print, '### STEP 2: SUCCESS!'
|
||||
|
||||
; ===================================================================
|
||||
; ### STEP 3: ORBITAL TREND REMOVAL ###
|
||||
; ===================================================================
|
||||
print, '======================================================'
|
||||
print, '### STEP 3: ORBITAL TREND REMOVAL ###'
|
||||
print, '======================================================'
|
||||
|
||||
reflat_output_root = workflow_root_name + '_rrpf'
|
||||
task3_obj = obj_new('SARscapeBatch', Module='InSARRemoveResidualPhaseFrequency')
|
||||
prefix_3 = 'MAIN_INSAR_REMOVE_RESIDUAL_PHASE_FREQUENCY.'
|
||||
task3_obj->SetParam, prefix_3 + 'INTERF_FILE_NAME', workflow_root_name + '_fint'
|
||||
task3_obj->SetParam, prefix_3 + 'COHERENCE_FILE_NAME', workflow_root_name + '_cc'
|
||||
task3_obj->SetParam, prefix_3 + 'OUT_ROOT_FILE_NAME', reflat_output_root
|
||||
ok_step3 = task3_obj->Execute()
|
||||
obj_destroy, task3_obj
|
||||
IF NOT ok_step3 THEN GOTO, WORKFLOW_EXIT_FAILURE
|
||||
print, '### STEP 3: SUCCESS!'
|
||||
|
||||
; ===================================================================
|
||||
; ### STEP 4: PHASE UNWRAPPING ###
|
||||
; ===================================================================
|
||||
print, '======================================================'
|
||||
print, '### STEP 4: PHASE UNWRAPPING ###'
|
||||
print, '======================================================'
|
||||
|
||||
reflat_upha_output_file = workflow_root_name + '_upha'
|
||||
task4_obj = obj_new('SARscapeBatch', Module='InSARPhaseUnwrapping')
|
||||
prefix_main4 = 'MAIN_INSAR_PHASE_UNWRAPPING_CMD.'
|
||||
prefix_upha4 = 'UPHA_CMD.'
|
||||
task4_obj->SetParam, prefix_main4 + 'INFILE_NAME', reflat_output_root + '_fint'
|
||||
task4_obj->SetParam, prefix_main4 + 'COHERENCEFILE_NAME', workflow_root_name + '_cc'
|
||||
task4_obj->SetParam, prefix_upha4 + 'UPHA_COH_THRESHOLD', string(unwrapping_coh_threshold)
|
||||
task4_obj->SetParam, prefix_main4 + 'OUTFILE_NAME', reflat_upha_output_file
|
||||
ok_step4 = task4_obj->Execute()
|
||||
obj_destroy, task4_obj
|
||||
IF NOT ok_step4 THEN GOTO, WORKFLOW_EXIT_FAILURE
|
||||
print, '### STEP 4: SUCCESS!'
|
||||
|
||||
; ===================================================================
|
||||
; ### STEP 5: REFINEMENT AND REFLATTENING ###
|
||||
; ===================================================================
|
||||
|
||||
; --- STEP 5A: Automatic GCP Generation ---
|
||||
print, '======================================================'
|
||||
print, '### STEP 5A: 正在自动生成GCP... ###'
|
||||
print, '======================================================'
|
||||
auto_gcp_shp = workflow_root_name + '_auto_gcp.shp'
|
||||
gcp_success = CREATE_GCPS_FROM_COHERENCE(workflow_root_name + '_cc', auto_gcp_shp, COH_THRESHOLD=gcp_coh_threshold, NUM_POINTS=gcp_number)
|
||||
|
||||
IF gcp_success EQ 0 THEN BEGIN
|
||||
PRINT, '错误: 自动生成GCP shapefile失败。流程终止。'
|
||||
GOTO, WORKFLOW_EXIT_FAILURE
|
||||
ENDIF
|
||||
PRINT, '### STEP 5A: SUCCESS!'
|
||||
|
||||
; --- STEP 5B: Refinement and Reflattening Execution ---
|
||||
print, '======================================================'
|
||||
print, '### STEP 5B: 正在执行 REFINEMENT AND REFLATTENING ###'
|
||||
print, '======================================================'
|
||||
ref_output_root = workflow_root_name + '_reflat'
|
||||
task_refine = obj_new('SARscapeBatch', Module='InSARRefinementAndReflattening')
|
||||
prefix_refine = 'MAIN_INSAR_REFINEMENT_AND_REFLATTENING_CMD.'
|
||||
|
||||
task_refine->SetParam, prefix_refine + 'INPUT_UPHA_FILE_NAME', reflat_upha_output_file
|
||||
task_refine->SetParam, prefix_refine + 'INPUT_REFERENCE_FILE_NAME', workflow_root_name + '_reference_pwr'
|
||||
task_refine->SetParam, prefix_refine + 'INPUT_SECONDARY_FILE_NAME', workflow_root_name + '_secondary_pwr'
|
||||
task_refine->SetParam, prefix_refine + 'SLANT_RANGE_DEM_FILE_NAME', workflow_root_name + '_srdem'
|
||||
task_refine->SetParam, prefix_refine + 'SYNTHETIC_FILE_NAME', workflow_root_name + '_sint'
|
||||
task_refine->SetParam, prefix_refine + 'COHERENCE_FILE_NAME', workflow_root_name + '_cc'
|
||||
task_refine->SetParam, prefix_refine + 'OUTPUT_ROOT_NAME', ref_output_root
|
||||
task_refine->SetParam, prefix_refine + 'DEM_FILE_NAME', dem_base_file
|
||||
task_refine->SetParam, 'REFINEMENT_CMD.REFINEMENT_GCP_FILE_NAME', auto_gcp_shp
|
||||
|
||||
ok_step5 = task_refine->Execute()
|
||||
obj_destroy, task_refine
|
||||
IF NOT ok_step5 THEN GOTO, WORKFLOW_EXIT_FAILURE
|
||||
print, '### STEP 5B: SUCCESS!'
|
||||
|
||||
; ===================================================================
|
||||
; ### STEP 6: PHASE TO DISPLACEMENT AND GEOCODING ###
|
||||
; ===================================================================
|
||||
print, '======================================================'
|
||||
print, '### STEP 6: PHASE TO DISPLACEMENT AND GEOCODING ###'
|
||||
print, '======================================================'
|
||||
|
||||
final_upha_file = ref_output_root + '_upha'
|
||||
geocoded_output_root = workflow_root_name + '_geo'
|
||||
|
||||
task6_obj = obj_new('SARscapeBatch', Module='InSARPhaseToDisplacement')
|
||||
|
||||
IF (~OBJ_VALID(task6_obj)) THEN BEGIN
|
||||
PRINT, '错误: 创建 SARscapeBatch 对象失败: InSARPhaseToDisplacement'
|
||||
GOTO, WORKFLOW_EXIT_FAILURE
|
||||
ENDIF
|
||||
|
||||
; --- Set Main Parameters ---
|
||||
prefix_main6 = 'MAIN_INSAR_PHASE_TO_DISPLACEMENT_CMD.'
|
||||
task6_obj->SetParam, prefix_main6 + 'INPUT_FILE_NAME', final_upha_file
|
||||
task6_obj->SetParam, prefix_main6 + 'OUTPUT_FILE', geocoded_output_root
|
||||
task6_obj->SetParam, prefix_main6 + 'COHERENCE_FILE', workflow_root_name + '_cc'
|
||||
task6_obj->SetParam, prefix_main6 + 'DEM_FILE_NAME', dem_base_file
|
||||
task6_obj->SetParam, prefix_main6 + 'COHERENCE_THRESHOLD', STRING(geocoding_coh_threshold)
|
||||
|
||||
; --- Set Other Required Parameters ---
|
||||
task6_obj->SetParam, prefix_main6 + 'INTERPOL_TYPE' , '4th_order_cc'
|
||||
task6_obj->SetParam, prefix_main6 + 'ALLOW_SKIP_REFINEMENT' , 'NotOK'
|
||||
task6_obj->SetParam, prefix_main6 + 'SARSCAPEENVIRONMENT' , 'IDL_ENVI_ENV'
|
||||
|
||||
; --- Set Geocoding Parameters ---
|
||||
prefix_geocode6 = 'GEOCODE_CMD.'
|
||||
task6_obj->SetParam, prefix_geocode6 + 'GEOCODE_RG_GRID_SIZE', STRING(geocoding_pixel_size_m)
|
||||
task6_obj->SetParam, prefix_geocode6 + 'GEOCODE_AZ_GRID_SIZE', STRING(geocoding_pixel_size_m)
|
||||
task6_obj->SetParam, prefix_geocode6 + 'GEOCODE_INTERPOL_BOX_SIZE', '7'
|
||||
|
||||
; --- Set Displacement Product Generation Parameters ---
|
||||
prefix_disp6 = 'DISPLACEMENT_PROJECTION_CMD.'
|
||||
task6_obj->SetParam, prefix_disp6 + 'GENERATE_LOS_FLAG', 'OK'
|
||||
task6_obj->SetParam, prefix_disp6 + 'GENERATE_VERTICAL_FLAG', 'NotOK'
|
||||
task6_obj->SetParam, prefix_disp6 + 'GENERATE_MAX_SLOPE_FLAG', 'NotOK'
|
||||
|
||||
print, '正在执行地理编码任务...'
|
||||
ok_step6 = task6_obj->Execute()
|
||||
obj_destroy, task6_obj
|
||||
IF NOT ok_step6 THEN GOTO, WORKFLOW_EXIT_FAILURE
|
||||
print, '### STEP 6: SUCCESS!'
|
||||
|
||||
; --- Final Success Message ---
|
||||
print, '======================================================='
|
||||
print, ' FULL D-InSAR WORKFLOW (STEPS 1-6) COMPLETED SUCCESSFULLY!'
|
||||
print, '======================================================='
|
||||
GOTO, WORKFLOW_EXIT_NORMAL
|
||||
|
||||
WORKFLOW_EXIT_FAILURE:
|
||||
print, 'Workflow terminated due to a failure in one of the steps.'
|
||||
WORKFLOW_EXIT_NORMAL:
|
||||
SARscape_Batch_Exit
|
||||
|
||||
end
|
||||
@@ -0,0 +1,316 @@
|
||||
# ENVI + IDL D-InSAR 批处理流水线
|
||||
|
||||
本仓库是一套基于 ENVI / IDL / SARscape 的 D-InSAR 自动化处理脚本,面向陆探一号等 SAR 数据的批量导入与差分干涉处理。代码包含两个主要阶段:
|
||||
|
||||
1. 批量导入原始数据,将 `*.meta.xml` 导入为 SARscape 可继续处理的产品。
|
||||
2. 对已经完成导入的主辅影像执行完整 D-InSAR 流程,直到位移反演与地理编码输出。
|
||||
|
||||
仓库当前核心脚本只有 3 个:
|
||||
|
||||
- `Run_Batch_Task_Import.pro`:批量导入 `Task_*` 目录中的原始数据。
|
||||
- `batch_dinsarworkflow_all.pro`:批量执行 D-InSAR 流程。
|
||||
- `DinsarWorkflow_ALL.pro`:单景测试版 / 调试版完整流程。
|
||||
|
||||
## 1. 依赖环境
|
||||
|
||||
运行本项目需要以下软件环境已经正确安装并可在 IDL 中调用:
|
||||
|
||||
- ENVI
|
||||
- IDL
|
||||
- SARscape
|
||||
- 对应 SARscape Batch 模块与 ENVI Task 环境
|
||||
|
||||
脚本中直接使用了以下对象 / 任务:
|
||||
|
||||
- `ENVI(/HEADLESS)`
|
||||
- `SARscape_Batch_Init`
|
||||
- `SARscapeBatch`
|
||||
- `ENVITask('SARsImportLuTan1')`
|
||||
- `IDLffShape`
|
||||
|
||||
因此该仓库不是纯 IDL 通用脚本,必须在装有 ENVI + SARscape 的环境中运行。
|
||||
|
||||
## 2. 仓库结构
|
||||
|
||||
```text
|
||||
ENVI_IDL_DInSAR/
|
||||
├─ Run_Batch_Task_Import.pro
|
||||
├─ batch_dinsarworkflow_all.pro
|
||||
├─ DinsarWorkflow_ALL.pro
|
||||
└─ README.md
|
||||
```
|
||||
|
||||
各脚本职责如下。
|
||||
|
||||
### 2.1 `Run_Batch_Task_Import.pro`
|
||||
|
||||
用途:批量扫描根目录下的 `Task_*` 文件夹,并分别处理其中的 `master` 与 `slave` 子目录。
|
||||
|
||||
处理逻辑:
|
||||
|
||||
- 在 `master` / `slave` 目录中查找 `*.meta.xml`
|
||||
- 使用 `ENVITask('SARsImportLuTan1')` 执行导入
|
||||
- 输出直接写回当前 `master` 或 `slave` 目录
|
||||
- 如果目录中已经存在 `*.sml`,则认为该目录已导入,自动跳过
|
||||
|
||||
适合放在整个流程的第一步,用来把原始数据准备成后续 D-InSAR 可识别的输入。
|
||||
|
||||
### 2.2 `batch_dinsarworkflow_all.pro`
|
||||
|
||||
用途:批量执行完整 D-InSAR 流程。
|
||||
|
||||
处理逻辑:
|
||||
|
||||
- 扫描根目录下所有 `Task_*` 文件夹
|
||||
- 检查每个任务目录下是否存在 `master` / `slave`
|
||||
- 在 `master`、`slave` 中各取第一个 `*.sml`
|
||||
- 去掉 `.sml` 后缀,得到 SARscape 输入基名
|
||||
- 在任务目录内创建 `dinsar_results` 输出目录
|
||||
- 调用 `Execute_Single_Dinsar_Workflow` 执行完整处理
|
||||
|
||||
这是生产使用时最重要的脚本。
|
||||
|
||||
### 2.3 `DinsarWorkflow_ALL.pro`
|
||||
|
||||
用途:单组主辅影像的完整 D-InSAR 处理脚本。
|
||||
|
||||
特点:
|
||||
|
||||
- 直接在脚本内手工指定 `master_base_file`、`slave_base_file`、`dem_base_file`
|
||||
- 更适合单景调试、流程验证、参数试验
|
||||
- 逻辑上与批处理版本一致,但不是批量入口
|
||||
|
||||
## 3. 建议的数据组织方式
|
||||
|
||||
根据代码,批处理脚本假定数据目录组织如下:
|
||||
|
||||
```text
|
||||
<rootDir>/
|
||||
├─ Task_20250504_20250601_1/
|
||||
│ ├─ master/
|
||||
│ │ ├─ *.meta.xml
|
||||
│ │ ├─ ... 导入后生成的 *.sml、影像及配套文件
|
||||
│ ├─ slave/
|
||||
│ │ ├─ *.meta.xml
|
||||
│ │ ├─ ... 导入后生成的 *.sml、影像及配套文件
|
||||
│ └─ dinsar_results/
|
||||
│ └─ workflow*
|
||||
├─ Task_xxx/
|
||||
└─ ...
|
||||
```
|
||||
|
||||
处理顺序通常是:
|
||||
|
||||
1. 先运行 `Run_Batch_Task_Import.pro`
|
||||
2. 确认每个 `master` / `slave` 目录中已经生成 `*.sml`
|
||||
3. 再运行 `batch_dinsarworkflow_all.pro`
|
||||
|
||||
## 4. D-InSAR 主流程说明
|
||||
|
||||
`batch_dinsarworkflow_all.pro` 中的 `Execute_Single_Dinsar_Workflow` 实现了完整 6 步流程:
|
||||
|
||||
### Step 1. 干涉图生成
|
||||
|
||||
- 从主影像和辅影像的 `.sml` 中读取:
|
||||
- `PixelSpacingRg`
|
||||
- `PixelSpacingAz`
|
||||
- `IncidenceAngle`
|
||||
- 根据目标地面分辨率自动计算多视数:
|
||||
- `RG_LOOKS_NBR`
|
||||
- `AZ_LOOKS_NBR`
|
||||
- 调用 `InSARInterferogramGeneration`
|
||||
|
||||
### Step 2. 滤波与相干图生成
|
||||
|
||||
- 调用 `InSARFilterAndCoherence`
|
||||
- 使用设定的滤波方法,例如 `GOLDSTEIN`
|
||||
- 生成滤波干涉图与相干图
|
||||
|
||||
### Step 3. 轨道残余相位频率去除
|
||||
|
||||
- 调用 `InSARRemoveResidualPhaseFrequency`
|
||||
- 输入为上一步生成的滤波干涉图和相干图
|
||||
|
||||
### Step 4. 相位解缠
|
||||
|
||||
- 调用 `InSARPhaseUnwrapping`
|
||||
- 使用相干阈值 `UPHA_COH_THRESHOLD`
|
||||
|
||||
### Step 5. 精化与再平地
|
||||
|
||||
分为两步:
|
||||
|
||||
1. 自动 GCP 生成
|
||||
2. Refinement and Reflattening
|
||||
|
||||
自动 GCP 生成的逻辑为:
|
||||
|
||||
- 打开相干图
|
||||
- 按网格均匀划分影像
|
||||
- 在每个网格中选取相干系数最大的像元
|
||||
- 当该像元相干值高于阈值时,将其写入 shapefile
|
||||
- 最终生成 `*_auto_gcp.shp`
|
||||
|
||||
随后调用 `InSARRefinementAndReflattening`,并将自动生成的 GCP 文件作为输入。
|
||||
|
||||
### Step 6. 相位转位移与地理编码
|
||||
|
||||
- 调用 `InSARPhaseToDisplacement`
|
||||
- 使用最终解缠结果生成 LOS 位移产品
|
||||
- 执行地理编码
|
||||
- 当前脚本仅开启 LOS 输出:
|
||||
- `GENERATE_LOS_FLAG = OK`
|
||||
- `GENERATE_VERTICAL_FLAG = NotOK`
|
||||
- `GENERATE_MAX_SLOPE_FLAG = NotOK`
|
||||
|
||||
## 5. 关键输入参数
|
||||
|
||||
以下参数需要在脚本中按实际工程修改。
|
||||
|
||||
### 5.1 导入脚本参数
|
||||
|
||||
文件:`Run_Batch_Task_Import.pro`
|
||||
|
||||
- `temporary_directory`:SARscape 临时目录
|
||||
- `rootDir`:待扫描的任务根目录
|
||||
- `numToProcess`:最多处理多少个任务,`0` 表示全部处理
|
||||
|
||||
### 5.2 批处理 D-InSAR 参数
|
||||
|
||||
文件:`batch_dinsarworkflow_all.pro`
|
||||
|
||||
- `temporary_directory`:SARscape 临时目录
|
||||
- `rootDir`:包含多个 `Task_*` 目录的根目录
|
||||
- `dem_base_file`:DEM 基础文件路径
|
||||
- `target_ground_resolution_m`:目标地面分辨率,控制自动多视
|
||||
- `filter_method`:滤波方法,默认 `GOLDSTEIN`
|
||||
- `unwrapping_coh_threshold`:解缠相干阈值
|
||||
- `gcp_coh_threshold`:自动 GCP 相干阈值
|
||||
- `gcp_number`:目标 GCP 数量
|
||||
- `geocoding_coh_threshold`:地理编码时使用的相干阈值
|
||||
- `geocoding_pixel_size_m`:地理编码输出像元大小
|
||||
- `numToProcess`:最多处理多少个任务,`0` 表示全部处理
|
||||
|
||||
### 5.3 单景调试参数
|
||||
|
||||
文件:`DinsarWorkflow_ALL.pro`
|
||||
|
||||
- `master_base_file`
|
||||
- `slave_base_file`
|
||||
- `dem_base_file`
|
||||
- `workflow_root_name`
|
||||
- 其余处理参数与批处理版本基本一致
|
||||
|
||||
注意:`master_base_file` 和 `slave_base_file` 填写的是“基名”,不是 `.sml` 文件全名。脚本内部会自动拼接 `.sml` 去读取参数。
|
||||
|
||||
## 6. 输出结果说明
|
||||
|
||||
批处理脚本默认在每个任务目录下创建:
|
||||
|
||||
```text
|
||||
Task_xxx/
|
||||
└─ dinsar_results/
|
||||
├─ workflow_dint
|
||||
├─ workflow_reference_pwr
|
||||
├─ workflow_secondary_pwr
|
||||
├─ workflow_fint
|
||||
├─ workflow_cc
|
||||
├─ workflow_rrpf_fint
|
||||
├─ workflow_upha
|
||||
├─ workflow_auto_gcp.shp
|
||||
├─ workflow_reflat_upha
|
||||
├─ workflow_geo*
|
||||
└─ ...
|
||||
```
|
||||
|
||||
其中常见关键中间结果包括:
|
||||
|
||||
- `*_dint`:初始干涉图
|
||||
- `*_fint`:滤波后干涉图
|
||||
- `*_cc`:相干图
|
||||
- `*_upha`:解缠结果
|
||||
- `*_auto_gcp.shp`:自动生成的 GCP 矢量文件
|
||||
- `*_geo*`:地理编码后的位移产品
|
||||
|
||||
实际生成的附属文件数量和后缀会受 SARscape 版本及模块设置影响。
|
||||
|
||||
## 7. 典型使用方式
|
||||
|
||||
### 7.1 第一步:批量导入
|
||||
|
||||
在 `Run_Batch_Task_Import.pro` 中修改:
|
||||
|
||||
- `temporary_directory`
|
||||
- `rootDir`
|
||||
- `numToProcess`
|
||||
|
||||
然后在 IDL 命令行或脚本环境中运行:
|
||||
|
||||
```idl
|
||||
.compile Run_Batch_Task_Import.pro
|
||||
Run_Batch_Task_Import
|
||||
```
|
||||
|
||||
### 7.2 第二步:批量执行 D-InSAR
|
||||
|
||||
在 `batch_dinsarworkflow_all.pro` 中修改:
|
||||
|
||||
- `temporary_directory`
|
||||
- `rootDir`
|
||||
- `dem_base_file`
|
||||
- 各处理参数
|
||||
|
||||
然后运行:
|
||||
|
||||
```idl
|
||||
.compile batch_dinsarworkflow_all.pro
|
||||
Batch_DinsarWorkflow_ALL
|
||||
```
|
||||
|
||||
### 7.3 单景测试
|
||||
|
||||
在 `DinsarWorkflow_ALL.pro` 中直接修改主影像、辅影像、DEM 和输出路径,然后运行:
|
||||
|
||||
```idl
|
||||
.compile DinsarWorkflow_ALL.pro
|
||||
DinsarWorkflow_ALL
|
||||
```
|
||||
|
||||
## 8. 代码实现要点
|
||||
|
||||
### 8.1 自动读取 SML 参数
|
||||
|
||||
`Read_SML_Parameter` 会直接从 `.sml` 文件中读取指定 XML 标签内容,用于自动计算多视参数。这意味着:
|
||||
|
||||
- 输入数据必须已经正确导入并生成 `.sml`
|
||||
- `.sml` 内部需要包含 `PixelSpacingRg`、`PixelSpacingAz`、`IncidenceAngle` 等字段
|
||||
|
||||
### 8.2 自动 GCP 生成策略
|
||||
|
||||
`CREATE_GCPS_FROM_COHERENCE` 不是随机取点,而是“规则网格 + 每格取最大相干值”的策略。这样做的优点是:
|
||||
|
||||
- GCP 空间分布更均匀
|
||||
- 能避免点位过度集中
|
||||
- 能通过相干阈值控制稳定区域
|
||||
|
||||
### 8.3 批处理脚本的选择策略
|
||||
|
||||
当前批处理版本在每个 `master` / `slave` 目录中只取“第一个找到的 `.sml`”作为输入,因此建议:
|
||||
|
||||
- 每个 `master` 目录内只保留一组待处理主影像
|
||||
- 每个 `slave` 目录内只保留一组待处理辅影像
|
||||
- 若目录中存在多份 `.sml`,需要先确认 `FILE_SEARCH` 返回顺序是否符合预期
|
||||
|
||||
## 9. 注意事项
|
||||
|
||||
1. 路径需要按本机环境手动修改,仓库中的路径示例均为作者本地路径。
|
||||
2. 脚本默认使用 `ENVI(/HEADLESS)`,适合批处理环境。
|
||||
3. `temporary_directory` 需要保证有足够磁盘空间。
|
||||
4. `gcp_coh_threshold` 过高时,可能导致无法生成足够 GCP,流程会在 Step 5A 失败。
|
||||
5. `unwrapping_coh_threshold`、`geocoding_coh_threshold` 需要结合研究区相干性调整。
|
||||
6. 本仓库目前没有统一的日志输出、配置文件和异常恢复机制,失败后通常需要根据控制台信息排查。
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,160 @@
|
||||
PRO Run_Batch_Task_Import
|
||||
|
||||
;===========================================================================
|
||||
; --- 用户配置区域 ---
|
||||
; 1. **数据根目录 (rootDir)**
|
||||
; 指定包含所有Task_开头文件夹的根目录路径。
|
||||
; !!! 重要: 路径末尾不要带斜杠,脚本会自动添加。
|
||||
; - 正确示例: 'C:\MyData\LuTan1' 或 '\\SERVER\Share\Data'
|
||||
|
||||
; 2. **处理数量 (numToProcess)**
|
||||
; 指定要处理的Task文件夹的最大数量。
|
||||
; - 如果设置为 0 或一个负数, 程序将处理所有找到的文件夹。
|
||||
|
||||
;===========================================================================
|
||||
temporary_directory = 'D:/Sarscape_IDL_Area'
|
||||
SARscape_Batch_Init,Temp_Directory=temporary_directory
|
||||
rootDir = 'K:/T2'
|
||||
numToProcess = 0
|
||||
;===========================================================================
|
||||
; --- 脚本执行区域 ---
|
||||
;===========================================================================
|
||||
pathSep = ''
|
||||
IF (!VERSION.OS_FAMILY EQ 'Windows') THEN pathSep = '\' ELSE pathSep = '/'
|
||||
e = ENVI(/HEADLESS)
|
||||
|
||||
IF rootDir EQ '' OR FILE_TEST(rootDir, /DIRECTORY) EQ 0 THEN BEGIN
|
||||
PRINT, '错误: "rootDir" 参数未设置或指定的目录不存在: ' + rootDir
|
||||
RETURN
|
||||
ENDIF
|
||||
|
||||
PRINT, '=============================================='
|
||||
PRINT, '开始批量导入陆探一号数据...'
|
||||
PRINT, '扫描目录: ' + rootDir
|
||||
PRINT, '=============================================='
|
||||
|
||||
; 查找所有Task_开头的文件夹 - 使用兼容的方法
|
||||
PRINT, '正在扫描Task_开头的文件夹...'
|
||||
|
||||
; 兼容旧版本IDL的方法:先搜索所有匹配项,然后筛选出文件夹
|
||||
allItems = FILE_SEARCH(rootDir + pathSep + 'Task_*', COUNT=nTotal)
|
||||
taskFolders = []
|
||||
nTasks = 0
|
||||
|
||||
IF nTotal GT 0 THEN BEGIN
|
||||
FOR i = 0, nTotal - 1 DO BEGIN
|
||||
IF FILE_TEST(allItems[i], /DIRECTORY) THEN BEGIN
|
||||
taskFolders = [taskFolders, allItems[i]]
|
||||
nTasks = nTasks + 1
|
||||
ENDIF
|
||||
ENDFOR
|
||||
ENDIF
|
||||
|
||||
IF nTasks EQ 0 THEN BEGIN
|
||||
PRINT, '在指定目录下没有找到任何Task_开头的文件夹。'
|
||||
RETURN
|
||||
ENDIF
|
||||
|
||||
PRINT, '共找到 ' + STRING(nTasks) + ' 个Task文件夹,准备开始处理...'
|
||||
PRINT, ''
|
||||
|
||||
processed_count = 0
|
||||
|
||||
; 使用标准的FOR循环替代FOREACH以确保兼容性
|
||||
FOR taskIndex = 0, nTasks - 1 DO BEGIN
|
||||
taskFolder = taskFolders[taskIndex]
|
||||
|
||||
IF (numToProcess GT 0) AND (processed_count GE numToProcess) THEN BEGIN
|
||||
PRINT, ''
|
||||
PRINT, '已达到指定的处理数量 (' + STRING(numToProcess) + '),程序终止。'
|
||||
BREAK
|
||||
ENDIF
|
||||
|
||||
taskName = FILE_BASENAME(taskFolder)
|
||||
PRINT, '=== 正在处理任务文件夹: ' + taskName + ' ==='
|
||||
|
||||
; 检查master和slave子文件夹
|
||||
masterDir = taskFolder + pathSep + 'master'
|
||||
slaveDir = taskFolder + pathSep + 'slave'
|
||||
|
||||
; 处理master文件夹
|
||||
IF FILE_TEST(masterDir, /DIRECTORY) THEN BEGIN
|
||||
PRINT, '--- 处理master文件夹 ---'
|
||||
processed_count = processed_count + Process_LuTan_Folder(masterDir, pathSep)
|
||||
ENDIF ELSE BEGIN
|
||||
PRINT, '警告: ' + taskName + ' 中未找到master文件夹'
|
||||
ENDELSE
|
||||
|
||||
; 处理slave文件夹
|
||||
IF FILE_TEST(slaveDir, /DIRECTORY) THEN BEGIN
|
||||
PRINT, '--- 处理slave文件夹 ---'
|
||||
processed_count = processed_count + Process_LuTan_Folder(slaveDir, pathSep)
|
||||
ENDIF ELSE BEGIN
|
||||
PRINT, '警告: ' + taskName + ' 中未找到slave文件夹'
|
||||
ENDELSE
|
||||
|
||||
PRINT, '完成处理任务文件夹: ' + taskName
|
||||
PRINT, ''
|
||||
ENDFOR
|
||||
|
||||
PRINT, '=============================================='
|
||||
PRINT, '所有任务已完成!'
|
||||
PRINT, '总共成功处理了 ' + STRING(processed_count) + ' 个数据文件夹。'
|
||||
PRINT, '=============================================='
|
||||
|
||||
END
|
||||
|
||||
; 处理单个陆探一号数据文件夹的子函数
|
||||
FUNCTION Process_LuTan_Folder, folderPath, pathSep
|
||||
COMPILE_OPT STRICTARR
|
||||
|
||||
baseName = FILE_BASENAME(folderPath)
|
||||
PRINT, '-> 正在检查文件夹: ' + baseName
|
||||
|
||||
; 直接使用master/slave文件夹作为输出目录,不再创建envi_import子目录
|
||||
outputDir = folderPath
|
||||
|
||||
; 检查是否已经处理过(通过检查是否存在.sml文件)
|
||||
check_sml_file = FILE_SEARCH(outputDir + pathSep + '*.sml', COUNT=sml_count)
|
||||
IF sml_count GT 0 THEN BEGIN
|
||||
PRINT, '-> 跳过: ' + baseName + ' 文件夹中已存在处理结果。'
|
||||
RETURN, 0
|
||||
ENDIF
|
||||
|
||||
; 查找元数据文件 - 查找文件夹中所有的.meta.xml文件
|
||||
metaFiles = FILE_SEARCH(folderPath + pathSep + '*.meta.xml', COUNT=nMetaFiles)
|
||||
|
||||
IF nMetaFiles EQ 0 THEN BEGIN
|
||||
PRINT, '-> 警告: 在文件夹 ' + baseName + ' 中未找到元数据文件(*.meta.xml),跳过。'
|
||||
RETURN, 0
|
||||
ENDIF
|
||||
|
||||
; 处理找到的元数据文件
|
||||
success_count = 0
|
||||
FOR i = 0, nMetaFiles - 1 DO BEGIN
|
||||
inputFile = metaFiles[i]
|
||||
metaFileName = FILE_BASENAME(inputFile)
|
||||
PRINT, '-> 准备导入: ' + metaFileName
|
||||
|
||||
; 执行导入任务
|
||||
oSB = obj_new('SARscapeBatch')
|
||||
task = ENVITask('SARsImportLuTan1')
|
||||
task.INPUT_FILE_LIST = [inputFile]
|
||||
task.ROOT_URI_FOR_OUTPUT = outputDir ; 直接输出到master/slave文件夹
|
||||
|
||||
CATCH, error_status
|
||||
IF error_status NE 0 THEN BEGIN
|
||||
PRINT, '-> 错误: 在处理 ' + metaFileName + ' 时发生严重错误。'
|
||||
PRINT, !ERROR_STATE.MSG
|
||||
CATCH, /CANCEL
|
||||
CONTINUE
|
||||
ENDIF
|
||||
|
||||
task.Execute
|
||||
CATCH, /CANCEL
|
||||
PRINT, '-> 成功: 数据已导入到 ' + baseName + ' 文件夹中。'
|
||||
success_count = success_count + 1
|
||||
ENDFOR
|
||||
|
||||
RETURN, success_count
|
||||
END
|
||||
@@ -0,0 +1,488 @@
|
||||
;================================================================================
|
||||
; ### HELPER FUNCTION: READ SML PARAMETER ###
|
||||
;================================================================================
|
||||
FUNCTION Read_SML_Parameter, sml_file, param_name_to_find
|
||||
IF N_ELEMENTS(sml_file) EQ 0 OR sml_file EQ '' THEN BEGIN
|
||||
PRINT, 'Error: An empty or undefined filename was passed to Read_SML_Parameter.'
|
||||
RETURN, ''
|
||||
ENDIF
|
||||
IF NOT FILE_TEST(sml_file) THEN BEGIN
|
||||
PRINT, 'Error: SML file not found -> ' + sml_file
|
||||
RETURN, ''
|
||||
ENDIF
|
||||
lun = -1
|
||||
ON_ERROR, 2
|
||||
OPENR, lun, sml_file, /GET_LUN, ERROR=err
|
||||
IF (err NE 0) THEN BEGIN
|
||||
PRINT, 'Error: Could not open SML file for reading -> ' + sml_file
|
||||
IF lun NE -1 THEN FREE_LUN, lun
|
||||
RETURN, ''
|
||||
ENDIF
|
||||
found_value = ''
|
||||
line = ''
|
||||
start_tag = '<' + STRUPCASE(param_name_to_find) + '>'
|
||||
end_tag = '</' + STRUPCASE(param_name_to_find) + '>'
|
||||
WHILE NOT EOF(lun) DO BEGIN
|
||||
READF, lun, line
|
||||
clean_line = STRTRIM(line, 2)
|
||||
line_upper = STRUPCASE(clean_line)
|
||||
start_pos = STRPOS(line_upper, start_tag)
|
||||
IF start_pos NE -1 THEN BEGIN
|
||||
end_pos = STRPOS(line_upper, end_tag, start_pos)
|
||||
IF end_pos NE -1 THEN BEGIN
|
||||
start_extract = start_pos + STRLEN(start_tag)
|
||||
value_length = end_pos - start_extract
|
||||
found_value = STRMID(clean_line, start_extract, value_length)
|
||||
BREAK
|
||||
ENDIF
|
||||
ENDIF
|
||||
ENDWHILE
|
||||
IF lun NE -1 THEN FREE_LUN, lun
|
||||
RETURN, found_value
|
||||
END
|
||||
|
||||
;================================================================================
|
||||
; ### HELPER FUNCTION: CREATE GCPs FROM COHERENCE ###
|
||||
;================================================================================
|
||||
FUNCTION CREATE_GCPS_FROM_COHERENCE, coherence_file, output_shp_file, COH_THRESHOLD=coh_threshold, NUM_POINTS=num_points
|
||||
|
||||
ON_ERROR, 2
|
||||
IF FILE_TEST(output_shp_file) THEN FILE_DELETE, output_shp_file, /QUIET
|
||||
|
||||
IF N_ELEMENTS(coh_threshold) EQ 0 THEN coh_threshold = 0.7
|
||||
IF N_ELEMENTS(num_points) EQ 0 THEN num_points = 100
|
||||
|
||||
e = ENVI(/HEADLESS)
|
||||
|
||||
PRINT, 'Opening coherence file: ' + coherence_file
|
||||
oRaster = e.OpenRaster(coherence_file)
|
||||
|
||||
ns = oRaster.NCOLUMNS
|
||||
nl = oRaster.NROWS
|
||||
data = oRaster.GetData(BANDS=[0])
|
||||
|
||||
PRINT, 'Searching for GCPs with coherence > ' + STRTRIM(coh_threshold, 2)
|
||||
|
||||
grid_dim = CEIL(SQRT(num_points))
|
||||
x_step = FLOOR(ns / grid_dim)
|
||||
y_step = FLOOR(nl / grid_dim)
|
||||
|
||||
pixel_points_x = [] & pixel_points_y = []
|
||||
|
||||
FOR j=0L, grid_dim-1 DO BEGIN
|
||||
FOR i=0L, grid_dim-1 DO BEGIN
|
||||
x_start = i * x_step & y_start = j * y_step
|
||||
x_end = (i+1) * x_step -1 < (ns-1)
|
||||
y_end = (j+1) * y_step -1 < (nl-1)
|
||||
IF (x_start GE ns) OR (y_start GE nl) THEN CONTINUE
|
||||
cell_data = data[x_start:x_end, y_start:y_end]
|
||||
max_val = MAX(cell_data, max_idx)
|
||||
IF max_val GE coh_threshold THEN BEGIN
|
||||
max_y_cell = max_idx / (x_end - x_start + 1)
|
||||
max_x_cell = max_idx MOD (x_end - x_start + 1)
|
||||
pixel_points_x = [pixel_points_x, x_start + max_x_cell]
|
||||
pixel_points_y = [pixel_points_y, y_start + max_y_cell]
|
||||
ENDIF
|
||||
ENDFOR
|
||||
ENDFOR
|
||||
|
||||
point_count = N_ELEMENTS(pixel_points_x)
|
||||
IF point_count EQ 0 THEN BEGIN
|
||||
PRINT, 'Error: No GCPs found.' & OBJ_DESTROY, oRaster & RETURN, 0
|
||||
ENDIF
|
||||
|
||||
PRINT, 'Found ' + STRTRIM(point_count, 2) + ' suitable GCPs.'
|
||||
PRINT, 'Creating shapefile using final authoritative methodology...'
|
||||
|
||||
oShape = OBJ_NEW('IDLffShape', output_shp_file, ENTITY_TYPE=1, /UPDATE)
|
||||
|
||||
oShape->AddAttribute, 'SHP_ID', 3, 10
|
||||
oShape->AddAttribute, 'GCP_LABEL', 7, 20
|
||||
oShape->AddAttribute, 'GCP_TYPE', 7, 20
|
||||
oShape->AddAttribute, 'GCP_COLUMN', 5, 24, PRECISION=6
|
||||
oShape->AddAttribute, 'GCP_ROW', 5, 24, PRECISION=6
|
||||
oShape->AddAttribute, 'GCP_OTHER_', 7, 10 ; User-verified field name
|
||||
|
||||
attr_template = oShape->GetAttributes(/ATTRIBUTE_STRUCTURE)
|
||||
|
||||
entNew = {IDL_SHAPE_ENTITY}
|
||||
entNew.SHAPE_TYPE = 1
|
||||
|
||||
FOR i=0, point_count-1 DO BEGIN
|
||||
x_float = FLOAT(pixel_points_x[i])
|
||||
y_float = FLOAT(pixel_points_y[i])
|
||||
|
||||
entNew.ISHAPE = i
|
||||
entNew.BOUNDS = [x_float, y_float, 0.0, 0.0, x_float, y_float, 0.0, 0.0]
|
||||
oShape->PutEntity, entNew
|
||||
|
||||
attr = attr_template
|
||||
attr.ATTRIBUTE_0 = i
|
||||
attr.ATTRIBUTE_1 = 'GCP_' + STRTRIM(i+1, 2)
|
||||
attr.ATTRIBUTE_2 = 'undefined'
|
||||
attr.ATTRIBUTE_3 = x_float
|
||||
attr.ATTRIBUTE_4 = y_float
|
||||
attr.ATTRIBUTE_5 = ''
|
||||
oShape->SetAttributes, i, attr
|
||||
ENDFOR
|
||||
|
||||
oShape->Close
|
||||
OBJ_DESTROY, [oShape, oRaster]
|
||||
|
||||
PRINT, 'Successfully created GCP shapefile: ' + output_shp_file
|
||||
RETURN, 1
|
||||
END
|
||||
|
||||
;================================================================================
|
||||
; ### 执行单个D-InSAR工作流的函数 ###
|
||||
;================================================================================
|
||||
FUNCTION Execute_Single_Dinsar_Workflow, master_base_file, slave_base_file, dem_base_file, workflow_root_name, $
|
||||
target_ground_resolution_m, filter_method, unwrapping_coh_threshold, $
|
||||
gcp_coh_threshold, gcp_number, geocoding_coh_threshold, geocoding_pixel_size_m
|
||||
|
||||
; ===================================================================
|
||||
; ### STEP 1: INTERFEROGRAM GENERATION ###
|
||||
; ===================================================================
|
||||
|
||||
PRINT, '======================================================'
|
||||
PRINT, '### STEP 1: INTERFEROGRAM GENERATION ###'
|
||||
PRINT, '======================================================'
|
||||
|
||||
master_sml = master_base_file + '.sml'
|
||||
slave_sml = slave_base_file + '.sml'
|
||||
|
||||
m_rg_sp_str = Read_SML_Parameter(master_sml, 'PixelSpacingRg')
|
||||
m_az_sp_str = Read_SML_Parameter(master_sml, 'PixelSpacingAz')
|
||||
m_inc_ang_str = Read_SML_Parameter(master_sml, 'IncidenceAngle')
|
||||
s_rg_sp_str = Read_SML_Parameter(slave_sml, 'PixelSpacingRg')
|
||||
s_az_sp_str = Read_SML_Parameter(slave_sml, 'PixelSpacingAz')
|
||||
s_inc_ang_str = Read_SML_Parameter(slave_sml, 'IncidenceAngle')
|
||||
|
||||
IF m_rg_sp_str EQ '' OR s_rg_sp_str EQ '' THEN RETURN, 0
|
||||
|
||||
m_rg_sp = FLOAT(m_rg_sp_str) & m_az_sp = FLOAT(m_az_sp_str) & m_inc_ang = FLOAT(m_inc_ang_str)
|
||||
s_rg_sp = FLOAT(s_rg_sp_str) & s_az_sp = FLOAT(s_az_sp_str) & s_inc_ang = FLOAT(s_inc_ang_str)
|
||||
|
||||
avg_az_spacing = (m_az_sp + s_az_sp) / 2.0
|
||||
azimuth_looks = (LONG(target_ground_resolution_m / avg_az_spacing)) > 1
|
||||
m_ground_rg_sp = m_rg_sp / SIN(m_inc_ang * !DPI / 180.0)
|
||||
s_ground_rg_sp = s_rg_sp / SIN(s_inc_ang * !DPI / 180.0)
|
||||
avg_ground_rg_sp = (m_ground_rg_sp + s_ground_rg_sp) / 2.0
|
||||
range_looks = (LONG(target_ground_resolution_m / avg_ground_rg_sp)) > 1
|
||||
|
||||
task1_obj = OBJ_NEW('SARscapeBatch', Module='InSARInterferogramGeneration')
|
||||
prefix_1 = 'MAIN_INSAR_INTERFEROGRAM_GENERATION_CMD.'
|
||||
task1_obj->SetParam, prefix_1 + 'INPUT_REFERENCE_FILE_NAME', master_base_file
|
||||
task1_obj->SetParam, prefix_1 + 'INPUT_SECONDARY_FILE_NAME', slave_base_file
|
||||
task1_obj->SetParam, prefix_1 + 'DEM_FILE_NAME', dem_base_file
|
||||
task1_obj->SetParam, prefix_1 + 'OUTPUT_ROOT_FILE_NAME', workflow_root_name
|
||||
task1_obj->SetParam, prefix_1 + 'RG_LOOKS_NBR', STRING(range_looks)
|
||||
task1_obj->SetParam, prefix_1 + 'AZ_LOOKS_NBR', STRING(azimuth_looks)
|
||||
task1_obj->SetParam, 'COREGISTRATION_CMD.COREGISTRATION_WITH_DEM_FLAG', 'OK'
|
||||
task1_obj->SetParam, 'FLAT_CMD.MAKE_FLATTENING_FLAG', 'OK'
|
||||
ok_step1 = task1_obj->Execute()
|
||||
OBJ_DESTROY, task1_obj
|
||||
IF NOT ok_step1 THEN RETURN, 0
|
||||
PRINT, '### STEP 1: SUCCESS!'
|
||||
|
||||
; ===================================================================
|
||||
; ### STEP 2: FILTERING AND COHERENCE GENERATION ###
|
||||
; ===================================================================
|
||||
|
||||
PRINT, '======================================================'
|
||||
PRINT, '### STEP 2: FILTERING AND COHERENCE GENERATION ###'
|
||||
PRINT, '======================================================'
|
||||
|
||||
task2_obj = OBJ_NEW('SARscapeBatch', Module='InSARFilterAndCoherence')
|
||||
prefix_main2 = 'MAIN_INSAR_FILTER_COHERENCE_CMD.'
|
||||
prefix_filt2 = 'FILTERING_CMD.'
|
||||
task2_obj->SetParam, prefix_main2 + 'INPUT_INTERF_FILE_NAME', workflow_root_name + '_dint'
|
||||
task2_obj->SetParam, prefix_main2 + 'INPUT_REFERENCE_FILE_NAME', workflow_root_name + '_reference_pwr'
|
||||
task2_obj->SetParam, prefix_main2 + 'INPUT_SECONDARY_FILE_NAME', workflow_root_name + '_secondary_pwr'
|
||||
task2_obj->SetParam, prefix_main2 + 'OUT_ROOT_FILE', workflow_root_name
|
||||
task2_obj->SetParam, prefix_filt2 + 'FILTERING_METHOD', filter_method
|
||||
task2_obj->SetParam, prefix_main2 + 'COHERENCE_FLAG', 'OK'
|
||||
task2_obj->SetParam, prefix_main2 + 'INTERF_FILT_FLAG', 'OK'
|
||||
ok_step2 = task2_obj->Execute()
|
||||
OBJ_DESTROY, task2_obj
|
||||
IF NOT ok_step2 THEN RETURN, 0
|
||||
PRINT, '### STEP 2: SUCCESS!'
|
||||
|
||||
; ===================================================================
|
||||
; ### STEP 3: ORBITAL TREND REMOVAL ###
|
||||
; ===================================================================
|
||||
|
||||
PRINT, '======================================================'
|
||||
PRINT, '### STEP 3: ORBITAL TREND REMOVAL ###'
|
||||
PRINT, '======================================================'
|
||||
|
||||
reflat_output_root = workflow_root_name + '_rrpf'
|
||||
task3_obj = OBJ_NEW('SARscapeBatch', Module='InSARRemoveResidualPhaseFrequency')
|
||||
prefix_3 = 'MAIN_INSAR_REMOVE_RESIDUAL_PHASE_FREQUENCY.'
|
||||
task3_obj->SetParam, prefix_3 + 'INTERF_FILE_NAME', workflow_root_name + '_fint'
|
||||
task3_obj->SetParam, prefix_3 + 'COHERENCE_FILE_NAME', workflow_root_name + '_cc'
|
||||
task3_obj->SetParam, prefix_3 + 'OUT_ROOT_FILE_NAME', reflat_output_root
|
||||
ok_step3 = task3_obj->Execute()
|
||||
OBJ_DESTROY, task3_obj
|
||||
IF NOT ok_step3 THEN RETURN, 0
|
||||
PRINT, '### STEP 3: SUCCESS!'
|
||||
|
||||
; ===================================================================
|
||||
; ### STEP 4: PHASE UNWRAPPING ###
|
||||
; ===================================================================
|
||||
|
||||
PRINT, '======================================================'
|
||||
PRINT, '### STEP 4: PHASE UNWRAPPING ###'
|
||||
PRINT, '======================================================'
|
||||
|
||||
reflat_upha_output_file = workflow_root_name + '_upha'
|
||||
task4_obj = OBJ_NEW('SARscapeBatch', Module='InSARPhaseUnwrapping')
|
||||
prefix_main4 = 'MAIN_INSAR_PHASE_UNWRAPPING_CMD.'
|
||||
prefix_upha4 = 'UPHA_CMD.'
|
||||
task4_obj->SetParam, prefix_main4 + 'INFILE_NAME', reflat_output_root + '_fint'
|
||||
task4_obj->SetParam, prefix_main4 + 'COHERENCEFILE_NAME', workflow_root_name + '_cc'
|
||||
task4_obj->SetParam, prefix_upha4 + 'UPHA_COH_THRESHOLD', STRING(unwrapping_coh_threshold)
|
||||
task4_obj->SetParam, prefix_main4 + 'OUTFILE_NAME', reflat_upha_output_file
|
||||
ok_step4 = task4_obj->Execute()
|
||||
OBJ_DESTROY, task4_obj
|
||||
IF NOT ok_step4 THEN RETURN, 0
|
||||
PRINT, '### STEP 4: SUCCESS!'
|
||||
|
||||
; ===================================================================
|
||||
; ### STEP 5: REFINEMENT AND REFLATTENING ###
|
||||
; ===================================================================
|
||||
|
||||
; --- STEP 5A: Automatic GCP Generation ---
|
||||
PRINT, '======================================================'
|
||||
PRINT, '### STEP 5A: 正在自动生成GCP... ###'
|
||||
PRINT, '======================================================'
|
||||
|
||||
auto_gcp_shp = workflow_root_name + '_auto_gcp.shp'
|
||||
gcp_success = CREATE_GCPS_FROM_COHERENCE(workflow_root_name + '_cc', auto_gcp_shp, COH_THRESHOLD=gcp_coh_threshold, NUM_POINTS=gcp_number)
|
||||
|
||||
IF gcp_success EQ 0 THEN BEGIN
|
||||
PRINT, '错误: 自动生成GCP shapefile失败。流程终止。'
|
||||
RETURN, 0
|
||||
ENDIF
|
||||
PRINT, '### STEP 5A: SUCCESS!'
|
||||
|
||||
; --- STEP 5B: Refinement and Reflattening Execution ---
|
||||
PRINT, '======================================================'
|
||||
PRINT, '### STEP 5B: 正在执行 REFINEMENT AND REFLATTENING ###'
|
||||
PRINT, '======================================================'
|
||||
|
||||
ref_output_root = workflow_root_name + '_reflat'
|
||||
task_refine = OBJ_NEW('SARscapeBatch', Module='InSARRefinementAndReflattening')
|
||||
prefix_refine = 'MAIN_INSAR_REFINEMENT_AND_REFLATTENING_CMD.'
|
||||
|
||||
task_refine->SetParam, prefix_refine + 'INPUT_UPHA_FILE_NAME', reflat_upha_output_file
|
||||
task_refine->SetParam, prefix_refine + 'INPUT_REFERENCE_FILE_NAME', workflow_root_name + '_reference_pwr'
|
||||
task_refine->SetParam, prefix_refine + 'INPUT_SECONDARY_FILE_NAME', workflow_root_name + '_secondary_pwr'
|
||||
task_refine->SetParam, prefix_refine + 'SLANT_RANGE_DEM_FILE_NAME', workflow_root_name + '_srdem'
|
||||
task_refine->SetParam, prefix_refine + 'SYNTHETIC_FILE_NAME', workflow_root_name + '_sint'
|
||||
task_refine->SetParam, prefix_refine + 'COHERENCE_FILE_NAME', workflow_root_name + '_cc'
|
||||
task_refine->SetParam, prefix_refine + 'OUTPUT_ROOT_NAME', ref_output_root
|
||||
task_refine->SetParam, prefix_refine + 'DEM_FILE_NAME', dem_base_file
|
||||
task_refine->SetParam, 'REFINEMENT_CMD.REFINEMENT_GCP_FILE_NAME', auto_gcp_shp
|
||||
|
||||
ok_step5 = task_refine->Execute()
|
||||
OBJ_DESTROY, task_refine
|
||||
IF NOT ok_step5 THEN RETURN, 0
|
||||
PRINT, '### STEP 5B: SUCCESS!'
|
||||
|
||||
; ===================================================================
|
||||
; ### STEP 6: PHASE TO DISPLACEMENT AND GEOCODING ###
|
||||
; ===================================================================
|
||||
|
||||
PRINT, '======================================================'
|
||||
PRINT, '### STEP 6: PHASE TO DISPLACEMENT AND GEOCODING ###'
|
||||
PRINT, '======================================================'
|
||||
|
||||
final_upha_file = ref_output_root + '_upha'
|
||||
geocoded_output_root = workflow_root_name + '_geo'
|
||||
|
||||
task6_obj = OBJ_NEW('SARscapeBatch', Module='InSARPhaseToDisplacement')
|
||||
|
||||
IF (~OBJ_VALID(task6_obj)) THEN BEGIN
|
||||
PRINT, '错误: 创建 SARscapeBatch 对象失败: InSARPhaseToDisplacement'
|
||||
RETURN, 0
|
||||
ENDIF
|
||||
|
||||
; --- Set Main Parameters ---
|
||||
prefix_main6 = 'MAIN_INSAR_PHASE_TO_DISPLACEMENT_CMD.'
|
||||
task6_obj->SetParam, prefix_main6 + 'INPUT_FILE_NAME', final_upha_file
|
||||
task6_obj->SetParam, prefix_main6 + 'OUTPUT_FILE', geocoded_output_root
|
||||
task6_obj->SetParam, prefix_main6 + 'COHERENCE_FILE', workflow_root_name + '_cc'
|
||||
task6_obj->SetParam, prefix_main6 + 'DEM_FILE_NAME', dem_base_file
|
||||
task6_obj->SetParam, prefix_main6 + 'COHERENCE_THRESHOLD', STRING(geocoding_coh_threshold)
|
||||
|
||||
; --- Set Other Required Parameters ---
|
||||
task6_obj->SetParam, prefix_main6 + 'INTERPOL_TYPE' , '4th_order_cc'
|
||||
task6_obj->SetParam, prefix_main6 + 'ALLOW_SKIP_REFINEMENT' , 'NotOK'
|
||||
task6_obj->SetParam, prefix_main6 + 'SARSCAPEENVIRONMENT' , 'IDL_ENVI_ENV'
|
||||
|
||||
; --- Set Geocoding Parameters ---
|
||||
prefix_geocode6 = 'GEOCODE_CMD.'
|
||||
task6_obj->SetParam, prefix_geocode6 + 'GEOCODE_RG_GRID_SIZE', STRING(geocoding_pixel_size_m)
|
||||
task6_obj->SetParam, prefix_geocode6 + 'GEOCODE_AZ_GRID_SIZE', STRING(geocoding_pixel_size_m)
|
||||
task6_obj->SetParam, prefix_geocode6 + 'GEOCODE_INTERPOL_BOX_SIZE', '7'
|
||||
|
||||
; --- Set Displacement Product Generation Parameters ---
|
||||
prefix_disp6 = 'DISPLACEMENT_PROJECTION_CMD.'
|
||||
task6_obj->SetParam, prefix_disp6 + 'GENERATE_LOS_FLAG', 'OK'
|
||||
task6_obj->SetParam, prefix_disp6 + 'GENERATE_VERTICAL_FLAG', 'NotOK'
|
||||
task6_obj->SetParam, prefix_disp6 + 'GENERATE_MAX_SLOPE_FLAG', 'NotOK'
|
||||
|
||||
PRINT, '正在执行地理编码任务...'
|
||||
ok_step6 = task6_obj->Execute()
|
||||
OBJ_DESTROY, task6_obj
|
||||
IF NOT ok_step6 THEN RETURN, 0
|
||||
PRINT, '### STEP 6: SUCCESS!'
|
||||
|
||||
; --- Final Success Message ---
|
||||
PRINT, '======================================================='
|
||||
PRINT, ' FULL D-InSAR WORKFLOW (STEPS 1-6) COMPLETED SUCCESSFULLY!'
|
||||
PRINT, '======================================================='
|
||||
|
||||
RETURN, 1
|
||||
|
||||
END
|
||||
|
||||
;================================================================================
|
||||
; ### 批量D-InSAR工作流主程序 ###
|
||||
;================================================================================
|
||||
PRO Batch_DinsarWorkflow_ALL
|
||||
|
||||
; --- 初始化SARscape环境 ---
|
||||
temporary_directory = 'D:\Sarscape_IDL_Area'
|
||||
SARscape_Batch_Init, Temp_Directory=temporary_directory
|
||||
e = ENVI(/HEADLESS)
|
||||
PRINT, 'ENVI 和 SARscape 环境初始化成功!'
|
||||
PRINT, ''
|
||||
|
||||
; ===================================================================
|
||||
; ### 全局配置 ###
|
||||
; ===================================================================
|
||||
rootDir = 'G:/Province_Result/new' ; 修改为您的根目录
|
||||
dem_base_file = 'F:/Data/SRTM30m/SRTMDEM_RSP_SARscape' ; DEM文件路径
|
||||
|
||||
; 处理参数配置
|
||||
target_ground_resolution_m = 10.0
|
||||
filter_method = 'GOLDSTEIN'
|
||||
unwrapping_coh_threshold = 0.05
|
||||
gcp_coh_threshold = 0.7
|
||||
gcp_number = 100
|
||||
geocoding_coh_threshold = 0.0
|
||||
geocoding_pixel_size_m = 10.0
|
||||
|
||||
; 最大处理数量 (0表示处理所有)
|
||||
numToProcess = 0
|
||||
|
||||
; ===================================================================
|
||||
; ### 查找并处理所有Task文件夹 ###
|
||||
; ===================================================================
|
||||
|
||||
pathSep = ''
|
||||
IF (!VERSION.OS_FAMILY EQ 'Windows') THEN pathSep = '\' ELSE pathSep = '/'
|
||||
|
||||
IF rootDir EQ '' OR FILE_TEST(rootDir, /DIRECTORY) EQ 0 THEN BEGIN
|
||||
PRINT, '错误: "rootDir" 参数未设置或指定的目录不存在: ' + rootDir
|
||||
RETURN
|
||||
ENDIF
|
||||
|
||||
PRINT, '=============================================='
|
||||
PRINT, '开始批量D-InSAR处理...'
|
||||
PRINT, '扫描目录: ' + rootDir
|
||||
PRINT, '=============================================='
|
||||
|
||||
; 查找所有Task_开头的文件夹
|
||||
PRINT, '正在扫描Task_开头的文件夹...'
|
||||
taskFolders = FILE_SEARCH(rootDir + pathSep + 'Task_*', /FOLDERS, COUNT=nTasks)
|
||||
|
||||
IF nTasks EQ 0 THEN BEGIN
|
||||
PRINT, '在指定目录下没有找到任何Task_开头的文件夹。'
|
||||
RETURN
|
||||
ENDIF
|
||||
|
||||
PRINT, '共找到 ' + STRING(nTasks) + ' 个Task文件夹,准备开始处理...'
|
||||
PRINT, ''
|
||||
|
||||
processed_count = 0
|
||||
|
||||
; 循环处理每个Task文件夹
|
||||
FOREACH taskFolder, taskFolders DO BEGIN
|
||||
IF (numToProcess GT 0) AND (processed_count GE numToProcess) THEN BEGIN
|
||||
PRINT, ''
|
||||
PRINT, '已达到指定的处理数量 (' + STRING(numToProcess) + '),程序终止。'
|
||||
BREAK
|
||||
ENDIF
|
||||
|
||||
taskName = FILE_BASENAME(taskFolder)
|
||||
PRINT, '=== 正在处理任务文件夹: ' + taskName + ' ==='
|
||||
|
||||
; 检查master和slave子文件夹
|
||||
masterDir = taskFolder + pathSep + 'master'
|
||||
slaveDir = taskFolder + pathSep + 'slave'
|
||||
|
||||
IF (NOT FILE_TEST(masterDir, /DIRECTORY)) OR (NOT FILE_TEST(slaveDir, /DIRECTORY)) THEN BEGIN
|
||||
PRINT, '警告: ' + taskName + ' 中缺少master或slave文件夹,跳过。'
|
||||
PRINT, ''
|
||||
CONTINUE
|
||||
ENDIF
|
||||
|
||||
; 查找master和slave文件夹中的SML文件
|
||||
masterSmlFiles = FILE_SEARCH(masterDir + pathSep + '*.sml', COUNT=nMaster)
|
||||
slaveSmlFiles = FILE_SEARCH(slaveDir + pathSep + '*.sml', COUNT=nSlave)
|
||||
|
||||
IF (nMaster EQ 0) OR (nSlave EQ 0) THEN BEGIN
|
||||
PRINT, '警告: ' + taskName + ' 中未找到足够的SML文件 (master: ' + STRING(nMaster) + ', slave: ' + STRING(nSlave) + '),跳过。'
|
||||
PRINT, ''
|
||||
CONTINUE
|
||||
ENDIF
|
||||
|
||||
; 取第一个找到的SML文件作为主从影像
|
||||
master_base_file = masterSmlFiles[0]
|
||||
slave_base_file = slaveSmlFiles[0]
|
||||
|
||||
; 移除.sml扩展名,得到基础文件名
|
||||
master_base_file = STRMID(master_base_file, 0, STRLEN(master_base_file) - 4)
|
||||
slave_base_file = STRMID(slave_base_file, 0, STRLEN(slave_base_file) - 4)
|
||||
|
||||
; 设置工作流输出根名称(在Task文件夹内创建结果)
|
||||
workflow_root_name = taskFolder + pathSep + 'dinsar_results' + pathSep + 'workflow'
|
||||
|
||||
; 创建输出目录
|
||||
outputDir = FILE_DIRNAME(workflow_root_name)
|
||||
IF NOT FILE_TEST(outputDir, /DIRECTORY) THEN BEGIN
|
||||
FILE_MKDIR, outputDir
|
||||
PRINT, '-> 创建输出目录: ' + outputDir
|
||||
ENDIF
|
||||
|
||||
PRINT, '-> Master文件: ' + FILE_BASENAME(master_base_file)
|
||||
PRINT, '-> Slave文件: ' + FILE_BASENAME(slave_base_file)
|
||||
PRINT, '-> 输出路径: ' + workflow_root_name
|
||||
PRINT, ''
|
||||
|
||||
; 执行单个D-InSAR工作流
|
||||
success = Execute_Single_Dinsar_Workflow(master_base_file, slave_base_file, dem_base_file, workflow_root_name, $
|
||||
target_ground_resolution_m, filter_method, unwrapping_coh_threshold, $
|
||||
gcp_coh_threshold, gcp_number, geocoding_coh_threshold, geocoding_pixel_size_m)
|
||||
|
||||
IF success THEN BEGIN
|
||||
PRINT, '### ' + taskName + ': D-InSAR工作流完成! ###'
|
||||
processed_count = processed_count + 1
|
||||
ENDIF ELSE BEGIN
|
||||
PRINT, '### ' + taskName + ': D-InSAR工作流失败! ###'
|
||||
ENDELSE
|
||||
|
||||
PRINT, ''
|
||||
ENDFOREACH
|
||||
|
||||
PRINT, '=============================================='
|
||||
PRINT, '批量D-InSAR处理完成!'
|
||||
PRINT, '总共成功处理了 ' + STRING(processed_count) + ' 个Task文件夹。'
|
||||
PRINT, '=============================================='
|
||||
|
||||
SARscape_Batch_Exit
|
||||
|
||||
END
|
||||
@@ -94,6 +94,51 @@ function Get-ListeningProcessId {
|
||||
return 0
|
||||
}
|
||||
|
||||
function Invoke-UiShutdown {
|
||||
param(
|
||||
[string]$Url,
|
||||
[string]$ShutdownToken
|
||||
)
|
||||
|
||||
if ([string]::IsNullOrWhiteSpace($Url) -or [string]::IsNullOrWhiteSpace($ShutdownToken)) {
|
||||
return $false
|
||||
}
|
||||
|
||||
try {
|
||||
$headers = @{
|
||||
"X-S1DL-Shutdown-Token" = $ShutdownToken
|
||||
}
|
||||
$shutdownUrl = $Url.TrimEnd("/") + "/api/shutdown"
|
||||
$null = Invoke-RestMethod -Uri $shutdownUrl -Method Post -Headers $headers -TimeoutSec 5 -ErrorAction Stop
|
||||
return $true
|
||||
} catch {
|
||||
return $false
|
||||
}
|
||||
}
|
||||
|
||||
function Wait-ForUiShutdown {
|
||||
param(
|
||||
[int]$ProcessId,
|
||||
[int]$Port,
|
||||
[int]$TimeoutSeconds = 10
|
||||
)
|
||||
|
||||
$deadline = (Get-Date).AddSeconds($TimeoutSeconds)
|
||||
while ((Get-Date) -lt $deadline) {
|
||||
$running = Get-RunningProcess -Id $ProcessId
|
||||
$listenerProcessId = 0
|
||||
if ($Port -gt 0) {
|
||||
$listenerProcessId = Get-ListeningProcessId -Port $Port
|
||||
}
|
||||
if (-not $running -and $listenerProcessId -le 0) {
|
||||
return $true
|
||||
}
|
||||
Start-Sleep -Milliseconds 500
|
||||
}
|
||||
|
||||
return $false
|
||||
}
|
||||
|
||||
function Load-State {
|
||||
param([string]$Path)
|
||||
|
||||
@@ -113,14 +158,16 @@ function Save-State {
|
||||
[string]$Path,
|
||||
[int]$ProcessId,
|
||||
[int]$Port,
|
||||
[string]$Url
|
||||
[string]$Url,
|
||||
[string]$ShutdownToken
|
||||
)
|
||||
|
||||
$payload = [ordered]@{
|
||||
pid = $ProcessId
|
||||
port = $Port
|
||||
url = $Url
|
||||
started_at = (Get-Date).ToString("s")
|
||||
pid = $ProcessId
|
||||
port = $Port
|
||||
url = $Url
|
||||
shutdown_token = $ShutdownToken
|
||||
started_at = (Get-Date).ToString("s")
|
||||
}
|
||||
|
||||
$payload | ConvertTo-Json | Set-Content -LiteralPath $Path -Encoding UTF8
|
||||
@@ -157,11 +204,21 @@ if ($state -and -not $ForceRestart) {
|
||||
}
|
||||
|
||||
if ($ForceRestart -and $state) {
|
||||
$oldProcess = Get-RunningProcess -Id ([int]$state.pid)
|
||||
if ($oldProcess) {
|
||||
Write-Info "Stopping previous tracked process $($oldProcess.Id)"
|
||||
Stop-Process -Id $oldProcess.Id -Force
|
||||
Start-Sleep -Seconds 1
|
||||
$existingStopped = $false
|
||||
if ($state.url -and $state.shutdown_token) {
|
||||
Write-Info "Requesting shutdown for previous tracked UI"
|
||||
if (Invoke-UiShutdown -Url $state.url -ShutdownToken $state.shutdown_token) {
|
||||
$existingStopped = Wait-ForUiShutdown -ProcessId ([int]$state.pid) -Port ([int]$state.port)
|
||||
}
|
||||
}
|
||||
|
||||
if (-not $existingStopped) {
|
||||
$oldProcess = Get-RunningProcess -Id ([int]$state.pid)
|
||||
if ($oldProcess) {
|
||||
Write-Info "Stopping previous tracked process $($oldProcess.Id)"
|
||||
Stop-Process -Id $oldProcess.Id -Force
|
||||
Start-Sleep -Seconds 1
|
||||
}
|
||||
}
|
||||
Remove-State -Path $statePath
|
||||
}
|
||||
@@ -183,7 +240,7 @@ foreach ($port in $PreferredPorts) {
|
||||
if (Test-Health -Port $port) {
|
||||
$url = "http://127.0.0.1:$port/"
|
||||
Write-Info "Found an existing running UI at $url"
|
||||
Save-State -Path $statePath -ProcessId 0 -Port $port -Url $url
|
||||
Save-State -Path $statePath -ProcessId 0 -Port $port -Url $url -ShutdownToken ""
|
||||
if (-not $NoBrowser) {
|
||||
Start-Process $url | Out-Null
|
||||
}
|
||||
@@ -206,6 +263,7 @@ if (-not (Test-Path -LiteralPath $pythonPath)) {
|
||||
|
||||
$portToUse = Get-FreePort -Candidates $PreferredPorts
|
||||
$url = "http://127.0.0.1:$portToUse/"
|
||||
$shutdownToken = [guid]::NewGuid().ToString("N")
|
||||
|
||||
if (Test-Path -LiteralPath $stdoutPath) {
|
||||
Remove-Item -LiteralPath $stdoutPath -Force
|
||||
@@ -225,14 +283,19 @@ $arguments = @(
|
||||
)
|
||||
|
||||
Write-Info "Starting UI at $url"
|
||||
$process = Start-Process `
|
||||
-FilePath $pythonPath `
|
||||
-ArgumentList $arguments `
|
||||
-WorkingDirectory $projectRoot `
|
||||
-WindowStyle Hidden `
|
||||
-RedirectStandardOutput $stdoutPath `
|
||||
-RedirectStandardError $stderrPath `
|
||||
-PassThru
|
||||
try {
|
||||
$env:S1DL_UI_SHUTDOWN_TOKEN = $shutdownToken
|
||||
$process = Start-Process `
|
||||
-FilePath $pythonPath `
|
||||
-ArgumentList $arguments `
|
||||
-WorkingDirectory $projectRoot `
|
||||
-WindowStyle Hidden `
|
||||
-RedirectStandardOutput $stdoutPath `
|
||||
-RedirectStandardError $stderrPath `
|
||||
-PassThru
|
||||
} finally {
|
||||
Remove-Item Env:S1DL_UI_SHUTDOWN_TOKEN -ErrorAction SilentlyContinue
|
||||
}
|
||||
|
||||
$started = $false
|
||||
for ($i = 0; $i -lt 30; $i++) {
|
||||
@@ -261,12 +324,12 @@ if (-not $started) {
|
||||
throw "UI failed to start."
|
||||
}
|
||||
|
||||
Save-State -Path $statePath -ProcessId $process.Id -Port $portToUse -Url $url
|
||||
Save-State -Path $statePath -ProcessId $process.Id -Port $portToUse -Url $url -ShutdownToken $shutdownToken
|
||||
$listenerProcessId = Get-ListeningProcessId -Port $portToUse
|
||||
if ($listenerProcessId -gt 0) {
|
||||
Save-State -Path $statePath -ProcessId $listenerProcessId -Port $portToUse -Url $url
|
||||
Save-State -Path $statePath -ProcessId $listenerProcessId -Port $portToUse -Url $url -ShutdownToken $shutdownToken
|
||||
} else {
|
||||
Save-State -Path $statePath -ProcessId $process.Id -Port $portToUse -Url $url
|
||||
Save-State -Path $statePath -ProcessId $process.Id -Port $portToUse -Url $url -ShutdownToken $shutdownToken
|
||||
}
|
||||
Write-Info "UI is ready: $url"
|
||||
|
||||
|
||||
@@ -44,6 +44,51 @@ function Get-ListeningProcessId {
|
||||
return 0
|
||||
}
|
||||
|
||||
function Invoke-UiShutdown {
|
||||
param(
|
||||
[string]$Url,
|
||||
[string]$ShutdownToken
|
||||
)
|
||||
|
||||
if ([string]::IsNullOrWhiteSpace($Url) -or [string]::IsNullOrWhiteSpace($ShutdownToken)) {
|
||||
return $false
|
||||
}
|
||||
|
||||
try {
|
||||
$headers = @{
|
||||
"X-S1DL-Shutdown-Token" = $ShutdownToken
|
||||
}
|
||||
$shutdownUrl = $Url.TrimEnd("/") + "/api/shutdown"
|
||||
$null = Invoke-RestMethod -Uri $shutdownUrl -Method Post -Headers $headers -TimeoutSec 5 -ErrorAction Stop
|
||||
return $true
|
||||
} catch {
|
||||
return $false
|
||||
}
|
||||
}
|
||||
|
||||
function Wait-ForUiShutdown {
|
||||
param(
|
||||
[int]$ProcessId,
|
||||
[int]$Port,
|
||||
[int]$TimeoutSeconds = 10
|
||||
)
|
||||
|
||||
$deadline = (Get-Date).AddSeconds($TimeoutSeconds)
|
||||
while ((Get-Date) -lt $deadline) {
|
||||
$running = Get-RunningProcess -Id $ProcessId
|
||||
$listenerProcessId = 0
|
||||
if ($Port -gt 0) {
|
||||
$listenerProcessId = Get-ListeningProcessId -Port $Port
|
||||
}
|
||||
if (-not $running -and $listenerProcessId -le 0) {
|
||||
return $true
|
||||
}
|
||||
Start-Sleep -Milliseconds 500
|
||||
}
|
||||
|
||||
return $false
|
||||
}
|
||||
|
||||
$projectRoot = Split-Path -Parent $PSCommandPath
|
||||
$statePath = Join-Path $projectRoot ".s1dl\web-ui-state.json"
|
||||
|
||||
@@ -53,23 +98,49 @@ if (-not (Test-Path -LiteralPath $statePath)) {
|
||||
}
|
||||
|
||||
$state = Get-Content -LiteralPath $statePath -Raw | ConvertFrom-Json
|
||||
$process = Get-RunningProcess -Id ([int]$state.pid)
|
||||
$stopped = $false
|
||||
|
||||
if ($process) {
|
||||
Write-Info "Stopping UI process $($process.Id)"
|
||||
Stop-Process -Id $process.Id -Force
|
||||
} else {
|
||||
$listenerProcessId = 0
|
||||
if ($state.port) {
|
||||
$listenerProcessId = Get-ListeningProcessId -Port ([int]$state.port)
|
||||
if ($state.url -and $state.shutdown_token) {
|
||||
Write-Info "Requesting UI shutdown via API"
|
||||
if (Invoke-UiShutdown -Url $state.url -ShutdownToken $state.shutdown_token) {
|
||||
$stopped = Wait-ForUiShutdown -ProcessId ([int]$state.pid) -Port ([int]$state.port)
|
||||
}
|
||||
if ($listenerProcessId -gt 0) {
|
||||
Write-Info "Stopping listener process $listenerProcessId"
|
||||
Stop-Process -Id $listenerProcessId -Force
|
||||
}
|
||||
|
||||
if (-not $stopped) {
|
||||
$process = Get-RunningProcess -Id ([int]$state.pid)
|
||||
|
||||
if ($process) {
|
||||
Write-Info "Stopping UI process $($process.Id)"
|
||||
try {
|
||||
Stop-Process -Id $process.Id -Force -ErrorAction Stop
|
||||
$stopped = $true
|
||||
} catch {
|
||||
throw "Failed to stop UI process $($process.Id). It may have been started from an elevated shell. Start or stop it with the same privilege level, or restart it with the updated launcher once so it can use API shutdown. $($_.Exception.Message)"
|
||||
}
|
||||
} else {
|
||||
Write-Info "Tracked process is not running."
|
||||
$listenerProcessId = 0
|
||||
if ($state.port) {
|
||||
$listenerProcessId = Get-ListeningProcessId -Port ([int]$state.port)
|
||||
}
|
||||
if ($listenerProcessId -gt 0) {
|
||||
Write-Info "Stopping listener process $listenerProcessId"
|
||||
try {
|
||||
Stop-Process -Id $listenerProcessId -Force -ErrorAction Stop
|
||||
$stopped = $true
|
||||
} catch {
|
||||
throw "Failed to stop UI listener process $listenerProcessId. It may have been started from an elevated shell. Start or stop it with the same privilege level, or restart it with the updated launcher once so it can use API shutdown. $($_.Exception.Message)"
|
||||
}
|
||||
} else {
|
||||
Write-Info "Tracked process is not running."
|
||||
$stopped = $true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (-not $stopped) {
|
||||
throw "UI shutdown did not complete."
|
||||
}
|
||||
|
||||
Remove-Item -LiteralPath $statePath -Force
|
||||
Write-Info "Done."
|
||||
|
||||
@@ -14,7 +14,11 @@ from sentinel_orbit_downloader.api.routes_projects import router as projects_rou
|
||||
from sentinel_orbit_downloader.api.routes_regions import router as regions_router
|
||||
from sentinel_orbit_downloader.api.routes_search import router as search_router
|
||||
from sentinel_orbit_downloader.api.routes_settings import router as settings_router
|
||||
from sentinel_orbit_downloader.core.config import configured_api_host, configured_api_port
|
||||
from sentinel_orbit_downloader.core.config import (
|
||||
configured_api_host,
|
||||
configured_api_port,
|
||||
configured_ui_shutdown_token,
|
||||
)
|
||||
|
||||
|
||||
def create_app() -> FastAPI:
|
||||
@@ -23,6 +27,7 @@ def create_app() -> FastAPI:
|
||||
version="0.1.0",
|
||||
description="Local API for Sentinel-1 search, orbit matching, and task management.",
|
||||
)
|
||||
app.state.shutdown_token = configured_ui_shutdown_token()
|
||||
|
||||
web_root = Path(__file__).resolve().parents[1] / "web"
|
||||
app.mount("/static", StaticFiles(directory=web_root), name="static")
|
||||
|
||||
@@ -1,14 +1,42 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
import os
|
||||
import signal
|
||||
import time
|
||||
|
||||
from fastapi import APIRouter, BackgroundTasks, Depends, Header, HTTPException, Request, status
|
||||
|
||||
from sentinel_orbit_downloader.api.deps import get_db_path
|
||||
from sentinel_orbit_downloader.api.schemas import HealthResponse
|
||||
from sentinel_orbit_downloader.api.schemas import HealthResponse, MessageResponse
|
||||
|
||||
|
||||
router = APIRouter(tags=["health"])
|
||||
|
||||
|
||||
def _terminate_current_process() -> None:
|
||||
time.sleep(0.2)
|
||||
try:
|
||||
os.kill(os.getpid(), signal.SIGTERM)
|
||||
except Exception:
|
||||
os._exit(0)
|
||||
|
||||
|
||||
@router.get("/health", response_model=HealthResponse)
|
||||
def health(db_path=Depends(get_db_path)) -> HealthResponse:
|
||||
return HealthResponse(status="ok", db_path=str(db_path))
|
||||
|
||||
|
||||
@router.post("/shutdown", response_model=MessageResponse, include_in_schema=False)
|
||||
def shutdown(
|
||||
background_tasks: BackgroundTasks,
|
||||
request: Request,
|
||||
x_s1dl_shutdown_token: str | None = Header(default=None, alias="X-S1DL-Shutdown-Token"),
|
||||
) -> MessageResponse:
|
||||
expected_token = getattr(request.app.state, "shutdown_token", None)
|
||||
if not expected_token:
|
||||
raise HTTPException(status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail="Shutdown is disabled.")
|
||||
if x_s1dl_shutdown_token != expected_token:
|
||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Invalid shutdown token.")
|
||||
|
||||
background_tasks.add_task(_terminate_current_process)
|
||||
return MessageResponse(message="Shutdown requested.")
|
||||
|
||||
@@ -25,3 +25,10 @@ def configured_api_port() -> int:
|
||||
if not raw:
|
||||
return DEFAULT_API_PORT
|
||||
return int(raw)
|
||||
|
||||
|
||||
def configured_ui_shutdown_token() -> str | None:
|
||||
raw = os.getenv("S1DL_UI_SHUTDOWN_TOKEN")
|
||||
if not raw:
|
||||
return None
|
||||
return raw
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
* text=auto
|
||||
|
||||
.gitattributes text eol=lf
|
||||
.gitignore text eol=lf
|
||||
*.md text eol=lf
|
||||
*.py text eol=lf
|
||||
*.go text eol=lf
|
||||
*.mod text eol=lf
|
||||
*.toml text eol=lf
|
||||
*.yml text eol=lf
|
||||
*.idl text eol=lf
|
||||
*.pro text eol=lf
|
||||
*.cmd text eol=crlf
|
||||
|
||||
*.exe binary
|
||||
*.sav binary
|
||||
@@ -0,0 +1,27 @@
|
||||
.venv/
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
.pytest_cache/
|
||||
.mypy_cache/
|
||||
.ruff_cache/
|
||||
/.tmp_tests/
|
||||
logs/
|
||||
GMTED2010.jp2
|
||||
|
||||
# Local wrapper state
|
||||
gf3wrapper.json
|
||||
.gf3_extract/
|
||||
.gf3_runtime/
|
||||
|
||||
# Production data and generated outputs
|
||||
/data/
|
||||
/outputs/
|
||||
*.tar.gz
|
||||
*.tgz
|
||||
*.zip
|
||||
*.tif
|
||||
*.tiff
|
||||
*.img
|
||||
*.dat
|
||||
*.sml
|
||||
*.hdr
|
||||
@@ -0,0 +1,30 @@
|
||||
"""Compatibility wrapper for archive extraction."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
_SRC = Path(__file__).resolve().parent / "src"
|
||||
if _SRC.is_dir():
|
||||
sys.path.insert(0, str(_SRC))
|
||||
|
||||
from gf3_l1a2l2.decompress import extract_compressed_files
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Extract GF3 archives")
|
||||
parser.add_argument("input", type=Path, help="folder containing .tar.gz or .zip archives")
|
||||
parser.add_argument("--overwrite", action="store_true", help="overwrite existing extracted folders")
|
||||
args = parser.parse_args()
|
||||
|
||||
results = extract_compressed_files(args.input, overwrite=args.overwrite)
|
||||
failed = [item for item in results if item.status == "failed"]
|
||||
for item in results:
|
||||
print(f"{item.status}\t{item.archive}\t{item.output_dir}\t{item.message}")
|
||||
return 1 if failed else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,103 @@
|
||||
# GF3 L1A to L2 Pipeline
|
||||
|
||||
This repository contains two GF-3 L1A to L2 processing paths:
|
||||
|
||||
- **SARscape Go wrapper**: Windows-oriented production wrapper around ENVI/IDL Runtime and SARscape. This is the recommended path when SARscape is available.
|
||||
- **Python pipeline**: GDAL/Rasterio/NumPy implementation kept as a pure Python package and CLI.
|
||||
|
||||
The SARscape wrapper currently has the more complete operational workflow for GF-3 scene import, multilook/filter/geocode processing, batch directory discovery, saved defaults, and progress reporting.
|
||||
|
||||
## Repository Layout
|
||||
|
||||
```text
|
||||
dist/windows/ Prebuilt Windows executable
|
||||
docs/ Operational documentation
|
||||
src/gf3_l1a2l2/ Python package
|
||||
tests/ Python unit tests and small fixtures
|
||||
tools/gf3_sarscape_wrapper/ Go wrapper source and embedded SAV asset
|
||||
tools/gf3_sarscape_wrapper/idl/ IDL/SARscape source scripts
|
||||
```
|
||||
|
||||
Large GF-3 archives, extracted scenes, SARscape outputs, and local runtime files are intentionally not tracked in Git.
|
||||
|
||||
## Recommended: SARscape Go Wrapper
|
||||
|
||||
Use the prebuilt executable:
|
||||
|
||||
```powershell
|
||||
.\dist\windows\gf3wrapper.exe `
|
||||
-input "D:\GF3\L1A_BATCH" `
|
||||
-output "E:\GF3\L2_SARscape" `
|
||||
-dem "D:\DEM\COPDEM_GLO30_China_4326_DEM" `
|
||||
-pol "HH,HV"
|
||||
```
|
||||
|
||||
Supported input forms:
|
||||
|
||||
- A single `*.tar.gz` GF-3 L1A package.
|
||||
- A single `*.meta.xml` scene metadata file.
|
||||
- A directory containing multiple `*.tar.gz` or `*.meta.xml` scenes.
|
||||
|
||||
The wrapper writes one output subdirectory per scene. On successful runs it saves defaults next to the executable in `gf3wrapper.json`, so double-click interactive mode can reuse the previous input/output/DEM/polarization settings.
|
||||
|
||||
Batch directory runs continue after per-scene failures. If one archive is truncated, invalid, missing metadata, or fails during SARscape processing, the wrapper prints the failed scene and reason, skips that scene, and continues with the remaining scenes. At the end it prints a batch summary and returns a non-zero exit code when any scene failed.
|
||||
|
||||
Before processing each scene, the wrapper checks the scene output directory for completed geocoded products for the requested polarizations. Already completed scenes are skipped, so rerunning the same input/output pair only processes missing or incomplete scenes.
|
||||
|
||||
More details: [docs/sarscape_go_wrapper.md](docs/sarscape_go_wrapper.md)
|
||||
|
||||
## Python Pipeline
|
||||
|
||||
Use an environment that already has working GDAL/OSGeo Python bindings. On Windows, Conda/Miniforge is usually the least fragile route.
|
||||
|
||||
```powershell
|
||||
conda env create -f environment.yml
|
||||
conda activate gf3-l1a2l2
|
||||
pip install -e . --no-deps
|
||||
```
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
gf3-l1a2l2 run --input D:\GF3\L1A_BATCH --output E:\GF3\L2_Image --dem .\GMTED2010.jp2
|
||||
```
|
||||
|
||||
Common options:
|
||||
|
||||
```powershell
|
||||
gf3-l1a2l2 run `
|
||||
--input D:\GF3\L1A_BATCH `
|
||||
--output E:\GF3\L2_Image `
|
||||
--dem .\GMTED2010.jp2 `
|
||||
--workers 1 `
|
||||
--x-res 0.0002 `
|
||||
--y-res 0.0002 `
|
||||
--dst-srs EPSG:4326
|
||||
```
|
||||
|
||||
Legacy wrappers remain available:
|
||||
|
||||
```powershell
|
||||
python gf3_L1A_To_L2.py run --input D:\GF3\L1A_BATCH --output E:\GF3\L2_Image --dem .\GMTED2010.jp2
|
||||
python Decompression.py D:\GF3\L1A_BATCH
|
||||
```
|
||||
|
||||
## Development Checks
|
||||
|
||||
Python:
|
||||
|
||||
```powershell
|
||||
python -m pytest
|
||||
```
|
||||
|
||||
Go wrapper:
|
||||
|
||||
```powershell
|
||||
cd .\tools\gf3_sarscape_wrapper
|
||||
$env:GOTELEMETRY = "off"
|
||||
go build -o ..\..\dist\windows\gf3wrapper.exe .
|
||||
```
|
||||
|
||||
## Data Policy
|
||||
|
||||
Do not commit production GF-3 archives or generated products. Keep large data on NAS/storage and document the path in run notes. Only small sanitized fixtures should be added under `tests/fixtures`.
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
# Windows Distribution
|
||||
|
||||
`gf3wrapper.exe` is the prebuilt SARscape Go wrapper.
|
||||
|
||||
Run from the repository root:
|
||||
|
||||
```powershell
|
||||
.\dist\windows\gf3wrapper.exe `
|
||||
-input "D:\GF3\L1A_BATCH" `
|
||||
-output "E:\GF3\L2_SARscape" `
|
||||
-dem "D:\DEM\COPDEM_GLO30_China_4326_DEM" `
|
||||
-pol "HH,HV"
|
||||
```
|
||||
|
||||
The executable requires ENVI/IDL Runtime and SARscape on the target machine.
|
||||
|
||||
In directory mode, failed scenes are skipped and listed in a final batch summary. The wrapper continues with remaining scenes, but exits non-zero if any scene failed.
|
||||
|
||||
Rerunning the same input/output pair skips scenes with complete geocoded outputs for the requested polarizations and processes only missing or incomplete scenes.
|
||||
|
||||
After successful runs, `gf3wrapper.json` is created in this directory to store defaults. That file is local runtime state and is ignored by Git.
|
||||
Binary file not shown.
@@ -0,0 +1,45 @@
|
||||
# Repository Layout
|
||||
|
||||
```text
|
||||
.
|
||||
|-- dist/
|
||||
| `-- windows/
|
||||
| `-- gf3wrapper.exe
|
||||
|-- docs/
|
||||
| |-- repository_layout.md
|
||||
| `-- sarscape_go_wrapper.md
|
||||
|-- src/
|
||||
| `-- gf3_l1a2l2/
|
||||
|-- tests/
|
||||
| `-- fixtures/
|
||||
|-- tools/
|
||||
| `-- gf3_sarscape_wrapper/
|
||||
| |-- assets/
|
||||
| |-- idl/
|
||||
| |-- go.mod
|
||||
| `-- main.go
|
||||
|-- README.md
|
||||
|-- environment.yml
|
||||
`-- pyproject.toml
|
||||
```
|
||||
|
||||
## What Belongs In Git
|
||||
|
||||
- Python package source.
|
||||
- Go wrapper source.
|
||||
- IDL/SARscape source scripts.
|
||||
- Small embedded runtime assets needed to build the wrapper.
|
||||
- Small sanitized tests and fixtures.
|
||||
- Documentation and reproducible build instructions.
|
||||
- Small convenience binaries when the team explicitly wants a ready-to-run Windows artifact.
|
||||
|
||||
## What Does Not Belong In Git
|
||||
|
||||
- Raw GF-3 production archives.
|
||||
- Extracted scenes.
|
||||
- SARscape output directories.
|
||||
- Local wrapper config files such as `gf3wrapper.json`.
|
||||
- Runtime folders such as `.gf3_extract` and `.gf3_runtime`.
|
||||
- DEM files and other large environment-specific dependencies.
|
||||
|
||||
Use NAS or object storage for production data and record exact paths in run logs or issue notes.
|
||||
@@ -0,0 +1,142 @@
|
||||
# SARscape Go Wrapper
|
||||
|
||||
`gf3wrapper.exe` is a Windows command-line wrapper for the SARscape/IDL GF-3 processing workflow. It embeds `gf3_sarscape_cli.sav`, starts `idlrt.exe`, and passes scene metadata, output directory, DEM, and polarization arguments to SARscape.
|
||||
|
||||
This path is intended for operational runs where ENVI/IDL Runtime and SARscape are installed and licensed.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Windows host.
|
||||
- ENVI/IDL Runtime, default path:
|
||||
`C:\Program Files\Harris\ENVI56\IDL88\bin\bin.x86_64\idlrt.exe`
|
||||
- SARscape available to the IDL Runtime environment.
|
||||
- DEM file, for example:
|
||||
`D:\DEM\COPDEM_GLO30_China_4326_DEM`
|
||||
- GF-3 L1A input as `*.tar.gz`, `*.meta.xml`, or a directory containing multiple scenes.
|
||||
|
||||
The IDL Runtime path can be overridden with `-idlrt` or the `IDLRT_PATH` environment variable.
|
||||
|
||||
## Quick Start
|
||||
|
||||
Use the prebuilt executable:
|
||||
|
||||
```powershell
|
||||
.\dist\windows\gf3wrapper.exe `
|
||||
-input "D:\GF3\L1A_BATCH" `
|
||||
-output "E:\GF3\L2_SARscape" `
|
||||
-dem "D:\DEM\COPDEM_GLO30_China_4326_DEM" `
|
||||
-pol "HH,HV"
|
||||
```
|
||||
|
||||
Double-click mode is also supported. When launched without arguments, the wrapper prompts for input, output, DEM, polarizations, and IDL Runtime path. Press Enter to reuse saved defaults.
|
||||
|
||||
## Inputs
|
||||
|
||||
The wrapper supports:
|
||||
|
||||
- A single GF-3 archive: `GF3_....tar.gz`
|
||||
- A single extracted scene metadata file: `GF3_....meta.xml`
|
||||
- A directory containing many archives and/or extracted scenes
|
||||
|
||||
Directory mode recursively scans for `*.tar.gz`, `*.tgz`, and `*.meta.xml`. If both archive and metadata exist for the same scene name, the archive is preferred. Internal `.gf3_extract` folders are skipped.
|
||||
|
||||
## Batch Failure Handling
|
||||
|
||||
Directory runs are fault-tolerant at the scene level. If a single scene cannot be extracted or processed, the wrapper records that scene as failed, prints the input path and error reason, and continues with the next scene.
|
||||
|
||||
Archive read errors include the archive path, archive size when available, and the archive member being read when the failure happened. For example, a truncated `.tar.gz` reports that the archive appears incomplete and should be re-copied or re-downloaded.
|
||||
|
||||
At the end of the run, the wrapper prints a summary:
|
||||
|
||||
```text
|
||||
Batch summary: 3 succeeded, 1 failed, 4 total
|
||||
Failed scenes:
|
||||
- GF3_... (<archive path>): <failure reason>
|
||||
```
|
||||
|
||||
If any scene failed, the process exits with a non-zero status after the summary. This lets scheduled jobs detect that the batch needs attention while still preserving successful outputs from other scenes.
|
||||
|
||||
## Resume Behavior
|
||||
|
||||
Before extracting or processing a scene, the wrapper checks the scene output directory for completed geocoded SARscape products for every requested polarization. A product is considered complete when both the final `*_geo.sml` metadata file and its matching data file exist and are non-empty.
|
||||
|
||||
Already completed scenes are skipped:
|
||||
|
||||
```text
|
||||
SKIPPED GF3_...
|
||||
Input D:\GF3\L1A_BATCH\GF3_....tar.gz
|
||||
Output already complete E:\GF3\L2_SARscape\GF3_...
|
||||
```
|
||||
|
||||
This is intended for interrupted or partially failed batch runs. For example, after replacing a truncated archive in the original input directory, rerun the same command with the same output directory; scenes that already have complete HH/HV geocoded outputs are skipped, and only missing or incomplete scenes are processed.
|
||||
|
||||
## Outputs
|
||||
|
||||
Each scene is processed into its own output directory:
|
||||
|
||||
```text
|
||||
E:\GF3\L2_SARscape\
|
||||
GF3_SCENE_NAME\
|
||||
.gf3_extract\ Extracted archive, when input was .tar.gz
|
||||
*_slc.sml
|
||||
*_ml.sml
|
||||
*_filt.sml
|
||||
*_geo.sml
|
||||
```
|
||||
|
||||
The embedded SAV is materialized under:
|
||||
|
||||
```text
|
||||
<output>\.gf3_runtime\gf3_sarscape_cli.sav
|
||||
```
|
||||
|
||||
## Saved Defaults
|
||||
|
||||
After a successful run, the wrapper writes `gf3wrapper.json` next to the executable:
|
||||
|
||||
```json
|
||||
{
|
||||
"idlrt_path": "C:\\Program Files\\Harris\\ENVI56\\IDL88\\bin\\bin.x86_64\\idlrt.exe",
|
||||
"dem_file": "D:\\DEM\\COPDEM_GLO30_China_4326_DEM",
|
||||
"polarizations": "HH,HV",
|
||||
"last_input": "D:\\GF3\\L1A_BATCH",
|
||||
"last_output": "E:\\GF3\\L2_SARscape",
|
||||
"keep_extracted": true
|
||||
}
|
||||
```
|
||||
|
||||
Command-line flags override saved defaults. If `-input`, `-output`, `-dem`, or `-pol` are omitted, saved values are reused when available.
|
||||
|
||||
## Build
|
||||
|
||||
From the repository root:
|
||||
|
||||
```powershell
|
||||
cd .\tools\gf3_sarscape_wrapper
|
||||
$env:GOTELEMETRY = "off"
|
||||
go build -o ..\..\dist\windows\gf3wrapper.exe .
|
||||
```
|
||||
|
||||
If Go cannot write to the default user cache directory, set local cache paths:
|
||||
|
||||
```powershell
|
||||
$env:GOCACHE = "C:\Temp\go-build-cache"
|
||||
$env:APPDATA = "C:\Temp\go-appdata"
|
||||
```
|
||||
|
||||
## Source Files
|
||||
|
||||
```text
|
||||
tools/gf3_sarscape_wrapper/main.go
|
||||
tools/gf3_sarscape_wrapper/assets/gf3_sarscape_cli.sav
|
||||
tools/gf3_sarscape_wrapper/idl/gf3_sarscape_cli.pro
|
||||
tools/gf3_sarscape_wrapper/idl/gf3_sarscape_pipeline.pro
|
||||
```
|
||||
|
||||
`assets/gf3_sarscape_cli.sav` is embedded into the Go executable at build time.
|
||||
|
||||
## Notes
|
||||
|
||||
- Keep production archives and generated products outside Git.
|
||||
- Keep only small sanitized fixtures in `tests/fixtures`.
|
||||
- The prebuilt executable in `dist/windows` is a convenience artifact for Windows users. Rebuild it after changing `main.go` or the embedded `.sav`.
|
||||
@@ -0,0 +1,10 @@
|
||||
name: gf3-l1a2l2
|
||||
channels:
|
||||
- conda-forge
|
||||
dependencies:
|
||||
- python=3.11
|
||||
- gdal
|
||||
- rasterio
|
||||
- numpy
|
||||
- tqdm
|
||||
- pip
|
||||
@@ -0,0 +1,16 @@
|
||||
"""Compatibility wrapper for the packaged GF3 L1A to L2 pipeline."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
_SRC = Path(__file__).resolve().parent / "src"
|
||||
if _SRC.is_dir():
|
||||
sys.path.insert(0, str(_SRC))
|
||||
|
||||
from gf3_l1a2l2.cli import main
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,24 @@
|
||||
[build-system]
|
||||
requires = ["setuptools>=68", "wheel"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "gf3-l1a2l2-pipeline"
|
||||
version = "0.1.0"
|
||||
description = "GF3 L1A to L2 processing pipeline"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10"
|
||||
dependencies = [
|
||||
"numpy>=1.22",
|
||||
"rasterio>=1.3",
|
||||
"tqdm>=4.64",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
gf3-l1a2l2 = "gf3_l1a2l2.cli:main"
|
||||
|
||||
[tool.setuptools]
|
||||
package-dir = {"" = "src"}
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src"]
|
||||
@@ -0,0 +1,16 @@
|
||||
"""GF3 L1A to L2 processing pipeline."""
|
||||
|
||||
__all__ = ["PipelineConfig", "run_pipeline"]
|
||||
__version__ = "0.1.0"
|
||||
|
||||
|
||||
def __getattr__(name: str):
|
||||
if name == "PipelineConfig":
|
||||
from gf3_l1a2l2.config import PipelineConfig
|
||||
|
||||
return PipelineConfig
|
||||
if name == "run_pipeline":
|
||||
from gf3_l1a2l2.pipeline import run_pipeline
|
||||
|
||||
return run_pipeline
|
||||
raise AttributeError(name)
|
||||
@@ -0,0 +1,5 @@
|
||||
from gf3_l1a2l2.cli import main
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,88 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import math
|
||||
from pathlib import Path
|
||||
from typing import Iterator, Optional
|
||||
|
||||
import numpy as np
|
||||
import rasterio
|
||||
from rasterio.windows import Window
|
||||
|
||||
from gf3_l1a2l2.errors import ProcessingError
|
||||
from gf3_l1a2l2.models import CalibrationValue
|
||||
from gf3_l1a2l2.paths import output_name
|
||||
|
||||
|
||||
def radiometric_calibrate_l1a_to_l1b(
|
||||
image_path: Path | str,
|
||||
output_dir: Path | str,
|
||||
calibration: CalibrationValue,
|
||||
*,
|
||||
block_size: int,
|
||||
overwrite: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
) -> Path:
|
||||
source = Path(image_path)
|
||||
out_dir = Path(output_dir)
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
l1b_path = out_dir / output_name(source, "L1A", "L1B")
|
||||
|
||||
if l1b_path.exists() and not overwrite:
|
||||
if logger:
|
||||
logger.info("Skip existing L1B: %s", l1b_path)
|
||||
return l1b_path
|
||||
|
||||
if not math.isfinite(calibration.qualify_value):
|
||||
raise ProcessingError(f"Invalid QualifyValue for {source.name}: {calibration.qualify_value}")
|
||||
if not math.isfinite(calibration.calibration_const):
|
||||
raise ProcessingError(
|
||||
f"Invalid CalibrationConst for {source.name}: {calibration.calibration_const}"
|
||||
)
|
||||
|
||||
if logger:
|
||||
logger.info("Calibrating %s -> %s", source, l1b_path)
|
||||
|
||||
try:
|
||||
with rasterio.open(source) as src:
|
||||
if src.count < 2:
|
||||
raise ProcessingError(f"Expected at least 2 bands in L1A image: {source}")
|
||||
|
||||
profile = src.profile.copy()
|
||||
profile.update(
|
||||
driver="GTiff",
|
||||
count=1,
|
||||
dtype="float32",
|
||||
compress="lzw",
|
||||
BIGTIFF="YES",
|
||||
tiled=True,
|
||||
)
|
||||
|
||||
with rasterio.open(l1b_path, "w", **profile) as dst:
|
||||
for window in _iter_windows(src.width, src.height, block_size):
|
||||
real = src.read(1, window=window, out_dtype="float32")
|
||||
imag = src.read(2, window=window, out_dtype="float32")
|
||||
db = _calibrate_window(real, imag, calibration)
|
||||
dst.write(db, 1, window=window)
|
||||
except ProcessingError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise ProcessingError(f"Failed to calibrate {source}") from exc
|
||||
|
||||
return l1b_path
|
||||
|
||||
|
||||
def _calibrate_window(real: np.ndarray, imag: np.ndarray, calibration: CalibrationValue) -> np.ndarray:
|
||||
amplitude = np.hypot(real, imag)
|
||||
scaled = (amplitude / np.float32(32767.0)) * np.float32(calibration.qualify_value)
|
||||
with np.errstate(divide="ignore", invalid="ignore"):
|
||||
db = np.float32(20.0) * np.log10(scaled) - np.float32(calibration.calibration_const)
|
||||
return db.astype("float32", copy=False)
|
||||
|
||||
|
||||
def _iter_windows(width: int, height: int, block_size: int) -> Iterator[Window]:
|
||||
for row_off in range(0, height, block_size):
|
||||
window_height = min(block_size, height - row_off)
|
||||
for col_off in range(0, width, block_size):
|
||||
window_width = min(block_size, width - col_off)
|
||||
yield Window(col_off, row_off, window_width, window_height)
|
||||
@@ -0,0 +1,107 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from gf3_l1a2l2.config import PipelineConfig
|
||||
from gf3_l1a2l2.decompress import extract_compressed_files
|
||||
from gf3_l1a2l2.discovery import discover_scenes
|
||||
from gf3_l1a2l2.errors import GF3PipelineError
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = build_parser()
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
if not hasattr(args, "handler"):
|
||||
parser.print_help()
|
||||
return 2
|
||||
|
||||
try:
|
||||
return args.handler(args)
|
||||
except GF3PipelineError as exc:
|
||||
print(f"ERROR: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(prog="gf3-l1a2l2", description="GF3 L1A to L2 pipeline")
|
||||
subparsers = parser.add_subparsers(dest="command")
|
||||
|
||||
run_parser = subparsers.add_parser("run", help="extract and process GF3 L1A products")
|
||||
run_parser.add_argument("--input", required=True, type=Path, help="input folder containing archives or scenes")
|
||||
run_parser.add_argument("--output", required=True, type=Path, help="output folder")
|
||||
run_parser.add_argument("--dem", type=Path, default=None, help="DEM file for RPC orthorectification")
|
||||
run_parser.add_argument("--x-res", type=float, default=0.0002, help="output x resolution")
|
||||
run_parser.add_argument("--y-res", type=float, default=0.0002, help="output y resolution")
|
||||
run_parser.add_argument("--dst-srs", default="EPSG:4326", help="destination spatial reference")
|
||||
run_parser.add_argument("--block-size", type=int, default=2048, help="calibration block size in pixels")
|
||||
run_parser.add_argument("--workers", type=int, default=1, help="number of scenes to process concurrently")
|
||||
run_parser.add_argument("--skip-decompress", action="store_true", help="do not extract archives before processing")
|
||||
run_parser.add_argument("--overwrite", action="store_true", help="overwrite existing extraction and output files")
|
||||
run_parser.add_argument("--flat-output", action="store_true", help="write all outputs directly to output folder")
|
||||
run_parser.add_argument("--fail-fast", action="store_true", help="stop on the first failed scene or polarization")
|
||||
run_parser.add_argument("--log-level", default="INFO", choices=["DEBUG", "INFO", "WARNING", "ERROR"])
|
||||
run_parser.set_defaults(handler=_run)
|
||||
|
||||
decompress_parser = subparsers.add_parser("decompress", help="extract .tar.gz and .zip archives")
|
||||
decompress_parser.add_argument("input", type=Path, help="folder containing archives")
|
||||
decompress_parser.add_argument("--overwrite", action="store_true", help="overwrite existing extracted folders")
|
||||
decompress_parser.set_defaults(handler=_decompress)
|
||||
|
||||
discover_parser = subparsers.add_parser("discover", help="list discovered GF3 scenes")
|
||||
discover_parser.add_argument("input", type=Path, help="input folder")
|
||||
discover_parser.set_defaults(handler=_discover)
|
||||
|
||||
return parser
|
||||
|
||||
|
||||
def _run(args: argparse.Namespace) -> int:
|
||||
from gf3_l1a2l2.pipeline import run_pipeline
|
||||
|
||||
dem_path = args.dem or _default_dem_path()
|
||||
config = PipelineConfig(
|
||||
input_dir=args.input,
|
||||
output_dir=args.output,
|
||||
dem_path=dem_path,
|
||||
x_res=args.x_res,
|
||||
y_res=args.y_res,
|
||||
dst_srs=args.dst_srs,
|
||||
block_size=args.block_size,
|
||||
workers=args.workers,
|
||||
skip_decompress=args.skip_decompress,
|
||||
overwrite=args.overwrite,
|
||||
flat_output=args.flat_output,
|
||||
fail_fast=args.fail_fast,
|
||||
log_level=args.log_level,
|
||||
)
|
||||
summary = run_pipeline(config)
|
||||
return 1 if summary.failed else 0
|
||||
|
||||
|
||||
def _decompress(args: argparse.Namespace) -> int:
|
||||
results = extract_compressed_files(args.input, overwrite=args.overwrite)
|
||||
failed = [item for item in results if item.status == "failed"]
|
||||
for item in results:
|
||||
print(f"{item.status}\t{item.archive}\t{item.output_dir}\t{item.message}")
|
||||
return 1 if failed else 0
|
||||
|
||||
|
||||
def _discover(args: argparse.Namespace) -> int:
|
||||
scenes = discover_scenes(args.input)
|
||||
for scene in scenes:
|
||||
polarizations = ",".join(scene.available_polarizations)
|
||||
print(f"{scene.root}\tmetadata={scene.metadata_xml.name}\tpol={polarizations}")
|
||||
return 0
|
||||
|
||||
|
||||
def _default_dem_path() -> Path | None:
|
||||
candidates = [
|
||||
Path.cwd() / "GMTED2010.jp2",
|
||||
Path(__file__).resolve().parents[2] / "GMTED2010.jp2",
|
||||
]
|
||||
for candidate in candidates:
|
||||
if candidate.is_file():
|
||||
return candidate
|
||||
return None
|
||||
@@ -0,0 +1,41 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, replace
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from gf3_l1a2l2.constants import (
|
||||
DEFAULT_BLOCK_SIZE,
|
||||
DEFAULT_DST_SRS,
|
||||
DEFAULT_X_RES,
|
||||
DEFAULT_Y_RES,
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PipelineConfig:
|
||||
input_dir: Path
|
||||
output_dir: Path
|
||||
dem_path: Optional[Path] = None
|
||||
x_res: float = DEFAULT_X_RES
|
||||
y_res: float = DEFAULT_Y_RES
|
||||
dst_srs: str = DEFAULT_DST_SRS
|
||||
block_size: int = DEFAULT_BLOCK_SIZE
|
||||
workers: int = 1
|
||||
skip_decompress: bool = False
|
||||
overwrite: bool = False
|
||||
flat_output: bool = False
|
||||
keep_l1b: bool = True
|
||||
fail_fast: bool = False
|
||||
log_level: str = "INFO"
|
||||
|
||||
def resolved(self) -> "PipelineConfig":
|
||||
dem_path = self.dem_path.resolve() if self.dem_path else None
|
||||
return replace(
|
||||
self,
|
||||
input_dir=self.input_dir.resolve(),
|
||||
output_dir=self.output_dir.resolve(),
|
||||
dem_path=dem_path,
|
||||
workers=max(1, int(self.workers)),
|
||||
block_size=max(128, int(self.block_size)),
|
||||
)
|
||||
@@ -0,0 +1,6 @@
|
||||
POLARIZATIONS = ("HH", "HV", "VH", "VV")
|
||||
|
||||
DEFAULT_X_RES = 0.0002
|
||||
DEFAULT_Y_RES = 0.0002
|
||||
DEFAULT_DST_SRS = "EPSG:4326"
|
||||
DEFAULT_BLOCK_SIZE = 2048
|
||||
@@ -0,0 +1,112 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import shutil
|
||||
import tarfile
|
||||
import zipfile
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Iterable, Optional
|
||||
|
||||
try:
|
||||
from tqdm import tqdm
|
||||
except Exception: # pragma: no cover - only used when tqdm is unavailable
|
||||
def tqdm(iterable: Iterable, **_: object) -> Iterable:
|
||||
return iterable
|
||||
|
||||
from gf3_l1a2l2.errors import ProcessingError
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ExtractionResult:
|
||||
archive: Path
|
||||
output_dir: Path
|
||||
status: str
|
||||
message: str = ""
|
||||
|
||||
|
||||
def extract_compressed_files(
|
||||
folder_path: Path | str,
|
||||
*,
|
||||
overwrite: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
) -> list[ExtractionResult]:
|
||||
folder = Path(folder_path).resolve()
|
||||
if not folder.is_dir():
|
||||
raise ProcessingError(f"Input folder does not exist: {folder}")
|
||||
|
||||
archives = sorted(
|
||||
path
|
||||
for path in folder.iterdir()
|
||||
if path.is_file()
|
||||
and (path.name.lower().endswith(".tar.gz") or path.suffix.lower() == ".zip")
|
||||
)
|
||||
|
||||
results: list[ExtractionResult] = []
|
||||
if not archives:
|
||||
if logger:
|
||||
logger.info("No .tar.gz or .zip archives found in %s", folder)
|
||||
return results
|
||||
|
||||
for archive in tqdm(archives, desc="Extract archives", unit="file"):
|
||||
output_dir = _archive_output_dir(archive)
|
||||
try:
|
||||
if output_dir.exists():
|
||||
if not overwrite:
|
||||
results.append(ExtractionResult(archive, output_dir, "skipped", "output exists"))
|
||||
if logger:
|
||||
logger.info("Skip existing extraction: %s", output_dir)
|
||||
continue
|
||||
shutil.rmtree(output_dir)
|
||||
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
if archive.name.lower().endswith(".tar.gz"):
|
||||
_extract_tar_gz(archive, output_dir)
|
||||
elif archive.suffix.lower() == ".zip":
|
||||
_extract_zip(archive, output_dir)
|
||||
else:
|
||||
continue
|
||||
|
||||
results.append(ExtractionResult(archive, output_dir, "done"))
|
||||
if logger:
|
||||
logger.info("Extracted %s -> %s", archive, output_dir)
|
||||
except Exception as exc:
|
||||
results.append(ExtractionResult(archive, output_dir, "failed", str(exc)))
|
||||
if logger:
|
||||
logger.exception("Failed to extract %s", archive)
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def _archive_output_dir(archive: Path) -> Path:
|
||||
if archive.name.lower().endswith(".tar.gz"):
|
||||
return archive.with_name(archive.name[:-7])
|
||||
return archive.with_suffix("")
|
||||
|
||||
|
||||
def _extract_tar_gz(archive: Path, output_dir: Path) -> None:
|
||||
base = output_dir.resolve()
|
||||
with tarfile.open(archive, "r:gz") as tar:
|
||||
members = tar.getmembers()
|
||||
for member in tqdm(members, desc=f"Extract {archive.name}", leave=False, unit="file"):
|
||||
if member.issym() or member.islnk():
|
||||
raise ProcessingError(f"Refusing to extract link from archive: {member.name}")
|
||||
_ensure_safe_target(base, member.name)
|
||||
tar.extract(member, path=base)
|
||||
|
||||
|
||||
def _extract_zip(archive: Path, output_dir: Path) -> None:
|
||||
base = output_dir.resolve()
|
||||
with zipfile.ZipFile(archive, "r") as zip_ref:
|
||||
names = zip_ref.namelist()
|
||||
for name in tqdm(names, desc=f"Extract {archive.name}", leave=False, unit="file"):
|
||||
_ensure_safe_target(base, name)
|
||||
zip_ref.extract(name, base)
|
||||
|
||||
|
||||
def _ensure_safe_target(base: Path, member_name: str) -> None:
|
||||
target = (base / member_name).resolve()
|
||||
if target == base:
|
||||
return
|
||||
if base not in target.parents:
|
||||
raise ProcessingError(f"Unsafe archive path: {member_name}")
|
||||
@@ -0,0 +1,90 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Iterable, Optional
|
||||
|
||||
from gf3_l1a2l2.errors import DiscoveryError
|
||||
from gf3_l1a2l2.models import SceneFiles
|
||||
from gf3_l1a2l2.paths import detect_polarization
|
||||
|
||||
SKIP_DIR_NAMES = {".git", ".venv", ".tmp_tests", "__pycache__", ".pytest_cache", "logs"}
|
||||
|
||||
|
||||
def discover_scenes(input_dir: Path | str, logger: Optional[logging.Logger] = None) -> list[SceneFiles]:
|
||||
root_dir = Path(input_dir).resolve()
|
||||
if not root_dir.is_dir():
|
||||
raise DiscoveryError(f"Input directory does not exist: {root_dir}")
|
||||
|
||||
scenes: list[SceneFiles] = []
|
||||
for directory in _walk_directories(root_dir):
|
||||
try:
|
||||
files = [path for path in directory.iterdir() if path.is_file()]
|
||||
except OSError as exc:
|
||||
if logger:
|
||||
logger.warning("Skipping unreadable directory %s: %s", directory, exc)
|
||||
continue
|
||||
|
||||
metadata_files = sorted(path for path in files if path.name.lower().endswith(".meta.xml"))
|
||||
image_files = sorted(
|
||||
path
|
||||
for path in files
|
||||
if path.suffix.lower() in (".tif", ".tiff") and detect_polarization(path)
|
||||
)
|
||||
|
||||
if not metadata_files or not image_files:
|
||||
continue
|
||||
|
||||
metadata_xml = metadata_files[0]
|
||||
if len(metadata_files) > 1 and logger:
|
||||
logger.warning("Multiple metadata files in %s; using %s", directory, metadata_xml.name)
|
||||
|
||||
images = _collect_by_polarization(image_files, logger)
|
||||
rpcs = _collect_by_polarization(
|
||||
sorted(path for path in files if path.suffix.lower() in (".rpc", ".rpb")),
|
||||
logger,
|
||||
)
|
||||
|
||||
scenes.append(SceneFiles(root=directory, metadata_xml=metadata_xml, images=images, rpcs=rpcs))
|
||||
|
||||
scenes.sort(key=lambda scene: str(scene.root).lower())
|
||||
if logger:
|
||||
logger.info("Discovered %s scene(s) under %s", len(scenes), root_dir)
|
||||
return scenes
|
||||
|
||||
|
||||
def _walk_directories(root_dir: Path) -> Iterable[Path]:
|
||||
stack = [root_dir]
|
||||
while stack:
|
||||
directory = stack.pop()
|
||||
if directory.name in SKIP_DIR_NAMES:
|
||||
continue
|
||||
yield directory
|
||||
try:
|
||||
children = list(directory.iterdir())
|
||||
except OSError:
|
||||
continue
|
||||
stack.extend(path for path in children if path.is_dir() and path.name not in SKIP_DIR_NAMES)
|
||||
|
||||
|
||||
def _collect_by_polarization(
|
||||
files: Iterable[Path],
|
||||
logger: Optional[logging.Logger] = None,
|
||||
) -> dict[str, Path]:
|
||||
collected: dict[str, Path] = {}
|
||||
for path in files:
|
||||
polarization = detect_polarization(path)
|
||||
if not polarization:
|
||||
continue
|
||||
if polarization in collected:
|
||||
if logger:
|
||||
logger.warning(
|
||||
"Duplicate %s file in %s; using %s and ignoring %s",
|
||||
polarization,
|
||||
path.parent,
|
||||
collected[polarization].name,
|
||||
path.name,
|
||||
)
|
||||
continue
|
||||
collected[polarization] = path
|
||||
return collected
|
||||
@@ -0,0 +1,18 @@
|
||||
class GF3PipelineError(Exception):
|
||||
"""Base error for pipeline failures."""
|
||||
|
||||
|
||||
class DiscoveryError(GF3PipelineError):
|
||||
"""Raised when input products cannot be discovered."""
|
||||
|
||||
|
||||
class MetadataError(GF3PipelineError):
|
||||
"""Raised when GF3 metadata cannot be parsed."""
|
||||
|
||||
|
||||
class RpcError(GF3PipelineError):
|
||||
"""Raised when RPC metadata cannot be parsed."""
|
||||
|
||||
|
||||
class ProcessingError(GF3PipelineError):
|
||||
"""Raised when image processing fails."""
|
||||
@@ -0,0 +1,78 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from gf3_l1a2l2.errors import ProcessingError
|
||||
from gf3_l1a2l2.paths import output_name
|
||||
from gf3_l1a2l2.rpc import read_rpc_metadata
|
||||
|
||||
|
||||
def geometric_correction(
|
||||
l1b_path: Path | str,
|
||||
rpc_path: Path | str,
|
||||
output_dir: Path | str,
|
||||
*,
|
||||
dem_path: Optional[Path],
|
||||
x_res: float,
|
||||
y_res: float,
|
||||
dst_srs: str,
|
||||
overwrite: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
) -> Path:
|
||||
source = Path(l1b_path)
|
||||
out_dir = Path(output_dir)
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
l2_path = out_dir / output_name(source, "L1B", "L2")
|
||||
|
||||
if l2_path.exists() and not overwrite:
|
||||
if logger:
|
||||
logger.info("Skip existing L2: %s", l2_path)
|
||||
return l2_path
|
||||
|
||||
if dem_path and not Path(dem_path).is_file():
|
||||
raise ProcessingError(f"DEM file does not exist: {dem_path}")
|
||||
|
||||
rpc_metadata = read_rpc_metadata(rpc_path)
|
||||
|
||||
try:
|
||||
from osgeo import gdal
|
||||
except Exception as exc:
|
||||
raise ProcessingError("Cannot import osgeo.gdal. Install GDAL Python bindings.") from exc
|
||||
|
||||
if logger:
|
||||
logger.info("Geocorrecting %s -> %s", source, l2_path)
|
||||
|
||||
dataset = gdal.Open(str(source), gdal.GA_Update)
|
||||
if dataset is None:
|
||||
raise ProcessingError(f"GDAL cannot open L1B image: {source}")
|
||||
|
||||
try:
|
||||
dataset.SetMetadata(rpc_metadata, "RPC")
|
||||
transformer_options = []
|
||||
if dem_path:
|
||||
transformer_options.append(f"RPC_DEM={Path(dem_path)}")
|
||||
|
||||
warp_options = gdal.WarpOptions(
|
||||
dstSRS=dst_srs,
|
||||
xRes=x_res,
|
||||
yRes=y_res,
|
||||
rpc=True,
|
||||
multithread=True,
|
||||
transformerOptions=transformer_options,
|
||||
creationOptions=["COMPRESS=LZW", "TILED=YES", "BIGTIFF=YES"],
|
||||
)
|
||||
result = gdal.Warp(str(l2_path), dataset, options=warp_options)
|
||||
if result is None:
|
||||
raise ProcessingError(f"GDAL Warp failed for {source}")
|
||||
result.FlushCache()
|
||||
result = None
|
||||
except ProcessingError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise ProcessingError(f"Failed to geocorrect {source}") from exc
|
||||
finally:
|
||||
dataset = None
|
||||
|
||||
return l2_path
|
||||
@@ -0,0 +1,34 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def configure_logging(output_dir: Path, level: str = "INFO") -> logging.Logger:
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
log_dir = output_dir / "logs"
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
logger = logging.getLogger("gf3_l1a2l2")
|
||||
logger.setLevel(getattr(logging, level.upper(), logging.INFO))
|
||||
logger.propagate = False
|
||||
logger.handlers.clear()
|
||||
|
||||
formatter = logging.Formatter(
|
||||
fmt="%(asctime)s %(levelname)s %(message)s",
|
||||
datefmt="%Y-%m-%d %H:%M:%S",
|
||||
)
|
||||
|
||||
console = logging.StreamHandler()
|
||||
console.setFormatter(formatter)
|
||||
console.setLevel(getattr(logging, level.upper(), logging.INFO))
|
||||
logger.addHandler(console)
|
||||
|
||||
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
file_handler = logging.FileHandler(log_dir / f"gf3_l1a2l2_{timestamp}.log", encoding="utf-8")
|
||||
file_handler.setFormatter(formatter)
|
||||
file_handler.setLevel(logging.DEBUG)
|
||||
logger.addHandler(file_handler)
|
||||
|
||||
return logger
|
||||
@@ -0,0 +1,145 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import xml.etree.ElementTree as ET
|
||||
from pathlib import Path
|
||||
from typing import Iterable, Optional
|
||||
|
||||
from gf3_l1a2l2.constants import POLARIZATIONS
|
||||
from gf3_l1a2l2.errors import MetadataError
|
||||
from gf3_l1a2l2.models import CalibrationMetadata, CalibrationValue
|
||||
from gf3_l1a2l2.paths import detect_polarization
|
||||
|
||||
|
||||
QUALIFY_KEYWORDS = ("qualifyvalue", "qualityvalue")
|
||||
CALIBRATION_KEYWORDS = ("calibrationconst", "calibrationconstant")
|
||||
|
||||
|
||||
def read_calibration_metadata(xml_path: Path | str) -> CalibrationMetadata:
|
||||
path = Path(xml_path)
|
||||
try:
|
||||
root = ET.parse(path).getroot()
|
||||
except ET.ParseError as exc:
|
||||
raise MetadataError(f"Invalid metadata XML: {path}") from exc
|
||||
except OSError as exc:
|
||||
raise MetadataError(f"Cannot read metadata XML: {path}") from exc
|
||||
|
||||
qualify_values = _find_values(root, QUALIFY_KEYWORDS) or _legacy_values(root, (17, 13))
|
||||
calibration_values = _find_values(root, CALIBRATION_KEYWORDS) or _legacy_values(root, (18, 3))
|
||||
|
||||
if not qualify_values:
|
||||
raise MetadataError(f"Cannot find QualifyValue values in {path}")
|
||||
if not calibration_values:
|
||||
raise MetadataError(f"Cannot find CalibrationConst values in {path}")
|
||||
|
||||
values: dict[str, CalibrationValue] = {}
|
||||
for polarization in POLARIZATIONS:
|
||||
if polarization not in qualify_values or polarization not in calibration_values:
|
||||
continue
|
||||
values[polarization] = CalibrationValue(
|
||||
qualify_value=qualify_values[polarization],
|
||||
calibration_const=calibration_values[polarization],
|
||||
)
|
||||
|
||||
if not values:
|
||||
raise MetadataError(f"No usable calibration values found in {path}")
|
||||
|
||||
return CalibrationMetadata(values=values)
|
||||
|
||||
|
||||
def _find_values(root: ET.Element, keywords: Iterable[str]) -> Optional[dict[str, float]]:
|
||||
lowered = tuple(keyword.lower() for keyword in keywords)
|
||||
for element in root.iter():
|
||||
name = _local_name(element.tag).lower()
|
||||
if not any(keyword in name for keyword in lowered):
|
||||
continue
|
||||
|
||||
values = _values_by_child_polarization(element)
|
||||
if values:
|
||||
return values
|
||||
|
||||
values = _values_by_child_order(element)
|
||||
if values:
|
||||
return values
|
||||
|
||||
values = _values_from_text(element.text)
|
||||
if values:
|
||||
return values
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def _legacy_values(root: ET.Element, indexes: tuple[int, int]) -> Optional[dict[str, float]]:
|
||||
try:
|
||||
node = root
|
||||
for index in indexes:
|
||||
node = list(node)[index]
|
||||
values = [_parse_float(child.text) for child in list(node)[:4]]
|
||||
except (IndexError, TypeError, ValueError):
|
||||
return None
|
||||
|
||||
if len(values) < 4:
|
||||
return None
|
||||
return dict(zip(POLARIZATIONS, values))
|
||||
|
||||
|
||||
def _values_by_child_polarization(element: ET.Element) -> Optional[dict[str, float]]:
|
||||
values: dict[str, float] = {}
|
||||
for child in list(element):
|
||||
polarization = _element_polarization(child)
|
||||
if not polarization:
|
||||
continue
|
||||
try:
|
||||
values[polarization] = _parse_float(child.text)
|
||||
except ValueError:
|
||||
continue
|
||||
return values if values else None
|
||||
|
||||
|
||||
def _values_by_child_order(element: ET.Element) -> Optional[dict[str, float]]:
|
||||
numeric_values: list[float] = []
|
||||
for child in list(element):
|
||||
try:
|
||||
numeric_values.append(_parse_float(child.text))
|
||||
except ValueError:
|
||||
continue
|
||||
if len(numeric_values) < 4:
|
||||
return None
|
||||
return dict(zip(POLARIZATIONS, numeric_values[:4]))
|
||||
|
||||
|
||||
def _values_from_text(text: Optional[str]) -> Optional[dict[str, float]]:
|
||||
if not text:
|
||||
return None
|
||||
normalized = text.replace(",", " ").replace(";", " ")
|
||||
parts = [part for part in normalized.split() if part]
|
||||
if len(parts) < 4:
|
||||
return None
|
||||
try:
|
||||
values = [_parse_float(part) for part in parts[:4]]
|
||||
except ValueError:
|
||||
return None
|
||||
return dict(zip(POLARIZATIONS, values))
|
||||
|
||||
|
||||
def _element_polarization(element: ET.Element) -> Optional[str]:
|
||||
for value in element.attrib.values():
|
||||
polarization = detect_polarization(value)
|
||||
if polarization:
|
||||
return polarization
|
||||
return detect_polarization(_local_name(element.tag))
|
||||
|
||||
|
||||
def _parse_float(text: Optional[str]) -> float:
|
||||
if text is None:
|
||||
raise ValueError("missing value")
|
||||
value = text.strip().strip("=")
|
||||
if not value or value.upper() == "NULL":
|
||||
return math.nan
|
||||
return float(value)
|
||||
|
||||
|
||||
def _local_name(tag: str) -> str:
|
||||
if "}" in tag:
|
||||
return tag.rsplit("}", 1)[1]
|
||||
return tag
|
||||
@@ -0,0 +1,66 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Mapping, Optional
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CalibrationValue:
|
||||
qualify_value: float
|
||||
calibration_const: float
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CalibrationMetadata:
|
||||
values: Mapping[str, CalibrationValue]
|
||||
|
||||
def for_polarization(self, polarization: str) -> CalibrationValue:
|
||||
return self.values[polarization]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SceneFiles:
|
||||
root: Path
|
||||
metadata_xml: Path
|
||||
images: Mapping[str, Path]
|
||||
rpcs: Mapping[str, Path]
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return self.root.name
|
||||
|
||||
@property
|
||||
def available_polarizations(self) -> tuple[str, ...]:
|
||||
return tuple(sorted(self.images))
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TaskResult:
|
||||
scene: str
|
||||
polarization: str
|
||||
status: str
|
||||
l1b_path: Optional[Path] = None
|
||||
l2_path: Optional[Path] = None
|
||||
message: str = ""
|
||||
|
||||
|
||||
@dataclass
|
||||
class PipelineSummary:
|
||||
results: list[TaskResult] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def succeeded(self) -> int:
|
||||
return sum(1 for item in self.results if item.status == "done")
|
||||
|
||||
@property
|
||||
def skipped(self) -> int:
|
||||
return sum(1 for item in self.results if item.status == "skipped")
|
||||
|
||||
@property
|
||||
def failed(self) -> int:
|
||||
return sum(1 for item in self.results if item.status == "failed")
|
||||
|
||||
@property
|
||||
def total(self) -> int:
|
||||
return len(self.results)
|
||||
@@ -0,0 +1,43 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from gf3_l1a2l2.constants import POLARIZATIONS
|
||||
|
||||
|
||||
def detect_polarization(path_or_name: Path | str) -> Optional[str]:
|
||||
name = Path(path_or_name).name.upper()
|
||||
for polarization in POLARIZATIONS:
|
||||
if polarization in name:
|
||||
return polarization
|
||||
return None
|
||||
|
||||
|
||||
def output_name(source: Path, source_level: str, target_level: str) -> str:
|
||||
name = source.name
|
||||
index = name.lower().find(source_level.lower())
|
||||
if index >= 0:
|
||||
return name[:index] + target_level + name[index + len(source_level) :]
|
||||
|
||||
stem = source.stem
|
||||
suffix = "".join(source.suffixes)
|
||||
if suffix and stem.endswith(suffix):
|
||||
stem = stem[: -len(suffix)]
|
||||
return f"{stem}_{target_level}{source.suffix}"
|
||||
|
||||
|
||||
def scene_output_dir(input_dir: Path, output_dir: Path, scene_dir: Path, flat: bool) -> Path:
|
||||
if flat:
|
||||
return output_dir
|
||||
|
||||
try:
|
||||
relative = scene_dir.relative_to(input_dir)
|
||||
except ValueError:
|
||||
relative = Path(scene_dir.name)
|
||||
|
||||
if str(relative) in ("", "."):
|
||||
relative = Path(scene_dir.name)
|
||||
|
||||
safe_name = "_".join(part for part in relative.parts if part not in ("", "."))
|
||||
return output_dir / safe_name
|
||||
@@ -0,0 +1,182 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import concurrent.futures
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Iterable
|
||||
|
||||
from gf3_l1a2l2.calibration import radiometric_calibrate_l1a_to_l1b
|
||||
from gf3_l1a2l2.config import PipelineConfig
|
||||
from gf3_l1a2l2.constants import POLARIZATIONS
|
||||
from gf3_l1a2l2.decompress import extract_compressed_files
|
||||
from gf3_l1a2l2.discovery import discover_scenes
|
||||
from gf3_l1a2l2.errors import GF3PipelineError
|
||||
from gf3_l1a2l2.geocorrection import geometric_correction
|
||||
from gf3_l1a2l2.log import configure_logging
|
||||
from gf3_l1a2l2.metadata import read_calibration_metadata
|
||||
from gf3_l1a2l2.models import PipelineSummary, SceneFiles, TaskResult
|
||||
from gf3_l1a2l2.paths import scene_output_dir
|
||||
|
||||
|
||||
def run_pipeline(config: PipelineConfig) -> PipelineSummary:
|
||||
config = config.resolved()
|
||||
logger = configure_logging(config.output_dir, config.log_level)
|
||||
_validate_config(config)
|
||||
|
||||
logger.info("Input: %s", config.input_dir)
|
||||
logger.info("Output: %s", config.output_dir)
|
||||
if config.dem_path:
|
||||
logger.info("DEM: %s", config.dem_path)
|
||||
|
||||
if not config.skip_decompress:
|
||||
extract_compressed_files(config.input_dir, overwrite=config.overwrite, logger=logger)
|
||||
|
||||
scenes = discover_scenes(config.input_dir, logger=logger)
|
||||
if not scenes:
|
||||
logger.warning("No GF3 scenes found under %s", config.input_dir)
|
||||
return PipelineSummary()
|
||||
|
||||
summary = PipelineSummary()
|
||||
for results in _process_scenes(scenes, config, logger):
|
||||
summary.results.extend(results)
|
||||
|
||||
logger.info(
|
||||
"Pipeline finished: total=%s done=%s skipped=%s failed=%s",
|
||||
summary.total,
|
||||
summary.succeeded,
|
||||
summary.skipped,
|
||||
summary.failed,
|
||||
)
|
||||
return summary
|
||||
|
||||
|
||||
def _validate_config(config: PipelineConfig) -> None:
|
||||
if not config.input_dir.is_dir():
|
||||
raise GF3PipelineError(f"Input directory does not exist: {config.input_dir}")
|
||||
if config.dem_path and not config.dem_path.is_file():
|
||||
raise GF3PipelineError(f"DEM file does not exist: {config.dem_path}")
|
||||
if config.x_res <= 0 or config.y_res <= 0:
|
||||
raise GF3PipelineError("x_res and y_res must be positive")
|
||||
|
||||
|
||||
def _process_scenes(
|
||||
scenes: list[SceneFiles],
|
||||
config: PipelineConfig,
|
||||
logger: logging.Logger,
|
||||
) -> Iterable[list[TaskResult]]:
|
||||
if config.workers == 1:
|
||||
for scene in scenes:
|
||||
yield _process_scene(scene, config, logger)
|
||||
return
|
||||
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=config.workers) as executor:
|
||||
future_map = {
|
||||
executor.submit(_process_scene, scene, config, logger): scene
|
||||
for scene in scenes
|
||||
}
|
||||
for future in concurrent.futures.as_completed(future_map):
|
||||
scene = future_map[future]
|
||||
try:
|
||||
yield future.result()
|
||||
except Exception as exc:
|
||||
logger.exception("Scene failed: %s", scene.root)
|
||||
if config.fail_fast:
|
||||
raise
|
||||
yield [
|
||||
TaskResult(
|
||||
scene=scene.name,
|
||||
polarization=polarization,
|
||||
status="failed",
|
||||
message=str(exc),
|
||||
)
|
||||
for polarization in scene.available_polarizations
|
||||
]
|
||||
|
||||
|
||||
def _process_scene(
|
||||
scene: SceneFiles,
|
||||
config: PipelineConfig,
|
||||
logger: logging.Logger,
|
||||
) -> list[TaskResult]:
|
||||
logger.info("Processing scene: %s", scene.root)
|
||||
out_dir = scene_output_dir(config.input_dir, config.output_dir, scene.root, config.flat_output)
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
try:
|
||||
metadata = read_calibration_metadata(scene.metadata_xml)
|
||||
except Exception as exc:
|
||||
logger.exception("Failed to parse metadata for %s", scene.root)
|
||||
if config.fail_fast:
|
||||
raise
|
||||
return [
|
||||
TaskResult(scene=scene.name, polarization=pol, status="failed", message=str(exc))
|
||||
for pol in scene.available_polarizations
|
||||
]
|
||||
|
||||
results: list[TaskResult] = []
|
||||
for polarization in POLARIZATIONS:
|
||||
image_path = scene.images.get(polarization)
|
||||
if not image_path:
|
||||
continue
|
||||
|
||||
rpc_path = scene.rpcs.get(polarization)
|
||||
if not rpc_path:
|
||||
message = f"Missing RPC/RPB file for {polarization}"
|
||||
logger.error("%s in %s", message, scene.root)
|
||||
results.append(TaskResult(scene=scene.name, polarization=polarization, status="failed", message=message))
|
||||
if config.fail_fast:
|
||||
raise GF3PipelineError(message)
|
||||
continue
|
||||
|
||||
if polarization not in metadata.values:
|
||||
message = f"Missing calibration metadata for {polarization}"
|
||||
logger.error("%s in %s", message, scene.metadata_xml)
|
||||
results.append(TaskResult(scene=scene.name, polarization=polarization, status="failed", message=message))
|
||||
if config.fail_fast:
|
||||
raise GF3PipelineError(message)
|
||||
continue
|
||||
|
||||
try:
|
||||
calibration = metadata.for_polarization(polarization)
|
||||
l1b_path = radiometric_calibrate_l1a_to_l1b(
|
||||
image_path,
|
||||
out_dir,
|
||||
calibration,
|
||||
block_size=config.block_size,
|
||||
overwrite=config.overwrite,
|
||||
logger=logger,
|
||||
)
|
||||
l2_path = geometric_correction(
|
||||
l1b_path,
|
||||
rpc_path,
|
||||
out_dir,
|
||||
dem_path=config.dem_path,
|
||||
x_res=config.x_res,
|
||||
y_res=config.y_res,
|
||||
dst_srs=config.dst_srs,
|
||||
overwrite=config.overwrite,
|
||||
logger=logger,
|
||||
)
|
||||
results.append(
|
||||
TaskResult(
|
||||
scene=scene.name,
|
||||
polarization=polarization,
|
||||
status="done",
|
||||
l1b_path=l1b_path,
|
||||
l2_path=l2_path,
|
||||
)
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.exception("Failed %s %s", scene.name, polarization)
|
||||
results.append(
|
||||
TaskResult(
|
||||
scene=scene.name,
|
||||
polarization=polarization,
|
||||
status="failed",
|
||||
message=str(exc),
|
||||
)
|
||||
)
|
||||
if config.fail_fast:
|
||||
raise
|
||||
|
||||
return results
|
||||
@@ -0,0 +1,70 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
from gf3_l1a2l2.errors import RpcError
|
||||
|
||||
|
||||
RPC_KEY_MAP = {
|
||||
"errBias": "ERR_BIAS",
|
||||
"errRand": "ERR_RAND",
|
||||
"lineOffset": "LINE_OFF",
|
||||
"sampOffset": "SAMP_OFF",
|
||||
"latOffset": "LAT_OFF",
|
||||
"longOffset": "LONG_OFF",
|
||||
"heightOffset": "HEIGHT_OFF",
|
||||
"lineScale": "LINE_SCALE",
|
||||
"sampScale": "SAMP_SCALE",
|
||||
"latScale": "LAT_SCALE",
|
||||
"longScale": "LONG_SCALE",
|
||||
"heightScale": "HEIGHT_SCALE",
|
||||
"lineNumCoef": "LINE_NUM_COEFF",
|
||||
"lineDenCoef": "LINE_DEN_COEFF",
|
||||
"sampNumCoef": "SAMP_NUM_COEFF",
|
||||
"sampDenCoef": "SAMP_DEN_COEFF",
|
||||
}
|
||||
|
||||
COEFFICIENT_KEYS = {
|
||||
"lineNumCoef",
|
||||
"lineDenCoef",
|
||||
"sampNumCoef",
|
||||
"sampDenCoef",
|
||||
}
|
||||
|
||||
|
||||
def read_rpc_metadata(rpc_path: Path | str) -> dict[str, str]:
|
||||
path = Path(rpc_path)
|
||||
try:
|
||||
text = path.read_text(encoding="utf-8", errors="ignore")
|
||||
except OSError as exc:
|
||||
raise RpcError(f"Cannot read RPC file: {path}") from exc
|
||||
|
||||
metadata: dict[str, str] = {}
|
||||
missing: list[str] = []
|
||||
for source_key, target_key in RPC_KEY_MAP.items():
|
||||
value = _find_rpc_value(text, source_key)
|
||||
if value is None:
|
||||
missing.append(source_key)
|
||||
continue
|
||||
metadata[target_key] = _normalize_rpc_value(source_key, value)
|
||||
|
||||
if missing:
|
||||
raise RpcError(f"RPC file {path} is missing fields: {', '.join(missing)}")
|
||||
|
||||
return metadata
|
||||
|
||||
|
||||
def _find_rpc_value(text: str, key: str) -> str | None:
|
||||
pattern = re.compile(rf"\b{re.escape(key)}\b\s*[:=]?\s*(.*?);", re.IGNORECASE | re.DOTALL)
|
||||
match = pattern.search(text)
|
||||
if not match:
|
||||
return None
|
||||
return match.group(1)
|
||||
|
||||
|
||||
def _normalize_rpc_value(key: str, raw_value: str) -> str:
|
||||
value = raw_value.strip().strip("=")
|
||||
if key in COEFFICIENT_KEYS:
|
||||
value = value.replace("(", " ").replace(")", " ").replace(",", " ")
|
||||
return " ".join(value.split())
|
||||
@@ -0,0 +1,8 @@
|
||||
<root>
|
||||
<QualifyValue>
|
||||
<HH>1.0</HH><HV>2.0</HV><VH>NULL</VH><VV>4.0</VV>
|
||||
</QualifyValue>
|
||||
<CalibrationConst>
|
||||
<HH>10.0</HH><HV>20.0</HV><VH>30.0</VH><VV>40.0</VV>
|
||||
</CalibrationConst>
|
||||
</root>
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
errBias = -1;
|
||||
errRand = -1;
|
||||
lineOffset = 10;
|
||||
sampOffset = 20;
|
||||
latOffset = 30;
|
||||
longOffset = 40;
|
||||
heightOffset = 50;
|
||||
lineScale = 60;
|
||||
sampScale = 70;
|
||||
latScale = 80;
|
||||
longScale = 90;
|
||||
heightScale = 100;
|
||||
lineNumCoef = (1, 2, 3);
|
||||
lineDenCoef = (4, 5, 6);
|
||||
sampNumCoef = (7, 8, 9);
|
||||
sampDenCoef = (10, 11, 12);
|
||||
@@ -0,0 +1,19 @@
|
||||
import math
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
from gf3_l1a2l2.metadata import read_calibration_metadata
|
||||
|
||||
|
||||
class MetadataTests(unittest.TestCase):
|
||||
def test_reads_named_calibration_values(self):
|
||||
path = Path(__file__).parent / "fixtures" / "sample.meta.xml"
|
||||
metadata = read_calibration_metadata(path)
|
||||
|
||||
self.assertEqual(metadata.for_polarization("HH").qualify_value, 1.0)
|
||||
self.assertEqual(metadata.for_polarization("HV").calibration_const, 20.0)
|
||||
self.assertTrue(math.isnan(metadata.for_polarization("VH").qualify_value))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,18 @@
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
from gf3_l1a2l2.rpc import read_rpc_metadata
|
||||
|
||||
|
||||
class RpcTests(unittest.TestCase):
|
||||
def test_reads_rpc_metadata(self):
|
||||
path = Path(__file__).parent / "fixtures" / "sample.rpb"
|
||||
metadata = read_rpc_metadata(path)
|
||||
|
||||
self.assertEqual(metadata["ERR_BIAS"], "-1")
|
||||
self.assertEqual(metadata["LINE_OFF"], "10")
|
||||
self.assertEqual(metadata["LINE_NUM_COEFF"], "1 2 3")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,51 @@
|
||||
# GF-3 SARscape Go Wrapper
|
||||
|
||||
This wrapper embeds `assets/gf3_sarscape_cli.sav` and calls the local ENVI/IDL Runtime:
|
||||
|
||||
```text
|
||||
idlrt.exe gf3_sarscape_cli.sav -args input_meta_xml out_dir dem_file HH,HV
|
||||
```
|
||||
|
||||
It is the operational Windows path for SARscape-based GF-3 processing.
|
||||
|
||||
## Build
|
||||
|
||||
From this directory:
|
||||
|
||||
```powershell
|
||||
$env:GOTELEMETRY = "off"
|
||||
go build -o ..\..\dist\windows\gf3wrapper.exe .
|
||||
```
|
||||
|
||||
## Run
|
||||
|
||||
Single archive:
|
||||
|
||||
```powershell
|
||||
..\..\dist\windows\gf3wrapper.exe `
|
||||
-input "D:\GF3\GF3_SCENE.tar.gz" `
|
||||
-output "E:\GF3\L2_SARscape" `
|
||||
-dem "D:\DEM\COPDEM_GLO30_China_4326_DEM" `
|
||||
-pol "HH,HV"
|
||||
```
|
||||
|
||||
Batch directory:
|
||||
|
||||
```powershell
|
||||
..\..\dist\windows\gf3wrapper.exe `
|
||||
-input "D:\GF3\L1A_BATCH" `
|
||||
-output "E:\GF3\L2_SARscape"
|
||||
```
|
||||
|
||||
If `gf3wrapper.json` exists next to the executable, omitted `-input`, `-output`, `-dem`, and `-pol` values are loaded from that file.
|
||||
|
||||
In batch directory mode, a failed scene does not stop the whole run. Invalid or truncated archives, missing metadata, and SARscape failures are reported with the scene input path and reason, then the wrapper continues with the next scene. A final summary lists succeeded and failed scenes, and the process exits non-zero when any scene failed.
|
||||
|
||||
Reruns with the same output directory skip scenes that already have complete `*_geo.sml` outputs and matching data files for the requested polarizations. This lets a repaired archive be placed back into the original input folder without reprocessing scenes that already finished.
|
||||
|
||||
## Notes
|
||||
|
||||
- The server still needs ENVI, IDL Runtime, and SARscape installed and licensed.
|
||||
- `.tar.gz` inputs are extracted under each scene output directory in `.gf3_extract`.
|
||||
- The embedded SAV is extracted to `<output>\.gf3_runtime\gf3_sarscape_cli.sav`.
|
||||
- More operational details are in `docs/sarscape_go_wrapper.md`.
|
||||
Binary file not shown.
@@ -0,0 +1,3 @@
|
||||
module gf3wrapper
|
||||
|
||||
go 1.22
|
||||
@@ -0,0 +1,5 @@
|
||||
.compile 'C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\gf3_sarscape_pipeline.pro'
|
||||
.compile 'C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\gf3_sarscape_cli.pro'
|
||||
resolve_all, skip_routines='envi', /continue_on_error
|
||||
save, /routines, filename='C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\gf3_sarscape_cli.sav'
|
||||
exit
|
||||
@@ -0,0 +1,5 @@
|
||||
.f
|
||||
.compile 'C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\gf3_sarscape_pipeline.pro'
|
||||
.compile 'C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\gf3_sarscape_cli.pro'
|
||||
save, /routines, filename='C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\gf3_sarscape_cli.sav'
|
||||
exit
|
||||
@@ -0,0 +1,79 @@
|
||||
function gf3_cli_split_csv, value
|
||||
compile_opt idl2
|
||||
|
||||
text = strtrim(value, 2)
|
||||
if text eq '' then return, ['HH', 'HV']
|
||||
|
||||
parts = ['']
|
||||
first = 1
|
||||
while 1 do begin
|
||||
pos = strpos(text, ',')
|
||||
if pos lt 0 then begin
|
||||
token = strtrim(text, 2)
|
||||
if first then parts = [token] else parts = [parts, token]
|
||||
break
|
||||
endif
|
||||
token = strtrim(strmid(text, 0, pos), 2)
|
||||
if first then begin
|
||||
parts = [token]
|
||||
first = 0
|
||||
endif else begin
|
||||
parts = [parts, token]
|
||||
endelse
|
||||
text = strtrim(strmid(text, pos + 1), 2)
|
||||
endwhile
|
||||
|
||||
return, parts
|
||||
end
|
||||
|
||||
|
||||
pro gf3_sarscape_cli
|
||||
compile_opt idl2
|
||||
|
||||
args = command_line_args(count=argc)
|
||||
if argc lt 3 then begin
|
||||
print, 'Usage: idlrt.exe gf3_sarscape_cli.sav -args input_meta_xml out_dir dem_file [polarizations]'
|
||||
print, 'Example polarizations: HH,HV or HH or HV'
|
||||
return
|
||||
endif
|
||||
|
||||
input_meta_xml = args[0]
|
||||
out_dir = args[1]
|
||||
dem_file = args[2]
|
||||
file_mkdir, out_dir
|
||||
|
||||
log_file = out_dir + '\gf3_sarscape_cli.log'
|
||||
openw, lun, log_file, /get_lun
|
||||
catch, err
|
||||
if err ne 0 then begin
|
||||
printf, lun, 'ERROR: ', !error_state.msg
|
||||
free_lun, lun
|
||||
catch, /cancel
|
||||
return
|
||||
endif
|
||||
|
||||
printf, lun, 'CLI start'
|
||||
printf, lun, 'input_meta_xml=', input_meta_xml
|
||||
printf, lun, 'out_dir=', out_dir
|
||||
printf, lun, 'dem_file=', dem_file
|
||||
|
||||
if argc ge 4 then begin
|
||||
polarizations = gf3_cli_split_csv(args[3])
|
||||
endif else begin
|
||||
polarizations = ['HH', 'HV']
|
||||
endelse
|
||||
|
||||
printf, lun, 'polarizations=', strjoin(polarizations, ',')
|
||||
|
||||
e = envi(/headless)
|
||||
printf, lun, 'ENVI headless initialized'
|
||||
|
||||
gf3_sarscape_run, $
|
||||
input_meta_xml=input_meta_xml, $
|
||||
out_dir=out_dir, $
|
||||
dem_file=dem_file, $
|
||||
polarizations=polarizations
|
||||
|
||||
printf, lun, 'CLI end'
|
||||
free_lun, lun
|
||||
end
|
||||
@@ -0,0 +1,506 @@
|
||||
function gf3_normalize_errmsg, value
|
||||
compile_opt idl2
|
||||
|
||||
if n_elements(value) eq 0 then return, ''
|
||||
|
||||
sz = size(value, /tname)
|
||||
|
||||
if sz eq 'STRING' then begin
|
||||
if n_elements(value) eq 1 then return, value
|
||||
return, strjoin(value, ' ')
|
||||
endif
|
||||
|
||||
if sz eq 'BYTE' then begin
|
||||
if n_elements(value) eq 0 then return, ''
|
||||
return, string(value)
|
||||
endif
|
||||
|
||||
return, string(value)
|
||||
end
|
||||
|
||||
|
||||
function gf3_float_str, value
|
||||
compile_opt idl2
|
||||
return, string(double(value), format='(F0.1)')
|
||||
end
|
||||
|
||||
|
||||
function gf3_base_root, polarization
|
||||
compile_opt idl2
|
||||
common gf3_pipeline_cfg, cfg
|
||||
|
||||
pol = strupcase(strtrim(polarization, 2))
|
||||
case pol of
|
||||
'HH': return, cfg.imported_hh_root
|
||||
'HV': return, cfg.imported_hv_root
|
||||
else: message, 'Unsupported polarization: ' + pol
|
||||
endcase
|
||||
end
|
||||
|
||||
|
||||
function gf3_stage_root, polarization, stage_name
|
||||
compile_opt idl2
|
||||
common gf3_pipeline_cfg, cfg
|
||||
|
||||
pol = strlowcase(strtrim(polarization, 2))
|
||||
prefix = cfg.output_prefix
|
||||
if prefix eq '' then prefix = 'gf3'
|
||||
|
||||
case strlowcase(strtrim(stage_name, 2)) of
|
||||
'multilook': return, cfg.out_dir + '\' + prefix + '_' + pol + '_ml'
|
||||
'filter': return, cfg.out_dir + '\' + prefix + '_' + pol + '_filt'
|
||||
'geocode': return, cfg.out_dir + '\' + prefix + '_' + pol + '_geo'
|
||||
else: message, 'Unsupported stage: ' + stage_name
|
||||
endcase
|
||||
end
|
||||
|
||||
|
||||
function gf3_scene_token, meta_xml
|
||||
compile_opt idl2
|
||||
|
||||
base = file_basename(meta_xml)
|
||||
pos = strpos(strlowcase(base), '.meta.xml')
|
||||
if pos ge 0 then base = strmid(base, 0, pos)
|
||||
return, base
|
||||
end
|
||||
|
||||
|
||||
function gf3_imported_root_for, polarization
|
||||
compile_opt idl2
|
||||
common gf3_pipeline_cfg, cfg
|
||||
|
||||
pol = strupcase(strtrim(polarization, 2))
|
||||
pattern = cfg.out_dir + '\*_' + pol + '_slc.sml'
|
||||
matches = file_search(pattern, count=count)
|
||||
if count eq 0 then return, ''
|
||||
|
||||
newest = matches[0]
|
||||
newest_time = (file_info(newest)).mtime
|
||||
for i = 1, count - 1 do begin
|
||||
info = file_info(matches[i])
|
||||
if info.mtime gt newest_time then begin
|
||||
newest = matches[i]
|
||||
newest_time = info.mtime
|
||||
endif
|
||||
endfor
|
||||
|
||||
return, strmid(newest, 0, strlen(newest) - 4)
|
||||
end
|
||||
|
||||
|
||||
pro gf3_refresh_import_roots
|
||||
compile_opt idl2
|
||||
common gf3_pipeline_cfg, cfg
|
||||
|
||||
hh = gf3_imported_root_for('HH')
|
||||
hv = gf3_imported_root_for('HV')
|
||||
|
||||
if hh ne '' then cfg.imported_hh_root = hh
|
||||
if hv ne '' then cfg.imported_hv_root = hv
|
||||
end
|
||||
|
||||
|
||||
pro gf3_run_module, module_name, set_params_proc, errmsg=errmsg, working_directory=working_directory
|
||||
compile_opt idl2
|
||||
|
||||
errmsg = ''
|
||||
ob = obj_new('SARscapeBatch', Module=module_name)
|
||||
if ~obj_valid(ob) then begin
|
||||
errmsg = 'Create object failed: ' + module_name
|
||||
print, errmsg
|
||||
return
|
||||
endif
|
||||
|
||||
ob->SetParam, 'GENERAL_PARAMETERS_CMD.AVAILABLE_MEMORY_SIZE_GB', '8.0000000'
|
||||
ob->SetParam, 'GENERAL_PARAMETERS_CMD.LOAD_IMAGES', 'OK'
|
||||
ob->SetParam, 'GENERAL_PARAMETERS_CMD.DELETE_TEMPORARY_FILES', 'OK'
|
||||
ob->SetParam, 'GENERAL_PARAMETERS_CMD.MAKE_TIFF', 'OK'
|
||||
ob->SetParam, 'GENERAL_PARAMETERS_CMD.QUICK_LOOK_FORMAT', 'ql_tiff'
|
||||
ob->SetParam, 'GENERAL_PARAMETERS_CMD.SARSCAPE_TRACE_LEVEL', '3.0000000'
|
||||
ob->SetParam, 'GENERAL_PARAMETERS_CMD.RENAME_THE_FILE_USING_PARAMETERS_FLAG', 'OK'
|
||||
ob->SetParam, 'GENERAL_PARAMETERS_CMD.INSERT_GEO_POINTS_FLAG', 'NotOK'
|
||||
ob->SetParam, 'GENERAL_PARAMETERS_CMD.VERBOSE_TRACE', 'OK'
|
||||
ob->SetParam, 'GENERAL_PARAMETERS_CMD.VERBOSE_STEP', 'OK'
|
||||
ob->SetParam, 'GENERAL_PARAMETERS_CMD.VERBOSE_BAR', 'OK'
|
||||
ob->SetParam, 'GENERAL_PARAMETERS_CMD.APPLY_CALIBRATION_CONSTANT_FLAG', 'OK'
|
||||
ob->SetParam, 'GENERAL_PARAMETERS_CMD.FILL_DUMMY_DURING_IMPORT', 'OK'
|
||||
|
||||
call_procedure, set_params_proc, ob
|
||||
|
||||
ok = ob->VerifyParams(Silent=0)
|
||||
if ~ok then begin
|
||||
errmsg = 'VerifyParams failed: ' + module_name
|
||||
print, errmsg
|
||||
return
|
||||
endif
|
||||
|
||||
if n_elements(working_directory) eq 0 then begin
|
||||
ok = ob->Execute()
|
||||
endif else begin
|
||||
exec_errmsg = ''
|
||||
ok = ob->Execute(WORKING_DIRECTORY=working_directory, ERRMSG=exec_errmsg)
|
||||
errmsg = gf3_normalize_errmsg(exec_errmsg)
|
||||
endelse
|
||||
|
||||
if ok then begin
|
||||
print, 'Success: ' + module_name
|
||||
endif else begin
|
||||
if errmsg eq '' then begin
|
||||
err_code = ''
|
||||
err_text = get_SARscape_error_string('OK', ERROR_CODE=err_code)
|
||||
errmsg = 'Execution failed [' + module_name + '] EC [' + err_code + ']: ' + err_text
|
||||
endif
|
||||
print, errmsg
|
||||
endelse
|
||||
end
|
||||
|
||||
|
||||
pro gf3_pipeline_set_import_params, ob
|
||||
compile_opt idl2
|
||||
common gf3_pipeline_cfg, cfg
|
||||
|
||||
ob->SetParam, 'MAIN_BASIC_IMPORT_GAOFEN3_CMD_CMD.SARSCAPEENVIRONMENT', 'IDL_ENVI_ENV'
|
||||
ob->SetParam, 'MAIN_BASIC_IMPORT_GAOFEN3_CMD_CMD.INPUT_FILE_LIST', [cfg.input_meta_xml]
|
||||
ob->SetParam, 'MAIN_BASIC_IMPORT_GAOFEN3_CMD_CMD.OUTPUT_FILE_LIST', [cfg.import_root]
|
||||
ob->SetParam, 'MAIN_BASIC_IMPORT_GAOFEN3_CMD_CMD.CROSS_COPOLARIZATION_FLAG', cfg.import_polarization_mode
|
||||
ob->SetParam, 'MAIN_BASIC_IMPORT_GAOFEN3_CMD_CMD.DOPP_ESTIMATION_FROM_RASTER_FLAG', 'OK'
|
||||
ob->SetParam, 'MAIN_BASIC_IMPORT_GAOFEN3_CMD_CMD.KEEP_ALL_LINES_FLAG', 'NotOK'
|
||||
end
|
||||
|
||||
|
||||
pro gf3_pipeline_set_multilook_params, ob
|
||||
compile_opt idl2
|
||||
common gf3_pipeline_cfg, cfg
|
||||
common gf3_pipeline_state, state
|
||||
|
||||
ob->SetParam, 'MAIN_BASIC_MULTILOOKING.SARSCAPEENVIRONMENT', 'IDL_ENVI_ENV'
|
||||
ob->SetParam, 'MAIN_BASIC_MULTILOOKING.INPUT_FILE_LIST', [state.current_input_root]
|
||||
ob->SetParam, 'MAIN_BASIC_MULTILOOKING.OUTPUT_FILE_LIST', [state.current_output_root]
|
||||
ob->SetParam, 'MAIN_BASIC_MULTILOOKING.MULTILOOK_METHOD', cfg.multilook_method
|
||||
ob->SetParam, 'MAIN_BASIC_MULTILOOKING.RANGE_MULTILOOK', gf3_float_str(cfg.range_looks)
|
||||
ob->SetParam, 'MAIN_BASIC_MULTILOOKING.AZIMUTH_MULTILOOK', gf3_float_str(cfg.azimuth_looks)
|
||||
ob->SetParam, 'MAIN_BASIC_MULTILOOKING.GRID_SIZE_FOR_SUGGESTED_LOOKS', gf3_float_str(cfg.grid_size_for_suggested_looks)
|
||||
ob->SetParam, 'MAIN_BASIC_MULTILOOKING.FILL_DUMMY_FLAG', 'OK'
|
||||
ob->SetParam, 'MAIN_BASIC_MULTILOOKING.FILL_DUMMY_METHOD', 'min_image_value'
|
||||
ob->SetParam, 'MAIN_BASIC_MULTILOOKING.ROWS_WINDOW_NUMBER', '3.0000000'
|
||||
ob->SetParam, 'MAIN_BASIC_MULTILOOKING.COLS_WINDOW_NUMBER', '3.0000000'
|
||||
end
|
||||
|
||||
|
||||
pro gf3_pipeline_set_filter_params, ob
|
||||
compile_opt idl2
|
||||
common gf3_pipeline_cfg, cfg
|
||||
common gf3_pipeline_state, state
|
||||
|
||||
ob->SetParam, 'MAIN_BASIC_DESPECKLE_CONVENTIONAL_CMD.SARSCAPEENVIRONMENT', 'IDL_ENVI_ENV'
|
||||
ob->SetParam, 'MAIN_BASIC_DESPECKLE_CONVENTIONAL_CMD.INPUT_FILE_LIST', [state.current_input_root]
|
||||
ob->SetParam, 'MAIN_BASIC_DESPECKLE_CONVENTIONAL_CMD.OUTPUT_FILE_LIST', [state.current_output_root]
|
||||
ob->SetParam, 'MAIN_BASIC_DESPECKLE_CONVENTIONAL_CMD.FILT_TYPE', cfg.filter_type
|
||||
ob->SetParam, 'MAIN_BASIC_DESPECKLE_CONVENTIONAL_CMD.EQUIVALENT_LOOKS', '-1.0000000'
|
||||
ob->SetParam, 'PARAMETERS_GENERIC_FILTERS.WIN_SIZE', gf3_float_str(cfg.filter_window)
|
||||
ob->SetParam, 'PARAMETERS_GENERIC_FILTERS.ROWS_WINDOW_NUMBER', gf3_float_str(cfg.filter_window)
|
||||
ob->SetParam, 'PARAMETERS_GENERIC_FILTERS.COLS_WINDOW_NUMBER', gf3_float_str(cfg.filter_window)
|
||||
ob->SetParam, 'PARAMETERS_GENERIC_FILTERS.WIN_MODE_SIZE', gf3_float_str(cfg.filter_window)
|
||||
end
|
||||
|
||||
|
||||
pro gf3_pipeline_set_geocode_params, ob
|
||||
compile_opt idl2
|
||||
common gf3_pipeline_cfg, cfg
|
||||
common gf3_pipeline_state, state
|
||||
|
||||
ob->SetParam, 'OUT_CARTOGRAPHIC_SYSTEM.OCS_CARTOGRAPHIC_SYSTEM_SPECIFICATION', ''
|
||||
ob->SetParam, 'OUT_CARTOGRAPHIC_SYSTEM.OCS_STATE', cfg.ocs_state
|
||||
ob->SetParam, 'OUT_CARTOGRAPHIC_SYSTEM.OCS_HEMISPHERE', cfg.ocs_hemisphere
|
||||
ob->SetParam, 'OUT_CARTOGRAPHIC_SYSTEM.OCS_PROJECTION', cfg.ocs_projection
|
||||
ob->SetParam, 'OUT_CARTOGRAPHIC_SYSTEM.OCS_ELLIPSOID', 'WGS84'
|
||||
ob->SetParam, 'OUT_CARTOGRAPHIC_SYSTEM.OCS_REFERENCE_HEIGHT', '0.0'
|
||||
|
||||
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.SARSCAPEENVIRONMENT', 'IDL_ENVI_ENV'
|
||||
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.INPUT_FILE_LIST', [state.current_input_root]
|
||||
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.OUTPUT_FILE_LIST', [state.current_output_root]
|
||||
if cfg.dem_file ne '' then ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.DEM_FILE_NAME', cfg.dem_file
|
||||
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GEOCODE_GRID_SIZE_X', gf3_float_str(cfg.geocode_grid_x)
|
||||
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GEOCODE_GRID_SIZE_Y', gf3_float_str(cfg.geocode_grid_y)
|
||||
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GEOCODE_RESAMPLING_TYPE', cfg.geocode_resampling
|
||||
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.CALIBRATION_FLAG', cfg.calibration_flag
|
||||
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GEO_SCATTERING_AREA_METHOD', 'sine_area_estimation'
|
||||
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.TRUE_AREA_EXPONENT', '1.0000000'
|
||||
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.MAX_VALUE_IN_CALIBRATION', '100.00000'
|
||||
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.RAD_NORMALIZATION_FLAG', cfg.rad_normalization_flag
|
||||
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GEOCODE_RAD_NORM_DEG', '2.0000000'
|
||||
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GEOCODE_RAD_NORM_ANG', '-1.0000000'
|
||||
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GEOCODE_SIGMA_FLAG', cfg.output_sigma_flag
|
||||
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GEOCODE_GAMMA_FLAG', cfg.output_gamma_flag
|
||||
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GEOCODE_BETA_FLAG', cfg.output_beta_flag
|
||||
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GEOCODE_COREG_IMAGES_FLAG', 'NotOK'
|
||||
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GENERATE_LIA_FLAG', 'NotOK'
|
||||
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.OUTPUT_TYPE', cfg.output_type
|
||||
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.DUMMY_REMOVAL_FLAG', 'NotOK'
|
||||
ob->SetParam, 'MAIN_BASIC_CALIBRATION_AND_GEO_CMD.GEOCODE_BLOCK_SIZE', '30000.000'
|
||||
end
|
||||
|
||||
|
||||
pro gf3_init_default_cfg
|
||||
compile_opt idl2
|
||||
common gf3_pipeline_cfg, cfg
|
||||
|
||||
cfg = { $
|
||||
input_meta_xml: 'C:\Users\Administrator\Desktop\GF3_Test\GF3_scene_20260513\GF3_SYC_FSI_051370_E124.6_N51.7_20260513_L1A_HHHV_L10007355955.meta.xml', $
|
||||
work_dir: 'C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\work', $
|
||||
temp_dir: 'C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\temp', $
|
||||
out_dir: 'C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\out', $
|
||||
output_prefix: '', $
|
||||
dem_file: 'D:\DEM\COPDEM_GLO30_China_4326_DEM', $
|
||||
import_root: 'C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\out\gf3_import', $
|
||||
import_polarization_mode: 'ALL_POL', $
|
||||
imported_hh_root: 'C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\out\gaofer3_20260513_214340051_D_HH_slc', $
|
||||
imported_hv_root: 'C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\out\gaofer3_20260513_214340051_D_HV_slc', $
|
||||
multilook_method: 'time_domain', $
|
||||
range_looks: 2.0, $
|
||||
azimuth_looks: 2.0, $
|
||||
grid_size_for_suggested_looks: 15.0, $
|
||||
filter_type: 'lee', $
|
||||
filter_window: 5.0, $
|
||||
geocode_grid_x: 15.0, $
|
||||
geocode_grid_y: 15.0, $
|
||||
geocode_resampling: '4th_order_cc', $
|
||||
calibration_flag: 'OK', $
|
||||
rad_normalization_flag: 'OK', $
|
||||
output_sigma_flag: 'OK', $
|
||||
output_gamma_flag: 'NotOK', $
|
||||
output_beta_flag: 'NotOK', $
|
||||
output_type: 'output_type_linear', $
|
||||
ocs_state: 'GEO-GLOBAL', $
|
||||
ocs_hemisphere: 'NONE', $
|
||||
ocs_projection: 'GEO' $
|
||||
}
|
||||
end
|
||||
|
||||
|
||||
pro gf3_configure, input_meta_xml=input_meta_xml, out_dir=out_dir, dem_file=dem_file, $
|
||||
work_dir=work_dir, temp_dir=temp_dir, range_looks=range_looks, $
|
||||
azimuth_looks=azimuth_looks, filter_type=filter_type, filter_window=filter_window, $
|
||||
geocode_grid_x=geocode_grid_x, geocode_grid_y=geocode_grid_y, $
|
||||
imported_hh_root=imported_hh_root, imported_hv_root=imported_hv_root
|
||||
|
||||
compile_opt idl2
|
||||
common gf3_pipeline_cfg, cfg
|
||||
|
||||
gf3_init_default_cfg
|
||||
|
||||
if n_elements(input_meta_xml) gt 0 then cfg.input_meta_xml = input_meta_xml
|
||||
if n_elements(out_dir) gt 0 then cfg.out_dir = out_dir
|
||||
if n_elements(dem_file) gt 0 then cfg.dem_file = dem_file
|
||||
if n_elements(work_dir) gt 0 then cfg.work_dir = work_dir
|
||||
if n_elements(temp_dir) gt 0 then cfg.temp_dir = temp_dir
|
||||
if n_elements(range_looks) gt 0 then cfg.range_looks = double(range_looks)
|
||||
if n_elements(azimuth_looks) gt 0 then cfg.azimuth_looks = double(azimuth_looks)
|
||||
if n_elements(filter_type) gt 0 then cfg.filter_type = filter_type
|
||||
if n_elements(filter_window) gt 0 then cfg.filter_window = double(filter_window)
|
||||
if n_elements(geocode_grid_x) gt 0 then cfg.geocode_grid_x = double(geocode_grid_x)
|
||||
if n_elements(geocode_grid_y) gt 0 then cfg.geocode_grid_y = double(geocode_grid_y)
|
||||
if n_elements(imported_hh_root) gt 0 then cfg.imported_hh_root = imported_hh_root
|
||||
if n_elements(imported_hv_root) gt 0 then cfg.imported_hv_root = imported_hv_root
|
||||
|
||||
cfg.import_root = cfg.out_dir + '\gf3_import'
|
||||
cfg.output_prefix = gf3_scene_token(cfg.input_meta_xml)
|
||||
|
||||
if n_elements(work_dir) eq 0 then cfg.work_dir = cfg.out_dir + '\work'
|
||||
if n_elements(temp_dir) eq 0 then cfg.temp_dir = cfg.out_dir + '\temp'
|
||||
|
||||
if n_elements(imported_hh_root) eq 0 then cfg.imported_hh_root = ''
|
||||
if n_elements(imported_hv_root) eq 0 then cfg.imported_hv_root = ''
|
||||
end
|
||||
|
||||
|
||||
pro gf3_prepare_dirs
|
||||
compile_opt idl2
|
||||
common gf3_pipeline_cfg, cfg
|
||||
|
||||
file_mkdir, cfg.work_dir
|
||||
file_mkdir, cfg.temp_dir
|
||||
file_mkdir, cfg.out_dir
|
||||
end
|
||||
|
||||
|
||||
pro gf3_start_batch
|
||||
compile_opt idl2
|
||||
common gf3_pipeline_cfg, cfg
|
||||
SARscape_Batch_Init, Temp_Directory=cfg.temp_dir
|
||||
end
|
||||
|
||||
|
||||
pro gf3_stop_batch
|
||||
compile_opt idl2
|
||||
SARscape_Batch_Exit
|
||||
end
|
||||
|
||||
|
||||
pro gf3_run_import, errmsg=errmsg
|
||||
compile_opt idl2
|
||||
common gf3_pipeline_cfg, cfg
|
||||
|
||||
errmsg = ''
|
||||
gf3_run_module, 'ImportGaofen3', 'gf3_pipeline_set_import_params', errmsg=errmsg, working_directory=cfg.work_dir
|
||||
end
|
||||
|
||||
|
||||
pro gf3_run_multilook, polarization, errmsg=errmsg
|
||||
compile_opt idl2
|
||||
common gf3_pipeline_cfg, cfg
|
||||
common gf3_pipeline_state, state
|
||||
|
||||
state = {current_input_root: gf3_base_root(polarization), current_output_root: gf3_stage_root(polarization, 'multilook')}
|
||||
errmsg = ''
|
||||
gf3_run_module, 'BaseMultilooking', 'gf3_pipeline_set_multilook_params', errmsg=errmsg, working_directory=cfg.work_dir
|
||||
end
|
||||
|
||||
|
||||
pro gf3_run_filter, polarization, errmsg=errmsg
|
||||
compile_opt idl2
|
||||
common gf3_pipeline_cfg, cfg
|
||||
common gf3_pipeline_state, state
|
||||
|
||||
state = {current_input_root: gf3_stage_root(polarization, 'multilook'), current_output_root: gf3_stage_root(polarization, 'filter')}
|
||||
errmsg = ''
|
||||
gf3_run_module, 'DespeckleConventionalSingle', 'gf3_pipeline_set_filter_params', errmsg=errmsg, working_directory=cfg.work_dir
|
||||
end
|
||||
|
||||
|
||||
pro gf3_run_geocode, polarization, errmsg=errmsg
|
||||
compile_opt idl2
|
||||
common gf3_pipeline_cfg, cfg
|
||||
common gf3_pipeline_state, state
|
||||
|
||||
state = {current_input_root: gf3_stage_root(polarization, 'filter'), current_output_root: gf3_stage_root(polarization, 'geocode')}
|
||||
errmsg = ''
|
||||
gf3_run_module, 'BasicGeocoding', 'gf3_pipeline_set_geocode_params', errmsg=errmsg, working_directory=cfg.work_dir
|
||||
end
|
||||
|
||||
|
||||
pro gf3_run_single_pol_pipeline, polarization, do_import=do_import, errmsg=errmsg
|
||||
compile_opt idl2
|
||||
|
||||
errmsg = ''
|
||||
if keyword_set(do_import) then begin
|
||||
gf3_run_import, errmsg=errmsg
|
||||
if errmsg ne '' then return
|
||||
gf3_refresh_import_roots
|
||||
endif
|
||||
|
||||
gf3_run_multilook, polarization, errmsg=errmsg
|
||||
if errmsg ne '' then return
|
||||
|
||||
gf3_run_filter, polarization, errmsg=errmsg
|
||||
if errmsg ne '' then return
|
||||
|
||||
gf3_run_geocode, polarization, errmsg=errmsg
|
||||
end
|
||||
|
||||
|
||||
function gf3_should_run_pol, polarizations, polarization
|
||||
compile_opt idl2
|
||||
|
||||
if n_elements(polarizations) eq 0 then return, 1
|
||||
|
||||
wanted = strupcase(strtrim(polarizations, 2))
|
||||
pol = strupcase(strtrim(polarization, 2))
|
||||
|
||||
for i = 0, n_elements(wanted) - 1 do begin
|
||||
if wanted[i] eq 'ALL' then return, 1
|
||||
if wanted[i] eq pol then return, 1
|
||||
endfor
|
||||
|
||||
return, 0
|
||||
end
|
||||
|
||||
|
||||
pro gf3_sarscape_run, input_meta_xml=input_meta_xml, out_dir=out_dir, dem_file=dem_file, $
|
||||
polarizations=polarizations, skip_import=skip_import, work_dir=work_dir, temp_dir=temp_dir, $
|
||||
range_looks=range_looks, azimuth_looks=azimuth_looks, filter_type=filter_type, $
|
||||
filter_window=filter_window, geocode_grid_x=geocode_grid_x, geocode_grid_y=geocode_grid_y
|
||||
|
||||
compile_opt idl2
|
||||
|
||||
gf3_configure, input_meta_xml=input_meta_xml, out_dir=out_dir, dem_file=dem_file, $
|
||||
work_dir=work_dir, temp_dir=temp_dir, range_looks=range_looks, $
|
||||
azimuth_looks=azimuth_looks, filter_type=filter_type, filter_window=filter_window, $
|
||||
geocode_grid_x=geocode_grid_x, geocode_grid_y=geocode_grid_y
|
||||
|
||||
gf3_prepare_dirs
|
||||
gf3_start_batch
|
||||
|
||||
errmsg = ''
|
||||
|
||||
if ~keyword_set(skip_import) then begin
|
||||
gf3_run_import, errmsg=errmsg
|
||||
if errmsg eq '' then gf3_refresh_import_roots
|
||||
endif else begin
|
||||
gf3_refresh_import_roots
|
||||
endelse
|
||||
|
||||
if errmsg eq '' && gf3_should_run_pol(polarizations, 'HH') then gf3_run_single_pol_pipeline, 'HH', errmsg=errmsg
|
||||
if errmsg eq '' && gf3_should_run_pol(polarizations, 'HV') then gf3_run_single_pol_pipeline, 'HV', errmsg=errmsg
|
||||
|
||||
gf3_stop_batch
|
||||
if errmsg ne '' then print, 'Pipeline stopped: ' + errmsg
|
||||
end
|
||||
|
||||
|
||||
pro gf3_sarscape_pipeline
|
||||
compile_opt idl2
|
||||
common gf3_pipeline_cfg, cfg
|
||||
|
||||
gf3_init_default_cfg
|
||||
gf3_prepare_dirs
|
||||
gf3_start_batch
|
||||
|
||||
errmsg = ''
|
||||
gf3_run_import, errmsg=errmsg
|
||||
if errmsg eq '' then gf3_refresh_import_roots
|
||||
if errmsg eq '' then gf3_run_single_pol_pipeline, 'HH', errmsg=errmsg
|
||||
if errmsg eq '' then gf3_run_single_pol_pipeline, 'HV', errmsg=errmsg
|
||||
|
||||
gf3_stop_batch
|
||||
if errmsg ne '' then print, 'Pipeline stopped: ' + errmsg
|
||||
end
|
||||
|
||||
|
||||
pro gf3_sarscape_import_only
|
||||
compile_opt idl2
|
||||
|
||||
gf3_init_default_cfg
|
||||
gf3_prepare_dirs
|
||||
gf3_start_batch
|
||||
errmsg = ''
|
||||
gf3_run_import, errmsg=errmsg
|
||||
gf3_stop_batch
|
||||
if errmsg ne '' then print, 'Import stopped: ' + errmsg
|
||||
end
|
||||
|
||||
|
||||
pro gf3_sarscape_hh_only
|
||||
compile_opt idl2
|
||||
|
||||
gf3_init_default_cfg
|
||||
gf3_prepare_dirs
|
||||
gf3_start_batch
|
||||
errmsg = ''
|
||||
gf3_run_single_pol_pipeline, 'HH', do_import=1, errmsg=errmsg
|
||||
gf3_stop_batch
|
||||
if errmsg ne '' then print, 'HH pipeline stopped: ' + errmsg
|
||||
end
|
||||
|
||||
|
||||
pro gf3_sarscape_hv_only
|
||||
compile_opt idl2
|
||||
|
||||
gf3_init_default_cfg
|
||||
gf3_prepare_dirs
|
||||
gf3_start_batch
|
||||
errmsg = ''
|
||||
gf3_run_single_pol_pipeline, 'HV', do_import=1, errmsg=errmsg
|
||||
gf3_stop_batch
|
||||
if errmsg ne '' then print, 'HV pipeline stopped: ' + errmsg
|
||||
end
|
||||
@@ -0,0 +1,8 @@
|
||||
@echo off
|
||||
set IDLRT=C:\Program Files\Harris\ENVI56\IDL88\bin\bin.x86_64\idlrt.exe
|
||||
set SAV=C:\Users\Administrator\Desktop\GF3_Test\GF3_ENVI_IDL\gf3_sarscape_cli.sav
|
||||
set META=C:\Users\Administrator\Desktop\GF3_Test\GF3_scene_20260513\GF3_SYC_FSI_051370_E124.6_N51.7_20260513_L1A_HHHV_L10007355955.meta.xml
|
||||
set OUT=C:\Users\Administrator\Desktop\GF3_Test\GF3_cli_test_out_20260513
|
||||
set DEM=D:\DEM\COPDEM_GLO30_China_4326_DEM
|
||||
|
||||
"%IDLRT%" "%SAV%" -args "%META%" "%OUT%" "%DEM%" "HH,HV"
|
||||
@@ -0,0 +1,768 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bufio"
|
||||
"compress/gzip"
|
||||
"embed"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
//go:embed assets/gf3_sarscape_cli.sav
|
||||
var embedded embed.FS
|
||||
|
||||
type config struct {
|
||||
configPath string `json:"-"`
|
||||
input string `json:"input"`
|
||||
output string `json:"output"`
|
||||
dem string `json:"dem"`
|
||||
polarizations string `json:"polarizations"`
|
||||
idlrt string `json:"idlrt"`
|
||||
keepExtracted bool `json:"keep_extracted"`
|
||||
}
|
||||
|
||||
type scene struct {
|
||||
metaPath string
|
||||
archivePath string
|
||||
name string
|
||||
}
|
||||
|
||||
type sceneFailure struct {
|
||||
name string
|
||||
input string
|
||||
err error
|
||||
}
|
||||
|
||||
type batchError struct {
|
||||
failures []sceneFailure
|
||||
}
|
||||
|
||||
func (e batchError) Error() string {
|
||||
return fmt.Sprintf("batch completed with %d failed scene(s)", len(e.failures))
|
||||
}
|
||||
|
||||
type completedProduct struct {
|
||||
polarization string
|
||||
path string
|
||||
}
|
||||
|
||||
func main() {
|
||||
cfg := parseFlags()
|
||||
if len(os.Args) == 1 {
|
||||
var err error
|
||||
cfg, err = promptConfig(cfg)
|
||||
if err != nil {
|
||||
fmt.Printf("ERROR: %v\n", err)
|
||||
waitForEnter()
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
if err := run(cfg); err != nil {
|
||||
fmt.Printf("ERROR: %v\n", err)
|
||||
if len(os.Args) == 1 {
|
||||
waitForEnter()
|
||||
}
|
||||
os.Exit(1)
|
||||
}
|
||||
if err := saveConfig(cfg.configPath, cfg); err != nil {
|
||||
fmt.Printf("WARNING: save config: %v\n", err)
|
||||
}
|
||||
if len(os.Args) == 1 {
|
||||
fmt.Println("Done.")
|
||||
waitForEnter()
|
||||
}
|
||||
}
|
||||
|
||||
func parseFlags() config {
|
||||
cfg := defaultConfig()
|
||||
configPath := preScanConfigPath()
|
||||
cfg.configPath = configPath
|
||||
_ = loadConfig(configPath, &cfg)
|
||||
|
||||
flag.StringVar(&cfg.configPath, "config", configPath, "Config file path")
|
||||
flag.StringVar(&cfg.input, "input", cfg.input, "GF-3 input: .meta.xml, .tar.gz, or directory")
|
||||
flag.StringVar(&cfg.output, "output", cfg.output, "Output directory")
|
||||
flag.StringVar(&cfg.dem, "dem", cfg.dem, "SARscape DEM file")
|
||||
flag.StringVar(&cfg.polarizations, "pol", cfg.polarizations, "Polarizations: HH, HV, or HH,HV")
|
||||
flag.StringVar(&cfg.idlrt, "idlrt", cfg.idlrt, "Path to idlrt.exe")
|
||||
flag.BoolVar(&cfg.keepExtracted, "keep-extracted", cfg.keepExtracted, "Keep extracted archives under output/.gf3_extract")
|
||||
flag.Parse()
|
||||
return cfg
|
||||
}
|
||||
|
||||
func defaultConfig() config {
|
||||
return config{
|
||||
dem: `D:\DEM\COPDEM_GLO30_China_4326_DEM`,
|
||||
polarizations: "HH,HV",
|
||||
idlrt: defaultIDLRT(),
|
||||
keepExtracted: true,
|
||||
}
|
||||
}
|
||||
|
||||
func preScanConfigPath() string {
|
||||
defaultPath := defaultConfigPath()
|
||||
for i := 1; i < len(os.Args)-1; i++ {
|
||||
if os.Args[i] == "-config" {
|
||||
return os.Args[i+1]
|
||||
}
|
||||
}
|
||||
return defaultPath
|
||||
}
|
||||
|
||||
func defaultConfigPath() string {
|
||||
exe, err := os.Executable()
|
||||
if err != nil {
|
||||
return "gf3wrapper.json"
|
||||
}
|
||||
return filepath.Join(filepath.Dir(exe), "gf3wrapper.json")
|
||||
}
|
||||
|
||||
func loadConfig(path string, cfg *config) error {
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var stored struct {
|
||||
IDLRTPath string `json:"idlrt_path"`
|
||||
DemFile string `json:"dem_file"`
|
||||
Polarizations string `json:"polarizations"`
|
||||
LastInput string `json:"last_input"`
|
||||
LastOutput string `json:"last_output"`
|
||||
IDLRT string `json:"idlrt"`
|
||||
Dem string `json:"dem"`
|
||||
Input string `json:"input"`
|
||||
Output string `json:"output"`
|
||||
KeepExtracted *bool `json:"keep_extracted"`
|
||||
}
|
||||
if err := json.Unmarshal(data, &stored); err != nil {
|
||||
return err
|
||||
}
|
||||
if stored.IDLRTPath != "" {
|
||||
cfg.idlrt = stored.IDLRTPath
|
||||
} else if stored.IDLRT != "" {
|
||||
cfg.idlrt = stored.IDLRT
|
||||
}
|
||||
if stored.DemFile != "" {
|
||||
cfg.dem = stored.DemFile
|
||||
} else if stored.Dem != "" {
|
||||
cfg.dem = stored.Dem
|
||||
}
|
||||
if stored.Polarizations != "" {
|
||||
cfg.polarizations = stored.Polarizations
|
||||
}
|
||||
if stored.LastInput != "" {
|
||||
cfg.input = stored.LastInput
|
||||
} else if stored.Input != "" {
|
||||
cfg.input = stored.Input
|
||||
}
|
||||
if stored.LastOutput != "" {
|
||||
cfg.output = stored.LastOutput
|
||||
} else if stored.Output != "" {
|
||||
cfg.output = stored.Output
|
||||
}
|
||||
if stored.KeepExtracted != nil {
|
||||
cfg.keepExtracted = *stored.KeepExtracted
|
||||
}
|
||||
cfg.configPath = path
|
||||
return nil
|
||||
}
|
||||
|
||||
func saveConfig(path string, cfg config) error {
|
||||
if path == "" {
|
||||
path = defaultConfigPath()
|
||||
}
|
||||
stored := struct {
|
||||
IDLRTPath string `json:"idlrt_path"`
|
||||
DemFile string `json:"dem_file"`
|
||||
Polarizations string `json:"polarizations"`
|
||||
LastInput string `json:"last_input"`
|
||||
LastOutput string `json:"last_output"`
|
||||
KeepExtracted bool `json:"keep_extracted"`
|
||||
}{
|
||||
IDLRTPath: cfg.idlrt,
|
||||
DemFile: cfg.dem,
|
||||
Polarizations: cfg.polarizations,
|
||||
LastInput: cfg.input,
|
||||
LastOutput: cfg.output,
|
||||
KeepExtracted: cfg.keepExtracted,
|
||||
}
|
||||
data, err := json.MarshalIndent(stored, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return os.WriteFile(path, data, 0o644)
|
||||
}
|
||||
|
||||
func promptConfig(defaults config) (config, error) {
|
||||
reader := bufio.NewReader(os.Stdin)
|
||||
cfg := defaults
|
||||
|
||||
fmt.Println("GF-3 SARscape Wrapper")
|
||||
fmt.Println()
|
||||
|
||||
cfg.input = prompt(reader, "Input .tar.gz / .meta.xml / folder", cfg.input)
|
||||
cfg.output = prompt(reader, "Output folder", cfg.output)
|
||||
cfg.dem = prompt(reader, "DEM file", cfg.dem)
|
||||
cfg.polarizations = prompt(reader, "Polarizations", cfg.polarizations)
|
||||
cfg.idlrt = prompt(reader, "IDL Runtime", cfg.idlrt)
|
||||
|
||||
if cfg.input == "" || cfg.output == "" || cfg.dem == "" {
|
||||
return cfg, errors.New("input, output, and DEM are required")
|
||||
}
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
func prompt(reader *bufio.Reader, label, defaultValue string) string {
|
||||
if defaultValue != "" {
|
||||
fmt.Printf("%s [%s]: ", label, defaultValue)
|
||||
} else {
|
||||
fmt.Printf("%s: ", label)
|
||||
}
|
||||
text, _ := reader.ReadString('\n')
|
||||
text = strings.TrimSpace(text)
|
||||
if text == "" {
|
||||
return defaultValue
|
||||
}
|
||||
return text
|
||||
}
|
||||
|
||||
func waitForEnter() {
|
||||
fmt.Println()
|
||||
fmt.Print("Press Enter to exit...")
|
||||
_, _ = bufio.NewReader(os.Stdin).ReadString('\n')
|
||||
}
|
||||
|
||||
func run(cfg config) error {
|
||||
if cfg.input == "" || cfg.output == "" || cfg.dem == "" {
|
||||
return errors.New("required flags: -input, -output, -dem")
|
||||
}
|
||||
if _, err := os.Stat(cfg.idlrt); err != nil {
|
||||
return fmt.Errorf("idlrt not found: %s: %w", cfg.idlrt, err)
|
||||
}
|
||||
if _, err := os.Stat(cfg.dem); err != nil {
|
||||
return fmt.Errorf("DEM not found: %s: %w", cfg.dem, err)
|
||||
}
|
||||
if err := os.MkdirAll(cfg.output, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
scenes, err := discoverScenes(cfg)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(scenes) == 0 {
|
||||
return fmt.Errorf("no *.meta.xml scenes found in %s", cfg.input)
|
||||
}
|
||||
|
||||
failures := make([]sceneFailure, 0)
|
||||
processed := 0
|
||||
skipped := 0
|
||||
savPath := ""
|
||||
for _, sc := range scenes {
|
||||
sceneOut := filepath.Join(cfg.output, sc.name)
|
||||
if products, ok := completedSceneOutputs(sceneOut, sc.name, cfg.polarizations); ok {
|
||||
recordSceneSkip(sc, sceneOut, products)
|
||||
skipped++
|
||||
continue
|
||||
}
|
||||
if err := os.MkdirAll(sceneOut, 0o755); err != nil {
|
||||
recordSceneFailure(&failures, sc, err)
|
||||
continue
|
||||
}
|
||||
if sc.archivePath != "" {
|
||||
extractDir := filepath.Join(sceneOut, ".gf3_extract")
|
||||
fmt.Printf("Extracting %s\n", sc.archivePath)
|
||||
if err := extractTarGz(sc.archivePath, extractDir); err != nil {
|
||||
recordSceneFailure(&failures, sc, fmt.Errorf("failed to extract archive before processing scene %s: %w", sc.archivePath, err))
|
||||
continue
|
||||
}
|
||||
extractedScenes, err := findMetaXML(extractDir)
|
||||
if err != nil {
|
||||
recordSceneFailure(&failures, sc, err)
|
||||
continue
|
||||
}
|
||||
if len(extractedScenes) == 0 {
|
||||
recordSceneFailure(&failures, sc, fmt.Errorf("no *.meta.xml found in archive: %s", sc.archivePath))
|
||||
continue
|
||||
}
|
||||
sc.metaPath = extractedScenes[0].metaPath
|
||||
sc.name = sceneName(sc.metaPath)
|
||||
}
|
||||
if savPath == "" {
|
||||
var err error
|
||||
savPath, err = materializeSAV(cfg.output)
|
||||
if err != nil {
|
||||
recordSceneFailure(&failures, sc, err)
|
||||
continue
|
||||
}
|
||||
}
|
||||
if err := runScene(cfg, savPath, sc, sceneOut); err != nil {
|
||||
recordSceneFailure(&failures, sc, err)
|
||||
continue
|
||||
}
|
||||
processed++
|
||||
}
|
||||
printBatchSummary(len(scenes), processed, skipped, failures)
|
||||
if len(failures) > 0 {
|
||||
return batchError{failures: failures}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func sceneInput(sc scene) string {
|
||||
if sc.archivePath != "" {
|
||||
return sc.archivePath
|
||||
}
|
||||
if sc.metaPath != "" {
|
||||
return sc.metaPath
|
||||
}
|
||||
return sc.name
|
||||
}
|
||||
|
||||
func recordSceneFailure(failures *[]sceneFailure, sc scene, err error) {
|
||||
input := sceneInput(sc)
|
||||
|
||||
failure := sceneFailure{name: sc.name, input: input, err: err}
|
||||
*failures = append(*failures, failure)
|
||||
|
||||
fmt.Printf("FAILED %s\n", sc.name)
|
||||
fmt.Printf("Input %s\n", input)
|
||||
fmt.Printf("Reason: %v\n", err)
|
||||
fmt.Println("Continuing with next scene.")
|
||||
}
|
||||
|
||||
func recordSceneSkip(sc scene, sceneOut string, products []completedProduct) {
|
||||
parts := make([]string, 0, len(products))
|
||||
for _, product := range products {
|
||||
parts = append(parts, product.polarization+"="+product.path)
|
||||
}
|
||||
|
||||
fmt.Printf("SKIPPED %s\n", sc.name)
|
||||
fmt.Printf("Input %s\n", sceneInput(sc))
|
||||
fmt.Printf("Output already complete %s\n", sceneOut)
|
||||
fmt.Printf("Products %s\n", strings.Join(parts, "; "))
|
||||
}
|
||||
|
||||
func printBatchSummary(total, processed, skipped int, failures []sceneFailure) {
|
||||
fmt.Println()
|
||||
fmt.Printf("Batch summary: %d processed, %d skipped, %d failed, %d total\n", processed, skipped, len(failures), total)
|
||||
if len(failures) == 0 {
|
||||
return
|
||||
}
|
||||
fmt.Println("Failed scenes:")
|
||||
for _, failure := range failures {
|
||||
fmt.Printf("- %s (%s): %v\n", failure.name, failure.input, failure.err)
|
||||
}
|
||||
}
|
||||
|
||||
func materializeSAV(outputDir string) (string, error) {
|
||||
data, err := embedded.ReadFile("assets/gf3_sarscape_cli.sav")
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
runtimeDir := filepath.Join(outputDir, ".gf3_runtime")
|
||||
if err := os.MkdirAll(runtimeDir, 0o755); err != nil {
|
||||
return "", err
|
||||
}
|
||||
savPath := filepath.Join(runtimeDir, "gf3_sarscape_cli.sav")
|
||||
if err := os.WriteFile(savPath, data, 0o644); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return savPath, nil
|
||||
}
|
||||
|
||||
func discoverScenes(cfg config) ([]scene, error) {
|
||||
info, err := os.Stat(cfg.input)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if info.IsDir() {
|
||||
return discoverScenesInDir(cfg.input, cfg.output)
|
||||
}
|
||||
lower := strings.ToLower(cfg.input)
|
||||
if strings.HasSuffix(lower, ".meta.xml") {
|
||||
return []scene{{metaPath: cfg.input, name: sceneName(cfg.input)}}, nil
|
||||
}
|
||||
if strings.HasSuffix(lower, ".tar.gz") || strings.HasSuffix(lower, ".tgz") {
|
||||
return []scene{{archivePath: cfg.input, name: safeName(trimArchiveExt(filepath.Base(cfg.input)))}}, nil
|
||||
}
|
||||
return nil, fmt.Errorf("unsupported input type: %s", cfg.input)
|
||||
}
|
||||
|
||||
func discoverScenesInDir(inputDir, outputDir string) ([]scene, error) {
|
||||
_ = outputDir
|
||||
scenesByName := make(map[string]scene)
|
||||
|
||||
err := filepath.WalkDir(inputDir, func(path string, d fs.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if d.IsDir() {
|
||||
if d.Name() == ".gf3_extract" {
|
||||
return filepath.SkipDir
|
||||
}
|
||||
return nil
|
||||
}
|
||||
lower := strings.ToLower(d.Name())
|
||||
switch {
|
||||
case strings.HasSuffix(lower, ".meta.xml"):
|
||||
sc := scene{metaPath: path, name: sceneName(path)}
|
||||
key := strings.ToLower(sc.name)
|
||||
if _, ok := scenesByName[key]; !ok {
|
||||
scenesByName[key] = sc
|
||||
}
|
||||
case strings.HasSuffix(lower, ".tar.gz") || strings.HasSuffix(lower, ".tgz"):
|
||||
sc := scene{archivePath: path, name: safeName(trimArchiveExt(filepath.Base(path)))}
|
||||
scenesByName[strings.ToLower(sc.name)] = sc
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
scenes := make([]scene, 0, len(scenesByName))
|
||||
for _, sc := range scenesByName {
|
||||
scenes = append(scenes, sc)
|
||||
}
|
||||
sort.Slice(scenes, func(i, j int) bool {
|
||||
return strings.ToLower(scenes[i].name) < strings.ToLower(scenes[j].name)
|
||||
})
|
||||
return scenes, nil
|
||||
}
|
||||
|
||||
func findMetaXML(root string) ([]scene, error) {
|
||||
var scenes []scene
|
||||
err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if d.IsDir() {
|
||||
return nil
|
||||
}
|
||||
if strings.HasSuffix(strings.ToLower(d.Name()), ".meta.xml") {
|
||||
scenes = append(scenes, scene{metaPath: path, name: sceneName(path)})
|
||||
}
|
||||
return nil
|
||||
})
|
||||
return scenes, err
|
||||
}
|
||||
|
||||
func uniqueScenes(scenes []scene) []scene {
|
||||
seen := make(map[string]bool, len(scenes))
|
||||
out := make([]scene, 0, len(scenes))
|
||||
for _, sc := range scenes {
|
||||
key := filepath.Clean(sc.metaPath)
|
||||
if seen[key] {
|
||||
continue
|
||||
}
|
||||
seen[key] = true
|
||||
out = append(out, sc)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func runScene(cfg config, savPath string, sc scene, sceneOut string) error {
|
||||
fmt.Printf("Processing %s\n", sc.metaPath)
|
||||
fmt.Printf("Output %s\n", sceneOut)
|
||||
|
||||
cmd := exec.Command(cfg.idlrt, savPath, "-args", sc.metaPath, sceneOut, cfg.dem, cfg.polarizations)
|
||||
cmd.Stdout = os.Stdout
|
||||
cmd.Stderr = os.Stderr
|
||||
cmd.Dir = sceneOut
|
||||
|
||||
if err := cmd.Start(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- cmd.Wait()
|
||||
}()
|
||||
|
||||
ticker := time.NewTicker(2 * time.Second)
|
||||
defer ticker.Stop()
|
||||
|
||||
for {
|
||||
select {
|
||||
case err := <-done:
|
||||
fmt.Printf("\r%-100s\r", "")
|
||||
if err != nil {
|
||||
return fmt.Errorf("scene failed %s: %w", sc.metaPath, err)
|
||||
}
|
||||
fmt.Println("Completed", sc.name)
|
||||
return nil
|
||||
case <-ticker.C:
|
||||
fmt.Printf("\r%s", progressLine(sceneOut, sc.name, cfg.polarizations))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func progressLine(sceneOut, sceneNameValue, polarizations string) string {
|
||||
stage, done, total := progressStage(sceneOut, sceneNameValue, polarizations)
|
||||
width := 28
|
||||
filled := 0
|
||||
if total > 0 {
|
||||
filled = done * width / total
|
||||
}
|
||||
if filled > width {
|
||||
filled = width
|
||||
}
|
||||
bar := strings.Repeat("#", filled) + strings.Repeat("-", width-filled)
|
||||
return fmt.Sprintf("[%s] %2d/%2d %s", bar, done, total, stage)
|
||||
}
|
||||
|
||||
func progressStage(sceneOut, sceneNameValue, polarizations string) (string, int, int) {
|
||||
pols := requestedPolarizations(polarizations)
|
||||
checks := make([]struct {
|
||||
label string
|
||||
path string
|
||||
}, 0, len(pols)*4)
|
||||
|
||||
for _, pol := range pols {
|
||||
checks = append(checks, struct {
|
||||
label string
|
||||
path string
|
||||
}{"import " + pol, filepath.Join(sceneOut, "*_"+pol+"_slc.sml")})
|
||||
}
|
||||
|
||||
for _, pol := range pols {
|
||||
lower := strings.ToLower(pol)
|
||||
prefix := sceneNameValue + "_" + lower
|
||||
checks = append(checks,
|
||||
struct {
|
||||
label string
|
||||
path string
|
||||
}{"multilook " + pol, filepath.Join(sceneOut, prefix+"_ml.sml")},
|
||||
struct {
|
||||
label string
|
||||
path string
|
||||
}{"filter " + pol, filepath.Join(sceneOut, prefix+"_filt.sml")},
|
||||
struct {
|
||||
label string
|
||||
path string
|
||||
}{"geocode " + pol, filepath.Join(sceneOut, prefix+"_geo.sml")},
|
||||
)
|
||||
}
|
||||
|
||||
done := 0
|
||||
for _, check := range checks {
|
||||
if matched(check.path) {
|
||||
done++
|
||||
continue
|
||||
}
|
||||
return check.label, done, len(checks)
|
||||
}
|
||||
return "finishing", done, len(checks)
|
||||
}
|
||||
|
||||
func completedSceneOutputs(sceneOut, sceneNameValue, polarizations string) ([]completedProduct, bool) {
|
||||
info, err := os.Stat(sceneOut)
|
||||
if err != nil || !info.IsDir() {
|
||||
return nil, false
|
||||
}
|
||||
|
||||
pols := requestedPolarizations(polarizations)
|
||||
products := make([]completedProduct, 0, len(pols))
|
||||
for _, pol := range pols {
|
||||
path, ok := completedGeoProduct(sceneOut, sceneNameValue, pol)
|
||||
if !ok {
|
||||
return nil, false
|
||||
}
|
||||
products = append(products, completedProduct{polarization: pol, path: path})
|
||||
}
|
||||
return products, true
|
||||
}
|
||||
|
||||
func completedGeoProduct(sceneOut, sceneNameValue, polarization string) (string, bool) {
|
||||
lower := strings.ToLower(polarization)
|
||||
exact := filepath.Join(sceneOut, sceneNameValue+"_"+lower+"_geo.sml")
|
||||
if completeGeoSML(exact) {
|
||||
return exact, true
|
||||
}
|
||||
|
||||
matches, err := filepath.Glob(filepath.Join(sceneOut, "*_"+lower+"_geo.sml"))
|
||||
if err != nil {
|
||||
return "", false
|
||||
}
|
||||
sort.Strings(matches)
|
||||
for _, match := range matches {
|
||||
if completeGeoSML(match) {
|
||||
return match, true
|
||||
}
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
func completeGeoSML(path string) bool {
|
||||
info, err := os.Stat(path)
|
||||
if err != nil || info.IsDir() || info.Size() == 0 {
|
||||
return false
|
||||
}
|
||||
|
||||
dataPath := strings.TrimSuffix(path, ".sml")
|
||||
dataInfo, err := os.Stat(dataPath)
|
||||
return err == nil && !dataInfo.IsDir() && dataInfo.Size() > 0
|
||||
}
|
||||
|
||||
func requestedPolarizations(value string) []string {
|
||||
parts := strings.Split(value, ",")
|
||||
out := make([]string, 0, len(parts))
|
||||
for _, part := range parts {
|
||||
pol := strings.ToUpper(strings.TrimSpace(part))
|
||||
if pol == "" {
|
||||
continue
|
||||
}
|
||||
if pol == "ALL" {
|
||||
return []string{"HH", "HV"}
|
||||
}
|
||||
out = append(out, pol)
|
||||
}
|
||||
if len(out) == 0 {
|
||||
return []string{"HH", "HV"}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func matched(pattern string) bool {
|
||||
matches, err := filepath.Glob(pattern)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
return len(matches) > 0
|
||||
}
|
||||
|
||||
func extractTarGz(src, dst string) error {
|
||||
dst = filepath.Clean(dst)
|
||||
tmpDst := dst + ".tmp"
|
||||
|
||||
if err := os.RemoveAll(tmpDst); err != nil {
|
||||
return fmt.Errorf("remove stale temporary extraction directory %s: %w", tmpDst, err)
|
||||
}
|
||||
if err := os.MkdirAll(tmpDst, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := extractTarGzToDir(src, tmpDst); err != nil {
|
||||
_ = os.RemoveAll(tmpDst)
|
||||
return err
|
||||
}
|
||||
if err := os.RemoveAll(dst); err != nil {
|
||||
_ = os.RemoveAll(tmpDst)
|
||||
return fmt.Errorf("replace extraction directory %s: %w", dst, err)
|
||||
}
|
||||
if err := os.Rename(tmpDst, dst); err != nil {
|
||||
_ = os.RemoveAll(tmpDst)
|
||||
return fmt.Errorf("move completed extraction into %s: %w", dst, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func extractTarGzToDir(src, dst string) error {
|
||||
f, err := os.Open(src)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
gz, err := gzip.NewReader(f)
|
||||
if err != nil {
|
||||
return archiveReadError(src, "", err)
|
||||
}
|
||||
defer gz.Close()
|
||||
|
||||
tr := tar.NewReader(gz)
|
||||
var currentEntry string
|
||||
for {
|
||||
hdr, err := tr.Next()
|
||||
if errors.Is(err, io.EOF) {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return archiveReadError(src, currentEntry, err)
|
||||
}
|
||||
currentEntry = hdr.Name
|
||||
target := filepath.Join(dst, filepath.Clean(hdr.Name))
|
||||
if !strings.HasPrefix(target, filepath.Clean(dst)+string(os.PathSeparator)) && filepath.Clean(target) != filepath.Clean(dst) {
|
||||
return fmt.Errorf("archive entry escapes destination: %s", hdr.Name)
|
||||
}
|
||||
switch hdr.Typeflag {
|
||||
case tar.TypeDir:
|
||||
if err := os.MkdirAll(target, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
case tar.TypeReg:
|
||||
if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
out, err := os.OpenFile(target, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o644)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, copyErr := io.Copy(out, tr)
|
||||
closeErr := out.Close()
|
||||
if copyErr != nil {
|
||||
return archiveReadError(src, hdr.Name, copyErr)
|
||||
}
|
||||
if closeErr != nil {
|
||||
return closeErr
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func archiveReadError(src, entry string, err error) error {
|
||||
detail := src
|
||||
if info, statErr := os.Stat(src); statErr == nil {
|
||||
detail = fmt.Sprintf("%s (%d bytes)", src, info.Size())
|
||||
}
|
||||
if entry != "" {
|
||||
detail = fmt.Sprintf("%s while reading %s", detail, entry)
|
||||
}
|
||||
if errors.Is(err, io.ErrUnexpectedEOF) || strings.Contains(strings.ToLower(err.Error()), "unexpected eof") {
|
||||
return fmt.Errorf("archive appears truncated or incomplete: %s: %w; re-copy or re-download the .tar.gz", detail, err)
|
||||
}
|
||||
return fmt.Errorf("read archive %s: %w", detail, err)
|
||||
}
|
||||
|
||||
func sceneName(metaPath string) string {
|
||||
base := filepath.Base(metaPath)
|
||||
base = strings.TrimSuffix(base, ".meta.xml")
|
||||
base = strings.TrimSuffix(base, ".META.XML")
|
||||
return safeName(base)
|
||||
}
|
||||
|
||||
func safeName(s string) string {
|
||||
replacer := strings.NewReplacer(" ", "_", ":", "_", "/", "_", "\\", "_")
|
||||
return replacer.Replace(s)
|
||||
}
|
||||
|
||||
func trimArchiveExt(name string) string {
|
||||
lower := strings.ToLower(name)
|
||||
switch {
|
||||
case strings.HasSuffix(lower, ".tar.gz"):
|
||||
return name[:len(name)-7]
|
||||
case strings.HasSuffix(lower, ".tgz"):
|
||||
return name[:len(name)-4]
|
||||
default:
|
||||
return strings.TrimSuffix(name, filepath.Ext(name))
|
||||
}
|
||||
}
|
||||
|
||||
func defaultIDLRT() string {
|
||||
if v := os.Getenv("IDLRT_PATH"); v != "" {
|
||||
return v
|
||||
}
|
||||
return `C:\Program Files\Harris\ENVI56\IDL88\bin\bin.x86_64\idlrt.exe`
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestCompletedSceneOutputsRequiresGeoSMLAndDataForAllPolarizations(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
sceneName := "GF3_TEST_SCENE"
|
||||
|
||||
writeFile(t, filepath.Join(dir, sceneName+"_hh_geo.sml"), "hh sml")
|
||||
writeFile(t, filepath.Join(dir, sceneName+"_hh_geo"), "hh data")
|
||||
writeFile(t, filepath.Join(dir, sceneName+"_hv_geo.sml"), "hv sml")
|
||||
writeFile(t, filepath.Join(dir, sceneName+"_hv_geo"), "hv data")
|
||||
|
||||
products, ok := completedSceneOutputs(dir, sceneName, "HH,HV")
|
||||
if !ok {
|
||||
t.Fatal("expected completed scene outputs")
|
||||
}
|
||||
if len(products) != 2 {
|
||||
t.Fatalf("expected 2 completed products, got %d", len(products))
|
||||
}
|
||||
}
|
||||
|
||||
func TestCompletedSceneOutputsRejectsMissingDataFile(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
sceneName := "GF3_TEST_SCENE"
|
||||
|
||||
writeFile(t, filepath.Join(dir, sceneName+"_hh_geo.sml"), "hh sml")
|
||||
writeFile(t, filepath.Join(dir, sceneName+"_hv_geo.sml"), "hv sml")
|
||||
writeFile(t, filepath.Join(dir, sceneName+"_hv_geo"), "hv data")
|
||||
|
||||
if _, ok := completedSceneOutputs(dir, sceneName, "HH,HV"); ok {
|
||||
t.Fatal("expected incomplete scene when one geocoded data file is missing")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCompletedSceneOutputsRequiresRequestedPolarizations(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
sceneName := "GF3_TEST_SCENE"
|
||||
|
||||
writeFile(t, filepath.Join(dir, sceneName+"_hh_geo.sml"), "hh sml")
|
||||
writeFile(t, filepath.Join(dir, sceneName+"_hh_geo"), "hh data")
|
||||
|
||||
if _, ok := completedSceneOutputs(dir, sceneName, "HH,HV"); ok {
|
||||
t.Fatal("expected incomplete scene when HV output is missing")
|
||||
}
|
||||
|
||||
if _, ok := completedSceneOutputs(dir, sceneName, "HH"); !ok {
|
||||
t.Fatal("expected completed scene for HH-only request")
|
||||
}
|
||||
}
|
||||
|
||||
func writeFile(t *testing.T, path, value string) {
|
||||
t.Helper()
|
||||
if err := os.WriteFile(path, []byte(value), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user