Rewrite the gateway around health, retries, slicing and a Windows service
Scanner - manifest lookup is now depth-bounded and prunes numeric zoom directories, so a scan no longer walks the whole tile payload - rescans only report a version as updated when something actually changed - overlapping runs are refused instead of interleaving their bookkeeping Hot path - token authorisation loads only that token grants and throttles the last_used_at write instead of touching it on every tile - SQLite runs in WAL with a busy timeout and a real connection pool - the catalog lookup used by /tiles no longer loads every basemap Security - the admin surface requires ADMIN_TOKEN (bearer) or ADMIN_USER/ADMIN_PASSWORD (basic), and the service refuses to bind off-host without credentials - CORS is an explicit allow-list; newly minted tokens never reach a redirect URL - client tokens are redacted from the access log Operations - resource health inspection flags a moved or deleted tile root before a 404 - pushes retry with backoff and can be re-queued by hand; the schedule lives in the database so a restart does not lose it - access log plus /metrics counters Image slicing - tile_jobs drive the GDAL bundled with QGIS via scripts/gdal2tiles-qgis.bat - layer_sources remembers what a layer is made of: submit one source to append a flight, or all of them to rebuild the layer as a VRT mosaic with no tile seams; the console exposes both, including a one-click rebuild - slicing is available to integrations through a narrow slice token scope (submit, poll, idempotent by code, completion callback) Console - rebuilt around a fixed rail and single-view navigation with health and outstanding work leading, instead of one long scrolling page Windows service - native install/start/stop/status with crash recovery, auto-start and a rotating log file, so no NSSM wrapper is needed - start-all.bat and stop-all.bat are removed: they killed the process by port with taskkill, which fought the service restart policy Verified with 49 tests and live runs against real tile data and real GDAL.
This commit is contained in:
+21
-22
@@ -1,32 +1,31 @@
|
||||
# Local machine configuration
|
||||
config/service.env
|
||||
|
||||
# Runtime data and generated assets
|
||||
# --- Runtime data (never commit) -------------------------------------------
|
||||
# data/ holds the production SQLite catalog, the tile tree and the vector
|
||||
# assets. It is machine state, not source.
|
||||
data/
|
||||
logs/
|
||||
temp/
|
||||
.tmp/
|
||||
|
||||
# Go local caches
|
||||
# --- Build and toolchain caches --------------------------------------------
|
||||
.gocache/
|
||||
.gomodcache/
|
||||
.gotmp/
|
||||
.gopath/
|
||||
.gotmp/
|
||||
bin/
|
||||
temp/
|
||||
.tmp/
|
||||
logs/
|
||||
|
||||
# Python cache
|
||||
# --- Machine-specific config -----------------------------------------------
|
||||
# config/service.env carries absolute paths for one machine; the .example file
|
||||
# is the template to copy from.
|
||||
config/service.env
|
||||
|
||||
# --- Tooling artifacts ------------------------------------------------------
|
||||
.claude/
|
||||
.codex/
|
||||
.impeccable/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.pyo
|
||||
|
||||
# Build and test artifacts
|
||||
*.exe
|
||||
*.test
|
||||
*.out
|
||||
*.err
|
||||
*.log
|
||||
*.pid
|
||||
coverage.out
|
||||
|
||||
# OS / editor noise
|
||||
.DS_Store
|
||||
# --- OS/editor noise --------------------------------------------------------
|
||||
Thumbs.db
|
||||
Desktop.ini
|
||||
*.swp
|
||||
+68
@@ -0,0 +1,68 @@
|
||||
# Product
|
||||
|
||||
<!-- impeccable:product-schema 1 -->
|
||||
|
||||
## Platform
|
||||
|
||||
web
|
||||
|
||||
## Stack
|
||||
|
||||
既有代码已确定：Go 1.26 标准库 HTTP 服务 + `modernc.org/sqlite`，运维界面是编译进二进制的单个 Go HTML 模板（`services/api-go/internal/basemapapi/admin_template.go`），无构建步骤、无 JS 框架。
|
||||
|
||||
*（决策来源：用户授权按推荐执行）* 运维界面继续维持"零依赖、单文件内嵌、无前端构建"的约束——单个 exe 可直接部署是本项目最大的运维优势，引入前端工具链会破坏它。允许使用原生 JS 与内联 CSS。
|
||||
|
||||
## Users
|
||||
|
||||
**主要用户：内部 GIS / 运维工程师。** 场景是日常运维与排障——盘点和核对底图、矢量资源，签发和回收访问令牌，处理推送失败，查看扫描与请求的异常。他们熟悉瓦片、XYZ 模板、token 这些概念，不需要教学，需要的是**一眼看到状态、一步完成操作**。
|
||||
|
||||
**次要用户：下游系统对接方。** 通过 token 拉取瓦片/矢量，或接收推送回调；他们主要消费 API 与回调负载，只在排查对接问题时才会打开运维台。
|
||||
|
||||
## Product Purpose
|
||||
|
||||
`basemap-pusher` 是底图与矢量资源的**编目与分发网关**。它不生成数据，只做四件事：扫描既有目录、编目、签发令牌、按令牌放行下载与推送描述。
|
||||
|
||||
存在的意义是把"谁能取哪张底图"这件事从散落的目录和口头约定，收敛成一份可审计的台账。成功 = 运维能自己回答"这张底图现在能不能取、谁有权限、上周谁改过"，而不必去翻服务器文件系统。
|
||||
|
||||
## Positioning
|
||||
|
||||
**它刻意不做切片。** 与上一代成果管理服务（PostgreSQL/PostGIS + Python worker + 切片流水线）相反，本服务把"生产"和"分发"切开：GIS 软件负责出片，产物落到 `data/tiles/<code>/z/x/y.ext`，网关只扫、只编目、只分发。
|
||||
|
||||
这个机制的价值是**部署面极小**：单个 Go 二进制 + 一个 SQLite 文件 + 两个目录。相邻的"成果平台"做不到这一点。
|
||||
|
||||
## Operating Context
|
||||
|
||||
- 数据以**目录即真相**：`BASEMAP_SCAN_ROOT` 下的目录结构就是事实来源，启动扫描或手动重扫即刷新台账。
|
||||
- 两种底图目录风格：带 `manifest.json` 的目录，和裸 `z/x/y.ext` 目录（后者按目录名自动入库）。
|
||||
- 矢量资源在 `VECTOR_SCAN_ROOT` 下递归扫描 `.geojson` / `.json` / `.zip` / `.shp`（shapefile 按需打包下载）。
|
||||
- 访问令牌通过 `Authorization: Bearer`、`X-Access-Token`、`token=` 查询参数投递——查询参数对瓦片客户端（地图库直接拼 URL）是必需的，不是可选项。
|
||||
- 推送是向下游系统发 HTTP 回调，携带资源描述符。
|
||||
- 部署形态：Windows 批处理脚本启停（`start-all.bat` / `stop-all.bat`），默认只绑 `127.0.0.1`。
|
||||
|
||||
## Capabilities and Constraints
|
||||
|
||||
**已确认能力**：目录扫描与编目、默认版本管理、令牌签发/停用/有效期、令牌鉴权取图、矢量下载（shapefile 动态打包）、兼容路由（`/geojson/...`、`/shp/...`、无版本号的瓦片路由）、下游推送、访问日志与 Prometheus 指标（`/metrics`）。管理面支持 HTTP Basic（浏览器）与 Bearer（API）。
|
||||
|
||||
**已确认约束**：
|
||||
- 存储是单文件 SQLite，单实例部署；扫描锁是进程内的。
|
||||
- 运维界面必须能表达**失败与待处理**，而不只是库存数量。
|
||||
- 不引入前端构建步骤。
|
||||
- **明确不做**：栅格切片、重投影、金字塔构建。影像转底图属于上游 GIS 工具的职责。
|
||||
|
||||
**明确未决**：影像切片是否以"作业化调用外部 GDAL"的形式纳入（已记为后续候选，当前不在范围内）。
|
||||
|
||||
## Evidence on Hand
|
||||
|
||||
真实的底图目录与矢量文件在 `data/` 下（`data/tiles/` 含 gaode_image / gaode_shp / google_image，实为 webp 瓦片；`data/geojson/` 含一份约 53MB 的 GeoJSON）。这些是真实数据，可用于验证界面，但**不得**在界面文案里虚构容量、QPS、SLA 或客户信息。
|
||||
|
||||
## Product Principles
|
||||
|
||||
1. **状态先于库存。** 打开界面第一眼要看到"有没有东西在报警、有什么待处理"，而不是资源总数。
|
||||
2. **台账要能验真。** 每一条资源都应能就地验证（看一眼图、复制 XYZ 模板、试拉一张），而不是只列一条记录。
|
||||
3. **危险操作要设防。** 停用令牌、重扫、改默认版本都会影响线上取图，必须有确认与后果说明。
|
||||
4. **生产与分发分离。** 任何新能力都要问一句：这是不是把"生产"塞进了网关？默认答案是不要。
|
||||
5. **小部署面是特性，不是妥协。** 新增依赖前先证明它值得牺牲单文件部署。
|
||||
|
||||
## Accessibility & Inclusion
|
||||
|
||||
内部工具，无正式合规要求。仍应满足：键盘可达、焦点可见、正文对比度 ≥ 4.5:1、不用纯颜色单独承载状态（状态同时有文字）。
|
||||
@@ -1,129 +1,42 @@
|
||||
# map-asset-gateway
|
||||
# Basemap Pusher
|
||||
|
||||
`map-asset-gateway` 是一个面向现有底图瓦片和矢量文件的目录管理与分发服务。
|
||||
An internal catalog and push service for existing basemap tiles, GeoJSON files, and shapefiles.
|
||||
|
||||
它不负责生成瓦片，不负责 GIS 生产流程，也不负责旧版遥感结果处理流水线。它只做这几件事：
|
||||
The service does not generate tiles, does not rebuild vector data, and does not bring back the old result-management pipeline. It only does:
|
||||
|
||||
- 扫描已有的底图瓦片目录
|
||||
- 扫描已有的 GeoJSON / Shapefile 文件
|
||||
- 将元数据写入 SQLite
|
||||
- 为底图和矢量资源签发访问 token
|
||||
- 按 token 提供 catalog、瓦片访问和矢量下载
|
||||
- 将底图或矢量描述推送给下游系统
|
||||
- scan existing basemap tile directories
|
||||
- scan existing vector files
|
||||
- manage metadata
|
||||
- issue tokens
|
||||
- control token access to basemaps and vector assets
|
||||
- push basemap or vector descriptors to downstream systems
|
||||
|
||||
## 技术栈
|
||||
## Stack
|
||||
|
||||
- Go
|
||||
- SQLite
|
||||
- Go 标准库 `net/http`
|
||||
- standard library HTTP server
|
||||
|
||||
## 适用场景
|
||||
## Current Capabilities
|
||||
|
||||
适合已经有离线瓦片目录、GeoJSON、SHP 数据，希望补一层统一管理、授权和分发接口的场景。
|
||||
- scan `BASEMAP_SCAN_ROOT` into SQLite
|
||||
- scan `VECTOR_SCAN_ROOT` into SQLite
|
||||
- support two basemap directory styles
|
||||
- manifest-based directories
|
||||
- raw `z/x/y.ext` tile roots
|
||||
- support vector asset formats
|
||||
- `.geojson`
|
||||
- `.json`
|
||||
- `.shp` plus sidecar files
|
||||
- `.zip` shapefile packages
|
||||
- token-protected basemap catalog, tile access, vector catalog, and vector download
|
||||
- token-protected compatibility routes for default-version tiles, GeoJSON paths, and shapefile paths
|
||||
- admin panel at `GET /admin/`
|
||||
- downstream callback push for basemaps and vectors
|
||||
|
||||
不适合下面这些需求：
|
||||
## Basemap Scan Rules
|
||||
|
||||
- 在线切片或重建瓦片
|
||||
- 栅格处理、预览图生成、图例生成
|
||||
- PostgreSQL / PostGIS 任务流水线
|
||||
- Python worker 驱动的结果管理系统
|
||||
|
||||
## 运行架构
|
||||
|
||||
服务启动时会按下面的顺序运行：
|
||||
|
||||
1. 读取 `config/service.env` 或环境变量
|
||||
2. 打开 SQLite 并自动建表
|
||||
3. 注册默认扫描源
|
||||
4. 如果 `AUTO_SCAN_ON_START=true`，自动扫描底图和矢量目录
|
||||
5. 启动 HTTP API 和管理页
|
||||
|
||||
这是一个单进程服务，没有额外 worker，没有外部数据库依赖。
|
||||
|
||||
## 目录结构
|
||||
|
||||
```text
|
||||
.
|
||||
├─ services/api-go/ Go 服务源码
|
||||
├─ config/ 环境变量配置
|
||||
├─ scripts/ 启动脚本
|
||||
├─ docs/ 接入文档
|
||||
├─ data/ 本地数据目录，不建议提交到 Git
|
||||
├─ logs/ 本地运行日志，不建议提交到 Git
|
||||
└─ temp/ 本地临时文件，不建议提交到 Git
|
||||
```
|
||||
|
||||
说明：
|
||||
|
||||
- `services/api-go` 是当前项目主体。
|
||||
- 历史遗留 demo 已从仓库中移除，避免和当前服务边界混淆。
|
||||
- `data/`、`logs/`、`temp/`、`.gocache/`、`.gomodcache/`、`.gotmp/`、`.gopath/` 都属于本地运行或构建产物。
|
||||
|
||||
## 快速启动
|
||||
|
||||
### 前置条件
|
||||
|
||||
- Go 1.26 或兼容版本
|
||||
- 可写的本地目录用于放 SQLite、扫描根目录和 Go 缓存
|
||||
- Windows 环境下可直接使用仓库自带批处理脚本
|
||||
|
||||
### 配置
|
||||
|
||||
复制示例配置：
|
||||
|
||||
```text
|
||||
config/service.env.example -> config/service.env
|
||||
```
|
||||
|
||||
关键配置项：
|
||||
|
||||
| 变量 | 说明 |
|
||||
| --- | --- |
|
||||
| `API_HOST` | 监听地址 |
|
||||
| `API_PORT` | 监听端口 |
|
||||
| `API_BASE_URL` | API 对外基地址 |
|
||||
| `TILE_PUBLIC_BASE_URL` | 底图下载地址前缀 |
|
||||
| `VECTOR_PUBLIC_BASE_URL` | 矢量下载地址前缀 |
|
||||
| `SQLITE_PATH` | SQLite 文件路径 |
|
||||
| `BASEMAP_SCAN_ROOT` | 底图扫描根目录 |
|
||||
| `VECTOR_SCAN_ROOT` | 矢量扫描根目录 |
|
||||
| `AUTO_SCAN_ON_START` | 启动时是否自动扫描 |
|
||||
|
||||
默认示例见 [config/service.env.example](./config/service.env.example)。
|
||||
|
||||
### 启动
|
||||
|
||||
Windows 下推荐：
|
||||
|
||||
```powershell
|
||||
.\start-all.bat
|
||||
```
|
||||
|
||||
或者直接运行 API：
|
||||
|
||||
```powershell
|
||||
.\scripts\run-api.bat
|
||||
```
|
||||
|
||||
停止：
|
||||
|
||||
```powershell
|
||||
.\stop-all.bat
|
||||
```
|
||||
|
||||
启动后可访问：
|
||||
|
||||
- 管理页: `http://127.0.0.1:8910/admin/`
|
||||
- 健康检查: `http://127.0.0.1:8910/health`
|
||||
- 就绪检查: `http://127.0.0.1:8910/ready`
|
||||
|
||||
## 数据组织与扫描规则
|
||||
|
||||
### 底图目录
|
||||
|
||||
服务支持两种底图目录形式。
|
||||
|
||||
### 1. 带 `manifest.json` 的目录
|
||||
### Manifest-based directory
|
||||
|
||||
```text
|
||||
data/tiles/gaode-image/
|
||||
@@ -133,7 +46,7 @@ data/tiles/gaode-image/
|
||||
1/0/0.png
|
||||
```
|
||||
|
||||
示例 `manifest.json`：
|
||||
Example `manifest.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -151,7 +64,7 @@ data/tiles/gaode-image/
|
||||
}
|
||||
```
|
||||
|
||||
### 2. 直接是 `z/x/y.ext` 的裸目录
|
||||
### Raw tile directory
|
||||
|
||||
```text
|
||||
data/tiles/google-image/
|
||||
@@ -159,22 +72,17 @@ data/tiles/google-image/
|
||||
1/0/0.webp
|
||||
```
|
||||
|
||||
这类目录会自动导入为：
|
||||
This is imported automatically as:
|
||||
|
||||
- `code = 目录名归一化后的值`
|
||||
- `code = directory name`
|
||||
- `version = current`
|
||||
- `type = xyz`
|
||||
|
||||
说明：
|
||||
## Vector Scan Rules
|
||||
|
||||
- `google-image`、`google_image`、`google.image` 最终都会被归一化成类似 `google-image` 的内部 code。
|
||||
- 默认版本路由 `/tiles/{code}/{z}/{x}/{y}.ext` 会解析到该底图当前默认版本。
|
||||
The service scans `VECTOR_SCAN_ROOT` recursively.
|
||||
|
||||
### 矢量目录
|
||||
|
||||
`VECTOR_SCAN_ROOT` 会被递归扫描。
|
||||
|
||||
示例：
|
||||
Examples:
|
||||
|
||||
```text
|
||||
data/geojson/admin-boundary.geojson
|
||||
@@ -184,95 +92,37 @@ data/geojson/landuse/parcel.dbf
|
||||
data/geojson/landuse/parcel.shx
|
||||
```
|
||||
|
||||
扫描规则：
|
||||
Rules:
|
||||
|
||||
- `.geojson` 和 `.json` 会被识别为 `geojson`
|
||||
- `.zip` 会被识别为 `shpzip`
|
||||
- `.shp` 会被识别为 `shapefile`
|
||||
- `shapefile` 下载时会按同名 sidecar 文件动态打包成 zip
|
||||
- `.geojson` and `.json` are imported as `geojson`
|
||||
- `.zip` is imported as `shpzip`
|
||||
- `.shp` is imported as `shapefile`
|
||||
- shapefile downloads are served as zip packages on demand
|
||||
|
||||
## 访问控制
|
||||
## Token Model
|
||||
|
||||
服务使用 token 做资源访问控制。
|
||||
A token can be granted access to:
|
||||
|
||||
一个 token 可以授权：
|
||||
- one or more basemaps
|
||||
- one or more vector assets
|
||||
|
||||
- 一个或多个底图
|
||||
- 一个或多个矢量资源
|
||||
|
||||
支持的 token 传递方式：
|
||||
Supported token delivery:
|
||||
|
||||
- `Authorization: Bearer <token>`
|
||||
- `X-Access-Token: <token>`
|
||||
- 查询参数 `token=`
|
||||
- 查询参数 `access_token=`
|
||||
- query parameter `token=`
|
||||
- query parameter `access_token=`
|
||||
|
||||
## 典型使用流程
|
||||
## API
|
||||
|
||||
### 1. 启动服务并扫描目录
|
||||
|
||||
如果开启了 `AUTO_SCAN_ON_START=true`，服务会在启动时自动扫描。
|
||||
|
||||
也可以通过管理接口手动触发：
|
||||
|
||||
- `POST /api/v1/admin/scan-sources/{code}/run`
|
||||
- `POST /api/v1/admin/vectors/rescan`
|
||||
|
||||
### 2. 创建 token
|
||||
|
||||
示例：
|
||||
|
||||
```json
|
||||
POST /api/v1/admin/tokens
|
||||
{
|
||||
"name": "insar-v2",
|
||||
"basemap_codes": ["google-image"],
|
||||
"vector_codes": ["全国行政区.geojson", "landuse/parcel.shp"]
|
||||
}
|
||||
```
|
||||
|
||||
### 3. 访问 catalog 或资源
|
||||
|
||||
```text
|
||||
GET /api/v1/catalog/basemaps?token=bmp_xxx
|
||||
GET /api/v1/catalog/vectors?token=bmp_xxx
|
||||
GET /tiles/google-image/10/843/388.webp?token=bmp_xxx
|
||||
GET /tiles/google-image/current/10/843/388.webp?token=bmp_xxx
|
||||
GET /geojson/全国行政区.geojson?token=bmp_xxx
|
||||
GET /shp/landuse/parcel.shp?token=bmp_xxx
|
||||
GET /vectors/landuse-parcel/download?token=bmp_xxx
|
||||
```
|
||||
|
||||
说明：
|
||||
|
||||
- `/tiles/{code}/{z}/{x}/{y}.ext` 走默认版本兼容路由
|
||||
- `/tiles/{code}/{version}/{z}/{x}/{y}.ext` 显式指定版本
|
||||
- `/geojson/{...}` 支持按原始相对路径、文件名或内部 code 解析
|
||||
- `/shp/{...}` 会把 `.shp + .dbf + .shx ...` 打成 zip 返回
|
||||
|
||||
### 4. 推送给下游系统
|
||||
|
||||
先注册目标系统：
|
||||
|
||||
- `GET /api/v1/admin/targets`
|
||||
- `POST /api/v1/admin/targets`
|
||||
|
||||
再触发推送：
|
||||
|
||||
- `POST /api/v1/admin/pushes`
|
||||
- `POST /api/v1/admin/vector-pushes`
|
||||
|
||||
推送内容是底图版本或矢量资源的描述信息，不是资源文件本体上传。
|
||||
|
||||
## API 概览
|
||||
|
||||
### 系统接口
|
||||
### System
|
||||
|
||||
- `GET /health`
|
||||
- `GET /ready`
|
||||
- `GET /metrics` (Prometheus text: request, tile, tile-miss and auth counters)
|
||||
- `GET /api/v1/system`
|
||||
|
||||
### 管理接口
|
||||
### Admin
|
||||
|
||||
- `GET /admin/`
|
||||
- `GET /api/v1/admin/dashboard`
|
||||
@@ -293,7 +143,7 @@ GET /vectors/landuse-parcel/download?token=bmp_xxx
|
||||
- `GET /api/v1/admin/vector-push-records`
|
||||
- `POST /api/v1/admin/vector-pushes`
|
||||
|
||||
### Catalog 与下载接口
|
||||
### Catalog and Downloads
|
||||
|
||||
- `GET /api/v1/catalog/basemaps`
|
||||
- `GET /api/v1/catalog/basemaps/{code}`
|
||||
@@ -305,7 +155,197 @@ GET /vectors/landuse-parcel/download?token=bmp_xxx
|
||||
- `GET /shp/{relative/path/to/file.shp}`
|
||||
- `GET /vectors/{code}/download`
|
||||
|
||||
## 数据库表
|
||||
Examples:
|
||||
|
||||
```text
|
||||
GET /tiles/gaode_image/10/843/388.png?token=bmp_xxx
|
||||
GET /tiles/gaode-image/current/10/843/388.png?token=bmp_xxx
|
||||
GET /geojson/全国行政区.geojson?token=bmp_xxx
|
||||
GET /shp/landuse/parcel.shp?token=bmp_xxx
|
||||
GET /vectors/admin-boundary/download?token=bmp_xxx
|
||||
```
|
||||
|
||||
Compatibility route notes:
|
||||
|
||||
- `/tiles/{code}/{z}/{x}/{y}.ext` resolves the current default version of the basemap
|
||||
- `/geojson/{...}` resolves vector assets by original relative path, file name, or internal code
|
||||
- `/shp/{...}` returns `.shp` packages as zip downloads and also supports `.zip` vector packages
|
||||
- token creation accepts vector references by code or by original relative path
|
||||
|
||||
## Slicing API for Other Systems
|
||||
|
||||
Integrations call a narrow, non-admin surface:
|
||||
|
||||
```text
|
||||
POST /api/v1/slice/jobs submit a job -> 202 + the job object
|
||||
GET /api/v1/slice/jobs/{id} poll one job -> 200 + the job object
|
||||
GET /api/v1/slice/layers/{code}/sources the layer's source rasters
|
||||
```
|
||||
|
||||
All three require a service token carrying the **`slice`** scope. Create one
|
||||
through the admin API or console; a slice-only token needs no basemap or vector
|
||||
grants, so it cannot read any tiles:
|
||||
|
||||
```json
|
||||
POST /api/v1/admin/tokens
|
||||
{"name": "integration", "scopes": ["slice"]}
|
||||
```
|
||||
|
||||
- **Accumulating layers** — a layer is the union of its source rasters, tracked in
|
||||
`layer_sources`. Submit **one** new source to append a flight (fast, what
|
||||
overlaps wins per tile), or submit **all** of `GET /api/v1/slice/layers/{code}/sources`
|
||||
to rebuild the layer as a VRT mosaic (per-pixel, no tile seams). The console's
|
||||
影像切片 view lists each layer's sources.
|
||||
- `TILE_JOB_COMMAND` slices one input; `TILE_JOB_VRT_COMMAND` runs first when
|
||||
several sources are submitted. `{inputs}` expands to one argument per source,
|
||||
and `{output_vrt}` is the temporary mosaic. Both ship as wrappers over the GDAL
|
||||
bundled with QGIS (`scripts/gdal2tiles-qgis.bat`,
|
||||
`scripts/gdalbuildvrt-qgis.bat`) because the Python bindings need the OSGeo4W
|
||||
environment; set `GDAL2TILES_QGIS_ROOT` when QGIS is not in `%ProgramFiles%`.
|
||||
The slicer default passes `--exclude`, so a new flight cannot punch transparent
|
||||
holes into the previous imagery.
|
||||
- **Rebuild from the console** — 影像切片 → 图层源清单 → 重建为镶嵌 slices every
|
||||
recorded source into one VRT, reusing the zoom range the layer is already
|
||||
published at. No need to copy paths by hand.
|
||||
- **Known limit** — a layer's source list only grows (there is no delete API yet),
|
||||
and a rebuild never clears the output directory. If a source is ever removed by
|
||||
hand, tiles it used to cover but no longer does stay behind; delete the layer
|
||||
directory and rebuild to clean up.
|
||||
- **Idempotent submit** — sending the same `code` twice returns the existing job
|
||||
instead of queueing a second one, so a retrying caller is safe.
|
||||
- **Completion callback** — pass `callback_url` when submitting and the service
|
||||
POSTs `{"event":"tile.job.finished","job":{...}}` when the job ends. Failures
|
||||
retry on the same 30s/60s backoff as push callbacks.
|
||||
- Jobs run one at a time, oldest first; the admin console's 影像切片 view shows
|
||||
the same queue.
|
||||
## Resource Health, Retries and Slicing
|
||||
|
||||
- **Resource health** — every catalogued tile root and vector file is stat'ed on
|
||||
`HEALTH_CHECK_INTERVAL` and on demand (`POST /api/v1/admin/health/check`).
|
||||
A path that vanished is recorded as `missing` and surfaces in the console's
|
||||
待处理 list instead of waiting for a client 404. Health is tracked separately
|
||||
from `status`: status is a catalog fact, health is whether the bytes are on
|
||||
disk right now.
|
||||
- **Push retries** — a failed callback is retried up to 3 times with backoff
|
||||
(30s, 60s) and stays `retrying` while queued; the schedule lives in the
|
||||
database so a restart does not lose it. A finished record can be re-queued
|
||||
with `POST /api/v1/admin/push-records/{id}/retry` (or
|
||||
`.../vector-push-records/{id}/retry`).
|
||||
- **Image slicing** — `scripts/gdal2tiles-qgis.bat` runs the GDAL bundled with
|
||||
QGIS (its Python bindings need the OSGeo4W environment, so a bare
|
||||
`gdal2tiles.exe` will not work; set `GDAL2TILES_QGIS_ROOT` if QGIS is not in
|
||||
`%ProgramFiles%`). `POST /api/v1/admin/tile-jobs` queues a job that runs
|
||||
`TILE_JOB_COMMAND` (see [config/service.env.example](./config/service.env.example))
|
||||
and, on success, rescans so the sliced basemap appears in the catalog. The
|
||||
service does the orchestration; the raster work is done by GDAL. The service
|
||||
account needs write access to the basemap scan root.
|
||||
## Running as a Windows Service
|
||||
|
||||
The binary manages its own service registration — no NSSM or WinSW wrapper:
|
||||
|
||||
```text
|
||||
basemap-pusher.exe --env-file <path\to\config\service.env> install
|
||||
basemap-pusher.exe start
|
||||
basemap-pusher.exe status
|
||||
basemap-pusher.exe stop
|
||||
basemap-pusher.exe uninstall
|
||||
```
|
||||
|
||||
- `install` bakes `--env-file` into the service command line, because the SCM
|
||||
starts services in `C:\Windows\System32` and the app would otherwise miss its
|
||||
config. Without `--env-file` it looks for `config/service.env` next to the
|
||||
executable and up to two directories above it.
|
||||
- A service has no console, so the standard logger is redirected to
|
||||
`bin\logs\basemap-pusher.log`, rolling to `.1` at 10 MB. Override the path with
|
||||
the `SERVICE_LOG_FILE` environment variable. Without this the service would
|
||||
discard every log line.
|
||||
- Recovery actions are registered on install: restart after 5s, 15s, then 60s,
|
||||
retrying for a day. The service starts automatically at boot.
|
||||
- `install` / `uninstall` / `start` / `stop` need an **elevated** console
|
||||
(Run as Administrator); `status` and the console mode do not.
|
||||
- Started with no verb the binary runs as a normal console app and handles
|
||||
Ctrl+C / SIGTERM with a 15s drain. Under the SCM the service handler owns the
|
||||
same shutdown path.
|
||||
|
||||
Build the binary the service will run (it must not depend on the Go toolchain):
|
||||
|
||||
```text
|
||||
scripts\build-api.bat
|
||||
```
|
||||
|
||||
The script loads `config/service.env` first, so the Go caches and `GOPROXY`
|
||||
match the rest of the project instead of the global defaults. Running `go build`
|
||||
by hand from the repository root will fail: the module lives in
|
||||
`services\api-go`, not at the root.
|
||||
## Access Control
|
||||
|
||||
The admin console and `/api/v1/admin/*` can mint tokens that read every
|
||||
basemap, so they are guarded:
|
||||
|
||||
- configure `ADMIN_USER` + `ADMIN_PASSWORD` for the browser console (HTTP Basic)
|
||||
- configure `ADMIN_TOKEN` for API clients (`Authorization: Bearer <token>`)
|
||||
- with neither set, the service **refuses to start** unless `API_HOST` is a
|
||||
loopback address (`127.0.0.1`, `::1`, `localhost`)
|
||||
|
||||
Cross-origin requests are denied by default; add the exact origins you trust to
|
||||
`CORS_ALLOWED_ORIGINS`. Client tokens never appear in redirect URLs, and tokens
|
||||
carried in a query string are redacted from the access log.
|
||||
## Config
|
||||
|
||||
See [config/service.env.example](./config/service.env.example).
|
||||
|
||||
For `Insar_management_system_v2` integration, see [docs/INSAR_MANAGEMENT_SYSTEM_V2.md](./docs/INSAR_MANAGEMENT_SYSTEM_V2.md).
|
||||
|
||||
Important variables:
|
||||
|
||||
- `API_HOST`
|
||||
- `API_PORT`
|
||||
- `API_BASE_URL`
|
||||
- `TILE_PUBLIC_BASE_URL`
|
||||
- `VECTOR_PUBLIC_BASE_URL`
|
||||
- `SQLITE_PATH`
|
||||
- `BASEMAP_SCAN_ROOT`
|
||||
- `VECTOR_SCAN_ROOT`
|
||||
- `AUTO_SCAN_ON_START`
|
||||
- `ADMIN_USER` / `ADMIN_PASSWORD` (HTTP Basic for the browser console)
|
||||
- `ADMIN_TOKEN` (bearer token for admin API clients)
|
||||
- `CORS_ALLOWED_ORIGINS` (comma-separated allow-list; empty means no CORS)
|
||||
- `TOKEN_DEFAULT_TTL` (lifetime for new tokens, e.g. `720h`; `0` disables expiry)
|
||||
- `HEALTH_CHECK_INTERVAL` (resource availability pass, e.g. `5m`; `0` disables)
|
||||
- `TILE_JOB_COMMAND` (image slicing command template; needs GDAL)
|
||||
- `TILE_JOB_LOG_DIR` (per-job log directory; empty = `<scan root>/.tile-jobs`)
|
||||
|
||||
## Start
|
||||
|
||||
### Windows service (production)
|
||||
|
||||
The service is the supported way to run this on Windows — it starts at boot,
|
||||
restarts on failure and keeps its own log file:
|
||||
|
||||
```text
|
||||
scripts\build-api.bat
|
||||
scripts\service.bat install (elevated)
|
||||
scripts\service.bat start
|
||||
scripts\service.bat status
|
||||
```
|
||||
|
||||
`service.bat` accepts `install | uninstall | start | stop | restart | status`
|
||||
and forwards to `bin\basemap-pusher.exe`. The install / uninstall / start / stop
|
||||
verbs need an **elevated** shell; `status` does not.
|
||||
|
||||
The old `start-all.bat` / `stop-all.bat` launchers were removed: they killed the
|
||||
process by port with `taskkill`, which fought the service's restart policy.
|
||||
|
||||
### Console (development)
|
||||
|
||||
```text
|
||||
scripts\run-api.bat
|
||||
```
|
||||
|
||||
This runs `go run` with the project's own caches, for a quick local loop. It is
|
||||
not supervised and does not survive a reboot.
|
||||
|
||||
## Database Tables
|
||||
|
||||
- `basemaps`
|
||||
- `basemap_versions`
|
||||
@@ -319,80 +359,12 @@ GET /vectors/landuse-parcel/download?token=bmp_xxx
|
||||
- `vector_token_grants`
|
||||
- `vector_push_records`
|
||||
|
||||
## 提交到 Git 前的约定
|
||||
## Design Boundaries
|
||||
|
||||
这个仓库里有很多目录本质上是本地运行数据或缓存，不建议提交到版本库。
|
||||
- no Python worker
|
||||
- no PostgreSQL
|
||||
- no tile generation pipeline
|
||||
- no result artifact management
|
||||
- no raster processing jobs
|
||||
|
||||
默认不建议提交：
|
||||
|
||||
- `data/`
|
||||
- `logs/`
|
||||
- `temp/`
|
||||
- `.tmp/`
|
||||
- `.gocache/`
|
||||
- `.gomodcache/`
|
||||
- `.gotmp/`
|
||||
- `.gopath/`
|
||||
- `config/service.env`
|
||||
|
||||
原因：
|
||||
|
||||
- `data/` 里通常是真实底图、真实 GeoJSON/SHP、以及本地 SQLite 数据库
|
||||
- `logs/` 和 `temp/` 是运行期产物
|
||||
- `.gocache/`、`.gomodcache/`、`.gotmp/`、`.gopath/` 是本地 Go 构建缓存
|
||||
- `config/service.env` 往往包含机器本地路径
|
||||
|
||||
如果需要给新同事一个可运行的仓库，建议提交：
|
||||
|
||||
- 源码
|
||||
- `config/service.env.example`
|
||||
- 文档
|
||||
- 一个最小化的目录约定说明
|
||||
|
||||
而不是直接提交大体积瓦片、数据库和本地缓存。
|
||||
|
||||
## 开发与测试
|
||||
|
||||
在仓库内使用本地 Go 缓存目录跑测试：
|
||||
|
||||
```powershell
|
||||
$env:GOCACHE='C:\path\to\map-asset-gateway\.gocache'
|
||||
$env:GOMODCACHE='C:\path\to\map-asset-gateway\.gomodcache'
|
||||
$env:GOTMPDIR='C:\path\to\map-asset-gateway\.gotmp'
|
||||
$env:GOPATH='C:\path\to\map-asset-gateway\.gopath'
|
||||
$env:GOPROXY='https://goproxy.cn,direct'
|
||||
$env:GOSUMDB='off'
|
||||
go test ./...
|
||||
```
|
||||
|
||||
当前测试主要覆盖：
|
||||
|
||||
- 底图扫描
|
||||
- 矢量扫描
|
||||
- token 鉴权
|
||||
- 默认版本瓦片路由
|
||||
- GeoJSON / SHP 兼容访问路径
|
||||
|
||||
## 相关文档
|
||||
|
||||
- [config/service.env.example](./config/service.env.example)
|
||||
|
||||
## 项目边界
|
||||
|
||||
再次强调，这个项目当前的边界是：
|
||||
|
||||
- 扫描
|
||||
- 建目录
|
||||
- 发 token
|
||||
- 鉴权
|
||||
- 下载
|
||||
- 推送资源描述
|
||||
|
||||
不包括：
|
||||
|
||||
- Python worker
|
||||
- PostgreSQL
|
||||
- PostGIS
|
||||
- 在线切片
|
||||
- 栅格处理任务
|
||||
- 结果资产管理流水线
|
||||
GIS software still produces the tiles and vector files. This project only scans, catalogs, authorizes, downloads, and pushes descriptors.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
APP_ENV=dev
|
||||
|
||||
SERVICE_NAME=map-asset-gateway
|
||||
SERVICE_NAME=basemap-pusher
|
||||
SERVICE_VERSION=0.2.0
|
||||
SERVICE_COMMIT=local
|
||||
|
||||
@@ -12,14 +12,56 @@ API_BASE_URL=http://127.0.0.1:8910
|
||||
TILE_PUBLIC_BASE_URL=http://127.0.0.1:8910
|
||||
VECTOR_PUBLIC_BASE_URL=http://127.0.0.1:8910
|
||||
|
||||
SQLITE_PATH=C:\path\to\map-asset-gateway\data\map-asset-gateway.db
|
||||
BASEMAP_SCAN_ROOT=C:\path\to\map-asset-gateway\data\tiles
|
||||
VECTOR_SCAN_ROOT=C:\path\to\map-asset-gateway\data\geojson
|
||||
SQLITE_PATH=Z:\Code\tile-server\data\basemap-pusher.db
|
||||
BASEMAP_SCAN_ROOT=Z:\Code\tile-server\data\tiles
|
||||
VECTOR_SCAN_ROOT=Z:\Code\tile-server\data\geojson
|
||||
AUTO_SCAN_ON_START=true
|
||||
|
||||
GOCACHE=C:\path\to\map-asset-gateway\.gocache
|
||||
GOMODCACHE=C:\path\to\map-asset-gateway\.gomodcache
|
||||
GOTMPDIR=C:\path\to\map-asset-gateway\.gotmp
|
||||
GOPATH=C:\path\to\map-asset-gateway\.gopath
|
||||
GOCACHE=Z:\Code\tile-server\.gocache
|
||||
GOMODCACHE=Z:\Code\tile-server\.gomodcache
|
||||
GOTMPDIR=Z:\Code\tile-server\.gotmp
|
||||
GOPATH=Z:\Code\tile-server\.gopath
|
||||
GOPROXY=https://goproxy.cn,direct
|
||||
GOSUMDB=off
|
||||
|
||||
# --- Admin access ---------------------------------------------------------
|
||||
# The admin console can mint tokens that read every basemap, so it must be
|
||||
# protected whenever the service is reachable off-host. Binding to a non
|
||||
# loopback API_HOST without credentials below makes startup fail on purpose.
|
||||
#
|
||||
# Browser console: set ADMIN_USER/ADMIN_PASSWORD (HTTP Basic).
|
||||
# API clients: set ADMIN_TOKEN (Authorization: Bearer <token>).
|
||||
# Leaving both empty is only allowed on 127.0.0.1 / ::1 / localhost.
|
||||
#ADMIN_USER=ops
|
||||
#ADMIN_PASSWORD=change-me
|
||||
#ADMIN_TOKEN=change-me-too
|
||||
|
||||
# Comma-separated list of browser origins allowed to call the API. Empty means
|
||||
# no cross-origin access at all; "*" allows every origin.
|
||||
#CORS_ALLOWED_ORIGINS=https://maps.example.com
|
||||
|
||||
# Lifetime applied to newly created tokens when no explicit expiry is given.
|
||||
# Use 0 to create tokens that never expire.
|
||||
TOKEN_DEFAULT_TTL=720h
|
||||
|
||||
# --- Resource health & tile jobs -----------------------------------------
|
||||
# How often catalogued paths are re-stat'ed so a moved/deleted resource is
|
||||
# flagged before a client hits a 404. Use 0 to disable the periodic pass
|
||||
# (the "立即巡检资源" button still works).
|
||||
HEALTH_CHECK_INTERVAL=5m
|
||||
|
||||
# Command template that slices a raster into XYZ tiles. Placeholders:
|
||||
# {input} {output} {zmin} {zmax} {tilesize} {format} {code}
|
||||
# scripts/gdal2tiles-qgis.bat locates the GDAL bundled with QGIS (the Python
|
||||
# bindings need the OSGeo4W environment, so a bare gdal2tiles.exe will not work).
|
||||
TILE_JOB_COMMAND=D:\Code\tile-server\scripts\gdal2tiles-qgis.bat -p mercator -z {zmin}-{zmax} --xyz -w none --tilesize={tilesize} --exclude {input} {output}
|
||||
|
||||
# Where per-job stdout/stderr is written. Empty means <scan root>/.tile-jobs.
|
||||
#TILE_JOB_LOG_DIR=
|
||||
|
||||
# Service tokens may carry the "slice" scope so an integration can submit tile
|
||||
# jobs through POST /api/v1/slice/jobs without holding the admin credential.
|
||||
|
||||
# 多源镶嵌：提交多个源影像时，先用它拼成一张 VRT 再切片（逐像素拼接，
|
||||
# 消除重叠区的瓦片接缝）。{inputs} 会展开成每个源各一个参数。
|
||||
TILE_JOB_VRT_COMMAND=D:\Code\tile-server\scripts\gdalbuildvrt-qgis.bat -resolution highest {output_vrt} {inputs}
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
# Demo Scripts
|
||||
|
||||
This folder contains runnable demos for the internal remote-sensing result service.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- PostgreSQL and PostGIS are available to the API through `config/service.env`.
|
||||
- The Python environment is available at `PYTHON_EXE`.
|
||||
- Go is installed and can run `go run`.
|
||||
|
||||
## Service Startup
|
||||
|
||||
You can start the stack manually from the repository root:
|
||||
|
||||
```powershell
|
||||
.\start-all.bat
|
||||
```
|
||||
|
||||
Or use the self-contained full-flow demo to start a temporary stack on demo ports:
|
||||
|
||||
```powershell
|
||||
.\demo\demo_full_flow.ps1 -StartStack -StartCallbackReceiver
|
||||
```
|
||||
|
||||
## Demo List
|
||||
|
||||
- `demo_import_and_slice.ps1`
|
||||
Imports a raster result, binds a style, and triggers COG, preview, legend, and tile generation.
|
||||
- `demo_register_target_system.ps1`
|
||||
Creates or updates a callback-based target system.
|
||||
- `demo_push_result.ps1`
|
||||
Pushes an existing managed result to a target system or callback URL.
|
||||
- `demo_geojson_flow.ps1`
|
||||
Registers a GeoJSON resource and optionally pushes it through the same delivery interface.
|
||||
- `demo_full_flow.ps1`
|
||||
Runs the main end-to-end flow: sample raster, import, slice, target registration, and callback push.
|
||||
- `demo_callback_receiver.py`
|
||||
Small local receiver for inspecting push payloads.
|
||||
|
||||
## Suggested Order
|
||||
|
||||
1. Start the services manually, or run `demo_full_flow.ps1 -StartStack -StartCallbackReceiver`.
|
||||
2. Run `demo_import_and_slice.ps1` to create a managed result.
|
||||
3. Run `demo_register_target_system.ps1` to register a callback receiver.
|
||||
4. Run `demo_push_result.ps1` to publish the managed result.
|
||||
5. Run `demo_geojson_flow.ps1 -Push` if you want to verify the retained GeoJSON delivery path.
|
||||
|
||||
## Example Commands
|
||||
|
||||
```powershell
|
||||
.\demo\demo_import_and_slice.ps1
|
||||
.\demo\demo_register_target_system.ps1 -CallbackUrl http://127.0.0.1:8920/callback
|
||||
.\demo\demo_push_result.ps1 -ResultCode demo_result_20260406_120000 -TargetSystemCode demo_callback
|
||||
.\demo\demo_geojson_flow.ps1 -Push -CallbackUrl http://127.0.0.1:8920/callback
|
||||
.\demo\demo_full_flow.ps1 -StartStack -StartCallbackReceiver
|
||||
```
|
||||
|
||||
## Output
|
||||
|
||||
Each demo prints a JSON summary that includes the main identifiers and URLs. Temporary run data is written under `temp/demo-runs/`.
|
||||
|
||||
`source_system_code` is intentionally left empty in the demos. If you want to populate it, seed the `source_systems` table first.
|
||||
+574
@@ -0,0 +1,574 @@
|
||||
param(
|
||||
[string]$EnvFile = (Join-Path $PSScriptRoot "..\config\service.env")
|
||||
)
|
||||
|
||||
$script:RsDemoRepoRoot = (Resolve-Path (Join-Path $PSScriptRoot "..")).Path
|
||||
$script:RsDemoProcesses = @()
|
||||
$script:RsDemoEnvLoaded = $false
|
||||
|
||||
function Get-RsRepoRoot {
|
||||
return $script:RsDemoRepoRoot
|
||||
}
|
||||
|
||||
function Import-RsDemoEnvironment {
|
||||
param([string]$Path = $EnvFile)
|
||||
|
||||
if ($script:RsDemoEnvLoaded) {
|
||||
return
|
||||
}
|
||||
|
||||
if (Test-Path -LiteralPath $Path) {
|
||||
foreach ($rawLine in (Get-Content -LiteralPath $Path)) {
|
||||
$line = $rawLine.Trim()
|
||||
if (-not $line -or $line.StartsWith("#")) {
|
||||
continue
|
||||
}
|
||||
if ($line.StartsWith("export ")) {
|
||||
$line = $line.Substring(7).Trim()
|
||||
}
|
||||
$parts = $line.Split("=", 2)
|
||||
if ($parts.Count -ne 2) {
|
||||
continue
|
||||
}
|
||||
$key = $parts[0].Trim()
|
||||
$value = $parts[1].Trim().Trim("`"").Trim("'")
|
||||
if (-not $key) {
|
||||
continue
|
||||
}
|
||||
if (-not (Test-Path "Env:$key")) {
|
||||
Set-Item -Path "Env:$key" -Value $value
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (-not $env:PYTHON_EXE) {
|
||||
$env:PYTHON_EXE = "C:\Users\Administrator\.conda\envs\InSAR\python.exe"
|
||||
}
|
||||
if (-not $env:API_BASE_URL) {
|
||||
$env:API_BASE_URL = "http://127.0.0.1:8910"
|
||||
}
|
||||
if (-not $env:TILE_STATIC_BASE_URL) {
|
||||
$env:TILE_STATIC_BASE_URL = "http://127.0.0.1:8900"
|
||||
}
|
||||
if (-not $env:RS_API_BASE_URL) {
|
||||
$env:RS_API_BASE_URL = $env:API_BASE_URL
|
||||
}
|
||||
if (-not $env:PYTHONIOENCODING) {
|
||||
$env:PYTHONIOENCODING = "utf-8"
|
||||
}
|
||||
if (-not $env:NO_PAUSE) {
|
||||
$env:NO_PAUSE = "1"
|
||||
}
|
||||
|
||||
$script:RsDemoEnvLoaded = $true
|
||||
}
|
||||
|
||||
function Get-RsApiBaseUrl {
|
||||
Import-RsDemoEnvironment
|
||||
return ([string]$env:API_BASE_URL).TrimEnd("/")
|
||||
}
|
||||
|
||||
function Get-RsTileBaseUrl {
|
||||
Import-RsDemoEnvironment
|
||||
return ([string]$env:TILE_STATIC_BASE_URL).TrimEnd("/")
|
||||
}
|
||||
|
||||
function Get-RsPythonExe {
|
||||
Import-RsDemoEnvironment
|
||||
if (-not (Test-Path -LiteralPath $env:PYTHON_EXE)) {
|
||||
throw "Python interpreter not found: $($env:PYTHON_EXE)"
|
||||
}
|
||||
return $env:PYTHON_EXE
|
||||
}
|
||||
|
||||
function Get-RsGoCommand {
|
||||
return (Get-Command go -ErrorAction Stop).Source
|
||||
}
|
||||
|
||||
function New-RsRunContext {
|
||||
param([string]$Prefix = "demo")
|
||||
|
||||
$runId = "{0}_{1}" -f $Prefix, (Get-Date -Format "yyyyMMdd_HHmmss")
|
||||
$runDir = Join-Path (Get-RsRepoRoot) ("temp\demo-runs\" + $runId)
|
||||
$logDir = Join-Path $runDir "logs"
|
||||
$outputDir = Join-Path $runDir "output"
|
||||
|
||||
New-Item -ItemType Directory -Force -Path $runDir | Out-Null
|
||||
New-Item -ItemType Directory -Force -Path $logDir | Out-Null
|
||||
New-Item -ItemType Directory -Force -Path $outputDir | Out-Null
|
||||
|
||||
return [pscustomobject]@{
|
||||
RunID = $runId
|
||||
RunDirectory = $runDir
|
||||
LogDirectory = $logDir
|
||||
OutputDirectory = $outputDir
|
||||
}
|
||||
}
|
||||
|
||||
function ConvertTo-RsFileUri {
|
||||
param([Parameter(Mandatory)] [string]$Path)
|
||||
|
||||
$fullPath = [System.IO.Path]::GetFullPath($Path)
|
||||
return ([System.Uri]$fullPath).AbsoluteUri
|
||||
}
|
||||
|
||||
function Encode-RsURIComponent {
|
||||
param([Parameter(Mandatory)] [string]$Value)
|
||||
|
||||
return [System.Uri]::EscapeDataString($Value)
|
||||
}
|
||||
|
||||
function Wait-RsUrl {
|
||||
param(
|
||||
[Parameter(Mandatory)] [string]$Url,
|
||||
[int]$TimeoutSec = 60
|
||||
)
|
||||
|
||||
$deadline = (Get-Date).AddSeconds($TimeoutSec)
|
||||
do {
|
||||
try {
|
||||
Invoke-RestMethod -Uri $Url -Method Get -TimeoutSec 5 | Out-Null
|
||||
return
|
||||
} catch {
|
||||
Start-Sleep -Milliseconds 500
|
||||
}
|
||||
} while ((Get-Date) -lt $deadline)
|
||||
|
||||
throw "Timed out waiting for $Url"
|
||||
}
|
||||
|
||||
function Invoke-RsApi {
|
||||
param(
|
||||
[Parameter(Mandatory)] [string]$Method,
|
||||
[Parameter(Mandatory)] [string]$Path,
|
||||
[object]$Body,
|
||||
[int]$TimeoutSec = 30
|
||||
)
|
||||
|
||||
Import-RsDemoEnvironment
|
||||
|
||||
$uri = if ($Path -match "^https?://") {
|
||||
$Path
|
||||
} else {
|
||||
"{0}{1}" -f (Get-RsApiBaseUrl), $Path
|
||||
}
|
||||
|
||||
$params = @{
|
||||
Uri = $uri
|
||||
Method = $Method
|
||||
TimeoutSec = $TimeoutSec
|
||||
Headers = @{
|
||||
Accept = "application/json"
|
||||
}
|
||||
}
|
||||
|
||||
if ($PSBoundParameters.ContainsKey("Body") -and $null -ne $Body) {
|
||||
$params.ContentType = "application/json"
|
||||
$params.Body = ($Body | ConvertTo-Json -Depth 32)
|
||||
}
|
||||
|
||||
try {
|
||||
$response = Invoke-RestMethod @params
|
||||
} catch {
|
||||
$message = $_.Exception.Message
|
||||
$webResponse = $_.Exception.Response
|
||||
if ($webResponse) {
|
||||
try {
|
||||
$stream = $webResponse.GetResponseStream()
|
||||
if ($stream) {
|
||||
$reader = New-Object System.IO.StreamReader($stream)
|
||||
$rawBody = $reader.ReadToEnd()
|
||||
$reader.Dispose()
|
||||
if ($rawBody) {
|
||||
try {
|
||||
$errorEnvelope = $rawBody | ConvertFrom-Json
|
||||
if ($errorEnvelope.error) {
|
||||
$message = "{0} {1}" -f $errorEnvelope.error.code, $errorEnvelope.error.message
|
||||
} else {
|
||||
$message = $rawBody
|
||||
}
|
||||
} catch {
|
||||
$message = $rawBody
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
}
|
||||
}
|
||||
throw "API call failed: $Method $uri`n$message"
|
||||
}
|
||||
|
||||
if ($null -eq $response) {
|
||||
return $null
|
||||
}
|
||||
if ($response.error) {
|
||||
throw "API error: $Method $uri => $($response.error.code) $($response.error.message)"
|
||||
}
|
||||
return $response.data
|
||||
}
|
||||
|
||||
function Wait-RsJob {
|
||||
param(
|
||||
[Parameter(Mandatory)] [string]$JobID,
|
||||
[int]$TimeoutSec = 300,
|
||||
[int]$PollIntervalSec = 2
|
||||
)
|
||||
|
||||
$deadline = (Get-Date).AddSeconds($TimeoutSec)
|
||||
do {
|
||||
$job = Invoke-RsApi -Method "GET" -Path ("/api/v1/jobs/{0}" -f $JobID)
|
||||
if ($job.status -eq "succeeded") {
|
||||
return $job
|
||||
}
|
||||
if ($job.status -in @("failed", "cancelled")) {
|
||||
throw "Job $JobID finished with status=$($job.status) error=$($job.last_error_message)"
|
||||
}
|
||||
Start-Sleep -Seconds $PollIntervalSec
|
||||
} while ((Get-Date) -lt $deadline)
|
||||
|
||||
throw "Timed out waiting for job $JobID"
|
||||
}
|
||||
|
||||
function Get-RsStyleByCode {
|
||||
param(
|
||||
[Parameter(Mandatory)] [string]$ProjectCode,
|
||||
[Parameter(Mandatory)] [string]$Code,
|
||||
[switch]$AllowMissing
|
||||
)
|
||||
|
||||
$project = Encode-RsURIComponent $ProjectCode
|
||||
$keyword = Encode-RsURIComponent $Code
|
||||
$items = @(Invoke-RsApi -Method "GET" -Path "/api/v1/styles?project_code=$project&keyword=$keyword")
|
||||
$match = $items | Where-Object { $_.code -eq $Code } | Select-Object -First 1
|
||||
if ($match) {
|
||||
return $match
|
||||
}
|
||||
if ($AllowMissing) {
|
||||
return $null
|
||||
}
|
||||
throw "Style not found: $Code"
|
||||
}
|
||||
|
||||
function Get-RsTargetSystemByCode {
|
||||
param(
|
||||
[Parameter(Mandatory)] [string]$ProjectCode,
|
||||
[Parameter(Mandatory)] [string]$Code,
|
||||
[switch]$AllowMissing
|
||||
)
|
||||
|
||||
$project = Encode-RsURIComponent $ProjectCode
|
||||
$items = @(Invoke-RsApi -Method "GET" -Path "/api/v1/target-systems?project_code=$project")
|
||||
$match = $items | Where-Object { $_.code -eq $Code } | Select-Object -First 1
|
||||
if ($match) {
|
||||
return $match
|
||||
}
|
||||
if ($AllowMissing) {
|
||||
return $null
|
||||
}
|
||||
throw "Target system not found: $Code"
|
||||
}
|
||||
|
||||
function Get-RsResultByCode {
|
||||
param(
|
||||
[Parameter(Mandatory)] [string]$ProjectCode,
|
||||
[Parameter(Mandatory)] [string]$Code,
|
||||
[switch]$AllowMissing
|
||||
)
|
||||
|
||||
$project = Encode-RsURIComponent $ProjectCode
|
||||
$keyword = Encode-RsURIComponent $Code
|
||||
$items = @(Invoke-RsApi -Method "GET" -Path "/api/v1/projects/$project/results?keyword=$keyword")
|
||||
$match = $items | Where-Object { $_.code -eq $Code } | Select-Object -First 1
|
||||
if ($match) {
|
||||
return Invoke-RsApi -Method "GET" -Path ("/api/v1/results/{0}" -f $match.id)
|
||||
}
|
||||
if ($AllowMissing) {
|
||||
return $null
|
||||
}
|
||||
throw "Result not found: $Code"
|
||||
}
|
||||
|
||||
function New-RsDemoStyleSpec {
|
||||
param(
|
||||
[string]$StyleID = "demo_continuous_style",
|
||||
[string]$Name = "Demo Continuous Style",
|
||||
[string]$Scope = "project",
|
||||
[double]$MinValue = -1.0,
|
||||
[double]$MaxValue = 1.0,
|
||||
[int]$TileSize = 256,
|
||||
[string]$TileFormat = "webp",
|
||||
[int]$MinZoom = 0,
|
||||
[int]$MaxZoom = 4
|
||||
)
|
||||
|
||||
return @{
|
||||
spec_version = "1.0"
|
||||
id = $StyleID
|
||||
name = $Name
|
||||
version = 1
|
||||
scope = $Scope
|
||||
render = @{
|
||||
mode = "continuous"
|
||||
band = 1
|
||||
resampling = "bilinear"
|
||||
value_domain = @{
|
||||
min = $MinValue
|
||||
max = $MaxValue
|
||||
}
|
||||
color_map = @{
|
||||
type = "linear"
|
||||
stops = @(
|
||||
@{ value = $MinValue; color = "#8c510a"; label = "low" },
|
||||
@{ value = (($MinValue + $MaxValue) / 2.0); color = "#f6e8c3"; label = "mid" },
|
||||
@{ value = $MaxValue; color = "#01665e"; label = "high" }
|
||||
)
|
||||
null_color = "#00000000"
|
||||
}
|
||||
}
|
||||
legend = @{
|
||||
title = $Name
|
||||
orientation = "horizontal"
|
||||
precision = 2
|
||||
}
|
||||
output = @{
|
||||
tile_format = $TileFormat
|
||||
tile_size = $TileSize
|
||||
min_zoom = $MinZoom
|
||||
max_zoom = $MaxZoom
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function Ensure-RsStyleVersion {
|
||||
param(
|
||||
[Parameter(Mandatory)] [string]$ProjectCode,
|
||||
[Parameter(Mandatory)] [string]$StyleCode,
|
||||
[string]$StyleName = "Demo Continuous Style",
|
||||
[hashtable]$StyleSpec,
|
||||
[switch]$CreateNewVersion
|
||||
)
|
||||
|
||||
$style = Get-RsStyleByCode -ProjectCode $ProjectCode -Code $StyleCode -AllowMissing
|
||||
if (-not $style) {
|
||||
$style = Invoke-RsApi -Method "POST" -Path "/api/v1/styles" -Body @{
|
||||
project_code = $ProjectCode
|
||||
scope = "project"
|
||||
code = $StyleCode
|
||||
name = $StyleName
|
||||
created_by = "demo"
|
||||
}
|
||||
}
|
||||
|
||||
if (-not $CreateNewVersion -and $style.latest_version_id) {
|
||||
return Invoke-RsApi -Method "GET" -Path ("/api/v1/style-versions/{0}" -f $style.latest_version_id)
|
||||
}
|
||||
|
||||
if (-not $StyleSpec) {
|
||||
$StyleSpec = New-RsDemoStyleSpec -StyleID $StyleCode -Name $StyleName
|
||||
}
|
||||
|
||||
return Invoke-RsApi -Method "POST" -Path ("/api/v1/styles/{0}/versions" -f $style.id) -Body @{
|
||||
style_spec = $StyleSpec
|
||||
}
|
||||
}
|
||||
|
||||
function Ensure-RsTargetSystem {
|
||||
param(
|
||||
[Parameter(Mandatory)] [string]$ProjectCode,
|
||||
[Parameter(Mandatory)] [string]$Code,
|
||||
[Parameter(Mandatory)] [string]$Name,
|
||||
[Parameter(Mandatory)] [string]$CallbackUrl,
|
||||
[string]$CallbackMethod = "POST",
|
||||
[hashtable]$CallbackHeaders,
|
||||
[int]$TimeoutSeconds = 30,
|
||||
[bool]$Enabled = $true,
|
||||
[string]$CreatedBy = "demo"
|
||||
)
|
||||
|
||||
$body = @{
|
||||
code = $Code
|
||||
name = $Name
|
||||
callback_url = $CallbackUrl
|
||||
callback_method = $CallbackMethod
|
||||
timeout_seconds = $TimeoutSeconds
|
||||
enabled = $Enabled
|
||||
callback_headers = $CallbackHeaders
|
||||
}
|
||||
|
||||
$existing = Get-RsTargetSystemByCode -ProjectCode $ProjectCode -Code $Code -AllowMissing
|
||||
if ($existing) {
|
||||
return Invoke-RsApi -Method "PATCH" -Path ("/api/v1/target-systems/{0}" -f $existing.id) -Body $body
|
||||
}
|
||||
|
||||
$body.project_code = $ProjectCode
|
||||
$body.created_by = $CreatedBy
|
||||
return Invoke-RsApi -Method "POST" -Path "/api/v1/target-systems" -Body $body
|
||||
}
|
||||
|
||||
function Start-RsBackgroundProcess {
|
||||
param(
|
||||
[Parameter(Mandatory)] [string]$FilePath,
|
||||
[string[]]$ArgumentList,
|
||||
[Parameter(Mandatory)] [string]$WorkingDirectory,
|
||||
[Parameter(Mandatory)] [string]$Name,
|
||||
[Parameter(Mandatory)] [string]$LogDirectory
|
||||
)
|
||||
|
||||
New-Item -ItemType Directory -Force -Path $LogDirectory | Out-Null
|
||||
|
||||
$stdout = Join-Path $LogDirectory ($Name + ".stdout.log")
|
||||
$stderr = Join-Path $LogDirectory ($Name + ".stderr.log")
|
||||
$process = Start-Process -FilePath $FilePath `
|
||||
-ArgumentList $ArgumentList `
|
||||
-WorkingDirectory $WorkingDirectory `
|
||||
-PassThru `
|
||||
-RedirectStandardOutput $stdout `
|
||||
-RedirectStandardError $stderr
|
||||
|
||||
$script:RsDemoProcesses += $process
|
||||
return $process
|
||||
}
|
||||
|
||||
function Stop-RsBackgroundProcesses {
|
||||
param([object[]]$Processes = $script:RsDemoProcesses)
|
||||
|
||||
foreach ($process in $Processes) {
|
||||
if ($process -and -not $process.HasExited) {
|
||||
Stop-Process -Id $process.Id -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function Start-RsDemoStack {
|
||||
param(
|
||||
[int]$ApiPort = 28910,
|
||||
[int]$TilePort = 28900,
|
||||
[switch]$IncludeWorker,
|
||||
[string]$RunID = ("stack_" + (Get-Date -Format "yyyyMMdd_HHmmss"))
|
||||
)
|
||||
|
||||
Import-RsDemoEnvironment
|
||||
|
||||
$runDir = Join-Path (Get-RsRepoRoot) ("temp\demo-runs\" + $RunID)
|
||||
$logDir = Join-Path $runDir "logs"
|
||||
New-Item -ItemType Directory -Force -Path $runDir | Out-Null
|
||||
New-Item -ItemType Directory -Force -Path $logDir | Out-Null
|
||||
|
||||
$apiBaseUrl = "http://127.0.0.1:$ApiPort"
|
||||
$tileBaseUrl = "http://127.0.0.1:$TilePort"
|
||||
|
||||
$env:API_HOST = "127.0.0.1"
|
||||
$env:API_PORT = "$ApiPort"
|
||||
$env:API_BASE_URL = $apiBaseUrl
|
||||
$env:TILE_HOST = "127.0.0.1"
|
||||
$env:TILE_PORT = "$TilePort"
|
||||
$env:TILE_STATIC_BASE_URL = $tileBaseUrl
|
||||
$env:RS_API_BASE_URL = $apiBaseUrl
|
||||
$env:NO_PAUSE = "1"
|
||||
|
||||
$pythonExe = Get-RsPythonExe
|
||||
$goExe = Get-RsGoCommand
|
||||
|
||||
$tileProcess = Start-RsBackgroundProcess `
|
||||
-FilePath $pythonExe `
|
||||
-ArgumentList @((Join-Path (Get-RsRepoRoot) "server.py")) `
|
||||
-WorkingDirectory (Get-RsRepoRoot) `
|
||||
-Name "tile-server" `
|
||||
-LogDirectory $logDir
|
||||
|
||||
$apiProcess = Start-RsBackgroundProcess `
|
||||
-FilePath $goExe `
|
||||
-ArgumentList @("run", ".\cmd\api") `
|
||||
-WorkingDirectory (Join-Path (Get-RsRepoRoot) "services\api-go") `
|
||||
-Name "api-service" `
|
||||
-LogDirectory $logDir
|
||||
|
||||
Wait-RsUrl -Url "$tileBaseUrl/health" -TimeoutSec 90
|
||||
Wait-RsUrl -Url "$apiBaseUrl/ready" -TimeoutSec 120
|
||||
|
||||
$workerProcess = $null
|
||||
if ($IncludeWorker) {
|
||||
$workerProcess = Start-RsBackgroundProcess `
|
||||
-FilePath $pythonExe `
|
||||
-ArgumentList @((Join-Path (Get-RsRepoRoot) "services\worker-py\main.py")) `
|
||||
-WorkingDirectory (Get-RsRepoRoot) `
|
||||
-Name "python-worker" `
|
||||
-LogDirectory $logDir
|
||||
Start-Sleep -Seconds 1
|
||||
}
|
||||
|
||||
$processes = @($tileProcess, $apiProcess)
|
||||
if ($workerProcess) {
|
||||
$processes += $workerProcess
|
||||
}
|
||||
|
||||
return [pscustomobject]@{
|
||||
RunID = $RunID
|
||||
RunDirectory = $runDir
|
||||
LogDirectory = $logDir
|
||||
ApiBaseUrl = $apiBaseUrl
|
||||
TileBaseUrl = $tileBaseUrl
|
||||
Processes = $processes
|
||||
}
|
||||
}
|
||||
|
||||
function Start-RsCallbackReceiver {
|
||||
param(
|
||||
[int]$Port = 28920,
|
||||
[string]$Path = "/callback",
|
||||
[string]$RunID = ("callback_" + (Get-Date -Format "yyyyMMdd_HHmmss"))
|
||||
)
|
||||
|
||||
$runDir = Join-Path (Get-RsRepoRoot) ("temp\demo-runs\" + $RunID)
|
||||
$logDir = Join-Path $runDir "logs"
|
||||
$outputDir = Join-Path $runDir "callback-records"
|
||||
New-Item -ItemType Directory -Force -Path $logDir | Out-Null
|
||||
New-Item -ItemType Directory -Force -Path $outputDir | Out-Null
|
||||
|
||||
$pythonExe = Get-RsPythonExe
|
||||
$process = Start-RsBackgroundProcess `
|
||||
-FilePath $pythonExe `
|
||||
-ArgumentList @(
|
||||
(Join-Path $PSScriptRoot "demo_callback_receiver.py"),
|
||||
"--host", "127.0.0.1",
|
||||
"--port", "$Port",
|
||||
"--path", $Path,
|
||||
"--output-dir", $outputDir
|
||||
) `
|
||||
-WorkingDirectory (Get-RsRepoRoot) `
|
||||
-Name "callback-receiver" `
|
||||
-LogDirectory $logDir
|
||||
|
||||
$callbackUrl = "http://127.0.0.1:$Port$Path"
|
||||
Wait-RsUrl -Url "http://127.0.0.1:$Port/health" -TimeoutSec 30
|
||||
|
||||
return [pscustomobject]@{
|
||||
RunID = $RunID
|
||||
RunDirectory = $runDir
|
||||
LogDirectory = $logDir
|
||||
OutputDirectory = $outputDir
|
||||
CallbackURL = $callbackUrl
|
||||
Process = $process
|
||||
}
|
||||
}
|
||||
|
||||
function Get-RsLatestCallbackRecord {
|
||||
param([Parameter(Mandatory)] [string]$OutputDirectory)
|
||||
|
||||
if (-not (Test-Path -LiteralPath $OutputDirectory)) {
|
||||
return $null
|
||||
}
|
||||
|
||||
$file = Get-ChildItem -LiteralPath $OutputDirectory -Filter "*.json" |
|
||||
Sort-Object LastWriteTimeUtc -Descending |
|
||||
Select-Object -First 1
|
||||
|
||||
if (-not $file) {
|
||||
return $null
|
||||
}
|
||||
|
||||
return [pscustomobject]@{
|
||||
Path = $file.FullName
|
||||
Payload = (Get-Content -LiteralPath $file.FullName -Raw | ConvertFrom-Json)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def bootstrap_gdal_env() -> None:
|
||||
env_root = Path(sys.executable).resolve().parent
|
||||
gdal_paths = [
|
||||
env_root / "Lib" / "site-packages" / "rasterio" / "gdal_data",
|
||||
env_root / "Library" / "share" / "gdal",
|
||||
env_root / "share" / "gdal",
|
||||
]
|
||||
proj_paths = [
|
||||
env_root / "Lib" / "site-packages" / "rasterio" / "proj_data",
|
||||
env_root / "Library" / "share" / "proj",
|
||||
env_root / "share" / "proj",
|
||||
]
|
||||
|
||||
for path in gdal_paths:
|
||||
if path.exists():
|
||||
os.environ["GDAL_DATA"] = str(path)
|
||||
break
|
||||
for path in proj_paths:
|
||||
if path.exists():
|
||||
os.environ["PROJ_LIB"] = str(path)
|
||||
os.environ["PROJ_DATA"] = str(path)
|
||||
break
|
||||
|
||||
|
||||
bootstrap_gdal_env()
|
||||
|
||||
import numpy as np
|
||||
import rasterio
|
||||
from rasterio.transform import from_origin
|
||||
|
||||
|
||||
def build_values(width: int, height: int, preset: str) -> np.ndarray:
|
||||
yy, xx = np.mgrid[0:height, 0:width]
|
||||
x = (xx / max(width - 1, 1)).astype("float32")
|
||||
y = (yy / max(height - 1, 1)).astype("float32")
|
||||
|
||||
if preset == "phase":
|
||||
values = np.sin((x * 8.0) + (y * 4.0)) * np.cos(y * 10.0)
|
||||
else:
|
||||
values = (x * 2.0 - 1.0) * 0.7 + np.sin(y * np.pi * 3.0) * 0.3
|
||||
|
||||
return values.astype("float32")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Create a sample GeoTIFF for demos.")
|
||||
parser.add_argument("--output", required=True, help="Output GeoTIFF path.")
|
||||
parser.add_argument("--width", type=int, default=512)
|
||||
parser.add_argument("--height", type=int, default=512)
|
||||
parser.add_argument("--preset", choices=["ndvi", "phase"], default="ndvi")
|
||||
parser.add_argument("--origin-x", type=float, default=110.0)
|
||||
parser.add_argument("--origin-y", type=float, default=22.0)
|
||||
parser.add_argument("--pixel-size", type=float, default=0.01)
|
||||
args = parser.parse_args()
|
||||
|
||||
output = Path(args.output).resolve()
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
data = build_values(args.width, args.height, args.preset)
|
||||
|
||||
with rasterio.open(
|
||||
output,
|
||||
"w",
|
||||
driver="GTiff",
|
||||
width=args.width,
|
||||
height=args.height,
|
||||
count=1,
|
||||
dtype="float32",
|
||||
crs="EPSG:4326",
|
||||
transform=from_origin(args.origin_x, args.origin_y, args.pixel_size, args.pixel_size),
|
||||
nodata=-9999.0,
|
||||
) as dst:
|
||||
dst.write(data, 1)
|
||||
|
||||
print(output)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,109 @@
|
||||
import argparse
|
||||
import json
|
||||
import threading
|
||||
import time
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="Simple callback receiver for tile-server demos.")
|
||||
parser.add_argument("--host", default="127.0.0.1")
|
||||
parser.add_argument("--port", type=int, default=8920)
|
||||
parser.add_argument("--path", default="/callback")
|
||||
parser.add_argument("--output-dir", required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
class State:
|
||||
def __init__(self, output_dir: Path, callback_path: str):
|
||||
self.output_dir = output_dir
|
||||
self.callback_path = callback_path
|
||||
self.counter = 0
|
||||
self.lock = threading.Lock()
|
||||
|
||||
def next_file(self) -> Path:
|
||||
with self.lock:
|
||||
self.counter += 1
|
||||
name = f"{time.strftime('%Y%m%d_%H%M%S')}_{self.counter:04d}.json"
|
||||
return self.output_dir / name
|
||||
|
||||
|
||||
def make_handler(state: State):
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
if self.path == "/health":
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "application/json; charset=utf-8")
|
||||
self.end_headers()
|
||||
self.wfile.write(json.dumps({"status": "ok"}).encode("utf-8"))
|
||||
return
|
||||
|
||||
self.send_response(404)
|
||||
self.end_headers()
|
||||
|
||||
def do_POST(self):
|
||||
self._handle_payload()
|
||||
|
||||
def do_PUT(self):
|
||||
self._handle_payload()
|
||||
|
||||
def _handle_payload(self):
|
||||
if self.path != state.callback_path:
|
||||
self.send_response(404)
|
||||
self.end_headers()
|
||||
return
|
||||
|
||||
length = int(self.headers.get("Content-Length", "0"))
|
||||
raw = self.rfile.read(length)
|
||||
text = raw.decode("utf-8", errors="replace")
|
||||
try:
|
||||
body_json = json.loads(text)
|
||||
except json.JSONDecodeError:
|
||||
body_json = None
|
||||
|
||||
record = {
|
||||
"received_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
"method": self.command,
|
||||
"path": self.path,
|
||||
"headers": {key: value for key, value in self.headers.items()},
|
||||
"body_text": text,
|
||||
"body_json": body_json,
|
||||
}
|
||||
|
||||
output_path = state.next_file()
|
||||
output_path.write_text(json.dumps(record, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print(f"[callback] saved {output_path}", flush=True)
|
||||
|
||||
response = {"saved": str(output_path), "ok": True}
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "application/json; charset=utf-8")
|
||||
self.end_headers()
|
||||
self.wfile.write(json.dumps(response).encode("utf-8"))
|
||||
|
||||
def log_message(self, fmt, *args):
|
||||
print("[callback] " + (fmt % args), flush=True)
|
||||
|
||||
return Handler
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
output_dir = Path(args.output_dir).resolve()
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
state = State(output_dir=output_dir, callback_path=args.path)
|
||||
server = ThreadingHTTPServer((args.host, args.port), make_handler(state))
|
||||
print(f"Callback receiver listening on http://{args.host}:{args.port}{args.path}", flush=True)
|
||||
print(f"Health endpoint: http://{args.host}:{args.port}/health", flush=True)
|
||||
print(f"Output dir: {output_dir}", flush=True)
|
||||
try:
|
||||
server.serve_forever()
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
finally:
|
||||
server.server_close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,213 @@
|
||||
param(
|
||||
[string]$ProjectCode = "default",
|
||||
[string]$ResultCode = ("demo_full_" + (Get-Date -Format "yyyyMMdd_HHmmss")),
|
||||
[string]$ResultType = "ndvi",
|
||||
[string]$SourceSystemCode = "",
|
||||
[string]$RequestedBy = "demo-user",
|
||||
[string]$StyleCode = "demo_continuous_style",
|
||||
[string]$TargetSystemCode = "demo_callback",
|
||||
[switch]$StartStack,
|
||||
[switch]$StartCallbackReceiver,
|
||||
[switch]$KeepProcesses,
|
||||
[int]$ApiPort = 28910,
|
||||
[int]$TilePort = 28900,
|
||||
[int]$CallbackPort = 28920,
|
||||
[int]$ZoomMin = 0,
|
||||
[int]$ZoomMax = 3
|
||||
)
|
||||
|
||||
. (Join-Path $PSScriptRoot "common.ps1")
|
||||
|
||||
$stack = $null
|
||||
$callback = $null
|
||||
$run = New-RsRunContext -Prefix "full_flow"
|
||||
|
||||
try {
|
||||
Import-RsDemoEnvironment
|
||||
|
||||
if ($StartStack) {
|
||||
$stack = Start-RsDemoStack -ApiPort $ApiPort -TilePort $TilePort -IncludeWorker -RunID $run.RunID
|
||||
}
|
||||
|
||||
$callbackUrl = ""
|
||||
if ($StartCallbackReceiver) {
|
||||
$callback = Start-RsCallbackReceiver -Port $CallbackPort -RunID ($run.RunID + "_callback")
|
||||
$callbackUrl = $callback.CallbackURL
|
||||
}
|
||||
|
||||
$inputPath = Join-Path $run.OutputDirectory ($ResultCode + ".tif")
|
||||
& (Get-RsPythonExe) (Join-Path $PSScriptRoot "create_sample_raster.py") `
|
||||
--output $inputPath `
|
||||
--preset "ndvi" | Out-Null
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "Failed to create sample raster"
|
||||
}
|
||||
|
||||
$inspection = Invoke-RsApi -Method "POST" -Path "/api/v1/admin/raster-inspections" -Body @{
|
||||
path = [System.IO.Path]::GetFullPath($inputPath)
|
||||
}
|
||||
|
||||
$styleVersion = Ensure-RsStyleVersion -ProjectCode $ProjectCode -StyleCode $StyleCode -StyleName "Demo Continuous Style"
|
||||
|
||||
$project = Encode-RsURIComponent $ProjectCode
|
||||
$importBody = @{
|
||||
requested_by = $RequestedBy
|
||||
asset = @{
|
||||
source_ref = $inspection.storage_uri
|
||||
asset_type = "raster"
|
||||
display_name = $ResultCode
|
||||
description = "Imported by demo_full_flow.ps1"
|
||||
metadata = @{
|
||||
demo_run = $run.RunID
|
||||
}
|
||||
}
|
||||
asset_version = @{
|
||||
format = $inspection.format
|
||||
storage_uri = $inspection.storage_uri
|
||||
mime_type = $inspection.mime_type
|
||||
size_bytes = $inspection.size_bytes
|
||||
crs = $inspection.crs
|
||||
width = $inspection.width
|
||||
height = $inspection.height
|
||||
band_count = $inspection.band_count
|
||||
dtype = $inspection.dtype
|
||||
nodata_value = $inspection.nodata_value
|
||||
bbox = $inspection.bbox
|
||||
resolution_x = $inspection.resolution_x
|
||||
resolution_y = $inspection.resolution_y
|
||||
metadata = $inspection.metadata
|
||||
}
|
||||
product = @{
|
||||
code = $ResultCode
|
||||
type = $ResultType
|
||||
name = $ResultCode
|
||||
description = "Demo end-to-end result"
|
||||
}
|
||||
product_version = @{
|
||||
value_type = "continuous"
|
||||
unit = "index"
|
||||
bbox = $inspection.bbox
|
||||
status = "ready"
|
||||
}
|
||||
style_binding = @{
|
||||
style_version_id = $styleVersion.id
|
||||
is_default = $true
|
||||
availability = "enabled"
|
||||
}
|
||||
}
|
||||
if ($SourceSystemCode) {
|
||||
$importBody.source_system_code = $SourceSystemCode
|
||||
}
|
||||
|
||||
$import = Invoke-RsApi -Method "POST" -Path "/api/v1/projects/$project/imports/results" -Body $importBody
|
||||
|
||||
$slice = Invoke-RsApi -Method "POST" -Path ("/api/v1/results/{0}/slice" -f $import.product.product.id) -Body @{
|
||||
requested_by = $RequestedBy
|
||||
build_cog = $true
|
||||
generate_preview = $true
|
||||
generate_legend = $true
|
||||
generate_tiles = $true
|
||||
build_cog_input = @{
|
||||
compression = "deflate"
|
||||
block_size = 256
|
||||
force_rebuild = $true
|
||||
}
|
||||
preview_input = @{
|
||||
style_version_id = $styleVersion.id
|
||||
width = 512
|
||||
height = 512
|
||||
format = "png"
|
||||
force_rebuild = $true
|
||||
}
|
||||
legend_input = @{
|
||||
style_version_id = $styleVersion.id
|
||||
width = 320
|
||||
height = 80
|
||||
format = "png"
|
||||
force_rebuild = $true
|
||||
}
|
||||
tiles_input = @{
|
||||
style_version_id = $styleVersion.id
|
||||
zoom_min = $ZoomMin
|
||||
zoom_max = $ZoomMax
|
||||
tile_size = 256
|
||||
tile_format = "webp"
|
||||
force_rebuild = $true
|
||||
}
|
||||
}
|
||||
|
||||
$sliceJobs = @()
|
||||
foreach ($job in @($slice.jobs)) {
|
||||
$sliceJobs += Wait-RsJob -JobID $job.id
|
||||
}
|
||||
|
||||
if ($callbackUrl) {
|
||||
Ensure-RsTargetSystem `
|
||||
-ProjectCode $ProjectCode `
|
||||
-Code $TargetSystemCode `
|
||||
-Name "Demo Callback Receiver" `
|
||||
-CallbackUrl $callbackUrl `
|
||||
-CallbackMethod "POST" `
|
||||
-TimeoutSeconds 30 `
|
||||
-CreatedBy $RequestedBy `
|
||||
-CallbackHeaders @{
|
||||
"X-Demo-Mode" = "full-flow"
|
||||
} | Out-Null
|
||||
}
|
||||
|
||||
$pushBody = @{
|
||||
requested_by = $RequestedBy
|
||||
style_version_id = $styleVersion.id
|
||||
include_artifacts = $true
|
||||
}
|
||||
if ($callbackUrl) {
|
||||
$pushBody.target_system_code = $TargetSystemCode
|
||||
} else {
|
||||
$pushBody.callback_url = "http://127.0.0.1:8920/callback"
|
||||
}
|
||||
|
||||
$push = Invoke-RsApi -Method "POST" -Path ("/api/v1/results/{0}/pushes" -f $import.product.product.id) -Body $pushBody
|
||||
$pushJob = Wait-RsJob -JobID $push.job.id
|
||||
|
||||
$delivery = Invoke-RsApi -Method "GET" -Path ("/api/v1/projects/{0}/products/{1}/delivery?version=latest&style_version_id={2}" -f `
|
||||
(Encode-RsURIComponent $ProjectCode), `
|
||||
(Encode-RsURIComponent $ResultCode), `
|
||||
(Encode-RsURIComponent $styleVersion.id))
|
||||
|
||||
$callbackRecord = $null
|
||||
if ($callback) {
|
||||
Start-Sleep -Seconds 1
|
||||
$callbackRecord = Get-RsLatestCallbackRecord -OutputDirectory $callback.OutputDirectory
|
||||
}
|
||||
|
||||
[ordered]@{
|
||||
run_id = $run.RunID
|
||||
api_base_url = Get-RsApiBaseUrl
|
||||
tile_base_url = Get-RsTileBaseUrl
|
||||
callback_url = if ($callback) { $callback.CallbackURL } else { $push.callback_url }
|
||||
result_id = $import.product.product.id
|
||||
result_code = $ResultCode
|
||||
product_version_id = $import.product_version.id
|
||||
style_version_id = $styleVersion.id
|
||||
slice_job_ids = @($sliceJobs | ForEach-Object { $_.id })
|
||||
push_job_id = $push.job.id
|
||||
push_job_status = $pushJob.status
|
||||
cog_url = $delivery.downloads.cog_url
|
||||
preview_url = if ($delivery.preview) { $delivery.preview.url } else { $null }
|
||||
legend_url = if ($delivery.legend) { $delivery.legend.url } else { $null }
|
||||
tileset_url = if ($delivery.tileset) { $delivery.tileset.url_template } else { $null }
|
||||
callback_record_path = if ($callbackRecord) { $callbackRecord.Path } else { $null }
|
||||
run_directory = $run.RunDirectory
|
||||
log_directory = if ($stack) { $stack.LogDirectory } else { $null }
|
||||
} | ConvertTo-Json -Depth 16
|
||||
}
|
||||
finally {
|
||||
if (-not $KeepProcesses) {
|
||||
if ($callback) {
|
||||
Stop-RsBackgroundProcesses -Processes @($callback.Process)
|
||||
}
|
||||
if ($stack) {
|
||||
Stop-RsBackgroundProcesses -Processes $stack.Processes
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,135 @@
|
||||
param(
|
||||
[string]$ProjectCode = "default",
|
||||
[string]$ResourceCode = ("demo_geojson_" + (Get-Date -Format "yyyyMMdd_HHmmss")),
|
||||
[string]$InputPath = "",
|
||||
[string]$RequestedBy = "demo-user",
|
||||
[switch]$Push,
|
||||
[string]$TargetSystemCode = "",
|
||||
[string]$CallbackUrl = "",
|
||||
[bool]$WaitForJob = $true,
|
||||
[switch]$StartStack,
|
||||
[switch]$StartCallbackReceiver,
|
||||
[switch]$KeepProcesses,
|
||||
[int]$ApiPort = 28910,
|
||||
[int]$TilePort = 28900,
|
||||
[int]$CallbackPort = 28920
|
||||
)
|
||||
|
||||
. (Join-Path $PSScriptRoot "common.ps1")
|
||||
|
||||
$stack = $null
|
||||
$callback = $null
|
||||
$run = New-RsRunContext -Prefix "geojson_flow"
|
||||
|
||||
try {
|
||||
Import-RsDemoEnvironment
|
||||
|
||||
if ($StartStack) {
|
||||
$stack = Start-RsDemoStack -ApiPort $ApiPort -TilePort $TilePort -IncludeWorker -RunID $run.RunID
|
||||
}
|
||||
if ($StartCallbackReceiver) {
|
||||
$callback = Start-RsCallbackReceiver -Port $CallbackPort -RunID ($run.RunID + "_callback")
|
||||
$Push = $true
|
||||
$CallbackUrl = $callback.CallbackURL
|
||||
}
|
||||
|
||||
if (-not $InputPath) {
|
||||
$geojsonDir = Join-Path (Get-RsRepoRoot) "data\geojson\demo"
|
||||
New-Item -ItemType Directory -Force -Path $geojsonDir | Out-Null
|
||||
$InputPath = Join-Path $geojsonDir ($ResourceCode + ".geojson")
|
||||
@"
|
||||
{
|
||||
"type": "FeatureCollection",
|
||||
"features": [
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {
|
||||
"code": "$ResourceCode",
|
||||
"kind": "demo"
|
||||
},
|
||||
"geometry": {
|
||||
"type": "Polygon",
|
||||
"coordinates": [
|
||||
[
|
||||
[110.0, 20.0],
|
||||
[112.0, 20.0],
|
||||
[112.0, 22.0],
|
||||
[110.0, 22.0],
|
||||
[110.0, 20.0]
|
||||
]
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
"@ | Set-Content -LiteralPath $InputPath -Encoding UTF8
|
||||
}
|
||||
|
||||
$publicUrl = "/geojson/demo/" + [System.IO.Path]::GetFileName($InputPath)
|
||||
$project = Encode-RsURIComponent $ProjectCode
|
||||
$resource = Invoke-RsApi -Method "POST" -Path "/api/v1/projects/$project/geojson-resources" -Body @{
|
||||
code = $ResourceCode
|
||||
name = $ResourceCode
|
||||
description = "Registered by demo_geojson_flow.ps1"
|
||||
storage_uri = ConvertTo-RsFileUri $InputPath
|
||||
public_url = $publicUrl
|
||||
metadata = @{
|
||||
demo_run = $run.RunID
|
||||
}
|
||||
}
|
||||
|
||||
$pushResponse = $null
|
||||
$job = $null
|
||||
if ($Push -or $TargetSystemCode -or $CallbackUrl) {
|
||||
$body = @{
|
||||
resource_kind = "geojson_resource"
|
||||
geojson_resource_id = $resource.id
|
||||
requested_by = $RequestedBy
|
||||
}
|
||||
if ($TargetSystemCode) {
|
||||
$body.target_system_code = $TargetSystemCode
|
||||
}
|
||||
if ($CallbackUrl) {
|
||||
$body.callback_url = $CallbackUrl
|
||||
}
|
||||
if (-not $TargetSystemCode -and -not $CallbackUrl) {
|
||||
$body.callback_url = "http://127.0.0.1:8920/callback"
|
||||
}
|
||||
|
||||
$pushResponse = Invoke-RsApi -Method "POST" -Path "/api/v1/projects/$project/delivery-pushes" -Body $body
|
||||
if ($WaitForJob) {
|
||||
$job = Wait-RsJob -JobID $pushResponse.job.id
|
||||
}
|
||||
}
|
||||
|
||||
$callbackRecord = $null
|
||||
if ($callback) {
|
||||
Start-Sleep -Seconds 1
|
||||
$callbackRecord = Get-RsLatestCallbackRecord -OutputDirectory $callback.OutputDirectory
|
||||
}
|
||||
|
||||
[ordered]@{
|
||||
run_id = $run.RunID
|
||||
api_base_url = Get-RsApiBaseUrl
|
||||
tile_base_url = Get-RsTileBaseUrl
|
||||
resource_id = $resource.id
|
||||
resource_code = $resource.code
|
||||
input_path = [System.IO.Path]::GetFullPath($InputPath)
|
||||
public_url = $resource.public_url
|
||||
download_url = $resource.download_url
|
||||
push_job_id = if ($pushResponse) { $pushResponse.job.id } else { $null }
|
||||
push_job_status = if ($job) { $job.status } elseif ($pushResponse) { $pushResponse.job.status } else { $null }
|
||||
callback_url = if ($pushResponse) { $pushResponse.callback_url } else { $CallbackUrl }
|
||||
callback_record_path = if ($callbackRecord) { $callbackRecord.Path } else { $null }
|
||||
} | ConvertTo-Json -Depth 12
|
||||
}
|
||||
finally {
|
||||
if (-not $KeepProcesses) {
|
||||
if ($callback) {
|
||||
Stop-RsBackgroundProcesses -Processes @($callback.Process)
|
||||
}
|
||||
if ($stack) {
|
||||
Stop-RsBackgroundProcesses -Processes $stack.Processes
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,190 @@
|
||||
param(
|
||||
[string]$ProjectCode = "default",
|
||||
[string]$ResultCode = ("demo_result_" + (Get-Date -Format "yyyyMMdd_HHmmss")),
|
||||
[string]$ResultType = "ndvi",
|
||||
[string]$InputPath = "",
|
||||
[string]$SourceSystemCode = "",
|
||||
[string]$RequestedBy = "demo-user",
|
||||
[string]$StyleCode = "demo_continuous_style",
|
||||
[string]$StyleName = "Demo Continuous Style",
|
||||
[int]$ZoomMin = 0,
|
||||
[int]$ZoomMax = 3,
|
||||
[bool]$WaitForJobs = $true,
|
||||
[switch]$StartStack,
|
||||
[switch]$KeepStack,
|
||||
[int]$ApiPort = 28910,
|
||||
[int]$TilePort = 28900
|
||||
)
|
||||
|
||||
. (Join-Path $PSScriptRoot "common.ps1")
|
||||
|
||||
$stack = $null
|
||||
$run = New-RsRunContext -Prefix "import_slice"
|
||||
|
||||
try {
|
||||
Import-RsDemoEnvironment
|
||||
|
||||
if ($StartStack) {
|
||||
$stack = Start-RsDemoStack -ApiPort $ApiPort -TilePort $TilePort -IncludeWorker -RunID $run.RunID
|
||||
}
|
||||
|
||||
if (-not $InputPath) {
|
||||
$InputPath = Join-Path $run.OutputDirectory ($ResultCode + ".tif")
|
||||
& (Get-RsPythonExe) (Join-Path $PSScriptRoot "create_sample_raster.py") `
|
||||
--output $InputPath `
|
||||
--preset "ndvi" | Out-Null
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "Failed to create sample raster"
|
||||
}
|
||||
}
|
||||
|
||||
$inspection = Invoke-RsApi -Method "POST" -Path "/api/v1/admin/raster-inspections" -Body @{
|
||||
path = [System.IO.Path]::GetFullPath($InputPath)
|
||||
}
|
||||
|
||||
$styleVersion = Ensure-RsStyleVersion -ProjectCode $ProjectCode -StyleCode $StyleCode -StyleName $StyleName
|
||||
|
||||
$assetVersionBody = @{
|
||||
format = $inspection.format
|
||||
storage_uri = $inspection.storage_uri
|
||||
mime_type = $inspection.mime_type
|
||||
size_bytes = $inspection.size_bytes
|
||||
crs = $inspection.crs
|
||||
width = $inspection.width
|
||||
height = $inspection.height
|
||||
band_count = $inspection.band_count
|
||||
dtype = $inspection.dtype
|
||||
bbox = $inspection.bbox
|
||||
metadata = $inspection.metadata
|
||||
}
|
||||
if ($inspection.PSObject.Properties.Name -contains "nodata_value" -and $null -ne $inspection.nodata_value) {
|
||||
$assetVersionBody.nodata_value = $inspection.nodata_value
|
||||
}
|
||||
if ($inspection.PSObject.Properties.Name -contains "resolution_x" -and $null -ne $inspection.resolution_x) {
|
||||
$assetVersionBody.resolution_x = $inspection.resolution_x
|
||||
}
|
||||
if ($inspection.PSObject.Properties.Name -contains "resolution_y" -and $null -ne $inspection.resolution_y) {
|
||||
$assetVersionBody.resolution_y = $inspection.resolution_y
|
||||
}
|
||||
|
||||
$project = Encode-RsURIComponent $ProjectCode
|
||||
$importBody = @{
|
||||
requested_by = $RequestedBy
|
||||
asset = @{
|
||||
source_ref = $inspection.storage_uri
|
||||
asset_type = "raster"
|
||||
display_name = $ResultCode
|
||||
description = "Imported by demo_import_and_slice.ps1"
|
||||
metadata = @{
|
||||
demo_run = $run.RunID
|
||||
}
|
||||
}
|
||||
asset_version = $assetVersionBody
|
||||
product = @{
|
||||
code = $ResultCode
|
||||
type = $ResultType
|
||||
name = $ResultCode
|
||||
description = "Demo managed result"
|
||||
}
|
||||
product_version = @{
|
||||
value_type = "continuous"
|
||||
unit = "index"
|
||||
bbox = $inspection.bbox
|
||||
status = "ready"
|
||||
}
|
||||
style_binding = @{
|
||||
style_version_id = $styleVersion.id
|
||||
is_default = $true
|
||||
availability = "enabled"
|
||||
}
|
||||
}
|
||||
if ($SourceSystemCode) {
|
||||
$importBody.source_system_code = $SourceSystemCode
|
||||
}
|
||||
|
||||
$import = Invoke-RsApi -Method "POST" -Path "/api/v1/projects/$project/imports/results" -Body $importBody
|
||||
|
||||
$slice = Invoke-RsApi -Method "POST" -Path ("/api/v1/results/{0}/slice" -f $import.product.product.id) -Body @{
|
||||
requested_by = $RequestedBy
|
||||
build_cog = $true
|
||||
generate_preview = $true
|
||||
generate_legend = $true
|
||||
generate_tiles = $true
|
||||
build_cog_input = @{
|
||||
compression = "deflate"
|
||||
block_size = 256
|
||||
force_rebuild = $true
|
||||
}
|
||||
preview_input = @{
|
||||
style_version_id = $styleVersion.id
|
||||
width = 512
|
||||
height = 512
|
||||
format = "png"
|
||||
force_rebuild = $true
|
||||
}
|
||||
legend_input = @{
|
||||
style_version_id = $styleVersion.id
|
||||
width = 320
|
||||
height = 80
|
||||
format = "png"
|
||||
force_rebuild = $true
|
||||
}
|
||||
tiles_input = @{
|
||||
style_version_id = $styleVersion.id
|
||||
zoom_min = $ZoomMin
|
||||
zoom_max = $ZoomMax
|
||||
tile_size = 256
|
||||
tile_format = "webp"
|
||||
force_rebuild = $true
|
||||
}
|
||||
}
|
||||
|
||||
$jobStates = @()
|
||||
if ($WaitForJobs) {
|
||||
foreach ($job in @($slice.jobs)) {
|
||||
$jobStates += Wait-RsJob -JobID $job.id
|
||||
}
|
||||
}
|
||||
|
||||
$result = Invoke-RsApi -Method "GET" -Path ("/api/v1/results/{0}" -f $import.product.product.id)
|
||||
$delivery = $null
|
||||
if ($WaitForJobs) {
|
||||
$delivery = Invoke-RsApi -Method "GET" -Path ("/api/v1/projects/{0}/products/{1}/delivery?version=latest&style_version_id={2}" -f `
|
||||
(Encode-RsURIComponent $ProjectCode), `
|
||||
(Encode-RsURIComponent $ResultCode), `
|
||||
(Encode-RsURIComponent $styleVersion.id))
|
||||
}
|
||||
|
||||
[ordered]@{
|
||||
run_id = $run.RunID
|
||||
api_base_url = Get-RsApiBaseUrl
|
||||
tile_base_url = Get-RsTileBaseUrl
|
||||
input_path = [System.IO.Path]::GetFullPath($InputPath)
|
||||
result_id = $import.product.product.id
|
||||
result_code = $ResultCode
|
||||
product_version_id = $import.product_version.id
|
||||
style_version_id = $styleVersion.id
|
||||
imported_asset_id = $import.asset.id
|
||||
imported_version_id = $import.asset_version.id
|
||||
slice_job_ids = @($slice.jobs | ForEach-Object { $_.id })
|
||||
slice_job_statuses = @($jobStates | ForEach-Object {
|
||||
[ordered]@{
|
||||
id = $_.id
|
||||
job_type = $_.job_type
|
||||
status = $_.status
|
||||
}
|
||||
})
|
||||
cog_url = if ($delivery) { $delivery.downloads.cog_url } else { $null }
|
||||
preview_url = if ($delivery -and $delivery.preview) { $delivery.preview.url } else { $null }
|
||||
legend_url = if ($delivery -and $delivery.legend) { $delivery.legend.url } else { $null }
|
||||
tileset_url = if ($delivery -and $delivery.tileset) { $delivery.tileset.url_template } else { $null }
|
||||
result_status = $result.status
|
||||
run_directory = $run.RunDirectory
|
||||
log_directory = if ($stack) { $stack.LogDirectory } else { $null }
|
||||
} | ConvertTo-Json -Depth 16
|
||||
}
|
||||
finally {
|
||||
if ($stack -and -not $KeepStack) {
|
||||
Stop-RsBackgroundProcesses -Processes $stack.Processes
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
param(
|
||||
[string]$ProjectCode = "default",
|
||||
[string]$ResultCode = "",
|
||||
[string]$ResultID = "",
|
||||
[string]$TargetSystemCode = "",
|
||||
[string]$CallbackUrl = "",
|
||||
[string]$StyleVersionID = "",
|
||||
[string]$RequestedBy = "demo-user",
|
||||
[bool]$WaitForJob = $true,
|
||||
[bool]$IncludeArtifacts = $true,
|
||||
[int]$TimeoutSec = 180
|
||||
)
|
||||
|
||||
. (Join-Path $PSScriptRoot "common.ps1")
|
||||
|
||||
Import-RsDemoEnvironment
|
||||
|
||||
if (-not $ResultID -and -not $ResultCode) {
|
||||
throw "ResultID or ResultCode is required"
|
||||
}
|
||||
|
||||
$result = if ($ResultID) {
|
||||
Invoke-RsApi -Method "GET" -Path ("/api/v1/results/{0}" -f $ResultID)
|
||||
} else {
|
||||
Get-RsResultByCode -ProjectCode $ProjectCode -Code $ResultCode
|
||||
}
|
||||
|
||||
if (-not $StyleVersionID) {
|
||||
if ($result.delivery -and $result.delivery.style_version -and $result.delivery.style_version.id) {
|
||||
$StyleVersionID = $result.delivery.style_version.id
|
||||
} elseif ($result.styles -and $result.styles.Count -gt 0) {
|
||||
$StyleVersionID = $result.styles[0].id
|
||||
}
|
||||
}
|
||||
if (-not $StyleVersionID) {
|
||||
throw "No style version is bound to the target result"
|
||||
}
|
||||
|
||||
$body = @{
|
||||
requested_by = $RequestedBy
|
||||
style_version_id = $StyleVersionID
|
||||
include_artifacts = $IncludeArtifacts
|
||||
}
|
||||
if ($TargetSystemCode) {
|
||||
$body.target_system_code = $TargetSystemCode
|
||||
}
|
||||
if ($CallbackUrl) {
|
||||
$body.callback_url = $CallbackUrl
|
||||
}
|
||||
if (-not $TargetSystemCode -and -not $CallbackUrl) {
|
||||
$body.callback_url = "http://127.0.0.1:8920/callback"
|
||||
}
|
||||
|
||||
$push = Invoke-RsApi -Method "POST" -Path ("/api/v1/results/{0}/pushes" -f $result.id) -Body $body
|
||||
|
||||
$job = $null
|
||||
if ($WaitForJob) {
|
||||
$job = Wait-RsJob -JobID $push.job.id -TimeoutSec $TimeoutSec
|
||||
}
|
||||
|
||||
[ordered]@{
|
||||
api_base_url = Get-RsApiBaseUrl
|
||||
result_id = $result.id
|
||||
result_code = $result.code
|
||||
style_version_id = $StyleVersionID
|
||||
target_system_code = $push.target_system_code
|
||||
callback_url = $push.callback_url
|
||||
publish_id = $push.publish_id
|
||||
job_id = $push.job.id
|
||||
job_status = if ($job) { $job.status } else { $push.job.status }
|
||||
resource_kind = $push.resource_kind
|
||||
} | ConvertTo-Json -Depth 8
|
||||
@@ -0,0 +1,27 @@
|
||||
param(
|
||||
[string]$ProjectCode = "default",
|
||||
[string]$Code = "demo_callback",
|
||||
[string]$Name = "Demo Callback Receiver",
|
||||
[string]$CallbackUrl = "http://127.0.0.1:8920/callback",
|
||||
[string]$CallbackMethod = "POST",
|
||||
[int]$TimeoutSeconds = 30,
|
||||
[string]$CreatedBy = "demo-user"
|
||||
)
|
||||
|
||||
. (Join-Path $PSScriptRoot "common.ps1")
|
||||
|
||||
Import-RsDemoEnvironment
|
||||
|
||||
$target = Ensure-RsTargetSystem `
|
||||
-ProjectCode $ProjectCode `
|
||||
-Code $Code `
|
||||
-Name $Name `
|
||||
-CallbackUrl $CallbackUrl `
|
||||
-CallbackMethod $CallbackMethod `
|
||||
-TimeoutSeconds $TimeoutSeconds `
|
||||
-CreatedBy $CreatedBy `
|
||||
-CallbackHeaders @{
|
||||
"X-Demo-System" = "tile-server-demo"
|
||||
}
|
||||
|
||||
$target | ConvertTo-Json -Depth 8
|
||||
@@ -0,0 +1,16 @@
|
||||
@echo off
|
||||
setlocal
|
||||
chcp 65001 >nul 2>&1
|
||||
|
||||
set "ROOT=%~dp0"
|
||||
if "%PYTHON_EXE%"=="" set "PYTHON_EXE=C:\Users\Administrator\.conda\envs\InSAR\python.exe"
|
||||
if "%DEMO_CALLBACK_PORT%"=="" set "DEMO_CALLBACK_PORT=8920"
|
||||
if "%DEMO_CALLBACK_PATH%"=="" set "DEMO_CALLBACK_PATH=/callback"
|
||||
if "%DEMO_CALLBACK_OUTPUT_DIR%"=="" set "DEMO_CALLBACK_OUTPUT_DIR=%ROOT%..\temp\demo-callback"
|
||||
|
||||
if not exist "%PYTHON_EXE%" (
|
||||
echo Python interpreter not found: %PYTHON_EXE%
|
||||
exit /b 1
|
||||
)
|
||||
|
||||
"%PYTHON_EXE%" "%ROOT%demo_callback_receiver.py" --host 127.0.0.1 --port %DEMO_CALLBACK_PORT% --path %DEMO_CALLBACK_PATH% --output-dir "%DEMO_CALLBACK_OUTPUT_DIR%"
|
||||
@@ -0,0 +1,201 @@
|
||||
# Insar_management_system_v2 接入说明
|
||||
|
||||
本文档说明 `Z:\Code\Insar_management_system_v2` 如何接入当前 `tile-server`，不需要修改推送服务外部的公开匿名路由，全部走带 token 的受保护接口。
|
||||
|
||||
## 结论
|
||||
|
||||
当前 `tile-server` 经过兼容后，可以支撑 `Insar_management_system_v2` 的底图和 GeoJSON/SHP 调用，但前提是目标系统把原先的固定静态路径改成带 token 的调用方式。
|
||||
|
||||
支持范围：
|
||||
|
||||
- 底图瓦片
|
||||
- GeoJSON 文件直连读取
|
||||
- Shapefile 或 `.zip` 方式下载
|
||||
|
||||
不需要目标系统实现“接收推送”的回调接口。
|
||||
|
||||
## 推荐调用方式
|
||||
|
||||
假设服务地址为：
|
||||
|
||||
```text
|
||||
http://127.0.0.1:8910
|
||||
```
|
||||
|
||||
假设访问 token 为：
|
||||
|
||||
```text
|
||||
bmp_xxxxxxxxxxxxxxxxx
|
||||
```
|
||||
|
||||
### 1. 底图瓦片
|
||||
|
||||
推荐直接使用默认版本兼容路由：
|
||||
|
||||
```text
|
||||
GET /tiles/{code}/{z}/{x}/{y}.ext?token=...
|
||||
```
|
||||
|
||||
示例：
|
||||
|
||||
```text
|
||||
http://127.0.0.1:8910/tiles/google_image/{z}/{x}/{y}.webp?token=bmp_xxx
|
||||
http://127.0.0.1:8910/tiles/gaode_image/{z}/{x}/{y}.webp?token=bmp_xxx
|
||||
http://127.0.0.1:8910/tiles/gaode_shp/{z}/{x}/{y}.webp?token=bmp_xxx
|
||||
```
|
||||
|
||||
说明：
|
||||
|
||||
- `google_image`、`gaode_image`、`gaode_shp` 可以直接保留下划线写法
|
||||
- 服务内部会归一化到对应底图库 code
|
||||
- 路由会自动解析该底图的默认版本
|
||||
|
||||
如果你希望显式锁定版本，也可以继续使用：
|
||||
|
||||
```text
|
||||
GET /tiles/{code}/{version}/{z}/{x}/{y}.ext?token=...
|
||||
```
|
||||
|
||||
### 2. GeoJSON
|
||||
|
||||
推荐直接按原始相对路径访问：
|
||||
|
||||
```text
|
||||
GET /geojson/{relative/path}.geojson?token=...
|
||||
```
|
||||
|
||||
示例：
|
||||
|
||||
```text
|
||||
http://127.0.0.1:8910/geojson/全国行政区.geojson?token=bmp_xxx
|
||||
```
|
||||
|
||||
说明：
|
||||
|
||||
- 支持按原始相对路径访问
|
||||
- 也支持按文件名访问
|
||||
- 也支持按内部 vector code 访问
|
||||
- 返回内容为实际 GeoJSON 文件内容，可直接 `fetch(...).json()`
|
||||
|
||||
### 3. SHP
|
||||
|
||||
推荐按原始相对路径下载：
|
||||
|
||||
```text
|
||||
GET /shp/{relative/path}.shp?token=...
|
||||
```
|
||||
|
||||
示例：
|
||||
|
||||
```text
|
||||
http://127.0.0.1:8910/shp/landuse/parcel.shp?token=bmp_xxx
|
||||
```
|
||||
|
||||
说明：
|
||||
|
||||
- 如果底层资源是 `.shp + .dbf + .shx ...` 组合，服务会自动打包成 zip 返回
|
||||
- 如果底层资源本来就是 `.zip`，则直接返回 zip
|
||||
|
||||
如果你希望继续按内部 code 下载，也可以使用：
|
||||
|
||||
```text
|
||||
GET /vectors/{code}/download?token=...
|
||||
```
|
||||
|
||||
## Token 传递方式
|
||||
|
||||
支持以下方式，推荐前端场景使用查询参数：
|
||||
|
||||
### 查询参数
|
||||
|
||||
```text
|
||||
?token=bmp_xxx
|
||||
```
|
||||
|
||||
### Header
|
||||
|
||||
```text
|
||||
Authorization: Bearer bmp_xxx
|
||||
```
|
||||
|
||||
或：
|
||||
|
||||
```text
|
||||
X-Access-Token: bmp_xxx
|
||||
```
|
||||
|
||||
## 前端改造建议
|
||||
|
||||
### 底图常量
|
||||
|
||||
把原先：
|
||||
|
||||
```js
|
||||
`${TILE_BASE}/tiles/google_image/{z}/{x}/{y}.webp`
|
||||
```
|
||||
|
||||
改成：
|
||||
|
||||
```js
|
||||
`${TILE_BASE}/tiles/google_image/{z}/{x}/{y}.webp?token=${encodeURIComponent(TILE_TOKEN)}`
|
||||
```
|
||||
|
||||
其他底图同理。
|
||||
|
||||
### 行政区 GeoJSON
|
||||
|
||||
把原先：
|
||||
|
||||
```js
|
||||
`${TILE_BASE}/geojson/全国行政区.geojson`
|
||||
```
|
||||
|
||||
改成：
|
||||
|
||||
```js
|
||||
`${TILE_BASE}/geojson/全国行政区.geojson?token=${encodeURIComponent(TILE_TOKEN)}`
|
||||
```
|
||||
|
||||
## 推荐接入流程
|
||||
|
||||
1. 在 `tile-server` 管理页或管理 API 扫描底图目录和矢量目录。
|
||||
2. 创建一个专供 `Insar_management_system_v2` 使用的 token。
|
||||
3. token 授权对应底图和需要访问的 GeoJSON/SHP 资源。
|
||||
4. 在 `Insar_management_system_v2` 中把静态地址改成带 token 的服务地址。
|
||||
|
||||
## 创建 token 时的资源填写规则
|
||||
|
||||
### 底图
|
||||
|
||||
填写底图库 code，例如：
|
||||
|
||||
```text
|
||||
google-image
|
||||
gaode-image
|
||||
gaode-shp
|
||||
```
|
||||
|
||||
### 矢量
|
||||
|
||||
现在既可以填内部 code，也可以直接填原始路径，例如：
|
||||
|
||||
```text
|
||||
全国行政区.geojson
|
||||
landuse/parcel.shp
|
||||
```
|
||||
|
||||
## 适配边界
|
||||
|
||||
当前服务负责的是：
|
||||
|
||||
- 扫描
|
||||
- 目录管理
|
||||
- token 鉴权
|
||||
- 底图瓦片输出
|
||||
- GeoJSON/SHP 输出
|
||||
|
||||
当前服务不负责：
|
||||
|
||||
- 生成切片
|
||||
- 把矢量转换成 GeoJSON
|
||||
- 修改 `Insar_management_system_v2` 代码
|
||||
@@ -0,0 +1,49 @@
|
||||
@echo off
|
||||
setlocal
|
||||
chcp 65001 >nul 2>&1
|
||||
title Build Basemap Pusher
|
||||
|
||||
set "ROOT=%~dp0..\\"
|
||||
pushd "%ROOT%" >nul 2>&1
|
||||
set "ROOT=%CD%\"
|
||||
popd
|
||||
call "%ROOT%scripts\load-env.bat" "%ROOT%config\service.env"
|
||||
|
||||
REM Keep the toolchain caches inside the repo, the same way run-api.bat does.
|
||||
REM Without this, go build uses the global module cache and reaches out to
|
||||
REM proxy.golang.org, which is blocked on many internal networks.
|
||||
if "%GOCACHE%"=="" set "GOCACHE=%ROOT%.gocache"
|
||||
if "%GOMODCACHE%"=="" set "GOMODCACHE=%ROOT%.gomodcache"
|
||||
if "%GOTMPDIR%"=="" set "GOTMPDIR=%ROOT%.gotmp"
|
||||
if "%GOPATH%"=="" set "GOPATH=%ROOT%.gopath"
|
||||
if "%GOPROXY%"=="" set "GOPROXY=https://goproxy.cn,direct"
|
||||
if "%GOSUMDB%"=="" set "GOSUMDB=off"
|
||||
|
||||
set "OUT=%ROOT%bin\basemap-pusher.exe"
|
||||
if not exist "%ROOT%bin" mkdir "%ROOT%bin" >nul 2>&1
|
||||
|
||||
echo ========================================
|
||||
echo Building Basemap Pusher
|
||||
echo ========================================
|
||||
echo Output: %OUT%
|
||||
echo.
|
||||
|
||||
pushd "%ROOT%services\api-go"
|
||||
go build -o "%OUT%" .\cmd\api
|
||||
set "BUILD_EXIT=%ERRORLEVEL%"
|
||||
popd
|
||||
|
||||
if not "%BUILD_EXIT%"=="0" (
|
||||
echo.
|
||||
echo [FAIL] Build failed with exit code %BUILD_EXIT%.
|
||||
) else (
|
||||
echo.
|
||||
echo [OK] Built %OUT%
|
||||
echo.
|
||||
echo Install as a Windows service ^(elevated shell^):
|
||||
echo "%OUT%" --env-file "%ROOT%config\service.env" install
|
||||
echo "%OUT%" start
|
||||
)
|
||||
|
||||
if "%NO_PAUSE%"=="" pause
|
||||
exit /b %BUILD_EXIT%
|
||||
@@ -0,0 +1,35 @@
|
||||
@echo off
|
||||
setlocal enableextensions
|
||||
REM Run gdal2tiles.py from the GDAL bundled with QGIS.
|
||||
REM
|
||||
REM QGIS ships GDAL but its Python bindings need the OSGeo4W environment
|
||||
REM (GDAL_DATA, PROJ_LIB, PYTHONHOME). Running gdal2tiles.exe from a bare
|
||||
REM shell fails to import osgeo, so this wrapper loads o4w_env.bat first.
|
||||
REM
|
||||
REM Override auto-detection with: set GDAL2TILES_QGIS_ROOT=C:\path\to\QGIS
|
||||
|
||||
set "QGIS_ROOT=%GDAL2TILES_QGIS_ROOT%"
|
||||
if not "%QGIS_ROOT%"=="" goto have_root
|
||||
|
||||
for /f "delims=" %%D in ('dir /b /ad /o-n "%ProgramFiles%\QGIS *" 2^>nul') do (
|
||||
if not defined QGIS_ROOT set "QGIS_ROOT=%ProgramFiles%\%%D"
|
||||
)
|
||||
|
||||
:have_root
|
||||
if "%QGIS_ROOT%"=="" (
|
||||
echo gdal2tiles wrapper: QGIS not found. Set GDAL2TILES_QGIS_ROOT to the QGIS install directory. 1>&2
|
||||
exit /b 2
|
||||
)
|
||||
|
||||
if not exist "%QGIS_ROOT%\bin\o4w_env.bat" (
|
||||
echo gdal2tiles wrapper: "%QGIS_ROOT%\bin\o4w_env.bat" is missing; is this a QGIS install? 1>&2
|
||||
exit /b 2
|
||||
)
|
||||
if not exist "%QGIS_ROOT%\apps\Python312\Scripts\gdal2tiles.exe" (
|
||||
echo gdal2tiles wrapper: gdal2tiles.exe not found under "%QGIS_ROOT%\apps\Python312\Scripts". 1>&2
|
||||
exit /b 2
|
||||
)
|
||||
|
||||
call "%QGIS_ROOT%\bin\o4w_env.bat" >nul 2>&1
|
||||
"%QGIS_ROOT%\apps\Python312\Scripts\gdal2tiles.exe" %*
|
||||
exit /b %ERRORLEVEL%
|
||||
@@ -0,0 +1,33 @@
|
||||
@echo off
|
||||
setlocal enableextensions
|
||||
REM Run gdalbuildvrt from the GDAL bundled with QGIS.
|
||||
REM
|
||||
REM QGIS ships GDAL but its Python bindings need the OSGeo4W environment
|
||||
REM (GDAL_DATA, PROJ_LIB). Loading o4w_env.bat first keeps PROJ working.
|
||||
REM
|
||||
REM Override auto-detection with: set GDAL2TILES_QGIS_ROOT=C:\path\to\QGIS
|
||||
|
||||
set "QGIS_ROOT=%GDAL2TILES_QGIS_ROOT%"
|
||||
if not "%QGIS_ROOT%"=="" goto have_root
|
||||
|
||||
for /f "delims=" %%D in ('dir /b /ad /o-n "%ProgramFiles%\QGIS *" 2^>nul') do (
|
||||
if not defined QGIS_ROOT set "QGIS_ROOT=%ProgramFiles%\%%D"
|
||||
)
|
||||
|
||||
:have_root
|
||||
if "%QGIS_ROOT%"=="" (
|
||||
echo gdalbuildvrt wrapper: QGIS not found. Set GDAL2TILES_QGIS_ROOT to the QGIS install directory. 1>&2
|
||||
exit /b 2
|
||||
)
|
||||
if not exist "%QGIS_ROOT%\bin\o4w_env.bat" (
|
||||
echo gdalbuildvrt wrapper: "%QGIS_ROOT%\bin\o4w_env.bat" is missing; is this a QGIS install? 1>&2
|
||||
exit /b 2
|
||||
)
|
||||
if not exist "%QGIS_ROOT%\bin\gdalbuildvrt.exe" (
|
||||
echo gdalbuildvrt wrapper: gdalbuildvrt.exe not found under "%QGIS_ROOT%\bin". 1>&2
|
||||
exit /b 2
|
||||
)
|
||||
|
||||
call "%QGIS_ROOT%\bin\o4w_env.bat" >nul 2>&1
|
||||
"%QGIS_ROOT%\bin\gdalbuildvrt.exe" %*
|
||||
exit /b %ERRORLEVEL%
|
||||
+3
-3
@@ -1,7 +1,7 @@
|
||||
@echo off
|
||||
setlocal
|
||||
chcp 65001 >nul 2>&1
|
||||
title Map Asset Gateway
|
||||
title Basemap Pusher
|
||||
|
||||
set "ROOT=%~dp0..\\"
|
||||
call "%ROOT%scripts\load-env.bat" "%ROOT%config\service.env"
|
||||
@@ -17,12 +17,12 @@ if "%API_PORT%"=="" set "API_PORT=8910"
|
||||
if "%API_BASE_URL%"=="" set "API_BASE_URL=http://127.0.0.1:%API_PORT%"
|
||||
if "%TILE_PUBLIC_BASE_URL%"=="" set "TILE_PUBLIC_BASE_URL=%API_BASE_URL%"
|
||||
if "%VECTOR_PUBLIC_BASE_URL%"=="" set "VECTOR_PUBLIC_BASE_URL=%API_BASE_URL%"
|
||||
if "%SQLITE_PATH%"=="" set "SQLITE_PATH=%ROOT%data\map-asset-gateway.db"
|
||||
if "%SQLITE_PATH%"=="" set "SQLITE_PATH=%ROOT%data\basemap-pusher.db"
|
||||
if "%BASEMAP_SCAN_ROOT%"=="" set "BASEMAP_SCAN_ROOT=%ROOT%data\tiles"
|
||||
if "%VECTOR_SCAN_ROOT%"=="" set "VECTOR_SCAN_ROOT=%ROOT%data\geojson"
|
||||
|
||||
echo ========================================
|
||||
echo Map Asset Gateway
|
||||
echo Basemap Pusher
|
||||
echo ========================================
|
||||
echo.
|
||||
echo Env file: %ROOT%config\service.env
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
@echo off
|
||||
setlocal
|
||||
chcp 65001 >nul 2>&1
|
||||
title Basemap Pusher Service
|
||||
|
||||
set "ROOT=%~dp0..\"
|
||||
pushd "%ROOT%" >nul 2>&1
|
||||
set "ROOT=%CD%\"
|
||||
popd
|
||||
|
||||
set "EXE=%ROOT%bin\basemap-pusher.exe"
|
||||
set "ENVFILE=%ROOT%config\service.env"
|
||||
|
||||
if not exist "%EXE%" (
|
||||
echo Service binary not found: %EXE%
|
||||
echo Build it first: scripts\build-api.bat
|
||||
if "%NO_PAUSE%"=="" pause
|
||||
exit /b 1
|
||||
)
|
||||
|
||||
set "VERB=%~1"
|
||||
if "%VERB%"=="" set "VERB=status"
|
||||
|
||||
if /i "%VERB%"=="install" goto do_install
|
||||
if /i "%VERB%"=="uninstall" goto do_uninstall
|
||||
if /i "%VERB%"=="start" goto do_start
|
||||
if /i "%VERB%"=="stop" goto do_stop
|
||||
if /i "%VERB%"=="restart" goto do_restart
|
||||
if /i "%VERB%"=="status" goto do_status
|
||||
|
||||
echo Usage: service.bat ^<install^|uninstall^|start^|stop^|restart^|status^>
|
||||
exit /b 2
|
||||
|
||||
:do_install
|
||||
"%EXE%" --env-file "%ENVFILE%" install
|
||||
goto finish
|
||||
|
||||
:do_uninstall
|
||||
"%EXE%" uninstall
|
||||
goto finish
|
||||
|
||||
:do_start
|
||||
"%EXE%" start
|
||||
goto finish
|
||||
|
||||
:do_stop
|
||||
"%EXE%" stop
|
||||
goto finish
|
||||
|
||||
:do_restart
|
||||
"%EXE%" stop
|
||||
"%EXE%" start
|
||||
goto finish
|
||||
|
||||
:do_status
|
||||
"%EXE%" status
|
||||
goto finish
|
||||
|
||||
:finish
|
||||
set "RC=%ERRORLEVEL%"
|
||||
if not "%RC%"=="0" (
|
||||
echo.
|
||||
echo [FAIL] exit code %RC%.
|
||||
echo install / uninstall / start / stop need an elevated shell ^(Run as Administrator^).
|
||||
)
|
||||
if "%NO_PAUSE%"=="" pause
|
||||
exit /b %RC%
|
||||
+136
-18
@@ -2,36 +2,111 @@ package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"map-asset-gateway/api-go/internal/basemap"
|
||||
"map-asset-gateway/api-go/internal/basemapapi"
|
||||
"map-asset-gateway/api-go/internal/config"
|
||||
"tile-server/api-go/internal/basemap"
|
||||
"tile-server/api-go/internal/basemapapi"
|
||||
"tile-server/api-go/internal/config"
|
||||
)
|
||||
|
||||
const shutdownGrace = 15 * time.Second
|
||||
|
||||
type cliArgs struct {
|
||||
command string
|
||||
envFile string
|
||||
}
|
||||
|
||||
func main() {
|
||||
args, err := parseArgs(os.Args[1:])
|
||||
if err != nil {
|
||||
log.Fatalf("parse arguments: %v", err)
|
||||
}
|
||||
if args.envFile != "" {
|
||||
if err := os.Setenv("APP_ENV_FILE", args.envFile); err != nil {
|
||||
log.Fatalf("set APP_ENV_FILE: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Service management verbs (install / uninstall / start / stop / status).
|
||||
if handled, err := handleServiceArgs(args.command, args.envFile); handled {
|
||||
if err != nil {
|
||||
log.Fatalf("%v", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// When the SCM launched us there is no console and no SIGINT, so the service
|
||||
// handler owns the lifecycle instead of the signal handler.
|
||||
if runAsWindowsService() {
|
||||
return
|
||||
}
|
||||
|
||||
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
||||
defer stop()
|
||||
if err := run(ctx); err != nil {
|
||||
log.Fatalf("%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func parseArgs(argv []string) (cliArgs, error) {
|
||||
var args cliArgs
|
||||
for i := 0; i < len(argv); i++ {
|
||||
switch argv[i] {
|
||||
case "--env-file":
|
||||
if i+1 >= len(argv) {
|
||||
return args, errors.New("--env-file requires a path")
|
||||
}
|
||||
args.envFile = argv[i+1]
|
||||
i++
|
||||
default:
|
||||
if args.command == "" {
|
||||
args.command = argv[i]
|
||||
}
|
||||
}
|
||||
}
|
||||
return args, nil
|
||||
}
|
||||
|
||||
// run boots the service and blocks until ctx is cancelled or the listener dies.
|
||||
// It owns every resource it creates.
|
||||
func run(ctx context.Context) error {
|
||||
cfg, err := config.Load()
|
||||
if err != nil {
|
||||
log.Fatalf("load config: %v", err)
|
||||
return fmt.Errorf("load config: %w", err)
|
||||
}
|
||||
|
||||
// The admin surface can create tokens that read every basemap, so it must
|
||||
// never be reachable off-host without credentials.
|
||||
if cfg.BindsToNonLoopback() && !cfg.HasAdminCredentials() {
|
||||
return fmt.Errorf("refusing to listen on %s without admin credentials: set ADMIN_TOKEN or ADMIN_USER/ADMIN_PASSWORD", cfg.ListenAddr())
|
||||
}
|
||||
|
||||
store, err := basemap.OpenStore(basemap.StoreConfig{
|
||||
SQLitePath: cfg.SQLitePath,
|
||||
ScanRoot: cfg.BasemapScanRoot,
|
||||
VectorScanRoot: cfg.VectorScanRoot,
|
||||
APIBaseURL: cfg.APIBaseURL,
|
||||
TileBaseURL: cfg.TilePublicBaseURL,
|
||||
VectorBaseURL: cfg.VectorPublicBaseURL,
|
||||
AutoScanOnStart: cfg.AutoScanOnStart,
|
||||
SQLitePath: cfg.SQLitePath,
|
||||
ScanRoot: cfg.BasemapScanRoot,
|
||||
VectorScanRoot: cfg.VectorScanRoot,
|
||||
APIBaseURL: cfg.APIBaseURL,
|
||||
TileBaseURL: cfg.TilePublicBaseURL,
|
||||
VectorBaseURL: cfg.VectorPublicBaseURL,
|
||||
AutoScanOnStart: cfg.AutoScanOnStart,
|
||||
TokenDefaultTTL: cfg.TokenDefaultTTL,
|
||||
TileJobCommand: cfg.TileJobCommand,
|
||||
TileJobVRTCommand: cfg.TileJobVRTCommand,
|
||||
TileJobLogDir: cfg.TileJobLogDir,
|
||||
HealthCheckInterval: cfg.HealthCheckInterval,
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatalf("open sqlite store: %v", err)
|
||||
return fmt.Errorf("open sqlite store: %w", err)
|
||||
}
|
||||
defer func() {
|
||||
if err := store.Close(); err != nil {
|
||||
@@ -50,6 +125,7 @@ func main() {
|
||||
SQLitePath: cfg.SQLitePath,
|
||||
ScanRoot: cfg.BasemapScanRoot,
|
||||
VectorScanRoot: cfg.VectorScanRoot,
|
||||
TileJobCommand: cfg.TileJobCommand,
|
||||
StartedAt: time.Now().UTC(),
|
||||
}
|
||||
|
||||
@@ -68,10 +144,21 @@ func main() {
|
||||
}
|
||||
}
|
||||
|
||||
server, err := basemapapi.New(store, systemInfo)
|
||||
// Health inspection, push retries and the tile queue run in the background.
|
||||
store.StartBackgroundWorkers()
|
||||
|
||||
server, err := basemapapi.New(store, systemInfo, basemapapi.Options{
|
||||
Admin: basemapapi.AdminAuth{
|
||||
Token: cfg.AdminToken,
|
||||
User: cfg.AdminUser,
|
||||
Password: cfg.AdminPassword,
|
||||
},
|
||||
CORSAllowedOrigins: cfg.CORSAllowedOrigins,
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatalf("init api server: %v", err)
|
||||
return fmt.Errorf("init api server: %w", err)
|
||||
}
|
||||
|
||||
httpServer := &http.Server{
|
||||
Addr: cfg.ListenAddr(),
|
||||
Handler: server.Handler(),
|
||||
@@ -79,22 +166,53 @@ func main() {
|
||||
}
|
||||
cleanupPID, err := writePIDFile(strings.TrimSpace(os.Getenv("RS_API_PID_FILE")))
|
||||
if err != nil {
|
||||
log.Fatalf("write pid file: %v", err)
|
||||
return fmt.Errorf("write pid file: %w", err)
|
||||
}
|
||||
if cleanupPID != nil {
|
||||
defer cleanupPID()
|
||||
}
|
||||
|
||||
log.Printf("map asset gateway listening on http://%s", cfg.ListenAddr())
|
||||
log.Printf("api base url: %s", cfg.APIBaseURL)
|
||||
log.Printf("tile public base url: %s", cfg.TilePublicBaseURL)
|
||||
log.Printf("vector public base url: %s", cfg.VectorPublicBaseURL)
|
||||
log.Printf("sqlite path: %s", cfg.SQLitePath)
|
||||
log.Printf("scan root: %s", cfg.BasemapScanRoot)
|
||||
log.Printf("vector scan root: %s", cfg.VectorScanRoot)
|
||||
if cfg.HasAdminCredentials() {
|
||||
log.Printf("admin auth: enabled")
|
||||
} else {
|
||||
log.Printf("admin auth: disabled (loopback only)")
|
||||
}
|
||||
|
||||
if err := httpServer.ListenAndServe(); err != nil && err != http.ErrServerClosed {
|
||||
log.Fatalf("listen and serve: %v", err)
|
||||
serveErr := make(chan error, 1)
|
||||
go func() {
|
||||
log.Printf("basemap pusher listening on http://%s", cfg.ListenAddr())
|
||||
if err := httpServer.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
|
||||
serveErr <- err
|
||||
return
|
||||
}
|
||||
serveErr <- nil
|
||||
}()
|
||||
|
||||
select {
|
||||
case err := <-serveErr:
|
||||
if err != nil {
|
||||
return fmt.Errorf("listen and serve: %w", err)
|
||||
}
|
||||
return nil
|
||||
case <-ctx.Done():
|
||||
log.Printf("shutdown signal received, draining connections")
|
||||
shutdownCtx, cancel := context.WithTimeout(context.Background(), shutdownGrace)
|
||||
defer cancel()
|
||||
if err := httpServer.Shutdown(shutdownCtx); err != nil {
|
||||
log.Printf("graceful shutdown incomplete: %v", err)
|
||||
_ = httpServer.Close()
|
||||
}
|
||||
if err := <-serveErr; err != nil {
|
||||
log.Printf("server stopped with error: %v", err)
|
||||
}
|
||||
log.Printf("shutdown complete")
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
//go:build !windows
|
||||
|
||||
package main
|
||||
|
||||
// Service management is Windows-only; other platforms run the console binary.
|
||||
func handleServiceArgs(command, envFile string) (bool, error) { return false, nil }
|
||||
|
||||
func runAsWindowsService() bool { return false }
|
||||
@@ -0,0 +1,376 @@
|
||||
//go:build windows
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"golang.org/x/sys/windows/svc"
|
||||
"golang.org/x/sys/windows/svc/mgr"
|
||||
)
|
||||
|
||||
const (
|
||||
serviceName = "BasemapPusher"
|
||||
serviceDisplayName = "Basemap Pusher"
|
||||
serviceDescription = "Internal catalog and push gateway for existing basemap tiles and vector assets."
|
||||
// appEnvFileFlag is baked into the service command line so the service finds
|
||||
// its config file even though the SCM starts it in C:\Windows\System32.
|
||||
appEnvFileFlag = "--env-file"
|
||||
)
|
||||
|
||||
// handleServiceArgs implements the install/uninstall/start/stop/status verbs.
|
||||
// The bool reports whether the verb was recognised.
|
||||
func handleServiceArgs(command, envFile string) (bool, error) {
|
||||
switch command {
|
||||
case "install":
|
||||
return true, installService(envFile)
|
||||
case "uninstall":
|
||||
return true, uninstallService()
|
||||
case "start":
|
||||
return true, startService()
|
||||
case "stop":
|
||||
return true, stopService()
|
||||
case "status":
|
||||
return true, serviceStatus()
|
||||
default:
|
||||
return false, nil
|
||||
}
|
||||
}
|
||||
|
||||
// runAsWindowsService reports whether the process was launched by the SCM, and
|
||||
// if so runs it to completion. It returns false for a normal console launch.
|
||||
func runAsWindowsService() bool {
|
||||
isService, err := svc.IsWindowsService()
|
||||
if err != nil || !isService {
|
||||
return false
|
||||
}
|
||||
restoreLog := setupServiceLogging()
|
||||
defer restoreLog()
|
||||
|
||||
if err := svc.Run(serviceName, &serviceHandler{}); err != nil {
|
||||
log.Printf("windows service run: %v", err)
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
type serviceHandler struct{}
|
||||
|
||||
func (h *serviceHandler) Execute(args []string, r <-chan svc.ChangeRequest, changes chan<- svc.Status) (bool, uint32) {
|
||||
changes <- svc.Status{State: svc.StartPending}
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- run(ctx) }()
|
||||
|
||||
changes <- svc.Status{State: svc.Running, Accepts: svc.AcceptStop | svc.AcceptShutdown}
|
||||
|
||||
for {
|
||||
select {
|
||||
case err := <-done:
|
||||
if err != nil {
|
||||
log.Printf("service stopped with error: %v", err)
|
||||
changes <- svc.Status{State: svc.Stopped}
|
||||
return false, 1
|
||||
}
|
||||
changes <- svc.Status{State: svc.Stopped}
|
||||
return false, 0
|
||||
case change := <-r:
|
||||
switch change.Cmd {
|
||||
case svc.Interrogate:
|
||||
changes <- change.CurrentStatus
|
||||
case svc.Stop, svc.Shutdown:
|
||||
changes <- svc.Status{State: svc.StopPending}
|
||||
cancel()
|
||||
// Give the HTTP server its drain window before reporting stopped.
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(shutdownGrace + 5*time.Second):
|
||||
log.Printf("service did not stop within the drain window")
|
||||
}
|
||||
changes <- svc.Status{State: svc.Stopped}
|
||||
return false, 0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func installService(envFile string) error {
|
||||
exe, err := os.Executable()
|
||||
if err != nil {
|
||||
return fmt.Errorf("resolve executable: %w", err)
|
||||
}
|
||||
exe, err = filepath.Abs(exe)
|
||||
if err != nil {
|
||||
return fmt.Errorf("resolve executable path: %w", err)
|
||||
}
|
||||
|
||||
resolvedEnv := envFile
|
||||
if resolvedEnv == "" {
|
||||
resolvedEnv = findDefaultEnvFile(filepath.Dir(exe))
|
||||
}
|
||||
if resolvedEnv == "" {
|
||||
return fmt.Errorf("no config/service.env found next to %s; pass --env-file <path> to install", exe)
|
||||
}
|
||||
resolvedEnv, err = filepath.Abs(resolvedEnv)
|
||||
if err != nil {
|
||||
return fmt.Errorf("resolve env file: %w", err)
|
||||
}
|
||||
if _, err := os.Stat(resolvedEnv); err != nil {
|
||||
return fmt.Errorf("env file %s is not readable: %w", resolvedEnv, err)
|
||||
}
|
||||
|
||||
manager, err := mgr.Connect()
|
||||
if err != nil {
|
||||
return fmt.Errorf("connect to the service manager (run this as Administrator): %w", err)
|
||||
}
|
||||
defer manager.Disconnect()
|
||||
|
||||
if existing, err := manager.OpenService(serviceName); err == nil {
|
||||
existing.Close()
|
||||
return fmt.Errorf("service %q already exists; run uninstall first", serviceName)
|
||||
}
|
||||
|
||||
service, err := manager.CreateService(serviceName, exe, mgr.Config{
|
||||
DisplayName: serviceDisplayName,
|
||||
Description: serviceDescription,
|
||||
StartType: mgr.StartAutomatic,
|
||||
}, appEnvFileFlag, resolvedEnv)
|
||||
if err != nil {
|
||||
return fmt.Errorf("create service: %w", err)
|
||||
}
|
||||
defer service.Close()
|
||||
|
||||
// Restart on failure: 5s, 15s, 60s, then keep trying within the reset window.
|
||||
if err := service.SetRecoveryActions([]mgr.RecoveryAction{
|
||||
{Type: mgr.ServiceRestart, Delay: 5 * time.Second},
|
||||
{Type: mgr.ServiceRestart, Delay: 15 * time.Second},
|
||||
{Type: mgr.ServiceRestart, Delay: 60 * time.Second},
|
||||
}, 86400); err != nil {
|
||||
log.Printf("warning: could not set recovery actions: %v", err)
|
||||
}
|
||||
|
||||
log.Printf("installed service %q", serviceName)
|
||||
log.Printf(" executable: %s", exe)
|
||||
log.Printf(" env file: %s", resolvedEnv)
|
||||
log.Printf(" start with: %s start", exe)
|
||||
return nil
|
||||
}
|
||||
|
||||
func uninstallService() error {
|
||||
manager, err := mgr.Connect()
|
||||
if err != nil {
|
||||
return fmt.Errorf("connect to the service manager (run this as Administrator): %w", err)
|
||||
}
|
||||
defer manager.Disconnect()
|
||||
|
||||
service, err := manager.OpenService(serviceName)
|
||||
if err != nil {
|
||||
return fmt.Errorf("service %q is not installed", serviceName)
|
||||
}
|
||||
defer service.Close()
|
||||
|
||||
if status, err := service.Query(); err == nil && status.State != svc.Stopped {
|
||||
_, _ = service.Control(svc.Stop)
|
||||
time.Sleep(2 * time.Second)
|
||||
}
|
||||
if err := service.Delete(); err != nil {
|
||||
return fmt.Errorf("delete service: %w", err)
|
||||
}
|
||||
log.Printf("uninstalled service %q", serviceName)
|
||||
return nil
|
||||
}
|
||||
|
||||
func startService() error {
|
||||
manager, err := mgr.Connect()
|
||||
if err != nil {
|
||||
return fmt.Errorf("connect to the service manager: %w", err)
|
||||
}
|
||||
defer manager.Disconnect()
|
||||
|
||||
service, err := manager.OpenService(serviceName)
|
||||
if err != nil {
|
||||
return fmt.Errorf("service %q is not installed", serviceName)
|
||||
}
|
||||
defer service.Close()
|
||||
|
||||
if err := service.Start(); err != nil {
|
||||
return fmt.Errorf("start service: %w", err)
|
||||
}
|
||||
log.Printf("started service %q", serviceName)
|
||||
return nil
|
||||
}
|
||||
|
||||
func stopService() error {
|
||||
manager, err := mgr.Connect()
|
||||
if err != nil {
|
||||
return fmt.Errorf("connect to the service manager: %w", err)
|
||||
}
|
||||
defer manager.Disconnect()
|
||||
|
||||
service, err := manager.OpenService(serviceName)
|
||||
if err != nil {
|
||||
return fmt.Errorf("service %q is not installed", serviceName)
|
||||
}
|
||||
defer service.Close()
|
||||
|
||||
status, err := service.Control(svc.Stop)
|
||||
if err != nil {
|
||||
return fmt.Errorf("stop service: %w", err)
|
||||
}
|
||||
deadline := time.Now().Add(30 * time.Second)
|
||||
for status.State != svc.Stopped && time.Now().Before(deadline) {
|
||||
time.Sleep(time.Second)
|
||||
status, err = service.Query()
|
||||
if err != nil {
|
||||
return fmt.Errorf("query service: %w", err)
|
||||
}
|
||||
}
|
||||
log.Printf("stopped service %q", serviceName)
|
||||
return nil
|
||||
}
|
||||
|
||||
func serviceStatus() error {
|
||||
manager, err := mgr.Connect()
|
||||
if err != nil {
|
||||
return fmt.Errorf("connect to the service manager: %w", err)
|
||||
}
|
||||
defer manager.Disconnect()
|
||||
|
||||
service, err := manager.OpenService(serviceName)
|
||||
if err != nil {
|
||||
return fmt.Errorf("service %q is not installed", serviceName)
|
||||
}
|
||||
defer service.Close()
|
||||
|
||||
status, err := service.Query()
|
||||
if err != nil {
|
||||
return fmt.Errorf("query service: %w", err)
|
||||
}
|
||||
names := map[svc.State]string{
|
||||
svc.Stopped: "stopped",
|
||||
svc.StartPending: "start pending",
|
||||
svc.StopPending: "stop pending",
|
||||
svc.Running: "running",
|
||||
svc.ContinuePending: "continue pending",
|
||||
svc.PausePending: "pause pending",
|
||||
svc.Paused: "paused",
|
||||
}
|
||||
name, ok := names[status.State]
|
||||
if !ok {
|
||||
name = fmt.Sprintf("state %d", status.State)
|
||||
}
|
||||
fmt.Printf("%s: %s\n", serviceName, name)
|
||||
return nil
|
||||
}
|
||||
|
||||
// findDefaultEnvFile looks in the usual layouts: next to the executable, then
|
||||
// one and two levels up (services/api-go -> repo root).
|
||||
func findDefaultEnvFile(startDir string) string {
|
||||
candidates := []string{
|
||||
filepath.Join(startDir, "config", "service.env"),
|
||||
filepath.Join(startDir, "..", "config", "service.env"),
|
||||
filepath.Join(startDir, "..", "..", "config", "service.env"),
|
||||
}
|
||||
for _, candidate := range candidates {
|
||||
if _, err := os.Stat(candidate); err == nil {
|
||||
return candidate
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// setupServiceLogging points the standard logger at a file, because a Windows
|
||||
// service has no console and would otherwise discard every log line. It returns
|
||||
// a restore func that is safe to call even when nothing was configured.
|
||||
func setupServiceLogging() func() {
|
||||
path := strings.TrimSpace(os.Getenv("SERVICE_LOG_FILE"))
|
||||
if path == "" {
|
||||
exe, err := os.Executable()
|
||||
if err != nil {
|
||||
return func() {}
|
||||
}
|
||||
path = filepath.Join(filepath.Dir(exe), "logs", "basemap-pusher.log")
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
return func() {}
|
||||
}
|
||||
writer, err := newRotatingFileWriter(path, serviceLogMaxBytes)
|
||||
if err != nil {
|
||||
return func() {}
|
||||
}
|
||||
log.SetOutput(writer)
|
||||
log.Printf("service log: %s (rotates at %d MB)", path, serviceLogMaxBytes>>20)
|
||||
return func() { _ = writer.Close() }
|
||||
}
|
||||
|
||||
const serviceLogMaxBytes = 10 << 20
|
||||
|
||||
// rotatingFileWriter appends to one file and rolls it to ".1" once it grows
|
||||
// past maxBytes, so a long-running service cannot fill the disk.
|
||||
type rotatingFileWriter struct {
|
||||
path string
|
||||
maxBytes int64
|
||||
file *os.File
|
||||
size int64
|
||||
}
|
||||
|
||||
func newRotatingFileWriter(path string, maxBytes int64) (*rotatingFileWriter, error) {
|
||||
writer := &rotatingFileWriter{path: path, maxBytes: maxBytes}
|
||||
if err := writer.open(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return writer, nil
|
||||
}
|
||||
|
||||
func (w *rotatingFileWriter) open() error {
|
||||
file, err := os.OpenFile(w.path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o644)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
info, err := file.Stat()
|
||||
if err != nil {
|
||||
file.Close()
|
||||
return err
|
||||
}
|
||||
w.file = file
|
||||
w.size = info.Size()
|
||||
return nil
|
||||
}
|
||||
|
||||
func (w *rotatingFileWriter) Write(p []byte) (int, error) {
|
||||
if w.maxBytes > 0 && w.size+int64(len(p)) > w.maxBytes {
|
||||
w.roll()
|
||||
}
|
||||
n, err := w.file.Write(p)
|
||||
w.size += int64(n)
|
||||
return n, err
|
||||
}
|
||||
|
||||
func (w *rotatingFileWriter) roll() {
|
||||
if w.file != nil {
|
||||
_ = w.file.Close()
|
||||
}
|
||||
// Windows refuses to rename onto an existing path, so drop the old roll.
|
||||
_ = os.Remove(w.path + ".1")
|
||||
_ = os.Rename(w.path, w.path+".1")
|
||||
w.size = 0
|
||||
if err := w.open(); err != nil {
|
||||
// Fall back to appending to the original path if the reopen failed.
|
||||
_ = err
|
||||
}
|
||||
}
|
||||
|
||||
func (w *rotatingFileWriter) Close() error {
|
||||
if w.file == nil {
|
||||
return nil
|
||||
}
|
||||
return w.file.Close()
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
//go:build windows
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestRotatingFileWriterRollsOver(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "svc.log")
|
||||
|
||||
writer, err := newRotatingFileWriter(path, 64)
|
||||
if err != nil {
|
||||
t.Fatalf("open writer: %v", err)
|
||||
}
|
||||
defer writer.Close()
|
||||
|
||||
for i := 0; i < 10; i++ {
|
||||
if _, err := writer.Write([]byte(strings.Repeat("x", 20) + "\n")); err != nil {
|
||||
t.Fatalf("write %d: %v", i, err)
|
||||
}
|
||||
}
|
||||
|
||||
if _, err := os.Stat(path); err != nil {
|
||||
t.Fatalf("current log missing: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(path + ".1"); err != nil {
|
||||
t.Fatalf("expected a rolled file at %s.1: %v", path, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSetupServiceLoggingUsesConfiguredPath(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "nested", "service.log")
|
||||
t.Setenv("SERVICE_LOG_FILE", path)
|
||||
|
||||
restore := setupServiceLogging()
|
||||
defer restore()
|
||||
|
||||
if _, err := os.Stat(path); err != nil {
|
||||
t.Fatalf("expected the log file at %s: %v", path, err)
|
||||
}
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
module map-asset-gateway/api-go
|
||||
module tile-server/api-go
|
||||
|
||||
go 1.26
|
||||
|
||||
|
||||
@@ -12,10 +12,15 @@ import (
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"map-asset-gateway/api-go/internal/uid"
|
||||
"tile-server/api-go/internal/uid"
|
||||
)
|
||||
|
||||
// lastUsedUpdateInterval throttles service_tokens.last_used_at writes so tile
|
||||
// traffic does not issue one UPDATE per tile request.
|
||||
const lastUsedUpdateInterval = time.Minute
|
||||
|
||||
var ErrBasemapVersionNotFound = errors.New("basemap version not found")
|
||||
|
||||
func (s *Store) CreateToken(ctx context.Context, input CreateTokenInput) (CreatedToken, error) {
|
||||
@@ -23,11 +28,16 @@ func (s *Store) CreateToken(ctx context.Context, input CreateTokenInput) (Create
|
||||
if name == "" {
|
||||
return CreatedToken{}, errors.New("token name is required")
|
||||
}
|
||||
if len(input.BasemapCodes) == 0 {
|
||||
if len(input.VectorCodes) == 0 {
|
||||
return CreatedToken{}, errors.New("at least one basemap code or vector code is required")
|
||||
// A token needs at least one capability: read grants, or the slice scope.
|
||||
hasSliceScope := false
|
||||
for _, value := range input.Scopes {
|
||||
if strings.EqualFold(strings.TrimSpace(value), slicePermission) {
|
||||
hasSliceScope = true
|
||||
}
|
||||
}
|
||||
if len(input.BasemapCodes) == 0 && len(input.VectorCodes) == 0 && !hasSliceScope {
|
||||
return CreatedToken{}, errors.New("at least one basemap code, vector code, or the slice scope is required")
|
||||
}
|
||||
|
||||
resolvedVectorAssets := map[string]VectorAsset{}
|
||||
for _, ref := range input.VectorCodes {
|
||||
@@ -54,15 +64,21 @@ func (s *Store) CreateToken(ctx context.Context, input CreateTokenInput) (Create
|
||||
}
|
||||
|
||||
now := nowUTC()
|
||||
// Fall back to the configured default lifetime so tokens do not live forever
|
||||
// unless an explicit expiry was supplied.
|
||||
if input.ExpiresAt == nil && s.cfg.TokenDefaultTTL > 0 {
|
||||
expiresAt := now.Add(s.cfg.TokenDefaultTTL)
|
||||
input.ExpiresAt = &expiresAt
|
||||
}
|
||||
tokenID := uid.New()
|
||||
prefix := token
|
||||
if len(prefix) > 16 {
|
||||
prefix = prefix[:16]
|
||||
}
|
||||
_, err = tx.ExecContext(ctx, `
|
||||
INSERT INTO service_tokens (id, name, token_hash, token_prefix, status, expires_at, last_used_at, created_at)
|
||||
VALUES (?, ?, ?, ?, 'active', ?, NULL, ?)
|
||||
`, tokenID, name, tokenHash, prefix, nullableTime(input.ExpiresAt), toRFC3339(now))
|
||||
INSERT INTO service_tokens (id, name, token_hash, token_prefix, scopes, status, expires_at, last_used_at, created_at)
|
||||
VALUES (?, ?, ?, ?, ?, 'active', ?, NULL, ?)
|
||||
`, tokenID, name, tokenHash, prefix, normalizeTokenScopes(input.Scopes), nullableTime(input.ExpiresAt), toRFC3339(now))
|
||||
if err != nil {
|
||||
return CreatedToken{}, err
|
||||
}
|
||||
@@ -75,7 +91,7 @@ func (s *Store) CreateToken(ctx context.Context, input CreateTokenInput) (Create
|
||||
}
|
||||
}
|
||||
if len(normalizedCodes) == 0 {
|
||||
if len(resolvedVectorAssets) == 0 {
|
||||
if len(resolvedVectorAssets) == 0 && !hasSliceScope {
|
||||
return CreatedToken{}, errors.New("no valid basemap codes provided")
|
||||
}
|
||||
}
|
||||
@@ -129,7 +145,7 @@ func (s *Store) CreateToken(ctx context.Context, input CreateTokenInput) (Create
|
||||
|
||||
func (s *Store) ListTokens(ctx context.Context) ([]ServiceToken, error) {
|
||||
rows, err := s.db.QueryContext(ctx, `
|
||||
SELECT id, name, token_hash, token_prefix, status, expires_at, last_used_at, created_at
|
||||
SELECT id, name, token_hash, token_prefix, scopes, status, expires_at, last_used_at, created_at
|
||||
FROM service_tokens
|
||||
ORDER BY created_at DESC
|
||||
`)
|
||||
@@ -226,6 +242,69 @@ func (s *Store) loadAllVectorTokenGrants(ctx context.Context) (map[string][]Vect
|
||||
return result, rows.Err()
|
||||
}
|
||||
|
||||
func (s *Store) loadTokenGrants(ctx context.Context, tokenID string) ([]TokenGrant, error) {
|
||||
rows, err := s.db.QueryContext(ctx, `
|
||||
SELECT
|
||||
g.id,
|
||||
g.token_id,
|
||||
g.basemap_id,
|
||||
b.code,
|
||||
g.basemap_version_id,
|
||||
v.version,
|
||||
g.permission,
|
||||
g.created_at
|
||||
FROM token_grants g
|
||||
JOIN basemaps b ON b.id = g.basemap_id
|
||||
LEFT JOIN basemap_versions v ON v.id = g.basemap_version_id
|
||||
WHERE g.token_id = ?
|
||||
ORDER BY g.created_at DESC
|
||||
`, tokenID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var items []TokenGrant
|
||||
for rows.Next() {
|
||||
item, err := scanTokenGrant(rows)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
items = append(items, item)
|
||||
}
|
||||
return items, rows.Err()
|
||||
}
|
||||
|
||||
func (s *Store) loadVectorTokenGrants(ctx context.Context, tokenID string) ([]VectorTokenGrant, error) {
|
||||
rows, err := s.db.QueryContext(ctx, `
|
||||
SELECT
|
||||
g.id,
|
||||
g.token_id,
|
||||
g.vector_asset_id,
|
||||
v.code,
|
||||
g.permission,
|
||||
g.created_at
|
||||
FROM vector_token_grants g
|
||||
JOIN vector_assets v ON v.id = g.vector_asset_id
|
||||
WHERE g.token_id = ?
|
||||
ORDER BY g.created_at DESC
|
||||
`, tokenID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var items []VectorTokenGrant
|
||||
for rows.Next() {
|
||||
item, err := scanVectorTokenGrant(rows)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
items = append(items, item)
|
||||
}
|
||||
return items, rows.Err()
|
||||
}
|
||||
|
||||
func (s *Store) DisableToken(ctx context.Context, tokenID string) error {
|
||||
result, err := s.db.ExecContext(ctx, `UPDATE service_tokens SET status = 'disabled' WHERE id = ?`, strings.TrimSpace(tokenID))
|
||||
if err != nil {
|
||||
@@ -243,7 +322,7 @@ func (s *Store) DisableToken(ctx context.Context, tokenID string) error {
|
||||
|
||||
func (s *Store) getTokenByID(ctx context.Context, tokenID string) (ServiceToken, error) {
|
||||
row := s.db.QueryRowContext(ctx, `
|
||||
SELECT id, name, token_hash, token_prefix, status, expires_at, last_used_at, created_at
|
||||
SELECT id, name, token_hash, token_prefix, scopes, status, expires_at, last_used_at, created_at
|
||||
FROM service_tokens
|
||||
WHERE id = ?
|
||||
`, tokenID)
|
||||
@@ -273,7 +352,7 @@ func (s *Store) AuthorizeToken(ctx context.Context, rawToken string) (TokenAuth,
|
||||
tokenHash := hex.EncodeToString(sum[:])
|
||||
|
||||
row := s.db.QueryRowContext(ctx, `
|
||||
SELECT id, name, token_hash, token_prefix, status, expires_at, last_used_at, created_at
|
||||
SELECT id, name, token_hash, token_prefix, scopes, status, expires_at, last_used_at, created_at
|
||||
FROM service_tokens
|
||||
WHERE token_hash = ?
|
||||
`, tokenHash)
|
||||
@@ -291,16 +370,16 @@ func (s *Store) AuthorizeToken(ctx context.Context, rawToken string) (TokenAuth,
|
||||
return TokenAuth{}, errors.New("token is expired")
|
||||
}
|
||||
|
||||
grantsByToken, err := s.loadAllTokenGrants(ctx)
|
||||
grants, err := s.loadTokenGrants(ctx, item.ID)
|
||||
if err != nil {
|
||||
return TokenAuth{}, err
|
||||
}
|
||||
item.Grants = grantsByToken[item.ID]
|
||||
vectorGrantsByToken, err := s.loadAllVectorTokenGrants(ctx)
|
||||
item.Grants = grants
|
||||
vectorGrants, err := s.loadVectorTokenGrants(ctx, item.ID)
|
||||
if err != nil {
|
||||
return TokenAuth{}, err
|
||||
}
|
||||
item.VectorGrants = vectorGrantsByToken[item.ID]
|
||||
item.VectorGrants = vectorGrants
|
||||
permissions := map[string]map[string]VersionScope{}
|
||||
for _, grant := range item.Grants {
|
||||
group, ok := permissions[grant.Permission]
|
||||
@@ -329,14 +408,64 @@ func (s *Store) AuthorizeToken(ctx context.Context, rawToken string) (TokenAuth,
|
||||
}
|
||||
group[grant.VectorCode] = struct{}{}
|
||||
}
|
||||
_, _ = s.db.ExecContext(ctx, `UPDATE service_tokens SET last_used_at = ? WHERE id = ?`, toRFC3339(nowUTC()), item.ID)
|
||||
if item.LastUsedAt == nil || nowUTC().Sub(*item.LastUsedAt) >= lastUsedUpdateInterval {
|
||||
_, _ = s.db.ExecContext(ctx, `UPDATE service_tokens SET last_used_at = ? WHERE id = ?`, toRFC3339(nowUTC()), item.ID)
|
||||
}
|
||||
return TokenAuth{
|
||||
Token: item,
|
||||
Permissions: permissions,
|
||||
VectorPermissions: vectorPermissions,
|
||||
Scopes: scopeSet(item.Scopes),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Token scopes are coarse capabilities layered on top of per-resource grants.
|
||||
// Every token can read; "slice" additionally allows submitting tile jobs.
|
||||
const slicePermission = "slice"
|
||||
|
||||
var allowedTokenScopes = map[string]struct{}{
|
||||
readPermission: {},
|
||||
slicePermission: {},
|
||||
}
|
||||
|
||||
// normalizeTokenScopes keeps only known scopes and always includes read, so the
|
||||
// stored column is a stable, sorted, comma-separated list.
|
||||
func normalizeTokenScopes(values []string) string {
|
||||
scopes := map[string]struct{}{readPermission: {}}
|
||||
for _, value := range values {
|
||||
scope := strings.ToLower(strings.TrimSpace(value))
|
||||
if _, ok := allowedTokenScopes[scope]; ok {
|
||||
scopes[scope] = struct{}{}
|
||||
}
|
||||
}
|
||||
return strings.Join(sortedKeys(scopes), ",")
|
||||
}
|
||||
|
||||
func splitTokenScopes(value string) []string {
|
||||
parts := strings.Split(value, ",")
|
||||
scopes := make([]string, 0, len(parts))
|
||||
for _, part := range parts {
|
||||
scope := strings.ToLower(strings.TrimSpace(part))
|
||||
if scope != "" {
|
||||
scopes = append(scopes, scope)
|
||||
}
|
||||
}
|
||||
return scopes
|
||||
}
|
||||
|
||||
func scopeSet(values []string) map[string]struct{} {
|
||||
scopes := make(map[string]struct{}, len(values))
|
||||
for _, value := range values {
|
||||
scopes[value] = struct{}{}
|
||||
}
|
||||
return scopes
|
||||
}
|
||||
|
||||
// CanSlice reports whether the token may submit tile jobs.
|
||||
func (auth TokenAuth) CanSlice() bool {
|
||||
_, ok := auth.Scopes[slicePermission]
|
||||
return ok
|
||||
}
|
||||
func (auth TokenAuth) CanReadBasemap(code string, version string) bool {
|
||||
group, ok := auth.Permissions[readPermission]
|
||||
if !ok {
|
||||
|
||||
@@ -8,7 +8,7 @@ import (
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"map-asset-gateway/api-go/internal/uid"
|
||||
"tile-server/api-go/internal/uid"
|
||||
)
|
||||
|
||||
func (s *Store) EnsureScanSource(ctx context.Context, input CreateScanSourceInput) (ScanSource, error) {
|
||||
@@ -61,7 +61,7 @@ func (s *Store) GetScanSourceByCode(ctx context.Context, code string) (ScanSourc
|
||||
item, err := scanScanSource(row)
|
||||
if err != nil {
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return ScanSource{}, fmt.Errorf("scan source %q not found", code)
|
||||
return ScanSource{}, &scanSourceError{sentinel: ErrScanSourceNotFound, code: code, reason: "not found"}
|
||||
}
|
||||
return ScanSource{}, err
|
||||
}
|
||||
@@ -168,17 +168,66 @@ func (s *Store) ListBasemaps(ctx context.Context) ([]Basemap, error) {
|
||||
}
|
||||
|
||||
func (s *Store) GetBasemapByCode(ctx context.Context, code string) (Basemap, error) {
|
||||
items, err := s.ListBasemaps(ctx)
|
||||
normalized := normalizeBasemapCode(code)
|
||||
row := s.db.QueryRowContext(ctx, `
|
||||
SELECT id, code, name, type, status, description, created_at, updated_at
|
||||
FROM basemaps
|
||||
WHERE code = ?
|
||||
`, normalized)
|
||||
item, err := scanBasemap(row)
|
||||
if err != nil {
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return Basemap{}, fmt.Errorf("basemap %q not found", code)
|
||||
}
|
||||
return Basemap{}, err
|
||||
}
|
||||
|
||||
versionRows, err := s.db.QueryContext(ctx, `
|
||||
SELECT
|
||||
v.id,
|
||||
v.basemap_id,
|
||||
b.code,
|
||||
v.version,
|
||||
v.status,
|
||||
v.is_default,
|
||||
v.manifest_path,
|
||||
v.tile_root_path,
|
||||
v.url_template,
|
||||
v.tile_format,
|
||||
v.tile_scheme,
|
||||
v.min_zoom,
|
||||
v.max_zoom,
|
||||
v.bbox_json,
|
||||
v.attribution,
|
||||
v.metadata_json,
|
||||
v.created_at,
|
||||
v.updated_at
|
||||
FROM basemap_versions v
|
||||
JOIN basemaps b ON b.id = v.basemap_id
|
||||
WHERE v.basemap_id = ?
|
||||
ORDER BY v.version
|
||||
`, item.ID)
|
||||
if err != nil {
|
||||
return Basemap{}, err
|
||||
}
|
||||
normalized := normalizeBasemapCode(code)
|
||||
for _, item := range items {
|
||||
if item.Code == normalized {
|
||||
return item, nil
|
||||
defer versionRows.Close()
|
||||
|
||||
for versionRows.Next() {
|
||||
row, err := scanBasemapVersionRow(versionRows)
|
||||
if err != nil {
|
||||
return Basemap{}, err
|
||||
}
|
||||
version := decodeBasemapVersion(row)
|
||||
item.Versions = append(item.Versions, version)
|
||||
if version.IsDefault {
|
||||
copyValue := version
|
||||
item.Default = ©Value
|
||||
}
|
||||
}
|
||||
return Basemap{}, fmt.Errorf("basemap %q not found", code)
|
||||
if err := versionRows.Err(); err != nil {
|
||||
return Basemap{}, err
|
||||
}
|
||||
return item, nil
|
||||
}
|
||||
|
||||
func (s *Store) SetDefaultVersion(ctx context.Context, basemapCode, version string) error {
|
||||
@@ -276,6 +325,8 @@ func (s *Store) ListPushRecords(ctx context.Context, limit int) ([]PushRecord, e
|
||||
p.response_status,
|
||||
p.response_body,
|
||||
p.error_message,
|
||||
p.attempt_count,
|
||||
p.next_attempt_at,
|
||||
p.pushed_at,
|
||||
p.finished_at
|
||||
FROM push_records p
|
||||
|
||||
@@ -35,6 +35,22 @@ func (s *Store) Dashboard(ctx context.Context, info SystemInfo, flash string, cr
|
||||
if err != nil {
|
||||
return DashboardData{}, err
|
||||
}
|
||||
versionHealth, err := s.loadVersionHealth(ctx)
|
||||
if err != nil {
|
||||
return DashboardData{}, err
|
||||
}
|
||||
vectorHealth, err := s.loadVectorHealth(ctx)
|
||||
if err != nil {
|
||||
return DashboardData{}, err
|
||||
}
|
||||
tileJobs, err := s.ListTileJobs(ctx, 10)
|
||||
if err != nil {
|
||||
return DashboardData{}, err
|
||||
}
|
||||
layers, err := s.ListLayerInventories(ctx)
|
||||
if err != nil {
|
||||
return DashboardData{}, err
|
||||
}
|
||||
return DashboardData{
|
||||
SystemInfo: info,
|
||||
Basemaps: basemaps,
|
||||
@@ -47,5 +63,9 @@ func (s *Store) Dashboard(ctx context.Context, info SystemInfo, flash string, cr
|
||||
VectorPushRecords: vectorPushRecords,
|
||||
Flash: flash,
|
||||
CreatedToken: createdToken,
|
||||
VersionHealth: versionHealth,
|
||||
VectorHealth: vectorHealth,
|
||||
TileJobs: tileJobs,
|
||||
Layers: layers,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,872 @@
|
||||
package basemap
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestCheckHealthFlagsMissingPaths(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
root := t.TempDir()
|
||||
tileRoot := filepath.Join(root, "tiles")
|
||||
vectorRoot := filepath.Join(root, "vectors")
|
||||
writeTestFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile")
|
||||
writeTestFile(t, filepath.Join(vectorRoot, "boundary.geojson"), `{"type":"FeatureCollection","features":[]}`)
|
||||
|
||||
store := openTestStore(t, StoreConfig{
|
||||
SQLitePath: filepath.Join(root, "basemap.db"),
|
||||
ScanRoot: tileRoot,
|
||||
VectorScanRoot: vectorRoot,
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
TileBaseURL: "http://127.0.0.1:8910",
|
||||
VectorBaseURL: "http://127.0.0.1:8910",
|
||||
})
|
||||
if _, err := store.RunScan(ctx, defaultScanSourceCode); err != nil {
|
||||
t.Fatalf("scan: %v", err)
|
||||
}
|
||||
if _, err := store.RunVectorScan(ctx); err != nil {
|
||||
t.Fatalf("vector scan: %v", err)
|
||||
}
|
||||
|
||||
report, err := store.CheckHealth(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("check health: %v", err)
|
||||
}
|
||||
if report.VersionsMissing != 0 || report.VectorsMissing != 0 {
|
||||
t.Fatalf("expected everything healthy, got %+v", report)
|
||||
}
|
||||
|
||||
// A directory that disappears must be reported before the first 404.
|
||||
if err := os.RemoveAll(filepath.Join(tileRoot, "google_image")); err != nil {
|
||||
t.Fatalf("remove tile root: %v", err)
|
||||
}
|
||||
if err := os.Remove(filepath.Join(vectorRoot, "boundary.geojson")); err != nil {
|
||||
t.Fatalf("remove vector: %v", err)
|
||||
}
|
||||
|
||||
report, err = store.CheckHealth(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("check health after removal: %v", err)
|
||||
}
|
||||
if report.VersionsMissing != 1 || report.VectorsMissing != 1 {
|
||||
t.Fatalf("expected one missing of each, got %+v", report)
|
||||
}
|
||||
|
||||
versionHealth, err := store.loadVersionHealth(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("load version health: %v", err)
|
||||
}
|
||||
assetHealth, err := store.loadVectorHealth(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("load vector health: %v", err)
|
||||
}
|
||||
foundMissing := false
|
||||
for _, health := range versionHealth {
|
||||
if health.Health == healthMissing {
|
||||
foundMissing = true
|
||||
}
|
||||
}
|
||||
if !foundMissing {
|
||||
t.Fatalf("expected a missing version verdict, got %+v", versionHealth)
|
||||
}
|
||||
foundMissing = false
|
||||
for _, health := range assetHealth {
|
||||
if health.Health == healthMissing {
|
||||
foundMissing = true
|
||||
}
|
||||
}
|
||||
if !foundMissing {
|
||||
t.Fatalf("expected a missing vector verdict, got %+v", assetHealth)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPushRetrySucceedsAfterBackoff(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
root := t.TempDir()
|
||||
tileRoot := filepath.Join(root, "tiles")
|
||||
writeTestFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile")
|
||||
|
||||
var attempts int
|
||||
callback := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
attempts++
|
||||
if attempts < 2 {
|
||||
w.WriteHeader(http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}))
|
||||
defer callback.Close()
|
||||
|
||||
store := openTestStore(t, StoreConfig{
|
||||
SQLitePath: filepath.Join(root, "basemap.db"),
|
||||
ScanRoot: tileRoot,
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
TileBaseURL: "http://127.0.0.1:8910",
|
||||
VectorBaseURL: "http://127.0.0.1:8910",
|
||||
})
|
||||
if _, err := store.RunScan(ctx, defaultScanSourceCode); err != nil {
|
||||
t.Fatalf("scan: %v", err)
|
||||
}
|
||||
if _, err := store.CreateTargetSystem(ctx, CreateTargetSystemInput{Code: "downstream", CallbackURL: callback.URL}); err != nil {
|
||||
t.Fatalf("create target: %v", err)
|
||||
}
|
||||
record, err := store.PushBasemapVersion(ctx, "downstream", "google_image", "current")
|
||||
if err != nil {
|
||||
t.Fatalf("push: %v", err)
|
||||
}
|
||||
|
||||
// First attempt fails, so the record must be parked in retrying with a
|
||||
// scheduled next attempt rather than being dropped as failed.
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
var parked bool
|
||||
for time.Now().Before(deadline) {
|
||||
records, err := store.ListPushRecords(ctx, 5)
|
||||
if err != nil {
|
||||
t.Fatalf("list: %v", err)
|
||||
}
|
||||
for _, item := range records {
|
||||
if item.ID == record.ID && item.Status == pushStatusRetrying && item.NextAttemptAt != nil {
|
||||
parked = true
|
||||
}
|
||||
}
|
||||
if parked {
|
||||
break
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
if !parked {
|
||||
t.Fatal("expected the failed push to be rescheduled rather than dropped")
|
||||
}
|
||||
|
||||
// Make the retry due immediately, then run the dispatcher.
|
||||
if _, err := store.db.ExecContext(ctx, `UPDATE push_records SET next_attempt_at = ? WHERE id = ?`, toRFC3339(nowUTC().Add(-time.Minute)), record.ID); err != nil {
|
||||
t.Fatalf("force due: %v", err)
|
||||
}
|
||||
if err := store.dispatchPushRetries(ctx); err != nil {
|
||||
t.Fatalf("dispatch retries: %v", err)
|
||||
}
|
||||
|
||||
records, err := store.ListPushRecords(ctx, 5)
|
||||
if err != nil {
|
||||
t.Fatalf("list after retry: %v", err)
|
||||
}
|
||||
var final PushRecord
|
||||
for _, item := range records {
|
||||
if item.ID == record.ID {
|
||||
final = item
|
||||
}
|
||||
}
|
||||
if final.Status != pushStatusSucceeded {
|
||||
t.Fatalf("expected the retry to succeed, got status=%q message=%q", final.Status, final.ErrorMessage)
|
||||
}
|
||||
if final.AttemptCount != 2 {
|
||||
t.Fatalf("expected 2 attempts, got %d", final.AttemptCount)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManualRetryRequeuesFailedPush(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
root := t.TempDir()
|
||||
tileRoot := filepath.Join(root, "tiles")
|
||||
writeTestFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile")
|
||||
|
||||
callback := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusBadGateway)
|
||||
}))
|
||||
defer callback.Close()
|
||||
|
||||
store := openTestStore(t, StoreConfig{
|
||||
SQLitePath: filepath.Join(root, "basemap.db"),
|
||||
ScanRoot: tileRoot,
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
TileBaseURL: "http://127.0.0.1:8910",
|
||||
VectorBaseURL: "http://127.0.0.1:8910",
|
||||
})
|
||||
if _, err := store.RunScan(ctx, defaultScanSourceCode); err != nil {
|
||||
t.Fatalf("scan: %v", err)
|
||||
}
|
||||
if _, err := store.CreateTargetSystem(ctx, CreateTargetSystemInput{Code: "downstream", CallbackURL: callback.URL}); err != nil {
|
||||
t.Fatalf("create target: %v", err)
|
||||
}
|
||||
record, err := store.PushBasemapVersion(ctx, "downstream", "google_image", "current")
|
||||
if err != nil {
|
||||
t.Fatalf("push: %v", err)
|
||||
}
|
||||
|
||||
// Exhaust the retry budget so the record lands in failed.
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
records, _ := store.ListPushRecords(ctx, 5)
|
||||
for _, item := range records {
|
||||
if item.ID == record.ID && item.Status == pushStatusFailed {
|
||||
goto failed
|
||||
}
|
||||
}
|
||||
if _, err := store.db.ExecContext(ctx, `UPDATE push_records SET next_attempt_at = ? WHERE id = ? AND status = ?`, toRFC3339(nowUTC().Add(-time.Minute)), record.ID, pushStatusRetrying); err != nil {
|
||||
t.Fatalf("force due: %v", err)
|
||||
}
|
||||
_ = store.dispatchPushRetries(ctx)
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
|
||||
failed:
|
||||
records, err := store.ListPushRecords(ctx, 5)
|
||||
if err != nil {
|
||||
t.Fatalf("list: %v", err)
|
||||
}
|
||||
var failedRecord PushRecord
|
||||
for _, item := range records {
|
||||
if item.ID == record.ID {
|
||||
failedRecord = item
|
||||
}
|
||||
}
|
||||
if failedRecord.Status != pushStatusFailed {
|
||||
t.Fatalf("expected a failed record, got %q", failedRecord.Status)
|
||||
}
|
||||
if failedRecord.AttemptCount != pushMaxAttempts {
|
||||
t.Fatalf("expected %d attempts, got %d", pushMaxAttempts, failedRecord.AttemptCount)
|
||||
}
|
||||
|
||||
if err := store.RetryPush(ctx, "basemap", record.ID); err != nil {
|
||||
t.Fatalf("manual retry: %v", err)
|
||||
}
|
||||
records, err = store.ListPushRecords(ctx, 5)
|
||||
if err != nil {
|
||||
t.Fatalf("list after manual retry: %v", err)
|
||||
}
|
||||
for _, item := range records {
|
||||
if item.ID != record.ID {
|
||||
continue
|
||||
}
|
||||
if item.AttemptCount != 0 {
|
||||
t.Fatalf("expected the manual retry to reset the attempt counter, got %d", item.AttemptCount)
|
||||
}
|
||||
if item.Status == pushStatusSucceeded {
|
||||
t.Fatal("a failing target must not report success")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateTileJobValidation(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
root := t.TempDir()
|
||||
tileRoot := filepath.Join(root, "tiles")
|
||||
source := filepath.Join(root, "input.tif")
|
||||
writeTestFile(t, source, "fake raster")
|
||||
|
||||
store := openTestStore(t, StoreConfig{
|
||||
SQLitePath: filepath.Join(root, "basemap.db"),
|
||||
ScanRoot: tileRoot,
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
TileBaseURL: "http://127.0.0.1:8910",
|
||||
VectorBaseURL: "http://127.0.0.1:8910",
|
||||
})
|
||||
|
||||
if _, err := store.CreateTileJob(ctx, CreateTileJobInput{TargetCode: "x", SourcePath: ""}); err == nil {
|
||||
t.Fatal("expected a missing source path to be rejected")
|
||||
}
|
||||
if _, err := store.CreateTileJob(ctx, CreateTileJobInput{SourcePath: source}); err == nil {
|
||||
t.Fatal("expected a missing target code to be rejected")
|
||||
}
|
||||
if _, err := store.CreateTileJob(ctx, CreateTileJobInput{SourcePath: source, TargetCode: "x", ZoomMin: 5, ZoomMax: 2}); err == nil {
|
||||
t.Fatal("expected an inverted zoom range to be rejected")
|
||||
}
|
||||
if _, err := store.CreateTileJob(ctx, CreateTileJobInput{SourcePath: filepath.Join(root, "missing.tif"), TargetCode: "x"}); err == nil {
|
||||
t.Fatal("expected a missing source file to be rejected")
|
||||
}
|
||||
|
||||
job, err := store.CreateTileJob(ctx, CreateTileJobInput{SourcePath: source, TargetCode: "demo", ZoomMin: 0, ZoomMax: 3})
|
||||
if err != nil {
|
||||
t.Fatalf("create job: %v", err)
|
||||
}
|
||||
if job.Status != tileJobPending {
|
||||
t.Fatalf("expected a pending job, got %q", job.Status)
|
||||
}
|
||||
if job.TileSize != 256 || job.TileFormat != "webp" {
|
||||
t.Fatalf("expected defaults to be applied, got %d/%s", job.TileSize, job.TileFormat)
|
||||
}
|
||||
if !strings.HasSuffix(job.OutputRoot, filepath.Join("tiles", "demo")) {
|
||||
t.Fatalf("unexpected output root %q", job.OutputRoot)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTileJobCommandArgsSubstitution(t *testing.T) {
|
||||
job := TileJob{
|
||||
SourcePath: "/data/in.tif",
|
||||
OutputRoot: "/data/out",
|
||||
ZoomMin: 1,
|
||||
ZoomMax: 5,
|
||||
TileSize: 512,
|
||||
TileFormat: "png",
|
||||
TargetCode: "demo",
|
||||
}
|
||||
args := tileJobCommandArgs(`gdal2tiles.py --xyz --zoom={zmin}-{zmax} --tilesize={tilesize} --format={format} {input} {output}`, job, job.SourcePath)
|
||||
want := []string{"gdal2tiles.py", "--xyz", "--zoom=1-5", "--tilesize=512", "--format=png", "/data/in.tif", "/data/out"}
|
||||
if len(args) != len(want) {
|
||||
t.Fatalf("expected %d args, got %v", len(want), args)
|
||||
}
|
||||
for i := range want {
|
||||
if args[i] != want[i] {
|
||||
t.Fatalf("arg %d = %q, want %q", i, args[i], want[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSplitCommandLineKeepsQuotedSegments(t *testing.T) {
|
||||
parts := splitCommandLine(`cmd /c mkdir "C:\Program Files\tiles\0"`)
|
||||
if len(parts) != 4 {
|
||||
t.Fatalf("expected 4 parts, got %v", parts)
|
||||
}
|
||||
if parts[3] != `C:\Program Files\tiles\0` {
|
||||
t.Fatalf("quoted segment not preserved: %q", parts[3])
|
||||
}
|
||||
}
|
||||
|
||||
func TestTileJobRunsCommandThenScans(t *testing.T) {
|
||||
if runtime.GOOS != "windows" {
|
||||
t.Skip("the stub slicer below uses cmd.exe")
|
||||
}
|
||||
ctx := context.Background()
|
||||
root := t.TempDir()
|
||||
tileRoot := filepath.Join(root, "tiles")
|
||||
source := filepath.Join(root, "input.tif")
|
||||
writeTestFile(t, source, "fake raster")
|
||||
if err := os.MkdirAll(tileRoot, 0o755); err != nil {
|
||||
t.Fatalf("mkdir tile root: %v", err)
|
||||
}
|
||||
|
||||
store := openTestStore(t, StoreConfig{
|
||||
SQLitePath: filepath.Join(root, "basemap.db"),
|
||||
ScanRoot: tileRoot,
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
TileBaseURL: "http://127.0.0.1:8910",
|
||||
VectorBaseURL: "http://127.0.0.1:8910",
|
||||
// Stand-in for gdal2tiles: lay down a z/x tree the scanner can read.
|
||||
TileJobCommand: `cmd /c mkdir "{output}\0\0"`,
|
||||
})
|
||||
if _, err := store.RunScan(ctx, defaultScanSourceCode); err != nil {
|
||||
t.Fatalf("initial scan: %v", err)
|
||||
}
|
||||
|
||||
job, err := store.CreateTileJob(ctx, CreateTileJobInput{
|
||||
SourcePath: source,
|
||||
TargetCode: "demo",
|
||||
ZoomMin: 0,
|
||||
ZoomMax: 0,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("create job: %v", err)
|
||||
}
|
||||
store.runTileJob(ctx, job)
|
||||
|
||||
tileJobs, err := store.ListTileJobs(ctx, 5)
|
||||
if err != nil {
|
||||
t.Fatalf("list jobs: %v", err)
|
||||
}
|
||||
if len(tileJobs) != 1 {
|
||||
t.Fatalf("expected 1 job, got %d", len(tileJobs))
|
||||
}
|
||||
if tileJobs[0].Status != tileJobSucceeded {
|
||||
t.Fatalf("expected success, got %q (%s)", tileJobs[0].Status, tileJobs[0].Message)
|
||||
}
|
||||
|
||||
basemaps, err := store.ListBasemaps(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("list basemaps: %v", err)
|
||||
}
|
||||
found := false
|
||||
for _, item := range basemaps {
|
||||
if item.Code == "demo" {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatalf("expected the sliced basemap to be scanned in, got %+v", basemaps)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTileJobFailureIsRecorded(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
root := t.TempDir()
|
||||
tileRoot := filepath.Join(root, "tiles")
|
||||
source := filepath.Join(root, "input.tif")
|
||||
writeTestFile(t, source, "fake raster")
|
||||
|
||||
store := openTestStore(t, StoreConfig{
|
||||
SQLitePath: filepath.Join(root, "basemap.db"),
|
||||
ScanRoot: tileRoot,
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
TileBaseURL: "http://127.0.0.1:8910",
|
||||
VectorBaseURL: "http://127.0.0.1:8910",
|
||||
TileJobCommand: "definitely-not-a-real-binary {input} {output}",
|
||||
})
|
||||
job, err := store.CreateTileJob(ctx, CreateTileJobInput{SourcePath: source, TargetCode: "demo"})
|
||||
if err != nil {
|
||||
t.Fatalf("create job: %v", err)
|
||||
}
|
||||
store.runTileJob(ctx, job)
|
||||
|
||||
jobs, err := store.ListTileJobs(ctx, 5)
|
||||
if err != nil {
|
||||
t.Fatalf("list jobs: %v", err)
|
||||
}
|
||||
if jobs[0].Status != tileJobFailed {
|
||||
t.Fatalf("expected failure, got %q", jobs[0].Status)
|
||||
}
|
||||
if jobs[0].Message == "" {
|
||||
t.Fatal("expected the failure reason to be recorded")
|
||||
}
|
||||
t.Logf("DEBUG job=%+v", jobs[0])
|
||||
if jobs[0].LogPath == "" {
|
||||
t.Fatal("expected a log path to be recorded")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetryPushRejectsUnknownKind(t *testing.T) {
|
||||
store := openTestStore(t, StoreConfig{SQLitePath: filepath.Join(t.TempDir(), "basemap.db")})
|
||||
if err := store.RetryPush(context.Background(), "nonsense", "x"); err == nil {
|
||||
t.Fatal("expected an unknown kind to be rejected")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetTargetSystemByIDMissing(t *testing.T) {
|
||||
store := openTestStore(t, StoreConfig{SQLitePath: filepath.Join(t.TempDir(), "basemap.db")})
|
||||
_, err := store.GetTargetSystemByID(context.Background(), "nope")
|
||||
if err == nil || errors.Is(err, os.ErrNotExist) {
|
||||
t.Fatalf("expected a not-found error, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBootstrapMigratesLegacySchema(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
dbPath := filepath.Join(root, "legacy.db")
|
||||
|
||||
// A database created before the health and retry columns existed.
|
||||
legacy, err := sql.Open("sqlite", "file:"+filepath.ToSlash(dbPath))
|
||||
if err != nil {
|
||||
t.Fatalf("open legacy db: %v", err)
|
||||
}
|
||||
legacyDDL := []string{
|
||||
`CREATE TABLE basemap_versions (
|
||||
id TEXT PRIMARY KEY,
|
||||
basemap_id TEXT NOT NULL,
|
||||
scan_source_id TEXT,
|
||||
version TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
is_default INTEGER NOT NULL DEFAULT 0,
|
||||
manifest_path TEXT NOT NULL DEFAULT '',
|
||||
tile_root_path TEXT NOT NULL,
|
||||
url_template TEXT NOT NULL,
|
||||
tile_format TEXT NOT NULL,
|
||||
tile_scheme TEXT NOT NULL,
|
||||
min_zoom INTEGER NOT NULL DEFAULT 0,
|
||||
max_zoom INTEGER NOT NULL DEFAULT 0,
|
||||
bbox_json TEXT NOT NULL DEFAULT '',
|
||||
attribution TEXT NOT NULL DEFAULT '',
|
||||
metadata_json TEXT NOT NULL DEFAULT '',
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
);`,
|
||||
`CREATE TABLE push_records (
|
||||
id TEXT PRIMARY KEY,
|
||||
target_system_id TEXT NOT NULL,
|
||||
basemap_version_id TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
request_json TEXT NOT NULL DEFAULT '',
|
||||
response_status INTEGER,
|
||||
response_body TEXT NOT NULL DEFAULT '',
|
||||
error_message TEXT NOT NULL DEFAULT '',
|
||||
pushed_at TEXT NOT NULL,
|
||||
finished_at TEXT
|
||||
);`,
|
||||
`INSERT INTO basemap_versions (id, basemap_id, version, status, tile_root_path, url_template, tile_format, tile_scheme, created_at, updated_at)
|
||||
VALUES ('v1', 'b1', 'current', 'ready', '/tiles/demo', '/tiles/demo/current/{z}/{x}/{y}.png', 'png', 'xyz', '2026-01-01T00:00:00Z', '2026-01-01T00:00:00Z');`,
|
||||
}
|
||||
for _, statement := range legacyDDL {
|
||||
if _, err := legacy.Exec(statement); err != nil {
|
||||
legacy.Close()
|
||||
t.Fatalf("seed legacy schema: %v", err)
|
||||
}
|
||||
}
|
||||
if err := legacy.Close(); err != nil {
|
||||
t.Fatalf("close legacy db: %v", err)
|
||||
}
|
||||
|
||||
store := openTestStore(t, StoreConfig{SQLitePath: dbPath, HealthCheckInterval: time.Minute})
|
||||
|
||||
// The pre-existing row must survive and gain default health values.
|
||||
var health, detail string
|
||||
if err := store.db.QueryRow(`SELECT health, health_detail FROM basemap_versions WHERE id = 'v1'`).Scan(&health, &detail); err != nil {
|
||||
t.Fatalf("legacy row lost or columns missing: %v", err)
|
||||
}
|
||||
if health != "" {
|
||||
t.Fatalf("expected a blank default health, got %q", health)
|
||||
}
|
||||
|
||||
var attempts int
|
||||
var nextAttempt sql.NullString
|
||||
if err := store.db.QueryRow(`SELECT attempt_count, next_attempt_at FROM push_records LIMIT 1`).Scan(&attempts, &nextAttempt); err != nil && !errors.Is(err, sql.ErrNoRows) {
|
||||
t.Fatalf("retry columns missing on push_records: %v", err)
|
||||
}
|
||||
|
||||
if err := store.ensureColumn(context.Background(), "basemap_versions", "health", "TEXT NOT NULL DEFAULT ''"); err != nil {
|
||||
t.Fatalf("ensureColumn must be idempotent: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTokenScopesNormalization(t *testing.T) {
|
||||
if got := normalizeTokenScopes(nil); got != "read" {
|
||||
t.Fatalf("expected read by default, got %q", got)
|
||||
}
|
||||
if got := normalizeTokenScopes([]string{"slice"}); got != "read,slice" {
|
||||
t.Fatalf("expected read,slice, got %q", got)
|
||||
}
|
||||
if got := normalizeTokenScopes([]string{"SLICE", "bogus", " read "}); got != "read,slice" {
|
||||
t.Fatalf("expected unknown scopes to be dropped, got %q", got)
|
||||
}
|
||||
|
||||
store := openTestStore(t, StoreConfig{SQLitePath: filepath.Join(t.TempDir(), "basemap.db")})
|
||||
token, err := store.CreateToken(context.Background(), CreateTokenInput{
|
||||
Name: "slicer",
|
||||
Scopes: []string{"slice"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("create token: %v", err)
|
||||
}
|
||||
auth, err := store.AuthorizeToken(context.Background(), token.Token)
|
||||
if err != nil {
|
||||
t.Fatalf("authorize: %v", err)
|
||||
}
|
||||
if !auth.CanSlice() {
|
||||
t.Fatal("expected the token to carry the slice scope after a round trip")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateTileJobIsIdempotentByCode(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
root := t.TempDir()
|
||||
tileRoot := filepath.Join(root, "tiles")
|
||||
source := filepath.Join(root, "input.tif")
|
||||
writeTestFile(t, source, "fake raster")
|
||||
|
||||
store := openTestStore(t, StoreConfig{
|
||||
SQLitePath: filepath.Join(root, "basemap.db"),
|
||||
ScanRoot: tileRoot,
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
TileBaseURL: "http://127.0.0.1:8910",
|
||||
VectorBaseURL: "http://127.0.0.1:8910",
|
||||
})
|
||||
|
||||
input := CreateTileJobInput{Code: "repeat-me", SourcePath: source, TargetCode: "demo"}
|
||||
first, err := store.CreateTileJob(ctx, input)
|
||||
if err != nil {
|
||||
t.Fatalf("first submit: %v", err)
|
||||
}
|
||||
second, err := store.CreateTileJob(ctx, input)
|
||||
if err != nil {
|
||||
t.Fatalf("second submit must be accepted: %v", err)
|
||||
}
|
||||
if first.ID != second.ID {
|
||||
t.Fatalf("expected the same job back, got %s then %s", first.ID, second.ID)
|
||||
}
|
||||
|
||||
jobs, err := store.ListTileJobs(ctx, 10)
|
||||
if err != nil {
|
||||
t.Fatalf("list: %v", err)
|
||||
}
|
||||
if len(jobs) != 1 {
|
||||
t.Fatalf("expected exactly one job, got %d", len(jobs))
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetTileJob(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
root := t.TempDir()
|
||||
source := filepath.Join(root, "input.tif")
|
||||
writeTestFile(t, source, "fake raster")
|
||||
|
||||
store := openTestStore(t, StoreConfig{
|
||||
SQLitePath: filepath.Join(root, "basemap.db"),
|
||||
ScanRoot: filepath.Join(root, "tiles"),
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
TileBaseURL: "http://127.0.0.1:8910",
|
||||
VectorBaseURL: "http://127.0.0.1:8910",
|
||||
})
|
||||
job, err := store.CreateTileJob(ctx, CreateTileJobInput{SourcePath: source, TargetCode: "demo"})
|
||||
if err != nil {
|
||||
t.Fatalf("create: %v", err)
|
||||
}
|
||||
found, err := store.GetTileJob(ctx, job.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("get: %v", err)
|
||||
}
|
||||
if found.Code != job.Code {
|
||||
t.Fatalf("unexpected job %+v", found)
|
||||
}
|
||||
if _, err := store.GetTileJob(ctx, "nope"); err == nil {
|
||||
t.Fatal("expected a missing job to error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTileJobCallbackFiresAndRetries(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
root := t.TempDir()
|
||||
tileRoot := filepath.Join(root, "tiles")
|
||||
source := filepath.Join(root, "input.tif")
|
||||
writeTestFile(t, source, "fake raster")
|
||||
|
||||
var attempts int
|
||||
received := make(chan map[string]any, 1)
|
||||
callback := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
attempts++
|
||||
if attempts < 2 {
|
||||
w.WriteHeader(http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
var payload map[string]any
|
||||
_ = json.NewDecoder(r.Body).Decode(&payload)
|
||||
select {
|
||||
case received <- payload:
|
||||
default:
|
||||
}
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}))
|
||||
defer callback.Close()
|
||||
|
||||
store := openTestStore(t, StoreConfig{
|
||||
SQLitePath: filepath.Join(root, "basemap.db"),
|
||||
ScanRoot: tileRoot,
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
TileBaseURL: "http://127.0.0.1:8910",
|
||||
VectorBaseURL: "http://127.0.0.1:8910",
|
||||
TileJobCommand: "definitely-not-a-real-binary {input} {output}",
|
||||
})
|
||||
if err := os.MkdirAll(tileRoot, 0o755); err != nil {
|
||||
t.Fatalf("mkdir: %v", err)
|
||||
}
|
||||
|
||||
job, err := store.CreateTileJob(ctx, CreateTileJobInput{
|
||||
SourcePath: source,
|
||||
TargetCode: "demo",
|
||||
CallbackURL: callback.URL,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("create: %v", err)
|
||||
}
|
||||
store.runTileJob(ctx, job)
|
||||
|
||||
// The first delivery fails, so the callback must be parked with a backoff.
|
||||
if err := store.dispatchTileJobCallbacks(ctx); err != nil {
|
||||
t.Fatalf("dispatch: %v", err)
|
||||
}
|
||||
jobs, err := store.ListTileJobs(ctx, 5)
|
||||
if err != nil {
|
||||
t.Fatalf("list: %v", err)
|
||||
}
|
||||
if jobs[0].CallbackStatus != pushStatusRetrying {
|
||||
t.Fatalf("expected the callback to be retrying, got %q", jobs[0].CallbackStatus)
|
||||
}
|
||||
if jobs[0].CallbackNextAttemptAt == nil {
|
||||
t.Fatal("expected a next attempt to be scheduled")
|
||||
}
|
||||
|
||||
if _, err := store.db.ExecContext(ctx, `UPDATE tile_jobs SET callback_next_attempt_at = ? WHERE id = ?`, toRFC3339(nowUTC().Add(-time.Minute)), job.ID); err != nil {
|
||||
t.Fatalf("force due: %v", err)
|
||||
}
|
||||
if err := store.dispatchTileJobCallbacks(ctx); err != nil {
|
||||
t.Fatalf("dispatch retry: %v", err)
|
||||
}
|
||||
|
||||
jobs, _ = store.ListTileJobs(ctx, 5)
|
||||
if jobs[0].CallbackStatus != pushStatusSucceeded {
|
||||
t.Fatalf("expected the callback to succeed, got %q (%s)", jobs[0].CallbackStatus, jobs[0].CallbackMessage)
|
||||
}
|
||||
select {
|
||||
case payload := <-received:
|
||||
if payload["event"] != "tile.job.finished" {
|
||||
t.Fatalf("unexpected callback payload: %v", payload)
|
||||
}
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("callback payload never arrived")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTileJobCallbackSkippedWhenNoURL(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
root := t.TempDir()
|
||||
tileRoot := filepath.Join(root, "tiles")
|
||||
source := filepath.Join(root, "input.tif")
|
||||
writeTestFile(t, source, "fake raster")
|
||||
if err := os.MkdirAll(tileRoot, 0o755); err != nil {
|
||||
t.Fatalf("mkdir: %v", err)
|
||||
}
|
||||
|
||||
store := openTestStore(t, StoreConfig{
|
||||
SQLitePath: filepath.Join(root, "basemap.db"),
|
||||
ScanRoot: tileRoot,
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
TileBaseURL: "http://127.0.0.1:8910",
|
||||
VectorBaseURL: "http://127.0.0.1:8910",
|
||||
TileJobCommand: "definitely-not-a-real-binary {input} {output}",
|
||||
})
|
||||
job, err := store.CreateTileJob(ctx, CreateTileJobInput{SourcePath: source, TargetCode: "demo"})
|
||||
if err != nil {
|
||||
t.Fatalf("create: %v", err)
|
||||
}
|
||||
store.runTileJob(ctx, job)
|
||||
|
||||
jobs, _ := store.ListTileJobs(ctx, 5)
|
||||
if jobs[0].CallbackStatus != "" {
|
||||
t.Fatalf("expected no callback scheduling, got %q", jobs[0].CallbackStatus)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTileJobTemplateExpandsInputList(t *testing.T) {
|
||||
job := TileJob{
|
||||
SourcePaths: []string{`C:\a\in1.tif`, `C:\b\in2.tif`},
|
||||
TargetCode: "demo",
|
||||
OutputRoot: `C:\out`,
|
||||
}
|
||||
got := tileJobVRTCommandArgs(`gdalbuildvrt -resolution highest {output_vrt} {inputs}`, job, `C:\work\m.vrt`)
|
||||
want := []string{"gdalbuildvrt", "-resolution", "highest", `C:\work\m.vrt`, `C:\a\in1.tif`, `C:\b\in2.tif`}
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("got %v, want %v", got, want)
|
||||
}
|
||||
for i := range want {
|
||||
if got[i] != want[i] {
|
||||
t.Fatalf("arg %d = %q, want %q", i, got[i], want[i])
|
||||
}
|
||||
}
|
||||
|
||||
// The slicer still gets one input path (the VRT) when mosaicking.
|
||||
slicer := tileJobCommandArgs(`gdal2tiles.py --exclude {input} {output}`, job, `C:\work\m.vrt`)
|
||||
if slicer[2] != `C:\work\m.vrt` {
|
||||
t.Fatalf("expected the VRT to be the slicer input, got %v", slicer)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTileJobMosaicsMultipleSources(t *testing.T) {
|
||||
if runtime.GOOS != "windows" {
|
||||
t.Skip("the stub commands below use cmd.exe")
|
||||
}
|
||||
ctx := context.Background()
|
||||
root := t.TempDir()
|
||||
tileRoot := filepath.Join(root, "tiles")
|
||||
if err := os.MkdirAll(tileRoot, 0o755); err != nil {
|
||||
t.Fatalf("mkdir: %v", err)
|
||||
}
|
||||
first := filepath.Join(root, "flight-a.tif")
|
||||
second := filepath.Join(root, "flight-b.tif")
|
||||
writeTestFile(t, first, "fake raster a")
|
||||
writeTestFile(t, second, "fake raster b")
|
||||
|
||||
store := openTestStore(t, StoreConfig{
|
||||
SQLitePath: filepath.Join(root, "basemap.db"),
|
||||
ScanRoot: tileRoot,
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
TileBaseURL: "http://127.0.0.1:8910",
|
||||
VectorBaseURL: "http://127.0.0.1:8910",
|
||||
// Stand-ins: the mosaic step writes the VRT, the slicer lays down a z tree.
|
||||
TileJobVRTCommand: `cmd /c echo vrt > {output_vrt}`,
|
||||
TileJobCommand: `cmd /c mkdir "{output}\0\0"`,
|
||||
})
|
||||
|
||||
job, err := store.CreateTileJob(ctx, CreateTileJobInput{
|
||||
SourcePaths: []string{first, second},
|
||||
TargetCode: "uav-2026",
|
||||
ZoomMin: 0,
|
||||
ZoomMax: 1,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("create job: %v", err)
|
||||
}
|
||||
if job.SourceCount != 2 || len(job.SourcePaths) != 2 {
|
||||
t.Fatalf("expected two sources on the job, got %d / %v", job.SourceCount, job.SourcePaths)
|
||||
}
|
||||
|
||||
store.runTileJob(ctx, job)
|
||||
|
||||
jobs, err := store.ListTileJobs(ctx, 5)
|
||||
if err != nil {
|
||||
t.Fatalf("list: %v", err)
|
||||
}
|
||||
if jobs[0].Status != tileJobSucceeded {
|
||||
t.Fatalf("expected success, got %q (%s)", jobs[0].Status, jobs[0].Message)
|
||||
}
|
||||
if jobs[0].SourceCount != 2 {
|
||||
t.Fatalf("expected source_count 2 after a round trip, got %d", jobs[0].SourceCount)
|
||||
}
|
||||
|
||||
// The mosaic step ran: its VRT sits beside the job log.
|
||||
vrt := filepath.Join(filepath.Dir(jobs[0].LogPath), job.Code+".vrt")
|
||||
if _, err := os.Stat(vrt); err != nil {
|
||||
t.Fatalf("expected the mosaic VRT at %s: %v", vrt, err)
|
||||
}
|
||||
|
||||
// Both flights are recorded against the layer.
|
||||
sources, err := store.ListLayerSources(ctx, "uav-2026")
|
||||
if err != nil {
|
||||
t.Fatalf("list layer sources: %v", err)
|
||||
}
|
||||
if len(sources) != 2 {
|
||||
t.Fatalf("expected 2 recorded sources, got %d", len(sources))
|
||||
}
|
||||
|
||||
// Adding the same image twice does not duplicate the registry entry.
|
||||
if err := store.AddLayerSources(ctx, "uav-2026", []string{first}); err != nil {
|
||||
t.Fatalf("re-add: %v", err)
|
||||
}
|
||||
sources, _ = store.ListLayerSources(ctx, "uav-2026")
|
||||
if len(sources) != 2 {
|
||||
t.Fatalf("expected re-adding to be a no-op, got %d entries", len(sources))
|
||||
}
|
||||
}
|
||||
|
||||
func TestTileJobSingleSourceSkipsMosaic(t *testing.T) {
|
||||
if runtime.GOOS != "windows" {
|
||||
t.Skip("the stub command below uses cmd.exe")
|
||||
}
|
||||
ctx := context.Background()
|
||||
root := t.TempDir()
|
||||
tileRoot := filepath.Join(root, "tiles")
|
||||
if err := os.MkdirAll(tileRoot, 0o755); err != nil {
|
||||
t.Fatalf("mkdir: %v", err)
|
||||
}
|
||||
source := filepath.Join(root, "flight-a.tif")
|
||||
writeTestFile(t, source, "fake raster")
|
||||
|
||||
store := openTestStore(t, StoreConfig{
|
||||
SQLitePath: filepath.Join(root, "basemap.db"),
|
||||
ScanRoot: tileRoot,
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
TileBaseURL: "http://127.0.0.1:8910",
|
||||
VectorBaseURL: "http://127.0.0.1:8910",
|
||||
// A failing mosaic command proves the single-source path never calls it.
|
||||
TileJobVRTCommand: `definitely-not-a-real-binary {output_vrt} {inputs}`,
|
||||
TileJobCommand: `cmd /c mkdir "{output}\0\0"`,
|
||||
})
|
||||
|
||||
job, err := store.CreateTileJob(ctx, CreateTileJobInput{SourcePath: source, TargetCode: "uav-2026"})
|
||||
if err != nil {
|
||||
t.Fatalf("create job: %v", err)
|
||||
}
|
||||
if job.SourceCount != 1 {
|
||||
t.Fatalf("expected source_count 1, got %d", job.SourceCount)
|
||||
}
|
||||
store.runTileJob(ctx, job)
|
||||
|
||||
jobs, _ := store.ListTileJobs(ctx, 5)
|
||||
if jobs[0].Status != tileJobSucceeded {
|
||||
t.Fatalf("expected the single source to slice directly, got %q (%s)", jobs[0].Status, jobs[0].Message)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,155 @@
|
||||
package basemap
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Availability verdicts recorded on catalog rows.
|
||||
const (
|
||||
healthOK = "ok"
|
||||
healthMissing = "missing"
|
||||
healthUnknown = "unknown"
|
||||
)
|
||||
|
||||
// HealthReport summarizes one inspection pass.
|
||||
type HealthReport struct {
|
||||
CheckedAt time.Time `json:"checked_at"`
|
||||
VersionsChecked int `json:"versions_checked"`
|
||||
VersionsMissing int `json:"versions_missing"`
|
||||
VectorsChecked int `json:"vectors_checked"`
|
||||
VectorsMissing int `json:"vectors_missing"`
|
||||
}
|
||||
|
||||
// CheckHealth stats every catalogued path and records whether it is still
|
||||
// there, so a directory that was moved or deleted surfaces before a client
|
||||
// hits a 404. It shares scanMu with the scanners because both rewrite catalog
|
||||
// rows.
|
||||
func (s *Store) CheckHealth(ctx context.Context) (HealthReport, error) {
|
||||
s.scanMu.Lock()
|
||||
defer s.scanMu.Unlock()
|
||||
|
||||
checkedAt := nowUTC()
|
||||
report := HealthReport{CheckedAt: checkedAt}
|
||||
|
||||
type verdict struct {
|
||||
id string
|
||||
health string
|
||||
detail string
|
||||
}
|
||||
|
||||
versions, err := s.collectHealth(ctx, `SELECT id, tile_root_path FROM basemap_versions`, true)
|
||||
if err != nil {
|
||||
return report, err
|
||||
}
|
||||
vectors, err := s.collectHealth(ctx, `SELECT id, file_path FROM vector_assets`, false)
|
||||
if err != nil {
|
||||
return report, err
|
||||
}
|
||||
|
||||
tx, err := s.db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return report, err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
|
||||
stamp := toRFC3339(checkedAt)
|
||||
for _, item := range versions {
|
||||
if _, err := tx.ExecContext(ctx, `
|
||||
UPDATE basemap_versions SET health = ?, health_checked_at = ?, health_detail = ? WHERE id = ?
|
||||
`, item.health, stamp, item.detail, item.id); err != nil {
|
||||
return report, err
|
||||
}
|
||||
report.VersionsChecked++
|
||||
if item.health != healthOK {
|
||||
report.VersionsMissing++
|
||||
}
|
||||
}
|
||||
for _, item := range vectors {
|
||||
if _, err := tx.ExecContext(ctx, `
|
||||
UPDATE vector_assets SET health = ?, health_checked_at = ?, health_detail = ? WHERE id = ?
|
||||
`, item.health, stamp, item.detail, item.id); err != nil {
|
||||
return report, err
|
||||
}
|
||||
report.VectorsChecked++
|
||||
if item.health != healthOK {
|
||||
report.VectorsMissing++
|
||||
}
|
||||
}
|
||||
if err := tx.Commit(); err != nil {
|
||||
return report, err
|
||||
}
|
||||
return report, nil
|
||||
}
|
||||
|
||||
type healthVerdict struct {
|
||||
id string
|
||||
health string
|
||||
detail string
|
||||
}
|
||||
|
||||
func (s *Store) collectHealth(ctx context.Context, query string, wantDir bool) ([]healthVerdict, error) {
|
||||
rows, err := s.db.QueryContext(ctx, query)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var verdicts []healthVerdict
|
||||
for rows.Next() {
|
||||
var id, path string
|
||||
if err := rows.Scan(&id, &path); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
health, detail := inspectPath(path, wantDir)
|
||||
verdicts = append(verdicts, healthVerdict{id: id, health: health, detail: detail})
|
||||
}
|
||||
return verdicts, rows.Err()
|
||||
}
|
||||
|
||||
func inspectPath(path string, wantDir bool) (string, string) {
|
||||
info, err := os.Stat(path)
|
||||
if err != nil {
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return healthMissing, "路径不存在"
|
||||
}
|
||||
return healthUnknown, err.Error()
|
||||
}
|
||||
if wantDir && !info.IsDir() {
|
||||
return healthMissing, "路径不是目录"
|
||||
}
|
||||
if !wantDir && info.IsDir() {
|
||||
return healthMissing, "路径是目录"
|
||||
}
|
||||
return healthOK, ""
|
||||
}
|
||||
|
||||
// The console reads health through these maps so the row scanners on the tile
|
||||
// request path stay untouched.
|
||||
func (s *Store) loadVersionHealth(ctx context.Context) (map[string]ResourceHealth, error) {
|
||||
return s.loadHealth(ctx, `SELECT id, health, health_detail, health_checked_at FROM basemap_versions`)
|
||||
}
|
||||
|
||||
func (s *Store) loadVectorHealth(ctx context.Context) (map[string]ResourceHealth, error) {
|
||||
return s.loadHealth(ctx, `SELECT id, health, health_detail, health_checked_at FROM vector_assets`)
|
||||
}
|
||||
|
||||
func (s *Store) loadHealth(ctx context.Context, query string) (map[string]ResourceHealth, error) {
|
||||
rows, err := s.db.QueryContext(ctx, query)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
result := map[string]ResourceHealth{}
|
||||
for rows.Next() {
|
||||
var id, health, detail, checkedAt string
|
||||
if err := rows.Scan(&id, &health, &detail, &checkedAt); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
result[id] = ResourceHealth{Health: health, Detail: detail, CheckedAt: parseTime(checkedAt)}
|
||||
}
|
||||
return result, rows.Err()
|
||||
}
|
||||
@@ -13,20 +13,21 @@ type SystemInfo struct {
|
||||
SQLitePath string `json:"sqlite_path"`
|
||||
ScanRoot string `json:"scan_root"`
|
||||
VectorScanRoot string `json:"vector_scan_root"`
|
||||
TileJobCommand string `json:"tile_job_command,omitempty"`
|
||||
StartedAt time.Time `json:"started_at"`
|
||||
}
|
||||
|
||||
type Basemap struct {
|
||||
ID string `json:"id"`
|
||||
Code string `json:"code"`
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Status string `json:"status"`
|
||||
Description string `json:"description,omitempty"`
|
||||
Default *BasemapVersion `json:"default_version,omitempty"`
|
||||
Versions []BasemapVersion `json:"versions,omitempty"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
ID string `json:"id"`
|
||||
Code string `json:"code"`
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Status string `json:"status"`
|
||||
Description string `json:"description,omitempty"`
|
||||
Default *BasemapVersion `json:"default_version,omitempty"`
|
||||
Versions []BasemapVersion `json:"versions,omitempty"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
}
|
||||
|
||||
type BasemapVersion struct {
|
||||
@@ -89,26 +90,27 @@ type TargetSystem struct {
|
||||
}
|
||||
|
||||
type ServiceToken struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
TokenPrefix string `json:"token_prefix"`
|
||||
Status string `json:"status"`
|
||||
ExpiresAt *time.Time `json:"expires_at,omitempty"`
|
||||
LastUsedAt *time.Time `json:"last_used_at,omitempty"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
Grants []TokenGrant `json:"grants,omitempty"`
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
TokenPrefix string `json:"token_prefix"`
|
||||
Scopes []string `json:"scopes"`
|
||||
Status string `json:"status"`
|
||||
ExpiresAt *time.Time `json:"expires_at,omitempty"`
|
||||
LastUsedAt *time.Time `json:"last_used_at,omitempty"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
Grants []TokenGrant `json:"grants,omitempty"`
|
||||
VectorGrants []VectorTokenGrant `json:"vector_grants,omitempty"`
|
||||
}
|
||||
|
||||
type TokenGrant struct {
|
||||
ID string `json:"id"`
|
||||
TokenID string `json:"token_id"`
|
||||
BasemapID string `json:"basemap_id"`
|
||||
BasemapCode string `json:"basemap_code"`
|
||||
BasemapVersionID *string `json:"basemap_version_id,omitempty"`
|
||||
BasemapVersion *string `json:"basemap_version,omitempty"`
|
||||
Permission string `json:"permission"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
ID string `json:"id"`
|
||||
TokenID string `json:"token_id"`
|
||||
BasemapID string `json:"basemap_id"`
|
||||
BasemapCode string `json:"basemap_code"`
|
||||
BasemapVersionID *string `json:"basemap_version_id,omitempty"`
|
||||
BasemapVersion *string `json:"basemap_version,omitempty"`
|
||||
Permission string `json:"permission"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
}
|
||||
|
||||
type PushRecord struct {
|
||||
@@ -123,6 +125,8 @@ type PushRecord struct {
|
||||
ResponseStatus *int `json:"response_status,omitempty"`
|
||||
ResponseBody string `json:"response_body,omitempty"`
|
||||
ErrorMessage string `json:"error_message,omitempty"`
|
||||
AttemptCount int `json:"attempt_count"`
|
||||
NextAttemptAt *time.Time `json:"next_attempt_at,omitempty"`
|
||||
PushedAt time.Time `json:"pushed_at"`
|
||||
FinishedAt *time.Time `json:"finished_at,omitempty"`
|
||||
}
|
||||
@@ -164,6 +168,8 @@ type VectorPushRecord struct {
|
||||
ResponseStatus *int `json:"response_status,omitempty"`
|
||||
ResponseBody string `json:"response_body,omitempty"`
|
||||
ErrorMessage string `json:"error_message,omitempty"`
|
||||
AttemptCount int `json:"attempt_count"`
|
||||
NextAttemptAt *time.Time `json:"next_attempt_at,omitempty"`
|
||||
PushedAt time.Time `json:"pushed_at"`
|
||||
FinishedAt *time.Time `json:"finished_at,omitempty"`
|
||||
}
|
||||
@@ -205,14 +211,17 @@ type CreateTokenInput struct {
|
||||
Name string
|
||||
BasemapCodes []string
|
||||
VectorCodes []string
|
||||
Scopes []string
|
||||
ExpiresAt *time.Time
|
||||
}
|
||||
|
||||
type TokenAuth struct {
|
||||
Token ServiceToken
|
||||
// permission -> basemap_code -> version scope
|
||||
Permissions map[string]map[string]VersionScope
|
||||
Permissions map[string]map[string]VersionScope
|
||||
VectorPermissions map[string]map[string]struct{}
|
||||
// Scopes are coarse capabilities beyond resource grants, e.g. "slice".
|
||||
Scopes map[string]struct{}
|
||||
}
|
||||
|
||||
type VersionScope struct {
|
||||
@@ -220,16 +229,41 @@ type VersionScope struct {
|
||||
Versions map[string]struct{}
|
||||
}
|
||||
|
||||
// ResourceHealth is the availability verdict recorded by the health inspector.
|
||||
// It is kept separate from Status: Status is a catalog fact, Health is whether
|
||||
// the bytes are still on disk right now.
|
||||
type ResourceHealth struct {
|
||||
Health string `json:"health"`
|
||||
Detail string `json:"detail,omitempty"`
|
||||
CheckedAt time.Time `json:"checked_at"`
|
||||
}
|
||||
|
||||
// LayerInventory is one layer and the imagery it is built from, ordered for
|
||||
// deterministic rendering in the console.
|
||||
type LayerInventory struct {
|
||||
TargetCode string `json:"target_code"`
|
||||
Sources []LayerSource `json:"sources"`
|
||||
// ZoomMin/ZoomMax/TileFormat mirror the layer's current basemap version so
|
||||
// a rebuild can reuse the range it is already published at.
|
||||
ZoomMin int `json:"zoom_min"`
|
||||
ZoomMax int `json:"zoom_max"`
|
||||
TileFormat string `json:"tile_format"`
|
||||
}
|
||||
|
||||
type DashboardData struct {
|
||||
SystemInfo SystemInfo
|
||||
Basemaps []Basemap
|
||||
VectorAssets []VectorAsset
|
||||
ScanSources []ScanSource
|
||||
ScanRuns []ScanRun
|
||||
TargetSystems []TargetSystem
|
||||
Tokens []ServiceToken
|
||||
PushRecords []PushRecord
|
||||
SystemInfo SystemInfo
|
||||
Basemaps []Basemap
|
||||
VectorAssets []VectorAsset
|
||||
ScanSources []ScanSource
|
||||
ScanRuns []ScanRun
|
||||
TargetSystems []TargetSystem
|
||||
Tokens []ServiceToken
|
||||
PushRecords []PushRecord
|
||||
VectorPushRecords []VectorPushRecord
|
||||
Flash string
|
||||
CreatedToken string
|
||||
Flash string
|
||||
CreatedToken string
|
||||
VersionHealth map[string]ResourceHealth
|
||||
VectorHealth map[string]ResourceHealth
|
||||
TileJobs []TileJob
|
||||
Layers []LayerInventory
|
||||
}
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
package basemap
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func writeTestFile(t *testing.T, path, body string) {
|
||||
t.Helper()
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
t.Fatalf("mkdir %s: %v", filepath.Dir(path), err)
|
||||
}
|
||||
if err := os.WriteFile(path, []byte(body), 0o644); err != nil {
|
||||
t.Fatalf("write %s: %v", path, err)
|
||||
}
|
||||
}
|
||||
|
||||
func manifestBody(code, name string) string {
|
||||
return `{"code":"` + code + `","name":"` + name + `","type":"xyz","version":"current","tile_format":"png","root_path":"tiles","is_default":true}`
|
||||
}
|
||||
|
||||
// The manifest lookup must not descend into the tile payload: a manifest nested
|
||||
// inside another basemap's "tiles" subtree must be ignored.
|
||||
func TestFindManifestVersionsPrunesTilePayload(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
writeTestFile(t, filepath.Join(root, "demo", "manifest.json"), manifestBody("demo", "Demo"))
|
||||
writeTestFile(t, filepath.Join(root, "demo", "tiles", "0", "0", "0.png"), "tile")
|
||||
writeTestFile(t, filepath.Join(root, "demo", "tiles", "nested", "manifest.json"), manifestBody("nested", "Nested"))
|
||||
|
||||
store := &Store{}
|
||||
items, covered, err := store.findManifestVersions(ScanSource{RootPath: root, ManifestName: "manifest.json"})
|
||||
if err != nil {
|
||||
t.Fatalf("find manifests: %v", err)
|
||||
}
|
||||
if len(items) != 1 {
|
||||
t.Fatalf("expected 1 manifest, got %d: %+v", len(items), items)
|
||||
}
|
||||
if items[0].BasemapCode != "demo" {
|
||||
t.Fatalf("expected code demo, got %q", items[0].BasemapCode)
|
||||
}
|
||||
if _, ok := covered["demo"]; !ok {
|
||||
t.Fatalf("expected demo to be marked as covered, got %v", covered)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsNumericDirName(t *testing.T) {
|
||||
for _, name := range []string{"0", "14", "007"} {
|
||||
if !isNumericDirName(name) {
|
||||
t.Fatalf("expected %q to be numeric", name)
|
||||
}
|
||||
}
|
||||
for _, name := range []string{"", "gaode-image", "2026-04", "tiles", "1a"} {
|
||||
if isNumericDirName(name) {
|
||||
t.Fatalf("expected %q to be non-numeric", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSQLiteDSNEnablesWALAndForeignKeys(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
store, err := OpenStore(StoreConfig{SQLitePath: filepath.Join(root, "basemap.db")})
|
||||
if err != nil {
|
||||
t.Fatalf("open store: %v", err)
|
||||
}
|
||||
defer store.Close()
|
||||
|
||||
var mode string
|
||||
if err := store.db.QueryRow("PRAGMA journal_mode;").Scan(&mode); err != nil {
|
||||
t.Fatalf("read journal_mode: %v", err)
|
||||
}
|
||||
if !strings.EqualFold(mode, "wal") {
|
||||
t.Fatalf("expected WAL journal mode, got %q", mode)
|
||||
}
|
||||
|
||||
var foreignKeys int
|
||||
if err := store.db.QueryRow("PRAGMA foreign_keys;").Scan(&foreignKeys); err != nil {
|
||||
t.Fatalf("read foreign_keys: %v", err)
|
||||
}
|
||||
if foreignKeys != 1 {
|
||||
t.Fatalf("expected foreign_keys=1, got %d", foreignKeys)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAuthorizeTokenThrottlesLastUsedWrite(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
root := t.TempDir()
|
||||
tileRoot := filepath.Join(root, "tiles")
|
||||
writeTestFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile")
|
||||
|
||||
store, err := OpenStore(StoreConfig{
|
||||
SQLitePath: filepath.Join(root, "basemap.db"),
|
||||
ScanRoot: tileRoot,
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
TileBaseURL: "http://127.0.0.1:8910",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("open store: %v", err)
|
||||
}
|
||||
defer store.Close()
|
||||
|
||||
if _, err := store.RunScan(ctx, defaultScanSourceCode); err != nil {
|
||||
t.Fatalf("run scan: %v", err)
|
||||
}
|
||||
created, err := store.CreateToken(ctx, CreateTokenInput{Name: "throttle", BasemapCodes: []string{"google_image"}})
|
||||
if err != nil {
|
||||
t.Fatalf("create token: %v", err)
|
||||
}
|
||||
|
||||
// Mark the token as unused long ago so the next authorization refreshes it.
|
||||
stale := toRFC3339(nowUTC().Add(-10 * time.Minute))
|
||||
if _, err := store.db.ExecContext(ctx, `UPDATE service_tokens SET last_used_at = ? WHERE id = ?`, stale, created.Meta.ID); err != nil {
|
||||
t.Fatalf("seed stale last_used_at: %v", err)
|
||||
}
|
||||
if _, err := store.AuthorizeToken(ctx, created.Token); err != nil {
|
||||
t.Fatalf("authorize token: %v", err)
|
||||
}
|
||||
first := readLastUsedAt(t, store, created.Meta.ID)
|
||||
if first == stale {
|
||||
t.Fatalf("expected last_used_at to be refreshed, still %q", first)
|
||||
}
|
||||
|
||||
if _, err := store.AuthorizeToken(ctx, created.Token); err != nil {
|
||||
t.Fatalf("authorize token again: %v", err)
|
||||
}
|
||||
second := readLastUsedAt(t, store, created.Meta.ID)
|
||||
if second != first {
|
||||
t.Fatalf("expected last_used_at to be throttled, got %q then %q", first, second)
|
||||
}
|
||||
}
|
||||
|
||||
func readLastUsedAt(t *testing.T, store *Store, tokenID string) string {
|
||||
t.Helper()
|
||||
var value string
|
||||
if err := store.db.QueryRow(`SELECT last_used_at FROM service_tokens WHERE id = ?`, tokenID).Scan(&value); err != nil {
|
||||
t.Fatalf("read last_used_at: %v", err)
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
func TestGetBasemapByCodeLoadsVersions(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
root := t.TempDir()
|
||||
tileRoot := filepath.Join(root, "tiles")
|
||||
writeTestFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile")
|
||||
writeTestFile(t, filepath.Join(tileRoot, "google_image", "1", "0", "0.webp"), "tile")
|
||||
|
||||
store, err := OpenStore(StoreConfig{
|
||||
SQLitePath: filepath.Join(root, "basemap.db"),
|
||||
ScanRoot: tileRoot,
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
TileBaseURL: "http://127.0.0.1:8910",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("open store: %v", err)
|
||||
}
|
||||
defer store.Close()
|
||||
|
||||
if _, err := store.RunScan(ctx, defaultScanSourceCode); err != nil {
|
||||
t.Fatalf("run scan: %v", err)
|
||||
}
|
||||
|
||||
item, err := store.GetBasemapByCode(ctx, "google_image")
|
||||
if err != nil {
|
||||
t.Fatalf("get basemap: %v", err)
|
||||
}
|
||||
if item.Code != "google-image" {
|
||||
t.Fatalf("unexpected code: %q", item.Code)
|
||||
}
|
||||
if len(item.Versions) != 1 {
|
||||
t.Fatalf("expected 1 version, got %d", len(item.Versions))
|
||||
}
|
||||
if item.Default == nil || item.Default.Version != "current" {
|
||||
t.Fatalf("expected default version current, got %+v", item.Default)
|
||||
}
|
||||
if item.Default.MinZoom != 0 || item.Default.MaxZoom != 1 {
|
||||
t.Fatalf("unexpected zoom range: %d-%d", item.Default.MinZoom, item.Default.MaxZoom)
|
||||
}
|
||||
if _, err := store.GetBasemapByCode(ctx, "missing"); err == nil {
|
||||
t.Fatal("expected missing basemap to error")
|
||||
}
|
||||
}
|
||||
@@ -7,10 +7,28 @@ import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"map-asset-gateway/api-go/internal/uid"
|
||||
"tile-server/api-go/internal/uid"
|
||||
)
|
||||
|
||||
const pushCallbackTimeout = 30 * time.Second
|
||||
|
||||
// Delivery outcomes. retrying is a transient state: the record stays queued
|
||||
// until next_attempt_at elapses.
|
||||
const (
|
||||
pushStatusSucceeded = "succeeded"
|
||||
pushStatusFailed = "failed"
|
||||
pushStatusRetrying = "retrying"
|
||||
|
||||
// pushMaxAttempts caps how many times one record is delivered.
|
||||
pushMaxAttempts = 3
|
||||
// pushRetryBase is the first backoff step; it doubles per attempt.
|
||||
pushRetryBase = 30 * time.Second
|
||||
)
|
||||
|
||||
func (s *Store) CreateTargetSystem(ctx context.Context, input CreateTargetSystemInput) (TargetSystem, error) {
|
||||
@@ -92,6 +110,9 @@ func (s *Store) ListTargetSystems(ctx context.Context) ([]TargetSystem, error) {
|
||||
return items, rows.Err()
|
||||
}
|
||||
|
||||
// PushBasemapVersion queues a downstream notification. The HTTP callback runs
|
||||
// off the request path so a slow target cannot block the caller; the returned
|
||||
// record starts as "pending" and is updated once the callback finishes.
|
||||
func (s *Store) PushBasemapVersion(ctx context.Context, targetCode, basemapCode, version string) (PushRecord, error) {
|
||||
target, err := s.GetTargetSystemByCode(ctx, targetCode)
|
||||
if err != nil {
|
||||
@@ -148,20 +169,6 @@ func (s *Store) PushBasemapVersion(ctx context.Context, targetCode, basemapCode,
|
||||
"pushed_at": toRFC3339(nowUTC()),
|
||||
"targetCode": target.Code,
|
||||
}
|
||||
body, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return PushRecord{}, err
|
||||
}
|
||||
request, err := http.NewRequestWithContext(ctx, target.CallbackMethod, target.CallbackURL, bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return PushRecord{}, err
|
||||
}
|
||||
request.Header.Set("Content-Type", "application/json")
|
||||
for key, value := range target.CallbackHeaders {
|
||||
request.Header.Set(key, value)
|
||||
}
|
||||
|
||||
response, err := s.httpClient.Do(request)
|
||||
record := PushRecord{
|
||||
ID: uid.New(),
|
||||
TargetSystemID: target.ID,
|
||||
@@ -169,41 +176,19 @@ func (s *Store) PushBasemapVersion(ctx context.Context, targetCode, basemapCode,
|
||||
BasemapVersionID: versionItem.ID,
|
||||
BasemapCode: basemapItem.Code,
|
||||
BasemapVersion: versionItem.Version,
|
||||
Status: "pending",
|
||||
Request: payload,
|
||||
PushedAt: nowUTC(),
|
||||
}
|
||||
if err != nil {
|
||||
record.Status = "failed"
|
||||
record.ErrorMessage = err.Error()
|
||||
} else {
|
||||
defer response.Body.Close()
|
||||
record.ResponseBody = limitedReadAll(response.Body, 256*1024)
|
||||
status := response.StatusCode
|
||||
record.ResponseStatus = &status
|
||||
if status >= 200 && status < 300 {
|
||||
record.Status = "succeeded"
|
||||
} else {
|
||||
record.Status = "failed"
|
||||
record.ErrorMessage = fmt.Sprintf("target returned status %d", status)
|
||||
}
|
||||
}
|
||||
finishedAt := nowUTC()
|
||||
record.FinishedAt = &finishedAt
|
||||
|
||||
_, insertErr := s.db.ExecContext(ctx, `
|
||||
if _, err := s.db.ExecContext(ctx, `
|
||||
INSERT INTO push_records (
|
||||
id, target_system_id, basemap_version_id, status, request_json, response_status, response_body, error_message, pushed_at, finished_at
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
`, record.ID, record.TargetSystemID, record.BasemapVersionID, record.Status, writeJSON(record.Request), record.ResponseStatus, record.ResponseBody, record.ErrorMessage, toRFC3339(record.PushedAt), nullableTime(record.FinishedAt))
|
||||
if insertErr != nil {
|
||||
return PushRecord{}, insertErr
|
||||
}
|
||||
if err != nil {
|
||||
return record, err
|
||||
}
|
||||
if record.Status != "succeeded" {
|
||||
return record, errors.New(record.ErrorMessage)
|
||||
) VALUES (?, ?, ?, ?, ?, NULL, '', '', ?, NULL)
|
||||
`, record.ID, record.TargetSystemID, record.BasemapVersionID, record.Status, writeJSON(record.Request), toRFC3339(record.PushedAt)); err != nil {
|
||||
return PushRecord{}, err
|
||||
}
|
||||
|
||||
s.enqueuePush("push_records", record.ID, target, payload)
|
||||
return record, nil
|
||||
}
|
||||
|
||||
@@ -223,6 +208,8 @@ func (s *Store) ListVectorPushRecords(ctx context.Context, limit int) ([]VectorP
|
||||
p.response_status,
|
||||
p.response_body,
|
||||
p.error_message,
|
||||
p.attempt_count,
|
||||
p.next_attempt_at,
|
||||
p.pushed_at,
|
||||
p.finished_at
|
||||
FROM vector_push_records p
|
||||
@@ -247,6 +234,8 @@ func (s *Store) ListVectorPushRecords(ctx context.Context, limit int) ([]VectorP
|
||||
return items, rows.Err()
|
||||
}
|
||||
|
||||
// PushVectorAsset queues a downstream notification, mirroring
|
||||
// PushBasemapVersion.
|
||||
func (s *Store) PushVectorAsset(ctx context.Context, targetCode, vectorCode string) (VectorPushRecord, error) {
|
||||
target, err := s.GetTargetSystemByCode(ctx, targetCode)
|
||||
if err != nil {
|
||||
@@ -268,60 +257,240 @@ func (s *Store) PushVectorAsset(ctx context.Context, targetCode, vectorCode stri
|
||||
"pushed_at": toRFC3339(nowUTC()),
|
||||
"targetCode": target.Code,
|
||||
}
|
||||
body, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return VectorPushRecord{}, err
|
||||
}
|
||||
request, err := http.NewRequestWithContext(ctx, target.CallbackMethod, target.CallbackURL, bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return VectorPushRecord{}, err
|
||||
}
|
||||
request.Header.Set("Content-Type", "application/json")
|
||||
for key, value := range target.CallbackHeaders {
|
||||
request.Header.Set(key, value)
|
||||
}
|
||||
|
||||
response, err := s.httpClient.Do(request)
|
||||
record := VectorPushRecord{
|
||||
ID: uid.New(),
|
||||
TargetSystemID: target.ID,
|
||||
TargetSystemCode: target.Code,
|
||||
VectorAssetID: asset.ID,
|
||||
VectorCode: asset.Code,
|
||||
Status: "pending",
|
||||
Request: payload,
|
||||
PushedAt: nowUTC(),
|
||||
}
|
||||
if err != nil {
|
||||
record.Status = "failed"
|
||||
record.ErrorMessage = err.Error()
|
||||
} else {
|
||||
defer response.Body.Close()
|
||||
record.ResponseBody = limitedReadAll(response.Body, 256*1024)
|
||||
status := response.StatusCode
|
||||
record.ResponseStatus = &status
|
||||
if status >= 200 && status < 300 {
|
||||
record.Status = "succeeded"
|
||||
} else {
|
||||
record.Status = "failed"
|
||||
record.ErrorMessage = fmt.Sprintf("target returned status %d", status)
|
||||
}
|
||||
}
|
||||
finishedAt := nowUTC()
|
||||
record.FinishedAt = &finishedAt
|
||||
|
||||
_, insertErr := s.db.ExecContext(ctx, `
|
||||
if _, err := s.db.ExecContext(ctx, `
|
||||
INSERT INTO vector_push_records (
|
||||
id, target_system_id, vector_asset_id, status, request_json, response_status, response_body, error_message, pushed_at, finished_at
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
`, record.ID, record.TargetSystemID, record.VectorAssetID, record.Status, writeJSON(record.Request), record.ResponseStatus, record.ResponseBody, record.ErrorMessage, toRFC3339(record.PushedAt), nullableTime(record.FinishedAt))
|
||||
if insertErr != nil {
|
||||
return VectorPushRecord{}, insertErr
|
||||
}
|
||||
if err != nil {
|
||||
return record, err
|
||||
}
|
||||
if record.Status != "succeeded" {
|
||||
return record, errors.New(record.ErrorMessage)
|
||||
) VALUES (?, ?, ?, ?, ?, NULL, '', '', ?, NULL)
|
||||
`, record.ID, record.TargetSystemID, record.VectorAssetID, record.Status, writeJSON(record.Request), toRFC3339(record.PushedAt)); err != nil {
|
||||
return VectorPushRecord{}, err
|
||||
}
|
||||
|
||||
s.enqueuePush("vector_push_records", record.ID, target, payload)
|
||||
return record, nil
|
||||
}
|
||||
|
||||
// enqueuePush performs the first delivery attempt off the request path. A
|
||||
// failure inside the retry budget is rescheduled by dispatchPushRetries.
|
||||
func (s *Store) enqueuePush(table, pushID string, target TargetSystem, payload map[string]any) {
|
||||
s.pushWG.Add(1)
|
||||
go func() {
|
||||
defer s.pushWG.Done()
|
||||
s.attemptPush(table, pushID, 1, target, payload)
|
||||
}()
|
||||
}
|
||||
|
||||
// attemptPush performs one delivery attempt and records the outcome. attempt is
|
||||
// the 1-based attempt number; table is a trusted internal literal.
|
||||
func (s *Store) attemptPush(table, pushID string, attempt int, target TargetSystem, payload map[string]any) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), pushCallbackTimeout)
|
||||
defer cancel()
|
||||
|
||||
status, body, err := s.deliverCallback(ctx, target, payload)
|
||||
errorMessage := ""
|
||||
if err != nil {
|
||||
errorMessage = err.Error()
|
||||
}
|
||||
var responseStatus any
|
||||
if status != 0 {
|
||||
responseStatus = status
|
||||
}
|
||||
|
||||
if err != nil && attempt < pushMaxAttempts {
|
||||
// Stay queued: the dispatcher picks it up once the backoff elapses.
|
||||
next := nowUTC().Add(pushRetryDelay(attempt))
|
||||
if _, updateErr := s.db.ExecContext(context.Background(), fmt.Sprintf(`
|
||||
UPDATE %s
|
||||
SET status = ?, attempt_count = ?, next_attempt_at = ?, response_status = ?, response_body = ?,
|
||||
error_message = ?, finished_at = NULL
|
||||
WHERE id = ?
|
||||
`, table), pushStatusRetrying, attempt, toRFC3339(next), responseStatus, body, errorMessage, pushID); updateErr != nil {
|
||||
log.Printf("reschedule push %s/%s: %v", table, pushID, updateErr)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
recordStatus := pushStatusSucceeded
|
||||
if err != nil {
|
||||
recordStatus = pushStatusFailed
|
||||
}
|
||||
if _, updateErr := s.db.ExecContext(context.Background(), fmt.Sprintf(`
|
||||
UPDATE %s
|
||||
SET status = ?, attempt_count = ?, next_attempt_at = NULL, response_status = ?, response_body = ?,
|
||||
error_message = ?, finished_at = ?
|
||||
WHERE id = ?
|
||||
`, table), recordStatus, attempt, responseStatus, body, errorMessage, toRFC3339(nowUTC()), pushID); updateErr != nil {
|
||||
log.Printf("record push %s/%s: %v", table, pushID, updateErr)
|
||||
}
|
||||
}
|
||||
|
||||
// pushRetryDelay backs off exponentially: 30s, 60s, ...
|
||||
func pushRetryDelay(attempt int) time.Duration {
|
||||
if attempt < 1 {
|
||||
attempt = 1
|
||||
}
|
||||
return pushRetryBase * time.Duration(1<<(attempt-1))
|
||||
}
|
||||
|
||||
// dispatchPushRetries finds attempts whose backoff has elapsed and runs them.
|
||||
func (s *Store) dispatchPushRetries(ctx context.Context) error {
|
||||
for _, table := range []string{"push_records", "vector_push_records"} {
|
||||
due, err := s.dueRetries(ctx, table, 5)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, item := range due {
|
||||
target, err := s.GetTargetSystemByID(ctx, item.targetID)
|
||||
if err != nil {
|
||||
s.failRetry(table, item.id, fmt.Sprintf("target system is gone: %v", err))
|
||||
continue
|
||||
}
|
||||
if !target.Enabled {
|
||||
s.failRetry(table, item.id, "target system is disabled")
|
||||
continue
|
||||
}
|
||||
s.attemptPush(table, item.id, item.attempt+1, target, item.payload)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type duePush struct {
|
||||
id string
|
||||
attempt int
|
||||
targetID string
|
||||
payload map[string]any
|
||||
}
|
||||
|
||||
func (s *Store) dueRetries(ctx context.Context, table string, limit int) ([]duePush, error) {
|
||||
now := toRFC3339(nowUTC())
|
||||
rows, err := s.db.QueryContext(ctx, fmt.Sprintf(`
|
||||
SELECT id, attempt_count, target_system_id, request_json
|
||||
FROM %s
|
||||
WHERE status = ? AND (next_attempt_at IS NULL OR next_attempt_at <= ?)
|
||||
ORDER BY next_attempt_at
|
||||
LIMIT ?
|
||||
`, table), pushStatusRetrying, now, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var items []duePush
|
||||
for rows.Next() {
|
||||
var item duePush
|
||||
var requestJSON string
|
||||
if err := rows.Scan(&item.id, &item.attempt, &item.targetID, &requestJSON); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if strings.TrimSpace(requestJSON) != "" {
|
||||
if err := readMaybeJSON(requestJSON, &item.payload); err != nil {
|
||||
item.payload = nil
|
||||
}
|
||||
}
|
||||
items = append(items, item)
|
||||
}
|
||||
return items, rows.Err()
|
||||
}
|
||||
|
||||
func (s *Store) failRetry(table, pushID, message string) {
|
||||
if _, err := s.db.ExecContext(context.Background(), fmt.Sprintf(`
|
||||
UPDATE %s SET status = ?, error_message = ?, next_attempt_at = NULL, finished_at = ? WHERE id = ?
|
||||
`, table), pushStatusFailed, message, toRFC3339(nowUTC()), pushID); err != nil {
|
||||
log.Printf("fail push %s/%s: %v", table, pushID, err)
|
||||
}
|
||||
}
|
||||
|
||||
// GetTargetSystemByID looks a target up by primary key for retry dispatch.
|
||||
func (s *Store) GetTargetSystemByID(ctx context.Context, id string) (TargetSystem, error) {
|
||||
row := s.db.QueryRowContext(ctx, `
|
||||
SELECT id, code, name, callback_url, callback_method, callback_headers_json, enabled, created_at, updated_at
|
||||
FROM target_systems
|
||||
WHERE id = ?
|
||||
`, strings.TrimSpace(id))
|
||||
item, err := scanTargetSystem(row)
|
||||
if err != nil {
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return TargetSystem{}, fmt.Errorf("target system %q not found", id)
|
||||
}
|
||||
return TargetSystem{}, err
|
||||
}
|
||||
return item, nil
|
||||
}
|
||||
|
||||
// RetryPush re-queues a finished push record. kind is "basemap" or "vector".
|
||||
func (s *Store) RetryPush(ctx context.Context, kind, recordID string) error {
|
||||
table := "push_records"
|
||||
if kind == "vector" {
|
||||
table = "vector_push_records"
|
||||
} else if kind != "basemap" {
|
||||
return fmt.Errorf("unknown push kind %q", kind)
|
||||
}
|
||||
result, err := s.db.ExecContext(ctx, fmt.Sprintf(`
|
||||
UPDATE %s
|
||||
SET status = ?, attempt_count = 0, next_attempt_at = ?, finished_at = NULL, error_message = ''
|
||||
WHERE id = ?
|
||||
`, table), pushStatusRetrying, toRFC3339(nowUTC()), strings.TrimSpace(recordID))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
affected, err := result.RowsAffected()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if affected == 0 {
|
||||
return fmt.Errorf("push record %q not found", recordID)
|
||||
}
|
||||
// Kick the dispatcher so a manual retry does not wait for the next tick.
|
||||
s.pushWG.Add(1)
|
||||
go func() {
|
||||
defer s.pushWG.Done()
|
||||
_ = s.dispatchPushRetries(context.Background())
|
||||
}()
|
||||
return nil
|
||||
}
|
||||
|
||||
// deliverCallback performs the downstream HTTP call for a target system.
|
||||
func (s *Store) deliverCallback(ctx context.Context, target TargetSystem, payload map[string]any) (int, string, error) {
|
||||
return s.postJSON(ctx, target.CallbackMethod, target.CallbackURL, target.CallbackHeaders, payload)
|
||||
}
|
||||
|
||||
// postJSON sends a JSON payload and returns the status code plus a truncated
|
||||
// response body. Shared by push callbacks and tile-job completion callbacks.
|
||||
func (s *Store) postJSON(ctx context.Context, method, url string, headers map[string]string, payload map[string]any) (int, string, error) {
|
||||
body, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return 0, "", err
|
||||
}
|
||||
request, err := http.NewRequestWithContext(ctx, method, url, bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return 0, "", err
|
||||
}
|
||||
request.Header.Set("Content-Type", "application/json")
|
||||
for key, value := range headers {
|
||||
request.Header.Set(key, value)
|
||||
}
|
||||
|
||||
response, err := s.httpClient.Do(request)
|
||||
if err != nil {
|
||||
return 0, "", err
|
||||
}
|
||||
defer response.Body.Close()
|
||||
|
||||
responseBody := limitedReadAll(response.Body, 256*1024)
|
||||
// Drain whatever is left so the connection can be reused.
|
||||
_, _ = io.Copy(io.Discard, response.Body)
|
||||
if response.StatusCode < 200 || response.StatusCode >= 300 {
|
||||
return response.StatusCode, responseBody, fmt.Errorf("target returned status %d", response.StatusCode)
|
||||
}
|
||||
return response.StatusCode, responseBody, nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,397 @@
|
||||
package basemap
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func openTestStore(t *testing.T, cfg StoreConfig) *Store {
|
||||
t.Helper()
|
||||
store, err := OpenStore(cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("open store: %v", err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
if err := store.Close(); err != nil {
|
||||
t.Fatalf("close store: %v", err)
|
||||
}
|
||||
})
|
||||
return store
|
||||
}
|
||||
|
||||
func TestRunScanRefusesConcurrentRuns(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
root := t.TempDir()
|
||||
tileRoot := filepath.Join(root, "tiles")
|
||||
writeTestFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile")
|
||||
vectorRoot := filepath.Join(root, "vectors")
|
||||
|
||||
store := openTestStore(t, StoreConfig{
|
||||
SQLitePath: filepath.Join(root, "basemap.db"),
|
||||
ScanRoot: tileRoot,
|
||||
VectorScanRoot: vectorRoot,
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
TileBaseURL: "http://127.0.0.1:8910",
|
||||
VectorBaseURL: "http://127.0.0.1:8910",
|
||||
})
|
||||
|
||||
store.scanMu.Lock()
|
||||
if _, err := store.RunScan(ctx, defaultScanSourceCode); !errors.Is(err, ErrScanInProgress) {
|
||||
store.scanMu.Unlock()
|
||||
t.Fatalf("expected ErrScanInProgress, got %v", err)
|
||||
}
|
||||
if _, err := store.RunVectorScan(ctx); !errors.Is(err, ErrScanInProgress) {
|
||||
store.scanMu.Unlock()
|
||||
t.Fatalf("expected ErrScanInProgress from vector scan, got %v", err)
|
||||
}
|
||||
store.scanMu.Unlock()
|
||||
|
||||
if _, err := store.RunScan(ctx, defaultScanSourceCode); err != nil {
|
||||
t.Fatalf("run scan after unlock: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunScanReportsUnknownSource(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
root := t.TempDir()
|
||||
store := openTestStore(t, StoreConfig{SQLitePath: filepath.Join(root, "basemap.db")})
|
||||
|
||||
if _, err := store.RunScan(ctx, "does-not-exist"); !errors.Is(err, ErrScanSourceNotFound) {
|
||||
t.Fatalf("expected ErrScanSourceNotFound, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRescanOnlyCountsRealChanges(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
root := t.TempDir()
|
||||
tileRoot := filepath.Join(root, "tiles")
|
||||
writeTestFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile")
|
||||
|
||||
store := openTestStore(t, StoreConfig{
|
||||
SQLitePath: filepath.Join(root, "basemap.db"),
|
||||
ScanRoot: tileRoot,
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
TileBaseURL: "http://127.0.0.1:8910",
|
||||
VectorBaseURL: "http://127.0.0.1:8910",
|
||||
})
|
||||
|
||||
first, err := store.RunScan(ctx, defaultScanSourceCode)
|
||||
if err != nil {
|
||||
t.Fatalf("first scan: %v", err)
|
||||
}
|
||||
if first.AddedCount != 1 {
|
||||
t.Fatalf("expected 1 added on first scan, got %d", first.AddedCount)
|
||||
}
|
||||
|
||||
second, err := store.RunScan(ctx, defaultScanSourceCode)
|
||||
if err != nil {
|
||||
t.Fatalf("second scan: %v", err)
|
||||
}
|
||||
if second.UpdatedCount != 0 {
|
||||
t.Fatalf("expected 0 updates on an unchanged rescan, got %d", second.UpdatedCount)
|
||||
}
|
||||
if second.AddedCount != 0 || second.RemovedCount != 0 {
|
||||
t.Fatalf("expected no additions/removals, got +%d/-%d", second.AddedCount, second.RemovedCount)
|
||||
}
|
||||
if got := second.Summary["unchanged_count"]; got != 1 {
|
||||
t.Fatalf("expected unchanged_count=1, got %v", got)
|
||||
}
|
||||
|
||||
// A new zoom level is a genuine change and must be reported as an update.
|
||||
writeTestFile(t, filepath.Join(tileRoot, "google_image", "1", "0", "0.webp"), "tile")
|
||||
third, err := store.RunScan(ctx, defaultScanSourceCode)
|
||||
if err != nil {
|
||||
t.Fatalf("third scan: %v", err)
|
||||
}
|
||||
if third.UpdatedCount != 1 {
|
||||
t.Fatalf("expected 1 update after adding a zoom level, got %d", third.UpdatedCount)
|
||||
}
|
||||
}
|
||||
|
||||
func TestVectorCodeCollisionKeepsBothAssets(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
root := t.TempDir()
|
||||
vectorRoot := filepath.Join(root, "vectors")
|
||||
writeTestFile(t, filepath.Join(vectorRoot, "a", "b.geojson"), `{"type":"FeatureCollection","features":[]}`)
|
||||
writeTestFile(t, filepath.Join(vectorRoot, "a-b.geojson"), `{"type":"FeatureCollection","features":[]}`)
|
||||
|
||||
store := openTestStore(t, StoreConfig{
|
||||
SQLitePath: filepath.Join(root, "basemap.db"),
|
||||
VectorScanRoot: vectorRoot,
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
VectorBaseURL: "http://127.0.0.1:8910",
|
||||
})
|
||||
|
||||
if _, err := store.RunVectorScan(ctx); err != nil {
|
||||
t.Fatalf("first vector scan: %v", err)
|
||||
}
|
||||
assets, err := store.ListVectorAssets(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("list assets: %v", err)
|
||||
}
|
||||
if len(assets) != 2 {
|
||||
t.Fatalf("expected both colliding assets to survive, got %d", len(assets))
|
||||
}
|
||||
if assets[0].Code == assets[1].Code {
|
||||
t.Fatalf("expected distinct codes, both are %q", assets[0].Code)
|
||||
}
|
||||
firstCodes := []string{assets[0].Code, assets[1].Code}
|
||||
|
||||
if _, err := store.RunVectorScan(ctx); err != nil {
|
||||
t.Fatalf("second vector scan: %v", err)
|
||||
}
|
||||
assets, err = store.ListVectorAssets(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("list assets after rescan: %v", err)
|
||||
}
|
||||
if len(assets) != 2 {
|
||||
t.Fatalf("rescan dropped an asset, now %d", len(assets))
|
||||
}
|
||||
secondCodes := map[string]bool{assets[0].Code: true, assets[1].Code: true}
|
||||
for _, code := range firstCodes {
|
||||
if !secondCodes[code] {
|
||||
t.Fatalf("disambiguated code %q was not stable across rescans", code)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestVectorScanReusesChecksumUntilFileChanges(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
root := t.TempDir()
|
||||
vectorRoot := filepath.Join(root, "vectors")
|
||||
file := filepath.Join(vectorRoot, "boundary.geojson")
|
||||
writeTestFile(t, file, `{"type":"FeatureCollection","features":[]}`)
|
||||
|
||||
store := openTestStore(t, StoreConfig{
|
||||
SQLitePath: filepath.Join(root, "basemap.db"),
|
||||
VectorScanRoot: vectorRoot,
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
VectorBaseURL: "http://127.0.0.1:8910",
|
||||
})
|
||||
|
||||
if _, err := store.RunVectorScan(ctx); err != nil {
|
||||
t.Fatalf("first scan: %v", err)
|
||||
}
|
||||
first, err := store.GetVectorAssetByCode(ctx, "boundary")
|
||||
if err != nil {
|
||||
t.Fatalf("get asset: %v", err)
|
||||
}
|
||||
if first.Metadata["hash_size"] == nil || first.Metadata["hash_mtime"] == nil {
|
||||
t.Fatalf("expected hash stamp in metadata, got %v", first.Metadata)
|
||||
}
|
||||
|
||||
second, err := store.RunVectorScan(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("second scan: %v", err)
|
||||
}
|
||||
if second["unchanged_count"] != 1 {
|
||||
t.Fatalf("expected unchanged_count=1, got %v", second["unchanged_count"])
|
||||
}
|
||||
reused, err := store.GetVectorAssetByCode(ctx, "boundary")
|
||||
if err != nil {
|
||||
t.Fatalf("get asset after rescan: %v", err)
|
||||
}
|
||||
if reused.Checksum != first.Checksum {
|
||||
t.Fatalf("checksum changed without a file change: %q -> %q", first.Checksum, reused.Checksum)
|
||||
}
|
||||
|
||||
// Rewriting the file with different content must invalidate the cached hash.
|
||||
writeTestFile(t, file, `{"type":"FeatureCollection","features":[{"type":"Feature"}]}`)
|
||||
if _, err := store.RunVectorScan(ctx); err != nil {
|
||||
t.Fatalf("third scan: %v", err)
|
||||
}
|
||||
changed, err := store.GetVectorAssetByCode(ctx, "boundary")
|
||||
if err != nil {
|
||||
t.Fatalf("get asset after change: %v", err)
|
||||
}
|
||||
if changed.Checksum == first.Checksum {
|
||||
t.Fatal("expected checksum to change after the file changed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateTokenAppliesDefaultTTL(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
root := t.TempDir()
|
||||
tileRoot := filepath.Join(root, "tiles")
|
||||
writeTestFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile")
|
||||
|
||||
store := openTestStore(t, StoreConfig{
|
||||
SQLitePath: filepath.Join(root, "basemap.db"),
|
||||
ScanRoot: tileRoot,
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
TileBaseURL: "http://127.0.0.1:8910",
|
||||
VectorBaseURL: "http://127.0.0.1:8910",
|
||||
TokenDefaultTTL: time.Hour,
|
||||
})
|
||||
if _, err := store.RunScan(ctx, defaultScanSourceCode); err != nil {
|
||||
t.Fatalf("run scan: %v", err)
|
||||
}
|
||||
|
||||
created, err := store.CreateToken(ctx, CreateTokenInput{Name: "ttl", BasemapCodes: []string{"google_image"}})
|
||||
if err != nil {
|
||||
t.Fatalf("create token: %v", err)
|
||||
}
|
||||
if created.Meta.ExpiresAt == nil {
|
||||
t.Fatal("expected a default expiry to be applied")
|
||||
}
|
||||
remaining := time.Until(*created.Meta.ExpiresAt)
|
||||
if remaining < 55*time.Minute || remaining > 65*time.Minute {
|
||||
t.Fatalf("expected ~1h expiry, got %s", remaining)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPushRunsAsynchronously(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
root := t.TempDir()
|
||||
tileRoot := filepath.Join(root, "tiles")
|
||||
writeTestFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile")
|
||||
|
||||
received := make(chan map[string]any, 1)
|
||||
callback := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
var payload map[string]any
|
||||
_ = json.NewDecoder(r.Body).Decode(&payload)
|
||||
select {
|
||||
case received <- payload:
|
||||
default:
|
||||
}
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}))
|
||||
defer callback.Close()
|
||||
|
||||
store := openTestStore(t, StoreConfig{
|
||||
SQLitePath: filepath.Join(root, "basemap.db"),
|
||||
ScanRoot: tileRoot,
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
TileBaseURL: "http://127.0.0.1:8910",
|
||||
VectorBaseURL: "http://127.0.0.1:8910",
|
||||
})
|
||||
if _, err := store.RunScan(ctx, defaultScanSourceCode); err != nil {
|
||||
t.Fatalf("run scan: %v", err)
|
||||
}
|
||||
if _, err := store.CreateTargetSystem(ctx, CreateTargetSystemInput{
|
||||
Code: "downstream",
|
||||
CallbackURL: callback.URL,
|
||||
}); err != nil {
|
||||
t.Fatalf("create target: %v", err)
|
||||
}
|
||||
|
||||
record, err := store.PushBasemapVersion(ctx, "downstream", "google_image", "current")
|
||||
if err != nil {
|
||||
t.Fatalf("push: %v", err)
|
||||
}
|
||||
if record.Status != "pending" {
|
||||
t.Fatalf("expected the push to be queued as pending, got %q", record.Status)
|
||||
}
|
||||
|
||||
payload := waitForPayload(t, received)
|
||||
if payload["event"] != "basemap.published" {
|
||||
t.Fatalf("unexpected callback payload: %v", payload)
|
||||
}
|
||||
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
records, err := store.ListPushRecords(ctx, 5)
|
||||
if err != nil {
|
||||
t.Fatalf("list push records: %v", err)
|
||||
}
|
||||
done := false
|
||||
for _, item := range records {
|
||||
if item.ID == record.ID && item.Status == "succeeded" {
|
||||
done = true
|
||||
}
|
||||
}
|
||||
if done {
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("push record %s never reached succeeded: %+v", record.ID, records)
|
||||
}
|
||||
time.Sleep(25 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
func waitForPayload(t *testing.T, received <-chan map[string]any) map[string]any {
|
||||
t.Helper()
|
||||
select {
|
||||
case payload := <-received:
|
||||
return payload
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("callback was never invoked")
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
func TestCloseWaitsForQueuedPush(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
root := t.TempDir()
|
||||
tileRoot := filepath.Join(root, "tiles")
|
||||
writeTestFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile")
|
||||
|
||||
started := make(chan struct{}, 1)
|
||||
release := make(chan struct{})
|
||||
callback := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
select {
|
||||
case started <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
<-release
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}))
|
||||
defer callback.Close()
|
||||
|
||||
store, err := OpenStore(StoreConfig{
|
||||
SQLitePath: filepath.Join(root, "basemap.db"),
|
||||
ScanRoot: tileRoot,
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
TileBaseURL: "http://127.0.0.1:8910",
|
||||
VectorBaseURL: "http://127.0.0.1:8910",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("open store: %v", err)
|
||||
}
|
||||
if _, err := store.RunScan(ctx, defaultScanSourceCode); err != nil {
|
||||
t.Fatalf("run scan: %v", err)
|
||||
}
|
||||
if _, err := store.CreateTargetSystem(ctx, CreateTargetSystemInput{Code: "downstream", CallbackURL: callback.URL}); err != nil {
|
||||
t.Fatalf("create target: %v", err)
|
||||
}
|
||||
if _, err := store.PushBasemapVersion(ctx, "downstream", "google_image", "current"); err != nil {
|
||||
t.Fatalf("push: %v", err)
|
||||
}
|
||||
|
||||
<-started
|
||||
closed := make(chan struct{})
|
||||
go func() {
|
||||
_ = store.Close()
|
||||
close(closed)
|
||||
}()
|
||||
select {
|
||||
case <-closed:
|
||||
t.Fatal("Close returned while a push callback was still in flight")
|
||||
case <-time.After(150 * time.Millisecond):
|
||||
}
|
||||
close(release)
|
||||
select {
|
||||
case <-closed:
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("Close did not return after the push finished")
|
||||
}
|
||||
}
|
||||
|
||||
func TestScanRootDirectoryIsNotRequiredAtOpen(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
store := openTestStore(t, StoreConfig{SQLitePath: filepath.Join(root, "basemap.db")})
|
||||
if store.ScanRoot() != "" {
|
||||
t.Fatalf("expected empty scan root, got %q", store.ScanRoot())
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(root, "basemap.db")); err != nil {
|
||||
t.Fatalf("expected the database file to exist: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -12,7 +12,7 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"map-asset-gateway/api-go/internal/uid"
|
||||
"tile-server/api-go/internal/uid"
|
||||
)
|
||||
|
||||
type scannedVersion struct {
|
||||
@@ -34,13 +34,30 @@ type scannedVersion struct {
|
||||
IsDefault bool
|
||||
}
|
||||
|
||||
// scanCounts reports what a scan actually changed, so operators can tell a real
|
||||
// update from a no-op rescan.
|
||||
type scanCounts struct {
|
||||
Scanned int
|
||||
Added int
|
||||
Updated int
|
||||
Removed int
|
||||
Unchanged int
|
||||
}
|
||||
|
||||
func (s *Store) RunScan(ctx context.Context, sourceCode string) (ScanRun, error) {
|
||||
// A scan reads the current catalog and then rewrites it, so two overlapping
|
||||
// runs could delete each other's rows. Refuse instead of interleaving.
|
||||
if !s.scanMu.TryLock() {
|
||||
return ScanRun{}, ErrScanInProgress
|
||||
}
|
||||
defer s.scanMu.Unlock()
|
||||
|
||||
source, err := s.GetScanSourceByCode(ctx, sourceCode)
|
||||
if err != nil {
|
||||
return ScanRun{}, err
|
||||
}
|
||||
if !source.Enabled {
|
||||
return ScanRun{}, fmt.Errorf("scan source %q is disabled", sourceCode)
|
||||
return ScanRun{}, &scanSourceError{sentinel: ErrScanSourceDisabled, code: sourceCode, reason: "is disabled"}
|
||||
}
|
||||
|
||||
startedAt := nowUTC()
|
||||
@@ -58,21 +75,25 @@ func (s *Store) RunScan(ctx context.Context, sourceCode string) (ScanRun, error)
|
||||
}
|
||||
|
||||
func (s *Store) runScan(ctx context.Context, runID string, source ScanSource, startedAt time.Time) (ScanRun, error) {
|
||||
fail := func(err error) (ScanRun, error) {
|
||||
return s.finishScanRun(ctx, runID, source, startedAt, "failed", nil, scanCounts{}, err)
|
||||
}
|
||||
|
||||
rootInfo, err := os.Stat(source.RootPath)
|
||||
if err != nil {
|
||||
return s.finishScanRun(ctx, runID, source, startedAt, "failed", nil, 0, 0, 0, 0, fmt.Errorf("stat scan root: %w", err))
|
||||
return fail(fmt.Errorf("stat scan root: %w", err))
|
||||
}
|
||||
if !rootInfo.IsDir() {
|
||||
return s.finishScanRun(ctx, runID, source, startedAt, "failed", nil, 0, 0, 0, 0, fmt.Errorf("scan root %s is not a directory", source.RootPath))
|
||||
return fail(fmt.Errorf("scan root %s is not a directory", source.RootPath))
|
||||
}
|
||||
|
||||
manifestItems, coveredRoots, err := s.findManifestVersions(source)
|
||||
if err != nil {
|
||||
return s.finishScanRun(ctx, runID, source, startedAt, "failed", nil, 0, 0, 0, 0, err)
|
||||
return fail(err)
|
||||
}
|
||||
fallbackItems, err := s.findDirectTileRoots(source, coveredRoots)
|
||||
if err != nil {
|
||||
return s.finishScanRun(ctx, runID, source, startedAt, "failed", nil, 0, 0, 0, 0, err)
|
||||
return fail(err)
|
||||
}
|
||||
|
||||
allItems := append(manifestItems, fallbackItems...)
|
||||
@@ -83,11 +104,12 @@ func (s *Store) runScan(ctx context.Context, runID string, source ScanSource, st
|
||||
return allItems[i].BasemapCode < allItems[j].BasemapCode
|
||||
})
|
||||
|
||||
addedCount, updatedCount, removedCount, err := s.persistScanResult(ctx, source, allItems)
|
||||
counts, err := s.persistScanResult(ctx, source, allItems)
|
||||
if err != nil {
|
||||
return s.finishScanRun(ctx, runID, source, startedAt, "failed", allItems, len(allItems), addedCount, updatedCount, removedCount, err)
|
||||
return s.finishScanRun(ctx, runID, source, startedAt, "failed", allItems, scanCounts{Scanned: len(allItems)}, err)
|
||||
}
|
||||
return s.finishScanRun(ctx, runID, source, startedAt, "succeeded", allItems, len(allItems), addedCount, updatedCount, removedCount, nil)
|
||||
counts.Scanned = len(allItems)
|
||||
return s.finishScanRun(ctx, runID, source, startedAt, "succeeded", allItems, counts, nil)
|
||||
}
|
||||
|
||||
func (s *Store) finishScanRun(
|
||||
@@ -97,20 +119,18 @@ func (s *Store) finishScanRun(
|
||||
startedAt time.Time,
|
||||
status string,
|
||||
items []scannedVersion,
|
||||
scannedCount int,
|
||||
addedCount int,
|
||||
updatedCount int,
|
||||
removedCount int,
|
||||
counts scanCounts,
|
||||
runErr error,
|
||||
) (ScanRun, error) {
|
||||
finishedAt := nowUTC()
|
||||
summary := map[string]any{
|
||||
"source_code": source.Code,
|
||||
"scan_root": source.RootPath,
|
||||
"scanned_count": scannedCount,
|
||||
"added_count": addedCount,
|
||||
"updated_count": updatedCount,
|
||||
"removed_count": removedCount,
|
||||
"source_code": source.Code,
|
||||
"scan_root": source.RootPath,
|
||||
"scanned_count": counts.Scanned,
|
||||
"added_count": counts.Added,
|
||||
"updated_count": counts.Updated,
|
||||
"removed_count": counts.Removed,
|
||||
"unchanged_count": counts.Unchanged,
|
||||
}
|
||||
if runErr != nil {
|
||||
summary["error"] = runErr.Error()
|
||||
@@ -127,7 +147,7 @@ func (s *Store) finishScanRun(
|
||||
UPDATE scan_runs
|
||||
SET status = ?, scanned_count = ?, added_count = ?, updated_count = ?, removed_count = ?, summary_json = ?, finished_at = ?
|
||||
WHERE id = ?
|
||||
`, status, scannedCount, addedCount, updatedCount, removedCount, writeJSON(summary), toRFC3339(finishedAt), runID)
|
||||
`, status, counts.Scanned, counts.Added, counts.Updated, counts.Removed, writeJSON(summary), toRFC3339(finishedAt), runID)
|
||||
if updateErr != nil {
|
||||
if runErr != nil {
|
||||
return ScanRun{}, fmt.Errorf("%v; update scan run: %w", runErr, updateErr)
|
||||
@@ -140,10 +160,10 @@ func (s *Store) finishScanRun(
|
||||
ScanSourceID: source.ID,
|
||||
SourceCode: source.Code,
|
||||
Status: status,
|
||||
ScannedCount: scannedCount,
|
||||
AddedCount: addedCount,
|
||||
UpdatedCount: updatedCount,
|
||||
RemovedCount: removedCount,
|
||||
ScannedCount: counts.Scanned,
|
||||
AddedCount: counts.Added,
|
||||
UpdatedCount: counts.Updated,
|
||||
RemovedCount: counts.Removed,
|
||||
Summary: summary,
|
||||
StartedAt: startedAt,
|
||||
FinishedAt: finishedAt,
|
||||
@@ -154,6 +174,11 @@ func (s *Store) finishScanRun(
|
||||
return run, nil
|
||||
}
|
||||
|
||||
// manifestScanMaxDepth bounds how deep the manifest lookup descends. Manifest
|
||||
// based basemaps place manifest.json at the basemap root or one version folder
|
||||
// below it, so anything deeper is tile payload that must not be walked.
|
||||
const manifestScanMaxDepth = 4
|
||||
|
||||
func (s *Store) findManifestVersions(source ScanSource) ([]scannedVersion, map[string]struct{}, error) {
|
||||
manifestName := strings.TrimSpace(source.ManifestName)
|
||||
if manifestName == "" {
|
||||
@@ -162,69 +187,103 @@ func (s *Store) findManifestVersions(source ScanSource) ([]scannedVersion, map[s
|
||||
|
||||
var items []scannedVersion
|
||||
coveredRoots := map[string]struct{}{}
|
||||
err := filepath.WalkDir(source.RootPath, func(path string, entry os.DirEntry, walkErr error) error {
|
||||
if walkErr != nil {
|
||||
return walkErr
|
||||
|
||||
var walk func(dir string, depth int) error
|
||||
walk = func(dir string, depth int) error {
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if entry.IsDir() {
|
||||
return nil
|
||||
}
|
||||
if !strings.EqualFold(entry.Name(), manifestName) {
|
||||
|
||||
// A directory holding a manifest is a basemap root: record it and stop,
|
||||
// so the tile payload underneath (the default "tiles" subtree) is never
|
||||
// traversed.
|
||||
for _, entry := range entries {
|
||||
if entry.IsDir() || !strings.EqualFold(entry.Name(), manifestName) {
|
||||
continue
|
||||
}
|
||||
manifestPath := filepath.Join(dir, entry.Name())
|
||||
manifest, err := LoadManifest(manifestPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tileRoot, err := resolveTileRoot(dir, manifest.RootPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
format, minZoom, maxZoom, found, err := inspectTileRoot(tileRoot, manifest.TileFormat)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if found {
|
||||
manifest.TileFormat = format
|
||||
manifest.MinZoom = minZoom
|
||||
manifest.MaxZoom = maxZoom
|
||||
}
|
||||
|
||||
if relPath, err := filepath.Rel(source.RootPath, tileRoot); err == nil {
|
||||
first := strings.Split(filepath.ToSlash(relPath), "/")[0]
|
||||
if first != "" && first != "." {
|
||||
coveredRoots[first] = struct{}{}
|
||||
}
|
||||
}
|
||||
|
||||
items = append(items, scannedVersion{
|
||||
BasemapCode: manifest.Code,
|
||||
Name: defaultText(manifest.Name, manifest.Code),
|
||||
Type: defaultText(manifest.Type, "xyz"),
|
||||
Status: defaultText(manifest.Status, "ready"),
|
||||
Description: manifest.Description,
|
||||
Version: defaultText(manifest.Version, "current"),
|
||||
Manifest: manifestPath,
|
||||
TileRoot: tileRoot,
|
||||
TileFormat: normalizeTileFormat(manifest.TileFormat),
|
||||
TileScheme: defaultText(manifest.TileScheme, "xyz"),
|
||||
MinZoom: manifest.MinZoom,
|
||||
MaxZoom: manifest.MaxZoom,
|
||||
BBox: manifest.BBox,
|
||||
Attribution: manifest.Attribution,
|
||||
Metadata: manifest.Metadata,
|
||||
IsDefault: manifest.IsDefault,
|
||||
})
|
||||
return nil
|
||||
}
|
||||
|
||||
manifest, err := LoadManifest(path)
|
||||
if err != nil {
|
||||
return err
|
||||
if depth >= manifestScanMaxDepth {
|
||||
return nil
|
||||
}
|
||||
tileRoot, err := resolveTileRoot(filepath.Dir(path), manifest.RootPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
format, minZoom, maxZoom, found, err := inspectTileRoot(tileRoot, manifest.TileFormat)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if found {
|
||||
manifest.TileFormat = format
|
||||
manifest.MinZoom = minZoom
|
||||
manifest.MaxZoom = maxZoom
|
||||
}
|
||||
|
||||
relPath, err := filepath.Rel(source.RootPath, tileRoot)
|
||||
if err == nil {
|
||||
first := strings.Split(filepath.ToSlash(relPath), "/")[0]
|
||||
if first != "" && first != "." {
|
||||
coveredRoots[first] = struct{}{}
|
||||
for _, entry := range entries {
|
||||
if !entry.IsDir() || isNumericDirName(entry.Name()) {
|
||||
continue
|
||||
}
|
||||
if err := walk(filepath.Join(dir, entry.Name()), depth+1); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
items = append(items, scannedVersion{
|
||||
BasemapCode: manifest.Code,
|
||||
Name: defaultText(manifest.Name, manifest.Code),
|
||||
Type: defaultText(manifest.Type, "xyz"),
|
||||
Status: defaultText(manifest.Status, "ready"),
|
||||
Description: manifest.Description,
|
||||
Version: defaultText(manifest.Version, "current"),
|
||||
Manifest: path,
|
||||
TileRoot: tileRoot,
|
||||
TileFormat: normalizeTileFormat(manifest.TileFormat),
|
||||
TileScheme: defaultText(manifest.TileScheme, "xyz"),
|
||||
MinZoom: manifest.MinZoom,
|
||||
MaxZoom: manifest.MaxZoom,
|
||||
BBox: manifest.BBox,
|
||||
Attribution: manifest.Attribution,
|
||||
Metadata: manifest.Metadata,
|
||||
IsDefault: manifest.IsDefault,
|
||||
})
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
}
|
||||
|
||||
if err := walk(source.RootPath, 0); err != nil {
|
||||
return nil, nil, fmt.Errorf("walk manifests: %w", err)
|
||||
}
|
||||
return items, coveredRoots, nil
|
||||
}
|
||||
|
||||
// isNumericDirName reports whether the directory name is a bare zoom level such
|
||||
// as "0" or "14". Raw tile roots nest z/x/y under the basemap directory, so
|
||||
// pruning these keeps the scan clear of the tile payload.
|
||||
func isNumericDirName(name string) bool {
|
||||
if name == "" {
|
||||
return false
|
||||
}
|
||||
for _, r := range name {
|
||||
if r < '0' || r > '9' {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func (s *Store) findDirectTileRoots(source ScanSource, skip map[string]struct{}) ([]scannedVersion, error) {
|
||||
entries, err := os.ReadDir(source.RootPath)
|
||||
if err != nil {
|
||||
@@ -352,57 +411,114 @@ func resolveTileRoot(baseDir, relativePath string) (string, error) {
|
||||
return resolved, nil
|
||||
}
|
||||
|
||||
func (s *Store) persistScanResult(ctx context.Context, source ScanSource, items []scannedVersion) (int, int, int, error) {
|
||||
// existingVersion is a snapshot of a stored version, used to detect whether a
|
||||
// rescan actually changed anything.
|
||||
type existingVersion struct {
|
||||
Name string
|
||||
Type string
|
||||
Status string
|
||||
Description string
|
||||
ManifestPath string
|
||||
TileRootPath string
|
||||
TileFormat string
|
||||
TileScheme string
|
||||
MinZoom int
|
||||
MaxZoom int
|
||||
IsDefault bool
|
||||
BBoxJSON string
|
||||
Attribution string
|
||||
MetadataJSON string
|
||||
}
|
||||
|
||||
func (s *Store) persistScanResult(ctx context.Context, source ScanSource, items []scannedVersion) (scanCounts, error) {
|
||||
tx, err := s.db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return 0, 0, 0, err
|
||||
return scanCounts{}, err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
|
||||
existing := map[string]struct{}{}
|
||||
existing := map[string]existingVersion{}
|
||||
rows, err := tx.QueryContext(ctx, `
|
||||
SELECT b.code, v.version
|
||||
SELECT
|
||||
b.code,
|
||||
v.version,
|
||||
b.name,
|
||||
b.type,
|
||||
b.status,
|
||||
b.description,
|
||||
v.manifest_path,
|
||||
v.tile_root_path,
|
||||
v.tile_format,
|
||||
v.tile_scheme,
|
||||
v.min_zoom,
|
||||
v.max_zoom,
|
||||
v.is_default,
|
||||
v.bbox_json,
|
||||
v.attribution,
|
||||
v.metadata_json
|
||||
FROM basemap_versions v
|
||||
JOIN basemaps b ON b.id = v.basemap_id
|
||||
WHERE v.scan_source_id = ?
|
||||
`, source.ID)
|
||||
if err != nil {
|
||||
return 0, 0, 0, err
|
||||
return scanCounts{}, err
|
||||
}
|
||||
for rows.Next() {
|
||||
var code string
|
||||
var version string
|
||||
if err := rows.Scan(&code, &version); err != nil {
|
||||
var item existingVersion
|
||||
var isDefault int
|
||||
if err := rows.Scan(
|
||||
&code,
|
||||
&version,
|
||||
&item.Name,
|
||||
&item.Type,
|
||||
&item.Status,
|
||||
&item.Description,
|
||||
&item.ManifestPath,
|
||||
&item.TileRootPath,
|
||||
&item.TileFormat,
|
||||
&item.TileScheme,
|
||||
&item.MinZoom,
|
||||
&item.MaxZoom,
|
||||
&isDefault,
|
||||
&item.BBoxJSON,
|
||||
&item.Attribution,
|
||||
&item.MetadataJSON,
|
||||
); err != nil {
|
||||
rows.Close()
|
||||
return 0, 0, 0, err
|
||||
return scanCounts{}, err
|
||||
}
|
||||
existing[code+"::"+version] = struct{}{}
|
||||
item.IsDefault = isDefault == 1
|
||||
existing[code+"::"+version] = item
|
||||
}
|
||||
rows.Close()
|
||||
if err := rows.Err(); err != nil {
|
||||
return 0, 0, 0, err
|
||||
return scanCounts{}, err
|
||||
}
|
||||
|
||||
now := nowUTC()
|
||||
found := map[string]struct{}{}
|
||||
touchedBasemaps := map[string]struct{}{}
|
||||
addedCount := 0
|
||||
updatedCount := 0
|
||||
counts := scanCounts{}
|
||||
for _, item := range items {
|
||||
key := item.BasemapCode + "::" + item.Version
|
||||
found[key] = struct{}{}
|
||||
if _, exists := existing[key]; exists {
|
||||
updatedCount++
|
||||
if prev, exists := existing[key]; exists {
|
||||
if scannedVersionMatches(item, prev) {
|
||||
counts.Unchanged++
|
||||
} else {
|
||||
counts.Updated++
|
||||
}
|
||||
} else {
|
||||
addedCount++
|
||||
counts.Added++
|
||||
}
|
||||
if err := s.upsertScannedVersionTx(ctx, tx, source, item, now); err != nil {
|
||||
return 0, 0, 0, err
|
||||
return scanCounts{}, err
|
||||
}
|
||||
touchedBasemaps[item.BasemapCode] = struct{}{}
|
||||
}
|
||||
|
||||
removedCount := 0
|
||||
for key := range existing {
|
||||
if _, ok := found[key]; ok {
|
||||
continue
|
||||
@@ -412,22 +528,41 @@ func (s *Store) persistScanResult(ctx context.Context, source ScanSource, items
|
||||
continue
|
||||
}
|
||||
if err := s.deleteVersionBySourceTx(ctx, tx, source.ID, parts[0], parts[1]); err != nil {
|
||||
return 0, 0, 0, err
|
||||
return scanCounts{}, err
|
||||
}
|
||||
touchedBasemaps[parts[0]] = struct{}{}
|
||||
removedCount++
|
||||
counts.Removed++
|
||||
}
|
||||
|
||||
for code := range touchedBasemaps {
|
||||
if err := s.normalizeBasemapDefaultsTx(ctx, tx, code, now); err != nil {
|
||||
return 0, 0, 0, err
|
||||
return scanCounts{}, err
|
||||
}
|
||||
}
|
||||
|
||||
if err := tx.Commit(); err != nil {
|
||||
return 0, 0, 0, err
|
||||
return scanCounts{}, err
|
||||
}
|
||||
return addedCount, updatedCount, removedCount, nil
|
||||
return counts, nil
|
||||
}
|
||||
|
||||
// scannedVersionMatches reports whether a freshly scanned version is identical
|
||||
// to what is already stored, so an unchanged rescan is not reported as updated.
|
||||
func scannedVersionMatches(item scannedVersion, prev existingVersion) bool {
|
||||
return item.Name == prev.Name &&
|
||||
item.Type == prev.Type &&
|
||||
item.Status == prev.Status &&
|
||||
item.Description == prev.Description &&
|
||||
item.Manifest == prev.ManifestPath &&
|
||||
item.TileRoot == prev.TileRootPath &&
|
||||
normalizeTileFormat(item.TileFormat) == prev.TileFormat &&
|
||||
item.TileScheme == prev.TileScheme &&
|
||||
item.MinZoom == prev.MinZoom &&
|
||||
item.MaxZoom == prev.MaxZoom &&
|
||||
item.IsDefault == prev.IsDefault &&
|
||||
writeJSON(item.BBox) == prev.BBoxJSON &&
|
||||
item.Attribution == prev.Attribution &&
|
||||
writeJSON(item.Metadata) == prev.MetadataJSON
|
||||
}
|
||||
|
||||
func (s *Store) upsertScannedVersionTx(ctx context.Context, tx *sql.Tx, source ScanSource, item scannedVersion, now time.Time) error {
|
||||
|
||||
@@ -11,37 +11,97 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
_ "modernc.org/sqlite"
|
||||
)
|
||||
|
||||
const (
|
||||
defaultScanSourceCode = "local-basemaps"
|
||||
defaultManifestName = "manifest.json"
|
||||
readPermission = "read"
|
||||
defaultScanSourceCode = "local-basemaps"
|
||||
defaultManifestName = "manifest.json"
|
||||
readPermission = "read"
|
||||
defaultVectorSourceCode = "local-vectors"
|
||||
)
|
||||
|
||||
const (
|
||||
// sqliteMaxOpenConns lets concurrent tile reads overlap instead of being
|
||||
// serialized behind a single connection.
|
||||
sqliteMaxOpenConns = 8
|
||||
)
|
||||
|
||||
// sqliteDSN tunes each pooled connection. busy_timeout makes concurrent
|
||||
// writers wait for the lock instead of failing immediately.
|
||||
func sqliteDSN(path string) string {
|
||||
params := url.Values{}
|
||||
params.Add("_pragma", "busy_timeout(5000)")
|
||||
params.Add("_pragma", "foreign_keys(1)")
|
||||
params.Add("_pragma", "synchronous(NORMAL)")
|
||||
return "file:" + filepath.ToSlash(path) + "?" + params.Encode()
|
||||
}
|
||||
|
||||
var (
|
||||
// ErrScanInProgress is returned when a scan is already running.
|
||||
ErrScanInProgress = errors.New("scan already in progress")
|
||||
// ErrScanSourceNotFound is returned when a scan source code is unknown.
|
||||
ErrScanSourceNotFound = errors.New("scan source not found")
|
||||
// ErrScanSourceDisabled is returned when a scan source is turned off.
|
||||
ErrScanSourceDisabled = errors.New("scan source disabled")
|
||||
)
|
||||
|
||||
// scanSourceError keeps the operator-facing message stable while still matching
|
||||
// errors.Is against a sentinel.
|
||||
type scanSourceError struct {
|
||||
sentinel error
|
||||
code string
|
||||
reason string
|
||||
}
|
||||
|
||||
func (e *scanSourceError) Error() string {
|
||||
return fmt.Sprintf("scan source %q %s", e.code, e.reason)
|
||||
}
|
||||
|
||||
func (e *scanSourceError) Is(target error) bool {
|
||||
return target == e.sentinel
|
||||
}
|
||||
|
||||
type StoreConfig struct {
|
||||
SQLitePath string
|
||||
ScanRoot string
|
||||
VectorScanRoot string
|
||||
APIBaseURL string
|
||||
TileBaseURL string
|
||||
VectorBaseURL string
|
||||
AutoScanOnStart bool
|
||||
SQLitePath string
|
||||
ScanRoot string
|
||||
VectorScanRoot string
|
||||
APIBaseURL string
|
||||
TileBaseURL string
|
||||
VectorBaseURL string
|
||||
AutoScanOnStart bool
|
||||
TokenDefaultTTL time.Duration
|
||||
TileJobCommand string
|
||||
TileJobVRTCommand string
|
||||
TileJobLogDir string
|
||||
HealthCheckInterval time.Duration
|
||||
}
|
||||
|
||||
type Store struct {
|
||||
db *sql.DB
|
||||
cfg StoreConfig
|
||||
httpClient *http.Client
|
||||
|
||||
// scanMu serializes scans so two runs cannot interleave their read-then-write
|
||||
// bookkeeping.
|
||||
scanMu sync.Mutex
|
||||
// pushWG tracks detached push callbacks so Close can drain them.
|
||||
pushWG sync.WaitGroup
|
||||
|
||||
bgOnce sync.Once
|
||||
bgCancel context.CancelFunc
|
||||
bgCtx context.Context
|
||||
bgWG sync.WaitGroup
|
||||
}
|
||||
|
||||
type CreatedToken struct {
|
||||
@@ -66,13 +126,21 @@ func OpenStore(cfg StoreConfig) (*Store, error) {
|
||||
return nil, fmt.Errorf("create sqlite dir: %w", err)
|
||||
}
|
||||
|
||||
db, err := sql.Open("sqlite", cfg.SQLitePath)
|
||||
db, err := sql.Open("sqlite", sqliteDSN(cfg.SQLitePath))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("open sqlite: %w", err)
|
||||
}
|
||||
db.SetConnMaxLifetime(10 * time.Minute)
|
||||
db.SetMaxOpenConns(1)
|
||||
db.SetMaxIdleConns(1)
|
||||
db.SetConnMaxLifetime(30 * time.Minute)
|
||||
db.SetConnMaxIdleTime(5 * time.Minute)
|
||||
db.SetMaxOpenConns(sqliteMaxOpenConns)
|
||||
db.SetMaxIdleConns(sqliteMaxOpenConns)
|
||||
|
||||
// WAL keeps tile reads from blocking on scans/pushes. It is persisted in the
|
||||
// database file, so enabling it once is enough; a filesystem that cannot
|
||||
// support WAL (some network shares) must not stop the service from starting.
|
||||
if _, err := db.Exec("PRAGMA journal_mode=WAL;"); err != nil {
|
||||
log.Printf("enable sqlite WAL skipped: %v", err)
|
||||
}
|
||||
|
||||
store := &Store{
|
||||
db: db,
|
||||
@@ -109,6 +177,13 @@ func (s *Store) Close() error {
|
||||
if s == nil || s.db == nil {
|
||||
return nil
|
||||
}
|
||||
// Stop the periodic loops, then let detached push callbacks finish before
|
||||
// the pool goes away.
|
||||
if s.bgCancel != nil {
|
||||
s.bgCancel()
|
||||
}
|
||||
s.bgWG.Wait()
|
||||
s.pushWG.Wait()
|
||||
return s.db.Close()
|
||||
}
|
||||
|
||||
@@ -142,6 +217,9 @@ func (s *Store) bootstrap(ctx context.Context) error {
|
||||
bbox_json TEXT NOT NULL DEFAULT '',
|
||||
attribution TEXT NOT NULL DEFAULT '',
|
||||
metadata_json TEXT NOT NULL DEFAULT '',
|
||||
health TEXT NOT NULL DEFAULT '',
|
||||
health_checked_at TEXT NOT NULL DEFAULT '',
|
||||
health_detail TEXT NOT NULL DEFAULT '',
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL,
|
||||
UNIQUE(basemap_id, version),
|
||||
@@ -188,6 +266,7 @@ func (s *Store) bootstrap(ctx context.Context) error {
|
||||
name TEXT NOT NULL,
|
||||
token_hash TEXT NOT NULL UNIQUE,
|
||||
token_prefix TEXT NOT NULL,
|
||||
scopes TEXT NOT NULL DEFAULT 'read',
|
||||
status TEXT NOT NULL,
|
||||
expires_at TEXT,
|
||||
last_used_at TEXT,
|
||||
@@ -213,6 +292,8 @@ func (s *Store) bootstrap(ctx context.Context) error {
|
||||
response_status INTEGER,
|
||||
response_body TEXT NOT NULL DEFAULT '',
|
||||
error_message TEXT NOT NULL DEFAULT '',
|
||||
attempt_count INTEGER NOT NULL DEFAULT 0,
|
||||
next_attempt_at TEXT,
|
||||
pushed_at TEXT NOT NULL,
|
||||
finished_at TEXT,
|
||||
FOREIGN KEY (target_system_id) REFERENCES target_systems(id) ON DELETE CASCADE,
|
||||
@@ -231,6 +312,9 @@ func (s *Store) bootstrap(ctx context.Context) error {
|
||||
checksum TEXT NOT NULL DEFAULT '',
|
||||
download_url TEXT NOT NULL,
|
||||
metadata_json TEXT NOT NULL DEFAULT '',
|
||||
health TEXT NOT NULL DEFAULT '',
|
||||
health_checked_at TEXT NOT NULL DEFAULT '',
|
||||
health_detail TEXT NOT NULL DEFAULT '',
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
);`,
|
||||
@@ -252,15 +336,55 @@ func (s *Store) bootstrap(ctx context.Context) error {
|
||||
response_status INTEGER,
|
||||
response_body TEXT NOT NULL DEFAULT '',
|
||||
error_message TEXT NOT NULL DEFAULT '',
|
||||
attempt_count INTEGER NOT NULL DEFAULT 0,
|
||||
next_attempt_at TEXT,
|
||||
pushed_at TEXT NOT NULL,
|
||||
finished_at TEXT,
|
||||
FOREIGN KEY (target_system_id) REFERENCES target_systems(id) ON DELETE CASCADE,
|
||||
FOREIGN KEY (vector_asset_id) REFERENCES vector_assets(id) ON DELETE CASCADE
|
||||
);`,
|
||||
`CREATE TABLE IF NOT EXISTS tile_jobs (
|
||||
id TEXT PRIMARY KEY,
|
||||
code TEXT NOT NULL UNIQUE,
|
||||
name TEXT NOT NULL,
|
||||
source_path TEXT NOT NULL,
|
||||
source_count INTEGER NOT NULL DEFAULT 1,
|
||||
sources_json TEXT NOT NULL DEFAULT '',
|
||||
target_code TEXT NOT NULL,
|
||||
output_root TEXT NOT NULL,
|
||||
zoom_min INTEGER NOT NULL DEFAULT 0,
|
||||
zoom_max INTEGER NOT NULL DEFAULT 0,
|
||||
tile_size INTEGER NOT NULL DEFAULT 256,
|
||||
tile_format TEXT NOT NULL DEFAULT 'webp',
|
||||
status TEXT NOT NULL,
|
||||
progress TEXT NOT NULL DEFAULT '',
|
||||
message TEXT NOT NULL DEFAULT '',
|
||||
log_path TEXT NOT NULL DEFAULT '',
|
||||
callback_url TEXT NOT NULL DEFAULT '',
|
||||
callback_status TEXT NOT NULL DEFAULT '',
|
||||
callback_attempts INTEGER NOT NULL DEFAULT 0,
|
||||
callback_next_attempt_at TEXT,
|
||||
callback_message TEXT NOT NULL DEFAULT '',
|
||||
requested_by TEXT NOT NULL DEFAULT '',
|
||||
created_at TEXT NOT NULL,
|
||||
started_at TEXT,
|
||||
finished_at TEXT
|
||||
);`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_tile_jobs_status ON tile_jobs(status, created_at);`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_tile_jobs_target ON tile_jobs(target_code);`,
|
||||
`CREATE TABLE IF NOT EXISTS layer_sources (
|
||||
id TEXT PRIMARY KEY,
|
||||
target_code TEXT NOT NULL,
|
||||
source_path TEXT NOT NULL,
|
||||
added_at TEXT NOT NULL,
|
||||
UNIQUE(target_code, source_path)
|
||||
);`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_layer_sources_target ON layer_sources(target_code);`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_versions_basemap ON basemap_versions(basemap_id);`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_versions_source ON basemap_versions(scan_source_id);`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_grants_token ON token_grants(token_id);`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_push_target ON push_records(target_system_id);`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_vector_file_name ON vector_assets(file_name COLLATE NOCASE);`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_vector_grants_token ON vector_token_grants(token_id);`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_vector_push_target ON vector_push_records(target_system_id);`,
|
||||
}
|
||||
@@ -270,9 +394,77 @@ func (s *Store) bootstrap(ctx context.Context) error {
|
||||
return fmt.Errorf("bootstrap sqlite: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
// CREATE TABLE IF NOT EXISTS never evolves an existing database, so columns
|
||||
// added after the first release are patched in here.
|
||||
migrations := []struct {
|
||||
table string
|
||||
column string
|
||||
definition string
|
||||
}{
|
||||
{"basemap_versions", "health", "TEXT NOT NULL DEFAULT ''"},
|
||||
{"basemap_versions", "health_checked_at", "TEXT NOT NULL DEFAULT ''"},
|
||||
{"basemap_versions", "health_detail", "TEXT NOT NULL DEFAULT ''"},
|
||||
{"vector_assets", "health", "TEXT NOT NULL DEFAULT ''"},
|
||||
{"vector_assets", "health_checked_at", "TEXT NOT NULL DEFAULT ''"},
|
||||
{"vector_assets", "health_detail", "TEXT NOT NULL DEFAULT ''"},
|
||||
{"push_records", "attempt_count", "INTEGER NOT NULL DEFAULT 0"},
|
||||
{"push_records", "next_attempt_at", "TEXT"},
|
||||
{"vector_push_records", "attempt_count", "INTEGER NOT NULL DEFAULT 0"},
|
||||
{"vector_push_records", "next_attempt_at", "TEXT"},
|
||||
{"service_tokens", "scopes", "TEXT NOT NULL DEFAULT 'read'"},
|
||||
{"tile_jobs", "source_count", "INTEGER NOT NULL DEFAULT 1"},
|
||||
{"tile_jobs", "sources_json", "TEXT NOT NULL DEFAULT ''"},
|
||||
{"tile_jobs", "callback_url", "TEXT NOT NULL DEFAULT ''"},
|
||||
{"tile_jobs", "callback_status", "TEXT NOT NULL DEFAULT ''"},
|
||||
{"tile_jobs", "callback_attempts", "INTEGER NOT NULL DEFAULT 0"},
|
||||
{"tile_jobs", "callback_next_attempt_at", "TEXT"},
|
||||
{"tile_jobs", "callback_message", "TEXT NOT NULL DEFAULT ''"},
|
||||
}
|
||||
for _, migration := range migrations {
|
||||
if err := s.ensureColumn(ctx, migration.table, migration.column, migration.definition); err != nil {
|
||||
return fmt.Errorf("bootstrap migrate %s.%s: %w", migration.table, migration.column, err)
|
||||
}
|
||||
}
|
||||
// These indexes reference migrated columns, so they run after the patch.
|
||||
for _, statement := range []string{
|
||||
`CREATE INDEX IF NOT EXISTS idx_push_next_attempt ON push_records(next_attempt_at);`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_vector_push_next_attempt ON vector_push_records(next_attempt_at);`,
|
||||
} {
|
||||
if _, err := s.db.ExecContext(ctx, statement); err != nil {
|
||||
return fmt.Errorf("bootstrap index: %w", err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ensureColumn adds a column when an existing table is missing it. PRAGMA
|
||||
// table_info is the portable way to test SQLite column presence.
|
||||
func (s *Store) ensureColumn(ctx context.Context, table, column, definition string) error {
|
||||
rows, err := s.db.QueryContext(ctx, "PRAGMA table_info("+table+")")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
for rows.Next() {
|
||||
var cid, notNull, primaryKey int
|
||||
var name, columnType string
|
||||
var defaultValue sql.NullString
|
||||
if err := rows.Scan(&cid, &name, &columnType, ¬Null, &defaultValue, &primaryKey); err != nil {
|
||||
return err
|
||||
}
|
||||
if name == column {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = s.db.ExecContext(ctx, "ALTER TABLE "+table+" ADD COLUMN "+column+" "+definition)
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *Store) AutoScanEnabled() bool {
|
||||
return s.cfg.AutoScanOnStart
|
||||
}
|
||||
|
||||
@@ -2,7 +2,6 @@ package basemap
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
@@ -228,6 +227,7 @@ func scanServiceToken(scanner interface {
|
||||
}) (ServiceToken, string, error) {
|
||||
var item ServiceToken
|
||||
var tokenHash string
|
||||
var scopes string
|
||||
var expiresAt sql.NullString
|
||||
var lastUsedAt sql.NullString
|
||||
var createdAt string
|
||||
@@ -236,6 +236,7 @@ func scanServiceToken(scanner interface {
|
||||
&item.Name,
|
||||
&tokenHash,
|
||||
&item.TokenPrefix,
|
||||
&scopes,
|
||||
&item.Status,
|
||||
&expiresAt,
|
||||
&lastUsedAt,
|
||||
@@ -244,6 +245,7 @@ func scanServiceToken(scanner interface {
|
||||
return ServiceToken{}, "", err
|
||||
}
|
||||
item.CreatedAt = parseTime(createdAt)
|
||||
item.Scopes = splitTokenScopes(scopes)
|
||||
if expiresAt.Valid {
|
||||
value := parseTime(expiresAt.String)
|
||||
item.ExpiresAt = &value
|
||||
@@ -290,6 +292,7 @@ func scanPushRecord(scanner interface {
|
||||
var item PushRecord
|
||||
var requestJSON string
|
||||
var responseStatus sql.NullInt64
|
||||
var nextAttemptAt sql.NullString
|
||||
var finishedAt sql.NullString
|
||||
var pushedAt string
|
||||
if err := scanner.Scan(
|
||||
@@ -304,6 +307,8 @@ func scanPushRecord(scanner interface {
|
||||
&responseStatus,
|
||||
&item.ResponseBody,
|
||||
&item.ErrorMessage,
|
||||
&item.AttemptCount,
|
||||
&nextAttemptAt,
|
||||
&pushedAt,
|
||||
&finishedAt,
|
||||
); err != nil {
|
||||
@@ -319,6 +324,10 @@ func scanPushRecord(scanner interface {
|
||||
value := int(responseStatus.Int64)
|
||||
item.ResponseStatus = &value
|
||||
}
|
||||
if nextAttemptAt.Valid {
|
||||
value := parseTime(nextAttemptAt.String)
|
||||
item.NextAttemptAt = &value
|
||||
}
|
||||
item.PushedAt = parseTime(pushedAt)
|
||||
if finishedAt.Valid {
|
||||
value := parseTime(finishedAt.String)
|
||||
@@ -326,19 +335,3 @@ func scanPushRecord(scanner interface {
|
||||
}
|
||||
return item, nil
|
||||
}
|
||||
|
||||
func mustOneRow[T any](item T, err error) (T, error) {
|
||||
if err != nil {
|
||||
return item, err
|
||||
}
|
||||
return item, nil
|
||||
}
|
||||
|
||||
func unexpectedRowsAffected(count int64, expected ...int64) error {
|
||||
for _, value := range expected {
|
||||
if count == value {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("unexpected rows affected: %d", count)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,714 @@
|
||||
package basemap
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"tile-server/api-go/internal/uid"
|
||||
)
|
||||
|
||||
// Tile job statuses.
|
||||
const (
|
||||
tileJobPending = "pending"
|
||||
tileJobRunning = "running"
|
||||
tileJobSucceeded = "succeeded"
|
||||
tileJobFailed = "failed"
|
||||
)
|
||||
|
||||
// defaultTileJobCommand slices a raster into XYZ tiles. {input} is the source
|
||||
// GeoTIFF, {output} the directory the gateway scans.
|
||||
const defaultTileJobCommand = "gdal2tiles.py --xyz --zoom={zmin}-{zmax} --tilesize={tilesize} --exclude {input} {output}"
|
||||
|
||||
// defaultTileJobVRTCommand mosaics a layer's sources into one virtual raster
|
||||
// so overlapping flights composite per pixel instead of per tile.
|
||||
const defaultTileJobVRTCommand = "gdalbuildvrt -resolution highest {output_vrt} {inputs}"
|
||||
|
||||
// tileJobColumns keeps every read path in step with the struct.
|
||||
const tileJobColumns = `id, code, name, source_path, source_count, sources_json, target_code, output_root, zoom_min, zoom_max,
|
||||
tile_size, tile_format, status, progress, message, log_path, callback_url, callback_status,
|
||||
callback_attempts, callback_next_attempt_at, callback_message, requested_by, created_at, started_at, finished_at`
|
||||
|
||||
type TileJob struct {
|
||||
ID string `json:"id"`
|
||||
Code string `json:"code"`
|
||||
Name string `json:"name"`
|
||||
SourcePath string `json:"source_path"`
|
||||
SourceCount int `json:"source_count"`
|
||||
SourcePaths []string `json:"source_paths,omitempty"`
|
||||
TargetCode string `json:"target_code"`
|
||||
OutputRoot string `json:"output_root"`
|
||||
ZoomMin int `json:"zoom_min"`
|
||||
ZoomMax int `json:"zoom_max"`
|
||||
TileSize int `json:"tile_size"`
|
||||
TileFormat string `json:"tile_format"`
|
||||
Status string `json:"status"`
|
||||
Progress string `json:"progress"`
|
||||
Message string `json:"message"`
|
||||
LogPath string `json:"log_path"`
|
||||
CallbackURL string `json:"callback_url,omitempty"`
|
||||
CallbackStatus string `json:"callback_status,omitempty"`
|
||||
CallbackAttempts int `json:"callback_attempts,omitempty"`
|
||||
CallbackMessage string `json:"callback_message,omitempty"`
|
||||
CallbackNextAttemptAt *time.Time `json:"callback_next_attempt_at,omitempty"`
|
||||
RequestedBy string `json:"requested_by"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
StartedAt *time.Time `json:"started_at,omitempty"`
|
||||
FinishedAt *time.Time `json:"finished_at,omitempty"`
|
||||
}
|
||||
|
||||
type CreateTileJobInput struct {
|
||||
Code string `json:"code"`
|
||||
Name string `json:"name"`
|
||||
SourcePath string `json:"source_path"`
|
||||
SourcePaths []string `json:"source_paths"`
|
||||
TargetCode string `json:"target_code"`
|
||||
ZoomMin int `json:"zoom_min"`
|
||||
ZoomMax int `json:"zoom_max"`
|
||||
TileSize int `json:"tile_size"`
|
||||
TileFormat string `json:"tile_format"`
|
||||
CallbackURL string `json:"callback_url"`
|
||||
RequestedBy string `json:"requested_by"`
|
||||
}
|
||||
|
||||
// CreateTileJob validates the request and queues it. Slicing itself runs in the
|
||||
// background so a long gdal2tiles run cannot block the console.
|
||||
func (s *Store) CreateTileJob(ctx context.Context, input CreateTileJobInput) (TileJob, error) {
|
||||
sources, err := resolveTileJobSources(input)
|
||||
if err != nil {
|
||||
return TileJob{}, err
|
||||
}
|
||||
absSource := sources[0]
|
||||
|
||||
targetCode := normalizeBasemapCode(input.TargetCode)
|
||||
if targetCode == "" {
|
||||
return TileJob{}, errors.New("target code is required")
|
||||
}
|
||||
scanRoot := strings.TrimSpace(s.cfg.ScanRoot)
|
||||
if scanRoot == "" {
|
||||
return TileJob{}, errors.New("basemap scan root is not configured")
|
||||
}
|
||||
outputRoot := filepath.Join(scanRoot, targetCode)
|
||||
|
||||
if input.ZoomMin < 0 || input.ZoomMax < 0 || input.ZoomMax < input.ZoomMin {
|
||||
return TileJob{}, errors.New("invalid zoom range")
|
||||
}
|
||||
if input.ZoomMax > 24 {
|
||||
return TileJob{}, errors.New("zoom must be 24 or lower")
|
||||
}
|
||||
tileSize := input.TileSize
|
||||
if tileSize <= 0 {
|
||||
tileSize = 256
|
||||
}
|
||||
tileFormat := strings.ToLower(strings.TrimSpace(input.TileFormat))
|
||||
if tileFormat == "" {
|
||||
tileFormat = "webp"
|
||||
}
|
||||
code := normalizeBasemapCode(input.Code)
|
||||
if code == "" {
|
||||
code = normalizeBasemapCode(targetCode + "-" + nowUTC().Format("20060102150405"))
|
||||
}
|
||||
name := strings.TrimSpace(input.Name)
|
||||
if name == "" {
|
||||
name = filepath.Base(absSource)
|
||||
}
|
||||
|
||||
now := nowUTC()
|
||||
job := TileJob{
|
||||
ID: uid.New(),
|
||||
Code: code,
|
||||
Name: name,
|
||||
SourcePath: absSource,
|
||||
SourceCount: len(sources),
|
||||
SourcePaths: sources,
|
||||
TargetCode: targetCode,
|
||||
OutputRoot: outputRoot,
|
||||
ZoomMin: input.ZoomMin,
|
||||
ZoomMax: input.ZoomMax,
|
||||
TileSize: tileSize,
|
||||
TileFormat: tileFormat,
|
||||
Status: tileJobPending,
|
||||
RequestedBy: strings.TrimSpace(input.RequestedBy),
|
||||
CreatedAt: now,
|
||||
}
|
||||
job.CallbackURL = strings.TrimSpace(input.CallbackURL)
|
||||
|
||||
// A caller-supplied code makes submission idempotent: sending the same code
|
||||
// again returns the job that already exists instead of queueing a second copy.
|
||||
if existing, ok, err := s.getTileJobByCode(ctx, job.Code); err != nil {
|
||||
return TileJob{}, err
|
||||
} else if ok {
|
||||
return existing, nil
|
||||
}
|
||||
|
||||
if _, err := s.db.ExecContext(ctx, `
|
||||
INSERT INTO tile_jobs (
|
||||
id, code, name, source_path, source_count, sources_json, target_code, output_root, zoom_min, zoom_max,
|
||||
tile_size, tile_format, status, progress, message, log_path, callback_url, requested_by, created_at
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, '', '', '', ?, ?, ?)
|
||||
`, job.ID, job.Code, job.Name, job.SourcePath, job.SourceCount, writeJSON(job.SourcePaths), job.TargetCode, job.OutputRoot,
|
||||
job.ZoomMin, job.ZoomMax, job.TileSize, job.TileFormat, job.Status,
|
||||
job.CallbackURL, job.RequestedBy, toRFC3339(now)); err != nil {
|
||||
if strings.Contains(err.Error(), "UNIQUE") {
|
||||
if existing, ok, getErr := s.getTileJobByCode(ctx, job.Code); getErr == nil && ok {
|
||||
return existing, nil
|
||||
}
|
||||
}
|
||||
return TileJob{}, err
|
||||
}
|
||||
// Remember what this layer is made of so it can be rebuilt later.
|
||||
if err := s.AddLayerSources(ctx, job.TargetCode, sources); err != nil {
|
||||
log.Printf("record layer sources for %s: %v", job.TargetCode, err)
|
||||
}
|
||||
|
||||
return job, nil
|
||||
}
|
||||
|
||||
func (s *Store) ListTileJobs(ctx context.Context, limit int) ([]TileJob, error) {
|
||||
if limit <= 0 {
|
||||
limit = 20
|
||||
}
|
||||
rows, err := s.db.QueryContext(ctx, "SELECT "+tileJobColumns+`
|
||||
FROM tile_jobs
|
||||
ORDER BY created_at DESC
|
||||
LIMIT ?
|
||||
`, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var items []TileJob
|
||||
for rows.Next() {
|
||||
job, err := scanTileJob(rows)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
items = append(items, job)
|
||||
}
|
||||
return items, rows.Err()
|
||||
}
|
||||
|
||||
func scanTileJob(scanner interface {
|
||||
Scan(dest ...any) error
|
||||
}) (TileJob, error) {
|
||||
var job TileJob
|
||||
var createdAt string
|
||||
var sourcesJSON string
|
||||
var startedAt, finishedAt, callbackNextAttemptAt sql.NullString
|
||||
if err := scanner.Scan(
|
||||
&job.ID, &job.Code, &job.Name, &job.SourcePath, &job.SourceCount, &sourcesJSON, &job.TargetCode, &job.OutputRoot,
|
||||
&job.ZoomMin, &job.ZoomMax, &job.TileSize, &job.TileFormat, &job.Status,
|
||||
&job.Progress, &job.Message, &job.LogPath, &job.CallbackURL, &job.CallbackStatus, &job.CallbackAttempts,
|
||||
&callbackNextAttemptAt, &job.CallbackMessage, &job.RequestedBy, &createdAt,
|
||||
&startedAt, &finishedAt,
|
||||
); err != nil {
|
||||
return TileJob{}, err
|
||||
}
|
||||
job.CreatedAt = parseTime(createdAt)
|
||||
if strings.TrimSpace(sourcesJSON) != "" {
|
||||
_ = readMaybeJSON(sourcesJSON, &job.SourcePaths)
|
||||
}
|
||||
if len(job.SourcePaths) == 0 && job.SourcePath != "" {
|
||||
job.SourcePaths = []string{job.SourcePath}
|
||||
}
|
||||
if startedAt.Valid {
|
||||
value := parseTime(startedAt.String)
|
||||
job.StartedAt = &value
|
||||
}
|
||||
if finishedAt.Valid {
|
||||
value := parseTime(finishedAt.String)
|
||||
job.FinishedAt = &value
|
||||
}
|
||||
if callbackNextAttemptAt.Valid {
|
||||
value := parseTime(callbackNextAttemptAt.String)
|
||||
job.CallbackNextAttemptAt = &value
|
||||
}
|
||||
return job, nil
|
||||
}
|
||||
|
||||
// GetTileJob returns one job by id so a caller can poll a submission.
|
||||
func (s *Store) GetTileJob(ctx context.Context, id string) (TileJob, error) {
|
||||
row := s.db.QueryRowContext(ctx, "SELECT "+tileJobColumns+" FROM tile_jobs WHERE id = ?", strings.TrimSpace(id))
|
||||
job, err := scanTileJob(row)
|
||||
if err != nil {
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return TileJob{}, fmt.Errorf("tile job %q not found", id)
|
||||
}
|
||||
return TileJob{}, err
|
||||
}
|
||||
return job, nil
|
||||
}
|
||||
|
||||
func (s *Store) getTileJobByCode(ctx context.Context, code string) (TileJob, bool, error) {
|
||||
row := s.db.QueryRowContext(ctx, "SELECT "+tileJobColumns+" FROM tile_jobs WHERE code = ?", code)
|
||||
job, err := scanTileJob(row)
|
||||
if err != nil {
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return TileJob{}, false, nil
|
||||
}
|
||||
return TileJob{}, false, err
|
||||
}
|
||||
return job, true, nil
|
||||
}
|
||||
|
||||
// runTileJob executes one queued job and records the outcome verbatim.
|
||||
func (s *Store) runTileJob(ctx context.Context, job TileJob) {
|
||||
startedAt := nowUTC()
|
||||
if _, err := s.db.ExecContext(ctx, `
|
||||
UPDATE tile_jobs SET status = ?, started_at = ?, message = '' WHERE id = ?
|
||||
`, tileJobRunning, toRFC3339(startedAt), job.ID); err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
logPath, _, runErr := s.executeTileJob(ctx, job)
|
||||
|
||||
status := tileJobSucceeded
|
||||
message := ""
|
||||
if runErr != nil {
|
||||
status = tileJobFailed
|
||||
message = runErr.Error()
|
||||
}
|
||||
finishedAt := nowUTC()
|
||||
if _, err := s.db.ExecContext(context.Background(), `
|
||||
UPDATE tile_jobs SET status = ?, message = ?, log_path = ?, progress = ?, finished_at = ? WHERE id = ?
|
||||
`, status, message, logPath, message, toRFC3339(finishedAt), job.ID); err != nil {
|
||||
return
|
||||
}
|
||||
// Tell the submitting system the job is done, if it asked to be told.
|
||||
if strings.TrimSpace(job.CallbackURL) != "" {
|
||||
s.queueTileJobCallback(context.Background(), job.ID)
|
||||
}
|
||||
|
||||
// A sliced basemap only becomes visible after the scan picks it up.
|
||||
if status == tileJobSucceeded && strings.TrimSpace(s.cfg.ScanRoot) != "" {
|
||||
if _, err := s.RunScan(context.Background(), defaultScanSourceCode); err != nil {
|
||||
if _, updateErr := s.db.ExecContext(context.Background(), `
|
||||
UPDATE tile_jobs SET message = ? WHERE id = ?
|
||||
`, "切片完成，但扫描入库失败："+err.Error(), job.ID); updateErr != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// executeTileJob builds the input (mosaicking when several sources were
|
||||
// submitted), runs the slicer and captures its output.
|
||||
func (s *Store) executeTileJob(ctx context.Context, job TileJob) (string, []string, error) {
|
||||
if err := os.MkdirAll(job.OutputRoot, 0o755); err != nil {
|
||||
return "", nil, fmt.Errorf(
|
||||
"create output root %s: %w (the account running the service needs write access to the basemap scan root)",
|
||||
job.OutputRoot, err)
|
||||
}
|
||||
logDir := strings.TrimSpace(s.cfg.TileJobLogDir)
|
||||
if logDir == "" {
|
||||
// Keep logs beside the data: a system temp directory is not always
|
||||
// writable for the service account, and this stays under the scan root
|
||||
// without introducing a zoom directory the scanner would pick up.
|
||||
logDir = filepath.Join(filepath.Dir(job.OutputRoot), ".tile-jobs")
|
||||
}
|
||||
if err := os.MkdirAll(logDir, 0o755); err != nil {
|
||||
return "", nil, fmt.Errorf("create log dir: %w", err)
|
||||
}
|
||||
logPath := filepath.Join(logDir, job.Code+".log")
|
||||
logFile, err := os.Create(logPath)
|
||||
if err != nil {
|
||||
return logPath, nil, fmt.Errorf("create job log: %w", err)
|
||||
}
|
||||
defer logFile.Close()
|
||||
|
||||
// One source slices directly; several are mosaicked first so overlapping
|
||||
// flights composite per pixel instead of per tile.
|
||||
input := job.SourcePath
|
||||
if len(job.SourcePaths) > 1 {
|
||||
vrtPath := filepath.Join(logDir, job.Code+".vrt")
|
||||
if err := s.buildTileJobMosaic(ctx, job, vrtPath, logFile); err != nil {
|
||||
return logPath, nil, err
|
||||
}
|
||||
input = vrtPath
|
||||
}
|
||||
|
||||
template := strings.TrimSpace(s.cfg.TileJobCommand)
|
||||
if template == "" {
|
||||
template = defaultTileJobCommand
|
||||
}
|
||||
resolved := tileJobCommandArgs(template, job, input)
|
||||
if len(resolved) == 0 {
|
||||
return logPath, nil, errors.New("tile job command template is empty")
|
||||
}
|
||||
|
||||
command := exec.CommandContext(ctx, resolved[0], resolved[1:]...)
|
||||
command.Stdout = logFile
|
||||
command.Stderr = logFile
|
||||
if err := command.Run(); err != nil {
|
||||
return logPath, resolved, fmt.Errorf("%s: %w", strings.Join(resolved, " "), err)
|
||||
}
|
||||
return logPath, resolved, nil
|
||||
}
|
||||
|
||||
// buildTileJobMosaic runs gdalbuildvrt over the job's sources.
|
||||
func (s *Store) buildTileJobMosaic(ctx context.Context, job TileJob, vrtPath string, logFile *os.File) error {
|
||||
template := strings.TrimSpace(s.cfg.TileJobVRTCommand)
|
||||
if template == "" {
|
||||
template = defaultTileJobVRTCommand
|
||||
}
|
||||
resolved := tileJobVRTCommandArgs(template, job, vrtPath)
|
||||
if len(resolved) == 0 {
|
||||
return errors.New("tile job vrt command template is empty")
|
||||
}
|
||||
command := exec.CommandContext(ctx, resolved[0], resolved[1:]...)
|
||||
command.Stdout = logFile
|
||||
command.Stderr = logFile
|
||||
if err := command.Run(); err != nil {
|
||||
return fmt.Errorf("build mosaic: %s: %w", strings.Join(resolved, " "), err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// resolveTileJobSources validates every submitted source and returns absolute
|
||||
// paths. SourcePaths wins over the single SourcePath field.
|
||||
func resolveTileJobSources(input CreateTileJobInput) ([]string, error) {
|
||||
raw := input.SourcePaths
|
||||
if len(raw) == 0 && strings.TrimSpace(input.SourcePath) != "" {
|
||||
raw = []string{input.SourcePath}
|
||||
}
|
||||
if len(raw) == 0 {
|
||||
return nil, errors.New("source path is required")
|
||||
}
|
||||
|
||||
seen := map[string]struct{}{}
|
||||
sources := make([]string, 0, len(raw))
|
||||
for _, candidate := range raw {
|
||||
candidate = strings.TrimSpace(candidate)
|
||||
if candidate == "" {
|
||||
continue
|
||||
}
|
||||
abs, err := filepath.Abs(candidate)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("resolve source path %q: %w", candidate, err)
|
||||
}
|
||||
info, err := os.Stat(abs)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("source path %q is not readable: %w", candidate, err)
|
||||
}
|
||||
if info.IsDir() {
|
||||
return nil, fmt.Errorf("source path %q must be a file, not a directory", candidate)
|
||||
}
|
||||
if _, dup := seen[abs]; dup {
|
||||
continue
|
||||
}
|
||||
seen[abs] = struct{}{}
|
||||
sources = append(sources, abs)
|
||||
}
|
||||
if len(sources) == 0 {
|
||||
return nil, errors.New("source path is required")
|
||||
}
|
||||
return sources, nil
|
||||
}
|
||||
|
||||
// tileJobCommandArgs resolves the slicer template. Quoted segments are kept
|
||||
// together so paths with spaces survive.
|
||||
func tileJobCommandArgs(template string, job TileJob, input string) []string {
|
||||
return expandTileJobTemplate(template, map[string]string{
|
||||
"{input}": input,
|
||||
"{output}": job.OutputRoot,
|
||||
"{zmin}": fmt.Sprintf("%d", job.ZoomMin),
|
||||
"{zmax}": fmt.Sprintf("%d", job.ZoomMax),
|
||||
"{tilesize}": fmt.Sprintf("%d", job.TileSize),
|
||||
"{format}": job.TileFormat,
|
||||
"{code}": job.TargetCode,
|
||||
}, nil)
|
||||
}
|
||||
|
||||
// tileJobVRTCommandArgs resolves the mosaic template. {inputs} expands into one
|
||||
// argument per source raster.
|
||||
func tileJobVRTCommandArgs(template string, job TileJob, vrtPath string) []string {
|
||||
return expandTileJobTemplate(template, map[string]string{
|
||||
"{output_vrt}": vrtPath,
|
||||
"{output}": job.OutputRoot,
|
||||
"{code}": job.TargetCode,
|
||||
}, map[string][]string{"{inputs}": job.SourcePaths})
|
||||
}
|
||||
|
||||
// expandTileJobTemplate splits the template and substitutes placeholders. A
|
||||
// token that is exactly a list placeholder expands into several arguments.
|
||||
func expandTileJobTemplate(template string, scalars map[string]string, lists map[string][]string) []string {
|
||||
parts := splitCommandLine(template)
|
||||
args := make([]string, 0, len(parts))
|
||||
for _, part := range parts {
|
||||
if values, ok := lists[part]; ok {
|
||||
args = append(args, values...)
|
||||
continue
|
||||
}
|
||||
for key, value := range scalars {
|
||||
part = strings.ReplaceAll(part, key, value)
|
||||
}
|
||||
args = append(args, part)
|
||||
}
|
||||
return args
|
||||
}
|
||||
|
||||
// splitCommandLine splits on whitespace while honouring double quotes.
|
||||
func splitCommandLine(value string) []string {
|
||||
var parts []string
|
||||
var current strings.Builder
|
||||
inQuotes := false
|
||||
for _, r := range value {
|
||||
switch {
|
||||
case r == '"':
|
||||
inQuotes = !inQuotes
|
||||
case (r == ' ' || r == '\t' || r == '\n' || r == '\r') && !inQuotes:
|
||||
if current.Len() > 0 {
|
||||
parts = append(parts, current.String())
|
||||
current.Reset()
|
||||
}
|
||||
default:
|
||||
current.WriteRune(r)
|
||||
}
|
||||
}
|
||||
if current.Len() > 0 {
|
||||
parts = append(parts, current.String())
|
||||
}
|
||||
return parts
|
||||
}
|
||||
|
||||
// dispatchTileJobs runs queued jobs one at a time, oldest first.
|
||||
func (s *Store) dispatchTileJobs(ctx context.Context) {
|
||||
for {
|
||||
job, ok, err := s.nextPendingTileJob(ctx)
|
||||
if err != nil || !ok {
|
||||
return
|
||||
}
|
||||
s.runTileJob(ctx, job)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Store) nextPendingTileJob(ctx context.Context) (TileJob, bool, error) {
|
||||
row := s.db.QueryRowContext(ctx, "SELECT "+tileJobColumns+`
|
||||
FROM tile_jobs
|
||||
WHERE status = ?
|
||||
ORDER BY created_at
|
||||
LIMIT 1
|
||||
`, tileJobPending)
|
||||
job, err := scanTileJob(row)
|
||||
if err != nil {
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return TileJob{}, false, nil
|
||||
}
|
||||
return TileJob{}, false, err
|
||||
}
|
||||
return job, true, nil
|
||||
}
|
||||
|
||||
// Callback delivery for tile jobs. A finished job can notify the submitting
|
||||
// system; failures back off on the same schedule as push callbacks.
|
||||
const tileCallbackMaxAttempts = 3
|
||||
|
||||
func (s *Store) queueTileJobCallback(ctx context.Context, jobID string) {
|
||||
if _, err := s.db.ExecContext(ctx, `
|
||||
UPDATE tile_jobs
|
||||
SET callback_status = ?, callback_attempts = 0, callback_next_attempt_at = ?, callback_message = ''
|
||||
WHERE id = ?
|
||||
`, pushStatusRetrying, toRFC3339(nowUTC()), jobID); err != nil {
|
||||
log.Printf("queue tile job callback %s: %v", jobID, err)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Store) dispatchTileJobCallbacks(ctx context.Context) error {
|
||||
rows, err := s.db.QueryContext(ctx, "SELECT "+tileJobColumns+`
|
||||
FROM tile_jobs
|
||||
WHERE callback_url <> '' AND callback_status = ?
|
||||
AND (callback_next_attempt_at IS NULL OR callback_next_attempt_at <= ?)
|
||||
ORDER BY callback_next_attempt_at
|
||||
LIMIT 5
|
||||
`, pushStatusRetrying, toRFC3339(nowUTC()))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var jobs []TileJob
|
||||
for rows.Next() {
|
||||
job, err := scanTileJob(rows)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
jobs = append(jobs, job)
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, job := range jobs {
|
||||
s.attemptTileJobCallback(job, job.CallbackAttempts+1)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *Store) attemptTileJobCallback(job TileJob, attempt int) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), pushCallbackTimeout)
|
||||
defer cancel()
|
||||
|
||||
payload := map[string]any{
|
||||
"event": "tile.job.finished",
|
||||
"job": job,
|
||||
"delivered": toRFC3339(nowUTC()),
|
||||
}
|
||||
_, _, err := s.postJSON(ctx, http.MethodPost, job.CallbackURL, nil, payload)
|
||||
message := ""
|
||||
if err != nil {
|
||||
message = err.Error()
|
||||
}
|
||||
|
||||
if err != nil && attempt < tileCallbackMaxAttempts {
|
||||
next := nowUTC().Add(pushRetryDelay(attempt))
|
||||
if _, updateErr := s.db.ExecContext(context.Background(), `
|
||||
UPDATE tile_jobs
|
||||
SET callback_status = ?, callback_attempts = ?, callback_next_attempt_at = ?, callback_message = ?
|
||||
WHERE id = ?
|
||||
`, pushStatusRetrying, attempt, toRFC3339(next), message, job.ID); updateErr != nil {
|
||||
log.Printf("reschedule tile job callback %s: %v", job.ID, updateErr)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
status := pushStatusSucceeded
|
||||
if err != nil {
|
||||
status = pushStatusFailed
|
||||
}
|
||||
if _, updateErr := s.db.ExecContext(context.Background(), `
|
||||
UPDATE tile_jobs
|
||||
SET callback_status = ?, callback_attempts = ?, callback_next_attempt_at = NULL, callback_message = ?
|
||||
WHERE id = ?
|
||||
`, status, attempt, message, job.ID); updateErr != nil {
|
||||
log.Printf("record tile job callback %s: %v", job.ID, updateErr)
|
||||
}
|
||||
}
|
||||
|
||||
// LayerSource records one raster that belongs to a layer. The layer's tile
|
||||
// directory is the union of its sources, so adding a source and re-slicing
|
||||
// accumulates imagery instead of replacing the layer.
|
||||
type LayerSource struct {
|
||||
ID string `json:"id"`
|
||||
TargetCode string `json:"target_code"`
|
||||
SourcePath string `json:"source_path"`
|
||||
AddedAt time.Time `json:"added_at"`
|
||||
}
|
||||
|
||||
// AddLayerSources records sources against a layer. Re-adding a known path is a
|
||||
// no-op, which keeps repeated submissions idempotent.
|
||||
func (s *Store) AddLayerSources(ctx context.Context, targetCode string, paths []string) error {
|
||||
code := normalizeBasemapCode(targetCode)
|
||||
if code == "" {
|
||||
return errors.New("target code is required")
|
||||
}
|
||||
for _, path := range paths {
|
||||
path = strings.TrimSpace(path)
|
||||
if path == "" {
|
||||
continue
|
||||
}
|
||||
if _, err := s.db.ExecContext(ctx, `
|
||||
INSERT INTO layer_sources (id, target_code, source_path, added_at)
|
||||
VALUES (?, ?, ?, ?)
|
||||
ON CONFLICT(target_code, source_path) DO NOTHING
|
||||
`, uid.New(), code, path, toRFC3339(nowUTC())); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ListLayerSources returns a layer's source inventory in the order it was added.
|
||||
func (s *Store) ListLayerSources(ctx context.Context, targetCode string) ([]LayerSource, error) {
|
||||
rows, err := s.db.QueryContext(ctx, `
|
||||
SELECT id, target_code, source_path, added_at
|
||||
FROM layer_sources
|
||||
WHERE target_code = ?
|
||||
ORDER BY added_at
|
||||
`, normalizeBasemapCode(targetCode))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var items []LayerSource
|
||||
for rows.Next() {
|
||||
var item LayerSource
|
||||
var addedAt string
|
||||
if err := rows.Scan(&item.ID, &item.TargetCode, &item.SourcePath, &addedAt); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
item.AddedAt = parseTime(addedAt)
|
||||
items = append(items, item)
|
||||
}
|
||||
return items, rows.Err()
|
||||
}
|
||||
|
||||
// ListLayerInventories returns every layer with its sources, ordered by layer
|
||||
// code so the console renders stably.
|
||||
func (s *Store) ListLayerInventories(ctx context.Context) ([]LayerInventory, error) {
|
||||
rows, err := s.db.QueryContext(ctx, `
|
||||
SELECT id, target_code, source_path, added_at
|
||||
FROM layer_sources
|
||||
ORDER BY target_code, added_at
|
||||
`)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
byCode := map[string][]LayerSource{}
|
||||
var codes []string
|
||||
for rows.Next() {
|
||||
var item LayerSource
|
||||
var addedAt string
|
||||
if err := rows.Scan(&item.ID, &item.TargetCode, &item.SourcePath, &addedAt); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
item.AddedAt = parseTime(addedAt)
|
||||
if _, seen := byCode[item.TargetCode]; !seen {
|
||||
codes = append(codes, item.TargetCode)
|
||||
}
|
||||
byCode[item.TargetCode] = append(byCode[item.TargetCode], item)
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
items := make([]LayerInventory, 0, len(codes))
|
||||
for _, code := range codes {
|
||||
item := LayerInventory{TargetCode: code, Sources: byCode[code], ZoomMin: 0, ZoomMax: 18, TileFormat: "webp"}
|
||||
if minZoom, maxZoom, format, ok := s.layerPublishedRange(ctx, code); ok {
|
||||
item.ZoomMin, item.ZoomMax, item.TileFormat = minZoom, maxZoom, format
|
||||
}
|
||||
items = append(items, item)
|
||||
}
|
||||
return items, nil
|
||||
}
|
||||
|
||||
// layerPublishedRange returns the zoom range and format the layer is currently
|
||||
// published at, so a rebuild keeps the same range instead of guessing.
|
||||
func (s *Store) layerPublishedRange(ctx context.Context, targetCode string) (int, int, string, bool) {
|
||||
var minZoom, maxZoom int
|
||||
var format string
|
||||
err := s.db.QueryRowContext(ctx, `
|
||||
SELECT v.min_zoom, v.max_zoom, v.tile_format
|
||||
FROM basemap_versions v
|
||||
JOIN basemaps b ON b.id = v.basemap_id
|
||||
WHERE b.code = ?
|
||||
ORDER BY v.is_default DESC, v.version
|
||||
LIMIT 1
|
||||
`, normalizeBasemapCode(targetCode)).Scan(&minZoom, &maxZoom, &format)
|
||||
if err != nil {
|
||||
return 0, 0, "", false
|
||||
}
|
||||
return minZoom, maxZoom, format, true
|
||||
}
|
||||
@@ -65,6 +65,7 @@ func scanVectorPushRecord(scanner interface {
|
||||
var item VectorPushRecord
|
||||
var requestJSON string
|
||||
var responseStatus sql.NullInt64
|
||||
var nextAttemptAt sql.NullString
|
||||
var finishedAt sql.NullString
|
||||
var pushedAt string
|
||||
if err := scanner.Scan(
|
||||
@@ -78,6 +79,8 @@ func scanVectorPushRecord(scanner interface {
|
||||
&responseStatus,
|
||||
&item.ResponseBody,
|
||||
&item.ErrorMessage,
|
||||
&item.AttemptCount,
|
||||
&nextAttemptAt,
|
||||
&pushedAt,
|
||||
&finishedAt,
|
||||
); err != nil {
|
||||
@@ -93,6 +96,10 @@ func scanVectorPushRecord(scanner interface {
|
||||
value := int(responseStatus.Int64)
|
||||
item.ResponseStatus = &value
|
||||
}
|
||||
if nextAttemptAt.Valid {
|
||||
value := parseTime(nextAttemptAt.String)
|
||||
item.NextAttemptAt = &value
|
||||
}
|
||||
item.PushedAt = parseTime(pushedAt)
|
||||
if finishedAt.Valid {
|
||||
value := parseTime(finishedAt.String)
|
||||
|
||||
@@ -18,7 +18,7 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"map-asset-gateway/api-go/internal/uid"
|
||||
"tile-server/api-go/internal/uid"
|
||||
)
|
||||
|
||||
type VectorDownloadDescriptor struct {
|
||||
@@ -73,6 +73,20 @@ func (s *Store) GetVectorAssetByRef(ctx context.Context, ref string) (VectorAsse
|
||||
if err != nil {
|
||||
return VectorAsset{}, err
|
||||
}
|
||||
|
||||
// The compatibility routes resolve by ref on every request, so try the
|
||||
// indexed single-row lookups before falling back to a full table scan.
|
||||
if item, ok, err := s.getVectorAssetByLookup(ctx, "code = ?", normalizeBasemapCode(cleaned)); err != nil {
|
||||
return VectorAsset{}, err
|
||||
} else if ok {
|
||||
return item, nil
|
||||
}
|
||||
if item, ok, err := s.getVectorAssetByLookup(ctx, "file_name = ? COLLATE NOCASE", path.Base(cleaned)); err != nil {
|
||||
return VectorAsset{}, err
|
||||
} else if ok {
|
||||
return item, nil
|
||||
}
|
||||
|
||||
items, err := s.ListVectorAssets(ctx)
|
||||
if err != nil {
|
||||
return VectorAsset{}, err
|
||||
@@ -85,16 +99,49 @@ func (s *Store) GetVectorAssetByRef(ctx context.Context, ref string) (VectorAsse
|
||||
return VectorAsset{}, fmt.Errorf("vector asset %q not found", ref)
|
||||
}
|
||||
|
||||
// getVectorAssetByLookup runs a single indexed lookup. where must be a trusted
|
||||
// internal literal, never user input.
|
||||
func (s *Store) getVectorAssetByLookup(ctx context.Context, where, value string) (VectorAsset, bool, error) {
|
||||
value = strings.TrimSpace(value)
|
||||
if value == "" {
|
||||
return VectorAsset{}, false, nil
|
||||
}
|
||||
query := fmt.Sprintf(`
|
||||
SELECT id, code, name, format, status, description, file_path, file_name, file_size, checksum, download_url, metadata_json, created_at, updated_at
|
||||
FROM vector_assets
|
||||
WHERE %s
|
||||
LIMIT 1
|
||||
`, where)
|
||||
item, err := scanVectorAsset(s.db.QueryRowContext(ctx, query, value))
|
||||
if err != nil {
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return VectorAsset{}, false, nil
|
||||
}
|
||||
return VectorAsset{}, false, err
|
||||
}
|
||||
return item, true, nil
|
||||
}
|
||||
|
||||
func (s *Store) RunVectorScan(ctx context.Context) (map[string]any, error) {
|
||||
root := strings.TrimSpace(s.cfg.VectorScanRoot)
|
||||
if root == "" {
|
||||
return nil, nil
|
||||
}
|
||||
if !s.scanMu.TryLock() {
|
||||
return nil, ErrScanInProgress
|
||||
}
|
||||
defer s.scanMu.Unlock()
|
||||
|
||||
if err := os.MkdirAll(root, 0o755); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
items, err := scanVectorRoot(root)
|
||||
hashCache, err := s.loadVectorHashCache(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
items, err := scanVectorRoot(root, hashCache)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -105,32 +152,22 @@ func (s *Store) RunVectorScan(ctx context.Context) (map[string]any, error) {
|
||||
}
|
||||
defer tx.Rollback()
|
||||
|
||||
rows, err := tx.QueryContext(ctx, `SELECT code FROM vector_assets`)
|
||||
existing, err := loadExistingVectors(ctx, tx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
existing := map[string]struct{}{}
|
||||
for rows.Next() {
|
||||
var code string
|
||||
if err := rows.Scan(&code); err != nil {
|
||||
rows.Close()
|
||||
return nil, err
|
||||
}
|
||||
existing[code] = struct{}{}
|
||||
}
|
||||
rows.Close()
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
found := map[string]struct{}{}
|
||||
added := 0
|
||||
updated := 0
|
||||
added, updated, unchanged := 0, 0, 0
|
||||
now := nowUTC()
|
||||
for _, item := range items {
|
||||
found[item.Code] = struct{}{}
|
||||
if _, ok := existing[item.Code]; ok {
|
||||
updated++
|
||||
if prev, ok := existing[item.Code]; ok {
|
||||
if s.scannedVectorMatches(item, prev) {
|
||||
unchanged++
|
||||
} else {
|
||||
updated++
|
||||
}
|
||||
} else {
|
||||
added++
|
||||
}
|
||||
@@ -154,14 +191,117 @@ func (s *Store) RunVectorScan(ctx context.Context) (map[string]any, error) {
|
||||
return nil, err
|
||||
}
|
||||
return map[string]any{
|
||||
"scan_root": root,
|
||||
"scanned_count": len(items),
|
||||
"added_count": added,
|
||||
"updated_count": updated,
|
||||
"removed_count": removed,
|
||||
"scan_root": root,
|
||||
"scanned_count": len(items),
|
||||
"added_count": added,
|
||||
"updated_count": updated,
|
||||
"unchanged_count": unchanged,
|
||||
"removed_count": removed,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// existingVector is a snapshot of a stored asset, used to tell a real update
|
||||
// from a no-op rescan.
|
||||
type existingVector struct {
|
||||
Name string
|
||||
Format string
|
||||
Status string
|
||||
Description string
|
||||
FilePath string
|
||||
FileName string
|
||||
FileSize int64
|
||||
Checksum string
|
||||
DownloadURL string
|
||||
MetadataJSON string
|
||||
}
|
||||
|
||||
func loadExistingVectors(ctx context.Context, tx *sql.Tx) (map[string]existingVector, error) {
|
||||
rows, err := tx.QueryContext(ctx, `
|
||||
SELECT code, name, format, status, description, file_path, file_name, file_size, checksum, download_url, metadata_json
|
||||
FROM vector_assets
|
||||
`)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
items := map[string]existingVector{}
|
||||
for rows.Next() {
|
||||
var code string
|
||||
var item existingVector
|
||||
if err := rows.Scan(
|
||||
&code,
|
||||
&item.Name,
|
||||
&item.Format,
|
||||
&item.Status,
|
||||
&item.Description,
|
||||
&item.FilePath,
|
||||
&item.FileName,
|
||||
&item.FileSize,
|
||||
&item.Checksum,
|
||||
&item.DownloadURL,
|
||||
&item.MetadataJSON,
|
||||
); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
items[code] = item
|
||||
}
|
||||
return items, rows.Err()
|
||||
}
|
||||
|
||||
func (s *Store) scannedVectorMatches(item scannedVector, prev existingVector) bool {
|
||||
return item.Name == prev.Name &&
|
||||
item.Format == prev.Format &&
|
||||
item.Status == prev.Status &&
|
||||
item.Description == prev.Description &&
|
||||
item.FilePath == prev.FilePath &&
|
||||
item.FileName == prev.FileName &&
|
||||
item.FileSize == prev.FileSize &&
|
||||
item.Checksum == prev.Checksum &&
|
||||
s.buildVectorDownloadURL(item.Code) == prev.DownloadURL &&
|
||||
writeJSON(item.Metadata) == prev.MetadataJSON
|
||||
}
|
||||
|
||||
// vectorHashEntry lets an unchanged file reuse the checksum recorded by the
|
||||
// previous scan instead of re-reading the whole file on every startup.
|
||||
type vectorHashEntry struct {
|
||||
Size int64
|
||||
ModTime int64
|
||||
Checksum string
|
||||
}
|
||||
|
||||
func (s *Store) loadVectorHashCache(ctx context.Context) (map[string]vectorHashEntry, error) {
|
||||
rows, err := s.db.QueryContext(ctx, `
|
||||
SELECT file_path, checksum, metadata_json
|
||||
FROM vector_assets
|
||||
`)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
cache := map[string]vectorHashEntry{}
|
||||
for rows.Next() {
|
||||
var filePath, checksum, metadataJSON string
|
||||
if err := rows.Scan(&filePath, &checksum, &metadataJSON); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if strings.TrimSpace(checksum) == "" {
|
||||
continue
|
||||
}
|
||||
var meta struct {
|
||||
HashSize int64 `json:"hash_size"`
|
||||
HashMTime int64 `json:"hash_mtime"`
|
||||
}
|
||||
_ = readMaybeJSON(metadataJSON, &meta)
|
||||
if meta.HashSize == 0 && meta.HashMTime == 0 {
|
||||
continue
|
||||
}
|
||||
cache[filePath] = vectorHashEntry{Size: meta.HashSize, ModTime: meta.HashMTime, Checksum: checksum}
|
||||
}
|
||||
return cache, rows.Err()
|
||||
}
|
||||
|
||||
type scannedVector struct {
|
||||
Code string
|
||||
Name string
|
||||
@@ -175,7 +315,7 @@ type scannedVector struct {
|
||||
Metadata map[string]any
|
||||
}
|
||||
|
||||
func scanVectorRoot(root string) ([]scannedVector, error) {
|
||||
func scanVectorRoot(root string, hashCache map[string]vectorHashEntry) ([]scannedVector, error) {
|
||||
shapefiles := map[string]string{}
|
||||
var items []scannedVector
|
||||
err := filepath.WalkDir(root, func(path string, entry os.DirEntry, walkErr error) error {
|
||||
@@ -188,7 +328,7 @@ func scanVectorRoot(root string) ([]scannedVector, error) {
|
||||
ext := strings.ToLower(filepath.Ext(entry.Name()))
|
||||
switch ext {
|
||||
case ".geojson", ".json", ".zip":
|
||||
item, err := buildScannedVector(root, path, ext)
|
||||
item, err := buildScannedVector(root, path, ext, hashCache)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -205,19 +345,51 @@ func scanVectorRoot(root string) ([]scannedVector, error) {
|
||||
|
||||
for _, base := range sortedKeysStringMap(shapefiles) {
|
||||
path := shapefiles[base]
|
||||
item, err := buildScannedVector(root, path, ".shp")
|
||||
item, err := buildScannedVector(root, path, ".shp", hashCache)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
items = append(items, item)
|
||||
}
|
||||
|
||||
items = disambiguateVectorCodes(items)
|
||||
sort.Slice(items, func(i, j int) bool {
|
||||
return items[i].Code < items[j].Code
|
||||
})
|
||||
return items, nil
|
||||
}
|
||||
|
||||
// disambiguateVectorCodes keeps both assets when two different relative paths
|
||||
// normalize to the same code (for example "a/b.geojson" and "a-b.geojson").
|
||||
// The suffix comes from the relative path, so it is stable across scans.
|
||||
func disambiguateVectorCodes(items []scannedVector) []scannedVector {
|
||||
byCode := map[string][]int{}
|
||||
for i, item := range items {
|
||||
byCode[item.Code] = append(byCode[item.Code], i)
|
||||
}
|
||||
for code, indexes := range byCode {
|
||||
if len(indexes) < 2 {
|
||||
continue
|
||||
}
|
||||
for _, index := range indexes {
|
||||
sum := sha256.Sum256([]byte(scannedVectorRelPath(items[index])))
|
||||
items[index].Code = code + "-" + hex.EncodeToString(sum[:4])
|
||||
}
|
||||
}
|
||||
return items
|
||||
}
|
||||
|
||||
func scannedVectorRelPath(item scannedVector) string {
|
||||
if item.Metadata == nil {
|
||||
return ""
|
||||
}
|
||||
value, ok := item.Metadata["relative_path"].(string)
|
||||
if !ok {
|
||||
return ""
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
func sortedKeysStringMap(values map[string]string) []string {
|
||||
keys := make([]string, 0, len(values))
|
||||
for key := range values {
|
||||
@@ -227,7 +399,7 @@ func sortedKeysStringMap(values map[string]string) []string {
|
||||
return keys
|
||||
}
|
||||
|
||||
func buildScannedVector(root, path, ext string) (scannedVector, error) {
|
||||
func buildScannedVector(root, path, ext string, hashCache map[string]vectorHashEntry) (scannedVector, error) {
|
||||
absPath, err := filepath.Abs(path)
|
||||
if err != nil {
|
||||
return scannedVector{}, err
|
||||
@@ -245,10 +417,24 @@ func buildScannedVector(root, path, ext string) (scannedVector, error) {
|
||||
format := vectorFormatFromExt(ext)
|
||||
fileName := filepath.Base(absPath)
|
||||
fileSize := info.Size()
|
||||
checksum, err := hashVector(absPath, ext)
|
||||
|
||||
hashSize, hashMTime, err := vectorHashStamp(absPath, ext)
|
||||
if err != nil {
|
||||
return scannedVector{}, err
|
||||
}
|
||||
checksum := ""
|
||||
if cached, ok := hashCache[absPath]; ok &&
|
||||
(hashSize > 0 || hashMTime > 0) &&
|
||||
cached.Size == hashSize && cached.ModTime == hashMTime {
|
||||
checksum = cached.Checksum
|
||||
}
|
||||
if checksum == "" {
|
||||
checksum, err = hashVector(absPath, ext)
|
||||
if err != nil {
|
||||
return scannedVector{}, err
|
||||
}
|
||||
}
|
||||
|
||||
if format == "shapefile" {
|
||||
fileName = name + ".zip"
|
||||
}
|
||||
@@ -263,10 +449,37 @@ func buildScannedVector(root, path, ext string) (scannedVector, error) {
|
||||
Checksum: checksum,
|
||||
Metadata: map[string]any{
|
||||
"relative_path": filepath.ToSlash(rel),
|
||||
"hash_size": hashSize,
|
||||
"hash_mtime": hashMTime,
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
// vectorHashStamp summarizes the inputs that feed the checksum, so an unchanged
|
||||
// file can reuse the previous checksum instead of being rehashed every startup.
|
||||
// Shapefiles fold in every sidecar so a changed .dbf/.shx also invalidates it.
|
||||
func vectorHashStamp(absPath, ext string) (int64, int64, error) {
|
||||
if strings.EqualFold(ext, ".shp") {
|
||||
var totalSize, maxModTime int64
|
||||
for _, sibling := range shapefilePackageFiles(absPath) {
|
||||
info, err := os.Stat(sibling)
|
||||
if err != nil || info.IsDir() {
|
||||
continue
|
||||
}
|
||||
totalSize += info.Size()
|
||||
if modTime := info.ModTime().Unix(); modTime > maxModTime {
|
||||
maxModTime = modTime
|
||||
}
|
||||
}
|
||||
return totalSize, maxModTime, nil
|
||||
}
|
||||
info, err := os.Stat(absPath)
|
||||
if err != nil {
|
||||
return 0, 0, err
|
||||
}
|
||||
return info.Size(), info.ModTime().Unix(), nil
|
||||
}
|
||||
|
||||
func vectorCodeFromRel(rel, ext string) string {
|
||||
rel = filepath.ToSlash(rel)
|
||||
rel = strings.TrimSuffix(rel, strings.ToLower(filepath.Ext(rel)))
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
package basemap
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
// pushRetryInterval is how often due retries are looked for.
|
||||
pushRetryInterval = 15 * time.Second
|
||||
// tileJobInterval is how often the queue is checked for new jobs.
|
||||
tileJobInterval = 2 * time.Second
|
||||
)
|
||||
|
||||
// StartBackgroundWorkers launches the periodic maintenance loops: resource
|
||||
// health inspection, push retries and the tile slicing queue. Close stops them.
|
||||
// Calling it more than once has no additional effect.
|
||||
func (s *Store) StartBackgroundWorkers() {
|
||||
s.bgOnce.Do(func() {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
s.bgCtx = ctx
|
||||
s.bgCancel = cancel
|
||||
|
||||
s.bgWG.Add(1)
|
||||
go func() {
|
||||
defer s.bgWG.Done()
|
||||
s.healthLoop(ctx)
|
||||
}()
|
||||
|
||||
s.bgWG.Add(1)
|
||||
go func() {
|
||||
defer s.bgWG.Done()
|
||||
s.pushRetryLoop(ctx)
|
||||
}()
|
||||
|
||||
s.bgWG.Add(1)
|
||||
go func() {
|
||||
defer s.bgWG.Done()
|
||||
s.tileJobLoop(ctx)
|
||||
}()
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Store) healthLoop(ctx context.Context) {
|
||||
// Prime the catalog immediately so a resource that vanished since the last
|
||||
// scan shows up without waiting a full interval.
|
||||
if _, err := s.CheckHealth(ctx); err != nil && ctx.Err() == nil {
|
||||
log.Printf("health check failed: %v", err)
|
||||
}
|
||||
interval := s.cfg.HealthCheckInterval
|
||||
if interval <= 0 {
|
||||
return
|
||||
}
|
||||
ticker := time.NewTicker(interval)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
if _, err := s.CheckHealth(ctx); err != nil && ctx.Err() == nil {
|
||||
log.Printf("health check failed: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Store) pushRetryLoop(ctx context.Context) {
|
||||
ticker := time.NewTicker(pushRetryInterval)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
if err := s.dispatchPushRetries(ctx); err != nil && ctx.Err() == nil {
|
||||
log.Printf("push retry dispatch failed: %v", err)
|
||||
}
|
||||
if err := s.dispatchTileJobCallbacks(ctx); err != nil && ctx.Err() == nil {
|
||||
log.Printf("tile job callback dispatch failed: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Store) tileJobLoop(ctx context.Context) {
|
||||
ticker := time.NewTicker(tileJobInterval)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
s.dispatchTileJobs(ctx)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
}
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -9,7 +9,7 @@ import (
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"map-asset-gateway/api-go/internal/basemap"
|
||||
"tile-server/api-go/internal/basemap"
|
||||
)
|
||||
|
||||
func TestCompatibilityRoutes(t *testing.T) {
|
||||
@@ -65,7 +65,7 @@ func TestCompatibilityRoutes(t *testing.T) {
|
||||
}
|
||||
|
||||
server, err := New(store, basemap.SystemInfo{
|
||||
ServiceName: "map-asset-gateway",
|
||||
ServiceName: "basemap-pusher",
|
||||
ServiceVersion: "test",
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
TileBaseURL: "http://127.0.0.1:8910",
|
||||
@@ -73,7 +73,7 @@ func TestCompatibilityRoutes(t *testing.T) {
|
||||
SQLitePath: sqlitePath,
|
||||
ScanRoot: tileRoot,
|
||||
VectorScanRoot: vectorRoot,
|
||||
})
|
||||
}, Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("new server: %v", err)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,279 @@
|
||||
package basemapapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"tile-server/api-go/internal/basemap"
|
||||
)
|
||||
|
||||
func newTestStore(t *testing.T, root string) *basemap.Store {
|
||||
t.Helper()
|
||||
tileRoot := filepath.Join(root, "tiles")
|
||||
writeFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile")
|
||||
vectorRoot := filepath.Join(root, "vectors")
|
||||
|
||||
store, err := basemap.OpenStore(basemap.StoreConfig{
|
||||
SQLitePath: filepath.Join(root, "basemap.db"),
|
||||
ScanRoot: tileRoot,
|
||||
VectorScanRoot: vectorRoot,
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
TileBaseURL: "http://127.0.0.1:8910",
|
||||
VectorBaseURL: "http://127.0.0.1:8910",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("open store: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = store.Close() })
|
||||
|
||||
if _, err := store.RunScan(context.Background(), "local-basemaps"); err != nil {
|
||||
t.Fatalf("run scan: %v", err)
|
||||
}
|
||||
return store
|
||||
}
|
||||
|
||||
func writeFile(t *testing.T, path, body string) {
|
||||
t.Helper()
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
t.Fatalf("mkdir: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(path, []byte(body), 0o644); err != nil {
|
||||
t.Fatalf("write file: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func newTestServer(t *testing.T, store *basemap.Store, options Options) *httptest.Server {
|
||||
t.Helper()
|
||||
server, err := New(store, basemap.SystemInfo{ServiceName: "basemap-pusher"}, options)
|
||||
if err != nil {
|
||||
t.Fatalf("new server: %v", err)
|
||||
}
|
||||
httpServer := httptest.NewServer(server.Handler())
|
||||
t.Cleanup(httpServer.Close)
|
||||
return httpServer
|
||||
}
|
||||
|
||||
func basicAuthHeader(user, password string) string {
|
||||
return "Basic " + base64.StdEncoding.EncodeToString([]byte(user+":"+password))
|
||||
}
|
||||
|
||||
func TestAdminRequiresBasicAuth(t *testing.T) {
|
||||
store := newTestStore(t, t.TempDir())
|
||||
httpServer := newTestServer(t, store, Options{Admin: AdminAuth{User: "ops", Password: "s3cret"}})
|
||||
|
||||
for _, path := range []string{"/api/v1/admin/basemaps", "/admin/"} {
|
||||
response, err := http.Get(httpServer.URL + path)
|
||||
if err != nil {
|
||||
t.Fatalf("unauthorized GET %s: %v", path, err)
|
||||
}
|
||||
response.Body.Close()
|
||||
if response.StatusCode != http.StatusUnauthorized {
|
||||
t.Fatalf("expected 401 for anonymous %s, got %d", path, response.StatusCode)
|
||||
}
|
||||
|
||||
request, err := http.NewRequest(http.MethodGet, httpServer.URL+path, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("build request: %v", err)
|
||||
}
|
||||
request.Header.Set("Authorization", basicAuthHeader("ops", "s3cret"))
|
||||
response, err = http.DefaultClient.Do(request)
|
||||
if err != nil {
|
||||
t.Fatalf("authorized GET %s: %v", path, err)
|
||||
}
|
||||
response.Body.Close()
|
||||
if response.StatusCode != http.StatusOK {
|
||||
t.Fatalf("expected 200 for authorized %s, got %d", path, response.StatusCode)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdminRejectsWrongBasicAuth(t *testing.T) {
|
||||
store := newTestStore(t, t.TempDir())
|
||||
httpServer := newTestServer(t, store, Options{Admin: AdminAuth{User: "ops", Password: "s3cret"}})
|
||||
|
||||
request, _ := http.NewRequest(http.MethodGet, httpServer.URL+"/api/v1/admin/basemaps", nil)
|
||||
request.Header.Set("Authorization", basicAuthHeader("ops", "wrong"))
|
||||
response, err := http.DefaultClient.Do(request)
|
||||
if err != nil {
|
||||
t.Fatalf("request: %v", err)
|
||||
}
|
||||
defer response.Body.Close()
|
||||
if response.StatusCode != http.StatusUnauthorized {
|
||||
t.Fatalf("expected 401, got %d", response.StatusCode)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdminAcceptsBearerToken(t *testing.T) {
|
||||
store := newTestStore(t, t.TempDir())
|
||||
httpServer := newTestServer(t, store, Options{Admin: AdminAuth{Token: "admin-token"}})
|
||||
|
||||
request, _ := http.NewRequest(http.MethodGet, httpServer.URL+"/api/v1/admin/basemaps", nil)
|
||||
request.Header.Set("Authorization", "Bearer admin-token")
|
||||
response, err := http.DefaultClient.Do(request)
|
||||
if err != nil {
|
||||
t.Fatalf("request: %v", err)
|
||||
}
|
||||
response.Body.Close()
|
||||
if response.StatusCode != http.StatusOK {
|
||||
t.Fatalf("expected 200 with bearer token, got %d", response.StatusCode)
|
||||
}
|
||||
|
||||
request, _ = http.NewRequest(http.MethodGet, httpServer.URL+"/api/v1/admin/basemaps", nil)
|
||||
request.Header.Set("Authorization", "Bearer nope")
|
||||
response, err = http.DefaultClient.Do(request)
|
||||
if err != nil {
|
||||
t.Fatalf("request: %v", err)
|
||||
}
|
||||
response.Body.Close()
|
||||
if response.StatusCode != http.StatusUnauthorized {
|
||||
t.Fatalf("expected 401 with a wrong bearer token, got %d", response.StatusCode)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdminOpenWhenUnconfigured(t *testing.T) {
|
||||
store := newTestStore(t, t.TempDir())
|
||||
httpServer := newTestServer(t, store, Options{})
|
||||
|
||||
response, err := http.Get(httpServer.URL + "/api/v1/admin/basemaps")
|
||||
if err != nil {
|
||||
t.Fatalf("request: %v", err)
|
||||
}
|
||||
defer response.Body.Close()
|
||||
if response.StatusCode != http.StatusOK {
|
||||
t.Fatalf("expected the admin API to be open without credentials, got %d", response.StatusCode)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCatalogStillRequiresTokenEvenWithAdminAuth(t *testing.T) {
|
||||
store := newTestStore(t, t.TempDir())
|
||||
httpServer := newTestServer(t, store, Options{Admin: AdminAuth{User: "ops", Password: "s3cret"}})
|
||||
|
||||
response, err := http.Get(httpServer.URL + "/api/v1/catalog/basemaps")
|
||||
if err != nil {
|
||||
t.Fatalf("request: %v", err)
|
||||
}
|
||||
defer response.Body.Close()
|
||||
if response.StatusCode != http.StatusUnauthorized {
|
||||
t.Fatalf("expected the catalog to still require a service token, got %d", response.StatusCode)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCORSAllowlist(t *testing.T) {
|
||||
store := newTestStore(t, t.TempDir())
|
||||
httpServer := newTestServer(t, store, Options{CORSAllowedOrigins: []string{"https://maps.example.com"}})
|
||||
|
||||
allowed, _ := http.NewRequest(http.MethodGet, httpServer.URL+"/health", nil)
|
||||
allowed.Header.Set("Origin", "https://maps.example.com")
|
||||
response, err := http.DefaultClient.Do(allowed)
|
||||
if err != nil {
|
||||
t.Fatalf("allowed origin: %v", err)
|
||||
}
|
||||
response.Body.Close()
|
||||
if got := response.Header.Get("Access-Control-Allow-Origin"); got != "https://maps.example.com" {
|
||||
t.Fatalf("expected the allowlisted origin to be reflected, got %q", got)
|
||||
}
|
||||
|
||||
blocked, _ := http.NewRequest(http.MethodGet, httpServer.URL+"/health", nil)
|
||||
blocked.Header.Set("Origin", "https://evil.example.com")
|
||||
response, err = http.DefaultClient.Do(blocked)
|
||||
if err != nil {
|
||||
t.Fatalf("blocked origin: %v", err)
|
||||
}
|
||||
response.Body.Close()
|
||||
if got := response.Header.Get("Access-Control-Allow-Origin"); got != "" {
|
||||
t.Fatalf("expected no CORS header for an unlisted origin, got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateTokenFormKeepsTokenOutOfURL(t *testing.T) {
|
||||
store := newTestStore(t, t.TempDir())
|
||||
httpServer := newTestServer(t, store, Options{})
|
||||
|
||||
form := url.Values{}
|
||||
form.Set("name", "console")
|
||||
form.Set("basemap_codes", "google_image")
|
||||
request, err := http.NewRequest(http.MethodPost, httpServer.URL+"/admin/tokens", strings.NewReader(form.Encode()))
|
||||
if err != nil {
|
||||
t.Fatalf("build request: %v", err)
|
||||
}
|
||||
request.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
|
||||
client := &http.Client{CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}
|
||||
response, err := client.Do(request)
|
||||
if err != nil {
|
||||
t.Fatalf("post form: %v", err)
|
||||
}
|
||||
defer response.Body.Close()
|
||||
body, _ := io.ReadAll(response.Body)
|
||||
|
||||
if location := response.Header.Get("Location"); strings.Contains(location, "bmp_") {
|
||||
t.Fatalf("token leaked into the redirect URL: %s", location)
|
||||
}
|
||||
if !strings.Contains(string(body), "bmp_") {
|
||||
t.Fatalf("expected the new token to be rendered in the response body")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetricsEndpointCountsTileMisses(t *testing.T) {
|
||||
store := newTestStore(t, t.TempDir())
|
||||
httpServer := newTestServer(t, store, Options{})
|
||||
|
||||
created, err := store.CreateToken(context.Background(), basemap.CreateTokenInput{
|
||||
Name: "metrics",
|
||||
BasemapCodes: []string{"google_image"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("create token: %v", err)
|
||||
}
|
||||
|
||||
response, err := http.Get(httpServer.URL + "/tiles/google_image/current/9/9/9.png?token=" + created.Token)
|
||||
if err != nil {
|
||||
t.Fatalf("tile request: %v", err)
|
||||
}
|
||||
response.Body.Close()
|
||||
if response.StatusCode != http.StatusNotFound {
|
||||
t.Fatalf("expected 404 for a missing tile, got %d", response.StatusCode)
|
||||
}
|
||||
|
||||
response, err = http.Get(httpServer.URL + "/metrics")
|
||||
if err != nil {
|
||||
t.Fatalf("metrics request: %v", err)
|
||||
}
|
||||
defer response.Body.Close()
|
||||
body, _ := io.ReadAll(response.Body)
|
||||
text := string(body)
|
||||
for _, expected := range []string{
|
||||
"basemap_pusher_requests_total",
|
||||
"basemap_pusher_tile_requests_total 1",
|
||||
"basemap_pusher_tile_misses_total 1",
|
||||
} {
|
||||
if !strings.Contains(text, expected) {
|
||||
t.Fatalf("expected /metrics to contain %q, got:\n%s", expected, text)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestScanErrorStatusMapping(t *testing.T) {
|
||||
cases := []struct {
|
||||
err error
|
||||
want int
|
||||
}{
|
||||
{basemap.ErrScanInProgress, http.StatusConflict},
|
||||
{basemap.ErrScanSourceNotFound, http.StatusNotFound},
|
||||
{basemap.ErrScanSourceDisabled, http.StatusBadRequest},
|
||||
{io.ErrUnexpectedEOF, http.StatusInternalServerError},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
if got := scanErrorStatus(tc.err); got != tc.want {
|
||||
t.Fatalf("scanErrorStatus(%v) = %d, want %d", tc.err, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,29 +1,63 @@
|
||||
package basemapapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/subtle"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"html/template"
|
||||
"log"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"map-asset-gateway/api-go/internal/basemap"
|
||||
"tile-server/api-go/internal/basemap"
|
||||
)
|
||||
|
||||
type Server struct {
|
||||
store *basemap.Store
|
||||
system basemap.SystemInfo
|
||||
startedAt time.Time
|
||||
template *template.Template
|
||||
mux *http.ServeMux
|
||||
// AdminAuth protects the admin console and admin API. When neither credential
|
||||
// is configured the admin surface is only safe on loopback, which main enforces.
|
||||
type AdminAuth struct {
|
||||
Token string
|
||||
User string
|
||||
Password string
|
||||
}
|
||||
|
||||
func New(store *basemap.Store, system basemap.SystemInfo) (*Server, error) {
|
||||
func (a AdminAuth) enabled() bool {
|
||||
return strings.TrimSpace(a.Token) != "" || (strings.TrimSpace(a.User) != "" && a.Password != "")
|
||||
}
|
||||
|
||||
// Options carries wiring that must not be exposed through the public system
|
||||
// info payload (credentials, CORS policy).
|
||||
type Options struct {
|
||||
Admin AdminAuth
|
||||
CORSAllowedOrigins []string
|
||||
}
|
||||
|
||||
// metrics holds cheap in-process counters for the /metrics endpoint.
|
||||
type metrics struct {
|
||||
requests atomic.Int64
|
||||
status2xx atomic.Int64
|
||||
status4xx atomic.Int64
|
||||
status5xx atomic.Int64
|
||||
tiles atomic.Int64
|
||||
tileMisses atomic.Int64
|
||||
authFailures atomic.Int64
|
||||
}
|
||||
type Server struct {
|
||||
store *basemap.Store
|
||||
system basemap.SystemInfo
|
||||
startedAt time.Time
|
||||
template *template.Template
|
||||
mux *http.ServeMux
|
||||
admin AdminAuth
|
||||
corsOrigins []string
|
||||
metrics *metrics
|
||||
}
|
||||
|
||||
func New(store *basemap.Store, system basemap.SystemInfo, options Options) (*Server, error) {
|
||||
tmpl, err := template.New("admin").Funcs(template.FuncMap{
|
||||
"fmtTime": func(value time.Time) string {
|
||||
if value.IsZero() {
|
||||
@@ -106,18 +140,143 @@ func New(store *basemap.Store, system basemap.SystemInfo) (*Server, error) {
|
||||
}
|
||||
|
||||
server := &Server{
|
||||
store: store,
|
||||
system: system,
|
||||
startedAt: time.Now().UTC(),
|
||||
template: tmpl,
|
||||
mux: http.NewServeMux(),
|
||||
store: store,
|
||||
system: system,
|
||||
startedAt: time.Now().UTC(),
|
||||
template: tmpl,
|
||||
mux: http.NewServeMux(),
|
||||
admin: options.Admin,
|
||||
corsOrigins: normalizeOrigins(options.CORSAllowedOrigins),
|
||||
metrics: &metrics{},
|
||||
}
|
||||
server.routes()
|
||||
return server, nil
|
||||
}
|
||||
|
||||
func (s *Server) Handler() http.Handler {
|
||||
return s.mux
|
||||
return withAccessLog(withCORS(s.mux, s.corsOrigins), s.metrics)
|
||||
}
|
||||
|
||||
func normalizeOrigins(values []string) []string {
|
||||
origins := make([]string, 0, len(values))
|
||||
for _, value := range values {
|
||||
value = strings.TrimSpace(value)
|
||||
if value != "" {
|
||||
origins = append(origins, value)
|
||||
}
|
||||
}
|
||||
return origins
|
||||
}
|
||||
|
||||
// withCORS only reflects origins that were explicitly allowed. An empty
|
||||
// allow-list means no cross-origin access at all, rather than echoing any site.
|
||||
func withCORS(next http.Handler, allowed []string) http.Handler {
|
||||
allowAll := false
|
||||
allowedSet := map[string]struct{}{}
|
||||
for _, origin := range allowed {
|
||||
if origin == "*" {
|
||||
allowAll = true
|
||||
}
|
||||
allowedSet[strings.ToLower(origin)] = struct{}{}
|
||||
}
|
||||
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
origin := strings.TrimSpace(r.Header.Get("Origin"))
|
||||
if origin != "" {
|
||||
_, ok := allowedSet[strings.ToLower(origin)]
|
||||
if ok || allowAll {
|
||||
w.Header().Set("Access-Control-Allow-Origin", origin)
|
||||
w.Header().Set("Vary", "Origin")
|
||||
w.Header().Set("Access-Control-Allow-Headers", "Authorization, Content-Type, X-Access-Token")
|
||||
w.Header().Set("Access-Control-Allow-Methods", "GET, POST, OPTIONS")
|
||||
}
|
||||
}
|
||||
if r.Method == http.MethodOptions {
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
return
|
||||
}
|
||||
next.ServeHTTP(w, r)
|
||||
})
|
||||
}
|
||||
|
||||
type statusRecorder struct {
|
||||
http.ResponseWriter
|
||||
status int
|
||||
bytes int
|
||||
}
|
||||
|
||||
func (r *statusRecorder) WriteHeader(code int) {
|
||||
if r.status == 0 {
|
||||
r.status = code
|
||||
}
|
||||
r.ResponseWriter.WriteHeader(code)
|
||||
}
|
||||
|
||||
func (r *statusRecorder) Write(p []byte) (int, error) {
|
||||
if r.status == 0 {
|
||||
r.status = http.StatusOK
|
||||
}
|
||||
n, err := r.ResponseWriter.Write(p)
|
||||
r.bytes += n
|
||||
return n, err
|
||||
}
|
||||
|
||||
// withAccessLog records one line per request and feeds /metrics. Credentials
|
||||
// carried in the query string are redacted before logging.
|
||||
func withAccessLog(next http.Handler, stats *metrics) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
startedAt := time.Now()
|
||||
recorder := &statusRecorder{ResponseWriter: w}
|
||||
next.ServeHTTP(recorder, r)
|
||||
if recorder.status == 0 {
|
||||
recorder.status = http.StatusOK
|
||||
}
|
||||
|
||||
stats.requests.Add(1)
|
||||
switch {
|
||||
case recorder.status >= 500:
|
||||
stats.status5xx.Add(1)
|
||||
case recorder.status >= 400:
|
||||
stats.status4xx.Add(1)
|
||||
case recorder.status >= 200:
|
||||
stats.status2xx.Add(1)
|
||||
}
|
||||
if strings.HasPrefix(r.URL.Path, "/tiles/") {
|
||||
stats.tiles.Add(1)
|
||||
if recorder.status == http.StatusNotFound {
|
||||
stats.tileMisses.Add(1)
|
||||
}
|
||||
}
|
||||
if recorder.status == http.StatusUnauthorized {
|
||||
stats.authFailures.Add(1)
|
||||
}
|
||||
|
||||
if r.URL.Path == "/health" || r.URL.Path == "/ready" || r.URL.Path == "/metrics" {
|
||||
return
|
||||
}
|
||||
log.Printf("%s %s %d %dB %s", r.Method, redactRequestURI(r), recorder.status, recorder.bytes, time.Since(startedAt).Round(time.Millisecond))
|
||||
})
|
||||
}
|
||||
|
||||
func redactRequestURI(r *http.Request) string {
|
||||
if r.URL.RawQuery == "" {
|
||||
return r.URL.Path
|
||||
}
|
||||
redacted := url.Values{}
|
||||
for key, values := range r.URL.Query() {
|
||||
switch strings.ToLower(key) {
|
||||
case "token", "access_token":
|
||||
redacted.Set(key, "***")
|
||||
default:
|
||||
for _, value := range values {
|
||||
redacted.Add(key, value)
|
||||
}
|
||||
}
|
||||
}
|
||||
if encoded := redacted.Encode(); encoded != "" {
|
||||
return r.URL.Path + "?" + encoded
|
||||
}
|
||||
return r.URL.Path
|
||||
}
|
||||
|
||||
func (s *Server) routes() {
|
||||
@@ -126,25 +285,38 @@ func (s *Server) routes() {
|
||||
})
|
||||
s.mux.HandleFunc("GET /health", s.handleHealth)
|
||||
s.mux.HandleFunc("GET /ready", s.handleReady)
|
||||
s.mux.HandleFunc("GET /metrics", s.handleMetrics)
|
||||
s.mux.HandleFunc("GET /api/v1/system", s.handleSystem)
|
||||
|
||||
s.mux.HandleFunc("GET /api/v1/admin/dashboard", s.handleDashboardJSON)
|
||||
s.mux.HandleFunc("GET /api/v1/admin/basemaps", s.handleBasemapsJSON)
|
||||
s.mux.HandleFunc("GET /api/v1/admin/vectors", s.handleVectorsJSON)
|
||||
s.mux.HandleFunc("GET /api/v1/admin/scan-sources", s.handleScanSourcesJSON)
|
||||
s.mux.HandleFunc("POST /api/v1/admin/scan-sources", s.handleCreateScanSourceJSON)
|
||||
s.mux.HandleFunc("POST /api/v1/admin/scan-sources/{code}/run", s.handleRunScanJSON)
|
||||
s.mux.HandleFunc("POST /api/v1/admin/vectors/rescan", s.handleRunVectorScanJSON)
|
||||
s.mux.HandleFunc("POST /api/v1/admin/basemaps/{code}/default-version", s.handleSetDefaultVersionJSON)
|
||||
s.mux.HandleFunc("GET /api/v1/admin/targets", s.handleTargetsJSON)
|
||||
s.mux.HandleFunc("POST /api/v1/admin/targets", s.handleCreateTargetJSON)
|
||||
s.mux.HandleFunc("GET /api/v1/admin/tokens", s.handleTokensJSON)
|
||||
s.mux.HandleFunc("POST /api/v1/admin/tokens", s.handleCreateTokenJSON)
|
||||
s.mux.HandleFunc("POST /api/v1/admin/tokens/{id}/disable", s.handleDisableTokenJSON)
|
||||
s.mux.HandleFunc("GET /api/v1/admin/push-records", s.handlePushRecordsJSON)
|
||||
s.mux.HandleFunc("POST /api/v1/admin/pushes", s.handlePushJSON)
|
||||
s.mux.HandleFunc("GET /api/v1/admin/vector-push-records", s.handleVectorPushRecordsJSON)
|
||||
s.mux.HandleFunc("POST /api/v1/admin/vector-pushes", s.handleVectorPushJSON)
|
||||
// Creating a token here grants read access to every basemap, so the whole
|
||||
// admin surface sits behind the admin credentials.
|
||||
s.mux.HandleFunc("GET /api/v1/admin/dashboard", s.requireAdmin(s.handleDashboardJSON))
|
||||
s.mux.HandleFunc("GET /api/v1/admin/basemaps", s.requireAdmin(s.handleBasemapsJSON))
|
||||
s.mux.HandleFunc("GET /api/v1/admin/vectors", s.requireAdmin(s.handleVectorsJSON))
|
||||
s.mux.HandleFunc("GET /api/v1/admin/scan-sources", s.requireAdmin(s.handleScanSourcesJSON))
|
||||
s.mux.HandleFunc("POST /api/v1/admin/scan-sources", s.requireAdmin(s.handleCreateScanSourceJSON))
|
||||
s.mux.HandleFunc("POST /api/v1/admin/scan-sources/{code}/run", s.requireAdmin(s.handleRunScanJSON))
|
||||
s.mux.HandleFunc("POST /api/v1/admin/vectors/rescan", s.requireAdmin(s.handleRunVectorScanJSON))
|
||||
s.mux.HandleFunc("POST /api/v1/admin/basemaps/{code}/default-version", s.requireAdmin(s.handleSetDefaultVersionJSON))
|
||||
s.mux.HandleFunc("GET /api/v1/admin/targets", s.requireAdmin(s.handleTargetsJSON))
|
||||
s.mux.HandleFunc("POST /api/v1/admin/targets", s.requireAdmin(s.handleCreateTargetJSON))
|
||||
s.mux.HandleFunc("GET /api/v1/admin/tokens", s.requireAdmin(s.handleTokensJSON))
|
||||
s.mux.HandleFunc("POST /api/v1/admin/tokens", s.requireAdmin(s.handleCreateTokenJSON))
|
||||
s.mux.HandleFunc("POST /api/v1/admin/tokens/{id}/disable", s.requireAdmin(s.handleDisableTokenJSON))
|
||||
s.mux.HandleFunc("GET /api/v1/admin/push-records", s.requireAdmin(s.handlePushRecordsJSON))
|
||||
s.mux.HandleFunc("POST /api/v1/admin/pushes", s.requireAdmin(s.handlePushJSON))
|
||||
s.mux.HandleFunc("GET /api/v1/admin/vector-push-records", s.requireAdmin(s.handleVectorPushRecordsJSON))
|
||||
s.mux.HandleFunc("POST /api/v1/admin/vector-pushes", s.requireAdmin(s.handleVectorPushJSON))
|
||||
s.mux.HandleFunc("POST /api/v1/admin/health/check", s.requireAdmin(s.handleCheckHealthJSON))
|
||||
s.mux.HandleFunc("GET /api/v1/admin/tile-jobs", s.requireAdmin(s.handleTileJobsJSON))
|
||||
s.mux.HandleFunc("POST /api/v1/admin/tile-jobs", s.requireAdmin(s.handleCreateTileJobJSON))
|
||||
s.mux.HandleFunc("POST /api/v1/admin/push-records/{id}/retry", s.requireAdmin(s.handleRetryPushJSON))
|
||||
s.mux.HandleFunc("POST /api/v1/admin/vector-push-records/{id}/retry", s.requireAdmin(s.handleRetryVectorPushJSON))
|
||||
|
||||
// Slicing for external callers: a token with the "slice" scope, not admin.
|
||||
s.mux.HandleFunc("POST /api/v1/slice/jobs", s.handleSliceJobCreate)
|
||||
s.mux.HandleFunc("GET /api/v1/slice/jobs/{id}", s.handleSliceJobGet)
|
||||
s.mux.HandleFunc("GET /api/v1/slice/layers/{code}/sources", s.handleSliceLayerSources)
|
||||
|
||||
s.mux.HandleFunc("GET /api/v1/catalog/basemaps", s.handleCatalogList)
|
||||
s.mux.HandleFunc("GET /api/v1/catalog/basemaps/{code}", s.handleCatalogGet)
|
||||
@@ -155,16 +327,78 @@ func (s *Server) routes() {
|
||||
s.mux.HandleFunc("GET /shp/{assetPath...}", s.handleShapeByPath)
|
||||
s.mux.HandleFunc("GET /vectors/{code}/download", s.handleVectorDownload)
|
||||
|
||||
s.mux.HandleFunc("GET /admin/", s.handleAdminPage)
|
||||
s.mux.HandleFunc("POST /admin/scan-sources", s.handleCreateScanSourceForm)
|
||||
s.mux.HandleFunc("POST /admin/scan-sources/{code}/run", s.handleRunScanForm)
|
||||
s.mux.HandleFunc("POST /admin/vectors/rescan", s.handleRunVectorScanForm)
|
||||
s.mux.HandleFunc("POST /admin/basemaps/{code}/default", s.handleSetDefaultVersionForm)
|
||||
s.mux.HandleFunc("POST /admin/targets", s.handleCreateTargetForm)
|
||||
s.mux.HandleFunc("POST /admin/tokens", s.handleCreateTokenForm)
|
||||
s.mux.HandleFunc("POST /admin/tokens/{id}/disable", s.handleDisableTokenForm)
|
||||
s.mux.HandleFunc("POST /admin/pushes", s.handlePushForm)
|
||||
s.mux.HandleFunc("POST /admin/vector-pushes", s.handleVectorPushForm)
|
||||
s.mux.HandleFunc("GET /admin/", s.requireAdmin(s.handleAdminPage))
|
||||
s.mux.HandleFunc("POST /admin/scan-sources", s.requireAdmin(s.handleCreateScanSourceForm))
|
||||
s.mux.HandleFunc("POST /admin/scan-sources/{code}/run", s.requireAdmin(s.handleRunScanForm))
|
||||
s.mux.HandleFunc("POST /admin/vectors/rescan", s.requireAdmin(s.handleRunVectorScanForm))
|
||||
s.mux.HandleFunc("POST /admin/basemaps/{code}/default", s.requireAdmin(s.handleSetDefaultVersionForm))
|
||||
s.mux.HandleFunc("POST /admin/targets", s.requireAdmin(s.handleCreateTargetForm))
|
||||
s.mux.HandleFunc("POST /admin/tokens", s.requireAdmin(s.handleCreateTokenForm))
|
||||
s.mux.HandleFunc("POST /admin/tokens/{id}/disable", s.requireAdmin(s.handleDisableTokenForm))
|
||||
s.mux.HandleFunc("POST /admin/pushes", s.requireAdmin(s.handlePushForm))
|
||||
s.mux.HandleFunc("POST /admin/vector-pushes", s.requireAdmin(s.handleVectorPushForm))
|
||||
s.mux.HandleFunc("POST /admin/health/check", s.requireAdmin(s.handleCheckHealthForm))
|
||||
s.mux.HandleFunc("POST /admin/tile-jobs", s.requireAdmin(s.handleCreateTileJobForm))
|
||||
s.mux.HandleFunc("POST /admin/tile-jobs/rebuild", s.requireAdmin(s.handleRebuildLayerForm))
|
||||
s.mux.HandleFunc("POST /admin/push-records/{id}/retry", s.requireAdmin(s.handleRetryPushForm))
|
||||
s.mux.HandleFunc("POST /admin/vector-push-records/{id}/retry", s.requireAdmin(s.handleRetryVectorPushForm))
|
||||
}
|
||||
|
||||
// requireAdmin guards the admin surface. With no credentials configured the
|
||||
// service is only reachable on loopback, which main enforces at startup.
|
||||
func (s *Server) requireAdmin(next http.HandlerFunc) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
if !s.admin.enabled() {
|
||||
next(w, r)
|
||||
return
|
||||
}
|
||||
if s.adminAllows(r) {
|
||||
next(w, r)
|
||||
return
|
||||
}
|
||||
w.Header().Set("WWW-Authenticate", `Basic realm="basemap-pusher admin"`)
|
||||
http.Error(w, "admin credentials required", http.StatusUnauthorized)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) adminAllows(r *http.Request) bool {
|
||||
if token := strings.TrimSpace(s.admin.Token); token != "" {
|
||||
header := strings.TrimSpace(r.Header.Get("Authorization"))
|
||||
if len(header) > 7 && strings.EqualFold(header[:7], "bearer ") {
|
||||
if subtle.ConstantTimeCompare([]byte(strings.TrimSpace(header[7:])), []byte(token)) == 1 {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
user := strings.TrimSpace(s.admin.User)
|
||||
if user == "" || s.admin.Password == "" {
|
||||
return false
|
||||
}
|
||||
givenUser, givenPassword, ok := r.BasicAuth()
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
userOK := subtle.ConstantTimeCompare([]byte(givenUser), []byte(user)) == 1
|
||||
passwordOK := subtle.ConstantTimeCompare([]byte(givenPassword), []byte(s.admin.Password)) == 1
|
||||
return userOK && passwordOK
|
||||
}
|
||||
|
||||
func (s *Server) handleMetrics(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/plain; version=0.0.4; charset=utf-8")
|
||||
fmt.Fprintf(w, "# TYPE basemap_pusher_requests_total counter\n")
|
||||
fmt.Fprintf(w, "basemap_pusher_requests_total %d\n", s.metrics.requests.Load())
|
||||
fmt.Fprintf(w, "# TYPE basemap_pusher_requests_by_status_total counter\n")
|
||||
fmt.Fprintf(w, "basemap_pusher_requests_by_status_total{class=\"2xx\"} %d\n", s.metrics.status2xx.Load())
|
||||
fmt.Fprintf(w, "basemap_pusher_requests_by_status_total{class=\"4xx\"} %d\n", s.metrics.status4xx.Load())
|
||||
fmt.Fprintf(w, "basemap_pusher_requests_by_status_total{class=\"5xx\"} %d\n", s.metrics.status5xx.Load())
|
||||
fmt.Fprintf(w, "# TYPE basemap_pusher_tile_requests_total counter\n")
|
||||
fmt.Fprintf(w, "basemap_pusher_tile_requests_total %d\n", s.metrics.tiles.Load())
|
||||
fmt.Fprintf(w, "# TYPE basemap_pusher_tile_misses_total counter\n")
|
||||
fmt.Fprintf(w, "basemap_pusher_tile_misses_total %d\n", s.metrics.tileMisses.Load())
|
||||
fmt.Fprintf(w, "# TYPE basemap_pusher_auth_failures_total counter\n")
|
||||
fmt.Fprintf(w, "basemap_pusher_auth_failures_total %d\n", s.metrics.authFailures.Load())
|
||||
fmt.Fprintf(w, "# TYPE basemap_pusher_uptime_seconds gauge\n")
|
||||
fmt.Fprintf(w, "basemap_pusher_uptime_seconds %.0f\n", time.Since(s.startedAt).Seconds())
|
||||
}
|
||||
|
||||
func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) {
|
||||
@@ -241,7 +475,7 @@ func (s *Server) handleCreateScanSourceJSON(w http.ResponseWriter, r *http.Reque
|
||||
func (s *Server) handleRunScanJSON(w http.ResponseWriter, r *http.Request) {
|
||||
run, err := s.store.RunScan(r.Context(), r.PathValue("code"))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err)
|
||||
writeError(w, scanErrorStatus(err), err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, run)
|
||||
@@ -250,7 +484,7 @@ func (s *Server) handleRunScanJSON(w http.ResponseWriter, r *http.Request) {
|
||||
func (s *Server) handleRunVectorScanJSON(w http.ResponseWriter, r *http.Request) {
|
||||
result, err := s.store.RunVectorScan(r.Context())
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err)
|
||||
writeError(w, scanErrorStatus(err), err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, result)
|
||||
@@ -358,7 +592,7 @@ func (s *Server) handlePushJSON(w http.ResponseWriter, r *http.Request) {
|
||||
writeError(w, http.StatusBadRequest, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusCreated, item)
|
||||
writeJSON(w, http.StatusAccepted, item)
|
||||
}
|
||||
|
||||
func (s *Server) handleVectorPushJSON(w http.ResponseWriter, r *http.Request) {
|
||||
@@ -375,7 +609,7 @@ func (s *Server) handleVectorPushJSON(w http.ResponseWriter, r *http.Request) {
|
||||
writeError(w, http.StatusBadRequest, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusCreated, item)
|
||||
writeJSON(w, http.StatusAccepted, item)
|
||||
}
|
||||
|
||||
func (s *Server) handleCatalogList(w http.ResponseWriter, r *http.Request) {
|
||||
@@ -456,6 +690,7 @@ func (s *Server) handleTile(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", item.ContentType)
|
||||
w.Header().Set("Cache-Control", "public, max-age=31536000, immutable")
|
||||
http.ServeFile(w, r, item.FilePath)
|
||||
}
|
||||
|
||||
@@ -515,6 +750,203 @@ func (s *Server) handleShapeByPath(w http.ResponseWriter, r *http.Request) {
|
||||
http.ServeFile(w, r, item.FilePath)
|
||||
}
|
||||
|
||||
func (s *Server) handleCheckHealthJSON(w http.ResponseWriter, r *http.Request) {
|
||||
report, err := s.store.CheckHealth(r.Context())
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, report)
|
||||
}
|
||||
|
||||
func (s *Server) handleTileJobsJSON(w http.ResponseWriter, r *http.Request) {
|
||||
items, err := s.store.ListTileJobs(r.Context(), 50)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, items)
|
||||
}
|
||||
|
||||
func (s *Server) handleCreateTileJobJSON(w http.ResponseWriter, r *http.Request) {
|
||||
var input basemap.CreateTileJobInput
|
||||
if err := json.NewDecoder(r.Body).Decode(&input); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err)
|
||||
return
|
||||
}
|
||||
job, err := s.store.CreateTileJob(r.Context(), input)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusAccepted, job)
|
||||
}
|
||||
|
||||
func (s *Server) handleRetryPushJSON(w http.ResponseWriter, r *http.Request) {
|
||||
if err := s.store.RetryPush(r.Context(), "basemap", r.PathValue("id")); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusAccepted, map[string]string{"status": "queued"})
|
||||
}
|
||||
|
||||
func (s *Server) handleRetryVectorPushJSON(w http.ResponseWriter, r *http.Request) {
|
||||
if err := s.store.RetryPush(r.Context(), "vector", r.PathValue("id")); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusAccepted, map[string]string{"status": "queued"})
|
||||
}
|
||||
|
||||
func (s *Server) handleCheckHealthForm(w http.ResponseWriter, r *http.Request) {
|
||||
report, err := s.store.CheckHealth(r.Context())
|
||||
flash := "资源巡检已完成"
|
||||
if err == nil {
|
||||
flash = fmt.Sprintf("资源巡检完成：底图缺 %d / %d，矢量缺 %d / %d",
|
||||
report.VersionsMissing, report.VersionsChecked, report.VectorsMissing, report.VectorsChecked)
|
||||
}
|
||||
s.redirectAdmin(w, r, flash, err)
|
||||
}
|
||||
|
||||
func (s *Server) handleCreateTileJobForm(w http.ResponseWriter, r *http.Request) {
|
||||
if err := r.ParseForm(); err != nil {
|
||||
s.redirectAdmin(w, r, "", err)
|
||||
return
|
||||
}
|
||||
zoomMin, _ := strconv.Atoi(strings.TrimSpace(r.FormValue("zoom_min")))
|
||||
zoomMax, _ := strconv.Atoi(strings.TrimSpace(r.FormValue("zoom_max")))
|
||||
tileSize, _ := strconv.Atoi(strings.TrimSpace(r.FormValue("tile_size")))
|
||||
_, err := s.store.CreateTileJob(r.Context(), basemap.CreateTileJobInput{
|
||||
Code: r.FormValue("code"),
|
||||
Name: r.FormValue("name"),
|
||||
SourcePath: r.FormValue("source_path"),
|
||||
TargetCode: r.FormValue("target_code"),
|
||||
ZoomMin: zoomMin,
|
||||
ZoomMax: zoomMax,
|
||||
TileSize: tileSize,
|
||||
TileFormat: r.FormValue("tile_format"),
|
||||
CallbackURL: r.FormValue("callback_url"),
|
||||
RequestedBy: r.FormValue("requested_by"),
|
||||
})
|
||||
s.redirectAdmin(w, r, "切片任务已提交，刷新查看进度", err)
|
||||
}
|
||||
|
||||
// handleRebuildLayerForm slices every source of a layer into one VRT mosaic,
|
||||
// reusing the zoom range the layer is already published at.
|
||||
func (s *Server) handleRebuildLayerForm(w http.ResponseWriter, r *http.Request) {
|
||||
if err := r.ParseForm(); err != nil {
|
||||
s.redirectAdmin(w, r, "", err)
|
||||
return
|
||||
}
|
||||
targetCode := strings.TrimSpace(r.FormValue("target_code"))
|
||||
if targetCode == "" {
|
||||
s.redirectAdmin(w, r, "", errors.New("target code is required"))
|
||||
return
|
||||
}
|
||||
|
||||
sources, err := s.store.ListLayerSources(r.Context(), targetCode)
|
||||
if err != nil {
|
||||
s.redirectAdmin(w, r, "", err)
|
||||
return
|
||||
}
|
||||
if len(sources) == 0 {
|
||||
s.redirectAdmin(w, r, "", fmt.Errorf("layer %q has no recorded sources", targetCode))
|
||||
return
|
||||
}
|
||||
|
||||
zoomMin := 0
|
||||
zoomMax := 18
|
||||
tileFormat := "webp"
|
||||
if items, err := s.store.ListLayerInventories(r.Context()); err == nil {
|
||||
for _, item := range items {
|
||||
if item.TargetCode == normalizeLayerCode(targetCode) {
|
||||
zoomMin, zoomMax, tileFormat = item.ZoomMin, item.ZoomMax, item.TileFormat
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
paths := make([]string, 0, len(sources))
|
||||
for _, source := range sources {
|
||||
paths = append(paths, source.SourcePath)
|
||||
}
|
||||
if _, err := s.store.CreateTileJob(r.Context(), basemap.CreateTileJobInput{
|
||||
Name: "重建 " + targetCode,
|
||||
TargetCode: targetCode,
|
||||
SourcePaths: paths,
|
||||
ZoomMin: zoomMin,
|
||||
ZoomMax: zoomMax,
|
||||
TileFormat: tileFormat,
|
||||
RequestedBy: "console",
|
||||
}); err != nil {
|
||||
s.redirectAdmin(w, r, "", err)
|
||||
return
|
||||
}
|
||||
s.redirectAdmin(w, r, fmt.Sprintf("已提交 %s 的镶嵌重建（%d 个源，级别 %d-%d）", targetCode, len(paths), zoomMin, zoomMax), nil)
|
||||
}
|
||||
func (s *Server) handleRetryPushForm(w http.ResponseWriter, r *http.Request) {
|
||||
err := s.store.RetryPush(r.Context(), "basemap", r.PathValue("id"))
|
||||
s.redirectAdmin(w, r, "已重新排队推送", err)
|
||||
}
|
||||
|
||||
func (s *Server) handleRetryVectorPushForm(w http.ResponseWriter, r *http.Request) {
|
||||
err := s.store.RetryPush(r.Context(), "vector", r.PathValue("id"))
|
||||
s.redirectAdmin(w, r, "已重新排队推送", err)
|
||||
}
|
||||
|
||||
// requireSliceToken authenticates and then checks the coarse slice capability,
|
||||
// so an integration can submit jobs without holding the admin credential.
|
||||
func (s *Server) requireSliceToken(w http.ResponseWriter, r *http.Request) (basemap.TokenAuth, bool) {
|
||||
auth, ok := s.requireToken(w, r)
|
||||
if !ok {
|
||||
return basemap.TokenAuth{}, false
|
||||
}
|
||||
if !auth.CanSlice() {
|
||||
writeError(w, http.StatusForbidden, errors.New("token does not carry the slice scope"))
|
||||
return basemap.TokenAuth{}, false
|
||||
}
|
||||
return auth, true
|
||||
}
|
||||
|
||||
func (s *Server) handleSliceJobCreate(w http.ResponseWriter, r *http.Request) {
|
||||
if _, ok := s.requireSliceToken(w, r); !ok {
|
||||
return
|
||||
}
|
||||
var input basemap.CreateTileJobInput
|
||||
if err := json.NewDecoder(r.Body).Decode(&input); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err)
|
||||
return
|
||||
}
|
||||
job, err := s.store.CreateTileJob(r.Context(), input)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusAccepted, job)
|
||||
}
|
||||
|
||||
func (s *Server) handleSliceLayerSources(w http.ResponseWriter, r *http.Request) {
|
||||
if _, ok := s.requireSliceToken(w, r); !ok {
|
||||
return
|
||||
}
|
||||
items, err := s.store.ListLayerSources(r.Context(), r.PathValue("code"))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, items)
|
||||
}
|
||||
func (s *Server) handleSliceJobGet(w http.ResponseWriter, r *http.Request) {
|
||||
if _, ok := s.requireSliceToken(w, r); !ok {
|
||||
return
|
||||
}
|
||||
job, err := s.store.GetTileJob(r.Context(), r.PathValue("id"))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusNotFound, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, job)
|
||||
}
|
||||
func (s *Server) handleAdminPage(w http.ResponseWriter, r *http.Request) {
|
||||
data, err := s.store.Dashboard(r.Context(), s.system, r.URL.Query().Get("flash"), r.URL.Query().Get("token"))
|
||||
if err != nil {
|
||||
@@ -529,7 +961,7 @@ func (s *Server) handleAdminPage(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
func (s *Server) handleCreateScanSourceForm(w http.ResponseWriter, r *http.Request) {
|
||||
if err := r.ParseForm(); err != nil {
|
||||
s.redirectAdmin(w, r, "", "", err)
|
||||
s.redirectAdmin(w, r, "", err)
|
||||
return
|
||||
}
|
||||
_, err := s.store.EnsureScanSource(r.Context(), basemap.CreateScanSourceInput{
|
||||
@@ -538,31 +970,31 @@ func (s *Server) handleCreateScanSourceForm(w http.ResponseWriter, r *http.Reque
|
||||
RootPath: r.FormValue("root_path"),
|
||||
ManifestName: r.FormValue("manifest_name"),
|
||||
})
|
||||
s.redirectAdmin(w, r, "扫描源已保存", "", err)
|
||||
s.redirectAdmin(w, r, "扫描源已保存", err)
|
||||
}
|
||||
|
||||
func (s *Server) handleRunScanForm(w http.ResponseWriter, r *http.Request) {
|
||||
_, err := s.store.RunScan(r.Context(), r.PathValue("code"))
|
||||
s.redirectAdmin(w, r, "底图扫描已完成", "", err)
|
||||
s.redirectAdmin(w, r, "底图扫描已完成", err)
|
||||
}
|
||||
|
||||
func (s *Server) handleRunVectorScanForm(w http.ResponseWriter, r *http.Request) {
|
||||
_, err := s.store.RunVectorScan(r.Context())
|
||||
s.redirectAdmin(w, r, "矢量扫描已完成", "", err)
|
||||
s.redirectAdmin(w, r, "矢量扫描已完成", err)
|
||||
}
|
||||
|
||||
func (s *Server) handleSetDefaultVersionForm(w http.ResponseWriter, r *http.Request) {
|
||||
if err := r.ParseForm(); err != nil {
|
||||
s.redirectAdmin(w, r, "", "", err)
|
||||
s.redirectAdmin(w, r, "", err)
|
||||
return
|
||||
}
|
||||
err := s.store.SetDefaultVersion(r.Context(), r.PathValue("code"), r.FormValue("version"))
|
||||
s.redirectAdmin(w, r, "默认版本已更新", "", err)
|
||||
s.redirectAdmin(w, r, "默认版本已更新", err)
|
||||
}
|
||||
|
||||
func (s *Server) handleCreateTargetForm(w http.ResponseWriter, r *http.Request) {
|
||||
if err := r.ParseForm(); err != nil {
|
||||
s.redirectAdmin(w, r, "", "", err)
|
||||
s.redirectAdmin(w, r, "", err)
|
||||
return
|
||||
}
|
||||
headers := map[string]string{}
|
||||
@@ -580,44 +1012,44 @@ func (s *Server) handleCreateTargetForm(w http.ResponseWriter, r *http.Request)
|
||||
CallbackMethod: r.FormValue("callback_method"),
|
||||
CallbackHeaders: headers,
|
||||
})
|
||||
s.redirectAdmin(w, r, "目标系统已保存", "", err)
|
||||
s.redirectAdmin(w, r, "目标系统已保存", err)
|
||||
}
|
||||
|
||||
func (s *Server) handleCreateTokenForm(w http.ResponseWriter, r *http.Request) {
|
||||
input, err := decodeTokenInputFromForm(r)
|
||||
if err != nil {
|
||||
s.redirectAdmin(w, r, "", "", err)
|
||||
s.redirectAdmin(w, r, "", err)
|
||||
return
|
||||
}
|
||||
item, err := s.store.CreateToken(r.Context(), input)
|
||||
if err != nil {
|
||||
s.redirectAdmin(w, r, "", "", err)
|
||||
s.redirectAdmin(w, r, "", err)
|
||||
return
|
||||
}
|
||||
s.redirectAdmin(w, r, "访问令牌已创建", item.Token, nil)
|
||||
s.renderAdmin(w, r, "访问令牌已创建", item.Token, nil)
|
||||
}
|
||||
|
||||
func (s *Server) handleDisableTokenForm(w http.ResponseWriter, r *http.Request) {
|
||||
err := s.store.DisableToken(r.Context(), r.PathValue("id"))
|
||||
s.redirectAdmin(w, r, "访问令牌已停用", "", err)
|
||||
s.redirectAdmin(w, r, "访问令牌已停用", err)
|
||||
}
|
||||
|
||||
func (s *Server) handlePushForm(w http.ResponseWriter, r *http.Request) {
|
||||
if err := r.ParseForm(); err != nil {
|
||||
s.redirectAdmin(w, r, "", "", err)
|
||||
s.redirectAdmin(w, r, "", err)
|
||||
return
|
||||
}
|
||||
_, err := s.store.PushBasemapVersion(r.Context(), r.FormValue("target_code"), r.FormValue("basemap_code"), r.FormValue("version"))
|
||||
s.redirectAdmin(w, r, "底图推送已完成", "", err)
|
||||
s.redirectAdmin(w, r, "底图推送已提交，稍后刷新查看结果", err)
|
||||
}
|
||||
|
||||
func (s *Server) handleVectorPushForm(w http.ResponseWriter, r *http.Request) {
|
||||
if err := r.ParseForm(); err != nil {
|
||||
s.redirectAdmin(w, r, "", "", err)
|
||||
s.redirectAdmin(w, r, "", err)
|
||||
return
|
||||
}
|
||||
_, err := s.store.PushVectorAsset(r.Context(), r.FormValue("target_code"), r.FormValue("vector_code"))
|
||||
s.redirectAdmin(w, r, "矢量推送已完成", "", err)
|
||||
s.redirectAdmin(w, r, "矢量推送已提交，稍后刷新查看结果", err)
|
||||
}
|
||||
|
||||
func (s *Server) requireToken(w http.ResponseWriter, r *http.Request) (basemap.TokenAuth, bool) {
|
||||
@@ -652,6 +1084,7 @@ func decodeTokenInput(r *http.Request) (basemap.CreateTokenInput, error) {
|
||||
Name string `json:"name"`
|
||||
BasemapCodes []string `json:"basemap_codes"`
|
||||
VectorCodes []string `json:"vector_codes"`
|
||||
Scopes []string `json:"scopes"`
|
||||
ExpiresAt string `json:"expires_at"`
|
||||
}
|
||||
if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
|
||||
@@ -661,6 +1094,7 @@ func decodeTokenInput(r *http.Request) (basemap.CreateTokenInput, error) {
|
||||
Name: payload.Name,
|
||||
BasemapCodes: payload.BasemapCodes,
|
||||
VectorCodes: payload.VectorCodes,
|
||||
Scopes: payload.Scopes,
|
||||
}
|
||||
if strings.TrimSpace(payload.ExpiresAt) != "" {
|
||||
parsed, err := time.Parse(time.RFC3339, payload.ExpiresAt)
|
||||
@@ -680,6 +1114,7 @@ func decodeTokenInputFromForm(r *http.Request) (basemap.CreateTokenInput, error)
|
||||
Name: r.FormValue("name"),
|
||||
BasemapCodes: splitCSV(r.FormValue("basemap_codes")),
|
||||
VectorCodes: splitCSV(r.FormValue("vector_codes")),
|
||||
Scopes: splitCSV(r.FormValue("scopes")),
|
||||
}
|
||||
if value := strings.TrimSpace(r.FormValue("expires_at")); value != "" {
|
||||
if minutes, err := strconv.Atoi(value); err == nil && minutes > 0 {
|
||||
@@ -744,6 +1179,8 @@ func statusText(value string) string {
|
||||
return "失败"
|
||||
case "pending":
|
||||
return "待处理"
|
||||
case "retrying":
|
||||
return "重试中"
|
||||
case "running":
|
||||
return "运行中"
|
||||
case "processing":
|
||||
@@ -761,7 +1198,7 @@ func statusTone(value string) string {
|
||||
switch strings.ToLower(strings.TrimSpace(value)) {
|
||||
case "active", "enabled", "ready", "succeeded", "current":
|
||||
return "success"
|
||||
case "pending", "running", "processing":
|
||||
case "pending", "running", "processing", "retrying":
|
||||
return "warning"
|
||||
case "disabled", "failed", "error", "expired", "invalid":
|
||||
return "danger"
|
||||
@@ -787,6 +1224,25 @@ func formatBytes(size int64) string {
|
||||
return fmt.Sprintf("%.1f %s", value, units[unit])
|
||||
}
|
||||
|
||||
// scanErrorStatus maps scan failures onto HTTP status codes: a busy scanner is
|
||||
// a conflict, an unknown source is a 404, and anything else is a server fault.
|
||||
// normalizeLayerCode mirrors the store's basemap code normalization so the
|
||||
// console can match a layer against its inventory entry.
|
||||
func normalizeLayerCode(value string) string {
|
||||
return strings.ToLower(strings.TrimSpace(value))
|
||||
}
|
||||
func scanErrorStatus(err error) int {
|
||||
switch {
|
||||
case errors.Is(err, basemap.ErrScanInProgress):
|
||||
return http.StatusConflict
|
||||
case errors.Is(err, basemap.ErrScanSourceNotFound):
|
||||
return http.StatusNotFound
|
||||
case errors.Is(err, basemap.ErrScanSourceDisabled):
|
||||
return http.StatusBadRequest
|
||||
default:
|
||||
return http.StatusInternalServerError
|
||||
}
|
||||
}
|
||||
func localizeAdminError(err error) string {
|
||||
if err == nil {
|
||||
return ""
|
||||
@@ -819,8 +1275,8 @@ func localizeAdminError(err error) string {
|
||||
return fmt.Sprintf("目标系统已停用：%s", quoted[0])
|
||||
case msg == "token name is required":
|
||||
return "令牌名称不能为空"
|
||||
case msg == "at least one basemap code or vector code is required":
|
||||
return "至少填写一个底图编码或矢量资源编码/路径"
|
||||
case msg == "at least one basemap code, vector code, or the slice scope is required":
|
||||
return "至少填写一个底图编码、矢量资源，或勾选切片能力"
|
||||
case msg == "no valid basemap codes provided":
|
||||
return "未提供有效的底图编码"
|
||||
case len(quoted) == 1 && strings.HasPrefix(msg, `token "`) && strings.HasSuffix(msg, `" not found`):
|
||||
@@ -849,24 +1305,38 @@ func quotedSegments(value string) []string {
|
||||
return result
|
||||
}
|
||||
|
||||
func (s *Server) redirectAdmin(w http.ResponseWriter, r *http.Request, flash string, token string, err error) {
|
||||
// redirectAdmin sends the browser back to the console with a flash message. It
|
||||
// deliberately never carries a secret in the URL.
|
||||
func (s *Server) redirectAdmin(w http.ResponseWriter, r *http.Request, flash string, err error) {
|
||||
if err != nil {
|
||||
flash = "操作失败：" + localizeAdminError(err)
|
||||
}
|
||||
values := url.Values{}
|
||||
if flash != "" {
|
||||
values.Set("flash", flash)
|
||||
}
|
||||
if token != "" {
|
||||
values.Set("token", token)
|
||||
}
|
||||
location := "/admin/"
|
||||
if encoded := values.Encode(); encoded != "" {
|
||||
location += "?" + encoded
|
||||
if strings.TrimSpace(flash) != "" {
|
||||
values := url.Values{}
|
||||
values.Set("flash", flash)
|
||||
location += "?" + values.Encode()
|
||||
}
|
||||
http.Redirect(w, r, location, http.StatusSeeOther)
|
||||
}
|
||||
|
||||
// renderAdmin answers the current POST with the console HTML so one-time
|
||||
// secrets (a freshly minted token) stay out of the URL, history and logs.
|
||||
func (s *Server) renderAdmin(w http.ResponseWriter, r *http.Request, flash string, token string, err error) {
|
||||
if err != nil {
|
||||
flash = "操作失败：" + localizeAdminError(err)
|
||||
}
|
||||
data, dataErr := s.store.Dashboard(r.Context(), s.system, flash, token)
|
||||
if dataErr != nil {
|
||||
writeError(w, http.StatusInternalServerError, dataErr)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
if execErr := s.template.Execute(w, data); execErr != nil {
|
||||
writeError(w, http.StatusInternalServerError, execErr)
|
||||
}
|
||||
}
|
||||
|
||||
func writeJSON(w http.ResponseWriter, status int, value any) {
|
||||
w.Header().Set("Content-Type", "application/json; charset=utf-8")
|
||||
w.WriteHeader(status)
|
||||
@@ -878,11 +1348,3 @@ func writeError(w http.ResponseWriter, status int, err error) {
|
||||
"error": err.Error(),
|
||||
})
|
||||
}
|
||||
|
||||
func runStartupScan(ctx context.Context, store *basemap.Store) error {
|
||||
if !store.AutoScanEnabled() || strings.TrimSpace(store.ScanRoot()) == "" {
|
||||
return nil
|
||||
}
|
||||
_, err := store.RunScan(ctx, "local-basemaps")
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -0,0 +1,181 @@
|
||||
package basemapapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"tile-server/api-go/internal/basemap"
|
||||
)
|
||||
|
||||
func newSliceServer(t *testing.T) (*basemap.Store, *httptest.Server) {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
tileRoot := filepath.Join(root, "tiles")
|
||||
if err := os.MkdirAll(tileRoot, 0o755); err != nil {
|
||||
t.Fatalf("mkdir tile root: %v", err)
|
||||
}
|
||||
source := filepath.Join(root, "input.tif")
|
||||
writeFile(t, source, "fake raster")
|
||||
// A scanned basemap gives the negative test case something to grant.
|
||||
writeFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile")
|
||||
|
||||
store, err := basemap.OpenStore(basemap.StoreConfig{
|
||||
SQLitePath: filepath.Join(root, "basemap.db"),
|
||||
ScanRoot: tileRoot,
|
||||
VectorScanRoot: filepath.Join(root, "vectors"),
|
||||
APIBaseURL: "http://127.0.0.1:8910",
|
||||
TileBaseURL: "http://127.0.0.1:8910",
|
||||
VectorBaseURL: "http://127.0.0.1:8910",
|
||||
TileJobCommand: "definitely-not-a-real-binary {input} {output}",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("open store: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = store.Close() })
|
||||
if _, err := store.RunScan(context.Background(), "local-basemaps"); err != nil {
|
||||
t.Fatalf("scan: %v", err)
|
||||
}
|
||||
|
||||
return store, newTestServerRaw(t, store)
|
||||
}
|
||||
|
||||
func newTestServerRaw(t *testing.T, store *basemap.Store) *httptest.Server {
|
||||
t.Helper()
|
||||
server, err := New(store, basemap.SystemInfo{ServiceName: "basemap-pusher"}, Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("new server: %v", err)
|
||||
}
|
||||
httpServer := httptest.NewServer(server.Handler())
|
||||
t.Cleanup(httpServer.Close)
|
||||
return httpServer
|
||||
}
|
||||
|
||||
func postJob(t *testing.T, url, token string, body map[string]any) *http.Response {
|
||||
t.Helper()
|
||||
encoded, err := json.Marshal(body)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal: %v", err)
|
||||
}
|
||||
request, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(encoded))
|
||||
if err != nil {
|
||||
t.Fatalf("request: %v", err)
|
||||
}
|
||||
request.Header.Set("Content-Type", "application/json")
|
||||
request.Header.Set("Authorization", "Bearer "+token)
|
||||
response, err := http.DefaultClient.Do(request)
|
||||
if err != nil {
|
||||
t.Fatalf("post: %v", err)
|
||||
}
|
||||
return response
|
||||
}
|
||||
|
||||
func TestSliceJobRequiresSliceScope(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
store, httpServer := newSliceServer(t)
|
||||
source := filepath.Join(filepath.Dir(store.ScanRoot()), "input.tif")
|
||||
|
||||
readOnly, err := store.CreateToken(ctx, basemap.CreateTokenInput{
|
||||
Name: "read-only",
|
||||
BasemapCodes: []string{"google_image"},
|
||||
Scopes: []string{"read"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("create read-only token: %v", err)
|
||||
}
|
||||
if _, err := store.CreateToken(ctx, basemap.CreateTokenInput{Name: "empty"}); err == nil {
|
||||
t.Fatal("expected a token with neither grants nor slice scope to be rejected")
|
||||
}
|
||||
|
||||
response := postJob(t, httpServer.URL+"/api/v1/slice/jobs", readOnly.Token, map[string]any{
|
||||
"source_path": source,
|
||||
"target_code": "demo",
|
||||
})
|
||||
response.Body.Close()
|
||||
if response.StatusCode != http.StatusForbidden {
|
||||
t.Fatalf("expected 403 for a read-only token, got %d", response.StatusCode)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSliceJobSubmitAndPoll(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
store, httpServer := newSliceServer(t)
|
||||
source := filepath.Join(filepath.Dir(store.ScanRoot()), "input.tif")
|
||||
|
||||
slicer, err := store.CreateToken(ctx, basemap.CreateTokenInput{
|
||||
Name: "integration",
|
||||
Scopes: []string{"slice"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("create slicer token: %v", err)
|
||||
}
|
||||
|
||||
body := map[string]any{
|
||||
"code": "integration-job",
|
||||
"source_path": source,
|
||||
"target_code": "demo",
|
||||
"zoom_min": 0,
|
||||
"zoom_max": 2,
|
||||
}
|
||||
response := postJob(t, httpServer.URL+"/api/v1/slice/jobs", slicer.Token, body)
|
||||
if response.StatusCode != http.StatusAccepted {
|
||||
response.Body.Close()
|
||||
t.Fatalf("expected 202, got %d", response.StatusCode)
|
||||
}
|
||||
var job basemap.TileJob
|
||||
if err := json.NewDecoder(response.Body).Decode(&job); err != nil {
|
||||
t.Fatalf("decode: %v", err)
|
||||
}
|
||||
response.Body.Close()
|
||||
|
||||
// Re-submitting the same code must not queue a second job.
|
||||
again := postJob(t, httpServer.URL+"/api/v1/slice/jobs", slicer.Token, body)
|
||||
again.Body.Close()
|
||||
if again.StatusCode != http.StatusAccepted {
|
||||
t.Fatalf("expected the repeat submit to be accepted, got %d", again.StatusCode)
|
||||
}
|
||||
|
||||
request, _ := http.NewRequest(http.MethodGet, httpServer.URL+"/api/v1/slice/jobs/"+job.ID, nil)
|
||||
request.Header.Set("Authorization", "Bearer "+slicer.Token)
|
||||
polled, err := http.DefaultClient.Do(request)
|
||||
if err != nil {
|
||||
t.Fatalf("poll: %v", err)
|
||||
}
|
||||
defer polled.Body.Close()
|
||||
if polled.StatusCode != http.StatusOK {
|
||||
t.Fatalf("expected 200 from the poll endpoint, got %d", polled.StatusCode)
|
||||
}
|
||||
var fetched basemap.TileJob
|
||||
if err := json.NewDecoder(polled.Body).Decode(&fetched); err != nil {
|
||||
t.Fatalf("decode polled job: %v", err)
|
||||
}
|
||||
if fetched.ID != job.ID {
|
||||
t.Fatalf("polled %s, submitted %s", fetched.ID, job.ID)
|
||||
}
|
||||
|
||||
jobs, err := store.ListTileJobs(ctx, 10)
|
||||
if err != nil {
|
||||
t.Fatalf("list: %v", err)
|
||||
}
|
||||
if len(jobs) != 1 {
|
||||
t.Fatalf("expected one job after a repeated submit, got %d", len(jobs))
|
||||
}
|
||||
}
|
||||
|
||||
func TestSliceJobRejectsAnonymous(t *testing.T) {
|
||||
_, httpServer := newSliceServer(t)
|
||||
request, _ := http.NewRequest(http.MethodGet, httpServer.URL+"/api/v1/slice/jobs/whatever", nil)
|
||||
response, err := http.DefaultClient.Do(request)
|
||||
if err != nil {
|
||||
t.Fatalf("request: %v", err)
|
||||
}
|
||||
defer response.Body.Close()
|
||||
if response.StatusCode != http.StatusUnauthorized {
|
||||
t.Fatalf("expected 401 without a token, got %d", response.StatusCode)
|
||||
}
|
||||
}
|
||||
@@ -2,9 +2,12 @@ package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
type Config struct {
|
||||
@@ -21,6 +24,17 @@ type Config struct {
|
||||
ServiceName string
|
||||
ServiceVersion string
|
||||
Commit string
|
||||
|
||||
AdminToken string
|
||||
AdminUser string
|
||||
AdminPassword string
|
||||
CORSAllowedOrigins []string
|
||||
TokenDefaultTTL time.Duration
|
||||
|
||||
HealthCheckInterval time.Duration
|
||||
TileJobCommand string
|
||||
TileJobVRTCommand string
|
||||
TileJobLogDir string
|
||||
}
|
||||
|
||||
func Load() (Config, error) {
|
||||
@@ -36,6 +50,14 @@ func Load() (Config, error) {
|
||||
if err != nil {
|
||||
return Config{}, fmt.Errorf("parse AUTO_SCAN_ON_START: %w", err)
|
||||
}
|
||||
tokenDefaultTTL, err := durationEnv("TOKEN_DEFAULT_TTL", 720*time.Hour)
|
||||
if err != nil {
|
||||
return Config{}, fmt.Errorf("parse TOKEN_DEFAULT_TTL: %w", err)
|
||||
}
|
||||
healthCheckInterval, err := durationEnv("HEALTH_CHECK_INTERVAL", 5*time.Minute)
|
||||
if err != nil {
|
||||
return Config{}, fmt.Errorf("parse HEALTH_CHECK_INTERVAL: %w", err)
|
||||
}
|
||||
|
||||
cfg := Config{
|
||||
Host: stringEnv("API_HOST", "127.0.0.1"),
|
||||
@@ -43,14 +65,23 @@ func Load() (Config, error) {
|
||||
APIBaseURL: stringEnv("API_BASE_URL", ""),
|
||||
TilePublicBaseURL: stringEnv("TILE_PUBLIC_BASE_URL", ""),
|
||||
VectorPublicBaseURL: stringEnv("VECTOR_PUBLIC_BASE_URL", ""),
|
||||
SQLitePath: stringEnv("SQLITE_PATH", filepath.Clean(filepath.Join("data", "map-asset-gateway.db"))),
|
||||
SQLitePath: stringEnv("SQLITE_PATH", filepath.Clean(filepath.Join("data", "basemap-pusher.db"))),
|
||||
BasemapScanRoot: stringEnv("BASEMAP_SCAN_ROOT", filepath.Clean(filepath.Join("data", "tiles"))),
|
||||
VectorScanRoot: stringEnv("VECTOR_SCAN_ROOT", filepath.Clean(filepath.Join("data", "geojson"))),
|
||||
AutoScanOnStart: autoScanOnStart,
|
||||
Env: stringEnv("APP_ENV", "dev"),
|
||||
ServiceName: stringEnv("SERVICE_NAME", "map-asset-gateway"),
|
||||
ServiceName: stringEnv("SERVICE_NAME", "basemap-pusher"),
|
||||
ServiceVersion: stringEnv("SERVICE_VERSION", "0.2.0"),
|
||||
Commit: stringEnv("SERVICE_COMMIT", "local"),
|
||||
AdminToken: strings.TrimSpace(stringEnv("ADMIN_TOKEN", "")),
|
||||
AdminUser: strings.TrimSpace(stringEnv("ADMIN_USER", "")),
|
||||
AdminPassword: stringEnv("ADMIN_PASSWORD", ""),
|
||||
CORSAllowedOrigins: listEnv("CORS_ALLOWED_ORIGINS"),
|
||||
TokenDefaultTTL: tokenDefaultTTL,
|
||||
HealthCheckInterval: healthCheckInterval,
|
||||
TileJobCommand: stringEnv("TILE_JOB_COMMAND", ""),
|
||||
TileJobVRTCommand: stringEnv("TILE_JOB_VRT_COMMAND", ""),
|
||||
TileJobLogDir: stringEnv("TILE_JOB_LOG_DIR", ""),
|
||||
}
|
||||
if cfg.APIBaseURL == "" {
|
||||
cfg.APIBaseURL = deriveBaseURL(cfg.Host, cfg.Port)
|
||||
@@ -68,13 +99,37 @@ func (c Config) ListenAddr() string {
|
||||
return fmt.Sprintf("%s:%d", c.Host, c.Port)
|
||||
}
|
||||
|
||||
// HasAdminCredentials reports whether the admin console has any credential
|
||||
// configured. Without one, the admin surface is only safe on loopback.
|
||||
func (c Config) HasAdminCredentials() bool {
|
||||
return c.AdminToken != "" || (c.AdminUser != "" && c.AdminPassword != "")
|
||||
}
|
||||
|
||||
// BindsToNonLoopback reports whether the listen address is reachable from
|
||||
// outside the host, which makes credentials mandatory.
|
||||
func (c Config) BindsToNonLoopback() bool {
|
||||
return !isLoopbackHost(c.Host)
|
||||
}
|
||||
|
||||
func deriveBaseURL(host string, port int) string {
|
||||
if host == "" || host == "0.0.0.0" {
|
||||
if host == "" || host == "0.0.0.0" || host == "::" {
|
||||
host = "127.0.0.1"
|
||||
}
|
||||
return fmt.Sprintf("http://%s:%d", host, port)
|
||||
}
|
||||
|
||||
func isLoopbackHost(host string) bool {
|
||||
host = strings.TrimSpace(host)
|
||||
if host == "" || strings.EqualFold(host, "localhost") {
|
||||
return true
|
||||
}
|
||||
ip := net.ParseIP(strings.Trim(host, "[]"))
|
||||
if ip == nil {
|
||||
return false
|
||||
}
|
||||
return ip.IsLoopback()
|
||||
}
|
||||
|
||||
func stringEnv(key, fallback string) string {
|
||||
value, ok := os.LookupEnv(key)
|
||||
if ok && value != "" {
|
||||
@@ -106,3 +161,29 @@ func boolEnv(key string, fallback bool) (bool, error) {
|
||||
}
|
||||
return fallback, nil
|
||||
}
|
||||
|
||||
func durationEnv(key string, fallback time.Duration) (time.Duration, error) {
|
||||
value, ok := os.LookupEnv(key)
|
||||
if ok && strings.TrimSpace(value) != "" {
|
||||
parsed, err := time.ParseDuration(strings.TrimSpace(value))
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return parsed, nil
|
||||
}
|
||||
return fallback, nil
|
||||
}
|
||||
|
||||
func listEnv(key string) []string {
|
||||
raw := strings.FieldsFunc(stringEnv(key, ""), func(r rune) bool {
|
||||
return r == ',' || r == ';' || r == '\n' || r == '\r' || r == '\t' || r == ' '
|
||||
})
|
||||
values := make([]string, 0, len(raw))
|
||||
for _, item := range raw {
|
||||
item = strings.TrimSpace(item)
|
||||
if item != "" {
|
||||
values = append(values, item)
|
||||
}
|
||||
}
|
||||
return values
|
||||
}
|
||||
|
||||
@@ -1,38 +0,0 @@
|
||||
@echo off
|
||||
setlocal
|
||||
chcp 65001 >nul 2>&1
|
||||
|
||||
set "ROOT=%~dp0"
|
||||
call "%ROOT%scripts\load-env.bat" "%ROOT%config\service.env"
|
||||
|
||||
set "LAUNCHER_DIR=%ROOT%temp\launcher"
|
||||
if not exist "%LAUNCHER_DIR%" mkdir "%LAUNCHER_DIR%" >nul 2>&1
|
||||
set "RS_API_PID_FILE=%LAUNCHER_DIR%\api.pid"
|
||||
del /q "%RS_API_PID_FILE%" >nul 2>&1
|
||||
|
||||
if "%API_HOST%"=="" set "API_HOST=127.0.0.1"
|
||||
if "%API_PORT%"=="" set "API_PORT=8910"
|
||||
if "%API_BASE_URL%"=="" set "API_BASE_URL=http://127.0.0.1:%API_PORT%"
|
||||
|
||||
echo ========================================
|
||||
echo Map Asset Gateway Launcher
|
||||
echo ========================================
|
||||
echo.
|
||||
echo Env file: %ROOT%config\service.env
|
||||
echo Starting Go service in a separate window...
|
||||
echo.
|
||||
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -Command "Start-Process -FilePath 'cmd.exe' -ArgumentList '/k','call ""%ROOT%scripts\run-api.bat""' -WorkingDirectory '%ROOT%' | Out-Null"
|
||||
|
||||
echo ========================================
|
||||
echo Management Entry
|
||||
echo ========================================
|
||||
echo.
|
||||
echo Admin Console : %API_BASE_URL%/admin/
|
||||
echo API Health : %API_BASE_URL%/health
|
||||
echo API Ready : %API_BASE_URL%/ready
|
||||
echo Stop All : %ROOT%stop-all.bat
|
||||
echo PID Files : %LAUNCHER_DIR%
|
||||
echo.
|
||||
|
||||
if "%NO_PAUSE%"=="" pause
|
||||
-101
@@ -1,101 +0,0 @@
|
||||
@echo off
|
||||
setlocal
|
||||
chcp 65001 >nul 2>&1
|
||||
|
||||
set "ROOT=%~dp0"
|
||||
if exist "%ROOT%scripts\load-env.bat" call "%ROOT%scripts\load-env.bat" "%ROOT%config\service.env"
|
||||
set "LAUNCHER_DIR=%ROOT%temp\launcher"
|
||||
if "%API_PORT%"=="" set "API_PORT=8910"
|
||||
|
||||
echo ========================================
|
||||
echo Map Asset Gateway Stopper
|
||||
echo ========================================
|
||||
echo.
|
||||
|
||||
call :stop_pid_file "Map Asset Gateway" "%LAUNCHER_DIR%\api.pid"
|
||||
if not "%STOP_RESULT%"=="stopped" call :stop_port "Map Asset Gateway" "%API_PORT%"
|
||||
call :stop_window_title "Map Asset Gateway"
|
||||
|
||||
echo.
|
||||
echo Finished.
|
||||
if "%NO_PAUSE%"=="" pause
|
||||
exit /b 0
|
||||
|
||||
:stop_pid_file
|
||||
set "STOP_RESULT=missing"
|
||||
set "SERVICE_NAME=%~1"
|
||||
set "PID_FILE=%~2"
|
||||
if not exist "%PID_FILE%" (
|
||||
echo [SKIP] %SERVICE_NAME% not running
|
||||
set "SERVICE_NAME="
|
||||
set "PID_FILE="
|
||||
exit /b 0
|
||||
)
|
||||
|
||||
set "TARGET_PID="
|
||||
for /f "usebackq delims=" %%I in ("%PID_FILE%") do (
|
||||
set "TARGET_PID=%%I"
|
||||
goto :pid_loaded
|
||||
)
|
||||
:pid_loaded
|
||||
if not defined TARGET_PID (
|
||||
echo [SKIP] %SERVICE_NAME% not running
|
||||
del /q "%PID_FILE%" >nul 2>&1
|
||||
set "SERVICE_NAME="
|
||||
set "PID_FILE="
|
||||
set "TARGET_PID="
|
||||
exit /b 0
|
||||
)
|
||||
|
||||
call :kill_pid "%TARGET_PID%"
|
||||
if not "%KILL_OK%"=="1" (
|
||||
echo [SKIP] %SERVICE_NAME% not running
|
||||
) else (
|
||||
echo [OK] Stopped %SERVICE_NAME% PID=%TARGET_PID%
|
||||
set "STOP_RESULT=stopped"
|
||||
)
|
||||
|
||||
del /q "%PID_FILE%" >nul 2>&1
|
||||
set "SERVICE_NAME="
|
||||
set "PID_FILE="
|
||||
set "TARGET_PID="
|
||||
exit /b 0
|
||||
|
||||
:stop_port
|
||||
set "SERVICE_NAME=%~1"
|
||||
set "TARGET_PORT=%~2"
|
||||
set "FOUND_PID="
|
||||
for /f "tokens=5" %%I in ('netstat -ano -p TCP ^| findstr /R /C:":%TARGET_PORT% .*LISTENING"') do set "FOUND_PID=%%I"
|
||||
if not defined FOUND_PID (
|
||||
set "SERVICE_NAME="
|
||||
set "TARGET_PORT="
|
||||
exit /b 0
|
||||
)
|
||||
|
||||
call :kill_pid "%FOUND_PID%"
|
||||
if "%KILL_OK%"=="1" (
|
||||
echo [OK] Stopped %SERVICE_NAME% by port %TARGET_PORT% PID=%FOUND_PID%
|
||||
set "STOP_RESULT=stopped"
|
||||
)
|
||||
set "SERVICE_NAME="
|
||||
set "TARGET_PORT="
|
||||
set "FOUND_PID="
|
||||
exit /b 0
|
||||
|
||||
:stop_window_title
|
||||
taskkill /FI "WINDOWTITLE eq %~1*" /T /F >nul 2>&1
|
||||
exit /b 0
|
||||
|
||||
:kill_pid
|
||||
set "KILL_OK=0"
|
||||
set "KILL_TARGET_PID=%~1"
|
||||
if "%KILL_TARGET_PID%"=="" exit /b 0
|
||||
taskkill /PID %KILL_TARGET_PID% /T /F >nul 2>&1
|
||||
if errorlevel 1 (
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -Command "try { Stop-Process -Id %KILL_TARGET_PID% -Force -ErrorAction Stop; exit 0 } catch { exit 1 }" >nul 2>&1
|
||||
if not errorlevel 1 set "KILL_OK=1"
|
||||
) else (
|
||||
set "KILL_OK=1"
|
||||
)
|
||||
set "KILL_TARGET_PID="
|
||||
exit /b 0
|
||||
Reference in New Issue
Block a user