diff --git a/.gitignore b/.gitignore index a05a314..6c0cfaa 100644 --- a/.gitignore +++ b/.gitignore @@ -1,32 +1,31 @@ -# Local machine configuration -config/service.env - -# Runtime data and generated assets +# --- Runtime data (never commit) ------------------------------------------- +# data/ holds the production SQLite catalog, the tile tree and the vector +# assets. It is machine state, not source. data/ -logs/ -temp/ -.tmp/ -# Go local caches +# --- Build and toolchain caches -------------------------------------------- .gocache/ .gomodcache/ -.gotmp/ .gopath/ +.gotmp/ +bin/ +temp/ +.tmp/ +logs/ -# Python cache +# --- Machine-specific config ----------------------------------------------- +# config/service.env carries absolute paths for one machine; the .example file +# is the template to copy from. +config/service.env + +# --- Tooling artifacts ------------------------------------------------------ +.claude/ +.codex/ +.impeccable/ __pycache__/ *.pyc -*.pyo -# Build and test artifacts -*.exe -*.test -*.out -*.err -*.log -*.pid -coverage.out - -# OS / editor noise -.DS_Store +# --- OS/editor noise -------------------------------------------------------- Thumbs.db +Desktop.ini +*.swp \ No newline at end of file diff --git a/PRODUCT.md b/PRODUCT.md new file mode 100644 index 0000000..4800bc4 --- /dev/null +++ b/PRODUCT.md @@ -0,0 +1,68 @@ +# Product + + + +## Platform + +web + +## Stack + +既有代码已确定：Go 1.26 标准库 HTTP 服务 + `modernc.org/sqlite`，运维界面是编译进二进制的单个 Go HTML 模板（`services/api-go/internal/basemapapi/admin_template.go`），无构建步骤、无 JS 框架。 + +*（决策来源：用户授权按推荐执行）* 运维界面继续维持"零依赖、单文件内嵌、无前端构建"的约束——单个 exe 可直接部署是本项目最大的运维优势，引入前端工具链会破坏它。允许使用原生 JS 与内联 CSS。 + +## Users + +**主要用户：内部 GIS / 运维工程师。** 场景是日常运维与排障——盘点和核对底图、矢量资源，签发和回收访问令牌，处理推送失败，查看扫描与请求的异常。他们熟悉瓦片、XYZ 模板、token 这些概念，不需要教学，需要的是**一眼看到状态、一步完成操作**。 + +**次要用户：下游系统对接方。** 通过 token 拉取瓦片/矢量，或接收推送回调；他们主要消费 API 与回调负载，只在排查对接问题时才会打开运维台。 + +## Product Purpose + +`basemap-pusher` 是底图与矢量资源的**编目与分发网关**。它不生成数据，只做四件事：扫描既有目录、编目、签发令牌、按令牌放行下载与推送描述。 + +存在的意义是把"谁能取哪张底图"这件事从散落的目录和口头约定，收敛成一份可审计的台账。成功 = 运维能自己回答"这张底图现在能不能取、谁有权限、上周谁改过"，而不必去翻服务器文件系统。 + +## Positioning + +**它刻意不做切片。** 与上一代成果管理服务（PostgreSQL/PostGIS + Python worker + 切片流水线）相反，本服务把"生产"和"分发"切开：GIS 软件负责出片，产物落到 `data/tiles//z/x/y.ext`，网关只扫、只编目、只分发。 + +这个机制的价值是**部署面极小**：单个 Go 二进制 + 一个 SQLite 文件 + 两个目录。相邻的"成果平台"做不到这一点。 + +## Operating Context + +- 数据以**目录即真相**：`BASEMAP_SCAN_ROOT` 下的目录结构就是事实来源，启动扫描或手动重扫即刷新台账。 +- 两种底图目录风格：带 `manifest.json` 的目录，和裸 `z/x/y.ext` 目录（后者按目录名自动入库）。 +- 矢量资源在 `VECTOR_SCAN_ROOT` 下递归扫描 `.geojson` / `.json` / `.zip` / `.shp`（shapefile 按需打包下载）。 +- 访问令牌通过 `Authorization: Bearer`、`X-Access-Token`、`token=` 查询参数投递——查询参数对瓦片客户端（地图库直接拼 URL）是必需的，不是可选项。 +- 推送是向下游系统发 HTTP 回调，携带资源描述符。 +- 部署形态：Windows 批处理脚本启停（`start-all.bat` / `stop-all.bat`），默认只绑 `127.0.0.1`。 + +## Capabilities and Constraints + +**已确认能力**：目录扫描与编目、默认版本管理、令牌签发/停用/有效期、令牌鉴权取图、矢量下载（shapefile 动态打包）、兼容路由（`/geojson/...`、`/shp/...`、无版本号的瓦片路由）、下游推送、访问日志与 Prometheus 指标（`/metrics`）。管理面支持 HTTP Basic（浏览器）与 Bearer（API）。 + +**已确认约束**： +- 存储是单文件 SQLite，单实例部署；扫描锁是进程内的。 +- 运维界面必须能表达**失败与待处理**，而不只是库存数量。 +- 不引入前端构建步骤。 +- **明确不做**：栅格切片、重投影、金字塔构建。影像转底图属于上游 GIS 工具的职责。 + +**明确未决**：影像切片是否以"作业化调用外部 GDAL"的形式纳入（已记为后续候选，当前不在范围内）。 + +## Evidence on Hand + +真实的底图目录与矢量文件在 `data/` 下（`data/tiles/` 含 gaode_image / gaode_shp / google_image，实为 webp 瓦片；`data/geojson/` 含一份约 53MB 的 GeoJSON）。这些是真实数据，可用于验证界面，但**不得**在界面文案里虚构容量、QPS、SLA 或客户信息。 + +## Product Principles + +1. **状态先于库存。** 打开界面第一眼要看到"有没有东西在报警、有什么待处理"，而不是资源总数。 +2. **台账要能验真。** 每一条资源都应能就地验证（看一眼图、复制 XYZ 模板、试拉一张），而不是只列一条记录。 +3. **危险操作要设防。** 停用令牌、重扫、改默认版本都会影响线上取图，必须有确认与后果说明。 +4. **生产与分发分离。** 任何新能力都要问一句：这是不是把"生产"塞进了网关？默认答案是不要。 +5. **小部署面是特性，不是妥协。** 新增依赖前先证明它值得牺牲单文件部署。 + +## Accessibility & Inclusion + +内部工具，无正式合规要求。仍应满足：键盘可达、焦点可见、正文对比度 ≥ 4.5:1、不用纯颜色单独承载状态（状态同时有文字）。 \ No newline at end of file diff --git a/README.md b/README.md index 1969ccd..abc1fac 100644 --- a/README.md +++ b/README.md @@ -1,129 +1,42 @@ -# map-asset-gateway +# Basemap Pusher -`map-asset-gateway` 是一个面向现有底图瓦片和矢量文件的目录管理与分发服务。 +An internal catalog and push service for existing basemap tiles, GeoJSON files, and shapefiles. -它不负责生成瓦片，不负责 GIS 生产流程，也不负责旧版遥感结果处理流水线。它只做这几件事： +The service does not generate tiles, does not rebuild vector data, and does not bring back the old result-management pipeline. It only does: -- 扫描已有的底图瓦片目录 -- 扫描已有的 GeoJSON / Shapefile 文件 -- 将元数据写入 SQLite -- 为底图和矢量资源签发访问 token -- 按 token 提供 catalog、瓦片访问和矢量下载 -- 将底图或矢量描述推送给下游系统 +- scan existing basemap tile directories +- scan existing vector files +- manage metadata +- issue tokens +- control token access to basemaps and vector assets +- push basemap or vector descriptors to downstream systems -## 技术栈 +## Stack - Go - SQLite -- Go 标准库 `net/http` +- standard library HTTP server -## 适用场景 +## Current Capabilities -适合已经有离线瓦片目录、GeoJSON、SHP 数据，希望补一层统一管理、授权和分发接口的场景。 +- scan `BASEMAP_SCAN_ROOT` into SQLite +- scan `VECTOR_SCAN_ROOT` into SQLite +- support two basemap directory styles + - manifest-based directories + - raw `z/x/y.ext` tile roots +- support vector asset formats + - `.geojson` + - `.json` + - `.shp` plus sidecar files + - `.zip` shapefile packages +- token-protected basemap catalog, tile access, vector catalog, and vector download +- token-protected compatibility routes for default-version tiles, GeoJSON paths, and shapefile paths +- admin panel at `GET /admin/` +- downstream callback push for basemaps and vectors -不适合下面这些需求： +## Basemap Scan Rules -- 在线切片或重建瓦片 -- 栅格处理、预览图生成、图例生成 -- PostgreSQL / PostGIS 任务流水线 -- Python worker 驱动的结果管理系统 - -## 运行架构 - -服务启动时会按下面的顺序运行： - -1. 读取 `config/service.env` 或环境变量 -2. 打开 SQLite 并自动建表 -3. 注册默认扫描源 -4. 如果 `AUTO_SCAN_ON_START=true`，自动扫描底图和矢量目录 -5. 启动 HTTP API 和管理页 - -这是一个单进程服务，没有额外 worker，没有外部数据库依赖。 - -## 目录结构 - -```text -. -├─ services/api-go/ Go 服务源码 -├─ config/ 环境变量配置 -├─ scripts/ 启动脚本 -├─ docs/ 接入文档 -├─ data/ 本地数据目录，不建议提交到 Git -├─ logs/ 本地运行日志，不建议提交到 Git -└─ temp/ 本地临时文件，不建议提交到 Git -``` - -说明： - -- `services/api-go` 是当前项目主体。 -- 历史遗留 demo 已从仓库中移除，避免和当前服务边界混淆。 -- `data/`、`logs/`、`temp/`、`.gocache/`、`.gomodcache/`、`.gotmp/`、`.gopath/` 都属于本地运行或构建产物。 - -## 快速启动 - -### 前置条件 - -- Go 1.26 或兼容版本 -- 可写的本地目录用于放 SQLite、扫描根目录和 Go 缓存 -- Windows 环境下可直接使用仓库自带批处理脚本 - -### 配置 - -复制示例配置： - -```text -config/service.env.example -> config/service.env -``` - -关键配置项： - -| 变量 | 说明 | -| --- | --- | -| `API_HOST` | 监听地址 | -| `API_PORT` | 监听端口 | -| `API_BASE_URL` | API 对外基地址 | -| `TILE_PUBLIC_BASE_URL` | 底图下载地址前缀 | -| `VECTOR_PUBLIC_BASE_URL` | 矢量下载地址前缀 | -| `SQLITE_PATH` | SQLite 文件路径 | -| `BASEMAP_SCAN_ROOT` | 底图扫描根目录 | -| `VECTOR_SCAN_ROOT` | 矢量扫描根目录 | -| `AUTO_SCAN_ON_START` | 启动时是否自动扫描 | - -默认示例见 [config/service.env.example](./config/service.env.example)。 - -### 启动 - -Windows 下推荐： - -```powershell -.\start-all.bat -``` - -或者直接运行 API： - -```powershell -.\scripts\run-api.bat -``` - -停止： - -```powershell -.\stop-all.bat -``` - -启动后可访问： - -- 管理页: `http://127.0.0.1:8910/admin/` -- 健康检查: `http://127.0.0.1:8910/health` -- 就绪检查: `http://127.0.0.1:8910/ready` - -## 数据组织与扫描规则 - -### 底图目录 - -服务支持两种底图目录形式。 - -### 1. 带 `manifest.json` 的目录 +### Manifest-based directory ```text data/tiles/gaode-image/ @@ -133,7 +46,7 @@ data/tiles/gaode-image/ 1/0/0.png ``` -示例 `manifest.json`： +Example `manifest.json`: ```json { @@ -151,7 +64,7 @@ data/tiles/gaode-image/ } ``` -### 2. 直接是 `z/x/y.ext` 的裸目录 +### Raw tile directory ```text data/tiles/google-image/ @@ -159,22 +72,17 @@ data/tiles/google-image/ 1/0/0.webp ``` -这类目录会自动导入为： +This is imported automatically as: -- `code = 目录名归一化后的值` +- `code = directory name` - `version = current` - `type = xyz` -说明： +## Vector Scan Rules -- `google-image`、`google_image`、`google.image` 最终都会被归一化成类似 `google-image` 的内部 code。 -- 默认版本路由 `/tiles/{code}/{z}/{x}/{y}.ext` 会解析到该底图当前默认版本。 +The service scans `VECTOR_SCAN_ROOT` recursively. -### 矢量目录 - -`VECTOR_SCAN_ROOT` 会被递归扫描。 - -示例： +Examples: ```text data/geojson/admin-boundary.geojson @@ -184,95 +92,37 @@ data/geojson/landuse/parcel.dbf data/geojson/landuse/parcel.shx ``` -扫描规则： +Rules: -- `.geojson` 和 `.json` 会被识别为 `geojson` -- `.zip` 会被识别为 `shpzip` -- `.shp` 会被识别为 `shapefile` -- `shapefile` 下载时会按同名 sidecar 文件动态打包成 zip +- `.geojson` and `.json` are imported as `geojson` +- `.zip` is imported as `shpzip` +- `.shp` is imported as `shapefile` +- shapefile downloads are served as zip packages on demand -## 访问控制 +## Token Model -服务使用 token 做资源访问控制。 +A token can be granted access to: -一个 token 可以授权： +- one or more basemaps +- one or more vector assets -- 一个或多个底图 -- 一个或多个矢量资源 - -支持的 token 传递方式： +Supported token delivery: - `Authorization: Bearer ` - `X-Access-Token: ` -- 查询参数 `token=` -- 查询参数 `access_token=` +- query parameter `token=` +- query parameter `access_token=` -## 典型使用流程 +## API -### 1. 启动服务并扫描目录 - -如果开启了 `AUTO_SCAN_ON_START=true`，服务会在启动时自动扫描。 - -也可以通过管理接口手动触发： - -- `POST /api/v1/admin/scan-sources/{code}/run` -- `POST /api/v1/admin/vectors/rescan` - -### 2. 创建 token - -示例： - -```json -POST /api/v1/admin/tokens -{ - "name": "insar-v2", - "basemap_codes": ["google-image"], - "vector_codes": ["全国行政区.geojson", "landuse/parcel.shp"] -} -``` - -### 3. 访问 catalog 或资源 - -```text -GET /api/v1/catalog/basemaps?token=bmp_xxx -GET /api/v1/catalog/vectors?token=bmp_xxx -GET /tiles/google-image/10/843/388.webp?token=bmp_xxx -GET /tiles/google-image/current/10/843/388.webp?token=bmp_xxx -GET /geojson/全国行政区.geojson?token=bmp_xxx -GET /shp/landuse/parcel.shp?token=bmp_xxx -GET /vectors/landuse-parcel/download?token=bmp_xxx -``` - -说明： - -- `/tiles/{code}/{z}/{x}/{y}.ext` 走默认版本兼容路由 -- `/tiles/{code}/{version}/{z}/{x}/{y}.ext` 显式指定版本 -- `/geojson/{...}` 支持按原始相对路径、文件名或内部 code 解析 -- `/shp/{...}` 会把 `.shp + .dbf + .shx ...` 打成 zip 返回 - -### 4. 推送给下游系统 - -先注册目标系统： - -- `GET /api/v1/admin/targets` -- `POST /api/v1/admin/targets` - -再触发推送： - -- `POST /api/v1/admin/pushes` -- `POST /api/v1/admin/vector-pushes` - -推送内容是底图版本或矢量资源的描述信息，不是资源文件本体上传。 - -## API 概览 - -### 系统接口 +### System - `GET /health` - `GET /ready` +- `GET /metrics` (Prometheus text: request, tile, tile-miss and auth counters) - `GET /api/v1/system` -### 管理接口 +### Admin - `GET /admin/` - `GET /api/v1/admin/dashboard` @@ -293,7 +143,7 @@ GET /vectors/landuse-parcel/download?token=bmp_xxx - `GET /api/v1/admin/vector-push-records` - `POST /api/v1/admin/vector-pushes` -### Catalog 与下载接口 +### Catalog and Downloads - `GET /api/v1/catalog/basemaps` - `GET /api/v1/catalog/basemaps/{code}` @@ -305,7 +155,197 @@ GET /vectors/landuse-parcel/download?token=bmp_xxx - `GET /shp/{relative/path/to/file.shp}` - `GET /vectors/{code}/download` -## 数据库表 +Examples: + +```text +GET /tiles/gaode_image/10/843/388.png?token=bmp_xxx +GET /tiles/gaode-image/current/10/843/388.png?token=bmp_xxx +GET /geojson/全国行政区.geojson?token=bmp_xxx +GET /shp/landuse/parcel.shp?token=bmp_xxx +GET /vectors/admin-boundary/download?token=bmp_xxx +``` + +Compatibility route notes: + +- `/tiles/{code}/{z}/{x}/{y}.ext` resolves the current default version of the basemap +- `/geojson/{...}` resolves vector assets by original relative path, file name, or internal code +- `/shp/{...}` returns `.shp` packages as zip downloads and also supports `.zip` vector packages +- token creation accepts vector references by code or by original relative path + +## Slicing API for Other Systems + +Integrations call a narrow, non-admin surface: + +```text +POST /api/v1/slice/jobs submit a job -> 202 + the job object +GET /api/v1/slice/jobs/{id} poll one job -> 200 + the job object +GET /api/v1/slice/layers/{code}/sources the layer's source rasters +``` + +All three require a service token carrying the **`slice`** scope. Create one +through the admin API or console; a slice-only token needs no basemap or vector +grants, so it cannot read any tiles: + +```json +POST /api/v1/admin/tokens +{"name": "integration", "scopes": ["slice"]} +``` + +- **Accumulating layers** — a layer is the union of its source rasters, tracked in + `layer_sources`. Submit **one** new source to append a flight (fast, what + overlaps wins per tile), or submit **all** of `GET /api/v1/slice/layers/{code}/sources` + to rebuild the layer as a VRT mosaic (per-pixel, no tile seams). The console's + 影像切片 view lists each layer's sources. +- `TILE_JOB_COMMAND` slices one input; `TILE_JOB_VRT_COMMAND` runs first when + several sources are submitted. `{inputs}` expands to one argument per source, + and `{output_vrt}` is the temporary mosaic. Both ship as wrappers over the GDAL + bundled with QGIS (`scripts/gdal2tiles-qgis.bat`, + `scripts/gdalbuildvrt-qgis.bat`) because the Python bindings need the OSGeo4W + environment; set `GDAL2TILES_QGIS_ROOT` when QGIS is not in `%ProgramFiles%`. + The slicer default passes `--exclude`, so a new flight cannot punch transparent + holes into the previous imagery. +- **Rebuild from the console** — 影像切片 → 图层源清单 → 重建为镶嵌 slices every + recorded source into one VRT, reusing the zoom range the layer is already + published at. No need to copy paths by hand. +- **Known limit** — a layer's source list only grows (there is no delete API yet), + and a rebuild never clears the output directory. If a source is ever removed by + hand, tiles it used to cover but no longer does stay behind; delete the layer + directory and rebuild to clean up. +- **Idempotent submit** — sending the same `code` twice returns the existing job + instead of queueing a second one, so a retrying caller is safe. +- **Completion callback** — pass `callback_url` when submitting and the service + POSTs `{"event":"tile.job.finished","job":{...}}` when the job ends. Failures + retry on the same 30s/60s backoff as push callbacks. +- Jobs run one at a time, oldest first; the admin console's 影像切片 view shows + the same queue. +## Resource Health, Retries and Slicing + +- **Resource health** — every catalogued tile root and vector file is stat'ed on + `HEALTH_CHECK_INTERVAL` and on demand (`POST /api/v1/admin/health/check`). + A path that vanished is recorded as `missing` and surfaces in the console's + 待处理 list instead of waiting for a client 404. Health is tracked separately + from `status`: status is a catalog fact, health is whether the bytes are on + disk right now. +- **Push retries** — a failed callback is retried up to 3 times with backoff + (30s, 60s) and stays `retrying` while queued; the schedule lives in the + database so a restart does not lose it. A finished record can be re-queued + with `POST /api/v1/admin/push-records/{id}/retry` (or + `.../vector-push-records/{id}/retry`). +- **Image slicing** — `scripts/gdal2tiles-qgis.bat` runs the GDAL bundled with + QGIS (its Python bindings need the OSGeo4W environment, so a bare + `gdal2tiles.exe` will not work; set `GDAL2TILES_QGIS_ROOT` if QGIS is not in + `%ProgramFiles%`). `POST /api/v1/admin/tile-jobs` queues a job that runs + `TILE_JOB_COMMAND` (see [config/service.env.example](./config/service.env.example)) + and, on success, rescans so the sliced basemap appears in the catalog. The + service does the orchestration; the raster work is done by GDAL. The service + account needs write access to the basemap scan root. +## Running as a Windows Service + +The binary manages its own service registration — no NSSM or WinSW wrapper: + +```text +basemap-pusher.exe --env-file install +basemap-pusher.exe start +basemap-pusher.exe status +basemap-pusher.exe stop +basemap-pusher.exe uninstall +``` + +- `install` bakes `--env-file` into the service command line, because the SCM + starts services in `C:\Windows\System32` and the app would otherwise miss its + config. Without `--env-file` it looks for `config/service.env` next to the + executable and up to two directories above it. +- A service has no console, so the standard logger is redirected to + `bin\logs\basemap-pusher.log`, rolling to `.1` at 10 MB. Override the path with + the `SERVICE_LOG_FILE` environment variable. Without this the service would + discard every log line. +- Recovery actions are registered on install: restart after 5s, 15s, then 60s, + retrying for a day. The service starts automatically at boot. +- `install` / `uninstall` / `start` / `stop` need an **elevated** console + (Run as Administrator); `status` and the console mode do not. +- Started with no verb the binary runs as a normal console app and handles + Ctrl+C / SIGTERM with a 15s drain. Under the SCM the service handler owns the + same shutdown path. + +Build the binary the service will run (it must not depend on the Go toolchain): + +```text +scripts\build-api.bat +``` + +The script loads `config/service.env` first, so the Go caches and `GOPROXY` +match the rest of the project instead of the global defaults. Running `go build` +by hand from the repository root will fail: the module lives in +`services\api-go`, not at the root. +## Access Control + +The admin console and `/api/v1/admin/*` can mint tokens that read every +basemap, so they are guarded: + +- configure `ADMIN_USER` + `ADMIN_PASSWORD` for the browser console (HTTP Basic) +- configure `ADMIN_TOKEN` for API clients (`Authorization: Bearer `) +- with neither set, the service **refuses to start** unless `API_HOST` is a + loopback address (`127.0.0.1`, `::1`, `localhost`) + +Cross-origin requests are denied by default; add the exact origins you trust to +`CORS_ALLOWED_ORIGINS`. Client tokens never appear in redirect URLs, and tokens +carried in a query string are redacted from the access log. +## Config + +See [config/service.env.example](./config/service.env.example). + +For `Insar_management_system_v2` integration, see [docs/INSAR_MANAGEMENT_SYSTEM_V2.md](./docs/INSAR_MANAGEMENT_SYSTEM_V2.md). + +Important variables: + +- `API_HOST` +- `API_PORT` +- `API_BASE_URL` +- `TILE_PUBLIC_BASE_URL` +- `VECTOR_PUBLIC_BASE_URL` +- `SQLITE_PATH` +- `BASEMAP_SCAN_ROOT` +- `VECTOR_SCAN_ROOT` +- `AUTO_SCAN_ON_START` +- `ADMIN_USER` / `ADMIN_PASSWORD` (HTTP Basic for the browser console) +- `ADMIN_TOKEN` (bearer token for admin API clients) +- `CORS_ALLOWED_ORIGINS` (comma-separated allow-list; empty means no CORS) +- `TOKEN_DEFAULT_TTL` (lifetime for new tokens, e.g. `720h`; `0` disables expiry) +- `HEALTH_CHECK_INTERVAL` (resource availability pass, e.g. `5m`; `0` disables) +- `TILE_JOB_COMMAND` (image slicing command template; needs GDAL) +- `TILE_JOB_LOG_DIR` (per-job log directory; empty = `/.tile-jobs`) + +## Start + +### Windows service (production) + +The service is the supported way to run this on Windows — it starts at boot, +restarts on failure and keeps its own log file: + +```text +scripts\build-api.bat +scripts\service.bat install (elevated) +scripts\service.bat start +scripts\service.bat status +``` + +`service.bat` accepts `install | uninstall | start | stop | restart | status` +and forwards to `bin\basemap-pusher.exe`. The install / uninstall / start / stop +verbs need an **elevated** shell; `status` does not. + +The old `start-all.bat` / `stop-all.bat` launchers were removed: they killed the +process by port with `taskkill`, which fought the service's restart policy. + +### Console (development) + +```text +scripts\run-api.bat +``` + +This runs `go run` with the project's own caches, for a quick local loop. It is +not supervised and does not survive a reboot. + +## Database Tables - `basemaps` - `basemap_versions` @@ -319,80 +359,12 @@ GET /vectors/landuse-parcel/download?token=bmp_xxx - `vector_token_grants` - `vector_push_records` -## 提交到 Git 前的约定 +## Design Boundaries -这个仓库里有很多目录本质上是本地运行数据或缓存，不建议提交到版本库。 +- no Python worker +- no PostgreSQL +- no tile generation pipeline +- no result artifact management +- no raster processing jobs -默认不建议提交： - -- `data/` -- `logs/` -- `temp/` -- `.tmp/` -- `.gocache/` -- `.gomodcache/` -- `.gotmp/` -- `.gopath/` -- `config/service.env` - -原因： - -- `data/` 里通常是真实底图、真实 GeoJSON/SHP、以及本地 SQLite 数据库 -- `logs/` 和 `temp/` 是运行期产物 -- `.gocache/`、`.gomodcache/`、`.gotmp/`、`.gopath/` 是本地 Go 构建缓存 -- `config/service.env` 往往包含机器本地路径 - -如果需要给新同事一个可运行的仓库，建议提交： - -- 源码 -- `config/service.env.example` -- 文档 -- 一个最小化的目录约定说明 - -而不是直接提交大体积瓦片、数据库和本地缓存。 - -## 开发与测试 - -在仓库内使用本地 Go 缓存目录跑测试： - -```powershell -$env:GOCACHE='C:\path\to\map-asset-gateway\.gocache' -$env:GOMODCACHE='C:\path\to\map-asset-gateway\.gomodcache' -$env:GOTMPDIR='C:\path\to\map-asset-gateway\.gotmp' -$env:GOPATH='C:\path\to\map-asset-gateway\.gopath' -$env:GOPROXY='https://goproxy.cn,direct' -$env:GOSUMDB='off' -go test ./... -``` - -当前测试主要覆盖： - -- 底图扫描 -- 矢量扫描 -- token 鉴权 -- 默认版本瓦片路由 -- GeoJSON / SHP 兼容访问路径 - -## 相关文档 - -- [config/service.env.example](./config/service.env.example) - -## 项目边界 - -再次强调，这个项目当前的边界是： - -- 扫描 -- 建目录 -- 发 token -- 鉴权 -- 下载 -- 推送资源描述 - -不包括： - -- Python worker -- PostgreSQL -- PostGIS -- 在线切片 -- 栅格处理任务 -- 结果资产管理流水线 +GIS software still produces the tiles and vector files. This project only scans, catalogs, authorizes, downloads, and pushes descriptors. diff --git a/config/service.env.example b/config/service.env.example index fa22626..171e913 100644 --- a/config/service.env.example +++ b/config/service.env.example @@ -2,7 +2,7 @@ APP_ENV=dev -SERVICE_NAME=map-asset-gateway +SERVICE_NAME=basemap-pusher SERVICE_VERSION=0.2.0 SERVICE_COMMIT=local @@ -12,14 +12,56 @@ API_BASE_URL=http://127.0.0.1:8910 TILE_PUBLIC_BASE_URL=http://127.0.0.1:8910 VECTOR_PUBLIC_BASE_URL=http://127.0.0.1:8910 -SQLITE_PATH=C:\path\to\map-asset-gateway\data\map-asset-gateway.db -BASEMAP_SCAN_ROOT=C:\path\to\map-asset-gateway\data\tiles -VECTOR_SCAN_ROOT=C:\path\to\map-asset-gateway\data\geojson +SQLITE_PATH=Z:\Code\tile-server\data\basemap-pusher.db +BASEMAP_SCAN_ROOT=Z:\Code\tile-server\data\tiles +VECTOR_SCAN_ROOT=Z:\Code\tile-server\data\geojson AUTO_SCAN_ON_START=true -GOCACHE=C:\path\to\map-asset-gateway\.gocache -GOMODCACHE=C:\path\to\map-asset-gateway\.gomodcache -GOTMPDIR=C:\path\to\map-asset-gateway\.gotmp -GOPATH=C:\path\to\map-asset-gateway\.gopath +GOCACHE=Z:\Code\tile-server\.gocache +GOMODCACHE=Z:\Code\tile-server\.gomodcache +GOTMPDIR=Z:\Code\tile-server\.gotmp +GOPATH=Z:\Code\tile-server\.gopath GOPROXY=https://goproxy.cn,direct GOSUMDB=off + +# --- Admin access --------------------------------------------------------- +# The admin console can mint tokens that read every basemap, so it must be +# protected whenever the service is reachable off-host. Binding to a non +# loopback API_HOST without credentials below makes startup fail on purpose. +# +# Browser console: set ADMIN_USER/ADMIN_PASSWORD (HTTP Basic). +# API clients: set ADMIN_TOKEN (Authorization: Bearer ). +# Leaving both empty is only allowed on 127.0.0.1 / ::1 / localhost. +#ADMIN_USER=ops +#ADMIN_PASSWORD=change-me +#ADMIN_TOKEN=change-me-too + +# Comma-separated list of browser origins allowed to call the API. Empty means +# no cross-origin access at all; "*" allows every origin. +#CORS_ALLOWED_ORIGINS=https://maps.example.com + +# Lifetime applied to newly created tokens when no explicit expiry is given. +# Use 0 to create tokens that never expire. +TOKEN_DEFAULT_TTL=720h + +# --- Resource health & tile jobs ----------------------------------------- +# How often catalogued paths are re-stat'ed so a moved/deleted resource is +# flagged before a client hits a 404. Use 0 to disable the periodic pass +# (the "立即巡检资源" button still works). +HEALTH_CHECK_INTERVAL=5m + +# Command template that slices a raster into XYZ tiles. Placeholders: +# {input} {output} {zmin} {zmax} {tilesize} {format} {code} +# scripts/gdal2tiles-qgis.bat locates the GDAL bundled with QGIS (the Python +# bindings need the OSGeo4W environment, so a bare gdal2tiles.exe will not work). +TILE_JOB_COMMAND=D:\Code\tile-server\scripts\gdal2tiles-qgis.bat -p mercator -z {zmin}-{zmax} --xyz -w none --tilesize={tilesize} --exclude {input} {output} + +# Where per-job stdout/stderr is written. Empty means /.tile-jobs. +#TILE_JOB_LOG_DIR= + +# Service tokens may carry the "slice" scope so an integration can submit tile +# jobs through POST /api/v1/slice/jobs without holding the admin credential. + +# 多源镶嵌：提交多个源影像时，先用它拼成一张 VRT 再切片（逐像素拼接， +# 消除重叠区的瓦片接缝）。{inputs} 会展开成每个源各一个参数。 +TILE_JOB_VRT_COMMAND=D:\Code\tile-server\scripts\gdalbuildvrt-qgis.bat -resolution highest {output_vrt} {inputs} diff --git a/demo/README.md b/demo/README.md new file mode 100644 index 0000000..8f1d66d --- /dev/null +++ b/demo/README.md @@ -0,0 +1,62 @@ +# Demo Scripts + +This folder contains runnable demos for the internal remote-sensing result service. + +## Prerequisites + +- PostgreSQL and PostGIS are available to the API through `config/service.env`. +- The Python environment is available at `PYTHON_EXE`. +- Go is installed and can run `go run`. + +## Service Startup + +You can start the stack manually from the repository root: + +```powershell +.\start-all.bat +``` + +Or use the self-contained full-flow demo to start a temporary stack on demo ports: + +```powershell +.\demo\demo_full_flow.ps1 -StartStack -StartCallbackReceiver +``` + +## Demo List + +- `demo_import_and_slice.ps1` + Imports a raster result, binds a style, and triggers COG, preview, legend, and tile generation. +- `demo_register_target_system.ps1` + Creates or updates a callback-based target system. +- `demo_push_result.ps1` + Pushes an existing managed result to a target system or callback URL. +- `demo_geojson_flow.ps1` + Registers a GeoJSON resource and optionally pushes it through the same delivery interface. +- `demo_full_flow.ps1` + Runs the main end-to-end flow: sample raster, import, slice, target registration, and callback push. +- `demo_callback_receiver.py` + Small local receiver for inspecting push payloads. + +## Suggested Order + +1. Start the services manually, or run `demo_full_flow.ps1 -StartStack -StartCallbackReceiver`. +2. Run `demo_import_and_slice.ps1` to create a managed result. +3. Run `demo_register_target_system.ps1` to register a callback receiver. +4. Run `demo_push_result.ps1` to publish the managed result. +5. Run `demo_geojson_flow.ps1 -Push` if you want to verify the retained GeoJSON delivery path. + +## Example Commands + +```powershell +.\demo\demo_import_and_slice.ps1 +.\demo\demo_register_target_system.ps1 -CallbackUrl http://127.0.0.1:8920/callback +.\demo\demo_push_result.ps1 -ResultCode demo_result_20260406_120000 -TargetSystemCode demo_callback +.\demo\demo_geojson_flow.ps1 -Push -CallbackUrl http://127.0.0.1:8920/callback +.\demo\demo_full_flow.ps1 -StartStack -StartCallbackReceiver +``` + +## Output + +Each demo prints a JSON summary that includes the main identifiers and URLs. Temporary run data is written under `temp/demo-runs/`. + +`source_system_code` is intentionally left empty in the demos. If you want to populate it, seed the `source_systems` table first. diff --git a/demo/common.ps1 b/demo/common.ps1 new file mode 100644 index 0000000..beaeae4 --- /dev/null +++ b/demo/common.ps1 @@ -0,0 +1,574 @@ +param( + [string]$EnvFile = (Join-Path $PSScriptRoot "..\config\service.env") +) + +$script:RsDemoRepoRoot = (Resolve-Path (Join-Path $PSScriptRoot "..")).Path +$script:RsDemoProcesses = @() +$script:RsDemoEnvLoaded = $false + +function Get-RsRepoRoot { + return $script:RsDemoRepoRoot +} + +function Import-RsDemoEnvironment { + param([string]$Path = $EnvFile) + + if ($script:RsDemoEnvLoaded) { + return + } + + if (Test-Path -LiteralPath $Path) { + foreach ($rawLine in (Get-Content -LiteralPath $Path)) { + $line = $rawLine.Trim() + if (-not $line -or $line.StartsWith("#")) { + continue + } + if ($line.StartsWith("export ")) { + $line = $line.Substring(7).Trim() + } + $parts = $line.Split("=", 2) + if ($parts.Count -ne 2) { + continue + } + $key = $parts[0].Trim() + $value = $parts[1].Trim().Trim("`"").Trim("'") + if (-not $key) { + continue + } + if (-not (Test-Path "Env:$key")) { + Set-Item -Path "Env:$key" -Value $value + } + } + } + + if (-not $env:PYTHON_EXE) { + $env:PYTHON_EXE = "C:\Users\Administrator\.conda\envs\InSAR\python.exe" + } + if (-not $env:API_BASE_URL) { + $env:API_BASE_URL = "http://127.0.0.1:8910" + } + if (-not $env:TILE_STATIC_BASE_URL) { + $env:TILE_STATIC_BASE_URL = "http://127.0.0.1:8900" + } + if (-not $env:RS_API_BASE_URL) { + $env:RS_API_BASE_URL = $env:API_BASE_URL + } + if (-not $env:PYTHONIOENCODING) { + $env:PYTHONIOENCODING = "utf-8" + } + if (-not $env:NO_PAUSE) { + $env:NO_PAUSE = "1" + } + + $script:RsDemoEnvLoaded = $true +} + +function Get-RsApiBaseUrl { + Import-RsDemoEnvironment + return ([string]$env:API_BASE_URL).TrimEnd("/") +} + +function Get-RsTileBaseUrl { + Import-RsDemoEnvironment + return ([string]$env:TILE_STATIC_BASE_URL).TrimEnd("/") +} + +function Get-RsPythonExe { + Import-RsDemoEnvironment + if (-not (Test-Path -LiteralPath $env:PYTHON_EXE)) { + throw "Python interpreter not found: $($env:PYTHON_EXE)" + } + return $env:PYTHON_EXE +} + +function Get-RsGoCommand { + return (Get-Command go -ErrorAction Stop).Source +} + +function New-RsRunContext { + param([string]$Prefix = "demo") + + $runId = "{0}_{1}" -f $Prefix, (Get-Date -Format "yyyyMMdd_HHmmss") + $runDir = Join-Path (Get-RsRepoRoot) ("temp\demo-runs\" + $runId) + $logDir = Join-Path $runDir "logs" + $outputDir = Join-Path $runDir "output" + + New-Item -ItemType Directory -Force -Path $runDir | Out-Null + New-Item -ItemType Directory -Force -Path $logDir | Out-Null + New-Item -ItemType Directory -Force -Path $outputDir | Out-Null + + return [pscustomobject]@{ + RunID = $runId + RunDirectory = $runDir + LogDirectory = $logDir + OutputDirectory = $outputDir + } +} + +function ConvertTo-RsFileUri { + param([Parameter(Mandatory)] [string]$Path) + + $fullPath = [System.IO.Path]::GetFullPath($Path) + return ([System.Uri]$fullPath).AbsoluteUri +} + +function Encode-RsURIComponent { + param([Parameter(Mandatory)] [string]$Value) + + return [System.Uri]::EscapeDataString($Value) +} + +function Wait-RsUrl { + param( + [Parameter(Mandatory)] [string]$Url, + [int]$TimeoutSec = 60 + ) + + $deadline = (Get-Date).AddSeconds($TimeoutSec) + do { + try { + Invoke-RestMethod -Uri $Url -Method Get -TimeoutSec 5 | Out-Null + return + } catch { + Start-Sleep -Milliseconds 500 + } + } while ((Get-Date) -lt $deadline) + + throw "Timed out waiting for $Url" +} + +function Invoke-RsApi { + param( + [Parameter(Mandatory)] [string]$Method, + [Parameter(Mandatory)] [string]$Path, + [object]$Body, + [int]$TimeoutSec = 30 + ) + + Import-RsDemoEnvironment + + $uri = if ($Path -match "^https?://") { + $Path + } else { + "{0}{1}" -f (Get-RsApiBaseUrl), $Path + } + + $params = @{ + Uri = $uri + Method = $Method + TimeoutSec = $TimeoutSec + Headers = @{ + Accept = "application/json" + } + } + + if ($PSBoundParameters.ContainsKey("Body") -and $null -ne $Body) { + $params.ContentType = "application/json" + $params.Body = ($Body | ConvertTo-Json -Depth 32) + } + + try { + $response = Invoke-RestMethod @params + } catch { + $message = $_.Exception.Message + $webResponse = $_.Exception.Response + if ($webResponse) { + try { + $stream = $webResponse.GetResponseStream() + if ($stream) { + $reader = New-Object System.IO.StreamReader($stream) + $rawBody = $reader.ReadToEnd() + $reader.Dispose() + if ($rawBody) { + try { + $errorEnvelope = $rawBody | ConvertFrom-Json + if ($errorEnvelope.error) { + $message = "{0} {1}" -f $errorEnvelope.error.code, $errorEnvelope.error.message + } else { + $message = $rawBody + } + } catch { + $message = $rawBody + } + } + } + } catch { + } + } + throw "API call failed: $Method $uri`n$message" + } + + if ($null -eq $response) { + return $null + } + if ($response.error) { + throw "API error: $Method $uri => $($response.error.code) $($response.error.message)" + } + return $response.data +} + +function Wait-RsJob { + param( + [Parameter(Mandatory)] [string]$JobID, + [int]$TimeoutSec = 300, + [int]$PollIntervalSec = 2 + ) + + $deadline = (Get-Date).AddSeconds($TimeoutSec) + do { + $job = Invoke-RsApi -Method "GET" -Path ("/api/v1/jobs/{0}" -f $JobID) + if ($job.status -eq "succeeded") { + return $job + } + if ($job.status -in @("failed", "cancelled")) { + throw "Job $JobID finished with status=$($job.status) error=$($job.last_error_message)" + } + Start-Sleep -Seconds $PollIntervalSec + } while ((Get-Date) -lt $deadline) + + throw "Timed out waiting for job $JobID" +} + +function Get-RsStyleByCode { + param( + [Parameter(Mandatory)] [string]$ProjectCode, + [Parameter(Mandatory)] [string]$Code, + [switch]$AllowMissing + ) + + $project = Encode-RsURIComponent $ProjectCode + $keyword = Encode-RsURIComponent $Code + $items = @(Invoke-RsApi -Method "GET" -Path "/api/v1/styles?project_code=$project&keyword=$keyword") + $match = $items | Where-Object { $_.code -eq $Code } | Select-Object -First 1 + if ($match) { + return $match + } + if ($AllowMissing) { + return $null + } + throw "Style not found: $Code" +} + +function Get-RsTargetSystemByCode { + param( + [Parameter(Mandatory)] [string]$ProjectCode, + [Parameter(Mandatory)] [string]$Code, + [switch]$AllowMissing + ) + + $project = Encode-RsURIComponent $ProjectCode + $items = @(Invoke-RsApi -Method "GET" -Path "/api/v1/target-systems?project_code=$project") + $match = $items | Where-Object { $_.code -eq $Code } | Select-Object -First 1 + if ($match) { + return $match + } + if ($AllowMissing) { + return $null + } + throw "Target system not found: $Code" +} + +function Get-RsResultByCode { + param( + [Parameter(Mandatory)] [string]$ProjectCode, + [Parameter(Mandatory)] [string]$Code, + [switch]$AllowMissing + ) + + $project = Encode-RsURIComponent $ProjectCode + $keyword = Encode-RsURIComponent $Code + $items = @(Invoke-RsApi -Method "GET" -Path "/api/v1/projects/$project/results?keyword=$keyword") + $match = $items | Where-Object { $_.code -eq $Code } | Select-Object -First 1 + if ($match) { + return Invoke-RsApi -Method "GET" -Path ("/api/v1/results/{0}" -f $match.id) + } + if ($AllowMissing) { + return $null + } + throw "Result not found: $Code" +} + +function New-RsDemoStyleSpec { + param( + [string]$StyleID = "demo_continuous_style", + [string]$Name = "Demo Continuous Style", + [string]$Scope = "project", + [double]$MinValue = -1.0, + [double]$MaxValue = 1.0, + [int]$TileSize = 256, + [string]$TileFormat = "webp", + [int]$MinZoom = 0, + [int]$MaxZoom = 4 + ) + + return @{ + spec_version = "1.0" + id = $StyleID + name = $Name + version = 1 + scope = $Scope + render = @{ + mode = "continuous" + band = 1 + resampling = "bilinear" + value_domain = @{ + min = $MinValue + max = $MaxValue + } + color_map = @{ + type = "linear" + stops = @( + @{ value = $MinValue; color = "#8c510a"; label = "low" }, + @{ value = (($MinValue + $MaxValue) / 2.0); color = "#f6e8c3"; label = "mid" }, + @{ value = $MaxValue; color = "#01665e"; label = "high" } + ) + null_color = "#00000000" + } + } + legend = @{ + title = $Name + orientation = "horizontal" + precision = 2 + } + output = @{ + tile_format = $TileFormat + tile_size = $TileSize + min_zoom = $MinZoom + max_zoom = $MaxZoom + } + } +} + +function Ensure-RsStyleVersion { + param( + [Parameter(Mandatory)] [string]$ProjectCode, + [Parameter(Mandatory)] [string]$StyleCode, + [string]$StyleName = "Demo Continuous Style", + [hashtable]$StyleSpec, + [switch]$CreateNewVersion + ) + + $style = Get-RsStyleByCode -ProjectCode $ProjectCode -Code $StyleCode -AllowMissing + if (-not $style) { + $style = Invoke-RsApi -Method "POST" -Path "/api/v1/styles" -Body @{ + project_code = $ProjectCode + scope = "project" + code = $StyleCode + name = $StyleName + created_by = "demo" + } + } + + if (-not $CreateNewVersion -and $style.latest_version_id) { + return Invoke-RsApi -Method "GET" -Path ("/api/v1/style-versions/{0}" -f $style.latest_version_id) + } + + if (-not $StyleSpec) { + $StyleSpec = New-RsDemoStyleSpec -StyleID $StyleCode -Name $StyleName + } + + return Invoke-RsApi -Method "POST" -Path ("/api/v1/styles/{0}/versions" -f $style.id) -Body @{ + style_spec = $StyleSpec + } +} + +function Ensure-RsTargetSystem { + param( + [Parameter(Mandatory)] [string]$ProjectCode, + [Parameter(Mandatory)] [string]$Code, + [Parameter(Mandatory)] [string]$Name, + [Parameter(Mandatory)] [string]$CallbackUrl, + [string]$CallbackMethod = "POST", + [hashtable]$CallbackHeaders, + [int]$TimeoutSeconds = 30, + [bool]$Enabled = $true, + [string]$CreatedBy = "demo" + ) + + $body = @{ + code = $Code + name = $Name + callback_url = $CallbackUrl + callback_method = $CallbackMethod + timeout_seconds = $TimeoutSeconds + enabled = $Enabled + callback_headers = $CallbackHeaders + } + + $existing = Get-RsTargetSystemByCode -ProjectCode $ProjectCode -Code $Code -AllowMissing + if ($existing) { + return Invoke-RsApi -Method "PATCH" -Path ("/api/v1/target-systems/{0}" -f $existing.id) -Body $body + } + + $body.project_code = $ProjectCode + $body.created_by = $CreatedBy + return Invoke-RsApi -Method "POST" -Path "/api/v1/target-systems" -Body $body +} + +function Start-RsBackgroundProcess { + param( + [Parameter(Mandatory)] [string]$FilePath, + [string[]]$ArgumentList, + [Parameter(Mandatory)] [string]$WorkingDirectory, + [Parameter(Mandatory)] [string]$Name, + [Parameter(Mandatory)] [string]$LogDirectory + ) + + New-Item -ItemType Directory -Force -Path $LogDirectory | Out-Null + + $stdout = Join-Path $LogDirectory ($Name + ".stdout.log") + $stderr = Join-Path $LogDirectory ($Name + ".stderr.log") + $process = Start-Process -FilePath $FilePath ` + -ArgumentList $ArgumentList ` + -WorkingDirectory $WorkingDirectory ` + -PassThru ` + -RedirectStandardOutput $stdout ` + -RedirectStandardError $stderr + + $script:RsDemoProcesses += $process + return $process +} + +function Stop-RsBackgroundProcesses { + param([object[]]$Processes = $script:RsDemoProcesses) + + foreach ($process in $Processes) { + if ($process -and -not $process.HasExited) { + Stop-Process -Id $process.Id -Force -ErrorAction SilentlyContinue + } + } +} + +function Start-RsDemoStack { + param( + [int]$ApiPort = 28910, + [int]$TilePort = 28900, + [switch]$IncludeWorker, + [string]$RunID = ("stack_" + (Get-Date -Format "yyyyMMdd_HHmmss")) + ) + + Import-RsDemoEnvironment + + $runDir = Join-Path (Get-RsRepoRoot) ("temp\demo-runs\" + $RunID) + $logDir = Join-Path $runDir "logs" + New-Item -ItemType Directory -Force -Path $runDir | Out-Null + New-Item -ItemType Directory -Force -Path $logDir | Out-Null + + $apiBaseUrl = "http://127.0.0.1:$ApiPort" + $tileBaseUrl = "http://127.0.0.1:$TilePort" + + $env:API_HOST = "127.0.0.1" + $env:API_PORT = "$ApiPort" + $env:API_BASE_URL = $apiBaseUrl + $env:TILE_HOST = "127.0.0.1" + $env:TILE_PORT = "$TilePort" + $env:TILE_STATIC_BASE_URL = $tileBaseUrl + $env:RS_API_BASE_URL = $apiBaseUrl + $env:NO_PAUSE = "1" + + $pythonExe = Get-RsPythonExe + $goExe = Get-RsGoCommand + + $tileProcess = Start-RsBackgroundProcess ` + -FilePath $pythonExe ` + -ArgumentList @((Join-Path (Get-RsRepoRoot) "server.py")) ` + -WorkingDirectory (Get-RsRepoRoot) ` + -Name "tile-server" ` + -LogDirectory $logDir + + $apiProcess = Start-RsBackgroundProcess ` + -FilePath $goExe ` + -ArgumentList @("run", ".\cmd\api") ` + -WorkingDirectory (Join-Path (Get-RsRepoRoot) "services\api-go") ` + -Name "api-service" ` + -LogDirectory $logDir + + Wait-RsUrl -Url "$tileBaseUrl/health" -TimeoutSec 90 + Wait-RsUrl -Url "$apiBaseUrl/ready" -TimeoutSec 120 + + $workerProcess = $null + if ($IncludeWorker) { + $workerProcess = Start-RsBackgroundProcess ` + -FilePath $pythonExe ` + -ArgumentList @((Join-Path (Get-RsRepoRoot) "services\worker-py\main.py")) ` + -WorkingDirectory (Get-RsRepoRoot) ` + -Name "python-worker" ` + -LogDirectory $logDir + Start-Sleep -Seconds 1 + } + + $processes = @($tileProcess, $apiProcess) + if ($workerProcess) { + $processes += $workerProcess + } + + return [pscustomobject]@{ + RunID = $RunID + RunDirectory = $runDir + LogDirectory = $logDir + ApiBaseUrl = $apiBaseUrl + TileBaseUrl = $tileBaseUrl + Processes = $processes + } +} + +function Start-RsCallbackReceiver { + param( + [int]$Port = 28920, + [string]$Path = "/callback", + [string]$RunID = ("callback_" + (Get-Date -Format "yyyyMMdd_HHmmss")) + ) + + $runDir = Join-Path (Get-RsRepoRoot) ("temp\demo-runs\" + $RunID) + $logDir = Join-Path $runDir "logs" + $outputDir = Join-Path $runDir "callback-records" + New-Item -ItemType Directory -Force -Path $logDir | Out-Null + New-Item -ItemType Directory -Force -Path $outputDir | Out-Null + + $pythonExe = Get-RsPythonExe + $process = Start-RsBackgroundProcess ` + -FilePath $pythonExe ` + -ArgumentList @( + (Join-Path $PSScriptRoot "demo_callback_receiver.py"), + "--host", "127.0.0.1", + "--port", "$Port", + "--path", $Path, + "--output-dir", $outputDir + ) ` + -WorkingDirectory (Get-RsRepoRoot) ` + -Name "callback-receiver" ` + -LogDirectory $logDir + + $callbackUrl = "http://127.0.0.1:$Port$Path" + Wait-RsUrl -Url "http://127.0.0.1:$Port/health" -TimeoutSec 30 + + return [pscustomobject]@{ + RunID = $RunID + RunDirectory = $runDir + LogDirectory = $logDir + OutputDirectory = $outputDir + CallbackURL = $callbackUrl + Process = $process + } +} + +function Get-RsLatestCallbackRecord { + param([Parameter(Mandatory)] [string]$OutputDirectory) + + if (-not (Test-Path -LiteralPath $OutputDirectory)) { + return $null + } + + $file = Get-ChildItem -LiteralPath $OutputDirectory -Filter "*.json" | + Sort-Object LastWriteTimeUtc -Descending | + Select-Object -First 1 + + if (-not $file) { + return $null + } + + return [pscustomobject]@{ + Path = $file.FullName + Payload = (Get-Content -LiteralPath $file.FullName -Raw | ConvertFrom-Json) + } +} diff --git a/demo/create_sample_raster.py b/demo/create_sample_raster.py new file mode 100644 index 0000000..f918790 --- /dev/null +++ b/demo/create_sample_raster.py @@ -0,0 +1,86 @@ +import argparse +import os +import sys +from pathlib import Path + + +def bootstrap_gdal_env() -> None: + env_root = Path(sys.executable).resolve().parent + gdal_paths = [ + env_root / "Lib" / "site-packages" / "rasterio" / "gdal_data", + env_root / "Library" / "share" / "gdal", + env_root / "share" / "gdal", + ] + proj_paths = [ + env_root / "Lib" / "site-packages" / "rasterio" / "proj_data", + env_root / "Library" / "share" / "proj", + env_root / "share" / "proj", + ] + + for path in gdal_paths: + if path.exists(): + os.environ["GDAL_DATA"] = str(path) + break + for path in proj_paths: + if path.exists(): + os.environ["PROJ_LIB"] = str(path) + os.environ["PROJ_DATA"] = str(path) + break + + +bootstrap_gdal_env() + +import numpy as np +import rasterio +from rasterio.transform import from_origin + + +def build_values(width: int, height: int, preset: str) -> np.ndarray: + yy, xx = np.mgrid[0:height, 0:width] + x = (xx / max(width - 1, 1)).astype("float32") + y = (yy / max(height - 1, 1)).astype("float32") + + if preset == "phase": + values = np.sin((x * 8.0) + (y * 4.0)) * np.cos(y * 10.0) + else: + values = (x * 2.0 - 1.0) * 0.7 + np.sin(y * np.pi * 3.0) * 0.3 + + return values.astype("float32") + + +def main() -> int: + parser = argparse.ArgumentParser(description="Create a sample GeoTIFF for demos.") + parser.add_argument("--output", required=True, help="Output GeoTIFF path.") + parser.add_argument("--width", type=int, default=512) + parser.add_argument("--height", type=int, default=512) + parser.add_argument("--preset", choices=["ndvi", "phase"], default="ndvi") + parser.add_argument("--origin-x", type=float, default=110.0) + parser.add_argument("--origin-y", type=float, default=22.0) + parser.add_argument("--pixel-size", type=float, default=0.01) + args = parser.parse_args() + + output = Path(args.output).resolve() + output.parent.mkdir(parents=True, exist_ok=True) + + data = build_values(args.width, args.height, args.preset) + + with rasterio.open( + output, + "w", + driver="GTiff", + width=args.width, + height=args.height, + count=1, + dtype="float32", + crs="EPSG:4326", + transform=from_origin(args.origin_x, args.origin_y, args.pixel_size, args.pixel_size), + nodata=-9999.0, + ) as dst: + dst.write(data, 1) + + print(output) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/demo/demo_callback_receiver.py b/demo/demo_callback_receiver.py new file mode 100644 index 0000000..8060591 --- /dev/null +++ b/demo/demo_callback_receiver.py @@ -0,0 +1,109 @@ +import argparse +import json +import threading +import time +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path + + +def parse_args(): + parser = argparse.ArgumentParser(description="Simple callback receiver for tile-server demos.") + parser.add_argument("--host", default="127.0.0.1") + parser.add_argument("--port", type=int, default=8920) + parser.add_argument("--path", default="/callback") + parser.add_argument("--output-dir", required=True) + return parser.parse_args() + + +class State: + def __init__(self, output_dir: Path, callback_path: str): + self.output_dir = output_dir + self.callback_path = callback_path + self.counter = 0 + self.lock = threading.Lock() + + def next_file(self) -> Path: + with self.lock: + self.counter += 1 + name = f"{time.strftime('%Y%m%d_%H%M%S')}_{self.counter:04d}.json" + return self.output_dir / name + + +def make_handler(state: State): + class Handler(BaseHTTPRequestHandler): + def do_GET(self): + if self.path == "/health": + self.send_response(200) + self.send_header("Content-Type", "application/json; charset=utf-8") + self.end_headers() + self.wfile.write(json.dumps({"status": "ok"}).encode("utf-8")) + return + + self.send_response(404) + self.end_headers() + + def do_POST(self): + self._handle_payload() + + def do_PUT(self): + self._handle_payload() + + def _handle_payload(self): + if self.path != state.callback_path: + self.send_response(404) + self.end_headers() + return + + length = int(self.headers.get("Content-Length", "0")) + raw = self.rfile.read(length) + text = raw.decode("utf-8", errors="replace") + try: + body_json = json.loads(text) + except json.JSONDecodeError: + body_json = None + + record = { + "received_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), + "method": self.command, + "path": self.path, + "headers": {key: value for key, value in self.headers.items()}, + "body_text": text, + "body_json": body_json, + } + + output_path = state.next_file() + output_path.write_text(json.dumps(record, ensure_ascii=False, indent=2), encoding="utf-8") + print(f"[callback] saved {output_path}", flush=True) + + response = {"saved": str(output_path), "ok": True} + self.send_response(200) + self.send_header("Content-Type", "application/json; charset=utf-8") + self.end_headers() + self.wfile.write(json.dumps(response).encode("utf-8")) + + def log_message(self, fmt, *args): + print("[callback] " + (fmt % args), flush=True) + + return Handler + + +def main(): + args = parse_args() + output_dir = Path(args.output_dir).resolve() + output_dir.mkdir(parents=True, exist_ok=True) + + state = State(output_dir=output_dir, callback_path=args.path) + server = ThreadingHTTPServer((args.host, args.port), make_handler(state)) + print(f"Callback receiver listening on http://{args.host}:{args.port}{args.path}", flush=True) + print(f"Health endpoint: http://{args.host}:{args.port}/health", flush=True) + print(f"Output dir: {output_dir}", flush=True) + try: + server.serve_forever() + except KeyboardInterrupt: + pass + finally: + server.server_close() + + +if __name__ == "__main__": + main() diff --git a/demo/demo_full_flow.ps1 b/demo/demo_full_flow.ps1 new file mode 100644 index 0000000..c0c3b0e --- /dev/null +++ b/demo/demo_full_flow.ps1 @@ -0,0 +1,213 @@ +param( + [string]$ProjectCode = "default", + [string]$ResultCode = ("demo_full_" + (Get-Date -Format "yyyyMMdd_HHmmss")), + [string]$ResultType = "ndvi", + [string]$SourceSystemCode = "", + [string]$RequestedBy = "demo-user", + [string]$StyleCode = "demo_continuous_style", + [string]$TargetSystemCode = "demo_callback", + [switch]$StartStack, + [switch]$StartCallbackReceiver, + [switch]$KeepProcesses, + [int]$ApiPort = 28910, + [int]$TilePort = 28900, + [int]$CallbackPort = 28920, + [int]$ZoomMin = 0, + [int]$ZoomMax = 3 +) + +. (Join-Path $PSScriptRoot "common.ps1") + +$stack = $null +$callback = $null +$run = New-RsRunContext -Prefix "full_flow" + +try { + Import-RsDemoEnvironment + + if ($StartStack) { + $stack = Start-RsDemoStack -ApiPort $ApiPort -TilePort $TilePort -IncludeWorker -RunID $run.RunID + } + + $callbackUrl = "" + if ($StartCallbackReceiver) { + $callback = Start-RsCallbackReceiver -Port $CallbackPort -RunID ($run.RunID + "_callback") + $callbackUrl = $callback.CallbackURL + } + + $inputPath = Join-Path $run.OutputDirectory ($ResultCode + ".tif") + & (Get-RsPythonExe) (Join-Path $PSScriptRoot "create_sample_raster.py") ` + --output $inputPath ` + --preset "ndvi" | Out-Null + if ($LASTEXITCODE -ne 0) { + throw "Failed to create sample raster" + } + + $inspection = Invoke-RsApi -Method "POST" -Path "/api/v1/admin/raster-inspections" -Body @{ + path = [System.IO.Path]::GetFullPath($inputPath) + } + + $styleVersion = Ensure-RsStyleVersion -ProjectCode $ProjectCode -StyleCode $StyleCode -StyleName "Demo Continuous Style" + + $project = Encode-RsURIComponent $ProjectCode + $importBody = @{ + requested_by = $RequestedBy + asset = @{ + source_ref = $inspection.storage_uri + asset_type = "raster" + display_name = $ResultCode + description = "Imported by demo_full_flow.ps1" + metadata = @{ + demo_run = $run.RunID + } + } + asset_version = @{ + format = $inspection.format + storage_uri = $inspection.storage_uri + mime_type = $inspection.mime_type + size_bytes = $inspection.size_bytes + crs = $inspection.crs + width = $inspection.width + height = $inspection.height + band_count = $inspection.band_count + dtype = $inspection.dtype + nodata_value = $inspection.nodata_value + bbox = $inspection.bbox + resolution_x = $inspection.resolution_x + resolution_y = $inspection.resolution_y + metadata = $inspection.metadata + } + product = @{ + code = $ResultCode + type = $ResultType + name = $ResultCode + description = "Demo end-to-end result" + } + product_version = @{ + value_type = "continuous" + unit = "index" + bbox = $inspection.bbox + status = "ready" + } + style_binding = @{ + style_version_id = $styleVersion.id + is_default = $true + availability = "enabled" + } + } + if ($SourceSystemCode) { + $importBody.source_system_code = $SourceSystemCode + } + + $import = Invoke-RsApi -Method "POST" -Path "/api/v1/projects/$project/imports/results" -Body $importBody + + $slice = Invoke-RsApi -Method "POST" -Path ("/api/v1/results/{0}/slice" -f $import.product.product.id) -Body @{ + requested_by = $RequestedBy + build_cog = $true + generate_preview = $true + generate_legend = $true + generate_tiles = $true + build_cog_input = @{ + compression = "deflate" + block_size = 256 + force_rebuild = $true + } + preview_input = @{ + style_version_id = $styleVersion.id + width = 512 + height = 512 + format = "png" + force_rebuild = $true + } + legend_input = @{ + style_version_id = $styleVersion.id + width = 320 + height = 80 + format = "png" + force_rebuild = $true + } + tiles_input = @{ + style_version_id = $styleVersion.id + zoom_min = $ZoomMin + zoom_max = $ZoomMax + tile_size = 256 + tile_format = "webp" + force_rebuild = $true + } + } + + $sliceJobs = @() + foreach ($job in @($slice.jobs)) { + $sliceJobs += Wait-RsJob -JobID $job.id + } + + if ($callbackUrl) { + Ensure-RsTargetSystem ` + -ProjectCode $ProjectCode ` + -Code $TargetSystemCode ` + -Name "Demo Callback Receiver" ` + -CallbackUrl $callbackUrl ` + -CallbackMethod "POST" ` + -TimeoutSeconds 30 ` + -CreatedBy $RequestedBy ` + -CallbackHeaders @{ + "X-Demo-Mode" = "full-flow" + } | Out-Null + } + + $pushBody = @{ + requested_by = $RequestedBy + style_version_id = $styleVersion.id + include_artifacts = $true + } + if ($callbackUrl) { + $pushBody.target_system_code = $TargetSystemCode + } else { + $pushBody.callback_url = "http://127.0.0.1:8920/callback" + } + + $push = Invoke-RsApi -Method "POST" -Path ("/api/v1/results/{0}/pushes" -f $import.product.product.id) -Body $pushBody + $pushJob = Wait-RsJob -JobID $push.job.id + + $delivery = Invoke-RsApi -Method "GET" -Path ("/api/v1/projects/{0}/products/{1}/delivery?version=latest&style_version_id={2}" -f ` + (Encode-RsURIComponent $ProjectCode), ` + (Encode-RsURIComponent $ResultCode), ` + (Encode-RsURIComponent $styleVersion.id)) + + $callbackRecord = $null + if ($callback) { + Start-Sleep -Seconds 1 + $callbackRecord = Get-RsLatestCallbackRecord -OutputDirectory $callback.OutputDirectory + } + + [ordered]@{ + run_id = $run.RunID + api_base_url = Get-RsApiBaseUrl + tile_base_url = Get-RsTileBaseUrl + callback_url = if ($callback) { $callback.CallbackURL } else { $push.callback_url } + result_id = $import.product.product.id + result_code = $ResultCode + product_version_id = $import.product_version.id + style_version_id = $styleVersion.id + slice_job_ids = @($sliceJobs | ForEach-Object { $_.id }) + push_job_id = $push.job.id + push_job_status = $pushJob.status + cog_url = $delivery.downloads.cog_url + preview_url = if ($delivery.preview) { $delivery.preview.url } else { $null } + legend_url = if ($delivery.legend) { $delivery.legend.url } else { $null } + tileset_url = if ($delivery.tileset) { $delivery.tileset.url_template } else { $null } + callback_record_path = if ($callbackRecord) { $callbackRecord.Path } else { $null } + run_directory = $run.RunDirectory + log_directory = if ($stack) { $stack.LogDirectory } else { $null } + } | ConvertTo-Json -Depth 16 +} +finally { + if (-not $KeepProcesses) { + if ($callback) { + Stop-RsBackgroundProcesses -Processes @($callback.Process) + } + if ($stack) { + Stop-RsBackgroundProcesses -Processes $stack.Processes + } + } +} diff --git a/demo/demo_geojson_flow.ps1 b/demo/demo_geojson_flow.ps1 new file mode 100644 index 0000000..d97bf21 --- /dev/null +++ b/demo/demo_geojson_flow.ps1 @@ -0,0 +1,135 @@ +param( + [string]$ProjectCode = "default", + [string]$ResourceCode = ("demo_geojson_" + (Get-Date -Format "yyyyMMdd_HHmmss")), + [string]$InputPath = "", + [string]$RequestedBy = "demo-user", + [switch]$Push, + [string]$TargetSystemCode = "", + [string]$CallbackUrl = "", + [bool]$WaitForJob = $true, + [switch]$StartStack, + [switch]$StartCallbackReceiver, + [switch]$KeepProcesses, + [int]$ApiPort = 28910, + [int]$TilePort = 28900, + [int]$CallbackPort = 28920 +) + +. (Join-Path $PSScriptRoot "common.ps1") + +$stack = $null +$callback = $null +$run = New-RsRunContext -Prefix "geojson_flow" + +try { + Import-RsDemoEnvironment + + if ($StartStack) { + $stack = Start-RsDemoStack -ApiPort $ApiPort -TilePort $TilePort -IncludeWorker -RunID $run.RunID + } + if ($StartCallbackReceiver) { + $callback = Start-RsCallbackReceiver -Port $CallbackPort -RunID ($run.RunID + "_callback") + $Push = $true + $CallbackUrl = $callback.CallbackURL + } + + if (-not $InputPath) { + $geojsonDir = Join-Path (Get-RsRepoRoot) "data\geojson\demo" + New-Item -ItemType Directory -Force -Path $geojsonDir | Out-Null + $InputPath = Join-Path $geojsonDir ($ResourceCode + ".geojson") + @" +{ + "type": "FeatureCollection", + "features": [ + { + "type": "Feature", + "properties": { + "code": "$ResourceCode", + "kind": "demo" + }, + "geometry": { + "type": "Polygon", + "coordinates": [ + [ + [110.0, 20.0], + [112.0, 20.0], + [112.0, 22.0], + [110.0, 22.0], + [110.0, 20.0] + ] + ] + } + } + ] +} +"@ | Set-Content -LiteralPath $InputPath -Encoding UTF8 + } + + $publicUrl = "/geojson/demo/" + [System.IO.Path]::GetFileName($InputPath) + $project = Encode-RsURIComponent $ProjectCode + $resource = Invoke-RsApi -Method "POST" -Path "/api/v1/projects/$project/geojson-resources" -Body @{ + code = $ResourceCode + name = $ResourceCode + description = "Registered by demo_geojson_flow.ps1" + storage_uri = ConvertTo-RsFileUri $InputPath + public_url = $publicUrl + metadata = @{ + demo_run = $run.RunID + } + } + + $pushResponse = $null + $job = $null + if ($Push -or $TargetSystemCode -or $CallbackUrl) { + $body = @{ + resource_kind = "geojson_resource" + geojson_resource_id = $resource.id + requested_by = $RequestedBy + } + if ($TargetSystemCode) { + $body.target_system_code = $TargetSystemCode + } + if ($CallbackUrl) { + $body.callback_url = $CallbackUrl + } + if (-not $TargetSystemCode -and -not $CallbackUrl) { + $body.callback_url = "http://127.0.0.1:8920/callback" + } + + $pushResponse = Invoke-RsApi -Method "POST" -Path "/api/v1/projects/$project/delivery-pushes" -Body $body + if ($WaitForJob) { + $job = Wait-RsJob -JobID $pushResponse.job.id + } + } + + $callbackRecord = $null + if ($callback) { + Start-Sleep -Seconds 1 + $callbackRecord = Get-RsLatestCallbackRecord -OutputDirectory $callback.OutputDirectory + } + + [ordered]@{ + run_id = $run.RunID + api_base_url = Get-RsApiBaseUrl + tile_base_url = Get-RsTileBaseUrl + resource_id = $resource.id + resource_code = $resource.code + input_path = [System.IO.Path]::GetFullPath($InputPath) + public_url = $resource.public_url + download_url = $resource.download_url + push_job_id = if ($pushResponse) { $pushResponse.job.id } else { $null } + push_job_status = if ($job) { $job.status } elseif ($pushResponse) { $pushResponse.job.status } else { $null } + callback_url = if ($pushResponse) { $pushResponse.callback_url } else { $CallbackUrl } + callback_record_path = if ($callbackRecord) { $callbackRecord.Path } else { $null } + } | ConvertTo-Json -Depth 12 +} +finally { + if (-not $KeepProcesses) { + if ($callback) { + Stop-RsBackgroundProcesses -Processes @($callback.Process) + } + if ($stack) { + Stop-RsBackgroundProcesses -Processes $stack.Processes + } + } +} diff --git a/demo/demo_import_and_slice.ps1 b/demo/demo_import_and_slice.ps1 new file mode 100644 index 0000000..cfb4b23 --- /dev/null +++ b/demo/demo_import_and_slice.ps1 @@ -0,0 +1,190 @@ +param( + [string]$ProjectCode = "default", + [string]$ResultCode = ("demo_result_" + (Get-Date -Format "yyyyMMdd_HHmmss")), + [string]$ResultType = "ndvi", + [string]$InputPath = "", + [string]$SourceSystemCode = "", + [string]$RequestedBy = "demo-user", + [string]$StyleCode = "demo_continuous_style", + [string]$StyleName = "Demo Continuous Style", + [int]$ZoomMin = 0, + [int]$ZoomMax = 3, + [bool]$WaitForJobs = $true, + [switch]$StartStack, + [switch]$KeepStack, + [int]$ApiPort = 28910, + [int]$TilePort = 28900 +) + +. (Join-Path $PSScriptRoot "common.ps1") + +$stack = $null +$run = New-RsRunContext -Prefix "import_slice" + +try { + Import-RsDemoEnvironment + + if ($StartStack) { + $stack = Start-RsDemoStack -ApiPort $ApiPort -TilePort $TilePort -IncludeWorker -RunID $run.RunID + } + + if (-not $InputPath) { + $InputPath = Join-Path $run.OutputDirectory ($ResultCode + ".tif") + & (Get-RsPythonExe) (Join-Path $PSScriptRoot "create_sample_raster.py") ` + --output $InputPath ` + --preset "ndvi" | Out-Null + if ($LASTEXITCODE -ne 0) { + throw "Failed to create sample raster" + } + } + + $inspection = Invoke-RsApi -Method "POST" -Path "/api/v1/admin/raster-inspections" -Body @{ + path = [System.IO.Path]::GetFullPath($InputPath) + } + + $styleVersion = Ensure-RsStyleVersion -ProjectCode $ProjectCode -StyleCode $StyleCode -StyleName $StyleName + + $assetVersionBody = @{ + format = $inspection.format + storage_uri = $inspection.storage_uri + mime_type = $inspection.mime_type + size_bytes = $inspection.size_bytes + crs = $inspection.crs + width = $inspection.width + height = $inspection.height + band_count = $inspection.band_count + dtype = $inspection.dtype + bbox = $inspection.bbox + metadata = $inspection.metadata + } + if ($inspection.PSObject.Properties.Name -contains "nodata_value" -and $null -ne $inspection.nodata_value) { + $assetVersionBody.nodata_value = $inspection.nodata_value + } + if ($inspection.PSObject.Properties.Name -contains "resolution_x" -and $null -ne $inspection.resolution_x) { + $assetVersionBody.resolution_x = $inspection.resolution_x + } + if ($inspection.PSObject.Properties.Name -contains "resolution_y" -and $null -ne $inspection.resolution_y) { + $assetVersionBody.resolution_y = $inspection.resolution_y + } + + $project = Encode-RsURIComponent $ProjectCode + $importBody = @{ + requested_by = $RequestedBy + asset = @{ + source_ref = $inspection.storage_uri + asset_type = "raster" + display_name = $ResultCode + description = "Imported by demo_import_and_slice.ps1" + metadata = @{ + demo_run = $run.RunID + } + } + asset_version = $assetVersionBody + product = @{ + code = $ResultCode + type = $ResultType + name = $ResultCode + description = "Demo managed result" + } + product_version = @{ + value_type = "continuous" + unit = "index" + bbox = $inspection.bbox + status = "ready" + } + style_binding = @{ + style_version_id = $styleVersion.id + is_default = $true + availability = "enabled" + } + } + if ($SourceSystemCode) { + $importBody.source_system_code = $SourceSystemCode + } + + $import = Invoke-RsApi -Method "POST" -Path "/api/v1/projects/$project/imports/results" -Body $importBody + + $slice = Invoke-RsApi -Method "POST" -Path ("/api/v1/results/{0}/slice" -f $import.product.product.id) -Body @{ + requested_by = $RequestedBy + build_cog = $true + generate_preview = $true + generate_legend = $true + generate_tiles = $true + build_cog_input = @{ + compression = "deflate" + block_size = 256 + force_rebuild = $true + } + preview_input = @{ + style_version_id = $styleVersion.id + width = 512 + height = 512 + format = "png" + force_rebuild = $true + } + legend_input = @{ + style_version_id = $styleVersion.id + width = 320 + height = 80 + format = "png" + force_rebuild = $true + } + tiles_input = @{ + style_version_id = $styleVersion.id + zoom_min = $ZoomMin + zoom_max = $ZoomMax + tile_size = 256 + tile_format = "webp" + force_rebuild = $true + } + } + + $jobStates = @() + if ($WaitForJobs) { + foreach ($job in @($slice.jobs)) { + $jobStates += Wait-RsJob -JobID $job.id + } + } + + $result = Invoke-RsApi -Method "GET" -Path ("/api/v1/results/{0}" -f $import.product.product.id) + $delivery = $null + if ($WaitForJobs) { + $delivery = Invoke-RsApi -Method "GET" -Path ("/api/v1/projects/{0}/products/{1}/delivery?version=latest&style_version_id={2}" -f ` + (Encode-RsURIComponent $ProjectCode), ` + (Encode-RsURIComponent $ResultCode), ` + (Encode-RsURIComponent $styleVersion.id)) + } + + [ordered]@{ + run_id = $run.RunID + api_base_url = Get-RsApiBaseUrl + tile_base_url = Get-RsTileBaseUrl + input_path = [System.IO.Path]::GetFullPath($InputPath) + result_id = $import.product.product.id + result_code = $ResultCode + product_version_id = $import.product_version.id + style_version_id = $styleVersion.id + imported_asset_id = $import.asset.id + imported_version_id = $import.asset_version.id + slice_job_ids = @($slice.jobs | ForEach-Object { $_.id }) + slice_job_statuses = @($jobStates | ForEach-Object { + [ordered]@{ + id = $_.id + job_type = $_.job_type + status = $_.status + } + }) + cog_url = if ($delivery) { $delivery.downloads.cog_url } else { $null } + preview_url = if ($delivery -and $delivery.preview) { $delivery.preview.url } else { $null } + legend_url = if ($delivery -and $delivery.legend) { $delivery.legend.url } else { $null } + tileset_url = if ($delivery -and $delivery.tileset) { $delivery.tileset.url_template } else { $null } + result_status = $result.status + run_directory = $run.RunDirectory + log_directory = if ($stack) { $stack.LogDirectory } else { $null } + } | ConvertTo-Json -Depth 16 +} +finally { + if ($stack -and -not $KeepStack) { + Stop-RsBackgroundProcesses -Processes $stack.Processes + } +} diff --git a/demo/demo_push_result.ps1 b/demo/demo_push_result.ps1 new file mode 100644 index 0000000..0333814 --- /dev/null +++ b/demo/demo_push_result.ps1 @@ -0,0 +1,72 @@ +param( + [string]$ProjectCode = "default", + [string]$ResultCode = "", + [string]$ResultID = "", + [string]$TargetSystemCode = "", + [string]$CallbackUrl = "", + [string]$StyleVersionID = "", + [string]$RequestedBy = "demo-user", + [bool]$WaitForJob = $true, + [bool]$IncludeArtifacts = $true, + [int]$TimeoutSec = 180 +) + +. (Join-Path $PSScriptRoot "common.ps1") + +Import-RsDemoEnvironment + +if (-not $ResultID -and -not $ResultCode) { + throw "ResultID or ResultCode is required" +} + +$result = if ($ResultID) { + Invoke-RsApi -Method "GET" -Path ("/api/v1/results/{0}" -f $ResultID) +} else { + Get-RsResultByCode -ProjectCode $ProjectCode -Code $ResultCode +} + +if (-not $StyleVersionID) { + if ($result.delivery -and $result.delivery.style_version -and $result.delivery.style_version.id) { + $StyleVersionID = $result.delivery.style_version.id + } elseif ($result.styles -and $result.styles.Count -gt 0) { + $StyleVersionID = $result.styles[0].id + } +} +if (-not $StyleVersionID) { + throw "No style version is bound to the target result" +} + +$body = @{ + requested_by = $RequestedBy + style_version_id = $StyleVersionID + include_artifacts = $IncludeArtifacts +} +if ($TargetSystemCode) { + $body.target_system_code = $TargetSystemCode +} +if ($CallbackUrl) { + $body.callback_url = $CallbackUrl +} +if (-not $TargetSystemCode -and -not $CallbackUrl) { + $body.callback_url = "http://127.0.0.1:8920/callback" +} + +$push = Invoke-RsApi -Method "POST" -Path ("/api/v1/results/{0}/pushes" -f $result.id) -Body $body + +$job = $null +if ($WaitForJob) { + $job = Wait-RsJob -JobID $push.job.id -TimeoutSec $TimeoutSec +} + +[ordered]@{ + api_base_url = Get-RsApiBaseUrl + result_id = $result.id + result_code = $result.code + style_version_id = $StyleVersionID + target_system_code = $push.target_system_code + callback_url = $push.callback_url + publish_id = $push.publish_id + job_id = $push.job.id + job_status = if ($job) { $job.status } else { $push.job.status } + resource_kind = $push.resource_kind +} | ConvertTo-Json -Depth 8 diff --git a/demo/demo_register_target_system.ps1 b/demo/demo_register_target_system.ps1 new file mode 100644 index 0000000..bf67969 --- /dev/null +++ b/demo/demo_register_target_system.ps1 @@ -0,0 +1,27 @@ +param( + [string]$ProjectCode = "default", + [string]$Code = "demo_callback", + [string]$Name = "Demo Callback Receiver", + [string]$CallbackUrl = "http://127.0.0.1:8920/callback", + [string]$CallbackMethod = "POST", + [int]$TimeoutSeconds = 30, + [string]$CreatedBy = "demo-user" +) + +. (Join-Path $PSScriptRoot "common.ps1") + +Import-RsDemoEnvironment + +$target = Ensure-RsTargetSystem ` + -ProjectCode $ProjectCode ` + -Code $Code ` + -Name $Name ` + -CallbackUrl $CallbackUrl ` + -CallbackMethod $CallbackMethod ` + -TimeoutSeconds $TimeoutSeconds ` + -CreatedBy $CreatedBy ` + -CallbackHeaders @{ + "X-Demo-System" = "tile-server-demo" + } + +$target | ConvertTo-Json -Depth 8 diff --git a/demo/start-callback-receiver.bat b/demo/start-callback-receiver.bat new file mode 100644 index 0000000..408ae51 --- /dev/null +++ b/demo/start-callback-receiver.bat @@ -0,0 +1,16 @@ +@echo off +setlocal +chcp 65001 >nul 2>&1 + +set "ROOT=%~dp0" +if "%PYTHON_EXE%"=="" set "PYTHON_EXE=C:\Users\Administrator\.conda\envs\InSAR\python.exe" +if "%DEMO_CALLBACK_PORT%"=="" set "DEMO_CALLBACK_PORT=8920" +if "%DEMO_CALLBACK_PATH%"=="" set "DEMO_CALLBACK_PATH=/callback" +if "%DEMO_CALLBACK_OUTPUT_DIR%"=="" set "DEMO_CALLBACK_OUTPUT_DIR=%ROOT%..\temp\demo-callback" + +if not exist "%PYTHON_EXE%" ( + echo Python interpreter not found: %PYTHON_EXE% + exit /b 1 +) + +"%PYTHON_EXE%" "%ROOT%demo_callback_receiver.py" --host 127.0.0.1 --port %DEMO_CALLBACK_PORT% --path %DEMO_CALLBACK_PATH% --output-dir "%DEMO_CALLBACK_OUTPUT_DIR%" diff --git a/docs/INSAR_MANAGEMENT_SYSTEM_V2.md b/docs/INSAR_MANAGEMENT_SYSTEM_V2.md new file mode 100644 index 0000000..9e7710f --- /dev/null +++ b/docs/INSAR_MANAGEMENT_SYSTEM_V2.md @@ -0,0 +1,201 @@ +# Insar_management_system_v2 接入说明 + +本文档说明 `Z:\Code\Insar_management_system_v2` 如何接入当前 `tile-server`，不需要修改推送服务外部的公开匿名路由，全部走带 token 的受保护接口。 + +## 结论 + +当前 `tile-server` 经过兼容后，可以支撑 `Insar_management_system_v2` 的底图和 GeoJSON/SHP 调用，但前提是目标系统把原先的固定静态路径改成带 token 的调用方式。 + +支持范围： + +- 底图瓦片 +- GeoJSON 文件直连读取 +- Shapefile 或 `.zip` 方式下载 + +不需要目标系统实现“接收推送”的回调接口。 + +## 推荐调用方式 + +假设服务地址为： + +```text +http://127.0.0.1:8910 +``` + +假设访问 token 为： + +```text +bmp_xxxxxxxxxxxxxxxxx +``` + +### 1. 底图瓦片 + +推荐直接使用默认版本兼容路由： + +```text +GET /tiles/{code}/{z}/{x}/{y}.ext?token=... +``` + +示例： + +```text +http://127.0.0.1:8910/tiles/google_image/{z}/{x}/{y}.webp?token=bmp_xxx +http://127.0.0.1:8910/tiles/gaode_image/{z}/{x}/{y}.webp?token=bmp_xxx +http://127.0.0.1:8910/tiles/gaode_shp/{z}/{x}/{y}.webp?token=bmp_xxx +``` + +说明： + +- `google_image`、`gaode_image`、`gaode_shp` 可以直接保留下划线写法 +- 服务内部会归一化到对应底图库 code +- 路由会自动解析该底图的默认版本 + +如果你希望显式锁定版本，也可以继续使用： + +```text +GET /tiles/{code}/{version}/{z}/{x}/{y}.ext?token=... +``` + +### 2. GeoJSON + +推荐直接按原始相对路径访问： + +```text +GET /geojson/{relative/path}.geojson?token=... +``` + +示例： + +```text +http://127.0.0.1:8910/geojson/全国行政区.geojson?token=bmp_xxx +``` + +说明： + +- 支持按原始相对路径访问 +- 也支持按文件名访问 +- 也支持按内部 vector code 访问 +- 返回内容为实际 GeoJSON 文件内容，可直接 `fetch(...).json()` + +### 3. SHP + +推荐按原始相对路径下载： + +```text +GET /shp/{relative/path}.shp?token=... +``` + +示例： + +```text +http://127.0.0.1:8910/shp/landuse/parcel.shp?token=bmp_xxx +``` + +说明： + +- 如果底层资源是 `.shp + .dbf + .shx ...` 组合，服务会自动打包成 zip 返回 +- 如果底层资源本来就是 `.zip`，则直接返回 zip + +如果你希望继续按内部 code 下载，也可以使用： + +```text +GET /vectors/{code}/download?token=... +``` + +## Token 传递方式 + +支持以下方式，推荐前端场景使用查询参数： + +### 查询参数 + +```text +?token=bmp_xxx +``` + +### Header + +```text +Authorization: Bearer bmp_xxx +``` + +或： + +```text +X-Access-Token: bmp_xxx +``` + +## 前端改造建议 + +### 底图常量 + +把原先： + +```js +`${TILE_BASE}/tiles/google_image/{z}/{x}/{y}.webp` +``` + +改成： + +```js +`${TILE_BASE}/tiles/google_image/{z}/{x}/{y}.webp?token=${encodeURIComponent(TILE_TOKEN)}` +``` + +其他底图同理。 + +### 行政区 GeoJSON + +把原先： + +```js +`${TILE_BASE}/geojson/全国行政区.geojson` +``` + +改成： + +```js +`${TILE_BASE}/geojson/全国行政区.geojson?token=${encodeURIComponent(TILE_TOKEN)}` +``` + +## 推荐接入流程 + +1. 在 `tile-server` 管理页或管理 API 扫描底图目录和矢量目录。 +2. 创建一个专供 `Insar_management_system_v2` 使用的 token。 +3. token 授权对应底图和需要访问的 GeoJSON/SHP 资源。 +4. 在 `Insar_management_system_v2` 中把静态地址改成带 token 的服务地址。 + +## 创建 token 时的资源填写规则 + +### 底图 + +填写底图库 code，例如： + +```text +google-image +gaode-image +gaode-shp +``` + +### 矢量 + +现在既可以填内部 code，也可以直接填原始路径，例如： + +```text +全国行政区.geojson +landuse/parcel.shp +``` + +## 适配边界 + +当前服务负责的是： + +- 扫描 +- 目录管理 +- token 鉴权 +- 底图瓦片输出 +- GeoJSON/SHP 输出 + +当前服务不负责： + +- 生成切片 +- 把矢量转换成 GeoJSON +- 修改 `Insar_management_system_v2` 代码 diff --git a/scripts/build-api.bat b/scripts/build-api.bat new file mode 100644 index 0000000..45d94af --- /dev/null +++ b/scripts/build-api.bat @@ -0,0 +1,49 @@ +@echo off +setlocal +chcp 65001 >nul 2>&1 +title Build Basemap Pusher + +set "ROOT=%~dp0..\\" +pushd "%ROOT%" >nul 2>&1 +set "ROOT=%CD%\" +popd +call "%ROOT%scripts\load-env.bat" "%ROOT%config\service.env" + +REM Keep the toolchain caches inside the repo, the same way run-api.bat does. +REM Without this, go build uses the global module cache and reaches out to +REM proxy.golang.org, which is blocked on many internal networks. +if "%GOCACHE%"=="" set "GOCACHE=%ROOT%.gocache" +if "%GOMODCACHE%"=="" set "GOMODCACHE=%ROOT%.gomodcache" +if "%GOTMPDIR%"=="" set "GOTMPDIR=%ROOT%.gotmp" +if "%GOPATH%"=="" set "GOPATH=%ROOT%.gopath" +if "%GOPROXY%"=="" set "GOPROXY=https://goproxy.cn,direct" +if "%GOSUMDB%"=="" set "GOSUMDB=off" + +set "OUT=%ROOT%bin\basemap-pusher.exe" +if not exist "%ROOT%bin" mkdir "%ROOT%bin" >nul 2>&1 + +echo ======================================== +echo Building Basemap Pusher +echo ======================================== +echo Output: %OUT% +echo. + +pushd "%ROOT%services\api-go" +go build -o "%OUT%" .\cmd\api +set "BUILD_EXIT=%ERRORLEVEL%" +popd + +if not "%BUILD_EXIT%"=="0" ( + echo. + echo [FAIL] Build failed with exit code %BUILD_EXIT%. +) else ( + echo. + echo [OK] Built %OUT% + echo. + echo Install as a Windows service ^(elevated shell^): + echo "%OUT%" --env-file "%ROOT%config\service.env" install + echo "%OUT%" start +) + +if "%NO_PAUSE%"=="" pause +exit /b %BUILD_EXIT% diff --git a/scripts/gdal2tiles-qgis.bat b/scripts/gdal2tiles-qgis.bat new file mode 100644 index 0000000..a71571c --- /dev/null +++ b/scripts/gdal2tiles-qgis.bat @@ -0,0 +1,35 @@ +@echo off +setlocal enableextensions +REM Run gdal2tiles.py from the GDAL bundled with QGIS. +REM +REM QGIS ships GDAL but its Python bindings need the OSGeo4W environment +REM (GDAL_DATA, PROJ_LIB, PYTHONHOME). Running gdal2tiles.exe from a bare +REM shell fails to import osgeo, so this wrapper loads o4w_env.bat first. +REM +REM Override auto-detection with: set GDAL2TILES_QGIS_ROOT=C:\path\to\QGIS + +set "QGIS_ROOT=%GDAL2TILES_QGIS_ROOT%" +if not "%QGIS_ROOT%"=="" goto have_root + +for /f "delims=" %%D in ('dir /b /ad /o-n "%ProgramFiles%\QGIS *" 2^>nul') do ( + if not defined QGIS_ROOT set "QGIS_ROOT=%ProgramFiles%\%%D" +) + +:have_root +if "%QGIS_ROOT%"=="" ( + echo gdal2tiles wrapper: QGIS not found. Set GDAL2TILES_QGIS_ROOT to the QGIS install directory. 1>&2 + exit /b 2 +) + +if not exist "%QGIS_ROOT%\bin\o4w_env.bat" ( + echo gdal2tiles wrapper: "%QGIS_ROOT%\bin\o4w_env.bat" is missing; is this a QGIS install? 1>&2 + exit /b 2 +) +if not exist "%QGIS_ROOT%\apps\Python312\Scripts\gdal2tiles.exe" ( + echo gdal2tiles wrapper: gdal2tiles.exe not found under "%QGIS_ROOT%\apps\Python312\Scripts". 1>&2 + exit /b 2 +) + +call "%QGIS_ROOT%\bin\o4w_env.bat" >nul 2>&1 +"%QGIS_ROOT%\apps\Python312\Scripts\gdal2tiles.exe" %* +exit /b %ERRORLEVEL% diff --git a/scripts/gdalbuildvrt-qgis.bat b/scripts/gdalbuildvrt-qgis.bat new file mode 100644 index 0000000..82959cd --- /dev/null +++ b/scripts/gdalbuildvrt-qgis.bat @@ -0,0 +1,33 @@ +@echo off +setlocal enableextensions +REM Run gdalbuildvrt from the GDAL bundled with QGIS. +REM +REM QGIS ships GDAL but its Python bindings need the OSGeo4W environment +REM (GDAL_DATA, PROJ_LIB). Loading o4w_env.bat first keeps PROJ working. +REM +REM Override auto-detection with: set GDAL2TILES_QGIS_ROOT=C:\path\to\QGIS + +set "QGIS_ROOT=%GDAL2TILES_QGIS_ROOT%" +if not "%QGIS_ROOT%"=="" goto have_root + +for /f "delims=" %%D in ('dir /b /ad /o-n "%ProgramFiles%\QGIS *" 2^>nul') do ( + if not defined QGIS_ROOT set "QGIS_ROOT=%ProgramFiles%\%%D" +) + +:have_root +if "%QGIS_ROOT%"=="" ( + echo gdalbuildvrt wrapper: QGIS not found. Set GDAL2TILES_QGIS_ROOT to the QGIS install directory. 1>&2 + exit /b 2 +) +if not exist "%QGIS_ROOT%\bin\o4w_env.bat" ( + echo gdalbuildvrt wrapper: "%QGIS_ROOT%\bin\o4w_env.bat" is missing; is this a QGIS install? 1>&2 + exit /b 2 +) +if not exist "%QGIS_ROOT%\bin\gdalbuildvrt.exe" ( + echo gdalbuildvrt wrapper: gdalbuildvrt.exe not found under "%QGIS_ROOT%\bin". 1>&2 + exit /b 2 +) + +call "%QGIS_ROOT%\bin\o4w_env.bat" >nul 2>&1 +"%QGIS_ROOT%\bin\gdalbuildvrt.exe" %* +exit /b %ERRORLEVEL% diff --git a/scripts/run-api.bat b/scripts/run-api.bat index dc4023f..63fa920 100644 --- a/scripts/run-api.bat +++ b/scripts/run-api.bat @@ -1,7 +1,7 @@ @echo off setlocal chcp 65001 >nul 2>&1 -title Map Asset Gateway +title Basemap Pusher set "ROOT=%~dp0..\\" call "%ROOT%scripts\load-env.bat" "%ROOT%config\service.env" @@ -17,12 +17,12 @@ if "%API_PORT%"=="" set "API_PORT=8910" if "%API_BASE_URL%"=="" set "API_BASE_URL=http://127.0.0.1:%API_PORT%" if "%TILE_PUBLIC_BASE_URL%"=="" set "TILE_PUBLIC_BASE_URL=%API_BASE_URL%" if "%VECTOR_PUBLIC_BASE_URL%"=="" set "VECTOR_PUBLIC_BASE_URL=%API_BASE_URL%" -if "%SQLITE_PATH%"=="" set "SQLITE_PATH=%ROOT%data\map-asset-gateway.db" +if "%SQLITE_PATH%"=="" set "SQLITE_PATH=%ROOT%data\basemap-pusher.db" if "%BASEMAP_SCAN_ROOT%"=="" set "BASEMAP_SCAN_ROOT=%ROOT%data\tiles" if "%VECTOR_SCAN_ROOT%"=="" set "VECTOR_SCAN_ROOT=%ROOT%data\geojson" echo ======================================== -echo Map Asset Gateway +echo Basemap Pusher echo ======================================== echo. echo Env file: %ROOT%config\service.env diff --git a/scripts/service.bat b/scripts/service.bat new file mode 100644 index 0000000..2243026 --- /dev/null +++ b/scripts/service.bat @@ -0,0 +1,67 @@ +@echo off +setlocal +chcp 65001 >nul 2>&1 +title Basemap Pusher Service + +set "ROOT=%~dp0..\" +pushd "%ROOT%" >nul 2>&1 +set "ROOT=%CD%\" +popd + +set "EXE=%ROOT%bin\basemap-pusher.exe" +set "ENVFILE=%ROOT%config\service.env" + +if not exist "%EXE%" ( + echo Service binary not found: %EXE% + echo Build it first: scripts\build-api.bat + if "%NO_PAUSE%"=="" pause + exit /b 1 +) + +set "VERB=%~1" +if "%VERB%"=="" set "VERB=status" + +if /i "%VERB%"=="install" goto do_install +if /i "%VERB%"=="uninstall" goto do_uninstall +if /i "%VERB%"=="start" goto do_start +if /i "%VERB%"=="stop" goto do_stop +if /i "%VERB%"=="restart" goto do_restart +if /i "%VERB%"=="status" goto do_status + +echo Usage: service.bat ^ +exit /b 2 + +:do_install +"%EXE%" --env-file "%ENVFILE%" install +goto finish + +:do_uninstall +"%EXE%" uninstall +goto finish + +:do_start +"%EXE%" start +goto finish + +:do_stop +"%EXE%" stop +goto finish + +:do_restart +"%EXE%" stop +"%EXE%" start +goto finish + +:do_status +"%EXE%" status +goto finish + +:finish +set "RC=%ERRORLEVEL%" +if not "%RC%"=="0" ( + echo. + echo [FAIL] exit code %RC%. + echo install / uninstall / start / stop need an elevated shell ^(Run as Administrator^). +) +if "%NO_PAUSE%"=="" pause +exit /b %RC% diff --git a/services/api-go/cmd/api/main.go b/services/api-go/cmd/api/main.go index a662962..241cf23 100644 --- a/services/api-go/cmd/api/main.go +++ b/services/api-go/cmd/api/main.go @@ -2,36 +2,111 @@ package main import ( "context" + "errors" + "fmt" "log" "net/http" "os" + "os/signal" "path/filepath" "strconv" "strings" + "syscall" "time" - "map-asset-gateway/api-go/internal/basemap" - "map-asset-gateway/api-go/internal/basemapapi" - "map-asset-gateway/api-go/internal/config" + "tile-server/api-go/internal/basemap" + "tile-server/api-go/internal/basemapapi" + "tile-server/api-go/internal/config" ) +const shutdownGrace = 15 * time.Second + +type cliArgs struct { + command string + envFile string +} + func main() { + args, err := parseArgs(os.Args[1:]) + if err != nil { + log.Fatalf("parse arguments: %v", err) + } + if args.envFile != "" { + if err := os.Setenv("APP_ENV_FILE", args.envFile); err != nil { + log.Fatalf("set APP_ENV_FILE: %v", err) + } + } + + // Service management verbs (install / uninstall / start / stop / status). + if handled, err := handleServiceArgs(args.command, args.envFile); handled { + if err != nil { + log.Fatalf("%v", err) + } + return + } + + // When the SCM launched us there is no console and no SIGINT, so the service + // handler owns the lifecycle instead of the signal handler. + if runAsWindowsService() { + return + } + + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer stop() + if err := run(ctx); err != nil { + log.Fatalf("%v", err) + } +} + +func parseArgs(argv []string) (cliArgs, error) { + var args cliArgs + for i := 0; i < len(argv); i++ { + switch argv[i] { + case "--env-file": + if i+1 >= len(argv) { + return args, errors.New("--env-file requires a path") + } + args.envFile = argv[i+1] + i++ + default: + if args.command == "" { + args.command = argv[i] + } + } + } + return args, nil +} + +// run boots the service and blocks until ctx is cancelled or the listener dies. +// It owns every resource it creates. +func run(ctx context.Context) error { cfg, err := config.Load() if err != nil { - log.Fatalf("load config: %v", err) + return fmt.Errorf("load config: %w", err) + } + + // The admin surface can create tokens that read every basemap, so it must + // never be reachable off-host without credentials. + if cfg.BindsToNonLoopback() && !cfg.HasAdminCredentials() { + return fmt.Errorf("refusing to listen on %s without admin credentials: set ADMIN_TOKEN or ADMIN_USER/ADMIN_PASSWORD", cfg.ListenAddr()) } store, err := basemap.OpenStore(basemap.StoreConfig{ - SQLitePath: cfg.SQLitePath, - ScanRoot: cfg.BasemapScanRoot, - VectorScanRoot: cfg.VectorScanRoot, - APIBaseURL: cfg.APIBaseURL, - TileBaseURL: cfg.TilePublicBaseURL, - VectorBaseURL: cfg.VectorPublicBaseURL, - AutoScanOnStart: cfg.AutoScanOnStart, + SQLitePath: cfg.SQLitePath, + ScanRoot: cfg.BasemapScanRoot, + VectorScanRoot: cfg.VectorScanRoot, + APIBaseURL: cfg.APIBaseURL, + TileBaseURL: cfg.TilePublicBaseURL, + VectorBaseURL: cfg.VectorPublicBaseURL, + AutoScanOnStart: cfg.AutoScanOnStart, + TokenDefaultTTL: cfg.TokenDefaultTTL, + TileJobCommand: cfg.TileJobCommand, + TileJobVRTCommand: cfg.TileJobVRTCommand, + TileJobLogDir: cfg.TileJobLogDir, + HealthCheckInterval: cfg.HealthCheckInterval, }) if err != nil { - log.Fatalf("open sqlite store: %v", err) + return fmt.Errorf("open sqlite store: %w", err) } defer func() { if err := store.Close(); err != nil { @@ -50,6 +125,7 @@ func main() { SQLitePath: cfg.SQLitePath, ScanRoot: cfg.BasemapScanRoot, VectorScanRoot: cfg.VectorScanRoot, + TileJobCommand: cfg.TileJobCommand, StartedAt: time.Now().UTC(), } @@ -68,10 +144,21 @@ func main() { } } - server, err := basemapapi.New(store, systemInfo) + // Health inspection, push retries and the tile queue run in the background. + store.StartBackgroundWorkers() + + server, err := basemapapi.New(store, systemInfo, basemapapi.Options{ + Admin: basemapapi.AdminAuth{ + Token: cfg.AdminToken, + User: cfg.AdminUser, + Password: cfg.AdminPassword, + }, + CORSAllowedOrigins: cfg.CORSAllowedOrigins, + }) if err != nil { - log.Fatalf("init api server: %v", err) + return fmt.Errorf("init api server: %w", err) } + httpServer := &http.Server{ Addr: cfg.ListenAddr(), Handler: server.Handler(), @@ -79,22 +166,53 @@ func main() { } cleanupPID, err := writePIDFile(strings.TrimSpace(os.Getenv("RS_API_PID_FILE"))) if err != nil { - log.Fatalf("write pid file: %v", err) + return fmt.Errorf("write pid file: %w", err) } if cleanupPID != nil { defer cleanupPID() } - log.Printf("map asset gateway listening on http://%s", cfg.ListenAddr()) log.Printf("api base url: %s", cfg.APIBaseURL) log.Printf("tile public base url: %s", cfg.TilePublicBaseURL) log.Printf("vector public base url: %s", cfg.VectorPublicBaseURL) log.Printf("sqlite path: %s", cfg.SQLitePath) log.Printf("scan root: %s", cfg.BasemapScanRoot) log.Printf("vector scan root: %s", cfg.VectorScanRoot) + if cfg.HasAdminCredentials() { + log.Printf("admin auth: enabled") + } else { + log.Printf("admin auth: disabled (loopback only)") + } - if err := httpServer.ListenAndServe(); err != nil && err != http.ErrServerClosed { - log.Fatalf("listen and serve: %v", err) + serveErr := make(chan error, 1) + go func() { + log.Printf("basemap pusher listening on http://%s", cfg.ListenAddr()) + if err := httpServer.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { + serveErr <- err + return + } + serveErr <- nil + }() + + select { + case err := <-serveErr: + if err != nil { + return fmt.Errorf("listen and serve: %w", err) + } + return nil + case <-ctx.Done(): + log.Printf("shutdown signal received, draining connections") + shutdownCtx, cancel := context.WithTimeout(context.Background(), shutdownGrace) + defer cancel() + if err := httpServer.Shutdown(shutdownCtx); err != nil { + log.Printf("graceful shutdown incomplete: %v", err) + _ = httpServer.Close() + } + if err := <-serveErr; err != nil { + log.Printf("server stopped with error: %v", err) + } + log.Printf("shutdown complete") + return nil } } diff --git a/services/api-go/cmd/api/service_other.go b/services/api-go/cmd/api/service_other.go new file mode 100644 index 0000000..11bc3e0 --- /dev/null +++ b/services/api-go/cmd/api/service_other.go @@ -0,0 +1,8 @@ +//go:build !windows + +package main + +// Service management is Windows-only; other platforms run the console binary. +func handleServiceArgs(command, envFile string) (bool, error) { return false, nil } + +func runAsWindowsService() bool { return false } diff --git a/services/api-go/cmd/api/service_windows.go b/services/api-go/cmd/api/service_windows.go new file mode 100644 index 0000000..d754bb0 --- /dev/null +++ b/services/api-go/cmd/api/service_windows.go @@ -0,0 +1,376 @@ +//go:build windows + +package main + +import ( + "context" + "fmt" + "log" + "os" + "path/filepath" + "strings" + "time" + + "golang.org/x/sys/windows/svc" + "golang.org/x/sys/windows/svc/mgr" +) + +const ( + serviceName = "BasemapPusher" + serviceDisplayName = "Basemap Pusher" + serviceDescription = "Internal catalog and push gateway for existing basemap tiles and vector assets." + // appEnvFileFlag is baked into the service command line so the service finds + // its config file even though the SCM starts it in C:\Windows\System32. + appEnvFileFlag = "--env-file" +) + +// handleServiceArgs implements the install/uninstall/start/stop/status verbs. +// The bool reports whether the verb was recognised. +func handleServiceArgs(command, envFile string) (bool, error) { + switch command { + case "install": + return true, installService(envFile) + case "uninstall": + return true, uninstallService() + case "start": + return true, startService() + case "stop": + return true, stopService() + case "status": + return true, serviceStatus() + default: + return false, nil + } +} + +// runAsWindowsService reports whether the process was launched by the SCM, and +// if so runs it to completion. It returns false for a normal console launch. +func runAsWindowsService() bool { + isService, err := svc.IsWindowsService() + if err != nil || !isService { + return false + } + restoreLog := setupServiceLogging() + defer restoreLog() + + if err := svc.Run(serviceName, &serviceHandler{}); err != nil { + log.Printf("windows service run: %v", err) + } + return true +} + +type serviceHandler struct{} + +func (h *serviceHandler) Execute(args []string, r <-chan svc.ChangeRequest, changes chan<- svc.Status) (bool, uint32) { + changes <- svc.Status{State: svc.StartPending} + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + done := make(chan error, 1) + go func() { done <- run(ctx) }() + + changes <- svc.Status{State: svc.Running, Accepts: svc.AcceptStop | svc.AcceptShutdown} + + for { + select { + case err := <-done: + if err != nil { + log.Printf("service stopped with error: %v", err) + changes <- svc.Status{State: svc.Stopped} + return false, 1 + } + changes <- svc.Status{State: svc.Stopped} + return false, 0 + case change := <-r: + switch change.Cmd { + case svc.Interrogate: + changes <- change.CurrentStatus + case svc.Stop, svc.Shutdown: + changes <- svc.Status{State: svc.StopPending} + cancel() + // Give the HTTP server its drain window before reporting stopped. + select { + case <-done: + case <-time.After(shutdownGrace + 5*time.Second): + log.Printf("service did not stop within the drain window") + } + changes <- svc.Status{State: svc.Stopped} + return false, 0 + } + } + } +} + +func installService(envFile string) error { + exe, err := os.Executable() + if err != nil { + return fmt.Errorf("resolve executable: %w", err) + } + exe, err = filepath.Abs(exe) + if err != nil { + return fmt.Errorf("resolve executable path: %w", err) + } + + resolvedEnv := envFile + if resolvedEnv == "" { + resolvedEnv = findDefaultEnvFile(filepath.Dir(exe)) + } + if resolvedEnv == "" { + return fmt.Errorf("no config/service.env found next to %s; pass --env-file to install", exe) + } + resolvedEnv, err = filepath.Abs(resolvedEnv) + if err != nil { + return fmt.Errorf("resolve env file: %w", err) + } + if _, err := os.Stat(resolvedEnv); err != nil { + return fmt.Errorf("env file %s is not readable: %w", resolvedEnv, err) + } + + manager, err := mgr.Connect() + if err != nil { + return fmt.Errorf("connect to the service manager (run this as Administrator): %w", err) + } + defer manager.Disconnect() + + if existing, err := manager.OpenService(serviceName); err == nil { + existing.Close() + return fmt.Errorf("service %q already exists; run uninstall first", serviceName) + } + + service, err := manager.CreateService(serviceName, exe, mgr.Config{ + DisplayName: serviceDisplayName, + Description: serviceDescription, + StartType: mgr.StartAutomatic, + }, appEnvFileFlag, resolvedEnv) + if err != nil { + return fmt.Errorf("create service: %w", err) + } + defer service.Close() + + // Restart on failure: 5s, 15s, 60s, then keep trying within the reset window. + if err := service.SetRecoveryActions([]mgr.RecoveryAction{ + {Type: mgr.ServiceRestart, Delay: 5 * time.Second}, + {Type: mgr.ServiceRestart, Delay: 15 * time.Second}, + {Type: mgr.ServiceRestart, Delay: 60 * time.Second}, + }, 86400); err != nil { + log.Printf("warning: could not set recovery actions: %v", err) + } + + log.Printf("installed service %q", serviceName) + log.Printf(" executable: %s", exe) + log.Printf(" env file: %s", resolvedEnv) + log.Printf(" start with: %s start", exe) + return nil +} + +func uninstallService() error { + manager, err := mgr.Connect() + if err != nil { + return fmt.Errorf("connect to the service manager (run this as Administrator): %w", err) + } + defer manager.Disconnect() + + service, err := manager.OpenService(serviceName) + if err != nil { + return fmt.Errorf("service %q is not installed", serviceName) + } + defer service.Close() + + if status, err := service.Query(); err == nil && status.State != svc.Stopped { + _, _ = service.Control(svc.Stop) + time.Sleep(2 * time.Second) + } + if err := service.Delete(); err != nil { + return fmt.Errorf("delete service: %w", err) + } + log.Printf("uninstalled service %q", serviceName) + return nil +} + +func startService() error { + manager, err := mgr.Connect() + if err != nil { + return fmt.Errorf("connect to the service manager: %w", err) + } + defer manager.Disconnect() + + service, err := manager.OpenService(serviceName) + if err != nil { + return fmt.Errorf("service %q is not installed", serviceName) + } + defer service.Close() + + if err := service.Start(); err != nil { + return fmt.Errorf("start service: %w", err) + } + log.Printf("started service %q", serviceName) + return nil +} + +func stopService() error { + manager, err := mgr.Connect() + if err != nil { + return fmt.Errorf("connect to the service manager: %w", err) + } + defer manager.Disconnect() + + service, err := manager.OpenService(serviceName) + if err != nil { + return fmt.Errorf("service %q is not installed", serviceName) + } + defer service.Close() + + status, err := service.Control(svc.Stop) + if err != nil { + return fmt.Errorf("stop service: %w", err) + } + deadline := time.Now().Add(30 * time.Second) + for status.State != svc.Stopped && time.Now().Before(deadline) { + time.Sleep(time.Second) + status, err = service.Query() + if err != nil { + return fmt.Errorf("query service: %w", err) + } + } + log.Printf("stopped service %q", serviceName) + return nil +} + +func serviceStatus() error { + manager, err := mgr.Connect() + if err != nil { + return fmt.Errorf("connect to the service manager: %w", err) + } + defer manager.Disconnect() + + service, err := manager.OpenService(serviceName) + if err != nil { + return fmt.Errorf("service %q is not installed", serviceName) + } + defer service.Close() + + status, err := service.Query() + if err != nil { + return fmt.Errorf("query service: %w", err) + } + names := map[svc.State]string{ + svc.Stopped: "stopped", + svc.StartPending: "start pending", + svc.StopPending: "stop pending", + svc.Running: "running", + svc.ContinuePending: "continue pending", + svc.PausePending: "pause pending", + svc.Paused: "paused", + } + name, ok := names[status.State] + if !ok { + name = fmt.Sprintf("state %d", status.State) + } + fmt.Printf("%s: %s\n", serviceName, name) + return nil +} + +// findDefaultEnvFile looks in the usual layouts: next to the executable, then +// one and two levels up (services/api-go -> repo root). +func findDefaultEnvFile(startDir string) string { + candidates := []string{ + filepath.Join(startDir, "config", "service.env"), + filepath.Join(startDir, "..", "config", "service.env"), + filepath.Join(startDir, "..", "..", "config", "service.env"), + } + for _, candidate := range candidates { + if _, err := os.Stat(candidate); err == nil { + return candidate + } + } + return "" +} + +// setupServiceLogging points the standard logger at a file, because a Windows +// service has no console and would otherwise discard every log line. It returns +// a restore func that is safe to call even when nothing was configured. +func setupServiceLogging() func() { + path := strings.TrimSpace(os.Getenv("SERVICE_LOG_FILE")) + if path == "" { + exe, err := os.Executable() + if err != nil { + return func() {} + } + path = filepath.Join(filepath.Dir(exe), "logs", "basemap-pusher.log") + } + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return func() {} + } + writer, err := newRotatingFileWriter(path, serviceLogMaxBytes) + if err != nil { + return func() {} + } + log.SetOutput(writer) + log.Printf("service log: %s (rotates at %d MB)", path, serviceLogMaxBytes>>20) + return func() { _ = writer.Close() } +} + +const serviceLogMaxBytes = 10 << 20 + +// rotatingFileWriter appends to one file and rolls it to ".1" once it grows +// past maxBytes, so a long-running service cannot fill the disk. +type rotatingFileWriter struct { + path string + maxBytes int64 + file *os.File + size int64 +} + +func newRotatingFileWriter(path string, maxBytes int64) (*rotatingFileWriter, error) { + writer := &rotatingFileWriter{path: path, maxBytes: maxBytes} + if err := writer.open(); err != nil { + return nil, err + } + return writer, nil +} + +func (w *rotatingFileWriter) open() error { + file, err := os.OpenFile(w.path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o644) + if err != nil { + return err + } + info, err := file.Stat() + if err != nil { + file.Close() + return err + } + w.file = file + w.size = info.Size() + return nil +} + +func (w *rotatingFileWriter) Write(p []byte) (int, error) { + if w.maxBytes > 0 && w.size+int64(len(p)) > w.maxBytes { + w.roll() + } + n, err := w.file.Write(p) + w.size += int64(n) + return n, err +} + +func (w *rotatingFileWriter) roll() { + if w.file != nil { + _ = w.file.Close() + } + // Windows refuses to rename onto an existing path, so drop the old roll. + _ = os.Remove(w.path + ".1") + _ = os.Rename(w.path, w.path+".1") + w.size = 0 + if err := w.open(); err != nil { + // Fall back to appending to the original path if the reopen failed. + _ = err + } +} + +func (w *rotatingFileWriter) Close() error { + if w.file == nil { + return nil + } + return w.file.Close() +} diff --git a/services/api-go/cmd/api/service_windows_test.go b/services/api-go/cmd/api/service_windows_test.go new file mode 100644 index 0000000..20fabf3 --- /dev/null +++ b/services/api-go/cmd/api/service_windows_test.go @@ -0,0 +1,47 @@ +//go:build windows + +package main + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +func TestRotatingFileWriterRollsOver(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "svc.log") + + writer, err := newRotatingFileWriter(path, 64) + if err != nil { + t.Fatalf("open writer: %v", err) + } + defer writer.Close() + + for i := 0; i < 10; i++ { + if _, err := writer.Write([]byte(strings.Repeat("x", 20) + "\n")); err != nil { + t.Fatalf("write %d: %v", i, err) + } + } + + if _, err := os.Stat(path); err != nil { + t.Fatalf("current log missing: %v", err) + } + if _, err := os.Stat(path + ".1"); err != nil { + t.Fatalf("expected a rolled file at %s.1: %v", path, err) + } +} + +func TestSetupServiceLoggingUsesConfiguredPath(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "nested", "service.log") + t.Setenv("SERVICE_LOG_FILE", path) + + restore := setupServiceLogging() + defer restore() + + if _, err := os.Stat(path); err != nil { + t.Fatalf("expected the log file at %s: %v", path, err) + } +} diff --git a/services/api-go/go.mod b/services/api-go/go.mod index 50234bd..beaba62 100644 --- a/services/api-go/go.mod +++ b/services/api-go/go.mod @@ -1,4 +1,4 @@ -module map-asset-gateway/api-go +module tile-server/api-go go 1.26 diff --git a/services/api-go/internal/basemap/auth.go b/services/api-go/internal/basemap/auth.go index edb5ac4..e6f30b5 100644 --- a/services/api-go/internal/basemap/auth.go +++ b/services/api-go/internal/basemap/auth.go @@ -12,10 +12,15 @@ import ( "path/filepath" "sort" "strings" + "time" - "map-asset-gateway/api-go/internal/uid" + "tile-server/api-go/internal/uid" ) +// lastUsedUpdateInterval throttles service_tokens.last_used_at writes so tile +// traffic does not issue one UPDATE per tile request. +const lastUsedUpdateInterval = time.Minute + var ErrBasemapVersionNotFound = errors.New("basemap version not found") func (s *Store) CreateToken(ctx context.Context, input CreateTokenInput) (CreatedToken, error) { @@ -23,11 +28,16 @@ func (s *Store) CreateToken(ctx context.Context, input CreateTokenInput) (Create if name == "" { return CreatedToken{}, errors.New("token name is required") } - if len(input.BasemapCodes) == 0 { - if len(input.VectorCodes) == 0 { - return CreatedToken{}, errors.New("at least one basemap code or vector code is required") + // A token needs at least one capability: read grants, or the slice scope. + hasSliceScope := false + for _, value := range input.Scopes { + if strings.EqualFold(strings.TrimSpace(value), slicePermission) { + hasSliceScope = true } } + if len(input.BasemapCodes) == 0 && len(input.VectorCodes) == 0 && !hasSliceScope { + return CreatedToken{}, errors.New("at least one basemap code, vector code, or the slice scope is required") + } resolvedVectorAssets := map[string]VectorAsset{} for _, ref := range input.VectorCodes { @@ -54,15 +64,21 @@ func (s *Store) CreateToken(ctx context.Context, input CreateTokenInput) (Create } now := nowUTC() + // Fall back to the configured default lifetime so tokens do not live forever + // unless an explicit expiry was supplied. + if input.ExpiresAt == nil && s.cfg.TokenDefaultTTL > 0 { + expiresAt := now.Add(s.cfg.TokenDefaultTTL) + input.ExpiresAt = &expiresAt + } tokenID := uid.New() prefix := token if len(prefix) > 16 { prefix = prefix[:16] } _, err = tx.ExecContext(ctx, ` - INSERT INTO service_tokens (id, name, token_hash, token_prefix, status, expires_at, last_used_at, created_at) - VALUES (?, ?, ?, ?, 'active', ?, NULL, ?) - `, tokenID, name, tokenHash, prefix, nullableTime(input.ExpiresAt), toRFC3339(now)) + INSERT INTO service_tokens (id, name, token_hash, token_prefix, scopes, status, expires_at, last_used_at, created_at) + VALUES (?, ?, ?, ?, ?, 'active', ?, NULL, ?) + `, tokenID, name, tokenHash, prefix, normalizeTokenScopes(input.Scopes), nullableTime(input.ExpiresAt), toRFC3339(now)) if err != nil { return CreatedToken{}, err } @@ -75,7 +91,7 @@ func (s *Store) CreateToken(ctx context.Context, input CreateTokenInput) (Create } } if len(normalizedCodes) == 0 { - if len(resolvedVectorAssets) == 0 { + if len(resolvedVectorAssets) == 0 && !hasSliceScope { return CreatedToken{}, errors.New("no valid basemap codes provided") } } @@ -129,7 +145,7 @@ func (s *Store) CreateToken(ctx context.Context, input CreateTokenInput) (Create func (s *Store) ListTokens(ctx context.Context) ([]ServiceToken, error) { rows, err := s.db.QueryContext(ctx, ` - SELECT id, name, token_hash, token_prefix, status, expires_at, last_used_at, created_at + SELECT id, name, token_hash, token_prefix, scopes, status, expires_at, last_used_at, created_at FROM service_tokens ORDER BY created_at DESC `) @@ -226,6 +242,69 @@ func (s *Store) loadAllVectorTokenGrants(ctx context.Context) (map[string][]Vect return result, rows.Err() } +func (s *Store) loadTokenGrants(ctx context.Context, tokenID string) ([]TokenGrant, error) { + rows, err := s.db.QueryContext(ctx, ` + SELECT + g.id, + g.token_id, + g.basemap_id, + b.code, + g.basemap_version_id, + v.version, + g.permission, + g.created_at + FROM token_grants g + JOIN basemaps b ON b.id = g.basemap_id + LEFT JOIN basemap_versions v ON v.id = g.basemap_version_id + WHERE g.token_id = ? + ORDER BY g.created_at DESC + `, tokenID) + if err != nil { + return nil, err + } + defer rows.Close() + + var items []TokenGrant + for rows.Next() { + item, err := scanTokenGrant(rows) + if err != nil { + return nil, err + } + items = append(items, item) + } + return items, rows.Err() +} + +func (s *Store) loadVectorTokenGrants(ctx context.Context, tokenID string) ([]VectorTokenGrant, error) { + rows, err := s.db.QueryContext(ctx, ` + SELECT + g.id, + g.token_id, + g.vector_asset_id, + v.code, + g.permission, + g.created_at + FROM vector_token_grants g + JOIN vector_assets v ON v.id = g.vector_asset_id + WHERE g.token_id = ? + ORDER BY g.created_at DESC + `, tokenID) + if err != nil { + return nil, err + } + defer rows.Close() + + var items []VectorTokenGrant + for rows.Next() { + item, err := scanVectorTokenGrant(rows) + if err != nil { + return nil, err + } + items = append(items, item) + } + return items, rows.Err() +} + func (s *Store) DisableToken(ctx context.Context, tokenID string) error { result, err := s.db.ExecContext(ctx, `UPDATE service_tokens SET status = 'disabled' WHERE id = ?`, strings.TrimSpace(tokenID)) if err != nil { @@ -243,7 +322,7 @@ func (s *Store) DisableToken(ctx context.Context, tokenID string) error { func (s *Store) getTokenByID(ctx context.Context, tokenID string) (ServiceToken, error) { row := s.db.QueryRowContext(ctx, ` - SELECT id, name, token_hash, token_prefix, status, expires_at, last_used_at, created_at + SELECT id, name, token_hash, token_prefix, scopes, status, expires_at, last_used_at, created_at FROM service_tokens WHERE id = ? `, tokenID) @@ -273,7 +352,7 @@ func (s *Store) AuthorizeToken(ctx context.Context, rawToken string) (TokenAuth, tokenHash := hex.EncodeToString(sum[:]) row := s.db.QueryRowContext(ctx, ` - SELECT id, name, token_hash, token_prefix, status, expires_at, last_used_at, created_at + SELECT id, name, token_hash, token_prefix, scopes, status, expires_at, last_used_at, created_at FROM service_tokens WHERE token_hash = ? `, tokenHash) @@ -291,16 +370,16 @@ func (s *Store) AuthorizeToken(ctx context.Context, rawToken string) (TokenAuth, return TokenAuth{}, errors.New("token is expired") } - grantsByToken, err := s.loadAllTokenGrants(ctx) + grants, err := s.loadTokenGrants(ctx, item.ID) if err != nil { return TokenAuth{}, err } - item.Grants = grantsByToken[item.ID] - vectorGrantsByToken, err := s.loadAllVectorTokenGrants(ctx) + item.Grants = grants + vectorGrants, err := s.loadVectorTokenGrants(ctx, item.ID) if err != nil { return TokenAuth{}, err } - item.VectorGrants = vectorGrantsByToken[item.ID] + item.VectorGrants = vectorGrants permissions := map[string]map[string]VersionScope{} for _, grant := range item.Grants { group, ok := permissions[grant.Permission] @@ -329,14 +408,64 @@ func (s *Store) AuthorizeToken(ctx context.Context, rawToken string) (TokenAuth, } group[grant.VectorCode] = struct{}{} } - _, _ = s.db.ExecContext(ctx, `UPDATE service_tokens SET last_used_at = ? WHERE id = ?`, toRFC3339(nowUTC()), item.ID) + if item.LastUsedAt == nil || nowUTC().Sub(*item.LastUsedAt) >= lastUsedUpdateInterval { + _, _ = s.db.ExecContext(ctx, `UPDATE service_tokens SET last_used_at = ? WHERE id = ?`, toRFC3339(nowUTC()), item.ID) + } return TokenAuth{ Token: item, Permissions: permissions, VectorPermissions: vectorPermissions, + Scopes: scopeSet(item.Scopes), }, nil } +// Token scopes are coarse capabilities layered on top of per-resource grants. +// Every token can read; "slice" additionally allows submitting tile jobs. +const slicePermission = "slice" + +var allowedTokenScopes = map[string]struct{}{ + readPermission: {}, + slicePermission: {}, +} + +// normalizeTokenScopes keeps only known scopes and always includes read, so the +// stored column is a stable, sorted, comma-separated list. +func normalizeTokenScopes(values []string) string { + scopes := map[string]struct{}{readPermission: {}} + for _, value := range values { + scope := strings.ToLower(strings.TrimSpace(value)) + if _, ok := allowedTokenScopes[scope]; ok { + scopes[scope] = struct{}{} + } + } + return strings.Join(sortedKeys(scopes), ",") +} + +func splitTokenScopes(value string) []string { + parts := strings.Split(value, ",") + scopes := make([]string, 0, len(parts)) + for _, part := range parts { + scope := strings.ToLower(strings.TrimSpace(part)) + if scope != "" { + scopes = append(scopes, scope) + } + } + return scopes +} + +func scopeSet(values []string) map[string]struct{} { + scopes := make(map[string]struct{}, len(values)) + for _, value := range values { + scopes[value] = struct{}{} + } + return scopes +} + +// CanSlice reports whether the token may submit tile jobs. +func (auth TokenAuth) CanSlice() bool { + _, ok := auth.Scopes[slicePermission] + return ok +} func (auth TokenAuth) CanReadBasemap(code string, version string) bool { group, ok := auth.Permissions[readPermission] if !ok { diff --git a/services/api-go/internal/basemap/catalog.go b/services/api-go/internal/basemap/catalog.go index 8a61616..d872c05 100644 --- a/services/api-go/internal/basemap/catalog.go +++ b/services/api-go/internal/basemap/catalog.go @@ -8,7 +8,7 @@ import ( "path/filepath" "strings" - "map-asset-gateway/api-go/internal/uid" + "tile-server/api-go/internal/uid" ) func (s *Store) EnsureScanSource(ctx context.Context, input CreateScanSourceInput) (ScanSource, error) { @@ -61,7 +61,7 @@ func (s *Store) GetScanSourceByCode(ctx context.Context, code string) (ScanSourc item, err := scanScanSource(row) if err != nil { if errors.Is(err, sql.ErrNoRows) { - return ScanSource{}, fmt.Errorf("scan source %q not found", code) + return ScanSource{}, &scanSourceError{sentinel: ErrScanSourceNotFound, code: code, reason: "not found"} } return ScanSource{}, err } @@ -168,17 +168,66 @@ func (s *Store) ListBasemaps(ctx context.Context) ([]Basemap, error) { } func (s *Store) GetBasemapByCode(ctx context.Context, code string) (Basemap, error) { - items, err := s.ListBasemaps(ctx) + normalized := normalizeBasemapCode(code) + row := s.db.QueryRowContext(ctx, ` + SELECT id, code, name, type, status, description, created_at, updated_at + FROM basemaps + WHERE code = ? + `, normalized) + item, err := scanBasemap(row) + if err != nil { + if errors.Is(err, sql.ErrNoRows) { + return Basemap{}, fmt.Errorf("basemap %q not found", code) + } + return Basemap{}, err + } + + versionRows, err := s.db.QueryContext(ctx, ` + SELECT + v.id, + v.basemap_id, + b.code, + v.version, + v.status, + v.is_default, + v.manifest_path, + v.tile_root_path, + v.url_template, + v.tile_format, + v.tile_scheme, + v.min_zoom, + v.max_zoom, + v.bbox_json, + v.attribution, + v.metadata_json, + v.created_at, + v.updated_at + FROM basemap_versions v + JOIN basemaps b ON b.id = v.basemap_id + WHERE v.basemap_id = ? + ORDER BY v.version + `, item.ID) if err != nil { return Basemap{}, err } - normalized := normalizeBasemapCode(code) - for _, item := range items { - if item.Code == normalized { - return item, nil + defer versionRows.Close() + + for versionRows.Next() { + row, err := scanBasemapVersionRow(versionRows) + if err != nil { + return Basemap{}, err + } + version := decodeBasemapVersion(row) + item.Versions = append(item.Versions, version) + if version.IsDefault { + copyValue := version + item.Default = ©Value } } - return Basemap{}, fmt.Errorf("basemap %q not found", code) + if err := versionRows.Err(); err != nil { + return Basemap{}, err + } + return item, nil } func (s *Store) SetDefaultVersion(ctx context.Context, basemapCode, version string) error { @@ -276,6 +325,8 @@ func (s *Store) ListPushRecords(ctx context.Context, limit int) ([]PushRecord, e p.response_status, p.response_body, p.error_message, + p.attempt_count, + p.next_attempt_at, p.pushed_at, p.finished_at FROM push_records p diff --git a/services/api-go/internal/basemap/dashboard.go b/services/api-go/internal/basemap/dashboard.go index 20e2eb9..c19368f 100644 --- a/services/api-go/internal/basemap/dashboard.go +++ b/services/api-go/internal/basemap/dashboard.go @@ -35,6 +35,22 @@ func (s *Store) Dashboard(ctx context.Context, info SystemInfo, flash string, cr if err != nil { return DashboardData{}, err } + versionHealth, err := s.loadVersionHealth(ctx) + if err != nil { + return DashboardData{}, err + } + vectorHealth, err := s.loadVectorHealth(ctx) + if err != nil { + return DashboardData{}, err + } + tileJobs, err := s.ListTileJobs(ctx, 10) + if err != nil { + return DashboardData{}, err + } + layers, err := s.ListLayerInventories(ctx) + if err != nil { + return DashboardData{}, err + } return DashboardData{ SystemInfo: info, Basemaps: basemaps, @@ -47,5 +63,9 @@ func (s *Store) Dashboard(ctx context.Context, info SystemInfo, flash string, cr VectorPushRecords: vectorPushRecords, Flash: flash, CreatedToken: createdToken, + VersionHealth: versionHealth, + VectorHealth: vectorHealth, + TileJobs: tileJobs, + Layers: layers, }, nil } diff --git a/services/api-go/internal/basemap/features_test.go b/services/api-go/internal/basemap/features_test.go new file mode 100644 index 0000000..658d759 --- /dev/null +++ b/services/api-go/internal/basemap/features_test.go @@ -0,0 +1,872 @@ +package basemap + +import ( + "context" + "database/sql" + "encoding/json" + "errors" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "runtime" + "strings" + "testing" + "time" +) + +func TestCheckHealthFlagsMissingPaths(t *testing.T) { + ctx := context.Background() + root := t.TempDir() + tileRoot := filepath.Join(root, "tiles") + vectorRoot := filepath.Join(root, "vectors") + writeTestFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile") + writeTestFile(t, filepath.Join(vectorRoot, "boundary.geojson"), `{"type":"FeatureCollection","features":[]}`) + + store := openTestStore(t, StoreConfig{ + SQLitePath: filepath.Join(root, "basemap.db"), + ScanRoot: tileRoot, + VectorScanRoot: vectorRoot, + APIBaseURL: "http://127.0.0.1:8910", + TileBaseURL: "http://127.0.0.1:8910", + VectorBaseURL: "http://127.0.0.1:8910", + }) + if _, err := store.RunScan(ctx, defaultScanSourceCode); err != nil { + t.Fatalf("scan: %v", err) + } + if _, err := store.RunVectorScan(ctx); err != nil { + t.Fatalf("vector scan: %v", err) + } + + report, err := store.CheckHealth(ctx) + if err != nil { + t.Fatalf("check health: %v", err) + } + if report.VersionsMissing != 0 || report.VectorsMissing != 0 { + t.Fatalf("expected everything healthy, got %+v", report) + } + + // A directory that disappears must be reported before the first 404. + if err := os.RemoveAll(filepath.Join(tileRoot, "google_image")); err != nil { + t.Fatalf("remove tile root: %v", err) + } + if err := os.Remove(filepath.Join(vectorRoot, "boundary.geojson")); err != nil { + t.Fatalf("remove vector: %v", err) + } + + report, err = store.CheckHealth(ctx) + if err != nil { + t.Fatalf("check health after removal: %v", err) + } + if report.VersionsMissing != 1 || report.VectorsMissing != 1 { + t.Fatalf("expected one missing of each, got %+v", report) + } + + versionHealth, err := store.loadVersionHealth(ctx) + if err != nil { + t.Fatalf("load version health: %v", err) + } + assetHealth, err := store.loadVectorHealth(ctx) + if err != nil { + t.Fatalf("load vector health: %v", err) + } + foundMissing := false + for _, health := range versionHealth { + if health.Health == healthMissing { + foundMissing = true + } + } + if !foundMissing { + t.Fatalf("expected a missing version verdict, got %+v", versionHealth) + } + foundMissing = false + for _, health := range assetHealth { + if health.Health == healthMissing { + foundMissing = true + } + } + if !foundMissing { + t.Fatalf("expected a missing vector verdict, got %+v", assetHealth) + } +} + +func TestPushRetrySucceedsAfterBackoff(t *testing.T) { + ctx := context.Background() + root := t.TempDir() + tileRoot := filepath.Join(root, "tiles") + writeTestFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile") + + var attempts int + callback := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + attempts++ + if attempts < 2 { + w.WriteHeader(http.StatusInternalServerError) + return + } + w.WriteHeader(http.StatusOK) + })) + defer callback.Close() + + store := openTestStore(t, StoreConfig{ + SQLitePath: filepath.Join(root, "basemap.db"), + ScanRoot: tileRoot, + APIBaseURL: "http://127.0.0.1:8910", + TileBaseURL: "http://127.0.0.1:8910", + VectorBaseURL: "http://127.0.0.1:8910", + }) + if _, err := store.RunScan(ctx, defaultScanSourceCode); err != nil { + t.Fatalf("scan: %v", err) + } + if _, err := store.CreateTargetSystem(ctx, CreateTargetSystemInput{Code: "downstream", CallbackURL: callback.URL}); err != nil { + t.Fatalf("create target: %v", err) + } + record, err := store.PushBasemapVersion(ctx, "downstream", "google_image", "current") + if err != nil { + t.Fatalf("push: %v", err) + } + + // First attempt fails, so the record must be parked in retrying with a + // scheduled next attempt rather than being dropped as failed. + deadline := time.Now().Add(5 * time.Second) + var parked bool + for time.Now().Before(deadline) { + records, err := store.ListPushRecords(ctx, 5) + if err != nil { + t.Fatalf("list: %v", err) + } + for _, item := range records { + if item.ID == record.ID && item.Status == pushStatusRetrying && item.NextAttemptAt != nil { + parked = true + } + } + if parked { + break + } + time.Sleep(20 * time.Millisecond) + } + if !parked { + t.Fatal("expected the failed push to be rescheduled rather than dropped") + } + + // Make the retry due immediately, then run the dispatcher. + if _, err := store.db.ExecContext(ctx, `UPDATE push_records SET next_attempt_at = ? WHERE id = ?`, toRFC3339(nowUTC().Add(-time.Minute)), record.ID); err != nil { + t.Fatalf("force due: %v", err) + } + if err := store.dispatchPushRetries(ctx); err != nil { + t.Fatalf("dispatch retries: %v", err) + } + + records, err := store.ListPushRecords(ctx, 5) + if err != nil { + t.Fatalf("list after retry: %v", err) + } + var final PushRecord + for _, item := range records { + if item.ID == record.ID { + final = item + } + } + if final.Status != pushStatusSucceeded { + t.Fatalf("expected the retry to succeed, got status=%q message=%q", final.Status, final.ErrorMessage) + } + if final.AttemptCount != 2 { + t.Fatalf("expected 2 attempts, got %d", final.AttemptCount) + } +} + +func TestManualRetryRequeuesFailedPush(t *testing.T) { + ctx := context.Background() + root := t.TempDir() + tileRoot := filepath.Join(root, "tiles") + writeTestFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile") + + callback := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusBadGateway) + })) + defer callback.Close() + + store := openTestStore(t, StoreConfig{ + SQLitePath: filepath.Join(root, "basemap.db"), + ScanRoot: tileRoot, + APIBaseURL: "http://127.0.0.1:8910", + TileBaseURL: "http://127.0.0.1:8910", + VectorBaseURL: "http://127.0.0.1:8910", + }) + if _, err := store.RunScan(ctx, defaultScanSourceCode); err != nil { + t.Fatalf("scan: %v", err) + } + if _, err := store.CreateTargetSystem(ctx, CreateTargetSystemInput{Code: "downstream", CallbackURL: callback.URL}); err != nil { + t.Fatalf("create target: %v", err) + } + record, err := store.PushBasemapVersion(ctx, "downstream", "google_image", "current") + if err != nil { + t.Fatalf("push: %v", err) + } + + // Exhaust the retry budget so the record lands in failed. + deadline := time.Now().Add(5 * time.Second) + for time.Now().Before(deadline) { + records, _ := store.ListPushRecords(ctx, 5) + for _, item := range records { + if item.ID == record.ID && item.Status == pushStatusFailed { + goto failed + } + } + if _, err := store.db.ExecContext(ctx, `UPDATE push_records SET next_attempt_at = ? WHERE id = ? AND status = ?`, toRFC3339(nowUTC().Add(-time.Minute)), record.ID, pushStatusRetrying); err != nil { + t.Fatalf("force due: %v", err) + } + _ = store.dispatchPushRetries(ctx) + time.Sleep(20 * time.Millisecond) + } + +failed: + records, err := store.ListPushRecords(ctx, 5) + if err != nil { + t.Fatalf("list: %v", err) + } + var failedRecord PushRecord + for _, item := range records { + if item.ID == record.ID { + failedRecord = item + } + } + if failedRecord.Status != pushStatusFailed { + t.Fatalf("expected a failed record, got %q", failedRecord.Status) + } + if failedRecord.AttemptCount != pushMaxAttempts { + t.Fatalf("expected %d attempts, got %d", pushMaxAttempts, failedRecord.AttemptCount) + } + + if err := store.RetryPush(ctx, "basemap", record.ID); err != nil { + t.Fatalf("manual retry: %v", err) + } + records, err = store.ListPushRecords(ctx, 5) + if err != nil { + t.Fatalf("list after manual retry: %v", err) + } + for _, item := range records { + if item.ID != record.ID { + continue + } + if item.AttemptCount != 0 { + t.Fatalf("expected the manual retry to reset the attempt counter, got %d", item.AttemptCount) + } + if item.Status == pushStatusSucceeded { + t.Fatal("a failing target must not report success") + } + } +} + +func TestCreateTileJobValidation(t *testing.T) { + ctx := context.Background() + root := t.TempDir() + tileRoot := filepath.Join(root, "tiles") + source := filepath.Join(root, "input.tif") + writeTestFile(t, source, "fake raster") + + store := openTestStore(t, StoreConfig{ + SQLitePath: filepath.Join(root, "basemap.db"), + ScanRoot: tileRoot, + APIBaseURL: "http://127.0.0.1:8910", + TileBaseURL: "http://127.0.0.1:8910", + VectorBaseURL: "http://127.0.0.1:8910", + }) + + if _, err := store.CreateTileJob(ctx, CreateTileJobInput{TargetCode: "x", SourcePath: ""}); err == nil { + t.Fatal("expected a missing source path to be rejected") + } + if _, err := store.CreateTileJob(ctx, CreateTileJobInput{SourcePath: source}); err == nil { + t.Fatal("expected a missing target code to be rejected") + } + if _, err := store.CreateTileJob(ctx, CreateTileJobInput{SourcePath: source, TargetCode: "x", ZoomMin: 5, ZoomMax: 2}); err == nil { + t.Fatal("expected an inverted zoom range to be rejected") + } + if _, err := store.CreateTileJob(ctx, CreateTileJobInput{SourcePath: filepath.Join(root, "missing.tif"), TargetCode: "x"}); err == nil { + t.Fatal("expected a missing source file to be rejected") + } + + job, err := store.CreateTileJob(ctx, CreateTileJobInput{SourcePath: source, TargetCode: "demo", ZoomMin: 0, ZoomMax: 3}) + if err != nil { + t.Fatalf("create job: %v", err) + } + if job.Status != tileJobPending { + t.Fatalf("expected a pending job, got %q", job.Status) + } + if job.TileSize != 256 || job.TileFormat != "webp" { + t.Fatalf("expected defaults to be applied, got %d/%s", job.TileSize, job.TileFormat) + } + if !strings.HasSuffix(job.OutputRoot, filepath.Join("tiles", "demo")) { + t.Fatalf("unexpected output root %q", job.OutputRoot) + } +} + +func TestTileJobCommandArgsSubstitution(t *testing.T) { + job := TileJob{ + SourcePath: "/data/in.tif", + OutputRoot: "/data/out", + ZoomMin: 1, + ZoomMax: 5, + TileSize: 512, + TileFormat: "png", + TargetCode: "demo", + } + args := tileJobCommandArgs(`gdal2tiles.py --xyz --zoom={zmin}-{zmax} --tilesize={tilesize} --format={format} {input} {output}`, job, job.SourcePath) + want := []string{"gdal2tiles.py", "--xyz", "--zoom=1-5", "--tilesize=512", "--format=png", "/data/in.tif", "/data/out"} + if len(args) != len(want) { + t.Fatalf("expected %d args, got %v", len(want), args) + } + for i := range want { + if args[i] != want[i] { + t.Fatalf("arg %d = %q, want %q", i, args[i], want[i]) + } + } +} + +func TestSplitCommandLineKeepsQuotedSegments(t *testing.T) { + parts := splitCommandLine(`cmd /c mkdir "C:\Program Files\tiles\0"`) + if len(parts) != 4 { + t.Fatalf("expected 4 parts, got %v", parts) + } + if parts[3] != `C:\Program Files\tiles\0` { + t.Fatalf("quoted segment not preserved: %q", parts[3]) + } +} + +func TestTileJobRunsCommandThenScans(t *testing.T) { + if runtime.GOOS != "windows" { + t.Skip("the stub slicer below uses cmd.exe") + } + ctx := context.Background() + root := t.TempDir() + tileRoot := filepath.Join(root, "tiles") + source := filepath.Join(root, "input.tif") + writeTestFile(t, source, "fake raster") + if err := os.MkdirAll(tileRoot, 0o755); err != nil { + t.Fatalf("mkdir tile root: %v", err) + } + + store := openTestStore(t, StoreConfig{ + SQLitePath: filepath.Join(root, "basemap.db"), + ScanRoot: tileRoot, + APIBaseURL: "http://127.0.0.1:8910", + TileBaseURL: "http://127.0.0.1:8910", + VectorBaseURL: "http://127.0.0.1:8910", + // Stand-in for gdal2tiles: lay down a z/x tree the scanner can read. + TileJobCommand: `cmd /c mkdir "{output}\0\0"`, + }) + if _, err := store.RunScan(ctx, defaultScanSourceCode); err != nil { + t.Fatalf("initial scan: %v", err) + } + + job, err := store.CreateTileJob(ctx, CreateTileJobInput{ + SourcePath: source, + TargetCode: "demo", + ZoomMin: 0, + ZoomMax: 0, + }) + if err != nil { + t.Fatalf("create job: %v", err) + } + store.runTileJob(ctx, job) + + tileJobs, err := store.ListTileJobs(ctx, 5) + if err != nil { + t.Fatalf("list jobs: %v", err) + } + if len(tileJobs) != 1 { + t.Fatalf("expected 1 job, got %d", len(tileJobs)) + } + if tileJobs[0].Status != tileJobSucceeded { + t.Fatalf("expected success, got %q (%s)", tileJobs[0].Status, tileJobs[0].Message) + } + + basemaps, err := store.ListBasemaps(ctx) + if err != nil { + t.Fatalf("list basemaps: %v", err) + } + found := false + for _, item := range basemaps { + if item.Code == "demo" { + found = true + } + } + if !found { + t.Fatalf("expected the sliced basemap to be scanned in, got %+v", basemaps) + } +} + +func TestTileJobFailureIsRecorded(t *testing.T) { + ctx := context.Background() + root := t.TempDir() + tileRoot := filepath.Join(root, "tiles") + source := filepath.Join(root, "input.tif") + writeTestFile(t, source, "fake raster") + + store := openTestStore(t, StoreConfig{ + SQLitePath: filepath.Join(root, "basemap.db"), + ScanRoot: tileRoot, + APIBaseURL: "http://127.0.0.1:8910", + TileBaseURL: "http://127.0.0.1:8910", + VectorBaseURL: "http://127.0.0.1:8910", + TileJobCommand: "definitely-not-a-real-binary {input} {output}", + }) + job, err := store.CreateTileJob(ctx, CreateTileJobInput{SourcePath: source, TargetCode: "demo"}) + if err != nil { + t.Fatalf("create job: %v", err) + } + store.runTileJob(ctx, job) + + jobs, err := store.ListTileJobs(ctx, 5) + if err != nil { + t.Fatalf("list jobs: %v", err) + } + if jobs[0].Status != tileJobFailed { + t.Fatalf("expected failure, got %q", jobs[0].Status) + } + if jobs[0].Message == "" { + t.Fatal("expected the failure reason to be recorded") + } + t.Logf("DEBUG job=%+v", jobs[0]) + if jobs[0].LogPath == "" { + t.Fatal("expected a log path to be recorded") + } +} + +func TestRetryPushRejectsUnknownKind(t *testing.T) { + store := openTestStore(t, StoreConfig{SQLitePath: filepath.Join(t.TempDir(), "basemap.db")}) + if err := store.RetryPush(context.Background(), "nonsense", "x"); err == nil { + t.Fatal("expected an unknown kind to be rejected") + } +} + +func TestGetTargetSystemByIDMissing(t *testing.T) { + store := openTestStore(t, StoreConfig{SQLitePath: filepath.Join(t.TempDir(), "basemap.db")}) + _, err := store.GetTargetSystemByID(context.Background(), "nope") + if err == nil || errors.Is(err, os.ErrNotExist) { + t.Fatalf("expected a not-found error, got %v", err) + } +} + +func TestBootstrapMigratesLegacySchema(t *testing.T) { + root := t.TempDir() + dbPath := filepath.Join(root, "legacy.db") + + // A database created before the health and retry columns existed. + legacy, err := sql.Open("sqlite", "file:"+filepath.ToSlash(dbPath)) + if err != nil { + t.Fatalf("open legacy db: %v", err) + } + legacyDDL := []string{ + `CREATE TABLE basemap_versions ( + id TEXT PRIMARY KEY, + basemap_id TEXT NOT NULL, + scan_source_id TEXT, + version TEXT NOT NULL, + status TEXT NOT NULL, + is_default INTEGER NOT NULL DEFAULT 0, + manifest_path TEXT NOT NULL DEFAULT '', + tile_root_path TEXT NOT NULL, + url_template TEXT NOT NULL, + tile_format TEXT NOT NULL, + tile_scheme TEXT NOT NULL, + min_zoom INTEGER NOT NULL DEFAULT 0, + max_zoom INTEGER NOT NULL DEFAULT 0, + bbox_json TEXT NOT NULL DEFAULT '', + attribution TEXT NOT NULL DEFAULT '', + metadata_json TEXT NOT NULL DEFAULT '', + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL + );`, + `CREATE TABLE push_records ( + id TEXT PRIMARY KEY, + target_system_id TEXT NOT NULL, + basemap_version_id TEXT NOT NULL, + status TEXT NOT NULL, + request_json TEXT NOT NULL DEFAULT '', + response_status INTEGER, + response_body TEXT NOT NULL DEFAULT '', + error_message TEXT NOT NULL DEFAULT '', + pushed_at TEXT NOT NULL, + finished_at TEXT + );`, + `INSERT INTO basemap_versions (id, basemap_id, version, status, tile_root_path, url_template, tile_format, tile_scheme, created_at, updated_at) + VALUES ('v1', 'b1', 'current', 'ready', '/tiles/demo', '/tiles/demo/current/{z}/{x}/{y}.png', 'png', 'xyz', '2026-01-01T00:00:00Z', '2026-01-01T00:00:00Z');`, + } + for _, statement := range legacyDDL { + if _, err := legacy.Exec(statement); err != nil { + legacy.Close() + t.Fatalf("seed legacy schema: %v", err) + } + } + if err := legacy.Close(); err != nil { + t.Fatalf("close legacy db: %v", err) + } + + store := openTestStore(t, StoreConfig{SQLitePath: dbPath, HealthCheckInterval: time.Minute}) + + // The pre-existing row must survive and gain default health values. + var health, detail string + if err := store.db.QueryRow(`SELECT health, health_detail FROM basemap_versions WHERE id = 'v1'`).Scan(&health, &detail); err != nil { + t.Fatalf("legacy row lost or columns missing: %v", err) + } + if health != "" { + t.Fatalf("expected a blank default health, got %q", health) + } + + var attempts int + var nextAttempt sql.NullString + if err := store.db.QueryRow(`SELECT attempt_count, next_attempt_at FROM push_records LIMIT 1`).Scan(&attempts, &nextAttempt); err != nil && !errors.Is(err, sql.ErrNoRows) { + t.Fatalf("retry columns missing on push_records: %v", err) + } + + if err := store.ensureColumn(context.Background(), "basemap_versions", "health", "TEXT NOT NULL DEFAULT ''"); err != nil { + t.Fatalf("ensureColumn must be idempotent: %v", err) + } +} + +func TestTokenScopesNormalization(t *testing.T) { + if got := normalizeTokenScopes(nil); got != "read" { + t.Fatalf("expected read by default, got %q", got) + } + if got := normalizeTokenScopes([]string{"slice"}); got != "read,slice" { + t.Fatalf("expected read,slice, got %q", got) + } + if got := normalizeTokenScopes([]string{"SLICE", "bogus", " read "}); got != "read,slice" { + t.Fatalf("expected unknown scopes to be dropped, got %q", got) + } + + store := openTestStore(t, StoreConfig{SQLitePath: filepath.Join(t.TempDir(), "basemap.db")}) + token, err := store.CreateToken(context.Background(), CreateTokenInput{ + Name: "slicer", + Scopes: []string{"slice"}, + }) + if err != nil { + t.Fatalf("create token: %v", err) + } + auth, err := store.AuthorizeToken(context.Background(), token.Token) + if err != nil { + t.Fatalf("authorize: %v", err) + } + if !auth.CanSlice() { + t.Fatal("expected the token to carry the slice scope after a round trip") + } +} + +func TestCreateTileJobIsIdempotentByCode(t *testing.T) { + ctx := context.Background() + root := t.TempDir() + tileRoot := filepath.Join(root, "tiles") + source := filepath.Join(root, "input.tif") + writeTestFile(t, source, "fake raster") + + store := openTestStore(t, StoreConfig{ + SQLitePath: filepath.Join(root, "basemap.db"), + ScanRoot: tileRoot, + APIBaseURL: "http://127.0.0.1:8910", + TileBaseURL: "http://127.0.0.1:8910", + VectorBaseURL: "http://127.0.0.1:8910", + }) + + input := CreateTileJobInput{Code: "repeat-me", SourcePath: source, TargetCode: "demo"} + first, err := store.CreateTileJob(ctx, input) + if err != nil { + t.Fatalf("first submit: %v", err) + } + second, err := store.CreateTileJob(ctx, input) + if err != nil { + t.Fatalf("second submit must be accepted: %v", err) + } + if first.ID != second.ID { + t.Fatalf("expected the same job back, got %s then %s", first.ID, second.ID) + } + + jobs, err := store.ListTileJobs(ctx, 10) + if err != nil { + t.Fatalf("list: %v", err) + } + if len(jobs) != 1 { + t.Fatalf("expected exactly one job, got %d", len(jobs)) + } +} + +func TestGetTileJob(t *testing.T) { + ctx := context.Background() + root := t.TempDir() + source := filepath.Join(root, "input.tif") + writeTestFile(t, source, "fake raster") + + store := openTestStore(t, StoreConfig{ + SQLitePath: filepath.Join(root, "basemap.db"), + ScanRoot: filepath.Join(root, "tiles"), + APIBaseURL: "http://127.0.0.1:8910", + TileBaseURL: "http://127.0.0.1:8910", + VectorBaseURL: "http://127.0.0.1:8910", + }) + job, err := store.CreateTileJob(ctx, CreateTileJobInput{SourcePath: source, TargetCode: "demo"}) + if err != nil { + t.Fatalf("create: %v", err) + } + found, err := store.GetTileJob(ctx, job.ID) + if err != nil { + t.Fatalf("get: %v", err) + } + if found.Code != job.Code { + t.Fatalf("unexpected job %+v", found) + } + if _, err := store.GetTileJob(ctx, "nope"); err == nil { + t.Fatal("expected a missing job to error") + } +} + +func TestTileJobCallbackFiresAndRetries(t *testing.T) { + ctx := context.Background() + root := t.TempDir() + tileRoot := filepath.Join(root, "tiles") + source := filepath.Join(root, "input.tif") + writeTestFile(t, source, "fake raster") + + var attempts int + received := make(chan map[string]any, 1) + callback := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + attempts++ + if attempts < 2 { + w.WriteHeader(http.StatusInternalServerError) + return + } + var payload map[string]any + _ = json.NewDecoder(r.Body).Decode(&payload) + select { + case received <- payload: + default: + } + w.WriteHeader(http.StatusOK) + })) + defer callback.Close() + + store := openTestStore(t, StoreConfig{ + SQLitePath: filepath.Join(root, "basemap.db"), + ScanRoot: tileRoot, + APIBaseURL: "http://127.0.0.1:8910", + TileBaseURL: "http://127.0.0.1:8910", + VectorBaseURL: "http://127.0.0.1:8910", + TileJobCommand: "definitely-not-a-real-binary {input} {output}", + }) + if err := os.MkdirAll(tileRoot, 0o755); err != nil { + t.Fatalf("mkdir: %v", err) + } + + job, err := store.CreateTileJob(ctx, CreateTileJobInput{ + SourcePath: source, + TargetCode: "demo", + CallbackURL: callback.URL, + }) + if err != nil { + t.Fatalf("create: %v", err) + } + store.runTileJob(ctx, job) + + // The first delivery fails, so the callback must be parked with a backoff. + if err := store.dispatchTileJobCallbacks(ctx); err != nil { + t.Fatalf("dispatch: %v", err) + } + jobs, err := store.ListTileJobs(ctx, 5) + if err != nil { + t.Fatalf("list: %v", err) + } + if jobs[0].CallbackStatus != pushStatusRetrying { + t.Fatalf("expected the callback to be retrying, got %q", jobs[0].CallbackStatus) + } + if jobs[0].CallbackNextAttemptAt == nil { + t.Fatal("expected a next attempt to be scheduled") + } + + if _, err := store.db.ExecContext(ctx, `UPDATE tile_jobs SET callback_next_attempt_at = ? WHERE id = ?`, toRFC3339(nowUTC().Add(-time.Minute)), job.ID); err != nil { + t.Fatalf("force due: %v", err) + } + if err := store.dispatchTileJobCallbacks(ctx); err != nil { + t.Fatalf("dispatch retry: %v", err) + } + + jobs, _ = store.ListTileJobs(ctx, 5) + if jobs[0].CallbackStatus != pushStatusSucceeded { + t.Fatalf("expected the callback to succeed, got %q (%s)", jobs[0].CallbackStatus, jobs[0].CallbackMessage) + } + select { + case payload := <-received: + if payload["event"] != "tile.job.finished" { + t.Fatalf("unexpected callback payload: %v", payload) + } + case <-time.After(3 * time.Second): + t.Fatal("callback payload never arrived") + } +} + +func TestTileJobCallbackSkippedWhenNoURL(t *testing.T) { + ctx := context.Background() + root := t.TempDir() + tileRoot := filepath.Join(root, "tiles") + source := filepath.Join(root, "input.tif") + writeTestFile(t, source, "fake raster") + if err := os.MkdirAll(tileRoot, 0o755); err != nil { + t.Fatalf("mkdir: %v", err) + } + + store := openTestStore(t, StoreConfig{ + SQLitePath: filepath.Join(root, "basemap.db"), + ScanRoot: tileRoot, + APIBaseURL: "http://127.0.0.1:8910", + TileBaseURL: "http://127.0.0.1:8910", + VectorBaseURL: "http://127.0.0.1:8910", + TileJobCommand: "definitely-not-a-real-binary {input} {output}", + }) + job, err := store.CreateTileJob(ctx, CreateTileJobInput{SourcePath: source, TargetCode: "demo"}) + if err != nil { + t.Fatalf("create: %v", err) + } + store.runTileJob(ctx, job) + + jobs, _ := store.ListTileJobs(ctx, 5) + if jobs[0].CallbackStatus != "" { + t.Fatalf("expected no callback scheduling, got %q", jobs[0].CallbackStatus) + } +} + +func TestTileJobTemplateExpandsInputList(t *testing.T) { + job := TileJob{ + SourcePaths: []string{`C:\a\in1.tif`, `C:\b\in2.tif`}, + TargetCode: "demo", + OutputRoot: `C:\out`, + } + got := tileJobVRTCommandArgs(`gdalbuildvrt -resolution highest {output_vrt} {inputs}`, job, `C:\work\m.vrt`) + want := []string{"gdalbuildvrt", "-resolution", "highest", `C:\work\m.vrt`, `C:\a\in1.tif`, `C:\b\in2.tif`} + if len(got) != len(want) { + t.Fatalf("got %v, want %v", got, want) + } + for i := range want { + if got[i] != want[i] { + t.Fatalf("arg %d = %q, want %q", i, got[i], want[i]) + } + } + + // The slicer still gets one input path (the VRT) when mosaicking. + slicer := tileJobCommandArgs(`gdal2tiles.py --exclude {input} {output}`, job, `C:\work\m.vrt`) + if slicer[2] != `C:\work\m.vrt` { + t.Fatalf("expected the VRT to be the slicer input, got %v", slicer) + } +} + +func TestTileJobMosaicsMultipleSources(t *testing.T) { + if runtime.GOOS != "windows" { + t.Skip("the stub commands below use cmd.exe") + } + ctx := context.Background() + root := t.TempDir() + tileRoot := filepath.Join(root, "tiles") + if err := os.MkdirAll(tileRoot, 0o755); err != nil { + t.Fatalf("mkdir: %v", err) + } + first := filepath.Join(root, "flight-a.tif") + second := filepath.Join(root, "flight-b.tif") + writeTestFile(t, first, "fake raster a") + writeTestFile(t, second, "fake raster b") + + store := openTestStore(t, StoreConfig{ + SQLitePath: filepath.Join(root, "basemap.db"), + ScanRoot: tileRoot, + APIBaseURL: "http://127.0.0.1:8910", + TileBaseURL: "http://127.0.0.1:8910", + VectorBaseURL: "http://127.0.0.1:8910", + // Stand-ins: the mosaic step writes the VRT, the slicer lays down a z tree. + TileJobVRTCommand: `cmd /c echo vrt > {output_vrt}`, + TileJobCommand: `cmd /c mkdir "{output}\0\0"`, + }) + + job, err := store.CreateTileJob(ctx, CreateTileJobInput{ + SourcePaths: []string{first, second}, + TargetCode: "uav-2026", + ZoomMin: 0, + ZoomMax: 1, + }) + if err != nil { + t.Fatalf("create job: %v", err) + } + if job.SourceCount != 2 || len(job.SourcePaths) != 2 { + t.Fatalf("expected two sources on the job, got %d / %v", job.SourceCount, job.SourcePaths) + } + + store.runTileJob(ctx, job) + + jobs, err := store.ListTileJobs(ctx, 5) + if err != nil { + t.Fatalf("list: %v", err) + } + if jobs[0].Status != tileJobSucceeded { + t.Fatalf("expected success, got %q (%s)", jobs[0].Status, jobs[0].Message) + } + if jobs[0].SourceCount != 2 { + t.Fatalf("expected source_count 2 after a round trip, got %d", jobs[0].SourceCount) + } + + // The mosaic step ran: its VRT sits beside the job log. + vrt := filepath.Join(filepath.Dir(jobs[0].LogPath), job.Code+".vrt") + if _, err := os.Stat(vrt); err != nil { + t.Fatalf("expected the mosaic VRT at %s: %v", vrt, err) + } + + // Both flights are recorded against the layer. + sources, err := store.ListLayerSources(ctx, "uav-2026") + if err != nil { + t.Fatalf("list layer sources: %v", err) + } + if len(sources) != 2 { + t.Fatalf("expected 2 recorded sources, got %d", len(sources)) + } + + // Adding the same image twice does not duplicate the registry entry. + if err := store.AddLayerSources(ctx, "uav-2026", []string{first}); err != nil { + t.Fatalf("re-add: %v", err) + } + sources, _ = store.ListLayerSources(ctx, "uav-2026") + if len(sources) != 2 { + t.Fatalf("expected re-adding to be a no-op, got %d entries", len(sources)) + } +} + +func TestTileJobSingleSourceSkipsMosaic(t *testing.T) { + if runtime.GOOS != "windows" { + t.Skip("the stub command below uses cmd.exe") + } + ctx := context.Background() + root := t.TempDir() + tileRoot := filepath.Join(root, "tiles") + if err := os.MkdirAll(tileRoot, 0o755); err != nil { + t.Fatalf("mkdir: %v", err) + } + source := filepath.Join(root, "flight-a.tif") + writeTestFile(t, source, "fake raster") + + store := openTestStore(t, StoreConfig{ + SQLitePath: filepath.Join(root, "basemap.db"), + ScanRoot: tileRoot, + APIBaseURL: "http://127.0.0.1:8910", + TileBaseURL: "http://127.0.0.1:8910", + VectorBaseURL: "http://127.0.0.1:8910", + // A failing mosaic command proves the single-source path never calls it. + TileJobVRTCommand: `definitely-not-a-real-binary {output_vrt} {inputs}`, + TileJobCommand: `cmd /c mkdir "{output}\0\0"`, + }) + + job, err := store.CreateTileJob(ctx, CreateTileJobInput{SourcePath: source, TargetCode: "uav-2026"}) + if err != nil { + t.Fatalf("create job: %v", err) + } + if job.SourceCount != 1 { + t.Fatalf("expected source_count 1, got %d", job.SourceCount) + } + store.runTileJob(ctx, job) + + jobs, _ := store.ListTileJobs(ctx, 5) + if jobs[0].Status != tileJobSucceeded { + t.Fatalf("expected the single source to slice directly, got %q (%s)", jobs[0].Status, jobs[0].Message) + } +} diff --git a/services/api-go/internal/basemap/health.go b/services/api-go/internal/basemap/health.go new file mode 100644 index 0000000..c276b57 --- /dev/null +++ b/services/api-go/internal/basemap/health.go @@ -0,0 +1,155 @@ +package basemap + +import ( + "context" + "errors" + "os" + "time" +) + +// Availability verdicts recorded on catalog rows. +const ( + healthOK = "ok" + healthMissing = "missing" + healthUnknown = "unknown" +) + +// HealthReport summarizes one inspection pass. +type HealthReport struct { + CheckedAt time.Time `json:"checked_at"` + VersionsChecked int `json:"versions_checked"` + VersionsMissing int `json:"versions_missing"` + VectorsChecked int `json:"vectors_checked"` + VectorsMissing int `json:"vectors_missing"` +} + +// CheckHealth stats every catalogued path and records whether it is still +// there, so a directory that was moved or deleted surfaces before a client +// hits a 404. It shares scanMu with the scanners because both rewrite catalog +// rows. +func (s *Store) CheckHealth(ctx context.Context) (HealthReport, error) { + s.scanMu.Lock() + defer s.scanMu.Unlock() + + checkedAt := nowUTC() + report := HealthReport{CheckedAt: checkedAt} + + type verdict struct { + id string + health string + detail string + } + + versions, err := s.collectHealth(ctx, `SELECT id, tile_root_path FROM basemap_versions`, true) + if err != nil { + return report, err + } + vectors, err := s.collectHealth(ctx, `SELECT id, file_path FROM vector_assets`, false) + if err != nil { + return report, err + } + + tx, err := s.db.BeginTx(ctx, nil) + if err != nil { + return report, err + } + defer tx.Rollback() + + stamp := toRFC3339(checkedAt) + for _, item := range versions { + if _, err := tx.ExecContext(ctx, ` + UPDATE basemap_versions SET health = ?, health_checked_at = ?, health_detail = ? WHERE id = ? + `, item.health, stamp, item.detail, item.id); err != nil { + return report, err + } + report.VersionsChecked++ + if item.health != healthOK { + report.VersionsMissing++ + } + } + for _, item := range vectors { + if _, err := tx.ExecContext(ctx, ` + UPDATE vector_assets SET health = ?, health_checked_at = ?, health_detail = ? WHERE id = ? + `, item.health, stamp, item.detail, item.id); err != nil { + return report, err + } + report.VectorsChecked++ + if item.health != healthOK { + report.VectorsMissing++ + } + } + if err := tx.Commit(); err != nil { + return report, err + } + return report, nil +} + +type healthVerdict struct { + id string + health string + detail string +} + +func (s *Store) collectHealth(ctx context.Context, query string, wantDir bool) ([]healthVerdict, error) { + rows, err := s.db.QueryContext(ctx, query) + if err != nil { + return nil, err + } + defer rows.Close() + + var verdicts []healthVerdict + for rows.Next() { + var id, path string + if err := rows.Scan(&id, &path); err != nil { + return nil, err + } + health, detail := inspectPath(path, wantDir) + verdicts = append(verdicts, healthVerdict{id: id, health: health, detail: detail}) + } + return verdicts, rows.Err() +} + +func inspectPath(path string, wantDir bool) (string, string) { + info, err := os.Stat(path) + if err != nil { + if errors.Is(err, os.ErrNotExist) { + return healthMissing, "路径不存在" + } + return healthUnknown, err.Error() + } + if wantDir && !info.IsDir() { + return healthMissing, "路径不是目录" + } + if !wantDir && info.IsDir() { + return healthMissing, "路径是目录" + } + return healthOK, "" +} + +// The console reads health through these maps so the row scanners on the tile +// request path stay untouched. +func (s *Store) loadVersionHealth(ctx context.Context) (map[string]ResourceHealth, error) { + return s.loadHealth(ctx, `SELECT id, health, health_detail, health_checked_at FROM basemap_versions`) +} + +func (s *Store) loadVectorHealth(ctx context.Context) (map[string]ResourceHealth, error) { + return s.loadHealth(ctx, `SELECT id, health, health_detail, health_checked_at FROM vector_assets`) +} + +func (s *Store) loadHealth(ctx context.Context, query string) (map[string]ResourceHealth, error) { + rows, err := s.db.QueryContext(ctx, query) + if err != nil { + return nil, err + } + defer rows.Close() + + result := map[string]ResourceHealth{} + for rows.Next() { + var id, health, detail, checkedAt string + if err := rows.Scan(&id, &health, &detail, &checkedAt); err != nil { + return nil, err + } + result[id] = ResourceHealth{Health: health, Detail: detail, CheckedAt: parseTime(checkedAt)} + } + return result, rows.Err() +} diff --git a/services/api-go/internal/basemap/models.go b/services/api-go/internal/basemap/models.go index c094549..6d15b73 100644 --- a/services/api-go/internal/basemap/models.go +++ b/services/api-go/internal/basemap/models.go @@ -13,20 +13,21 @@ type SystemInfo struct { SQLitePath string `json:"sqlite_path"` ScanRoot string `json:"scan_root"` VectorScanRoot string `json:"vector_scan_root"` + TileJobCommand string `json:"tile_job_command,omitempty"` StartedAt time.Time `json:"started_at"` } type Basemap struct { - ID string `json:"id"` - Code string `json:"code"` - Name string `json:"name"` - Type string `json:"type"` - Status string `json:"status"` - Description string `json:"description,omitempty"` - Default *BasemapVersion `json:"default_version,omitempty"` - Versions []BasemapVersion `json:"versions,omitempty"` - CreatedAt time.Time `json:"created_at"` - UpdatedAt time.Time `json:"updated_at"` + ID string `json:"id"` + Code string `json:"code"` + Name string `json:"name"` + Type string `json:"type"` + Status string `json:"status"` + Description string `json:"description,omitempty"` + Default *BasemapVersion `json:"default_version,omitempty"` + Versions []BasemapVersion `json:"versions,omitempty"` + CreatedAt time.Time `json:"created_at"` + UpdatedAt time.Time `json:"updated_at"` } type BasemapVersion struct { @@ -89,26 +90,27 @@ type TargetSystem struct { } type ServiceToken struct { - ID string `json:"id"` - Name string `json:"name"` - TokenPrefix string `json:"token_prefix"` - Status string `json:"status"` - ExpiresAt *time.Time `json:"expires_at,omitempty"` - LastUsedAt *time.Time `json:"last_used_at,omitempty"` - CreatedAt time.Time `json:"created_at"` - Grants []TokenGrant `json:"grants,omitempty"` + ID string `json:"id"` + Name string `json:"name"` + TokenPrefix string `json:"token_prefix"` + Scopes []string `json:"scopes"` + Status string `json:"status"` + ExpiresAt *time.Time `json:"expires_at,omitempty"` + LastUsedAt *time.Time `json:"last_used_at,omitempty"` + CreatedAt time.Time `json:"created_at"` + Grants []TokenGrant `json:"grants,omitempty"` VectorGrants []VectorTokenGrant `json:"vector_grants,omitempty"` } type TokenGrant struct { - ID string `json:"id"` - TokenID string `json:"token_id"` - BasemapID string `json:"basemap_id"` - BasemapCode string `json:"basemap_code"` - BasemapVersionID *string `json:"basemap_version_id,omitempty"` - BasemapVersion *string `json:"basemap_version,omitempty"` - Permission string `json:"permission"` - CreatedAt time.Time `json:"created_at"` + ID string `json:"id"` + TokenID string `json:"token_id"` + BasemapID string `json:"basemap_id"` + BasemapCode string `json:"basemap_code"` + BasemapVersionID *string `json:"basemap_version_id,omitempty"` + BasemapVersion *string `json:"basemap_version,omitempty"` + Permission string `json:"permission"` + CreatedAt time.Time `json:"created_at"` } type PushRecord struct { @@ -123,6 +125,8 @@ type PushRecord struct { ResponseStatus *int `json:"response_status,omitempty"` ResponseBody string `json:"response_body,omitempty"` ErrorMessage string `json:"error_message,omitempty"` + AttemptCount int `json:"attempt_count"` + NextAttemptAt *time.Time `json:"next_attempt_at,omitempty"` PushedAt time.Time `json:"pushed_at"` FinishedAt *time.Time `json:"finished_at,omitempty"` } @@ -164,6 +168,8 @@ type VectorPushRecord struct { ResponseStatus *int `json:"response_status,omitempty"` ResponseBody string `json:"response_body,omitempty"` ErrorMessage string `json:"error_message,omitempty"` + AttemptCount int `json:"attempt_count"` + NextAttemptAt *time.Time `json:"next_attempt_at,omitempty"` PushedAt time.Time `json:"pushed_at"` FinishedAt *time.Time `json:"finished_at,omitempty"` } @@ -205,14 +211,17 @@ type CreateTokenInput struct { Name string BasemapCodes []string VectorCodes []string + Scopes []string ExpiresAt *time.Time } type TokenAuth struct { Token ServiceToken // permission -> basemap_code -> version scope - Permissions map[string]map[string]VersionScope + Permissions map[string]map[string]VersionScope VectorPermissions map[string]map[string]struct{} + // Scopes are coarse capabilities beyond resource grants, e.g. "slice". + Scopes map[string]struct{} } type VersionScope struct { @@ -220,16 +229,41 @@ type VersionScope struct { Versions map[string]struct{} } +// ResourceHealth is the availability verdict recorded by the health inspector. +// It is kept separate from Status: Status is a catalog fact, Health is whether +// the bytes are still on disk right now. +type ResourceHealth struct { + Health string `json:"health"` + Detail string `json:"detail,omitempty"` + CheckedAt time.Time `json:"checked_at"` +} + +// LayerInventory is one layer and the imagery it is built from, ordered for +// deterministic rendering in the console. +type LayerInventory struct { + TargetCode string `json:"target_code"` + Sources []LayerSource `json:"sources"` + // ZoomMin/ZoomMax/TileFormat mirror the layer's current basemap version so + // a rebuild can reuse the range it is already published at. + ZoomMin int `json:"zoom_min"` + ZoomMax int `json:"zoom_max"` + TileFormat string `json:"tile_format"` +} + type DashboardData struct { - SystemInfo SystemInfo - Basemaps []Basemap - VectorAssets []VectorAsset - ScanSources []ScanSource - ScanRuns []ScanRun - TargetSystems []TargetSystem - Tokens []ServiceToken - PushRecords []PushRecord + SystemInfo SystemInfo + Basemaps []Basemap + VectorAssets []VectorAsset + ScanSources []ScanSource + ScanRuns []ScanRun + TargetSystems []TargetSystem + Tokens []ServiceToken + PushRecords []PushRecord VectorPushRecords []VectorPushRecord - Flash string - CreatedToken string + Flash string + CreatedToken string + VersionHealth map[string]ResourceHealth + VectorHealth map[string]ResourceHealth + TileJobs []TileJob + Layers []LayerInventory } diff --git a/services/api-go/internal/basemap/optimize_test.go b/services/api-go/internal/basemap/optimize_test.go new file mode 100644 index 0000000..15ac5b1 --- /dev/null +++ b/services/api-go/internal/basemap/optimize_test.go @@ -0,0 +1,185 @@ +package basemap + +import ( + "context" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +func writeTestFile(t *testing.T, path, body string) { + t.Helper() + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatalf("mkdir %s: %v", filepath.Dir(path), err) + } + if err := os.WriteFile(path, []byte(body), 0o644); err != nil { + t.Fatalf("write %s: %v", path, err) + } +} + +func manifestBody(code, name string) string { + return `{"code":"` + code + `","name":"` + name + `","type":"xyz","version":"current","tile_format":"png","root_path":"tiles","is_default":true}` +} + +// The manifest lookup must not descend into the tile payload: a manifest nested +// inside another basemap's "tiles" subtree must be ignored. +func TestFindManifestVersionsPrunesTilePayload(t *testing.T) { + root := t.TempDir() + writeTestFile(t, filepath.Join(root, "demo", "manifest.json"), manifestBody("demo", "Demo")) + writeTestFile(t, filepath.Join(root, "demo", "tiles", "0", "0", "0.png"), "tile") + writeTestFile(t, filepath.Join(root, "demo", "tiles", "nested", "manifest.json"), manifestBody("nested", "Nested")) + + store := &Store{} + items, covered, err := store.findManifestVersions(ScanSource{RootPath: root, ManifestName: "manifest.json"}) + if err != nil { + t.Fatalf("find manifests: %v", err) + } + if len(items) != 1 { + t.Fatalf("expected 1 manifest, got %d: %+v", len(items), items) + } + if items[0].BasemapCode != "demo" { + t.Fatalf("expected code demo, got %q", items[0].BasemapCode) + } + if _, ok := covered["demo"]; !ok { + t.Fatalf("expected demo to be marked as covered, got %v", covered) + } +} + +func TestIsNumericDirName(t *testing.T) { + for _, name := range []string{"0", "14", "007"} { + if !isNumericDirName(name) { + t.Fatalf("expected %q to be numeric", name) + } + } + for _, name := range []string{"", "gaode-image", "2026-04", "tiles", "1a"} { + if isNumericDirName(name) { + t.Fatalf("expected %q to be non-numeric", name) + } + } +} + +func TestSQLiteDSNEnablesWALAndForeignKeys(t *testing.T) { + root := t.TempDir() + store, err := OpenStore(StoreConfig{SQLitePath: filepath.Join(root, "basemap.db")}) + if err != nil { + t.Fatalf("open store: %v", err) + } + defer store.Close() + + var mode string + if err := store.db.QueryRow("PRAGMA journal_mode;").Scan(&mode); err != nil { + t.Fatalf("read journal_mode: %v", err) + } + if !strings.EqualFold(mode, "wal") { + t.Fatalf("expected WAL journal mode, got %q", mode) + } + + var foreignKeys int + if err := store.db.QueryRow("PRAGMA foreign_keys;").Scan(&foreignKeys); err != nil { + t.Fatalf("read foreign_keys: %v", err) + } + if foreignKeys != 1 { + t.Fatalf("expected foreign_keys=1, got %d", foreignKeys) + } +} + +func TestAuthorizeTokenThrottlesLastUsedWrite(t *testing.T) { + ctx := context.Background() + root := t.TempDir() + tileRoot := filepath.Join(root, "tiles") + writeTestFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile") + + store, err := OpenStore(StoreConfig{ + SQLitePath: filepath.Join(root, "basemap.db"), + ScanRoot: tileRoot, + APIBaseURL: "http://127.0.0.1:8910", + TileBaseURL: "http://127.0.0.1:8910", + }) + if err != nil { + t.Fatalf("open store: %v", err) + } + defer store.Close() + + if _, err := store.RunScan(ctx, defaultScanSourceCode); err != nil { + t.Fatalf("run scan: %v", err) + } + created, err := store.CreateToken(ctx, CreateTokenInput{Name: "throttle", BasemapCodes: []string{"google_image"}}) + if err != nil { + t.Fatalf("create token: %v", err) + } + + // Mark the token as unused long ago so the next authorization refreshes it. + stale := toRFC3339(nowUTC().Add(-10 * time.Minute)) + if _, err := store.db.ExecContext(ctx, `UPDATE service_tokens SET last_used_at = ? WHERE id = ?`, stale, created.Meta.ID); err != nil { + t.Fatalf("seed stale last_used_at: %v", err) + } + if _, err := store.AuthorizeToken(ctx, created.Token); err != nil { + t.Fatalf("authorize token: %v", err) + } + first := readLastUsedAt(t, store, created.Meta.ID) + if first == stale { + t.Fatalf("expected last_used_at to be refreshed, still %q", first) + } + + if _, err := store.AuthorizeToken(ctx, created.Token); err != nil { + t.Fatalf("authorize token again: %v", err) + } + second := readLastUsedAt(t, store, created.Meta.ID) + if second != first { + t.Fatalf("expected last_used_at to be throttled, got %q then %q", first, second) + } +} + +func readLastUsedAt(t *testing.T, store *Store, tokenID string) string { + t.Helper() + var value string + if err := store.db.QueryRow(`SELECT last_used_at FROM service_tokens WHERE id = ?`, tokenID).Scan(&value); err != nil { + t.Fatalf("read last_used_at: %v", err) + } + return value +} + +func TestGetBasemapByCodeLoadsVersions(t *testing.T) { + ctx := context.Background() + root := t.TempDir() + tileRoot := filepath.Join(root, "tiles") + writeTestFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile") + writeTestFile(t, filepath.Join(tileRoot, "google_image", "1", "0", "0.webp"), "tile") + + store, err := OpenStore(StoreConfig{ + SQLitePath: filepath.Join(root, "basemap.db"), + ScanRoot: tileRoot, + APIBaseURL: "http://127.0.0.1:8910", + TileBaseURL: "http://127.0.0.1:8910", + }) + if err != nil { + t.Fatalf("open store: %v", err) + } + defer store.Close() + + if _, err := store.RunScan(ctx, defaultScanSourceCode); err != nil { + t.Fatalf("run scan: %v", err) + } + + item, err := store.GetBasemapByCode(ctx, "google_image") + if err != nil { + t.Fatalf("get basemap: %v", err) + } + if item.Code != "google-image" { + t.Fatalf("unexpected code: %q", item.Code) + } + if len(item.Versions) != 1 { + t.Fatalf("expected 1 version, got %d", len(item.Versions)) + } + if item.Default == nil || item.Default.Version != "current" { + t.Fatalf("expected default version current, got %+v", item.Default) + } + if item.Default.MinZoom != 0 || item.Default.MaxZoom != 1 { + t.Fatalf("unexpected zoom range: %d-%d", item.Default.MinZoom, item.Default.MaxZoom) + } + if _, err := store.GetBasemapByCode(ctx, "missing"); err == nil { + t.Fatal("expected missing basemap to error") + } +} diff --git a/services/api-go/internal/basemap/push.go b/services/api-go/internal/basemap/push.go index 00ae71d..0b53c83 100644 --- a/services/api-go/internal/basemap/push.go +++ b/services/api-go/internal/basemap/push.go @@ -7,10 +7,28 @@ import ( "encoding/json" "errors" "fmt" + "io" + "log" "net/http" "strings" + "time" - "map-asset-gateway/api-go/internal/uid" + "tile-server/api-go/internal/uid" +) + +const pushCallbackTimeout = 30 * time.Second + +// Delivery outcomes. retrying is a transient state: the record stays queued +// until next_attempt_at elapses. +const ( + pushStatusSucceeded = "succeeded" + pushStatusFailed = "failed" + pushStatusRetrying = "retrying" + + // pushMaxAttempts caps how many times one record is delivered. + pushMaxAttempts = 3 + // pushRetryBase is the first backoff step; it doubles per attempt. + pushRetryBase = 30 * time.Second ) func (s *Store) CreateTargetSystem(ctx context.Context, input CreateTargetSystemInput) (TargetSystem, error) { @@ -92,6 +110,9 @@ func (s *Store) ListTargetSystems(ctx context.Context) ([]TargetSystem, error) { return items, rows.Err() } +// PushBasemapVersion queues a downstream notification. The HTTP callback runs +// off the request path so a slow target cannot block the caller; the returned +// record starts as "pending" and is updated once the callback finishes. func (s *Store) PushBasemapVersion(ctx context.Context, targetCode, basemapCode, version string) (PushRecord, error) { target, err := s.GetTargetSystemByCode(ctx, targetCode) if err != nil { @@ -148,20 +169,6 @@ func (s *Store) PushBasemapVersion(ctx context.Context, targetCode, basemapCode, "pushed_at": toRFC3339(nowUTC()), "targetCode": target.Code, } - body, err := json.Marshal(payload) - if err != nil { - return PushRecord{}, err - } - request, err := http.NewRequestWithContext(ctx, target.CallbackMethod, target.CallbackURL, bytes.NewReader(body)) - if err != nil { - return PushRecord{}, err - } - request.Header.Set("Content-Type", "application/json") - for key, value := range target.CallbackHeaders { - request.Header.Set(key, value) - } - - response, err := s.httpClient.Do(request) record := PushRecord{ ID: uid.New(), TargetSystemID: target.ID, @@ -169,41 +176,19 @@ func (s *Store) PushBasemapVersion(ctx context.Context, targetCode, basemapCode, BasemapVersionID: versionItem.ID, BasemapCode: basemapItem.Code, BasemapVersion: versionItem.Version, + Status: "pending", Request: payload, PushedAt: nowUTC(), } - if err != nil { - record.Status = "failed" - record.ErrorMessage = err.Error() - } else { - defer response.Body.Close() - record.ResponseBody = limitedReadAll(response.Body, 256*1024) - status := response.StatusCode - record.ResponseStatus = &status - if status >= 200 && status < 300 { - record.Status = "succeeded" - } else { - record.Status = "failed" - record.ErrorMessage = fmt.Sprintf("target returned status %d", status) - } - } - finishedAt := nowUTC() - record.FinishedAt = &finishedAt - - _, insertErr := s.db.ExecContext(ctx, ` + if _, err := s.db.ExecContext(ctx, ` INSERT INTO push_records ( id, target_system_id, basemap_version_id, status, request_json, response_status, response_body, error_message, pushed_at, finished_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - `, record.ID, record.TargetSystemID, record.BasemapVersionID, record.Status, writeJSON(record.Request), record.ResponseStatus, record.ResponseBody, record.ErrorMessage, toRFC3339(record.PushedAt), nullableTime(record.FinishedAt)) - if insertErr != nil { - return PushRecord{}, insertErr - } - if err != nil { - return record, err - } - if record.Status != "succeeded" { - return record, errors.New(record.ErrorMessage) + ) VALUES (?, ?, ?, ?, ?, NULL, '', '', ?, NULL) + `, record.ID, record.TargetSystemID, record.BasemapVersionID, record.Status, writeJSON(record.Request), toRFC3339(record.PushedAt)); err != nil { + return PushRecord{}, err } + + s.enqueuePush("push_records", record.ID, target, payload) return record, nil } @@ -223,6 +208,8 @@ func (s *Store) ListVectorPushRecords(ctx context.Context, limit int) ([]VectorP p.response_status, p.response_body, p.error_message, + p.attempt_count, + p.next_attempt_at, p.pushed_at, p.finished_at FROM vector_push_records p @@ -247,6 +234,8 @@ func (s *Store) ListVectorPushRecords(ctx context.Context, limit int) ([]VectorP return items, rows.Err() } +// PushVectorAsset queues a downstream notification, mirroring +// PushBasemapVersion. func (s *Store) PushVectorAsset(ctx context.Context, targetCode, vectorCode string) (VectorPushRecord, error) { target, err := s.GetTargetSystemByCode(ctx, targetCode) if err != nil { @@ -268,60 +257,240 @@ func (s *Store) PushVectorAsset(ctx context.Context, targetCode, vectorCode stri "pushed_at": toRFC3339(nowUTC()), "targetCode": target.Code, } - body, err := json.Marshal(payload) - if err != nil { - return VectorPushRecord{}, err - } - request, err := http.NewRequestWithContext(ctx, target.CallbackMethod, target.CallbackURL, bytes.NewReader(body)) - if err != nil { - return VectorPushRecord{}, err - } - request.Header.Set("Content-Type", "application/json") - for key, value := range target.CallbackHeaders { - request.Header.Set(key, value) - } - - response, err := s.httpClient.Do(request) record := VectorPushRecord{ ID: uid.New(), TargetSystemID: target.ID, TargetSystemCode: target.Code, VectorAssetID: asset.ID, VectorCode: asset.Code, + Status: "pending", Request: payload, PushedAt: nowUTC(), } - if err != nil { - record.Status = "failed" - record.ErrorMessage = err.Error() - } else { - defer response.Body.Close() - record.ResponseBody = limitedReadAll(response.Body, 256*1024) - status := response.StatusCode - record.ResponseStatus = &status - if status >= 200 && status < 300 { - record.Status = "succeeded" - } else { - record.Status = "failed" - record.ErrorMessage = fmt.Sprintf("target returned status %d", status) - } - } - finishedAt := nowUTC() - record.FinishedAt = &finishedAt - - _, insertErr := s.db.ExecContext(ctx, ` + if _, err := s.db.ExecContext(ctx, ` INSERT INTO vector_push_records ( id, target_system_id, vector_asset_id, status, request_json, response_status, response_body, error_message, pushed_at, finished_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - `, record.ID, record.TargetSystemID, record.VectorAssetID, record.Status, writeJSON(record.Request), record.ResponseStatus, record.ResponseBody, record.ErrorMessage, toRFC3339(record.PushedAt), nullableTime(record.FinishedAt)) - if insertErr != nil { - return VectorPushRecord{}, insertErr - } - if err != nil { - return record, err - } - if record.Status != "succeeded" { - return record, errors.New(record.ErrorMessage) + ) VALUES (?, ?, ?, ?, ?, NULL, '', '', ?, NULL) + `, record.ID, record.TargetSystemID, record.VectorAssetID, record.Status, writeJSON(record.Request), toRFC3339(record.PushedAt)); err != nil { + return VectorPushRecord{}, err } + + s.enqueuePush("vector_push_records", record.ID, target, payload) return record, nil } + +// enqueuePush performs the first delivery attempt off the request path. A +// failure inside the retry budget is rescheduled by dispatchPushRetries. +func (s *Store) enqueuePush(table, pushID string, target TargetSystem, payload map[string]any) { + s.pushWG.Add(1) + go func() { + defer s.pushWG.Done() + s.attemptPush(table, pushID, 1, target, payload) + }() +} + +// attemptPush performs one delivery attempt and records the outcome. attempt is +// the 1-based attempt number; table is a trusted internal literal. +func (s *Store) attemptPush(table, pushID string, attempt int, target TargetSystem, payload map[string]any) { + ctx, cancel := context.WithTimeout(context.Background(), pushCallbackTimeout) + defer cancel() + + status, body, err := s.deliverCallback(ctx, target, payload) + errorMessage := "" + if err != nil { + errorMessage = err.Error() + } + var responseStatus any + if status != 0 { + responseStatus = status + } + + if err != nil && attempt < pushMaxAttempts { + // Stay queued: the dispatcher picks it up once the backoff elapses. + next := nowUTC().Add(pushRetryDelay(attempt)) + if _, updateErr := s.db.ExecContext(context.Background(), fmt.Sprintf(` + UPDATE %s + SET status = ?, attempt_count = ?, next_attempt_at = ?, response_status = ?, response_body = ?, + error_message = ?, finished_at = NULL + WHERE id = ? + `, table), pushStatusRetrying, attempt, toRFC3339(next), responseStatus, body, errorMessage, pushID); updateErr != nil { + log.Printf("reschedule push %s/%s: %v", table, pushID, updateErr) + } + return + } + + recordStatus := pushStatusSucceeded + if err != nil { + recordStatus = pushStatusFailed + } + if _, updateErr := s.db.ExecContext(context.Background(), fmt.Sprintf(` + UPDATE %s + SET status = ?, attempt_count = ?, next_attempt_at = NULL, response_status = ?, response_body = ?, + error_message = ?, finished_at = ? + WHERE id = ? + `, table), recordStatus, attempt, responseStatus, body, errorMessage, toRFC3339(nowUTC()), pushID); updateErr != nil { + log.Printf("record push %s/%s: %v", table, pushID, updateErr) + } +} + +// pushRetryDelay backs off exponentially: 30s, 60s, ... +func pushRetryDelay(attempt int) time.Duration { + if attempt < 1 { + attempt = 1 + } + return pushRetryBase * time.Duration(1<<(attempt-1)) +} + +// dispatchPushRetries finds attempts whose backoff has elapsed and runs them. +func (s *Store) dispatchPushRetries(ctx context.Context) error { + for _, table := range []string{"push_records", "vector_push_records"} { + due, err := s.dueRetries(ctx, table, 5) + if err != nil { + return err + } + for _, item := range due { + target, err := s.GetTargetSystemByID(ctx, item.targetID) + if err != nil { + s.failRetry(table, item.id, fmt.Sprintf("target system is gone: %v", err)) + continue + } + if !target.Enabled { + s.failRetry(table, item.id, "target system is disabled") + continue + } + s.attemptPush(table, item.id, item.attempt+1, target, item.payload) + } + } + return nil +} + +type duePush struct { + id string + attempt int + targetID string + payload map[string]any +} + +func (s *Store) dueRetries(ctx context.Context, table string, limit int) ([]duePush, error) { + now := toRFC3339(nowUTC()) + rows, err := s.db.QueryContext(ctx, fmt.Sprintf(` + SELECT id, attempt_count, target_system_id, request_json + FROM %s + WHERE status = ? AND (next_attempt_at IS NULL OR next_attempt_at <= ?) + ORDER BY next_attempt_at + LIMIT ? + `, table), pushStatusRetrying, now, limit) + if err != nil { + return nil, err + } + defer rows.Close() + + var items []duePush + for rows.Next() { + var item duePush + var requestJSON string + if err := rows.Scan(&item.id, &item.attempt, &item.targetID, &requestJSON); err != nil { + return nil, err + } + if strings.TrimSpace(requestJSON) != "" { + if err := readMaybeJSON(requestJSON, &item.payload); err != nil { + item.payload = nil + } + } + items = append(items, item) + } + return items, rows.Err() +} + +func (s *Store) failRetry(table, pushID, message string) { + if _, err := s.db.ExecContext(context.Background(), fmt.Sprintf(` + UPDATE %s SET status = ?, error_message = ?, next_attempt_at = NULL, finished_at = ? WHERE id = ? + `, table), pushStatusFailed, message, toRFC3339(nowUTC()), pushID); err != nil { + log.Printf("fail push %s/%s: %v", table, pushID, err) + } +} + +// GetTargetSystemByID looks a target up by primary key for retry dispatch. +func (s *Store) GetTargetSystemByID(ctx context.Context, id string) (TargetSystem, error) { + row := s.db.QueryRowContext(ctx, ` + SELECT id, code, name, callback_url, callback_method, callback_headers_json, enabled, created_at, updated_at + FROM target_systems + WHERE id = ? + `, strings.TrimSpace(id)) + item, err := scanTargetSystem(row) + if err != nil { + if errors.Is(err, sql.ErrNoRows) { + return TargetSystem{}, fmt.Errorf("target system %q not found", id) + } + return TargetSystem{}, err + } + return item, nil +} + +// RetryPush re-queues a finished push record. kind is "basemap" or "vector". +func (s *Store) RetryPush(ctx context.Context, kind, recordID string) error { + table := "push_records" + if kind == "vector" { + table = "vector_push_records" + } else if kind != "basemap" { + return fmt.Errorf("unknown push kind %q", kind) + } + result, err := s.db.ExecContext(ctx, fmt.Sprintf(` + UPDATE %s + SET status = ?, attempt_count = 0, next_attempt_at = ?, finished_at = NULL, error_message = '' + WHERE id = ? + `, table), pushStatusRetrying, toRFC3339(nowUTC()), strings.TrimSpace(recordID)) + if err != nil { + return err + } + affected, err := result.RowsAffected() + if err != nil { + return err + } + if affected == 0 { + return fmt.Errorf("push record %q not found", recordID) + } + // Kick the dispatcher so a manual retry does not wait for the next tick. + s.pushWG.Add(1) + go func() { + defer s.pushWG.Done() + _ = s.dispatchPushRetries(context.Background()) + }() + return nil +} + +// deliverCallback performs the downstream HTTP call for a target system. +func (s *Store) deliverCallback(ctx context.Context, target TargetSystem, payload map[string]any) (int, string, error) { + return s.postJSON(ctx, target.CallbackMethod, target.CallbackURL, target.CallbackHeaders, payload) +} + +// postJSON sends a JSON payload and returns the status code plus a truncated +// response body. Shared by push callbacks and tile-job completion callbacks. +func (s *Store) postJSON(ctx context.Context, method, url string, headers map[string]string, payload map[string]any) (int, string, error) { + body, err := json.Marshal(payload) + if err != nil { + return 0, "", err + } + request, err := http.NewRequestWithContext(ctx, method, url, bytes.NewReader(body)) + if err != nil { + return 0, "", err + } + request.Header.Set("Content-Type", "application/json") + for key, value := range headers { + request.Header.Set(key, value) + } + + response, err := s.httpClient.Do(request) + if err != nil { + return 0, "", err + } + defer response.Body.Close() + + responseBody := limitedReadAll(response.Body, 256*1024) + // Drain whatever is left so the connection can be reused. + _, _ = io.Copy(io.Discard, response.Body) + if response.StatusCode < 200 || response.StatusCode >= 300 { + return response.StatusCode, responseBody, fmt.Errorf("target returned status %d", response.StatusCode) + } + return response.StatusCode, responseBody, nil +} diff --git a/services/api-go/internal/basemap/regression_test.go b/services/api-go/internal/basemap/regression_test.go new file mode 100644 index 0000000..eb91c00 --- /dev/null +++ b/services/api-go/internal/basemap/regression_test.go @@ -0,0 +1,397 @@ +package basemap + +import ( + "context" + "encoding/json" + "errors" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "testing" + "time" +) + +func openTestStore(t *testing.T, cfg StoreConfig) *Store { + t.Helper() + store, err := OpenStore(cfg) + if err != nil { + t.Fatalf("open store: %v", err) + } + t.Cleanup(func() { + if err := store.Close(); err != nil { + t.Fatalf("close store: %v", err) + } + }) + return store +} + +func TestRunScanRefusesConcurrentRuns(t *testing.T) { + ctx := context.Background() + root := t.TempDir() + tileRoot := filepath.Join(root, "tiles") + writeTestFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile") + vectorRoot := filepath.Join(root, "vectors") + + store := openTestStore(t, StoreConfig{ + SQLitePath: filepath.Join(root, "basemap.db"), + ScanRoot: tileRoot, + VectorScanRoot: vectorRoot, + APIBaseURL: "http://127.0.0.1:8910", + TileBaseURL: "http://127.0.0.1:8910", + VectorBaseURL: "http://127.0.0.1:8910", + }) + + store.scanMu.Lock() + if _, err := store.RunScan(ctx, defaultScanSourceCode); !errors.Is(err, ErrScanInProgress) { + store.scanMu.Unlock() + t.Fatalf("expected ErrScanInProgress, got %v", err) + } + if _, err := store.RunVectorScan(ctx); !errors.Is(err, ErrScanInProgress) { + store.scanMu.Unlock() + t.Fatalf("expected ErrScanInProgress from vector scan, got %v", err) + } + store.scanMu.Unlock() + + if _, err := store.RunScan(ctx, defaultScanSourceCode); err != nil { + t.Fatalf("run scan after unlock: %v", err) + } +} + +func TestRunScanReportsUnknownSource(t *testing.T) { + ctx := context.Background() + root := t.TempDir() + store := openTestStore(t, StoreConfig{SQLitePath: filepath.Join(root, "basemap.db")}) + + if _, err := store.RunScan(ctx, "does-not-exist"); !errors.Is(err, ErrScanSourceNotFound) { + t.Fatalf("expected ErrScanSourceNotFound, got %v", err) + } +} + +func TestRescanOnlyCountsRealChanges(t *testing.T) { + ctx := context.Background() + root := t.TempDir() + tileRoot := filepath.Join(root, "tiles") + writeTestFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile") + + store := openTestStore(t, StoreConfig{ + SQLitePath: filepath.Join(root, "basemap.db"), + ScanRoot: tileRoot, + APIBaseURL: "http://127.0.0.1:8910", + TileBaseURL: "http://127.0.0.1:8910", + VectorBaseURL: "http://127.0.0.1:8910", + }) + + first, err := store.RunScan(ctx, defaultScanSourceCode) + if err != nil { + t.Fatalf("first scan: %v", err) + } + if first.AddedCount != 1 { + t.Fatalf("expected 1 added on first scan, got %d", first.AddedCount) + } + + second, err := store.RunScan(ctx, defaultScanSourceCode) + if err != nil { + t.Fatalf("second scan: %v", err) + } + if second.UpdatedCount != 0 { + t.Fatalf("expected 0 updates on an unchanged rescan, got %d", second.UpdatedCount) + } + if second.AddedCount != 0 || second.RemovedCount != 0 { + t.Fatalf("expected no additions/removals, got +%d/-%d", second.AddedCount, second.RemovedCount) + } + if got := second.Summary["unchanged_count"]; got != 1 { + t.Fatalf("expected unchanged_count=1, got %v", got) + } + + // A new zoom level is a genuine change and must be reported as an update. + writeTestFile(t, filepath.Join(tileRoot, "google_image", "1", "0", "0.webp"), "tile") + third, err := store.RunScan(ctx, defaultScanSourceCode) + if err != nil { + t.Fatalf("third scan: %v", err) + } + if third.UpdatedCount != 1 { + t.Fatalf("expected 1 update after adding a zoom level, got %d", third.UpdatedCount) + } +} + +func TestVectorCodeCollisionKeepsBothAssets(t *testing.T) { + ctx := context.Background() + root := t.TempDir() + vectorRoot := filepath.Join(root, "vectors") + writeTestFile(t, filepath.Join(vectorRoot, "a", "b.geojson"), `{"type":"FeatureCollection","features":[]}`) + writeTestFile(t, filepath.Join(vectorRoot, "a-b.geojson"), `{"type":"FeatureCollection","features":[]}`) + + store := openTestStore(t, StoreConfig{ + SQLitePath: filepath.Join(root, "basemap.db"), + VectorScanRoot: vectorRoot, + APIBaseURL: "http://127.0.0.1:8910", + VectorBaseURL: "http://127.0.0.1:8910", + }) + + if _, err := store.RunVectorScan(ctx); err != nil { + t.Fatalf("first vector scan: %v", err) + } + assets, err := store.ListVectorAssets(ctx) + if err != nil { + t.Fatalf("list assets: %v", err) + } + if len(assets) != 2 { + t.Fatalf("expected both colliding assets to survive, got %d", len(assets)) + } + if assets[0].Code == assets[1].Code { + t.Fatalf("expected distinct codes, both are %q", assets[0].Code) + } + firstCodes := []string{assets[0].Code, assets[1].Code} + + if _, err := store.RunVectorScan(ctx); err != nil { + t.Fatalf("second vector scan: %v", err) + } + assets, err = store.ListVectorAssets(ctx) + if err != nil { + t.Fatalf("list assets after rescan: %v", err) + } + if len(assets) != 2 { + t.Fatalf("rescan dropped an asset, now %d", len(assets)) + } + secondCodes := map[string]bool{assets[0].Code: true, assets[1].Code: true} + for _, code := range firstCodes { + if !secondCodes[code] { + t.Fatalf("disambiguated code %q was not stable across rescans", code) + } + } +} + +func TestVectorScanReusesChecksumUntilFileChanges(t *testing.T) { + ctx := context.Background() + root := t.TempDir() + vectorRoot := filepath.Join(root, "vectors") + file := filepath.Join(vectorRoot, "boundary.geojson") + writeTestFile(t, file, `{"type":"FeatureCollection","features":[]}`) + + store := openTestStore(t, StoreConfig{ + SQLitePath: filepath.Join(root, "basemap.db"), + VectorScanRoot: vectorRoot, + APIBaseURL: "http://127.0.0.1:8910", + VectorBaseURL: "http://127.0.0.1:8910", + }) + + if _, err := store.RunVectorScan(ctx); err != nil { + t.Fatalf("first scan: %v", err) + } + first, err := store.GetVectorAssetByCode(ctx, "boundary") + if err != nil { + t.Fatalf("get asset: %v", err) + } + if first.Metadata["hash_size"] == nil || first.Metadata["hash_mtime"] == nil { + t.Fatalf("expected hash stamp in metadata, got %v", first.Metadata) + } + + second, err := store.RunVectorScan(ctx) + if err != nil { + t.Fatalf("second scan: %v", err) + } + if second["unchanged_count"] != 1 { + t.Fatalf("expected unchanged_count=1, got %v", second["unchanged_count"]) + } + reused, err := store.GetVectorAssetByCode(ctx, "boundary") + if err != nil { + t.Fatalf("get asset after rescan: %v", err) + } + if reused.Checksum != first.Checksum { + t.Fatalf("checksum changed without a file change: %q -> %q", first.Checksum, reused.Checksum) + } + + // Rewriting the file with different content must invalidate the cached hash. + writeTestFile(t, file, `{"type":"FeatureCollection","features":[{"type":"Feature"}]}`) + if _, err := store.RunVectorScan(ctx); err != nil { + t.Fatalf("third scan: %v", err) + } + changed, err := store.GetVectorAssetByCode(ctx, "boundary") + if err != nil { + t.Fatalf("get asset after change: %v", err) + } + if changed.Checksum == first.Checksum { + t.Fatal("expected checksum to change after the file changed") + } +} + +func TestCreateTokenAppliesDefaultTTL(t *testing.T) { + ctx := context.Background() + root := t.TempDir() + tileRoot := filepath.Join(root, "tiles") + writeTestFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile") + + store := openTestStore(t, StoreConfig{ + SQLitePath: filepath.Join(root, "basemap.db"), + ScanRoot: tileRoot, + APIBaseURL: "http://127.0.0.1:8910", + TileBaseURL: "http://127.0.0.1:8910", + VectorBaseURL: "http://127.0.0.1:8910", + TokenDefaultTTL: time.Hour, + }) + if _, err := store.RunScan(ctx, defaultScanSourceCode); err != nil { + t.Fatalf("run scan: %v", err) + } + + created, err := store.CreateToken(ctx, CreateTokenInput{Name: "ttl", BasemapCodes: []string{"google_image"}}) + if err != nil { + t.Fatalf("create token: %v", err) + } + if created.Meta.ExpiresAt == nil { + t.Fatal("expected a default expiry to be applied") + } + remaining := time.Until(*created.Meta.ExpiresAt) + if remaining < 55*time.Minute || remaining > 65*time.Minute { + t.Fatalf("expected ~1h expiry, got %s", remaining) + } +} + +func TestPushRunsAsynchronously(t *testing.T) { + ctx := context.Background() + root := t.TempDir() + tileRoot := filepath.Join(root, "tiles") + writeTestFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile") + + received := make(chan map[string]any, 1) + callback := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var payload map[string]any + _ = json.NewDecoder(r.Body).Decode(&payload) + select { + case received <- payload: + default: + } + w.WriteHeader(http.StatusOK) + })) + defer callback.Close() + + store := openTestStore(t, StoreConfig{ + SQLitePath: filepath.Join(root, "basemap.db"), + ScanRoot: tileRoot, + APIBaseURL: "http://127.0.0.1:8910", + TileBaseURL: "http://127.0.0.1:8910", + VectorBaseURL: "http://127.0.0.1:8910", + }) + if _, err := store.RunScan(ctx, defaultScanSourceCode); err != nil { + t.Fatalf("run scan: %v", err) + } + if _, err := store.CreateTargetSystem(ctx, CreateTargetSystemInput{ + Code: "downstream", + CallbackURL: callback.URL, + }); err != nil { + t.Fatalf("create target: %v", err) + } + + record, err := store.PushBasemapVersion(ctx, "downstream", "google_image", "current") + if err != nil { + t.Fatalf("push: %v", err) + } + if record.Status != "pending" { + t.Fatalf("expected the push to be queued as pending, got %q", record.Status) + } + + payload := waitForPayload(t, received) + if payload["event"] != "basemap.published" { + t.Fatalf("unexpected callback payload: %v", payload) + } + + deadline := time.Now().Add(5 * time.Second) + for { + records, err := store.ListPushRecords(ctx, 5) + if err != nil { + t.Fatalf("list push records: %v", err) + } + done := false + for _, item := range records { + if item.ID == record.ID && item.Status == "succeeded" { + done = true + } + } + if done { + break + } + if time.Now().After(deadline) { + t.Fatalf("push record %s never reached succeeded: %+v", record.ID, records) + } + time.Sleep(25 * time.Millisecond) + } +} + +func waitForPayload(t *testing.T, received <-chan map[string]any) map[string]any { + t.Helper() + select { + case payload := <-received: + return payload + case <-time.After(5 * time.Second): + t.Fatal("callback was never invoked") + return nil + } +} + +func TestCloseWaitsForQueuedPush(t *testing.T) { + ctx := context.Background() + root := t.TempDir() + tileRoot := filepath.Join(root, "tiles") + writeTestFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile") + + started := make(chan struct{}, 1) + release := make(chan struct{}) + callback := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + select { + case started <- struct{}{}: + default: + } + <-release + w.WriteHeader(http.StatusOK) + })) + defer callback.Close() + + store, err := OpenStore(StoreConfig{ + SQLitePath: filepath.Join(root, "basemap.db"), + ScanRoot: tileRoot, + APIBaseURL: "http://127.0.0.1:8910", + TileBaseURL: "http://127.0.0.1:8910", + VectorBaseURL: "http://127.0.0.1:8910", + }) + if err != nil { + t.Fatalf("open store: %v", err) + } + if _, err := store.RunScan(ctx, defaultScanSourceCode); err != nil { + t.Fatalf("run scan: %v", err) + } + if _, err := store.CreateTargetSystem(ctx, CreateTargetSystemInput{Code: "downstream", CallbackURL: callback.URL}); err != nil { + t.Fatalf("create target: %v", err) + } + if _, err := store.PushBasemapVersion(ctx, "downstream", "google_image", "current"); err != nil { + t.Fatalf("push: %v", err) + } + + <-started + closed := make(chan struct{}) + go func() { + _ = store.Close() + close(closed) + }() + select { + case <-closed: + t.Fatal("Close returned while a push callback was still in flight") + case <-time.After(150 * time.Millisecond): + } + close(release) + select { + case <-closed: + case <-time.After(5 * time.Second): + t.Fatal("Close did not return after the push finished") + } +} + +func TestScanRootDirectoryIsNotRequiredAtOpen(t *testing.T) { + root := t.TempDir() + store := openTestStore(t, StoreConfig{SQLitePath: filepath.Join(root, "basemap.db")}) + if store.ScanRoot() != "" { + t.Fatalf("expected empty scan root, got %q", store.ScanRoot()) + } + if _, err := os.Stat(filepath.Join(root, "basemap.db")); err != nil { + t.Fatalf("expected the database file to exist: %v", err) + } +} diff --git a/services/api-go/internal/basemap/scan.go b/services/api-go/internal/basemap/scan.go index 961a062..059c91e 100644 --- a/services/api-go/internal/basemap/scan.go +++ b/services/api-go/internal/basemap/scan.go @@ -12,7 +12,7 @@ import ( "strings" "time" - "map-asset-gateway/api-go/internal/uid" + "tile-server/api-go/internal/uid" ) type scannedVersion struct { @@ -34,13 +34,30 @@ type scannedVersion struct { IsDefault bool } +// scanCounts reports what a scan actually changed, so operators can tell a real +// update from a no-op rescan. +type scanCounts struct { + Scanned int + Added int + Updated int + Removed int + Unchanged int +} + func (s *Store) RunScan(ctx context.Context, sourceCode string) (ScanRun, error) { + // A scan reads the current catalog and then rewrites it, so two overlapping + // runs could delete each other's rows. Refuse instead of interleaving. + if !s.scanMu.TryLock() { + return ScanRun{}, ErrScanInProgress + } + defer s.scanMu.Unlock() + source, err := s.GetScanSourceByCode(ctx, sourceCode) if err != nil { return ScanRun{}, err } if !source.Enabled { - return ScanRun{}, fmt.Errorf("scan source %q is disabled", sourceCode) + return ScanRun{}, &scanSourceError{sentinel: ErrScanSourceDisabled, code: sourceCode, reason: "is disabled"} } startedAt := nowUTC() @@ -58,21 +75,25 @@ func (s *Store) RunScan(ctx context.Context, sourceCode string) (ScanRun, error) } func (s *Store) runScan(ctx context.Context, runID string, source ScanSource, startedAt time.Time) (ScanRun, error) { + fail := func(err error) (ScanRun, error) { + return s.finishScanRun(ctx, runID, source, startedAt, "failed", nil, scanCounts{}, err) + } + rootInfo, err := os.Stat(source.RootPath) if err != nil { - return s.finishScanRun(ctx, runID, source, startedAt, "failed", nil, 0, 0, 0, 0, fmt.Errorf("stat scan root: %w", err)) + return fail(fmt.Errorf("stat scan root: %w", err)) } if !rootInfo.IsDir() { - return s.finishScanRun(ctx, runID, source, startedAt, "failed", nil, 0, 0, 0, 0, fmt.Errorf("scan root %s is not a directory", source.RootPath)) + return fail(fmt.Errorf("scan root %s is not a directory", source.RootPath)) } manifestItems, coveredRoots, err := s.findManifestVersions(source) if err != nil { - return s.finishScanRun(ctx, runID, source, startedAt, "failed", nil, 0, 0, 0, 0, err) + return fail(err) } fallbackItems, err := s.findDirectTileRoots(source, coveredRoots) if err != nil { - return s.finishScanRun(ctx, runID, source, startedAt, "failed", nil, 0, 0, 0, 0, err) + return fail(err) } allItems := append(manifestItems, fallbackItems...) @@ -83,11 +104,12 @@ func (s *Store) runScan(ctx context.Context, runID string, source ScanSource, st return allItems[i].BasemapCode < allItems[j].BasemapCode }) - addedCount, updatedCount, removedCount, err := s.persistScanResult(ctx, source, allItems) + counts, err := s.persistScanResult(ctx, source, allItems) if err != nil { - return s.finishScanRun(ctx, runID, source, startedAt, "failed", allItems, len(allItems), addedCount, updatedCount, removedCount, err) + return s.finishScanRun(ctx, runID, source, startedAt, "failed", allItems, scanCounts{Scanned: len(allItems)}, err) } - return s.finishScanRun(ctx, runID, source, startedAt, "succeeded", allItems, len(allItems), addedCount, updatedCount, removedCount, nil) + counts.Scanned = len(allItems) + return s.finishScanRun(ctx, runID, source, startedAt, "succeeded", allItems, counts, nil) } func (s *Store) finishScanRun( @@ -97,20 +119,18 @@ func (s *Store) finishScanRun( startedAt time.Time, status string, items []scannedVersion, - scannedCount int, - addedCount int, - updatedCount int, - removedCount int, + counts scanCounts, runErr error, ) (ScanRun, error) { finishedAt := nowUTC() summary := map[string]any{ - "source_code": source.Code, - "scan_root": source.RootPath, - "scanned_count": scannedCount, - "added_count": addedCount, - "updated_count": updatedCount, - "removed_count": removedCount, + "source_code": source.Code, + "scan_root": source.RootPath, + "scanned_count": counts.Scanned, + "added_count": counts.Added, + "updated_count": counts.Updated, + "removed_count": counts.Removed, + "unchanged_count": counts.Unchanged, } if runErr != nil { summary["error"] = runErr.Error() @@ -127,7 +147,7 @@ func (s *Store) finishScanRun( UPDATE scan_runs SET status = ?, scanned_count = ?, added_count = ?, updated_count = ?, removed_count = ?, summary_json = ?, finished_at = ? WHERE id = ? - `, status, scannedCount, addedCount, updatedCount, removedCount, writeJSON(summary), toRFC3339(finishedAt), runID) + `, status, counts.Scanned, counts.Added, counts.Updated, counts.Removed, writeJSON(summary), toRFC3339(finishedAt), runID) if updateErr != nil { if runErr != nil { return ScanRun{}, fmt.Errorf("%v; update scan run: %w", runErr, updateErr) @@ -140,10 +160,10 @@ func (s *Store) finishScanRun( ScanSourceID: source.ID, SourceCode: source.Code, Status: status, - ScannedCount: scannedCount, - AddedCount: addedCount, - UpdatedCount: updatedCount, - RemovedCount: removedCount, + ScannedCount: counts.Scanned, + AddedCount: counts.Added, + UpdatedCount: counts.Updated, + RemovedCount: counts.Removed, Summary: summary, StartedAt: startedAt, FinishedAt: finishedAt, @@ -154,6 +174,11 @@ func (s *Store) finishScanRun( return run, nil } +// manifestScanMaxDepth bounds how deep the manifest lookup descends. Manifest +// based basemaps place manifest.json at the basemap root or one version folder +// below it, so anything deeper is tile payload that must not be walked. +const manifestScanMaxDepth = 4 + func (s *Store) findManifestVersions(source ScanSource) ([]scannedVersion, map[string]struct{}, error) { manifestName := strings.TrimSpace(source.ManifestName) if manifestName == "" { @@ -162,69 +187,103 @@ func (s *Store) findManifestVersions(source ScanSource) ([]scannedVersion, map[s var items []scannedVersion coveredRoots := map[string]struct{}{} - err := filepath.WalkDir(source.RootPath, func(path string, entry os.DirEntry, walkErr error) error { - if walkErr != nil { - return walkErr + + var walk func(dir string, depth int) error + walk = func(dir string, depth int) error { + entries, err := os.ReadDir(dir) + if err != nil { + return err } - if entry.IsDir() { - return nil - } - if !strings.EqualFold(entry.Name(), manifestName) { + + // A directory holding a manifest is a basemap root: record it and stop, + // so the tile payload underneath (the default "tiles" subtree) is never + // traversed. + for _, entry := range entries { + if entry.IsDir() || !strings.EqualFold(entry.Name(), manifestName) { + continue + } + manifestPath := filepath.Join(dir, entry.Name()) + manifest, err := LoadManifest(manifestPath) + if err != nil { + return err + } + tileRoot, err := resolveTileRoot(dir, manifest.RootPath) + if err != nil { + return err + } + format, minZoom, maxZoom, found, err := inspectTileRoot(tileRoot, manifest.TileFormat) + if err != nil { + return err + } + if found { + manifest.TileFormat = format + manifest.MinZoom = minZoom + manifest.MaxZoom = maxZoom + } + + if relPath, err := filepath.Rel(source.RootPath, tileRoot); err == nil { + first := strings.Split(filepath.ToSlash(relPath), "/")[0] + if first != "" && first != "." { + coveredRoots[first] = struct{}{} + } + } + + items = append(items, scannedVersion{ + BasemapCode: manifest.Code, + Name: defaultText(manifest.Name, manifest.Code), + Type: defaultText(manifest.Type, "xyz"), + Status: defaultText(manifest.Status, "ready"), + Description: manifest.Description, + Version: defaultText(manifest.Version, "current"), + Manifest: manifestPath, + TileRoot: tileRoot, + TileFormat: normalizeTileFormat(manifest.TileFormat), + TileScheme: defaultText(manifest.TileScheme, "xyz"), + MinZoom: manifest.MinZoom, + MaxZoom: manifest.MaxZoom, + BBox: manifest.BBox, + Attribution: manifest.Attribution, + Metadata: manifest.Metadata, + IsDefault: manifest.IsDefault, + }) return nil } - manifest, err := LoadManifest(path) - if err != nil { - return err + if depth >= manifestScanMaxDepth { + return nil } - tileRoot, err := resolveTileRoot(filepath.Dir(path), manifest.RootPath) - if err != nil { - return err - } - format, minZoom, maxZoom, found, err := inspectTileRoot(tileRoot, manifest.TileFormat) - if err != nil { - return err - } - if found { - manifest.TileFormat = format - manifest.MinZoom = minZoom - manifest.MaxZoom = maxZoom - } - - relPath, err := filepath.Rel(source.RootPath, tileRoot) - if err == nil { - first := strings.Split(filepath.ToSlash(relPath), "/")[0] - if first != "" && first != "." { - coveredRoots[first] = struct{}{} + for _, entry := range entries { + if !entry.IsDir() || isNumericDirName(entry.Name()) { + continue + } + if err := walk(filepath.Join(dir, entry.Name()), depth+1); err != nil { + return err } } - - items = append(items, scannedVersion{ - BasemapCode: manifest.Code, - Name: defaultText(manifest.Name, manifest.Code), - Type: defaultText(manifest.Type, "xyz"), - Status: defaultText(manifest.Status, "ready"), - Description: manifest.Description, - Version: defaultText(manifest.Version, "current"), - Manifest: path, - TileRoot: tileRoot, - TileFormat: normalizeTileFormat(manifest.TileFormat), - TileScheme: defaultText(manifest.TileScheme, "xyz"), - MinZoom: manifest.MinZoom, - MaxZoom: manifest.MaxZoom, - BBox: manifest.BBox, - Attribution: manifest.Attribution, - Metadata: manifest.Metadata, - IsDefault: manifest.IsDefault, - }) return nil - }) - if err != nil { + } + + if err := walk(source.RootPath, 0); err != nil { return nil, nil, fmt.Errorf("walk manifests: %w", err) } return items, coveredRoots, nil } +// isNumericDirName reports whether the directory name is a bare zoom level such +// as "0" or "14". Raw tile roots nest z/x/y under the basemap directory, so +// pruning these keeps the scan clear of the tile payload. +func isNumericDirName(name string) bool { + if name == "" { + return false + } + for _, r := range name { + if r < '0' || r > '9' { + return false + } + } + return true +} + func (s *Store) findDirectTileRoots(source ScanSource, skip map[string]struct{}) ([]scannedVersion, error) { entries, err := os.ReadDir(source.RootPath) if err != nil { @@ -352,57 +411,114 @@ func resolveTileRoot(baseDir, relativePath string) (string, error) { return resolved, nil } -func (s *Store) persistScanResult(ctx context.Context, source ScanSource, items []scannedVersion) (int, int, int, error) { +// existingVersion is a snapshot of a stored version, used to detect whether a +// rescan actually changed anything. +type existingVersion struct { + Name string + Type string + Status string + Description string + ManifestPath string + TileRootPath string + TileFormat string + TileScheme string + MinZoom int + MaxZoom int + IsDefault bool + BBoxJSON string + Attribution string + MetadataJSON string +} + +func (s *Store) persistScanResult(ctx context.Context, source ScanSource, items []scannedVersion) (scanCounts, error) { tx, err := s.db.BeginTx(ctx, nil) if err != nil { - return 0, 0, 0, err + return scanCounts{}, err } defer tx.Rollback() - existing := map[string]struct{}{} + existing := map[string]existingVersion{} rows, err := tx.QueryContext(ctx, ` - SELECT b.code, v.version + SELECT + b.code, + v.version, + b.name, + b.type, + b.status, + b.description, + v.manifest_path, + v.tile_root_path, + v.tile_format, + v.tile_scheme, + v.min_zoom, + v.max_zoom, + v.is_default, + v.bbox_json, + v.attribution, + v.metadata_json FROM basemap_versions v JOIN basemaps b ON b.id = v.basemap_id WHERE v.scan_source_id = ? `, source.ID) if err != nil { - return 0, 0, 0, err + return scanCounts{}, err } for rows.Next() { var code string var version string - if err := rows.Scan(&code, &version); err != nil { + var item existingVersion + var isDefault int + if err := rows.Scan( + &code, + &version, + &item.Name, + &item.Type, + &item.Status, + &item.Description, + &item.ManifestPath, + &item.TileRootPath, + &item.TileFormat, + &item.TileScheme, + &item.MinZoom, + &item.MaxZoom, + &isDefault, + &item.BBoxJSON, + &item.Attribution, + &item.MetadataJSON, + ); err != nil { rows.Close() - return 0, 0, 0, err + return scanCounts{}, err } - existing[code+"::"+version] = struct{}{} + item.IsDefault = isDefault == 1 + existing[code+"::"+version] = item } rows.Close() if err := rows.Err(); err != nil { - return 0, 0, 0, err + return scanCounts{}, err } now := nowUTC() found := map[string]struct{}{} touchedBasemaps := map[string]struct{}{} - addedCount := 0 - updatedCount := 0 + counts := scanCounts{} for _, item := range items { key := item.BasemapCode + "::" + item.Version found[key] = struct{}{} - if _, exists := existing[key]; exists { - updatedCount++ + if prev, exists := existing[key]; exists { + if scannedVersionMatches(item, prev) { + counts.Unchanged++ + } else { + counts.Updated++ + } } else { - addedCount++ + counts.Added++ } if err := s.upsertScannedVersionTx(ctx, tx, source, item, now); err != nil { - return 0, 0, 0, err + return scanCounts{}, err } touchedBasemaps[item.BasemapCode] = struct{}{} } - removedCount := 0 for key := range existing { if _, ok := found[key]; ok { continue @@ -412,22 +528,41 @@ func (s *Store) persistScanResult(ctx context.Context, source ScanSource, items continue } if err := s.deleteVersionBySourceTx(ctx, tx, source.ID, parts[0], parts[1]); err != nil { - return 0, 0, 0, err + return scanCounts{}, err } touchedBasemaps[parts[0]] = struct{}{} - removedCount++ + counts.Removed++ } for code := range touchedBasemaps { if err := s.normalizeBasemapDefaultsTx(ctx, tx, code, now); err != nil { - return 0, 0, 0, err + return scanCounts{}, err } } if err := tx.Commit(); err != nil { - return 0, 0, 0, err + return scanCounts{}, err } - return addedCount, updatedCount, removedCount, nil + return counts, nil +} + +// scannedVersionMatches reports whether a freshly scanned version is identical +// to what is already stored, so an unchanged rescan is not reported as updated. +func scannedVersionMatches(item scannedVersion, prev existingVersion) bool { + return item.Name == prev.Name && + item.Type == prev.Type && + item.Status == prev.Status && + item.Description == prev.Description && + item.Manifest == prev.ManifestPath && + item.TileRoot == prev.TileRootPath && + normalizeTileFormat(item.TileFormat) == prev.TileFormat && + item.TileScheme == prev.TileScheme && + item.MinZoom == prev.MinZoom && + item.MaxZoom == prev.MaxZoom && + item.IsDefault == prev.IsDefault && + writeJSON(item.BBox) == prev.BBoxJSON && + item.Attribution == prev.Attribution && + writeJSON(item.Metadata) == prev.MetadataJSON } func (s *Store) upsertScannedVersionTx(ctx context.Context, tx *sql.Tx, source ScanSource, item scannedVersion, now time.Time) error { diff --git a/services/api-go/internal/basemap/store.go b/services/api-go/internal/basemap/store.go index ea934ed..44d3168 100644 --- a/services/api-go/internal/basemap/store.go +++ b/services/api-go/internal/basemap/store.go @@ -11,37 +11,97 @@ import ( "errors" "fmt" "io" + "log" "net/http" + "net/url" "os" "path/filepath" "sort" "strings" + "sync" "time" _ "modernc.org/sqlite" ) const ( - defaultScanSourceCode = "local-basemaps" - defaultManifestName = "manifest.json" - readPermission = "read" + defaultScanSourceCode = "local-basemaps" + defaultManifestName = "manifest.json" + readPermission = "read" defaultVectorSourceCode = "local-vectors" ) +const ( + // sqliteMaxOpenConns lets concurrent tile reads overlap instead of being + // serialized behind a single connection. + sqliteMaxOpenConns = 8 +) + +// sqliteDSN tunes each pooled connection. busy_timeout makes concurrent +// writers wait for the lock instead of failing immediately. +func sqliteDSN(path string) string { + params := url.Values{} + params.Add("_pragma", "busy_timeout(5000)") + params.Add("_pragma", "foreign_keys(1)") + params.Add("_pragma", "synchronous(NORMAL)") + return "file:" + filepath.ToSlash(path) + "?" + params.Encode() +} + +var ( + // ErrScanInProgress is returned when a scan is already running. + ErrScanInProgress = errors.New("scan already in progress") + // ErrScanSourceNotFound is returned when a scan source code is unknown. + ErrScanSourceNotFound = errors.New("scan source not found") + // ErrScanSourceDisabled is returned when a scan source is turned off. + ErrScanSourceDisabled = errors.New("scan source disabled") +) + +// scanSourceError keeps the operator-facing message stable while still matching +// errors.Is against a sentinel. +type scanSourceError struct { + sentinel error + code string + reason string +} + +func (e *scanSourceError) Error() string { + return fmt.Sprintf("scan source %q %s", e.code, e.reason) +} + +func (e *scanSourceError) Is(target error) bool { + return target == e.sentinel +} + type StoreConfig struct { - SQLitePath string - ScanRoot string - VectorScanRoot string - APIBaseURL string - TileBaseURL string - VectorBaseURL string - AutoScanOnStart bool + SQLitePath string + ScanRoot string + VectorScanRoot string + APIBaseURL string + TileBaseURL string + VectorBaseURL string + AutoScanOnStart bool + TokenDefaultTTL time.Duration + TileJobCommand string + TileJobVRTCommand string + TileJobLogDir string + HealthCheckInterval time.Duration } type Store struct { db *sql.DB cfg StoreConfig httpClient *http.Client + + // scanMu serializes scans so two runs cannot interleave their read-then-write + // bookkeeping. + scanMu sync.Mutex + // pushWG tracks detached push callbacks so Close can drain them. + pushWG sync.WaitGroup + + bgOnce sync.Once + bgCancel context.CancelFunc + bgCtx context.Context + bgWG sync.WaitGroup } type CreatedToken struct { @@ -66,13 +126,21 @@ func OpenStore(cfg StoreConfig) (*Store, error) { return nil, fmt.Errorf("create sqlite dir: %w", err) } - db, err := sql.Open("sqlite", cfg.SQLitePath) + db, err := sql.Open("sqlite", sqliteDSN(cfg.SQLitePath)) if err != nil { return nil, fmt.Errorf("open sqlite: %w", err) } - db.SetConnMaxLifetime(10 * time.Minute) - db.SetMaxOpenConns(1) - db.SetMaxIdleConns(1) + db.SetConnMaxLifetime(30 * time.Minute) + db.SetConnMaxIdleTime(5 * time.Minute) + db.SetMaxOpenConns(sqliteMaxOpenConns) + db.SetMaxIdleConns(sqliteMaxOpenConns) + + // WAL keeps tile reads from blocking on scans/pushes. It is persisted in the + // database file, so enabling it once is enough; a filesystem that cannot + // support WAL (some network shares) must not stop the service from starting. + if _, err := db.Exec("PRAGMA journal_mode=WAL;"); err != nil { + log.Printf("enable sqlite WAL skipped: %v", err) + } store := &Store{ db: db, @@ -109,6 +177,13 @@ func (s *Store) Close() error { if s == nil || s.db == nil { return nil } + // Stop the periodic loops, then let detached push callbacks finish before + // the pool goes away. + if s.bgCancel != nil { + s.bgCancel() + } + s.bgWG.Wait() + s.pushWG.Wait() return s.db.Close() } @@ -142,6 +217,9 @@ func (s *Store) bootstrap(ctx context.Context) error { bbox_json TEXT NOT NULL DEFAULT '', attribution TEXT NOT NULL DEFAULT '', metadata_json TEXT NOT NULL DEFAULT '', + health TEXT NOT NULL DEFAULT '', + health_checked_at TEXT NOT NULL DEFAULT '', + health_detail TEXT NOT NULL DEFAULT '', created_at TEXT NOT NULL, updated_at TEXT NOT NULL, UNIQUE(basemap_id, version), @@ -188,6 +266,7 @@ func (s *Store) bootstrap(ctx context.Context) error { name TEXT NOT NULL, token_hash TEXT NOT NULL UNIQUE, token_prefix TEXT NOT NULL, + scopes TEXT NOT NULL DEFAULT 'read', status TEXT NOT NULL, expires_at TEXT, last_used_at TEXT, @@ -213,6 +292,8 @@ func (s *Store) bootstrap(ctx context.Context) error { response_status INTEGER, response_body TEXT NOT NULL DEFAULT '', error_message TEXT NOT NULL DEFAULT '', + attempt_count INTEGER NOT NULL DEFAULT 0, + next_attempt_at TEXT, pushed_at TEXT NOT NULL, finished_at TEXT, FOREIGN KEY (target_system_id) REFERENCES target_systems(id) ON DELETE CASCADE, @@ -231,6 +312,9 @@ func (s *Store) bootstrap(ctx context.Context) error { checksum TEXT NOT NULL DEFAULT '', download_url TEXT NOT NULL, metadata_json TEXT NOT NULL DEFAULT '', + health TEXT NOT NULL DEFAULT '', + health_checked_at TEXT NOT NULL DEFAULT '', + health_detail TEXT NOT NULL DEFAULT '', created_at TEXT NOT NULL, updated_at TEXT NOT NULL );`, @@ -252,15 +336,55 @@ func (s *Store) bootstrap(ctx context.Context) error { response_status INTEGER, response_body TEXT NOT NULL DEFAULT '', error_message TEXT NOT NULL DEFAULT '', + attempt_count INTEGER NOT NULL DEFAULT 0, + next_attempt_at TEXT, pushed_at TEXT NOT NULL, finished_at TEXT, FOREIGN KEY (target_system_id) REFERENCES target_systems(id) ON DELETE CASCADE, FOREIGN KEY (vector_asset_id) REFERENCES vector_assets(id) ON DELETE CASCADE );`, + `CREATE TABLE IF NOT EXISTS tile_jobs ( + id TEXT PRIMARY KEY, + code TEXT NOT NULL UNIQUE, + name TEXT NOT NULL, + source_path TEXT NOT NULL, + source_count INTEGER NOT NULL DEFAULT 1, + sources_json TEXT NOT NULL DEFAULT '', + target_code TEXT NOT NULL, + output_root TEXT NOT NULL, + zoom_min INTEGER NOT NULL DEFAULT 0, + zoom_max INTEGER NOT NULL DEFAULT 0, + tile_size INTEGER NOT NULL DEFAULT 256, + tile_format TEXT NOT NULL DEFAULT 'webp', + status TEXT NOT NULL, + progress TEXT NOT NULL DEFAULT '', + message TEXT NOT NULL DEFAULT '', + log_path TEXT NOT NULL DEFAULT '', + callback_url TEXT NOT NULL DEFAULT '', + callback_status TEXT NOT NULL DEFAULT '', + callback_attempts INTEGER NOT NULL DEFAULT 0, + callback_next_attempt_at TEXT, + callback_message TEXT NOT NULL DEFAULT '', + requested_by TEXT NOT NULL DEFAULT '', + created_at TEXT NOT NULL, + started_at TEXT, + finished_at TEXT + );`, + `CREATE INDEX IF NOT EXISTS idx_tile_jobs_status ON tile_jobs(status, created_at);`, + `CREATE INDEX IF NOT EXISTS idx_tile_jobs_target ON tile_jobs(target_code);`, + `CREATE TABLE IF NOT EXISTS layer_sources ( + id TEXT PRIMARY KEY, + target_code TEXT NOT NULL, + source_path TEXT NOT NULL, + added_at TEXT NOT NULL, + UNIQUE(target_code, source_path) + );`, + `CREATE INDEX IF NOT EXISTS idx_layer_sources_target ON layer_sources(target_code);`, `CREATE INDEX IF NOT EXISTS idx_versions_basemap ON basemap_versions(basemap_id);`, `CREATE INDEX IF NOT EXISTS idx_versions_source ON basemap_versions(scan_source_id);`, `CREATE INDEX IF NOT EXISTS idx_grants_token ON token_grants(token_id);`, `CREATE INDEX IF NOT EXISTS idx_push_target ON push_records(target_system_id);`, + `CREATE INDEX IF NOT EXISTS idx_vector_file_name ON vector_assets(file_name COLLATE NOCASE);`, `CREATE INDEX IF NOT EXISTS idx_vector_grants_token ON vector_token_grants(token_id);`, `CREATE INDEX IF NOT EXISTS idx_vector_push_target ON vector_push_records(target_system_id);`, } @@ -270,9 +394,77 @@ func (s *Store) bootstrap(ctx context.Context) error { return fmt.Errorf("bootstrap sqlite: %w", err) } } + + // CREATE TABLE IF NOT EXISTS never evolves an existing database, so columns + // added after the first release are patched in here. + migrations := []struct { + table string + column string + definition string + }{ + {"basemap_versions", "health", "TEXT NOT NULL DEFAULT ''"}, + {"basemap_versions", "health_checked_at", "TEXT NOT NULL DEFAULT ''"}, + {"basemap_versions", "health_detail", "TEXT NOT NULL DEFAULT ''"}, + {"vector_assets", "health", "TEXT NOT NULL DEFAULT ''"}, + {"vector_assets", "health_checked_at", "TEXT NOT NULL DEFAULT ''"}, + {"vector_assets", "health_detail", "TEXT NOT NULL DEFAULT ''"}, + {"push_records", "attempt_count", "INTEGER NOT NULL DEFAULT 0"}, + {"push_records", "next_attempt_at", "TEXT"}, + {"vector_push_records", "attempt_count", "INTEGER NOT NULL DEFAULT 0"}, + {"vector_push_records", "next_attempt_at", "TEXT"}, + {"service_tokens", "scopes", "TEXT NOT NULL DEFAULT 'read'"}, + {"tile_jobs", "source_count", "INTEGER NOT NULL DEFAULT 1"}, + {"tile_jobs", "sources_json", "TEXT NOT NULL DEFAULT ''"}, + {"tile_jobs", "callback_url", "TEXT NOT NULL DEFAULT ''"}, + {"tile_jobs", "callback_status", "TEXT NOT NULL DEFAULT ''"}, + {"tile_jobs", "callback_attempts", "INTEGER NOT NULL DEFAULT 0"}, + {"tile_jobs", "callback_next_attempt_at", "TEXT"}, + {"tile_jobs", "callback_message", "TEXT NOT NULL DEFAULT ''"}, + } + for _, migration := range migrations { + if err := s.ensureColumn(ctx, migration.table, migration.column, migration.definition); err != nil { + return fmt.Errorf("bootstrap migrate %s.%s: %w", migration.table, migration.column, err) + } + } + // These indexes reference migrated columns, so they run after the patch. + for _, statement := range []string{ + `CREATE INDEX IF NOT EXISTS idx_push_next_attempt ON push_records(next_attempt_at);`, + `CREATE INDEX IF NOT EXISTS idx_vector_push_next_attempt ON vector_push_records(next_attempt_at);`, + } { + if _, err := s.db.ExecContext(ctx, statement); err != nil { + return fmt.Errorf("bootstrap index: %w", err) + } + } return nil } +// ensureColumn adds a column when an existing table is missing it. PRAGMA +// table_info is the portable way to test SQLite column presence. +func (s *Store) ensureColumn(ctx context.Context, table, column, definition string) error { + rows, err := s.db.QueryContext(ctx, "PRAGMA table_info("+table+")") + if err != nil { + return err + } + defer rows.Close() + + for rows.Next() { + var cid, notNull, primaryKey int + var name, columnType string + var defaultValue sql.NullString + if err := rows.Scan(&cid, &name, &columnType, ¬Null, &defaultValue, &primaryKey); err != nil { + return err + } + if name == column { + return nil + } + } + if err := rows.Err(); err != nil { + return err + } + _, err = s.db.ExecContext(ctx, "ALTER TABLE "+table+" ADD COLUMN "+column+" "+definition) + return err +} + func (s *Store) AutoScanEnabled() bool { return s.cfg.AutoScanOnStart } diff --git a/services/api-go/internal/basemap/store_rows.go b/services/api-go/internal/basemap/store_rows.go index 1c26142..d1f13de 100644 --- a/services/api-go/internal/basemap/store_rows.go +++ b/services/api-go/internal/basemap/store_rows.go @@ -2,7 +2,6 @@ package basemap import ( "database/sql" - "fmt" "strings" "time" ) @@ -228,6 +227,7 @@ func scanServiceToken(scanner interface { }) (ServiceToken, string, error) { var item ServiceToken var tokenHash string + var scopes string var expiresAt sql.NullString var lastUsedAt sql.NullString var createdAt string @@ -236,6 +236,7 @@ func scanServiceToken(scanner interface { &item.Name, &tokenHash, &item.TokenPrefix, + &scopes, &item.Status, &expiresAt, &lastUsedAt, @@ -244,6 +245,7 @@ func scanServiceToken(scanner interface { return ServiceToken{}, "", err } item.CreatedAt = parseTime(createdAt) + item.Scopes = splitTokenScopes(scopes) if expiresAt.Valid { value := parseTime(expiresAt.String) item.ExpiresAt = &value @@ -290,6 +292,7 @@ func scanPushRecord(scanner interface { var item PushRecord var requestJSON string var responseStatus sql.NullInt64 + var nextAttemptAt sql.NullString var finishedAt sql.NullString var pushedAt string if err := scanner.Scan( @@ -304,6 +307,8 @@ func scanPushRecord(scanner interface { &responseStatus, &item.ResponseBody, &item.ErrorMessage, + &item.AttemptCount, + &nextAttemptAt, &pushedAt, &finishedAt, ); err != nil { @@ -319,6 +324,10 @@ func scanPushRecord(scanner interface { value := int(responseStatus.Int64) item.ResponseStatus = &value } + if nextAttemptAt.Valid { + value := parseTime(nextAttemptAt.String) + item.NextAttemptAt = &value + } item.PushedAt = parseTime(pushedAt) if finishedAt.Valid { value := parseTime(finishedAt.String) @@ -326,19 +335,3 @@ func scanPushRecord(scanner interface { } return item, nil } - -func mustOneRow[T any](item T, err error) (T, error) { - if err != nil { - return item, err - } - return item, nil -} - -func unexpectedRowsAffected(count int64, expected ...int64) error { - for _, value := range expected { - if count == value { - return nil - } - } - return fmt.Errorf("unexpected rows affected: %d", count) -} diff --git a/services/api-go/internal/basemap/tilejobs.go b/services/api-go/internal/basemap/tilejobs.go new file mode 100644 index 0000000..35d5c75 --- /dev/null +++ b/services/api-go/internal/basemap/tilejobs.go @@ -0,0 +1,714 @@ +package basemap + +import ( + "context" + "database/sql" + "errors" + "fmt" + "log" + "net/http" + "os" + "os/exec" + "path/filepath" + "strings" + "time" + + "tile-server/api-go/internal/uid" +) + +// Tile job statuses. +const ( + tileJobPending = "pending" + tileJobRunning = "running" + tileJobSucceeded = "succeeded" + tileJobFailed = "failed" +) + +// defaultTileJobCommand slices a raster into XYZ tiles. {input} is the source +// GeoTIFF, {output} the directory the gateway scans. +const defaultTileJobCommand = "gdal2tiles.py --xyz --zoom={zmin}-{zmax} --tilesize={tilesize} --exclude {input} {output}" + +// defaultTileJobVRTCommand mosaics a layer's sources into one virtual raster +// so overlapping flights composite per pixel instead of per tile. +const defaultTileJobVRTCommand = "gdalbuildvrt -resolution highest {output_vrt} {inputs}" + +// tileJobColumns keeps every read path in step with the struct. +const tileJobColumns = `id, code, name, source_path, source_count, sources_json, target_code, output_root, zoom_min, zoom_max, + tile_size, tile_format, status, progress, message, log_path, callback_url, callback_status, + callback_attempts, callback_next_attempt_at, callback_message, requested_by, created_at, started_at, finished_at` + +type TileJob struct { + ID string `json:"id"` + Code string `json:"code"` + Name string `json:"name"` + SourcePath string `json:"source_path"` + SourceCount int `json:"source_count"` + SourcePaths []string `json:"source_paths,omitempty"` + TargetCode string `json:"target_code"` + OutputRoot string `json:"output_root"` + ZoomMin int `json:"zoom_min"` + ZoomMax int `json:"zoom_max"` + TileSize int `json:"tile_size"` + TileFormat string `json:"tile_format"` + Status string `json:"status"` + Progress string `json:"progress"` + Message string `json:"message"` + LogPath string `json:"log_path"` + CallbackURL string `json:"callback_url,omitempty"` + CallbackStatus string `json:"callback_status,omitempty"` + CallbackAttempts int `json:"callback_attempts,omitempty"` + CallbackMessage string `json:"callback_message,omitempty"` + CallbackNextAttemptAt *time.Time `json:"callback_next_attempt_at,omitempty"` + RequestedBy string `json:"requested_by"` + CreatedAt time.Time `json:"created_at"` + StartedAt *time.Time `json:"started_at,omitempty"` + FinishedAt *time.Time `json:"finished_at,omitempty"` +} + +type CreateTileJobInput struct { + Code string `json:"code"` + Name string `json:"name"` + SourcePath string `json:"source_path"` + SourcePaths []string `json:"source_paths"` + TargetCode string `json:"target_code"` + ZoomMin int `json:"zoom_min"` + ZoomMax int `json:"zoom_max"` + TileSize int `json:"tile_size"` + TileFormat string `json:"tile_format"` + CallbackURL string `json:"callback_url"` + RequestedBy string `json:"requested_by"` +} + +// CreateTileJob validates the request and queues it. Slicing itself runs in the +// background so a long gdal2tiles run cannot block the console. +func (s *Store) CreateTileJob(ctx context.Context, input CreateTileJobInput) (TileJob, error) { + sources, err := resolveTileJobSources(input) + if err != nil { + return TileJob{}, err + } + absSource := sources[0] + + targetCode := normalizeBasemapCode(input.TargetCode) + if targetCode == "" { + return TileJob{}, errors.New("target code is required") + } + scanRoot := strings.TrimSpace(s.cfg.ScanRoot) + if scanRoot == "" { + return TileJob{}, errors.New("basemap scan root is not configured") + } + outputRoot := filepath.Join(scanRoot, targetCode) + + if input.ZoomMin < 0 || input.ZoomMax < 0 || input.ZoomMax < input.ZoomMin { + return TileJob{}, errors.New("invalid zoom range") + } + if input.ZoomMax > 24 { + return TileJob{}, errors.New("zoom must be 24 or lower") + } + tileSize := input.TileSize + if tileSize <= 0 { + tileSize = 256 + } + tileFormat := strings.ToLower(strings.TrimSpace(input.TileFormat)) + if tileFormat == "" { + tileFormat = "webp" + } + code := normalizeBasemapCode(input.Code) + if code == "" { + code = normalizeBasemapCode(targetCode + "-" + nowUTC().Format("20060102150405")) + } + name := strings.TrimSpace(input.Name) + if name == "" { + name = filepath.Base(absSource) + } + + now := nowUTC() + job := TileJob{ + ID: uid.New(), + Code: code, + Name: name, + SourcePath: absSource, + SourceCount: len(sources), + SourcePaths: sources, + TargetCode: targetCode, + OutputRoot: outputRoot, + ZoomMin: input.ZoomMin, + ZoomMax: input.ZoomMax, + TileSize: tileSize, + TileFormat: tileFormat, + Status: tileJobPending, + RequestedBy: strings.TrimSpace(input.RequestedBy), + CreatedAt: now, + } + job.CallbackURL = strings.TrimSpace(input.CallbackURL) + + // A caller-supplied code makes submission idempotent: sending the same code + // again returns the job that already exists instead of queueing a second copy. + if existing, ok, err := s.getTileJobByCode(ctx, job.Code); err != nil { + return TileJob{}, err + } else if ok { + return existing, nil + } + + if _, err := s.db.ExecContext(ctx, ` + INSERT INTO tile_jobs ( + id, code, name, source_path, source_count, sources_json, target_code, output_root, zoom_min, zoom_max, + tile_size, tile_format, status, progress, message, log_path, callback_url, requested_by, created_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, '', '', '', ?, ?, ?) + `, job.ID, job.Code, job.Name, job.SourcePath, job.SourceCount, writeJSON(job.SourcePaths), job.TargetCode, job.OutputRoot, + job.ZoomMin, job.ZoomMax, job.TileSize, job.TileFormat, job.Status, + job.CallbackURL, job.RequestedBy, toRFC3339(now)); err != nil { + if strings.Contains(err.Error(), "UNIQUE") { + if existing, ok, getErr := s.getTileJobByCode(ctx, job.Code); getErr == nil && ok { + return existing, nil + } + } + return TileJob{}, err + } + // Remember what this layer is made of so it can be rebuilt later. + if err := s.AddLayerSources(ctx, job.TargetCode, sources); err != nil { + log.Printf("record layer sources for %s: %v", job.TargetCode, err) + } + + return job, nil +} + +func (s *Store) ListTileJobs(ctx context.Context, limit int) ([]TileJob, error) { + if limit <= 0 { + limit = 20 + } + rows, err := s.db.QueryContext(ctx, "SELECT "+tileJobColumns+` + FROM tile_jobs + ORDER BY created_at DESC + LIMIT ? + `, limit) + if err != nil { + return nil, err + } + defer rows.Close() + + var items []TileJob + for rows.Next() { + job, err := scanTileJob(rows) + if err != nil { + return nil, err + } + items = append(items, job) + } + return items, rows.Err() +} + +func scanTileJob(scanner interface { + Scan(dest ...any) error +}) (TileJob, error) { + var job TileJob + var createdAt string + var sourcesJSON string + var startedAt, finishedAt, callbackNextAttemptAt sql.NullString + if err := scanner.Scan( + &job.ID, &job.Code, &job.Name, &job.SourcePath, &job.SourceCount, &sourcesJSON, &job.TargetCode, &job.OutputRoot, + &job.ZoomMin, &job.ZoomMax, &job.TileSize, &job.TileFormat, &job.Status, + &job.Progress, &job.Message, &job.LogPath, &job.CallbackURL, &job.CallbackStatus, &job.CallbackAttempts, + &callbackNextAttemptAt, &job.CallbackMessage, &job.RequestedBy, &createdAt, + &startedAt, &finishedAt, + ); err != nil { + return TileJob{}, err + } + job.CreatedAt = parseTime(createdAt) + if strings.TrimSpace(sourcesJSON) != "" { + _ = readMaybeJSON(sourcesJSON, &job.SourcePaths) + } + if len(job.SourcePaths) == 0 && job.SourcePath != "" { + job.SourcePaths = []string{job.SourcePath} + } + if startedAt.Valid { + value := parseTime(startedAt.String) + job.StartedAt = &value + } + if finishedAt.Valid { + value := parseTime(finishedAt.String) + job.FinishedAt = &value + } + if callbackNextAttemptAt.Valid { + value := parseTime(callbackNextAttemptAt.String) + job.CallbackNextAttemptAt = &value + } + return job, nil +} + +// GetTileJob returns one job by id so a caller can poll a submission. +func (s *Store) GetTileJob(ctx context.Context, id string) (TileJob, error) { + row := s.db.QueryRowContext(ctx, "SELECT "+tileJobColumns+" FROM tile_jobs WHERE id = ?", strings.TrimSpace(id)) + job, err := scanTileJob(row) + if err != nil { + if errors.Is(err, sql.ErrNoRows) { + return TileJob{}, fmt.Errorf("tile job %q not found", id) + } + return TileJob{}, err + } + return job, nil +} + +func (s *Store) getTileJobByCode(ctx context.Context, code string) (TileJob, bool, error) { + row := s.db.QueryRowContext(ctx, "SELECT "+tileJobColumns+" FROM tile_jobs WHERE code = ?", code) + job, err := scanTileJob(row) + if err != nil { + if errors.Is(err, sql.ErrNoRows) { + return TileJob{}, false, nil + } + return TileJob{}, false, err + } + return job, true, nil +} + +// runTileJob executes one queued job and records the outcome verbatim. +func (s *Store) runTileJob(ctx context.Context, job TileJob) { + startedAt := nowUTC() + if _, err := s.db.ExecContext(ctx, ` + UPDATE tile_jobs SET status = ?, started_at = ?, message = '' WHERE id = ? + `, tileJobRunning, toRFC3339(startedAt), job.ID); err != nil { + return + } + + logPath, _, runErr := s.executeTileJob(ctx, job) + + status := tileJobSucceeded + message := "" + if runErr != nil { + status = tileJobFailed + message = runErr.Error() + } + finishedAt := nowUTC() + if _, err := s.db.ExecContext(context.Background(), ` + UPDATE tile_jobs SET status = ?, message = ?, log_path = ?, progress = ?, finished_at = ? WHERE id = ? + `, status, message, logPath, message, toRFC3339(finishedAt), job.ID); err != nil { + return + } + // Tell the submitting system the job is done, if it asked to be told. + if strings.TrimSpace(job.CallbackURL) != "" { + s.queueTileJobCallback(context.Background(), job.ID) + } + + // A sliced basemap only becomes visible after the scan picks it up. + if status == tileJobSucceeded && strings.TrimSpace(s.cfg.ScanRoot) != "" { + if _, err := s.RunScan(context.Background(), defaultScanSourceCode); err != nil { + if _, updateErr := s.db.ExecContext(context.Background(), ` + UPDATE tile_jobs SET message = ? WHERE id = ? + `, "切片完成，但扫描入库失败："+err.Error(), job.ID); updateErr != nil { + return + } + } + } +} + +// executeTileJob builds the input (mosaicking when several sources were +// submitted), runs the slicer and captures its output. +func (s *Store) executeTileJob(ctx context.Context, job TileJob) (string, []string, error) { + if err := os.MkdirAll(job.OutputRoot, 0o755); err != nil { + return "", nil, fmt.Errorf( + "create output root %s: %w (the account running the service needs write access to the basemap scan root)", + job.OutputRoot, err) + } + logDir := strings.TrimSpace(s.cfg.TileJobLogDir) + if logDir == "" { + // Keep logs beside the data: a system temp directory is not always + // writable for the service account, and this stays under the scan root + // without introducing a zoom directory the scanner would pick up. + logDir = filepath.Join(filepath.Dir(job.OutputRoot), ".tile-jobs") + } + if err := os.MkdirAll(logDir, 0o755); err != nil { + return "", nil, fmt.Errorf("create log dir: %w", err) + } + logPath := filepath.Join(logDir, job.Code+".log") + logFile, err := os.Create(logPath) + if err != nil { + return logPath, nil, fmt.Errorf("create job log: %w", err) + } + defer logFile.Close() + + // One source slices directly; several are mosaicked first so overlapping + // flights composite per pixel instead of per tile. + input := job.SourcePath + if len(job.SourcePaths) > 1 { + vrtPath := filepath.Join(logDir, job.Code+".vrt") + if err := s.buildTileJobMosaic(ctx, job, vrtPath, logFile); err != nil { + return logPath, nil, err + } + input = vrtPath + } + + template := strings.TrimSpace(s.cfg.TileJobCommand) + if template == "" { + template = defaultTileJobCommand + } + resolved := tileJobCommandArgs(template, job, input) + if len(resolved) == 0 { + return logPath, nil, errors.New("tile job command template is empty") + } + + command := exec.CommandContext(ctx, resolved[0], resolved[1:]...) + command.Stdout = logFile + command.Stderr = logFile + if err := command.Run(); err != nil { + return logPath, resolved, fmt.Errorf("%s: %w", strings.Join(resolved, " "), err) + } + return logPath, resolved, nil +} + +// buildTileJobMosaic runs gdalbuildvrt over the job's sources. +func (s *Store) buildTileJobMosaic(ctx context.Context, job TileJob, vrtPath string, logFile *os.File) error { + template := strings.TrimSpace(s.cfg.TileJobVRTCommand) + if template == "" { + template = defaultTileJobVRTCommand + } + resolved := tileJobVRTCommandArgs(template, job, vrtPath) + if len(resolved) == 0 { + return errors.New("tile job vrt command template is empty") + } + command := exec.CommandContext(ctx, resolved[0], resolved[1:]...) + command.Stdout = logFile + command.Stderr = logFile + if err := command.Run(); err != nil { + return fmt.Errorf("build mosaic: %s: %w", strings.Join(resolved, " "), err) + } + return nil +} + +// resolveTileJobSources validates every submitted source and returns absolute +// paths. SourcePaths wins over the single SourcePath field. +func resolveTileJobSources(input CreateTileJobInput) ([]string, error) { + raw := input.SourcePaths + if len(raw) == 0 && strings.TrimSpace(input.SourcePath) != "" { + raw = []string{input.SourcePath} + } + if len(raw) == 0 { + return nil, errors.New("source path is required") + } + + seen := map[string]struct{}{} + sources := make([]string, 0, len(raw)) + for _, candidate := range raw { + candidate = strings.TrimSpace(candidate) + if candidate == "" { + continue + } + abs, err := filepath.Abs(candidate) + if err != nil { + return nil, fmt.Errorf("resolve source path %q: %w", candidate, err) + } + info, err := os.Stat(abs) + if err != nil { + return nil, fmt.Errorf("source path %q is not readable: %w", candidate, err) + } + if info.IsDir() { + return nil, fmt.Errorf("source path %q must be a file, not a directory", candidate) + } + if _, dup := seen[abs]; dup { + continue + } + seen[abs] = struct{}{} + sources = append(sources, abs) + } + if len(sources) == 0 { + return nil, errors.New("source path is required") + } + return sources, nil +} + +// tileJobCommandArgs resolves the slicer template. Quoted segments are kept +// together so paths with spaces survive. +func tileJobCommandArgs(template string, job TileJob, input string) []string { + return expandTileJobTemplate(template, map[string]string{ + "{input}": input, + "{output}": job.OutputRoot, + "{zmin}": fmt.Sprintf("%d", job.ZoomMin), + "{zmax}": fmt.Sprintf("%d", job.ZoomMax), + "{tilesize}": fmt.Sprintf("%d", job.TileSize), + "{format}": job.TileFormat, + "{code}": job.TargetCode, + }, nil) +} + +// tileJobVRTCommandArgs resolves the mosaic template. {inputs} expands into one +// argument per source raster. +func tileJobVRTCommandArgs(template string, job TileJob, vrtPath string) []string { + return expandTileJobTemplate(template, map[string]string{ + "{output_vrt}": vrtPath, + "{output}": job.OutputRoot, + "{code}": job.TargetCode, + }, map[string][]string{"{inputs}": job.SourcePaths}) +} + +// expandTileJobTemplate splits the template and substitutes placeholders. A +// token that is exactly a list placeholder expands into several arguments. +func expandTileJobTemplate(template string, scalars map[string]string, lists map[string][]string) []string { + parts := splitCommandLine(template) + args := make([]string, 0, len(parts)) + for _, part := range parts { + if values, ok := lists[part]; ok { + args = append(args, values...) + continue + } + for key, value := range scalars { + part = strings.ReplaceAll(part, key, value) + } + args = append(args, part) + } + return args +} + +// splitCommandLine splits on whitespace while honouring double quotes. +func splitCommandLine(value string) []string { + var parts []string + var current strings.Builder + inQuotes := false + for _, r := range value { + switch { + case r == '"': + inQuotes = !inQuotes + case (r == ' ' || r == '\t' || r == '\n' || r == '\r') && !inQuotes: + if current.Len() > 0 { + parts = append(parts, current.String()) + current.Reset() + } + default: + current.WriteRune(r) + } + } + if current.Len() > 0 { + parts = append(parts, current.String()) + } + return parts +} + +// dispatchTileJobs runs queued jobs one at a time, oldest first. +func (s *Store) dispatchTileJobs(ctx context.Context) { + for { + job, ok, err := s.nextPendingTileJob(ctx) + if err != nil || !ok { + return + } + s.runTileJob(ctx, job) + } +} + +func (s *Store) nextPendingTileJob(ctx context.Context) (TileJob, bool, error) { + row := s.db.QueryRowContext(ctx, "SELECT "+tileJobColumns+` + FROM tile_jobs + WHERE status = ? + ORDER BY created_at + LIMIT 1 + `, tileJobPending) + job, err := scanTileJob(row) + if err != nil { + if errors.Is(err, sql.ErrNoRows) { + return TileJob{}, false, nil + } + return TileJob{}, false, err + } + return job, true, nil +} + +// Callback delivery for tile jobs. A finished job can notify the submitting +// system; failures back off on the same schedule as push callbacks. +const tileCallbackMaxAttempts = 3 + +func (s *Store) queueTileJobCallback(ctx context.Context, jobID string) { + if _, err := s.db.ExecContext(ctx, ` + UPDATE tile_jobs + SET callback_status = ?, callback_attempts = 0, callback_next_attempt_at = ?, callback_message = '' + WHERE id = ? + `, pushStatusRetrying, toRFC3339(nowUTC()), jobID); err != nil { + log.Printf("queue tile job callback %s: %v", jobID, err) + } +} + +func (s *Store) dispatchTileJobCallbacks(ctx context.Context) error { + rows, err := s.db.QueryContext(ctx, "SELECT "+tileJobColumns+` + FROM tile_jobs + WHERE callback_url <> '' AND callback_status = ? + AND (callback_next_attempt_at IS NULL OR callback_next_attempt_at <= ?) + ORDER BY callback_next_attempt_at + LIMIT 5 + `, pushStatusRetrying, toRFC3339(nowUTC())) + if err != nil { + return err + } + defer rows.Close() + + var jobs []TileJob + for rows.Next() { + job, err := scanTileJob(rows) + if err != nil { + return err + } + jobs = append(jobs, job) + } + if err := rows.Err(); err != nil { + return err + } + + for _, job := range jobs { + s.attemptTileJobCallback(job, job.CallbackAttempts+1) + } + return nil +} + +func (s *Store) attemptTileJobCallback(job TileJob, attempt int) { + ctx, cancel := context.WithTimeout(context.Background(), pushCallbackTimeout) + defer cancel() + + payload := map[string]any{ + "event": "tile.job.finished", + "job": job, + "delivered": toRFC3339(nowUTC()), + } + _, _, err := s.postJSON(ctx, http.MethodPost, job.CallbackURL, nil, payload) + message := "" + if err != nil { + message = err.Error() + } + + if err != nil && attempt < tileCallbackMaxAttempts { + next := nowUTC().Add(pushRetryDelay(attempt)) + if _, updateErr := s.db.ExecContext(context.Background(), ` + UPDATE tile_jobs + SET callback_status = ?, callback_attempts = ?, callback_next_attempt_at = ?, callback_message = ? + WHERE id = ? + `, pushStatusRetrying, attempt, toRFC3339(next), message, job.ID); updateErr != nil { + log.Printf("reschedule tile job callback %s: %v", job.ID, updateErr) + } + return + } + + status := pushStatusSucceeded + if err != nil { + status = pushStatusFailed + } + if _, updateErr := s.db.ExecContext(context.Background(), ` + UPDATE tile_jobs + SET callback_status = ?, callback_attempts = ?, callback_next_attempt_at = NULL, callback_message = ? + WHERE id = ? + `, status, attempt, message, job.ID); updateErr != nil { + log.Printf("record tile job callback %s: %v", job.ID, updateErr) + } +} + +// LayerSource records one raster that belongs to a layer. The layer's tile +// directory is the union of its sources, so adding a source and re-slicing +// accumulates imagery instead of replacing the layer. +type LayerSource struct { + ID string `json:"id"` + TargetCode string `json:"target_code"` + SourcePath string `json:"source_path"` + AddedAt time.Time `json:"added_at"` +} + +// AddLayerSources records sources against a layer. Re-adding a known path is a +// no-op, which keeps repeated submissions idempotent. +func (s *Store) AddLayerSources(ctx context.Context, targetCode string, paths []string) error { + code := normalizeBasemapCode(targetCode) + if code == "" { + return errors.New("target code is required") + } + for _, path := range paths { + path = strings.TrimSpace(path) + if path == "" { + continue + } + if _, err := s.db.ExecContext(ctx, ` + INSERT INTO layer_sources (id, target_code, source_path, added_at) + VALUES (?, ?, ?, ?) + ON CONFLICT(target_code, source_path) DO NOTHING + `, uid.New(), code, path, toRFC3339(nowUTC())); err != nil { + return err + } + } + return nil +} + +// ListLayerSources returns a layer's source inventory in the order it was added. +func (s *Store) ListLayerSources(ctx context.Context, targetCode string) ([]LayerSource, error) { + rows, err := s.db.QueryContext(ctx, ` + SELECT id, target_code, source_path, added_at + FROM layer_sources + WHERE target_code = ? + ORDER BY added_at + `, normalizeBasemapCode(targetCode)) + if err != nil { + return nil, err + } + defer rows.Close() + + var items []LayerSource + for rows.Next() { + var item LayerSource + var addedAt string + if err := rows.Scan(&item.ID, &item.TargetCode, &item.SourcePath, &addedAt); err != nil { + return nil, err + } + item.AddedAt = parseTime(addedAt) + items = append(items, item) + } + return items, rows.Err() +} + +// ListLayerInventories returns every layer with its sources, ordered by layer +// code so the console renders stably. +func (s *Store) ListLayerInventories(ctx context.Context) ([]LayerInventory, error) { + rows, err := s.db.QueryContext(ctx, ` + SELECT id, target_code, source_path, added_at + FROM layer_sources + ORDER BY target_code, added_at + `) + if err != nil { + return nil, err + } + defer rows.Close() + + byCode := map[string][]LayerSource{} + var codes []string + for rows.Next() { + var item LayerSource + var addedAt string + if err := rows.Scan(&item.ID, &item.TargetCode, &item.SourcePath, &addedAt); err != nil { + return nil, err + } + item.AddedAt = parseTime(addedAt) + if _, seen := byCode[item.TargetCode]; !seen { + codes = append(codes, item.TargetCode) + } + byCode[item.TargetCode] = append(byCode[item.TargetCode], item) + } + if err := rows.Err(); err != nil { + return nil, err + } + + items := make([]LayerInventory, 0, len(codes)) + for _, code := range codes { + item := LayerInventory{TargetCode: code, Sources: byCode[code], ZoomMin: 0, ZoomMax: 18, TileFormat: "webp"} + if minZoom, maxZoom, format, ok := s.layerPublishedRange(ctx, code); ok { + item.ZoomMin, item.ZoomMax, item.TileFormat = minZoom, maxZoom, format + } + items = append(items, item) + } + return items, nil +} + +// layerPublishedRange returns the zoom range and format the layer is currently +// published at, so a rebuild keeps the same range instead of guessing. +func (s *Store) layerPublishedRange(ctx context.Context, targetCode string) (int, int, string, bool) { + var minZoom, maxZoom int + var format string + err := s.db.QueryRowContext(ctx, ` + SELECT v.min_zoom, v.max_zoom, v.tile_format + FROM basemap_versions v + JOIN basemaps b ON b.id = v.basemap_id + WHERE b.code = ? + ORDER BY v.is_default DESC, v.version + LIMIT 1 + `, normalizeBasemapCode(targetCode)).Scan(&minZoom, &maxZoom, &format) + if err != nil { + return 0, 0, "", false + } + return minZoom, maxZoom, format, true +} diff --git a/services/api-go/internal/basemap/vector_rows.go b/services/api-go/internal/basemap/vector_rows.go index 96e3658..2a51ced 100644 --- a/services/api-go/internal/basemap/vector_rows.go +++ b/services/api-go/internal/basemap/vector_rows.go @@ -65,6 +65,7 @@ func scanVectorPushRecord(scanner interface { var item VectorPushRecord var requestJSON string var responseStatus sql.NullInt64 + var nextAttemptAt sql.NullString var finishedAt sql.NullString var pushedAt string if err := scanner.Scan( @@ -78,6 +79,8 @@ func scanVectorPushRecord(scanner interface { &responseStatus, &item.ResponseBody, &item.ErrorMessage, + &item.AttemptCount, + &nextAttemptAt, &pushedAt, &finishedAt, ); err != nil { @@ -93,6 +96,10 @@ func scanVectorPushRecord(scanner interface { value := int(responseStatus.Int64) item.ResponseStatus = &value } + if nextAttemptAt.Valid { + value := parseTime(nextAttemptAt.String) + item.NextAttemptAt = &value + } item.PushedAt = parseTime(pushedAt) if finishedAt.Valid { value := parseTime(finishedAt.String) diff --git a/services/api-go/internal/basemap/vectors.go b/services/api-go/internal/basemap/vectors.go index d526460..b93e0c1 100644 --- a/services/api-go/internal/basemap/vectors.go +++ b/services/api-go/internal/basemap/vectors.go @@ -18,7 +18,7 @@ import ( "strings" "time" - "map-asset-gateway/api-go/internal/uid" + "tile-server/api-go/internal/uid" ) type VectorDownloadDescriptor struct { @@ -73,6 +73,20 @@ func (s *Store) GetVectorAssetByRef(ctx context.Context, ref string) (VectorAsse if err != nil { return VectorAsset{}, err } + + // The compatibility routes resolve by ref on every request, so try the + // indexed single-row lookups before falling back to a full table scan. + if item, ok, err := s.getVectorAssetByLookup(ctx, "code = ?", normalizeBasemapCode(cleaned)); err != nil { + return VectorAsset{}, err + } else if ok { + return item, nil + } + if item, ok, err := s.getVectorAssetByLookup(ctx, "file_name = ? COLLATE NOCASE", path.Base(cleaned)); err != nil { + return VectorAsset{}, err + } else if ok { + return item, nil + } + items, err := s.ListVectorAssets(ctx) if err != nil { return VectorAsset{}, err @@ -85,16 +99,49 @@ func (s *Store) GetVectorAssetByRef(ctx context.Context, ref string) (VectorAsse return VectorAsset{}, fmt.Errorf("vector asset %q not found", ref) } +// getVectorAssetByLookup runs a single indexed lookup. where must be a trusted +// internal literal, never user input. +func (s *Store) getVectorAssetByLookup(ctx context.Context, where, value string) (VectorAsset, bool, error) { + value = strings.TrimSpace(value) + if value == "" { + return VectorAsset{}, false, nil + } + query := fmt.Sprintf(` + SELECT id, code, name, format, status, description, file_path, file_name, file_size, checksum, download_url, metadata_json, created_at, updated_at + FROM vector_assets + WHERE %s + LIMIT 1 + `, where) + item, err := scanVectorAsset(s.db.QueryRowContext(ctx, query, value)) + if err != nil { + if errors.Is(err, sql.ErrNoRows) { + return VectorAsset{}, false, nil + } + return VectorAsset{}, false, err + } + return item, true, nil +} + func (s *Store) RunVectorScan(ctx context.Context) (map[string]any, error) { root := strings.TrimSpace(s.cfg.VectorScanRoot) if root == "" { return nil, nil } + if !s.scanMu.TryLock() { + return nil, ErrScanInProgress + } + defer s.scanMu.Unlock() + if err := os.MkdirAll(root, 0o755); err != nil { return nil, err } - items, err := scanVectorRoot(root) + hashCache, err := s.loadVectorHashCache(ctx) + if err != nil { + return nil, err + } + + items, err := scanVectorRoot(root, hashCache) if err != nil { return nil, err } @@ -105,32 +152,22 @@ func (s *Store) RunVectorScan(ctx context.Context) (map[string]any, error) { } defer tx.Rollback() - rows, err := tx.QueryContext(ctx, `SELECT code FROM vector_assets`) + existing, err := loadExistingVectors(ctx, tx) if err != nil { return nil, err } - existing := map[string]struct{}{} - for rows.Next() { - var code string - if err := rows.Scan(&code); err != nil { - rows.Close() - return nil, err - } - existing[code] = struct{}{} - } - rows.Close() - if err := rows.Err(); err != nil { - return nil, err - } found := map[string]struct{}{} - added := 0 - updated := 0 + added, updated, unchanged := 0, 0, 0 now := nowUTC() for _, item := range items { found[item.Code] = struct{}{} - if _, ok := existing[item.Code]; ok { - updated++ + if prev, ok := existing[item.Code]; ok { + if s.scannedVectorMatches(item, prev) { + unchanged++ + } else { + updated++ + } } else { added++ } @@ -154,14 +191,117 @@ func (s *Store) RunVectorScan(ctx context.Context) (map[string]any, error) { return nil, err } return map[string]any{ - "scan_root": root, - "scanned_count": len(items), - "added_count": added, - "updated_count": updated, - "removed_count": removed, + "scan_root": root, + "scanned_count": len(items), + "added_count": added, + "updated_count": updated, + "unchanged_count": unchanged, + "removed_count": removed, }, nil } +// existingVector is a snapshot of a stored asset, used to tell a real update +// from a no-op rescan. +type existingVector struct { + Name string + Format string + Status string + Description string + FilePath string + FileName string + FileSize int64 + Checksum string + DownloadURL string + MetadataJSON string +} + +func loadExistingVectors(ctx context.Context, tx *sql.Tx) (map[string]existingVector, error) { + rows, err := tx.QueryContext(ctx, ` + SELECT code, name, format, status, description, file_path, file_name, file_size, checksum, download_url, metadata_json + FROM vector_assets + `) + if err != nil { + return nil, err + } + defer rows.Close() + + items := map[string]existingVector{} + for rows.Next() { + var code string + var item existingVector + if err := rows.Scan( + &code, + &item.Name, + &item.Format, + &item.Status, + &item.Description, + &item.FilePath, + &item.FileName, + &item.FileSize, + &item.Checksum, + &item.DownloadURL, + &item.MetadataJSON, + ); err != nil { + return nil, err + } + items[code] = item + } + return items, rows.Err() +} + +func (s *Store) scannedVectorMatches(item scannedVector, prev existingVector) bool { + return item.Name == prev.Name && + item.Format == prev.Format && + item.Status == prev.Status && + item.Description == prev.Description && + item.FilePath == prev.FilePath && + item.FileName == prev.FileName && + item.FileSize == prev.FileSize && + item.Checksum == prev.Checksum && + s.buildVectorDownloadURL(item.Code) == prev.DownloadURL && + writeJSON(item.Metadata) == prev.MetadataJSON +} + +// vectorHashEntry lets an unchanged file reuse the checksum recorded by the +// previous scan instead of re-reading the whole file on every startup. +type vectorHashEntry struct { + Size int64 + ModTime int64 + Checksum string +} + +func (s *Store) loadVectorHashCache(ctx context.Context) (map[string]vectorHashEntry, error) { + rows, err := s.db.QueryContext(ctx, ` + SELECT file_path, checksum, metadata_json + FROM vector_assets + `) + if err != nil { + return nil, err + } + defer rows.Close() + + cache := map[string]vectorHashEntry{} + for rows.Next() { + var filePath, checksum, metadataJSON string + if err := rows.Scan(&filePath, &checksum, &metadataJSON); err != nil { + return nil, err + } + if strings.TrimSpace(checksum) == "" { + continue + } + var meta struct { + HashSize int64 `json:"hash_size"` + HashMTime int64 `json:"hash_mtime"` + } + _ = readMaybeJSON(metadataJSON, &meta) + if meta.HashSize == 0 && meta.HashMTime == 0 { + continue + } + cache[filePath] = vectorHashEntry{Size: meta.HashSize, ModTime: meta.HashMTime, Checksum: checksum} + } + return cache, rows.Err() +} + type scannedVector struct { Code string Name string @@ -175,7 +315,7 @@ type scannedVector struct { Metadata map[string]any } -func scanVectorRoot(root string) ([]scannedVector, error) { +func scanVectorRoot(root string, hashCache map[string]vectorHashEntry) ([]scannedVector, error) { shapefiles := map[string]string{} var items []scannedVector err := filepath.WalkDir(root, func(path string, entry os.DirEntry, walkErr error) error { @@ -188,7 +328,7 @@ func scanVectorRoot(root string) ([]scannedVector, error) { ext := strings.ToLower(filepath.Ext(entry.Name())) switch ext { case ".geojson", ".json", ".zip": - item, err := buildScannedVector(root, path, ext) + item, err := buildScannedVector(root, path, ext, hashCache) if err != nil { return err } @@ -205,19 +345,51 @@ func scanVectorRoot(root string) ([]scannedVector, error) { for _, base := range sortedKeysStringMap(shapefiles) { path := shapefiles[base] - item, err := buildScannedVector(root, path, ".shp") + item, err := buildScannedVector(root, path, ".shp", hashCache) if err != nil { return nil, err } items = append(items, item) } + items = disambiguateVectorCodes(items) sort.Slice(items, func(i, j int) bool { return items[i].Code < items[j].Code }) return items, nil } +// disambiguateVectorCodes keeps both assets when two different relative paths +// normalize to the same code (for example "a/b.geojson" and "a-b.geojson"). +// The suffix comes from the relative path, so it is stable across scans. +func disambiguateVectorCodes(items []scannedVector) []scannedVector { + byCode := map[string][]int{} + for i, item := range items { + byCode[item.Code] = append(byCode[item.Code], i) + } + for code, indexes := range byCode { + if len(indexes) < 2 { + continue + } + for _, index := range indexes { + sum := sha256.Sum256([]byte(scannedVectorRelPath(items[index]))) + items[index].Code = code + "-" + hex.EncodeToString(sum[:4]) + } + } + return items +} + +func scannedVectorRelPath(item scannedVector) string { + if item.Metadata == nil { + return "" + } + value, ok := item.Metadata["relative_path"].(string) + if !ok { + return "" + } + return value +} + func sortedKeysStringMap(values map[string]string) []string { keys := make([]string, 0, len(values)) for key := range values { @@ -227,7 +399,7 @@ func sortedKeysStringMap(values map[string]string) []string { return keys } -func buildScannedVector(root, path, ext string) (scannedVector, error) { +func buildScannedVector(root, path, ext string, hashCache map[string]vectorHashEntry) (scannedVector, error) { absPath, err := filepath.Abs(path) if err != nil { return scannedVector{}, err @@ -245,10 +417,24 @@ func buildScannedVector(root, path, ext string) (scannedVector, error) { format := vectorFormatFromExt(ext) fileName := filepath.Base(absPath) fileSize := info.Size() - checksum, err := hashVector(absPath, ext) + + hashSize, hashMTime, err := vectorHashStamp(absPath, ext) if err != nil { return scannedVector{}, err } + checksum := "" + if cached, ok := hashCache[absPath]; ok && + (hashSize > 0 || hashMTime > 0) && + cached.Size == hashSize && cached.ModTime == hashMTime { + checksum = cached.Checksum + } + if checksum == "" { + checksum, err = hashVector(absPath, ext) + if err != nil { + return scannedVector{}, err + } + } + if format == "shapefile" { fileName = name + ".zip" } @@ -263,10 +449,37 @@ func buildScannedVector(root, path, ext string) (scannedVector, error) { Checksum: checksum, Metadata: map[string]any{ "relative_path": filepath.ToSlash(rel), + "hash_size": hashSize, + "hash_mtime": hashMTime, }, }, nil } +// vectorHashStamp summarizes the inputs that feed the checksum, so an unchanged +// file can reuse the previous checksum instead of being rehashed every startup. +// Shapefiles fold in every sidecar so a changed .dbf/.shx also invalidates it. +func vectorHashStamp(absPath, ext string) (int64, int64, error) { + if strings.EqualFold(ext, ".shp") { + var totalSize, maxModTime int64 + for _, sibling := range shapefilePackageFiles(absPath) { + info, err := os.Stat(sibling) + if err != nil || info.IsDir() { + continue + } + totalSize += info.Size() + if modTime := info.ModTime().Unix(); modTime > maxModTime { + maxModTime = modTime + } + } + return totalSize, maxModTime, nil + } + info, err := os.Stat(absPath) + if err != nil { + return 0, 0, err + } + return info.Size(), info.ModTime().Unix(), nil +} + func vectorCodeFromRel(rel, ext string) string { rel = filepath.ToSlash(rel) rel = strings.TrimSuffix(rel, strings.ToLower(filepath.Ext(rel))) diff --git a/services/api-go/internal/basemap/workers.go b/services/api-go/internal/basemap/workers.go new file mode 100644 index 0000000..4837e58 --- /dev/null +++ b/services/api-go/internal/basemap/workers.go @@ -0,0 +1,98 @@ +package basemap + +import ( + "context" + "log" + "time" +) + +const ( + // pushRetryInterval is how often due retries are looked for. + pushRetryInterval = 15 * time.Second + // tileJobInterval is how often the queue is checked for new jobs. + tileJobInterval = 2 * time.Second +) + +// StartBackgroundWorkers launches the periodic maintenance loops: resource +// health inspection, push retries and the tile slicing queue. Close stops them. +// Calling it more than once has no additional effect. +func (s *Store) StartBackgroundWorkers() { + s.bgOnce.Do(func() { + ctx, cancel := context.WithCancel(context.Background()) + s.bgCtx = ctx + s.bgCancel = cancel + + s.bgWG.Add(1) + go func() { + defer s.bgWG.Done() + s.healthLoop(ctx) + }() + + s.bgWG.Add(1) + go func() { + defer s.bgWG.Done() + s.pushRetryLoop(ctx) + }() + + s.bgWG.Add(1) + go func() { + defer s.bgWG.Done() + s.tileJobLoop(ctx) + }() + }) +} + +func (s *Store) healthLoop(ctx context.Context) { + // Prime the catalog immediately so a resource that vanished since the last + // scan shows up without waiting a full interval. + if _, err := s.CheckHealth(ctx); err != nil && ctx.Err() == nil { + log.Printf("health check failed: %v", err) + } + interval := s.cfg.HealthCheckInterval + if interval <= 0 { + return + } + ticker := time.NewTicker(interval) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + if _, err := s.CheckHealth(ctx); err != nil && ctx.Err() == nil { + log.Printf("health check failed: %v", err) + } + } + } +} + +func (s *Store) pushRetryLoop(ctx context.Context) { + ticker := time.NewTicker(pushRetryInterval) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + if err := s.dispatchPushRetries(ctx); err != nil && ctx.Err() == nil { + log.Printf("push retry dispatch failed: %v", err) + } + if err := s.dispatchTileJobCallbacks(ctx); err != nil && ctx.Err() == nil { + log.Printf("tile job callback dispatch failed: %v", err) + } + } + } +} + +func (s *Store) tileJobLoop(ctx context.Context) { + ticker := time.NewTicker(tileJobInterval) + defer ticker.Stop() + for { + s.dispatchTileJobs(ctx) + select { + case <-ctx.Done(): + return + case <-ticker.C: + } + } +} diff --git a/services/api-go/internal/basemapapi/admin_template.go b/services/api-go/internal/basemapapi/admin_template.go index bd02ab7..087280d 100644 --- a/services/api-go/internal/basemapapi/admin_template.go +++ b/services/api-go/internal/basemapapi/admin_template.go @@ -3,999 +3,1123 @@ package basemapapi const adminTemplate = ` - - - 底图资源管理平台 - + + +底图网关 · 运维台 + - -
-
-
-
-
Tile Server Admin Console
-

底图资源管理平台

-

统一管理已接入的底图瓦片、GeoJSON 与 Shapefile 资源，负责目录扫描、元数据治理、访问授权以及下游系统推送记录，满足内部服务化管理场景的日常运维需求。

-
本系统仅维护资源台账、访问控制与投递链路，不承担切片生成、矢量转换或重建任务。界面中的编码、路径与 URL 保持技术原值，便于与现有业务系统直接对接。
+ + +
+ + +
+
+

总览

+ {{if .SystemInfo.Env}}{{.SystemInfo.Env}}{{else}}未配置环境{{end}} + 启动 {{fmtTime .SystemInfo.StartedAt}} + + 指标加载中 +
+
+ {{if .Flash}}
{{.Flash}}
{{end}} + {{if .CreatedToken}} +
+

访问令牌已创建

+

明文仅显示这一次，服务端只保存哈希。请立即复制并交付给使用者。

+ {{.CreatedToken}} +
+ +
-
+ {{end}} - - -
-
服务版本{{.SystemInfo.ServiceVersion}}
-
底图资源数{{len .Basemaps}}
-
矢量资源数{{len .VectorAssets}}
-
扫描源数{{len .ScanSources}}
-
目标系统数{{len .TargetSystems}}
-
令牌总数{{len .Tokens}}
-
-
- - {{if .Flash}}
{{.Flash}}
{{end}} - {{if .CreatedToken}}
新建访问令牌
{{.CreatedToken}}
令牌明文仅展示一次，后续系统仅保存哈希前缀，请立即妥善保管。
{{end}} - -
-
-
-
-

底图扫描源

-

登记底图目录与清单文件规则，并按需执行增量扫描。

+
-
-
-
-

矢量资源

-

递归扫描 GeoJSON、Shapefile 及压缩包，并维护可下载资源清单。

+
+

近期扫描

越靠右越新
+
+
+ {{range .ScanRuns}}{{end}} +
+
最近 {{len .ScanRuns}} 次现在
+
+ 成功 + 进行中 + 失败 +
+
- {{len .VectorAssets}} 项
-
- -
-
扫描根目录：{{.SystemInfo.VectorScanRoot}}
-
- -
-
- - - - {{range .VectorAssets}} - - - - - - - {{else}} - - {{end}} - - -
资源格式与状态大小下载地址
- {{.Code}} -
{{.Name}}
-
{{vectorPath .}}
- 		- {{.Format}} -
状态：{{statusText .Status}}
-
{{.FileName}}
- 		{{formatBytes .FileSize}} -
- 查看下载地址 -
-
{{.DownloadURL}}
-
- - -
-
-
-
当前未扫描到矢量资源文件。
-
-
-
-
-
-

目标系统

-

维护下游回调地址与请求头配置，用于资源推送对接。

+
+
+

服务指标

+ 来自 /metrics，进程内计数 + + 查看原始输出
- {{len .TargetSystems}} 个 -
-
- -
-
-
-
-
+
+
-请求总数
+
-4xx 响应
+
-5xx 响应
+
-鉴权失败
+
-瓦片请求
+
-瓦片 404
+
-运行时长
-
-
-
-
-
- - -
- - - - {{range .TargetSystems}} - - - - - - {{else}} - - {{end}} - - -
目标系统回调配置状态
{{.Code}}
{{.Name}}
 -
{{.CallbackMethod}}
-
- 查看回调地址 -
-
{{.CallbackURL}}
-
- - -
-
-
- 		{{enabledText .Enabled}}
暂无下游目标系统配置。
+

指标在服务重启后归零。

-
+
-
-
-
+
-
+ -
-
-
-

访问令牌

-

为下游系统或前端应用分配底图与矢量资源访问权限。

+ +
-
-
-
-

底图推送

-

向下游系统发送底图元数据与瓦片 URL 模板，不直接传输瓦片内容。

-
+
+

已签发

{{len .Tokens}} 枚
+
+ + + + {{range .Tokens}} + + + + + + + + {{else}} + + {{end}} + + +
令牌授权范围状态有效期
+ {{.Name}} +
{{.TokenPrefix}}
+
创建 {{fmtTime .CreatedAt}}
+ 		+
底图：{{tokenGrants .Grants}}
+
矢量：{{vectorTokenGrants .VectorGrants}}
+
能力：{{if .Scopes}}{{range $i, $s := .Scopes}}{{if $i}} · {{end}}{{$s}}{{end}}{{else}}-{{end}}
+ 		{{statusText .Status}} +
到期 {{fmtTimePtr .ExpiresAt}}
+
最近使用 {{fmtTimePtr .LastUsedAt}}
+ 		+
+ +
+
还没有令牌。签发一枚才能对外提供取图地址。
+
-
-
-
-
- -
-
+ -
-
-
-

矢量推送

-

向下游系统发送矢量元数据与受控下载地址，Shapefile 将以压缩包方式提供。

+
-
-
-
-

近期运行记录

-

汇总展示扫描任务、底图推送与矢量推送的最新执行情况，便于审计与排障。

+
+

目标系统

{{len .TargetSystems}} 个
+
+
+
+
+
+
+
+
+
+
每行一条 名字: 值。
+
+
+
+
+ + + + {{range .TargetSystems}} + + + + + + {{else}} + + {{end}} + +
系统回调状态
{{.Code}}
{{.Name}}
 +
{{.CallbackMethod}}
+
{{.CallbackURL}}
+
+ + +
+ 		{{enabledText .Enabled}}
还没有目标系统。
+
-
- + +
+

推送记录

最近 10 条
+
+ + + + {{range .PushRecords}} + + + + + + + + + {{end}} + {{range .VectorPushRecords}} + + + + + + + + + {{end}} + {{if and (eq (len .PushRecords) 0) (eq (len .VectorPushRecords) 0)}} + + {{end}} + +
类型目标资源状态时间
底图{{.TargetSystemCode}}{{.BasemapCode}} / {{.BasemapVersion}} + {{statusText .Status}} + {{if gt .AttemptCount 0}}
已尝试 {{.AttemptCount}} 次
{{end}} + {{if .NextAttemptAt}}
下次 {{fmtTimePtr .NextAttemptAt}}
{{end}} + {{if .ErrorMessage}}
{{.ErrorMessage}}
{{end}} + 		{{fmtTime .PushedAt}} + {{if or (eq .Status "failed") (eq .Status "retrying")}} +
+ {{end}} +
矢量{{.TargetSystemCode}}{{.VectorCode}} + {{statusText .Status}} + {{if gt .AttemptCount 0}}
已尝试 {{.AttemptCount}} 次
{{end}} + {{if .NextAttemptAt}}
下次 {{fmtTimePtr .NextAttemptAt}}
{{end}} + {{if .ErrorMessage}}
{{.ErrorMessage}}
{{end}} + 		{{fmtTime .PushedAt}} + {{if or (eq .Status "failed") (eq .Status "retrying")}} +
+ {{end}} +
还没有推送记录。
+
-
- - - - {{range .ScanRuns}} - - - - - - - {{end}} - {{range .PushRecords}} - - - - - - - {{end}} - {{range .VectorPushRecords}} - - - - - - - {{end}} - {{if and (eq (len .ScanRuns) 0) (eq (len .PushRecords) 0) (eq (len .VectorPushRecords) 0)}} - - {{end}} - - -
记录类型对象状态时间
底图扫描{{.SourceCode}}
新增 {{.AddedCount}} · 更新 {{.UpdatedCount}} · 删除 {{.RemovedCount}}
{{statusText .Status}}{{fmtTime .StartedAt}}
底图推送{{.TargetSystemCode}}
{{.BasemapCode}} / {{.BasemapVersion}}
{{statusText .Status}}{{fmtTime .PushedAt}}
矢量推送{{.TargetSystemCode}}
{{.VectorCode}}
{{statusText .Status}}{{fmtTime .PushedAt}}
暂无运行记录。
+ +
- + +
+

扫描源

{{len .ScanSources}} 个
+
+ + + + {{range .ScanSources}} + + + + + + + {{else}} + + {{end}} + +
来源根目录状态操作
{{.Code}}
{{.Name}}
 +
{{.RootPath}}
+
清单文件：{{.ManifestName}}
+ 		{{enabledText .Enabled}} +
+ +
+
还没有扫描源。
+
+
+ +
+

扫描记录

最近 10 次
+
+
+
+ {{range .ScanRuns}}{{end}} +
+
更早现在
+
+
+ + + + {{range .ScanRuns}} + + + + + + + {{else}} + + {{end}} + +
来源变更状态开始时间
{{.SourceCode}}新增 {{.AddedCount}} · 更新 {{.UpdatedCount}} · 删除 {{.RemovedCount}}{{if index .Summary "unchanged_count"}} · 未变 {{index .Summary "unchanged_count"}}{{end}}{{statusText .Status}}{{if index .Summary "error"}}
{{index .Summary "error"}}
{{end}}		{{fmtTime .StartedAt}}
还没有扫描记录。
+
+
+
+ + + + +
- + var chip = document.getElementById("metrics-chip"); + var chipText = document.getElementById("metrics-chip-text"); + if (chip && chipText) { + chipText.textContent = "瓦片 404 " + (miss === null ? "-" : miss) + " / " + (tiles === null ? "-" : tiles); + chip.classList.add(five > 0 ? "is-bad" : "is-ok"); + } + }) + .catch(function () { + var chipText = document.getElementById("metrics-chip-text"); + if (chipText) chipText.textContent = "指标不可用"; + }); +})(); + ` diff --git a/services/api-go/internal/basemapapi/compat_test.go b/services/api-go/internal/basemapapi/compat_test.go index fcafaf8..7c675e1 100644 --- a/services/api-go/internal/basemapapi/compat_test.go +++ b/services/api-go/internal/basemapapi/compat_test.go @@ -9,7 +9,7 @@ import ( "path/filepath" "testing" - "map-asset-gateway/api-go/internal/basemap" + "tile-server/api-go/internal/basemap" ) func TestCompatibilityRoutes(t *testing.T) { @@ -65,7 +65,7 @@ func TestCompatibilityRoutes(t *testing.T) { } server, err := New(store, basemap.SystemInfo{ - ServiceName: "map-asset-gateway", + ServiceName: "basemap-pusher", ServiceVersion: "test", APIBaseURL: "http://127.0.0.1:8910", TileBaseURL: "http://127.0.0.1:8910", @@ -73,7 +73,7 @@ func TestCompatibilityRoutes(t *testing.T) { SQLitePath: sqlitePath, ScanRoot: tileRoot, VectorScanRoot: vectorRoot, - }) + }, Options{}) if err != nil { t.Fatalf("new server: %v", err) } diff --git a/services/api-go/internal/basemapapi/security_test.go b/services/api-go/internal/basemapapi/security_test.go new file mode 100644 index 0000000..b2904a6 --- /dev/null +++ b/services/api-go/internal/basemapapi/security_test.go @@ -0,0 +1,279 @@ +package basemapapi + +import ( + "context" + "encoding/base64" + "io" + "net/http" + "net/http/httptest" + "net/url" + "os" + "path/filepath" + "strings" + "testing" + + "tile-server/api-go/internal/basemap" +) + +func newTestStore(t *testing.T, root string) *basemap.Store { + t.Helper() + tileRoot := filepath.Join(root, "tiles") + writeFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile") + vectorRoot := filepath.Join(root, "vectors") + + store, err := basemap.OpenStore(basemap.StoreConfig{ + SQLitePath: filepath.Join(root, "basemap.db"), + ScanRoot: tileRoot, + VectorScanRoot: vectorRoot, + APIBaseURL: "http://127.0.0.1:8910", + TileBaseURL: "http://127.0.0.1:8910", + VectorBaseURL: "http://127.0.0.1:8910", + }) + if err != nil { + t.Fatalf("open store: %v", err) + } + t.Cleanup(func() { _ = store.Close() }) + + if _, err := store.RunScan(context.Background(), "local-basemaps"); err != nil { + t.Fatalf("run scan: %v", err) + } + return store +} + +func writeFile(t *testing.T, path, body string) { + t.Helper() + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatalf("mkdir: %v", err) + } + if err := os.WriteFile(path, []byte(body), 0o644); err != nil { + t.Fatalf("write file: %v", err) + } +} + +func newTestServer(t *testing.T, store *basemap.Store, options Options) *httptest.Server { + t.Helper() + server, err := New(store, basemap.SystemInfo{ServiceName: "basemap-pusher"}, options) + if err != nil { + t.Fatalf("new server: %v", err) + } + httpServer := httptest.NewServer(server.Handler()) + t.Cleanup(httpServer.Close) + return httpServer +} + +func basicAuthHeader(user, password string) string { + return "Basic " + base64.StdEncoding.EncodeToString([]byte(user+":"+password)) +} + +func TestAdminRequiresBasicAuth(t *testing.T) { + store := newTestStore(t, t.TempDir()) + httpServer := newTestServer(t, store, Options{Admin: AdminAuth{User: "ops", Password: "s3cret"}}) + + for _, path := range []string{"/api/v1/admin/basemaps", "/admin/"} { + response, err := http.Get(httpServer.URL + path) + if err != nil { + t.Fatalf("unauthorized GET %s: %v", path, err) + } + response.Body.Close() + if response.StatusCode != http.StatusUnauthorized { + t.Fatalf("expected 401 for anonymous %s, got %d", path, response.StatusCode) + } + + request, err := http.NewRequest(http.MethodGet, httpServer.URL+path, nil) + if err != nil { + t.Fatalf("build request: %v", err) + } + request.Header.Set("Authorization", basicAuthHeader("ops", "s3cret")) + response, err = http.DefaultClient.Do(request) + if err != nil { + t.Fatalf("authorized GET %s: %v", path, err) + } + response.Body.Close() + if response.StatusCode != http.StatusOK { + t.Fatalf("expected 200 for authorized %s, got %d", path, response.StatusCode) + } + } +} + +func TestAdminRejectsWrongBasicAuth(t *testing.T) { + store := newTestStore(t, t.TempDir()) + httpServer := newTestServer(t, store, Options{Admin: AdminAuth{User: "ops", Password: "s3cret"}}) + + request, _ := http.NewRequest(http.MethodGet, httpServer.URL+"/api/v1/admin/basemaps", nil) + request.Header.Set("Authorization", basicAuthHeader("ops", "wrong")) + response, err := http.DefaultClient.Do(request) + if err != nil { + t.Fatalf("request: %v", err) + } + defer response.Body.Close() + if response.StatusCode != http.StatusUnauthorized { + t.Fatalf("expected 401, got %d", response.StatusCode) + } +} + +func TestAdminAcceptsBearerToken(t *testing.T) { + store := newTestStore(t, t.TempDir()) + httpServer := newTestServer(t, store, Options{Admin: AdminAuth{Token: "admin-token"}}) + + request, _ := http.NewRequest(http.MethodGet, httpServer.URL+"/api/v1/admin/basemaps", nil) + request.Header.Set("Authorization", "Bearer admin-token") + response, err := http.DefaultClient.Do(request) + if err != nil { + t.Fatalf("request: %v", err) + } + response.Body.Close() + if response.StatusCode != http.StatusOK { + t.Fatalf("expected 200 with bearer token, got %d", response.StatusCode) + } + + request, _ = http.NewRequest(http.MethodGet, httpServer.URL+"/api/v1/admin/basemaps", nil) + request.Header.Set("Authorization", "Bearer nope") + response, err = http.DefaultClient.Do(request) + if err != nil { + t.Fatalf("request: %v", err) + } + response.Body.Close() + if response.StatusCode != http.StatusUnauthorized { + t.Fatalf("expected 401 with a wrong bearer token, got %d", response.StatusCode) + } +} + +func TestAdminOpenWhenUnconfigured(t *testing.T) { + store := newTestStore(t, t.TempDir()) + httpServer := newTestServer(t, store, Options{}) + + response, err := http.Get(httpServer.URL + "/api/v1/admin/basemaps") + if err != nil { + t.Fatalf("request: %v", err) + } + defer response.Body.Close() + if response.StatusCode != http.StatusOK { + t.Fatalf("expected the admin API to be open without credentials, got %d", response.StatusCode) + } +} + +func TestCatalogStillRequiresTokenEvenWithAdminAuth(t *testing.T) { + store := newTestStore(t, t.TempDir()) + httpServer := newTestServer(t, store, Options{Admin: AdminAuth{User: "ops", Password: "s3cret"}}) + + response, err := http.Get(httpServer.URL + "/api/v1/catalog/basemaps") + if err != nil { + t.Fatalf("request: %v", err) + } + defer response.Body.Close() + if response.StatusCode != http.StatusUnauthorized { + t.Fatalf("expected the catalog to still require a service token, got %d", response.StatusCode) + } +} + +func TestCORSAllowlist(t *testing.T) { + store := newTestStore(t, t.TempDir()) + httpServer := newTestServer(t, store, Options{CORSAllowedOrigins: []string{"https://maps.example.com"}}) + + allowed, _ := http.NewRequest(http.MethodGet, httpServer.URL+"/health", nil) + allowed.Header.Set("Origin", "https://maps.example.com") + response, err := http.DefaultClient.Do(allowed) + if err != nil { + t.Fatalf("allowed origin: %v", err) + } + response.Body.Close() + if got := response.Header.Get("Access-Control-Allow-Origin"); got != "https://maps.example.com" { + t.Fatalf("expected the allowlisted origin to be reflected, got %q", got) + } + + blocked, _ := http.NewRequest(http.MethodGet, httpServer.URL+"/health", nil) + blocked.Header.Set("Origin", "https://evil.example.com") + response, err = http.DefaultClient.Do(blocked) + if err != nil { + t.Fatalf("blocked origin: %v", err) + } + response.Body.Close() + if got := response.Header.Get("Access-Control-Allow-Origin"); got != "" { + t.Fatalf("expected no CORS header for an unlisted origin, got %q", got) + } +} + +func TestCreateTokenFormKeepsTokenOutOfURL(t *testing.T) { + store := newTestStore(t, t.TempDir()) + httpServer := newTestServer(t, store, Options{}) + + form := url.Values{} + form.Set("name", "console") + form.Set("basemap_codes", "google_image") + request, err := http.NewRequest(http.MethodPost, httpServer.URL+"/admin/tokens", strings.NewReader(form.Encode())) + if err != nil { + t.Fatalf("build request: %v", err) + } + request.Header.Set("Content-Type", "application/x-www-form-urlencoded") + + client := &http.Client{CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }} + response, err := client.Do(request) + if err != nil { + t.Fatalf("post form: %v", err) + } + defer response.Body.Close() + body, _ := io.ReadAll(response.Body) + + if location := response.Header.Get("Location"); strings.Contains(location, "bmp_") { + t.Fatalf("token leaked into the redirect URL: %s", location) + } + if !strings.Contains(string(body), "bmp_") { + t.Fatalf("expected the new token to be rendered in the response body") + } +} + +func TestMetricsEndpointCountsTileMisses(t *testing.T) { + store := newTestStore(t, t.TempDir()) + httpServer := newTestServer(t, store, Options{}) + + created, err := store.CreateToken(context.Background(), basemap.CreateTokenInput{ + Name: "metrics", + BasemapCodes: []string{"google_image"}, + }) + if err != nil { + t.Fatalf("create token: %v", err) + } + + response, err := http.Get(httpServer.URL + "/tiles/google_image/current/9/9/9.png?token=" + created.Token) + if err != nil { + t.Fatalf("tile request: %v", err) + } + response.Body.Close() + if response.StatusCode != http.StatusNotFound { + t.Fatalf("expected 404 for a missing tile, got %d", response.StatusCode) + } + + response, err = http.Get(httpServer.URL + "/metrics") + if err != nil { + t.Fatalf("metrics request: %v", err) + } + defer response.Body.Close() + body, _ := io.ReadAll(response.Body) + text := string(body) + for _, expected := range []string{ + "basemap_pusher_requests_total", + "basemap_pusher_tile_requests_total 1", + "basemap_pusher_tile_misses_total 1", + } { + if !strings.Contains(text, expected) { + t.Fatalf("expected /metrics to contain %q, got:\n%s", expected, text) + } + } +} + +func TestScanErrorStatusMapping(t *testing.T) { + cases := []struct { + err error + want int + }{ + {basemap.ErrScanInProgress, http.StatusConflict}, + {basemap.ErrScanSourceNotFound, http.StatusNotFound}, + {basemap.ErrScanSourceDisabled, http.StatusBadRequest}, + {io.ErrUnexpectedEOF, http.StatusInternalServerError}, + } + for _, tc := range cases { + if got := scanErrorStatus(tc.err); got != tc.want { + t.Fatalf("scanErrorStatus(%v) = %d, want %d", tc.err, got, tc.want) + } + } +} diff --git a/services/api-go/internal/basemapapi/server.go b/services/api-go/internal/basemapapi/server.go index 2ef2a63..b9a23fb 100644 --- a/services/api-go/internal/basemapapi/server.go +++ b/services/api-go/internal/basemapapi/server.go @@ -1,29 +1,63 @@ package basemapapi import ( - "context" + "crypto/subtle" "encoding/json" "errors" "fmt" "html/template" + "log" "net/http" "net/url" "strconv" "strings" + "sync/atomic" "time" - "map-asset-gateway/api-go/internal/basemap" + "tile-server/api-go/internal/basemap" ) -type Server struct { - store *basemap.Store - system basemap.SystemInfo - startedAt time.Time - template *template.Template - mux *http.ServeMux +// AdminAuth protects the admin console and admin API. When neither credential +// is configured the admin surface is only safe on loopback, which main enforces. +type AdminAuth struct { + Token string + User string + Password string } -func New(store *basemap.Store, system basemap.SystemInfo) (*Server, error) { +func (a AdminAuth) enabled() bool { + return strings.TrimSpace(a.Token) != "" || (strings.TrimSpace(a.User) != "" && a.Password != "") +} + +// Options carries wiring that must not be exposed through the public system +// info payload (credentials, CORS policy). +type Options struct { + Admin AdminAuth + CORSAllowedOrigins []string +} + +// metrics holds cheap in-process counters for the /metrics endpoint. +type metrics struct { + requests atomic.Int64 + status2xx atomic.Int64 + status4xx atomic.Int64 + status5xx atomic.Int64 + tiles atomic.Int64 + tileMisses atomic.Int64 + authFailures atomic.Int64 +} +type Server struct { + store *basemap.Store + system basemap.SystemInfo + startedAt time.Time + template *template.Template + mux *http.ServeMux + admin AdminAuth + corsOrigins []string + metrics *metrics +} + +func New(store *basemap.Store, system basemap.SystemInfo, options Options) (*Server, error) { tmpl, err := template.New("admin").Funcs(template.FuncMap{ "fmtTime": func(value time.Time) string { if value.IsZero() { @@ -106,18 +140,143 @@ func New(store *basemap.Store, system basemap.SystemInfo) (*Server, error) { } server := &Server{ - store: store, - system: system, - startedAt: time.Now().UTC(), - template: tmpl, - mux: http.NewServeMux(), + store: store, + system: system, + startedAt: time.Now().UTC(), + template: tmpl, + mux: http.NewServeMux(), + admin: options.Admin, + corsOrigins: normalizeOrigins(options.CORSAllowedOrigins), + metrics: &metrics{}, } server.routes() return server, nil } func (s *Server) Handler() http.Handler { - return s.mux + return withAccessLog(withCORS(s.mux, s.corsOrigins), s.metrics) +} + +func normalizeOrigins(values []string) []string { + origins := make([]string, 0, len(values)) + for _, value := range values { + value = strings.TrimSpace(value) + if value != "" { + origins = append(origins, value) + } + } + return origins +} + +// withCORS only reflects origins that were explicitly allowed. An empty +// allow-list means no cross-origin access at all, rather than echoing any site. +func withCORS(next http.Handler, allowed []string) http.Handler { + allowAll := false + allowedSet := map[string]struct{}{} + for _, origin := range allowed { + if origin == "*" { + allowAll = true + } + allowedSet[strings.ToLower(origin)] = struct{}{} + } + + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + origin := strings.TrimSpace(r.Header.Get("Origin")) + if origin != "" { + _, ok := allowedSet[strings.ToLower(origin)] + if ok || allowAll { + w.Header().Set("Access-Control-Allow-Origin", origin) + w.Header().Set("Vary", "Origin") + w.Header().Set("Access-Control-Allow-Headers", "Authorization, Content-Type, X-Access-Token") + w.Header().Set("Access-Control-Allow-Methods", "GET, POST, OPTIONS") + } + } + if r.Method == http.MethodOptions { + w.WriteHeader(http.StatusNoContent) + return + } + next.ServeHTTP(w, r) + }) +} + +type statusRecorder struct { + http.ResponseWriter + status int + bytes int +} + +func (r *statusRecorder) WriteHeader(code int) { + if r.status == 0 { + r.status = code + } + r.ResponseWriter.WriteHeader(code) +} + +func (r *statusRecorder) Write(p []byte) (int, error) { + if r.status == 0 { + r.status = http.StatusOK + } + n, err := r.ResponseWriter.Write(p) + r.bytes += n + return n, err +} + +// withAccessLog records one line per request and feeds /metrics. Credentials +// carried in the query string are redacted before logging. +func withAccessLog(next http.Handler, stats *metrics) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + startedAt := time.Now() + recorder := &statusRecorder{ResponseWriter: w} + next.ServeHTTP(recorder, r) + if recorder.status == 0 { + recorder.status = http.StatusOK + } + + stats.requests.Add(1) + switch { + case recorder.status >= 500: + stats.status5xx.Add(1) + case recorder.status >= 400: + stats.status4xx.Add(1) + case recorder.status >= 200: + stats.status2xx.Add(1) + } + if strings.HasPrefix(r.URL.Path, "/tiles/") { + stats.tiles.Add(1) + if recorder.status == http.StatusNotFound { + stats.tileMisses.Add(1) + } + } + if recorder.status == http.StatusUnauthorized { + stats.authFailures.Add(1) + } + + if r.URL.Path == "/health" || r.URL.Path == "/ready" || r.URL.Path == "/metrics" { + return + } + log.Printf("%s %s %d %dB %s", r.Method, redactRequestURI(r), recorder.status, recorder.bytes, time.Since(startedAt).Round(time.Millisecond)) + }) +} + +func redactRequestURI(r *http.Request) string { + if r.URL.RawQuery == "" { + return r.URL.Path + } + redacted := url.Values{} + for key, values := range r.URL.Query() { + switch strings.ToLower(key) { + case "token", "access_token": + redacted.Set(key, "***") + default: + for _, value := range values { + redacted.Add(key, value) + } + } + } + if encoded := redacted.Encode(); encoded != "" { + return r.URL.Path + "?" + encoded + } + return r.URL.Path } func (s *Server) routes() { @@ -126,25 +285,38 @@ func (s *Server) routes() { }) s.mux.HandleFunc("GET /health", s.handleHealth) s.mux.HandleFunc("GET /ready", s.handleReady) + s.mux.HandleFunc("GET /metrics", s.handleMetrics) s.mux.HandleFunc("GET /api/v1/system", s.handleSystem) - s.mux.HandleFunc("GET /api/v1/admin/dashboard", s.handleDashboardJSON) - s.mux.HandleFunc("GET /api/v1/admin/basemaps", s.handleBasemapsJSON) - s.mux.HandleFunc("GET /api/v1/admin/vectors", s.handleVectorsJSON) - s.mux.HandleFunc("GET /api/v1/admin/scan-sources", s.handleScanSourcesJSON) - s.mux.HandleFunc("POST /api/v1/admin/scan-sources", s.handleCreateScanSourceJSON) - s.mux.HandleFunc("POST /api/v1/admin/scan-sources/{code}/run", s.handleRunScanJSON) - s.mux.HandleFunc("POST /api/v1/admin/vectors/rescan", s.handleRunVectorScanJSON) - s.mux.HandleFunc("POST /api/v1/admin/basemaps/{code}/default-version", s.handleSetDefaultVersionJSON) - s.mux.HandleFunc("GET /api/v1/admin/targets", s.handleTargetsJSON) - s.mux.HandleFunc("POST /api/v1/admin/targets", s.handleCreateTargetJSON) - s.mux.HandleFunc("GET /api/v1/admin/tokens", s.handleTokensJSON) - s.mux.HandleFunc("POST /api/v1/admin/tokens", s.handleCreateTokenJSON) - s.mux.HandleFunc("POST /api/v1/admin/tokens/{id}/disable", s.handleDisableTokenJSON) - s.mux.HandleFunc("GET /api/v1/admin/push-records", s.handlePushRecordsJSON) - s.mux.HandleFunc("POST /api/v1/admin/pushes", s.handlePushJSON) - s.mux.HandleFunc("GET /api/v1/admin/vector-push-records", s.handleVectorPushRecordsJSON) - s.mux.HandleFunc("POST /api/v1/admin/vector-pushes", s.handleVectorPushJSON) + // Creating a token here grants read access to every basemap, so the whole + // admin surface sits behind the admin credentials. + s.mux.HandleFunc("GET /api/v1/admin/dashboard", s.requireAdmin(s.handleDashboardJSON)) + s.mux.HandleFunc("GET /api/v1/admin/basemaps", s.requireAdmin(s.handleBasemapsJSON)) + s.mux.HandleFunc("GET /api/v1/admin/vectors", s.requireAdmin(s.handleVectorsJSON)) + s.mux.HandleFunc("GET /api/v1/admin/scan-sources", s.requireAdmin(s.handleScanSourcesJSON)) + s.mux.HandleFunc("POST /api/v1/admin/scan-sources", s.requireAdmin(s.handleCreateScanSourceJSON)) + s.mux.HandleFunc("POST /api/v1/admin/scan-sources/{code}/run", s.requireAdmin(s.handleRunScanJSON)) + s.mux.HandleFunc("POST /api/v1/admin/vectors/rescan", s.requireAdmin(s.handleRunVectorScanJSON)) + s.mux.HandleFunc("POST /api/v1/admin/basemaps/{code}/default-version", s.requireAdmin(s.handleSetDefaultVersionJSON)) + s.mux.HandleFunc("GET /api/v1/admin/targets", s.requireAdmin(s.handleTargetsJSON)) + s.mux.HandleFunc("POST /api/v1/admin/targets", s.requireAdmin(s.handleCreateTargetJSON)) + s.mux.HandleFunc("GET /api/v1/admin/tokens", s.requireAdmin(s.handleTokensJSON)) + s.mux.HandleFunc("POST /api/v1/admin/tokens", s.requireAdmin(s.handleCreateTokenJSON)) + s.mux.HandleFunc("POST /api/v1/admin/tokens/{id}/disable", s.requireAdmin(s.handleDisableTokenJSON)) + s.mux.HandleFunc("GET /api/v1/admin/push-records", s.requireAdmin(s.handlePushRecordsJSON)) + s.mux.HandleFunc("POST /api/v1/admin/pushes", s.requireAdmin(s.handlePushJSON)) + s.mux.HandleFunc("GET /api/v1/admin/vector-push-records", s.requireAdmin(s.handleVectorPushRecordsJSON)) + s.mux.HandleFunc("POST /api/v1/admin/vector-pushes", s.requireAdmin(s.handleVectorPushJSON)) + s.mux.HandleFunc("POST /api/v1/admin/health/check", s.requireAdmin(s.handleCheckHealthJSON)) + s.mux.HandleFunc("GET /api/v1/admin/tile-jobs", s.requireAdmin(s.handleTileJobsJSON)) + s.mux.HandleFunc("POST /api/v1/admin/tile-jobs", s.requireAdmin(s.handleCreateTileJobJSON)) + s.mux.HandleFunc("POST /api/v1/admin/push-records/{id}/retry", s.requireAdmin(s.handleRetryPushJSON)) + s.mux.HandleFunc("POST /api/v1/admin/vector-push-records/{id}/retry", s.requireAdmin(s.handleRetryVectorPushJSON)) + + // Slicing for external callers: a token with the "slice" scope, not admin. + s.mux.HandleFunc("POST /api/v1/slice/jobs", s.handleSliceJobCreate) + s.mux.HandleFunc("GET /api/v1/slice/jobs/{id}", s.handleSliceJobGet) + s.mux.HandleFunc("GET /api/v1/slice/layers/{code}/sources", s.handleSliceLayerSources) s.mux.HandleFunc("GET /api/v1/catalog/basemaps", s.handleCatalogList) s.mux.HandleFunc("GET /api/v1/catalog/basemaps/{code}", s.handleCatalogGet) @@ -155,16 +327,78 @@ func (s *Server) routes() { s.mux.HandleFunc("GET /shp/{assetPath...}", s.handleShapeByPath) s.mux.HandleFunc("GET /vectors/{code}/download", s.handleVectorDownload) - s.mux.HandleFunc("GET /admin/", s.handleAdminPage) - s.mux.HandleFunc("POST /admin/scan-sources", s.handleCreateScanSourceForm) - s.mux.HandleFunc("POST /admin/scan-sources/{code}/run", s.handleRunScanForm) - s.mux.HandleFunc("POST /admin/vectors/rescan", s.handleRunVectorScanForm) - s.mux.HandleFunc("POST /admin/basemaps/{code}/default", s.handleSetDefaultVersionForm) - s.mux.HandleFunc("POST /admin/targets", s.handleCreateTargetForm) - s.mux.HandleFunc("POST /admin/tokens", s.handleCreateTokenForm) - s.mux.HandleFunc("POST /admin/tokens/{id}/disable", s.handleDisableTokenForm) - s.mux.HandleFunc("POST /admin/pushes", s.handlePushForm) - s.mux.HandleFunc("POST /admin/vector-pushes", s.handleVectorPushForm) + s.mux.HandleFunc("GET /admin/", s.requireAdmin(s.handleAdminPage)) + s.mux.HandleFunc("POST /admin/scan-sources", s.requireAdmin(s.handleCreateScanSourceForm)) + s.mux.HandleFunc("POST /admin/scan-sources/{code}/run", s.requireAdmin(s.handleRunScanForm)) + s.mux.HandleFunc("POST /admin/vectors/rescan", s.requireAdmin(s.handleRunVectorScanForm)) + s.mux.HandleFunc("POST /admin/basemaps/{code}/default", s.requireAdmin(s.handleSetDefaultVersionForm)) + s.mux.HandleFunc("POST /admin/targets", s.requireAdmin(s.handleCreateTargetForm)) + s.mux.HandleFunc("POST /admin/tokens", s.requireAdmin(s.handleCreateTokenForm)) + s.mux.HandleFunc("POST /admin/tokens/{id}/disable", s.requireAdmin(s.handleDisableTokenForm)) + s.mux.HandleFunc("POST /admin/pushes", s.requireAdmin(s.handlePushForm)) + s.mux.HandleFunc("POST /admin/vector-pushes", s.requireAdmin(s.handleVectorPushForm)) + s.mux.HandleFunc("POST /admin/health/check", s.requireAdmin(s.handleCheckHealthForm)) + s.mux.HandleFunc("POST /admin/tile-jobs", s.requireAdmin(s.handleCreateTileJobForm)) + s.mux.HandleFunc("POST /admin/tile-jobs/rebuild", s.requireAdmin(s.handleRebuildLayerForm)) + s.mux.HandleFunc("POST /admin/push-records/{id}/retry", s.requireAdmin(s.handleRetryPushForm)) + s.mux.HandleFunc("POST /admin/vector-push-records/{id}/retry", s.requireAdmin(s.handleRetryVectorPushForm)) +} + +// requireAdmin guards the admin surface. With no credentials configured the +// service is only reachable on loopback, which main enforces at startup. +func (s *Server) requireAdmin(next http.HandlerFunc) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + if !s.admin.enabled() { + next(w, r) + return + } + if s.adminAllows(r) { + next(w, r) + return + } + w.Header().Set("WWW-Authenticate", `Basic realm="basemap-pusher admin"`) + http.Error(w, "admin credentials required", http.StatusUnauthorized) + } +} + +func (s *Server) adminAllows(r *http.Request) bool { + if token := strings.TrimSpace(s.admin.Token); token != "" { + header := strings.TrimSpace(r.Header.Get("Authorization")) + if len(header) > 7 && strings.EqualFold(header[:7], "bearer ") { + if subtle.ConstantTimeCompare([]byte(strings.TrimSpace(header[7:])), []byte(token)) == 1 { + return true + } + } + } + user := strings.TrimSpace(s.admin.User) + if user == "" || s.admin.Password == "" { + return false + } + givenUser, givenPassword, ok := r.BasicAuth() + if !ok { + return false + } + userOK := subtle.ConstantTimeCompare([]byte(givenUser), []byte(user)) == 1 + passwordOK := subtle.ConstantTimeCompare([]byte(givenPassword), []byte(s.admin.Password)) == 1 + return userOK && passwordOK +} + +func (s *Server) handleMetrics(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "text/plain; version=0.0.4; charset=utf-8") + fmt.Fprintf(w, "# TYPE basemap_pusher_requests_total counter\n") + fmt.Fprintf(w, "basemap_pusher_requests_total %d\n", s.metrics.requests.Load()) + fmt.Fprintf(w, "# TYPE basemap_pusher_requests_by_status_total counter\n") + fmt.Fprintf(w, "basemap_pusher_requests_by_status_total{class=\"2xx\"} %d\n", s.metrics.status2xx.Load()) + fmt.Fprintf(w, "basemap_pusher_requests_by_status_total{class=\"4xx\"} %d\n", s.metrics.status4xx.Load()) + fmt.Fprintf(w, "basemap_pusher_requests_by_status_total{class=\"5xx\"} %d\n", s.metrics.status5xx.Load()) + fmt.Fprintf(w, "# TYPE basemap_pusher_tile_requests_total counter\n") + fmt.Fprintf(w, "basemap_pusher_tile_requests_total %d\n", s.metrics.tiles.Load()) + fmt.Fprintf(w, "# TYPE basemap_pusher_tile_misses_total counter\n") + fmt.Fprintf(w, "basemap_pusher_tile_misses_total %d\n", s.metrics.tileMisses.Load()) + fmt.Fprintf(w, "# TYPE basemap_pusher_auth_failures_total counter\n") + fmt.Fprintf(w, "basemap_pusher_auth_failures_total %d\n", s.metrics.authFailures.Load()) + fmt.Fprintf(w, "# TYPE basemap_pusher_uptime_seconds gauge\n") + fmt.Fprintf(w, "basemap_pusher_uptime_seconds %.0f\n", time.Since(s.startedAt).Seconds()) } func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) { @@ -241,7 +475,7 @@ func (s *Server) handleCreateScanSourceJSON(w http.ResponseWriter, r *http.Reque func (s *Server) handleRunScanJSON(w http.ResponseWriter, r *http.Request) { run, err := s.store.RunScan(r.Context(), r.PathValue("code")) if err != nil { - writeError(w, http.StatusBadRequest, err) + writeError(w, scanErrorStatus(err), err) return } writeJSON(w, http.StatusOK, run) @@ -250,7 +484,7 @@ func (s *Server) handleRunScanJSON(w http.ResponseWriter, r *http.Request) { func (s *Server) handleRunVectorScanJSON(w http.ResponseWriter, r *http.Request) { result, err := s.store.RunVectorScan(r.Context()) if err != nil { - writeError(w, http.StatusBadRequest, err) + writeError(w, scanErrorStatus(err), err) return } writeJSON(w, http.StatusOK, result) @@ -358,7 +592,7 @@ func (s *Server) handlePushJSON(w http.ResponseWriter, r *http.Request) { writeError(w, http.StatusBadRequest, err) return } - writeJSON(w, http.StatusCreated, item) + writeJSON(w, http.StatusAccepted, item) } func (s *Server) handleVectorPushJSON(w http.ResponseWriter, r *http.Request) { @@ -375,7 +609,7 @@ func (s *Server) handleVectorPushJSON(w http.ResponseWriter, r *http.Request) { writeError(w, http.StatusBadRequest, err) return } - writeJSON(w, http.StatusCreated, item) + writeJSON(w, http.StatusAccepted, item) } func (s *Server) handleCatalogList(w http.ResponseWriter, r *http.Request) { @@ -456,6 +690,7 @@ func (s *Server) handleTile(w http.ResponseWriter, r *http.Request) { return } w.Header().Set("Content-Type", item.ContentType) + w.Header().Set("Cache-Control", "public, max-age=31536000, immutable") http.ServeFile(w, r, item.FilePath) } @@ -515,6 +750,203 @@ func (s *Server) handleShapeByPath(w http.ResponseWriter, r *http.Request) { http.ServeFile(w, r, item.FilePath) } +func (s *Server) handleCheckHealthJSON(w http.ResponseWriter, r *http.Request) { + report, err := s.store.CheckHealth(r.Context()) + if err != nil { + writeError(w, http.StatusInternalServerError, err) + return + } + writeJSON(w, http.StatusOK, report) +} + +func (s *Server) handleTileJobsJSON(w http.ResponseWriter, r *http.Request) { + items, err := s.store.ListTileJobs(r.Context(), 50) + if err != nil { + writeError(w, http.StatusInternalServerError, err) + return + } + writeJSON(w, http.StatusOK, items) +} + +func (s *Server) handleCreateTileJobJSON(w http.ResponseWriter, r *http.Request) { + var input basemap.CreateTileJobInput + if err := json.NewDecoder(r.Body).Decode(&input); err != nil { + writeError(w, http.StatusBadRequest, err) + return + } + job, err := s.store.CreateTileJob(r.Context(), input) + if err != nil { + writeError(w, http.StatusBadRequest, err) + return + } + writeJSON(w, http.StatusAccepted, job) +} + +func (s *Server) handleRetryPushJSON(w http.ResponseWriter, r *http.Request) { + if err := s.store.RetryPush(r.Context(), "basemap", r.PathValue("id")); err != nil { + writeError(w, http.StatusBadRequest, err) + return + } + writeJSON(w, http.StatusAccepted, map[string]string{"status": "queued"}) +} + +func (s *Server) handleRetryVectorPushJSON(w http.ResponseWriter, r *http.Request) { + if err := s.store.RetryPush(r.Context(), "vector", r.PathValue("id")); err != nil { + writeError(w, http.StatusBadRequest, err) + return + } + writeJSON(w, http.StatusAccepted, map[string]string{"status": "queued"}) +} + +func (s *Server) handleCheckHealthForm(w http.ResponseWriter, r *http.Request) { + report, err := s.store.CheckHealth(r.Context()) + flash := "资源巡检已完成" + if err == nil { + flash = fmt.Sprintf("资源巡检完成：底图缺 %d / %d，矢量缺 %d / %d", + report.VersionsMissing, report.VersionsChecked, report.VectorsMissing, report.VectorsChecked) + } + s.redirectAdmin(w, r, flash, err) +} + +func (s *Server) handleCreateTileJobForm(w http.ResponseWriter, r *http.Request) { + if err := r.ParseForm(); err != nil { + s.redirectAdmin(w, r, "", err) + return + } + zoomMin, _ := strconv.Atoi(strings.TrimSpace(r.FormValue("zoom_min"))) + zoomMax, _ := strconv.Atoi(strings.TrimSpace(r.FormValue("zoom_max"))) + tileSize, _ := strconv.Atoi(strings.TrimSpace(r.FormValue("tile_size"))) + _, err := s.store.CreateTileJob(r.Context(), basemap.CreateTileJobInput{ + Code: r.FormValue("code"), + Name: r.FormValue("name"), + SourcePath: r.FormValue("source_path"), + TargetCode: r.FormValue("target_code"), + ZoomMin: zoomMin, + ZoomMax: zoomMax, + TileSize: tileSize, + TileFormat: r.FormValue("tile_format"), + CallbackURL: r.FormValue("callback_url"), + RequestedBy: r.FormValue("requested_by"), + }) + s.redirectAdmin(w, r, "切片任务已提交，刷新查看进度", err) +} + +// handleRebuildLayerForm slices every source of a layer into one VRT mosaic, +// reusing the zoom range the layer is already published at. +func (s *Server) handleRebuildLayerForm(w http.ResponseWriter, r *http.Request) { + if err := r.ParseForm(); err != nil { + s.redirectAdmin(w, r, "", err) + return + } + targetCode := strings.TrimSpace(r.FormValue("target_code")) + if targetCode == "" { + s.redirectAdmin(w, r, "", errors.New("target code is required")) + return + } + + sources, err := s.store.ListLayerSources(r.Context(), targetCode) + if err != nil { + s.redirectAdmin(w, r, "", err) + return + } + if len(sources) == 0 { + s.redirectAdmin(w, r, "", fmt.Errorf("layer %q has no recorded sources", targetCode)) + return + } + + zoomMin := 0 + zoomMax := 18 + tileFormat := "webp" + if items, err := s.store.ListLayerInventories(r.Context()); err == nil { + for _, item := range items { + if item.TargetCode == normalizeLayerCode(targetCode) { + zoomMin, zoomMax, tileFormat = item.ZoomMin, item.ZoomMax, item.TileFormat + break + } + } + } + + paths := make([]string, 0, len(sources)) + for _, source := range sources { + paths = append(paths, source.SourcePath) + } + if _, err := s.store.CreateTileJob(r.Context(), basemap.CreateTileJobInput{ + Name: "重建 " + targetCode, + TargetCode: targetCode, + SourcePaths: paths, + ZoomMin: zoomMin, + ZoomMax: zoomMax, + TileFormat: tileFormat, + RequestedBy: "console", + }); err != nil { + s.redirectAdmin(w, r, "", err) + return + } + s.redirectAdmin(w, r, fmt.Sprintf("已提交 %s 的镶嵌重建（%d 个源，级别 %d-%d）", targetCode, len(paths), zoomMin, zoomMax), nil) +} +func (s *Server) handleRetryPushForm(w http.ResponseWriter, r *http.Request) { + err := s.store.RetryPush(r.Context(), "basemap", r.PathValue("id")) + s.redirectAdmin(w, r, "已重新排队推送", err) +} + +func (s *Server) handleRetryVectorPushForm(w http.ResponseWriter, r *http.Request) { + err := s.store.RetryPush(r.Context(), "vector", r.PathValue("id")) + s.redirectAdmin(w, r, "已重新排队推送", err) +} + +// requireSliceToken authenticates and then checks the coarse slice capability, +// so an integration can submit jobs without holding the admin credential. +func (s *Server) requireSliceToken(w http.ResponseWriter, r *http.Request) (basemap.TokenAuth, bool) { + auth, ok := s.requireToken(w, r) + if !ok { + return basemap.TokenAuth{}, false + } + if !auth.CanSlice() { + writeError(w, http.StatusForbidden, errors.New("token does not carry the slice scope")) + return basemap.TokenAuth{}, false + } + return auth, true +} + +func (s *Server) handleSliceJobCreate(w http.ResponseWriter, r *http.Request) { + if _, ok := s.requireSliceToken(w, r); !ok { + return + } + var input basemap.CreateTileJobInput + if err := json.NewDecoder(r.Body).Decode(&input); err != nil { + writeError(w, http.StatusBadRequest, err) + return + } + job, err := s.store.CreateTileJob(r.Context(), input) + if err != nil { + writeError(w, http.StatusBadRequest, err) + return + } + writeJSON(w, http.StatusAccepted, job) +} + +func (s *Server) handleSliceLayerSources(w http.ResponseWriter, r *http.Request) { + if _, ok := s.requireSliceToken(w, r); !ok { + return + } + items, err := s.store.ListLayerSources(r.Context(), r.PathValue("code")) + if err != nil { + writeError(w, http.StatusInternalServerError, err) + return + } + writeJSON(w, http.StatusOK, items) +} +func (s *Server) handleSliceJobGet(w http.ResponseWriter, r *http.Request) { + if _, ok := s.requireSliceToken(w, r); !ok { + return + } + job, err := s.store.GetTileJob(r.Context(), r.PathValue("id")) + if err != nil { + writeError(w, http.StatusNotFound, err) + return + } + writeJSON(w, http.StatusOK, job) +} func (s *Server) handleAdminPage(w http.ResponseWriter, r *http.Request) { data, err := s.store.Dashboard(r.Context(), s.system, r.URL.Query().Get("flash"), r.URL.Query().Get("token")) if err != nil { @@ -529,7 +961,7 @@ func (s *Server) handleAdminPage(w http.ResponseWriter, r *http.Request) { func (s *Server) handleCreateScanSourceForm(w http.ResponseWriter, r *http.Request) { if err := r.ParseForm(); err != nil { - s.redirectAdmin(w, r, "", "", err) + s.redirectAdmin(w, r, "", err) return } _, err := s.store.EnsureScanSource(r.Context(), basemap.CreateScanSourceInput{ @@ -538,31 +970,31 @@ func (s *Server) handleCreateScanSourceForm(w http.ResponseWriter, r *http.Reque RootPath: r.FormValue("root_path"), ManifestName: r.FormValue("manifest_name"), }) - s.redirectAdmin(w, r, "扫描源已保存", "", err) + s.redirectAdmin(w, r, "扫描源已保存", err) } func (s *Server) handleRunScanForm(w http.ResponseWriter, r *http.Request) { _, err := s.store.RunScan(r.Context(), r.PathValue("code")) - s.redirectAdmin(w, r, "底图扫描已完成", "", err) + s.redirectAdmin(w, r, "底图扫描已完成", err) } func (s *Server) handleRunVectorScanForm(w http.ResponseWriter, r *http.Request) { _, err := s.store.RunVectorScan(r.Context()) - s.redirectAdmin(w, r, "矢量扫描已完成", "", err) + s.redirectAdmin(w, r, "矢量扫描已完成", err) } func (s *Server) handleSetDefaultVersionForm(w http.ResponseWriter, r *http.Request) { if err := r.ParseForm(); err != nil { - s.redirectAdmin(w, r, "", "", err) + s.redirectAdmin(w, r, "", err) return } err := s.store.SetDefaultVersion(r.Context(), r.PathValue("code"), r.FormValue("version")) - s.redirectAdmin(w, r, "默认版本已更新", "", err) + s.redirectAdmin(w, r, "默认版本已更新", err) } func (s *Server) handleCreateTargetForm(w http.ResponseWriter, r *http.Request) { if err := r.ParseForm(); err != nil { - s.redirectAdmin(w, r, "", "", err) + s.redirectAdmin(w, r, "", err) return } headers := map[string]string{} @@ -580,44 +1012,44 @@ func (s *Server) handleCreateTargetForm(w http.ResponseWriter, r *http.Request) CallbackMethod: r.FormValue("callback_method"), CallbackHeaders: headers, }) - s.redirectAdmin(w, r, "目标系统已保存", "", err) + s.redirectAdmin(w, r, "目标系统已保存", err) } func (s *Server) handleCreateTokenForm(w http.ResponseWriter, r *http.Request) { input, err := decodeTokenInputFromForm(r) if err != nil { - s.redirectAdmin(w, r, "", "", err) + s.redirectAdmin(w, r, "", err) return } item, err := s.store.CreateToken(r.Context(), input) if err != nil { - s.redirectAdmin(w, r, "", "", err) + s.redirectAdmin(w, r, "", err) return } - s.redirectAdmin(w, r, "访问令牌已创建", item.Token, nil) + s.renderAdmin(w, r, "访问令牌已创建", item.Token, nil) } func (s *Server) handleDisableTokenForm(w http.ResponseWriter, r *http.Request) { err := s.store.DisableToken(r.Context(), r.PathValue("id")) - s.redirectAdmin(w, r, "访问令牌已停用", "", err) + s.redirectAdmin(w, r, "访问令牌已停用", err) } func (s *Server) handlePushForm(w http.ResponseWriter, r *http.Request) { if err := r.ParseForm(); err != nil { - s.redirectAdmin(w, r, "", "", err) + s.redirectAdmin(w, r, "", err) return } _, err := s.store.PushBasemapVersion(r.Context(), r.FormValue("target_code"), r.FormValue("basemap_code"), r.FormValue("version")) - s.redirectAdmin(w, r, "底图推送已完成", "", err) + s.redirectAdmin(w, r, "底图推送已提交，稍后刷新查看结果", err) } func (s *Server) handleVectorPushForm(w http.ResponseWriter, r *http.Request) { if err := r.ParseForm(); err != nil { - s.redirectAdmin(w, r, "", "", err) + s.redirectAdmin(w, r, "", err) return } _, err := s.store.PushVectorAsset(r.Context(), r.FormValue("target_code"), r.FormValue("vector_code")) - s.redirectAdmin(w, r, "矢量推送已完成", "", err) + s.redirectAdmin(w, r, "矢量推送已提交，稍后刷新查看结果", err) } func (s *Server) requireToken(w http.ResponseWriter, r *http.Request) (basemap.TokenAuth, bool) { @@ -652,6 +1084,7 @@ func decodeTokenInput(r *http.Request) (basemap.CreateTokenInput, error) { Name string `json:"name"` BasemapCodes []string `json:"basemap_codes"` VectorCodes []string `json:"vector_codes"` + Scopes []string `json:"scopes"` ExpiresAt string `json:"expires_at"` } if err := json.NewDecoder(r.Body).Decode(&payload); err != nil { @@ -661,6 +1094,7 @@ func decodeTokenInput(r *http.Request) (basemap.CreateTokenInput, error) { Name: payload.Name, BasemapCodes: payload.BasemapCodes, VectorCodes: payload.VectorCodes, + Scopes: payload.Scopes, } if strings.TrimSpace(payload.ExpiresAt) != "" { parsed, err := time.Parse(time.RFC3339, payload.ExpiresAt) @@ -680,6 +1114,7 @@ func decodeTokenInputFromForm(r *http.Request) (basemap.CreateTokenInput, error) Name: r.FormValue("name"), BasemapCodes: splitCSV(r.FormValue("basemap_codes")), VectorCodes: splitCSV(r.FormValue("vector_codes")), + Scopes: splitCSV(r.FormValue("scopes")), } if value := strings.TrimSpace(r.FormValue("expires_at")); value != "" { if minutes, err := strconv.Atoi(value); err == nil && minutes > 0 { @@ -744,6 +1179,8 @@ func statusText(value string) string { return "失败" case "pending": return "待处理" + case "retrying": + return "重试中" case "running": return "运行中" case "processing": @@ -761,7 +1198,7 @@ func statusTone(value string) string { switch strings.ToLower(strings.TrimSpace(value)) { case "active", "enabled", "ready", "succeeded", "current": return "success" - case "pending", "running", "processing": + case "pending", "running", "processing", "retrying": return "warning" case "disabled", "failed", "error", "expired", "invalid": return "danger" @@ -787,6 +1224,25 @@ func formatBytes(size int64) string { return fmt.Sprintf("%.1f %s", value, units[unit]) } +// scanErrorStatus maps scan failures onto HTTP status codes: a busy scanner is +// a conflict, an unknown source is a 404, and anything else is a server fault. +// normalizeLayerCode mirrors the store's basemap code normalization so the +// console can match a layer against its inventory entry. +func normalizeLayerCode(value string) string { + return strings.ToLower(strings.TrimSpace(value)) +} +func scanErrorStatus(err error) int { + switch { + case errors.Is(err, basemap.ErrScanInProgress): + return http.StatusConflict + case errors.Is(err, basemap.ErrScanSourceNotFound): + return http.StatusNotFound + case errors.Is(err, basemap.ErrScanSourceDisabled): + return http.StatusBadRequest + default: + return http.StatusInternalServerError + } +} func localizeAdminError(err error) string { if err == nil { return "" @@ -819,8 +1275,8 @@ func localizeAdminError(err error) string { return fmt.Sprintf("目标系统已停用：%s", quoted[0]) case msg == "token name is required": return "令牌名称不能为空" - case msg == "at least one basemap code or vector code is required": - return "至少填写一个底图编码或矢量资源编码/路径" + case msg == "at least one basemap code, vector code, or the slice scope is required": + return "至少填写一个底图编码、矢量资源，或勾选切片能力" case msg == "no valid basemap codes provided": return "未提供有效的底图编码" case len(quoted) == 1 && strings.HasPrefix(msg, `token "`) && strings.HasSuffix(msg, `" not found`): @@ -849,24 +1305,38 @@ func quotedSegments(value string) []string { return result } -func (s *Server) redirectAdmin(w http.ResponseWriter, r *http.Request, flash string, token string, err error) { +// redirectAdmin sends the browser back to the console with a flash message. It +// deliberately never carries a secret in the URL. +func (s *Server) redirectAdmin(w http.ResponseWriter, r *http.Request, flash string, err error) { if err != nil { flash = "操作失败：" + localizeAdminError(err) } - values := url.Values{} - if flash != "" { - values.Set("flash", flash) - } - if token != "" { - values.Set("token", token) - } location := "/admin/" - if encoded := values.Encode(); encoded != "" { - location += "?" + encoded + if strings.TrimSpace(flash) != "" { + values := url.Values{} + values.Set("flash", flash) + location += "?" + values.Encode() } http.Redirect(w, r, location, http.StatusSeeOther) } +// renderAdmin answers the current POST with the console HTML so one-time +// secrets (a freshly minted token) stay out of the URL, history and logs. +func (s *Server) renderAdmin(w http.ResponseWriter, r *http.Request, flash string, token string, err error) { + if err != nil { + flash = "操作失败：" + localizeAdminError(err) + } + data, dataErr := s.store.Dashboard(r.Context(), s.system, flash, token) + if dataErr != nil { + writeError(w, http.StatusInternalServerError, dataErr) + return + } + w.Header().Set("Content-Type", "text/html; charset=utf-8") + if execErr := s.template.Execute(w, data); execErr != nil { + writeError(w, http.StatusInternalServerError, execErr) + } +} + func writeJSON(w http.ResponseWriter, status int, value any) { w.Header().Set("Content-Type", "application/json; charset=utf-8") w.WriteHeader(status) @@ -878,11 +1348,3 @@ func writeError(w http.ResponseWriter, status int, err error) { "error": err.Error(), }) } - -func runStartupScan(ctx context.Context, store *basemap.Store) error { - if !store.AutoScanEnabled() || strings.TrimSpace(store.ScanRoot()) == "" { - return nil - } - _, err := store.RunScan(ctx, "local-basemaps") - return err -} diff --git a/services/api-go/internal/basemapapi/slice_test.go b/services/api-go/internal/basemapapi/slice_test.go new file mode 100644 index 0000000..59996f0 --- /dev/null +++ b/services/api-go/internal/basemapapi/slice_test.go @@ -0,0 +1,181 @@ +package basemapapi + +import ( + "bytes" + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "testing" + + "tile-server/api-go/internal/basemap" +) + +func newSliceServer(t *testing.T) (*basemap.Store, *httptest.Server) { + t.Helper() + root := t.TempDir() + tileRoot := filepath.Join(root, "tiles") + if err := os.MkdirAll(tileRoot, 0o755); err != nil { + t.Fatalf("mkdir tile root: %v", err) + } + source := filepath.Join(root, "input.tif") + writeFile(t, source, "fake raster") + // A scanned basemap gives the negative test case something to grant. + writeFile(t, filepath.Join(tileRoot, "google_image", "0", "0", "0.webp"), "tile") + + store, err := basemap.OpenStore(basemap.StoreConfig{ + SQLitePath: filepath.Join(root, "basemap.db"), + ScanRoot: tileRoot, + VectorScanRoot: filepath.Join(root, "vectors"), + APIBaseURL: "http://127.0.0.1:8910", + TileBaseURL: "http://127.0.0.1:8910", + VectorBaseURL: "http://127.0.0.1:8910", + TileJobCommand: "definitely-not-a-real-binary {input} {output}", + }) + if err != nil { + t.Fatalf("open store: %v", err) + } + t.Cleanup(func() { _ = store.Close() }) + if _, err := store.RunScan(context.Background(), "local-basemaps"); err != nil { + t.Fatalf("scan: %v", err) + } + + return store, newTestServerRaw(t, store) +} + +func newTestServerRaw(t *testing.T, store *basemap.Store) *httptest.Server { + t.Helper() + server, err := New(store, basemap.SystemInfo{ServiceName: "basemap-pusher"}, Options{}) + if err != nil { + t.Fatalf("new server: %v", err) + } + httpServer := httptest.NewServer(server.Handler()) + t.Cleanup(httpServer.Close) + return httpServer +} + +func postJob(t *testing.T, url, token string, body map[string]any) *http.Response { + t.Helper() + encoded, err := json.Marshal(body) + if err != nil { + t.Fatalf("marshal: %v", err) + } + request, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(encoded)) + if err != nil { + t.Fatalf("request: %v", err) + } + request.Header.Set("Content-Type", "application/json") + request.Header.Set("Authorization", "Bearer "+token) + response, err := http.DefaultClient.Do(request) + if err != nil { + t.Fatalf("post: %v", err) + } + return response +} + +func TestSliceJobRequiresSliceScope(t *testing.T) { + ctx := context.Background() + store, httpServer := newSliceServer(t) + source := filepath.Join(filepath.Dir(store.ScanRoot()), "input.tif") + + readOnly, err := store.CreateToken(ctx, basemap.CreateTokenInput{ + Name: "read-only", + BasemapCodes: []string{"google_image"}, + Scopes: []string{"read"}, + }) + if err != nil { + t.Fatalf("create read-only token: %v", err) + } + if _, err := store.CreateToken(ctx, basemap.CreateTokenInput{Name: "empty"}); err == nil { + t.Fatal("expected a token with neither grants nor slice scope to be rejected") + } + + response := postJob(t, httpServer.URL+"/api/v1/slice/jobs", readOnly.Token, map[string]any{ + "source_path": source, + "target_code": "demo", + }) + response.Body.Close() + if response.StatusCode != http.StatusForbidden { + t.Fatalf("expected 403 for a read-only token, got %d", response.StatusCode) + } +} + +func TestSliceJobSubmitAndPoll(t *testing.T) { + ctx := context.Background() + store, httpServer := newSliceServer(t) + source := filepath.Join(filepath.Dir(store.ScanRoot()), "input.tif") + + slicer, err := store.CreateToken(ctx, basemap.CreateTokenInput{ + Name: "integration", + Scopes: []string{"slice"}, + }) + if err != nil { + t.Fatalf("create slicer token: %v", err) + } + + body := map[string]any{ + "code": "integration-job", + "source_path": source, + "target_code": "demo", + "zoom_min": 0, + "zoom_max": 2, + } + response := postJob(t, httpServer.URL+"/api/v1/slice/jobs", slicer.Token, body) + if response.StatusCode != http.StatusAccepted { + response.Body.Close() + t.Fatalf("expected 202, got %d", response.StatusCode) + } + var job basemap.TileJob + if err := json.NewDecoder(response.Body).Decode(&job); err != nil { + t.Fatalf("decode: %v", err) + } + response.Body.Close() + + // Re-submitting the same code must not queue a second job. + again := postJob(t, httpServer.URL+"/api/v1/slice/jobs", slicer.Token, body) + again.Body.Close() + if again.StatusCode != http.StatusAccepted { + t.Fatalf("expected the repeat submit to be accepted, got %d", again.StatusCode) + } + + request, _ := http.NewRequest(http.MethodGet, httpServer.URL+"/api/v1/slice/jobs/"+job.ID, nil) + request.Header.Set("Authorization", "Bearer "+slicer.Token) + polled, err := http.DefaultClient.Do(request) + if err != nil { + t.Fatalf("poll: %v", err) + } + defer polled.Body.Close() + if polled.StatusCode != http.StatusOK { + t.Fatalf("expected 200 from the poll endpoint, got %d", polled.StatusCode) + } + var fetched basemap.TileJob + if err := json.NewDecoder(polled.Body).Decode(&fetched); err != nil { + t.Fatalf("decode polled job: %v", err) + } + if fetched.ID != job.ID { + t.Fatalf("polled %s, submitted %s", fetched.ID, job.ID) + } + + jobs, err := store.ListTileJobs(ctx, 10) + if err != nil { + t.Fatalf("list: %v", err) + } + if len(jobs) != 1 { + t.Fatalf("expected one job after a repeated submit, got %d", len(jobs)) + } +} + +func TestSliceJobRejectsAnonymous(t *testing.T) { + _, httpServer := newSliceServer(t) + request, _ := http.NewRequest(http.MethodGet, httpServer.URL+"/api/v1/slice/jobs/whatever", nil) + response, err := http.DefaultClient.Do(request) + if err != nil { + t.Fatalf("request: %v", err) + } + defer response.Body.Close() + if response.StatusCode != http.StatusUnauthorized { + t.Fatalf("expected 401 without a token, got %d", response.StatusCode) + } +} diff --git a/services/api-go/internal/config/config.go b/services/api-go/internal/config/config.go index 49622e6..a3110ef 100644 --- a/services/api-go/internal/config/config.go +++ b/services/api-go/internal/config/config.go @@ -2,9 +2,12 @@ package config import ( "fmt" + "net" "os" "path/filepath" "strconv" + "strings" + "time" ) type Config struct { @@ -21,6 +24,17 @@ type Config struct { ServiceName string ServiceVersion string Commit string + + AdminToken string + AdminUser string + AdminPassword string + CORSAllowedOrigins []string + TokenDefaultTTL time.Duration + + HealthCheckInterval time.Duration + TileJobCommand string + TileJobVRTCommand string + TileJobLogDir string } func Load() (Config, error) { @@ -36,6 +50,14 @@ func Load() (Config, error) { if err != nil { return Config{}, fmt.Errorf("parse AUTO_SCAN_ON_START: %w", err) } + tokenDefaultTTL, err := durationEnv("TOKEN_DEFAULT_TTL", 720*time.Hour) + if err != nil { + return Config{}, fmt.Errorf("parse TOKEN_DEFAULT_TTL: %w", err) + } + healthCheckInterval, err := durationEnv("HEALTH_CHECK_INTERVAL", 5*time.Minute) + if err != nil { + return Config{}, fmt.Errorf("parse HEALTH_CHECK_INTERVAL: %w", err) + } cfg := Config{ Host: stringEnv("API_HOST", "127.0.0.1"), @@ -43,14 +65,23 @@ func Load() (Config, error) { APIBaseURL: stringEnv("API_BASE_URL", ""), TilePublicBaseURL: stringEnv("TILE_PUBLIC_BASE_URL", ""), VectorPublicBaseURL: stringEnv("VECTOR_PUBLIC_BASE_URL", ""), - SQLitePath: stringEnv("SQLITE_PATH", filepath.Clean(filepath.Join("data", "map-asset-gateway.db"))), + SQLitePath: stringEnv("SQLITE_PATH", filepath.Clean(filepath.Join("data", "basemap-pusher.db"))), BasemapScanRoot: stringEnv("BASEMAP_SCAN_ROOT", filepath.Clean(filepath.Join("data", "tiles"))), VectorScanRoot: stringEnv("VECTOR_SCAN_ROOT", filepath.Clean(filepath.Join("data", "geojson"))), AutoScanOnStart: autoScanOnStart, Env: stringEnv("APP_ENV", "dev"), - ServiceName: stringEnv("SERVICE_NAME", "map-asset-gateway"), + ServiceName: stringEnv("SERVICE_NAME", "basemap-pusher"), ServiceVersion: stringEnv("SERVICE_VERSION", "0.2.0"), Commit: stringEnv("SERVICE_COMMIT", "local"), + AdminToken: strings.TrimSpace(stringEnv("ADMIN_TOKEN", "")), + AdminUser: strings.TrimSpace(stringEnv("ADMIN_USER", "")), + AdminPassword: stringEnv("ADMIN_PASSWORD", ""), + CORSAllowedOrigins: listEnv("CORS_ALLOWED_ORIGINS"), + TokenDefaultTTL: tokenDefaultTTL, + HealthCheckInterval: healthCheckInterval, + TileJobCommand: stringEnv("TILE_JOB_COMMAND", ""), + TileJobVRTCommand: stringEnv("TILE_JOB_VRT_COMMAND", ""), + TileJobLogDir: stringEnv("TILE_JOB_LOG_DIR", ""), } if cfg.APIBaseURL == "" { cfg.APIBaseURL = deriveBaseURL(cfg.Host, cfg.Port) @@ -68,13 +99,37 @@ func (c Config) ListenAddr() string { return fmt.Sprintf("%s:%d", c.Host, c.Port) } +// HasAdminCredentials reports whether the admin console has any credential +// configured. Without one, the admin surface is only safe on loopback. +func (c Config) HasAdminCredentials() bool { + return c.AdminToken != "" || (c.AdminUser != "" && c.AdminPassword != "") +} + +// BindsToNonLoopback reports whether the listen address is reachable from +// outside the host, which makes credentials mandatory. +func (c Config) BindsToNonLoopback() bool { + return !isLoopbackHost(c.Host) +} + func deriveBaseURL(host string, port int) string { - if host == "" || host == "0.0.0.0" { + if host == "" || host == "0.0.0.0" || host == "::" { host = "127.0.0.1" } return fmt.Sprintf("http://%s:%d", host, port) } +func isLoopbackHost(host string) bool { + host = strings.TrimSpace(host) + if host == "" || strings.EqualFold(host, "localhost") { + return true + } + ip := net.ParseIP(strings.Trim(host, "[]")) + if ip == nil { + return false + } + return ip.IsLoopback() +} + func stringEnv(key, fallback string) string { value, ok := os.LookupEnv(key) if ok && value != "" { @@ -106,3 +161,29 @@ func boolEnv(key string, fallback bool) (bool, error) { } return fallback, nil } + +func durationEnv(key string, fallback time.Duration) (time.Duration, error) { + value, ok := os.LookupEnv(key) + if ok && strings.TrimSpace(value) != "" { + parsed, err := time.ParseDuration(strings.TrimSpace(value)) + if err != nil { + return 0, err + } + return parsed, nil + } + return fallback, nil +} + +func listEnv(key string) []string { + raw := strings.FieldsFunc(stringEnv(key, ""), func(r rune) bool { + return r == ',' || r == ';' || r == '\n' || r == '\r' || r == '\t' || r == ' ' + }) + values := make([]string, 0, len(raw)) + for _, item := range raw { + item = strings.TrimSpace(item) + if item != "" { + values = append(values, item) + } + } + return values +} diff --git a/start-all.bat b/start-all.bat deleted file mode 100644 index bf3f63c..0000000 --- a/start-all.bat +++ /dev/null @@ -1,38 +0,0 @@ -@echo off -setlocal -chcp 65001 >nul 2>&1 - -set "ROOT=%~dp0" -call "%ROOT%scripts\load-env.bat" "%ROOT%config\service.env" - -set "LAUNCHER_DIR=%ROOT%temp\launcher" -if not exist "%LAUNCHER_DIR%" mkdir "%LAUNCHER_DIR%" >nul 2>&1 -set "RS_API_PID_FILE=%LAUNCHER_DIR%\api.pid" -del /q "%RS_API_PID_FILE%" >nul 2>&1 - -if "%API_HOST%"=="" set "API_HOST=127.0.0.1" -if "%API_PORT%"=="" set "API_PORT=8910" -if "%API_BASE_URL%"=="" set "API_BASE_URL=http://127.0.0.1:%API_PORT%" - -echo ======================================== -echo Map Asset Gateway Launcher -echo ======================================== -echo. -echo Env file: %ROOT%config\service.env -echo Starting Go service in a separate window... -echo. - -powershell -NoProfile -ExecutionPolicy Bypass -Command "Start-Process -FilePath 'cmd.exe' -ArgumentList '/k','call ""%ROOT%scripts\run-api.bat""' -WorkingDirectory '%ROOT%' | Out-Null" - -echo ======================================== -echo Management Entry -echo ======================================== -echo. -echo Admin Console : %API_BASE_URL%/admin/ -echo API Health : %API_BASE_URL%/health -echo API Ready : %API_BASE_URL%/ready -echo Stop All : %ROOT%stop-all.bat -echo PID Files : %LAUNCHER_DIR% -echo. - -if "%NO_PAUSE%"=="" pause diff --git a/stop-all.bat b/stop-all.bat deleted file mode 100644 index a2432ca..0000000 --- a/stop-all.bat +++ /dev/null @@ -1,101 +0,0 @@ -@echo off -setlocal -chcp 65001 >nul 2>&1 - -set "ROOT=%~dp0" -if exist "%ROOT%scripts\load-env.bat" call "%ROOT%scripts\load-env.bat" "%ROOT%config\service.env" -set "LAUNCHER_DIR=%ROOT%temp\launcher" -if "%API_PORT%"=="" set "API_PORT=8910" - -echo ======================================== -echo Map Asset Gateway Stopper -echo ======================================== -echo. - -call :stop_pid_file "Map Asset Gateway" "%LAUNCHER_DIR%\api.pid" -if not "%STOP_RESULT%"=="stopped" call :stop_port "Map Asset Gateway" "%API_PORT%" -call :stop_window_title "Map Asset Gateway" - -echo. -echo Finished. -if "%NO_PAUSE%"=="" pause -exit /b 0 - -:stop_pid_file -set "STOP_RESULT=missing" -set "SERVICE_NAME=%~1" -set "PID_FILE=%~2" -if not exist "%PID_FILE%" ( - echo [SKIP] %SERVICE_NAME% not running - set "SERVICE_NAME=" - set "PID_FILE=" - exit /b 0 -) - -set "TARGET_PID=" -for /f "usebackq delims=" %%I in ("%PID_FILE%") do ( - set "TARGET_PID=%%I" - goto :pid_loaded -) -:pid_loaded -if not defined TARGET_PID ( - echo [SKIP] %SERVICE_NAME% not running - del /q "%PID_FILE%" >nul 2>&1 - set "SERVICE_NAME=" - set "PID_FILE=" - set "TARGET_PID=" - exit /b 0 -) - -call :kill_pid "%TARGET_PID%" -if not "%KILL_OK%"=="1" ( - echo [SKIP] %SERVICE_NAME% not running -) else ( - echo [OK] Stopped %SERVICE_NAME% PID=%TARGET_PID% - set "STOP_RESULT=stopped" -) - -del /q "%PID_FILE%" >nul 2>&1 -set "SERVICE_NAME=" -set "PID_FILE=" -set "TARGET_PID=" -exit /b 0 - -:stop_port -set "SERVICE_NAME=%~1" -set "TARGET_PORT=%~2" -set "FOUND_PID=" -for /f "tokens=5" %%I in ('netstat -ano -p TCP ^| findstr /R /C:":%TARGET_PORT% .*LISTENING"') do set "FOUND_PID=%%I" -if not defined FOUND_PID ( - set "SERVICE_NAME=" - set "TARGET_PORT=" - exit /b 0 -) - -call :kill_pid "%FOUND_PID%" -if "%KILL_OK%"=="1" ( - echo [OK] Stopped %SERVICE_NAME% by port %TARGET_PORT% PID=%FOUND_PID% - set "STOP_RESULT=stopped" -) -set "SERVICE_NAME=" -set "TARGET_PORT=" -set "FOUND_PID=" -exit /b 0 - -:stop_window_title -taskkill /FI "WINDOWTITLE eq %~1*" /T /F >nul 2>&1 -exit /b 0 - -:kill_pid -set "KILL_OK=0" -set "KILL_TARGET_PID=%~1" -if "%KILL_TARGET_PID%"=="" exit /b 0 -taskkill /PID %KILL_TARGET_PID% /T /F >nul 2>&1 -if errorlevel 1 ( - powershell -NoProfile -ExecutionPolicy Bypass -Command "try { Stop-Process -Id %KILL_TARGET_PID% -Force -ErrorAction Stop; exit 0 } catch { exit 1 }" >nul 2>&1 - if not errorlevel 1 set "KILL_OK=1" -) else ( - set "KILL_OK=1" -) -set "KILL_TARGET_PID=" -exit /b 0