Harmon/map-asset-gateway
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rewrite the gateway around health, retries, slicing and a Windows service

Browse Source 
Scanner
- manifest lookup is now depth-bounded and prunes numeric zoom directories, so
  a scan no longer walks the whole tile payload
- rescans only report a version as updated when something actually changed
- overlapping runs are refused instead of interleaving their bookkeeping

Hot path
- token authorisation loads only that token grants and throttles the
  last_used_at write instead of touching it on every tile
- SQLite runs in WAL with a busy timeout and a real connection pool
- the catalog lookup used by /tiles no longer loads every basemap

Security
- the admin surface requires ADMIN_TOKEN (bearer) or ADMIN_USER/ADMIN_PASSWORD
  (basic), and the service refuses to bind off-host without credentials
- CORS is an explicit allow-list; newly minted tokens never reach a redirect URL
- client tokens are redacted from the access log

Operations
- resource health inspection flags a moved or deleted tile root before a 404
- pushes retry with backoff and can be re-queued by hand; the schedule lives in
  the database so a restart does not lose it
- access log plus /metrics counters

Image slicing
- tile_jobs drive the GDAL bundled with QGIS via scripts/gdal2tiles-qgis.bat
- layer_sources remembers what a layer is made of: submit one source to append
  a flight, or all of them to rebuild the layer as a VRT mosaic with no tile
  seams; the console exposes both, including a one-click rebuild
- slicing is available to integrations through a narrow slice token scope
  (submit, poll, idempotent by code, completion callback)

Console
- rebuilt around a fixed rail and single-view navigation with health and
  outstanding work leading, instead of one long scrolling page

Windows service
- native install/start/stop/status with crash recovery, auto-start and a
  rotating log file, so no NSSM wrapper is needed
- start-all.bat and stop-all.bat are removed: they killed the process by port
  with taskkill, which fought the service restart policy

Verified with 49 tests and live runs against real tile data and real GDAL.
This commit is contained in:
Harmon
2026-10-16 10:35:30 +08:00
parent 8faf25b436
commit 73ad03fa59
49 changed files with 8668 additions and 1817 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+250 -278
View File
@@ -1,129 +1,42 @@
# map-asset-gateway
# Basemap Pusher
`map-asset-gateway` 是一个面向现有底图瓦片和矢量文件的目录管理与分发服务。
An internal catalog and push service for existing basemap tiles, GeoJSON files, and shapefiles.
它不负责生成瓦片，不负责 GIS 生产流程，也不负责旧版遥感结果处理流水线。它只做这几件事：
The service does not generate tiles, does not rebuild vector data, and does not bring back the old result-management pipeline. It only does:
- 扫描已有的底图瓦片目录
- 扫描已有的 GeoJSON / Shapefile 文件
- 将元数据写入 SQLite
- 为底图和矢量资源签发访问 token
- 按 token 提供 catalog、瓦片访问和矢量下载
- 将底图或矢量描述推送给下游系统
- scan existing basemap tile directories
- scan existing vector files
- manage metadata
- issue tokens
- control token access to basemaps and vector assets
- push basemap or vector descriptors to downstream systems
## 技术栈
## Stack
- Go
- SQLite
- Go 标准库 `net/http`
- standard library HTTP server
## 适用场景
## Current Capabilities
适合已经有离线瓦片目录、GeoJSON、SHP 数据，希望补一层统一管理、授权和分发接口的场景。
- scan `BASEMAP_SCAN_ROOT` into SQLite
- scan `VECTOR_SCAN_ROOT` into SQLite
- support two basemap directory styles
- manifest-based directories
- raw `z/x/y.ext` tile roots
- support vector asset formats
- `.geojson`
- `.json`
- `.shp` plus sidecar files
- `.zip` shapefile packages
- token-protected basemap catalog, tile access, vector catalog, and vector download
- token-protected compatibility routes for default-version tiles, GeoJSON paths, and shapefile paths
- admin panel at `GET /admin/`
- downstream callback push for basemaps and vectors
不适合下面这些需求：
## Basemap Scan Rules
- 在线切片或重建瓦片
- 栅格处理、预览图生成、图例生成
- PostgreSQL / PostGIS 任务流水线
- Python worker 驱动的结果管理系统
## 运行架构
服务启动时会按下面的顺序运行：
1. 读取 `config/service.env` 或环境变量
2. 打开 SQLite 并自动建表
3. 注册默认扫描源
4. 如果 `AUTO_SCAN_ON_START=true`，自动扫描底图和矢量目录
5. 启动 HTTP API 和管理页
这是一个单进程服务，没有额外 worker，没有外部数据库依赖。
## 目录结构
```text
.
├─ services/api-go/ Go 服务源码
├─ config/ 环境变量配置
├─ scripts/ 启动脚本
├─ docs/ 接入文档
├─ data/ 本地数据目录，不建议提交到 Git
├─ logs/ 本地运行日志，不建议提交到 Git
└─ temp/ 本地临时文件，不建议提交到 Git
```
说明：
- `services/api-go` 是当前项目主体。
- 历史遗留 demo 已从仓库中移除，避免和当前服务边界混淆。
- `data/`、`logs/`、`temp/`、`.gocache/`、`.gomodcache/`、`.gotmp/`、`.gopath/` 都属于本地运行或构建产物。
## 快速启动
### 前置条件
- Go 1.26 或兼容版本
- 可写的本地目录用于放 SQLite、扫描根目录和 Go 缓存
- Windows 环境下可直接使用仓库自带批处理脚本
### 配置
复制示例配置：
```text
config/service.env.example -> config/service.env
```
关键配置项：
| 变量 | 说明 |
| --- | --- |
| `API_HOST` | 监听地址 |
| `API_PORT` | 监听端口 |
| `API_BASE_URL` | API 对外基地址 |
| `TILE_PUBLIC_BASE_URL` | 底图下载地址前缀 |
| `VECTOR_PUBLIC_BASE_URL` | 矢量下载地址前缀 |
| `SQLITE_PATH` | SQLite 文件路径 |
| `BASEMAP_SCAN_ROOT` | 底图扫描根目录 |
| `VECTOR_SCAN_ROOT` | 矢量扫描根目录 |
| `AUTO_SCAN_ON_START` | 启动时是否自动扫描 |
默认示例见 [config/service.env.example](./config/service.env.example)。
### 启动
Windows 下推荐：
```powershell
.\start-all.bat
```
或者直接运行 API：
```powershell
.\scripts\run-api.bat
```
停止：
```powershell
.\stop-all.bat
```
启动后可访问：
- 管理页: `http://127.0.0.1:8910/admin/`
- 健康检查: `http://127.0.0.1:8910/health`
- 就绪检查: `http://127.0.0.1:8910/ready`
## 数据组织与扫描规则
### 底图目录
服务支持两种底图目录形式。
### 1. 带 `manifest.json` 的目录
### Manifest-based directory
```text
data/tiles/gaode-image/
@@ -133,7 +46,7 @@ data/tiles/gaode-image/
1/0/0.png
```
示例 `manifest.json`：
Example `manifest.json`:
```json
{
@@ -151,7 +64,7 @@ data/tiles/gaode-image/
}
```
### 2. 直接是 `z/x/y.ext` 的裸目录
### Raw tile directory
```text
data/tiles/google-image/
@@ -159,22 +72,17 @@ data/tiles/google-image/
1/0/0.webp
```
这类目录会自动导入为：
This is imported automatically as:
- `code = 目录名归一化后的值`
- `code = directory name`
- `version = current`
- `type = xyz`
说明：
## Vector Scan Rules
- `google-image`、`google_image`、`google.image` 最终都会被归一化成类似 `google-image` 的内部 code。
- 默认版本路由 `/tiles/{code}/{z}/{x}/{y}.ext` 会解析到该底图当前默认版本。
The service scans `VECTOR_SCAN_ROOT` recursively.
### 矢量目录
`VECTOR_SCAN_ROOT` 会被递归扫描。
示例：
Examples:
```text
data/geojson/admin-boundary.geojson
@@ -184,95 +92,37 @@ data/geojson/landuse/parcel.dbf
data/geojson/landuse/parcel.shx
```
扫描规则：
Rules:
- `.geojson` 和 `.json` 会被识别为 `geojson`
- `.zip` 会被识别为 `shpzip`
- `.shp` 会被识别为 `shapefile`
- `shapefile` 下载时会按同名 sidecar 文件动态打包成 zip
- `.geojson` and `.json` are imported as `geojson`
- `.zip` is imported as `shpzip`
- `.shp` is imported as `shapefile`
- shapefile downloads are served as zip packages on demand
## 访问控制
## Token Model
服务使用 token 做资源访问控制。
A token can be granted access to:
一个 token 可以授权：
- one or more basemaps
- one or more vector assets
- 一个或多个底图
- 一个或多个矢量资源
支持的 token 传递方式：
Supported token delivery:
- `Authorization: Bearer <token>`
- `X-Access-Token: <token>`
- 查询参数 `token=`
- 查询参数 `access_token=`
- query parameter `token=`
- query parameter `access_token=`
## 典型使用流程
## API
### 1. 启动服务并扫描目录
如果开启了 `AUTO_SCAN_ON_START=true`，服务会在启动时自动扫描。
也可以通过管理接口手动触发：
- `POST /api/v1/admin/scan-sources/{code}/run`
- `POST /api/v1/admin/vectors/rescan`
### 2. 创建 token
示例：
```json
POST /api/v1/admin/tokens
{
"name": "insar-v2",
"basemap_codes": ["google-image"],
"vector_codes": ["全国行政区.geojson", "landuse/parcel.shp"]
}
```
### 3. 访问 catalog 或资源
```text
GET /api/v1/catalog/basemaps?token=bmp_xxx
GET /api/v1/catalog/vectors?token=bmp_xxx
GET /tiles/google-image/10/843/388.webp?token=bmp_xxx
GET /tiles/google-image/current/10/843/388.webp?token=bmp_xxx
GET /geojson/全国行政区.geojson?token=bmp_xxx
GET /shp/landuse/parcel.shp?token=bmp_xxx
GET /vectors/landuse-parcel/download?token=bmp_xxx
```
说明：
- `/tiles/{code}/{z}/{x}/{y}.ext` 走默认版本兼容路由
- `/tiles/{code}/{version}/{z}/{x}/{y}.ext` 显式指定版本
- `/geojson/{...}` 支持按原始相对路径、文件名或内部 code 解析
- `/shp/{...}` 会把 `.shp + .dbf + .shx ...` 打成 zip 返回
### 4. 推送给下游系统
先注册目标系统：
- `GET /api/v1/admin/targets`
- `POST /api/v1/admin/targets`
再触发推送：
- `POST /api/v1/admin/pushes`
- `POST /api/v1/admin/vector-pushes`
推送内容是底图版本或矢量资源的描述信息，不是资源文件本体上传。
## API 概览
### 系统接口
### System
- `GET /health`
- `GET /ready`
- `GET /metrics` (Prometheus text: request, tile, tile-miss and auth counters)
- `GET /api/v1/system`
### 管理接口
### Admin
- `GET /admin/`
- `GET /api/v1/admin/dashboard`
@@ -293,7 +143,7 @@ GET /vectors/landuse-parcel/download?token=bmp_xxx
- `GET /api/v1/admin/vector-push-records`
- `POST /api/v1/admin/vector-pushes`
### Catalog 与下载接口
### Catalog and Downloads
- `GET /api/v1/catalog/basemaps`
- `GET /api/v1/catalog/basemaps/{code}`
@@ -305,7 +155,197 @@ GET /vectors/landuse-parcel/download?token=bmp_xxx
- `GET /shp/{relative/path/to/file.shp}`
- `GET /vectors/{code}/download`
## 数据库表
Examples:
```text
GET /tiles/gaode_image/10/843/388.png?token=bmp_xxx
GET /tiles/gaode-image/current/10/843/388.png?token=bmp_xxx
GET /geojson/全国行政区.geojson?token=bmp_xxx
GET /shp/landuse/parcel.shp?token=bmp_xxx
GET /vectors/admin-boundary/download?token=bmp_xxx
```
Compatibility route notes:
- `/tiles/{code}/{z}/{x}/{y}.ext` resolves the current default version of the basemap
- `/geojson/{...}` resolves vector assets by original relative path, file name, or internal code
- `/shp/{...}` returns `.shp` packages as zip downloads and also supports `.zip` vector packages
- token creation accepts vector references by code or by original relative path
## Slicing API for Other Systems
Integrations call a narrow, non-admin surface:
```text
POST /api/v1/slice/jobs submit a job -> 202 + the job object
GET /api/v1/slice/jobs/{id} poll one job -> 200 + the job object
GET /api/v1/slice/layers/{code}/sources the layer's source rasters
```
All three require a service token carrying the **`slice`** scope. Create one
through the admin API or console; a slice-only token needs no basemap or vector
grants, so it cannot read any tiles:
```json
POST /api/v1/admin/tokens
{"name": "integration", "scopes": ["slice"]}
```
- **Accumulating layers** — a layer is the union of its source rasters, tracked in
`layer_sources`. Submit **one** new source to append a flight (fast, what
overlaps wins per tile), or submit **all** of `GET /api/v1/slice/layers/{code}/sources`
to rebuild the layer as a VRT mosaic (per-pixel, no tile seams). The console's
影像切片 view lists each layer's sources.
- `TILE_JOB_COMMAND` slices one input; `TILE_JOB_VRT_COMMAND` runs first when
several sources are submitted. `{inputs}` expands to one argument per source,
and `{output_vrt}` is the temporary mosaic. Both ship as wrappers over the GDAL
bundled with QGIS (`scripts/gdal2tiles-qgis.bat`,
`scripts/gdalbuildvrt-qgis.bat`) because the Python bindings need the OSGeo4W
environment; set `GDAL2TILES_QGIS_ROOT` when QGIS is not in `%ProgramFiles%`.
The slicer default passes `--exclude`, so a new flight cannot punch transparent
holes into the previous imagery.
- **Rebuild from the console** — 影像切片 → 图层源清单 → 重建为镶嵌 slices every
recorded source into one VRT, reusing the zoom range the layer is already
published at. No need to copy paths by hand.
- **Known limit** — a layer's source list only grows (there is no delete API yet),
and a rebuild never clears the output directory. If a source is ever removed by
hand, tiles it used to cover but no longer does stay behind; delete the layer
directory and rebuild to clean up.
- **Idempotent submit** — sending the same `code` twice returns the existing job
instead of queueing a second one, so a retrying caller is safe.
- **Completion callback** — pass `callback_url` when submitting and the service
POSTs `{"event":"tile.job.finished","job":{...}}` when the job ends. Failures
retry on the same 30s/60s backoff as push callbacks.
- Jobs run one at a time, oldest first; the admin console's 影像切片 view shows
the same queue.
## Resource Health, Retries and Slicing
- **Resource health** — every catalogued tile root and vector file is stat'ed on
`HEALTH_CHECK_INTERVAL` and on demand (`POST /api/v1/admin/health/check`).
A path that vanished is recorded as `missing` and surfaces in the console's
待处理 list instead of waiting for a client 404. Health is tracked separately
from `status`: status is a catalog fact, health is whether the bytes are on
disk right now.
- **Push retries** — a failed callback is retried up to 3 times with backoff
(30s, 60s) and stays `retrying` while queued; the schedule lives in the
database so a restart does not lose it. A finished record can be re-queued
with `POST /api/v1/admin/push-records/{id}/retry` (or
`.../vector-push-records/{id}/retry`).
- **Image slicing** — `scripts/gdal2tiles-qgis.bat` runs the GDAL bundled with
QGIS (its Python bindings need the OSGeo4W environment, so a bare
`gdal2tiles.exe` will not work; set `GDAL2TILES_QGIS_ROOT` if QGIS is not in
`%ProgramFiles%`). `POST /api/v1/admin/tile-jobs` queues a job that runs
`TILE_JOB_COMMAND` (see [config/service.env.example](./config/service.env.example))
and, on success, rescans so the sliced basemap appears in the catalog. The
service does the orchestration; the raster work is done by GDAL. The service
account needs write access to the basemap scan root.
## Running as a Windows Service
The binary manages its own service registration — no NSSM or WinSW wrapper:
```text
basemap-pusher.exe --env-file <path\to\config\service.env> install
basemap-pusher.exe start
basemap-pusher.exe status
basemap-pusher.exe stop
basemap-pusher.exe uninstall
```
- `install` bakes `--env-file` into the service command line, because the SCM
starts services in `C:\Windows\System32` and the app would otherwise miss its
config. Without `--env-file` it looks for `config/service.env` next to the
executable and up to two directories above it.
- A service has no console, so the standard logger is redirected to
`bin\logs\basemap-pusher.log`, rolling to `.1` at 10 MB. Override the path with
the `SERVICE_LOG_FILE` environment variable. Without this the service would
discard every log line.
- Recovery actions are registered on install: restart after 5s, 15s, then 60s,
retrying for a day. The service starts automatically at boot.
- `install` / `uninstall` / `start` / `stop` need an **elevated** console
(Run as Administrator); `status` and the console mode do not.
- Started with no verb the binary runs as a normal console app and handles
Ctrl+C / SIGTERM with a 15s drain. Under the SCM the service handler owns the
same shutdown path.
Build the binary the service will run (it must not depend on the Go toolchain):
```text
scripts\build-api.bat
```
The script loads `config/service.env` first, so the Go caches and `GOPROXY`
match the rest of the project instead of the global defaults. Running `go build`
by hand from the repository root will fail: the module lives in
`services\api-go`, not at the root.
## Access Control
The admin console and `/api/v1/admin/*` can mint tokens that read every
basemap, so they are guarded:
- configure `ADMIN_USER` + `ADMIN_PASSWORD` for the browser console (HTTP Basic)
- configure `ADMIN_TOKEN` for API clients (`Authorization: Bearer <token>`)
- with neither set, the service **refuses to start** unless `API_HOST` is a
loopback address (`127.0.0.1`, `::1`, `localhost`)
Cross-origin requests are denied by default; add the exact origins you trust to
`CORS_ALLOWED_ORIGINS`. Client tokens never appear in redirect URLs, and tokens
carried in a query string are redacted from the access log.
## Config
See [config/service.env.example](./config/service.env.example).
For `Insar_management_system_v2` integration, see [docs/INSAR_MANAGEMENT_SYSTEM_V2.md](./docs/INSAR_MANAGEMENT_SYSTEM_V2.md).
Important variables:
- `API_HOST`
- `API_PORT`
- `API_BASE_URL`
- `TILE_PUBLIC_BASE_URL`
- `VECTOR_PUBLIC_BASE_URL`
- `SQLITE_PATH`
- `BASEMAP_SCAN_ROOT`
- `VECTOR_SCAN_ROOT`
- `AUTO_SCAN_ON_START`
- `ADMIN_USER` / `ADMIN_PASSWORD` (HTTP Basic for the browser console)
- `ADMIN_TOKEN` (bearer token for admin API clients)
- `CORS_ALLOWED_ORIGINS` (comma-separated allow-list; empty means no CORS)
- `TOKEN_DEFAULT_TTL` (lifetime for new tokens, e.g. `720h`; `0` disables expiry)
- `HEALTH_CHECK_INTERVAL` (resource availability pass, e.g. `5m`; `0` disables)
- `TILE_JOB_COMMAND` (image slicing command template; needs GDAL)
- `TILE_JOB_LOG_DIR` (per-job log directory; empty = `<scan root>/.tile-jobs`)
## Start
### Windows service (production)
The service is the supported way to run this on Windows — it starts at boot,
restarts on failure and keeps its own log file:
```text
scripts\build-api.bat
scripts\service.bat install (elevated)
scripts\service.bat start
scripts\service.bat status
```
`service.bat` accepts `install | uninstall | start | stop | restart | status`
and forwards to `bin\basemap-pusher.exe`. The install / uninstall / start / stop
verbs need an **elevated** shell; `status` does not.
The old `start-all.bat` / `stop-all.bat` launchers were removed: they killed the
process by port with `taskkill`, which fought the service's restart policy.
### Console (development)
```text
scripts\run-api.bat
```
This runs `go run` with the project's own caches, for a quick local loop. It is
not supervised and does not survive a reboot.
## Database Tables
- `basemaps`
- `basemap_versions`
@@ -319,80 +359,12 @@ GET /vectors/landuse-parcel/download?token=bmp_xxx
- `vector_token_grants`
- `vector_push_records`
## 提交到 Git 前的约定
## Design Boundaries
这个仓库里有很多目录本质上是本地运行数据或缓存，不建议提交到版本库。
- no Python worker
- no PostgreSQL
- no tile generation pipeline
- no result artifact management
- no raster processing jobs
默认不建议提交：
- `data/`
- `logs/`
- `temp/`
- `.tmp/`
- `.gocache/`
- `.gomodcache/`
- `.gotmp/`
- `.gopath/`
- `config/service.env`
原因：
- `data/` 里通常是真实底图、真实 GeoJSON/SHP、以及本地 SQLite 数据库
- `logs/` 和 `temp/` 是运行期产物
- `.gocache/`、`.gomodcache/`、`.gotmp/`、`.gopath/` 是本地 Go 构建缓存
- `config/service.env` 往往包含机器本地路径
如果需要给新同事一个可运行的仓库，建议提交：
- 源码
- `config/service.env.example`
- 文档
- 一个最小化的目录约定说明
而不是直接提交大体积瓦片、数据库和本地缓存。
## 开发与测试
在仓库内使用本地 Go 缓存目录跑测试：
```powershell
$env:GOCACHE='C:\path\to\map-asset-gateway\.gocache'
$env:GOMODCACHE='C:\path\to\map-asset-gateway\.gomodcache'
$env:GOTMPDIR='C:\path\to\map-asset-gateway\.gotmp'
$env:GOPATH='C:\path\to\map-asset-gateway\.gopath'
$env:GOPROXY='https://goproxy.cn,direct'
$env:GOSUMDB='off'
go test ./...
```
当前测试主要覆盖：
- 底图扫描
- 矢量扫描
- token 鉴权
- 默认版本瓦片路由
- GeoJSON / SHP 兼容访问路径
## 相关文档
- [config/service.env.example](./config/service.env.example)
## 项目边界
再次强调，这个项目当前的边界是：
- 扫描
- 建目录
- 发 token
- 鉴权
- 下载
- 推送资源描述
不包括：
- Python worker
- PostgreSQL
- PostGIS
- 在线切片
- 栅格处理任务
- 结果资产管理流水线
GIS software still produces the tiles and vector files. This project only scans, catalogs, authorizes, downloads, and pushes descriptors.