Basemap Pusher: catalog and distribution gateway for existing tiles

A Go service that scans existing basemap tile directories and vector files, catalogues them in SQLite, issues scoped access tokens, and serves tiles and downloads to downstream systems. It deliberately does not generate or process raster data: GIS software produces the tiles, this service catalogues, authorises and distributes them. Capabilities - scans basemap tile directories (manifest-based or raw z/x/y) and vector assets (.geojson, .json, .zip, shapefiles with sidecars) - token auth with per-resource grants, expiry and coarse scopes (read, slice) - resource health inspection flags a moved or deleted tile root before a client hits a 404 - downstream push with exponential-backoff retries and manual re-queue - image slicing jobs driven by the GDAL bundled with QGIS, including VRT mosaicking so successive drone flights accumulate into one seamless layer - a narrow, non-admin slicing API for external callers: submit, poll, idempotent by job code, completion callback - ops console covering health, ledger, tokens, pushes, scans, slicing and service info - native Windows service install/start/stop with crash recovery, auto-start and a rotating log file, so no NSSM or WinSW wrapper is needed Verified with 49 tests plus live runs against real tile data and real GDAL.
2026-10-16 10:36:12 +08:00
commit 747eed63b0
55 changed files with 13526 additions and 0 deletions
@@ -0,0 +1,370 @@
+# Basemap Pusher
+
+An internal catalog and push service for existing basemap tiles, GeoJSON files, and shapefiles.
+
+The service does not generate tiles, does not rebuild vector data, and does not bring back the old result-management pipeline. It only does:
+
+- scan existing basemap tile directories
+- scan existing vector files
+- manage metadata
+- issue tokens
+- control token access to basemaps and vector assets
+- push basemap or vector descriptors to downstream systems
+
+## Stack
+
+- Go
+- SQLite
+- standard library HTTP server
+
+## Current Capabilities
+
+- scan `BASEMAP_SCAN_ROOT` into SQLite
+- scan `VECTOR_SCAN_ROOT` into SQLite
+- support two basemap directory styles
+  - manifest-based directories
+  - raw `z/x/y.ext` tile roots
+- support vector asset formats
+  - `.geojson`
+  - `.json`
+  - `.shp` plus sidecar files
+  - `.zip` shapefile packages
+- token-protected basemap catalog, tile access, vector catalog, and vector download
+- token-protected compatibility routes for default-version tiles, GeoJSON paths, and shapefile paths
+- admin panel at `GET /admin/`
+- downstream callback push for basemaps and vectors
+
+## Basemap Scan Rules
+
+### Manifest-based directory
+
+```text
+data/tiles/gaode-image/
+  manifest.json
+  tiles/
+    0/0/0.png
+    1/0/0.png
+```
+
+Example `manifest.json`:
+
+```json
+{
+  "code": "gaode-image",
+  "name": "Gaode Image",
+  "type": "xyz",
+  "version": "2026-04",
+  "status": "ready",
+  "tile_format": "png",
+  "tile_scheme": "xyz",
+  "min_zoom": 0,
+  "max_zoom": 18,
+  "root_path": "tiles",
+  "is_default": true
+}
+```
+
+### Raw tile directory
+
+```text
+data/tiles/google-image/
+  0/0/0.webp
+  1/0/0.webp
+```
+
+This is imported automatically as:
+
+- `code = directory name`
+- `version = current`
+- `type = xyz`
+
+## Vector Scan Rules
+
+The service scans `VECTOR_SCAN_ROOT` recursively.
+
+Examples:
+
+```text
+data/geojson/admin-boundary.geojson
+data/geojson/roads.zip
+data/geojson/landuse/parcel.shp
+data/geojson/landuse/parcel.dbf
+data/geojson/landuse/parcel.shx
+```
+
+Rules:
+
+- `.geojson` and `.json` are imported as `geojson`
+- `.zip` is imported as `shpzip`
+- `.shp` is imported as `shapefile`
+- shapefile downloads are served as zip packages on demand
+
+## Token Model
+
+A token can be granted access to:
+
+- one or more basemaps
+- one or more vector assets
+
+Supported token delivery:
+
+- `Authorization: Bearer <token>`
+- `X-Access-Token: <token>`
+- query parameter `token=`
+- query parameter `access_token=`
+
+## API
+
+### System
+
+- `GET /health`
+- `GET /ready`
+- `GET /metrics` (Prometheus text: request, tile, tile-miss and auth counters)
+- `GET /api/v1/system`
+
+### Admin
+
+- `GET /admin/`
+- `GET /api/v1/admin/dashboard`
+- `GET /api/v1/admin/basemaps`
+- `GET /api/v1/admin/vectors`
+- `GET /api/v1/admin/scan-sources`
+- `POST /api/v1/admin/scan-sources`
+- `POST /api/v1/admin/scan-sources/{code}/run`
+- `POST /api/v1/admin/vectors/rescan`
+- `POST /api/v1/admin/basemaps/{code}/default-version`
+- `GET /api/v1/admin/targets`
+- `POST /api/v1/admin/targets`
+- `GET /api/v1/admin/tokens`
+- `POST /api/v1/admin/tokens`
+- `POST /api/v1/admin/tokens/{id}/disable`
+- `GET /api/v1/admin/push-records`
+- `POST /api/v1/admin/pushes`
+- `GET /api/v1/admin/vector-push-records`
+- `POST /api/v1/admin/vector-pushes`
+
+### Catalog and Downloads
+
+- `GET /api/v1/catalog/basemaps`
+- `GET /api/v1/catalog/basemaps/{code}`
+- `GET /api/v1/catalog/vectors`
+- `GET /api/v1/catalog/vectors/{code}`
+- `GET /tiles/{code}/{z}/{x}/{y}.ext`
+- `GET /tiles/{code}/{version}/{z}/{x}/{y}.ext`
+- `GET /geojson/{relative/path/to/file.geojson}`
+- `GET /shp/{relative/path/to/file.shp}`
+- `GET /vectors/{code}/download`
+
+Examples:
+
+```text
+GET /tiles/gaode_image/10/843/388.png?token=bmp_xxx
+GET /tiles/gaode-image/current/10/843/388.png?token=bmp_xxx
+GET /geojson/全国行政区.geojson?token=bmp_xxx
+GET /shp/landuse/parcel.shp?token=bmp_xxx
+GET /vectors/admin-boundary/download?token=bmp_xxx
+```
+
+Compatibility route notes:
+
+- `/tiles/{code}/{z}/{x}/{y}.ext` resolves the current default version of the basemap
+- `/geojson/{...}` resolves vector assets by original relative path, file name, or internal code
+- `/shp/{...}` returns `.shp` packages as zip downloads and also supports `.zip` vector packages
+- token creation accepts vector references by code or by original relative path
+
+## Slicing API for Other Systems
+
+Integrations call a narrow, non-admin surface:
+
+```text
+POST /api/v1/slice/jobs                     submit a job  -> 202 + the job object
+GET  /api/v1/slice/jobs/{id}                poll one job  -> 200 + the job object
+GET  /api/v1/slice/layers/{code}/sources    the layer's source rasters
+```
+
+All three require a service token carrying the **`slice`** scope. Create one
+through the admin API or console; a slice-only token needs no basemap or vector
+grants, so it cannot read any tiles:
+
+```json
+POST /api/v1/admin/tokens
+{"name": "integration", "scopes": ["slice"]}
+```
+
+- **Accumulating layers** — a layer is the union of its source rasters, tracked in
+  `layer_sources`. Submit **one** new source to append a flight (fast, what
+  overlaps wins per tile), or submit **all** of `GET /api/v1/slice/layers/{code}/sources`
+  to rebuild the layer as a VRT mosaic (per-pixel, no tile seams). The console's
+  影像切片 view lists each layer's sources.
+- `TILE_JOB_COMMAND` slices one input; `TILE_JOB_VRT_COMMAND` runs first when
+  several sources are submitted. `{inputs}` expands to one argument per source,
+  and `{output_vrt}` is the temporary mosaic. Both ship as wrappers over the GDAL
+  bundled with QGIS (`scripts/gdal2tiles-qgis.bat`,
+  `scripts/gdalbuildvrt-qgis.bat`) because the Python bindings need the OSGeo4W
+  environment; set `GDAL2TILES_QGIS_ROOT` when QGIS is not in `%ProgramFiles%`.
+  The slicer default passes `--exclude`, so a new flight cannot punch transparent
+  holes into the previous imagery.
+- **Rebuild from the console** — 影像切片 → 图层源清单 → 重建为镶嵌 slices every
+  recorded source into one VRT, reusing the zoom range the layer is already
+  published at. No need to copy paths by hand.
+- **Known limit** — a layer's source list only grows (there is no delete API yet),
+  and a rebuild never clears the output directory. If a source is ever removed by
+  hand, tiles it used to cover but no longer does stay behind; delete the layer
+  directory and rebuild to clean up.
+- **Idempotent submit** — sending the same `code` twice returns the existing job
+  instead of queueing a second one, so a retrying caller is safe.
+- **Completion callback** — pass `callback_url` when submitting and the service
+  POSTs `{"event":"tile.job.finished","job":{...}}` when the job ends. Failures
+  retry on the same 30s/60s backoff as push callbacks.
+- Jobs run one at a time, oldest first; the admin console's 影像切片 view shows
+  the same queue.
+## Resource Health, Retries and Slicing
+
+- **Resource health** — every catalogued tile root and vector file is stat'ed on
+  `HEALTH_CHECK_INTERVAL` and on demand (`POST /api/v1/admin/health/check`).
+  A path that vanished is recorded as `missing` and surfaces in the console's
+  待处理 list instead of waiting for a client 404. Health is tracked separately
+  from `status`: status is a catalog fact, health is whether the bytes are on
+  disk right now.
+- **Push retries** — a failed callback is retried up to 3 times with backoff
+  (30s, 60s) and stays `retrying` while queued; the schedule lives in the
+  database so a restart does not lose it. A finished record can be re-queued
+  with `POST /api/v1/admin/push-records/{id}/retry` (or
+  `.../vector-push-records/{id}/retry`).
+- **Image slicing** — `scripts/gdal2tiles-qgis.bat` runs the GDAL bundled with
+  QGIS (its Python bindings need the OSGeo4W environment, so a bare
+  `gdal2tiles.exe` will not work; set `GDAL2TILES_QGIS_ROOT` if QGIS is not in
+  `%ProgramFiles%`). `POST /api/v1/admin/tile-jobs` queues a job that runs
+  `TILE_JOB_COMMAND` (see [config/service.env.example](./config/service.env.example))
+  and, on success, rescans so the sliced basemap appears in the catalog. The
+  service does the orchestration; the raster work is done by GDAL. The service
+  account needs write access to the basemap scan root.
+## Running as a Windows Service
+
+The binary manages its own service registration — no NSSM or WinSW wrapper:
+
+```text
+basemap-pusher.exe --env-file <path\to\config\service.env> install
+basemap-pusher.exe start
+basemap-pusher.exe status
+basemap-pusher.exe stop
+basemap-pusher.exe uninstall
+```
+
+- `install` bakes `--env-file` into the service command line, because the SCM
+  starts services in `C:\Windows\System32` and the app would otherwise miss its
+  config. Without `--env-file` it looks for `config/service.env` next to the
+  executable and up to two directories above it.
+- A service has no console, so the standard logger is redirected to
+  `bin\logs\basemap-pusher.log`, rolling to `.1` at 10 MB. Override the path with
+  the `SERVICE_LOG_FILE` environment variable. Without this the service would
+  discard every log line.
+- Recovery actions are registered on install: restart after 5s, 15s, then 60s,
+  retrying for a day. The service starts automatically at boot.
+- `install` / `uninstall` / `start` / `stop` need an **elevated** console
+  (Run as Administrator); `status` and the console mode do not.
+- Started with no verb the binary runs as a normal console app and handles
+  Ctrl+C / SIGTERM with a 15s drain. Under the SCM the service handler owns the
+  same shutdown path.
+
+Build the binary the service will run (it must not depend on the Go toolchain):
+
+```text
+scripts\build-api.bat
+```
+
+The script loads `config/service.env` first, so the Go caches and `GOPROXY`
+match the rest of the project instead of the global defaults. Running `go build`
+by hand from the repository root will fail: the module lives in
+`services\api-go`, not at the root.
+## Access Control
+
+The admin console and `/api/v1/admin/*` can mint tokens that read every
+basemap, so they are guarded:
+
+- configure `ADMIN_USER` + `ADMIN_PASSWORD` for the browser console (HTTP Basic)
+- configure `ADMIN_TOKEN` for API clients (`Authorization: Bearer <token>`)
+- with neither set, the service **refuses to start** unless `API_HOST` is a
+  loopback address (`127.0.0.1`, `::1`, `localhost`)
+
+Cross-origin requests are denied by default; add the exact origins you trust to
+`CORS_ALLOWED_ORIGINS`. Client tokens never appear in redirect URLs, and tokens
+carried in a query string are redacted from the access log.
+## Config
+
+See [config/service.env.example](./config/service.env.example).
+
+For `Insar_management_system_v2` integration, see [docs/INSAR_MANAGEMENT_SYSTEM_V2.md](./docs/INSAR_MANAGEMENT_SYSTEM_V2.md).
+
+Important variables:
+
+- `API_HOST`
+- `API_PORT`
+- `API_BASE_URL`
+- `TILE_PUBLIC_BASE_URL`
+- `VECTOR_PUBLIC_BASE_URL`
+- `SQLITE_PATH`
+- `BASEMAP_SCAN_ROOT`
+- `VECTOR_SCAN_ROOT`
+- `AUTO_SCAN_ON_START`
+- `ADMIN_USER` / `ADMIN_PASSWORD` (HTTP Basic for the browser console)
+- `ADMIN_TOKEN` (bearer token for admin API clients)
+- `CORS_ALLOWED_ORIGINS` (comma-separated allow-list; empty means no CORS)
+- `TOKEN_DEFAULT_TTL` (lifetime for new tokens, e.g. `720h`; `0` disables expiry)
+- `HEALTH_CHECK_INTERVAL` (resource availability pass, e.g. `5m`; `0` disables)
+- `TILE_JOB_COMMAND` (image slicing command template; needs GDAL)
+- `TILE_JOB_LOG_DIR` (per-job log directory; empty = `<scan root>/.tile-jobs`)
+
+## Start
+
+### Windows service (production)
+
+The service is the supported way to run this on Windows — it starts at boot,
+restarts on failure and keeps its own log file:
+
+```text
+scripts\build-api.bat
+scripts\service.bat install     (elevated)
+scripts\service.bat start
+scripts\service.bat status
+```
+
+`service.bat` accepts `install | uninstall | start | stop | restart | status`
+and forwards to `bin\basemap-pusher.exe`. The install / uninstall / start / stop
+verbs need an **elevated** shell; `status` does not.
+
+The old `start-all.bat` / `stop-all.bat` launchers were removed: they killed the
+process by port with `taskkill`, which fought the service's restart policy.
+
+### Console (development)
+
+```text
+scripts\run-api.bat
+```
+
+This runs `go run` with the project's own caches, for a quick local loop. It is
+not supervised and does not survive a reboot.
+
+## Database Tables
+
+- `basemaps`
+- `basemap_versions`
+- `scan_sources`
+- `scan_runs`
+- `target_systems`
+- `service_tokens`
+- `token_grants`
+- `push_records`
+- `vector_assets`
+- `vector_token_grants`
+- `vector_push_records`
+
+## Design Boundaries
+
+- no Python worker
+- no PostgreSQL
+- no tile generation pipeline
+- no result artifact management
+- no raster processing jobs
+
+GIS software still produces the tiles and vector files. This project only scans, catalogs, authorizes, downloads, and pushes descriptors.