Overview
Build the monitoring system I could explain end to end
After moving my applications onto shared infrastructure, I wanted one place to see whether they were responding, how long requests took, and whether a failure had become a sustained outage. Status monitors my portfolio, Tarkov Tracker, Nades, Meals, Quiz, and UI Lab, and publishes their observed health at status.asmale.me.
This was also a deliberate Go learning project. I built it incrementally with AI assistance, reviewing configuration, HTTP checks, state transitions, persistence, scheduling, and rendering as separate milestones. The goal was a small system whose behavior I could understand and verify, rather than a replacement for a commercial observability platform.
Scope and constraints
One process, one database, existing infrastructure
Keep operation inexpensive
Use one Go process and a local SQLite database. Reuse Docker, GitLab Runners, and Caddy without adding Redis, Kubernetes, or external monitoring services.
Configure instead of administering
Define service IDs, URLs, intervals, and request timeout in a validated JSON file. Stable IDs preserve history when display names change; configuration changes take effect on restart.
Render on the server
Use templ and vanilla CSS for the public dashboard. Responsive cards, native expandable tables, and inline SVG charts provide the interface without a JavaScript framework.
Separate monitoring from traffic
Background loops collect observations regardless of visitors. Opening the dashboard reads persisted results and never triggers another health check.
Architecture
Two paths through a shared source of truth
The monitoring path turns configuration into scheduled HTTP requests, then saves each observation with its derived state and incident changes. The public path reads a consistent database snapshot and renders it as HTML. Both paths run in one binary, with Caddy handling public HTTPS in front of the container.
01
Configuration and scheduling
Validated service definitions feed one sequential loop per endpoint. A shared semaphore bounds concurrent check-and-save cycles.
02
HTTP observation
A reusable HTTP client applies timeouts, verifies TLS, follows a bounded number of redirects, and measures time to response headers.
03
State and persistence
A pure transition function interprets consecutive results. SQLite commits the check, current state, and incident change together.
04
Public dashboard
Go handlers query persisted history and templ renders overall health, service cards, daily history, response times, and incidents.
Concurrency
Bound the work and make shutdown predictable
Sequential work per service
Each service has one goroutine, so its checks cannot overlap. A buffered channel limits concurrent cycles across services; the default is three.
No catch-up bursts
Initial checks are staggered. If a request or database write misses an interval, the loop skips elapsed slots instead of replaying a queue of overdue requests.
Cancellation is not an outage
Context cancellation stops requests and waiting loops without recording a failed observation. A request timeout, by contrast, is evidence about the endpoint and is recorded.
Finish completed work
A completed observation gets a bounded opportunity to commit during shutdown. WaitGroups join background work before the database closes, while the HTTP server drains active requests.
Incident detection
A failed request is evidence, not a verdict
The first or second consecutive failure makes a service degraded. The third declares it down and opens an incident. One successful recovery check moves it to recovering; a second consecutive success restores operational status and resolves the incident. A failed recovery keeps the original incident open.
Incident records distinguish the first failure from the check that confirmed the outage. Resolution is tied to confirmed recovery. These are observation timestamps, not exact measurements of when the application changed state.
Long monitoring gaps reset consecutive-check evidence without silently closing open incidents. The dashboard overlays unknown status when observations are missing or stale, preserving the last known phase instead of displaying old success as current health.
Persistence
Keep observations and conclusions consistent
Four focused tables
Services store stable identities and configuration. Checks retain raw observations. Service state preserves counters and important timestamps across restarts. Incidents retain confirmed outage history.
Atomic state changes
Each write transaction loads the latest committed state, applies the transition, and saves all related changes. A unique partial index prevents multiple open incidents for one service.
Simple connection policy
A single database connection serializes local work. WAL, foreign keys, a busy timeout, and short transactions keep the persistence behavior explicit. Embedded migrations travel with the binary.
Bounded retention
Raw checks are retained for 30 days and resolved incidents for 365 days after resolution. Daily cleanup deletes in batches of at most 1,000 rows per table; open incidents and current state are preserved.
Observability
Show what was measured, including what is missing
The dashboard combines an overall health summary with responsive service cards, observed uptime, daily history bars, recent response-time charts, and expandable check details. The display stays useful on smaller screens without an administration UI or client-side application.
Uptime means successful checks divided by completed checks over the last 30 days. It is not an exact time-based SLA. Missing observations are excluded, empty days are marked as no data, and daily counts expose how much evidence supports each block. Charts show recent successful request durations on a per-service scale, with failures represented as gaps.
Operational logs record check results, persistence outcomes, phase changes, and shutdown events as structured JSON. Readiness measures whether monitoring is making progress and can access its database: an application can be down while the monitor itself remains healthy.
Interface design and progressive enhancement
The public dashboard evolved from a conventional dark monitoring interface into a Liquid Glass-inspired design. Layered gradients, translucent panels, and subtle highlights add visual depth while keeping operational status, latency, and incident information prominent. The implementation uses templ and vanilla CSS rather than introducing a client-side UI framework.
Validation
Test failure paths as deliberately as successful checks
Controlled HTTP behavior
Local test servers cover response codes, redirect limits, timeouts, cancellation, untrusted certificates, and streaming bodies without depending on production endpoints.
Deterministic timing
Table-driven tests exercise outage and recovery sequences. Go's testing/synctest verifies concurrency limits, skipped intervals, shutdown coordination, and retention timing using simulated time.
Real database failures
SQLite integration tests reopen databases to verify restart continuity and inject an incident-write failure to prove that check history and state roll back together.
Public behavior
Handler tests cover empty, healthy, stale, and down states, escaped content, and database-unavailable responses. Browser checks exercised one-, two-, and three-column layouts and expanded tables.
Deployment
Fit the service into a working delivery pipeline
An Alpine Go build produces a static binary for a non-root scratch image. CA certificates support outbound HTTPS; templates, CSS, and migrations are embedded. Named volumes hold configuration and SQLite data while the runtime filesystem stays read-only.
GitLab generates templates, runs vet and tests, and builds images tagged with the commit SHA and latest. The existing production runner deploys the commit-tagged image on localhost port 3005, with Caddy routing status.asmale.me to it. The deployment waits for container health before reporting success.
The DNS cutover provided a useful operational lesson: HTTPS requests were succeeding while a browser still reported a missing hostname. Comparing public resolvers exposed inconsistent cached DNS results. Once those caches refreshed, the browser could reach the deployed site without another application change.
Results and limits
A live monitor with an explicit observation boundary
The system is deployed at status.asmale.me and collects recurring observations for six applications. It brings HTTP behavior, bounded concurrency, incident detection, durable state, and public reporting together in one small Go service.
The deployment also has deliberate limits. A single monitoring location cannot independently report the failure of the server or network it shares with the applications. A homepage returning HTTP 200 does not prove that every dependency or business operation works. The current deployment replaces the container with a short interruption rather than providing zero-downtime releases.
Lessons and next steps
Reliability depends on the meaning of the data
The most useful decisions were about evidence: when failures justify an incident, whether a canceled check should count, how old observations should be displayed, and what must commit together. Keeping those decisions explicit made the system easier to test and explain.
Next steps include verifying backup and restore against a separate volume, documenting rollback, and adding application-specific health endpoints where homepage checks are insufficient. Notifications, certificate-expiration checks, maintenance announcements, and Prometheus metrics remain future enhancements. Race detection requires a C compiler and is not part of the current Alpine pipeline.