Observability Hub

Problem

prj--personal-portfolio--v3 is a static platform behind CloudFront - four HTML surfaces (portfolio, blog, quiz, news-feed) plus two CDNs (news-data, assets) - not a process you can scrape. There is no app server, no /metrics, no request log on a box you SSH into. "Is it healthy?" lived in the AWS console. "Who is reading the blog?" defaulted to a third-party JS tracker and a cookie banner. "Did the quiz PWA just 5xx at the edge?" was a guess until someone emailed. "Is this month still cheap?" was another console, opened after the fact.

That is not an observability gap you close with Datadog because you have used Datadog at work. A personal JAMstack platform has a different failure mode: almost no runtime, almost no budget, and a privacy posture you actually have to mean. Prometheus against S3 is theatre. Google Analytics on a site that argues for craft is a contradiction. Kubernetes on a single VPS is resume-driven architecture. Cost Explorer as a daily habit is the AWS console again.

I needed a platform contract for telemetry: which questions each signal can answer, which it cannot, what ships before any hosting invoice, and how a fifth project joins later without a second stack.

Solution

prj--observability-hub is a separate repo on purpose. The portfolio Terraform owns how sites are served (distributions, private buckets, logging_config). This repo owns how those sites are observed - identity, local tooling, and later the always-on VPS. That split is the same seam as content-vs-app in the content pipeline: ownership follows change cadence, not a single "infra" folder.

The stack is phased with exit gates. Phase 0 runs entirely on the laptop against production CloudFront data and account-level CloudWatch billing estimates. Phase 0b is a JS beacon in the four HTML apps, queried into the same Grafana via Cloudflare's GraphQL API - still $0 hosting, still no VPS. Phases 1-6 are designed, spiked, and ADR'd; they are not shipped. Shipping a VPS before the CDN, cost, and visitor questions are answerable would be spending to feel busy.

Target shape, once the gates are met:

Scope and features

In scope

• Six CloudFront hostnames: four HTML surfaces (paulserban.eu, blog., quiz., news-feed.) plus news-data. and assets. CDNs
• CDN health without living in the AWS console (requests, 4xx/5xx, bytes, cache behaviour)
• Path-level traffic, referrers, status breakdown, and forensic access-log search
• Month-to-date estimated AWS spend in Grafana (not the invoice - CloudWatch EstimatedCharges)
• Privacy-friendly page analytics across the four HTML surfaces without a cookie banner - Cloudflare Web Analytics in Grafana now (Phase 0b), Umami later if we want to own the rows
• Real-user Core Web Vitals in the same pane as CDN requests, not only Lighthouse on a laptop or a second vendor UI
• Uptime checks that are not "curl from the same machine that is down"
• One Grafana as the eventual single pane; domain-agnostic so a fifth project is a site entry + a monitor, not a new platform
• Cost and complexity bounded to a single-node Compose stack

Out of scope (named, not forgotten)

• Distributed tracing / Grafana Tempo - static SSG has no server spans; the quiz PWA is not a mesh
• Real-time CloudFront logs via Kinesis - priced for a problem I do not have
• Multi-node HA, k3s, EKS - rejected in ADR-001
• Athena + Glue as the Phase 0 query path - superseded by ADR-002
• Selling this as a product or a team SRE platform - it is personal-scale, solo-operated, and should read that way

Honest privacy note: "privacy-friendly" here means no cookie consent theatre, not "no third party". Phase 0b sends beacons to Cloudflare. CloudFront standard logs still contain client IPs. Loki stores them in the log line (never as labels). ClickHouse can uniqExact(c_ip). That is CDN forensics plus a vendor dashboard, not "we have no identifiers". Pretending otherwise would be the kind of product copy I would reject in a design review. Umami (Phase 2) is the path to hold the event rows ourselves - it does not erase the Phase 0b choice.

Architecture

Two diagrams matter: the target platform, and what is actually running today.

Target (Phases 0-6)

Visitors
   │
   ▼
Route 53 -> CloudFront x 6 -> private S3
              │                    │
              │                    └── site objects (SSG / quiz SPA / JSON / media)
              ├── CloudWatch metrics (AWS/CloudFront, Region=Global)
              ├── CloudWatch billing (AWS/Billing EstimatedCharges, us-east-1)
              └── standard access logs -> s3://cf-access-logs.paulserban.eu
                                            │
                     ┌──────────────────────┼──────────────────────┐
                     ▼                      ▼                      ▼
              Grafana CloudWatch     Alloy -> Loki            clickhouse-local
              (health + estimated    (Grafana log UI)        (ad-hoc SQL CLI)
               spend)

Browser JS (Phase 0b, prod only)
  beacon.min.js -> Cloudflare Web Analytics
       │
       └── GraphQL rumPageload / rumWebVitals
             -> Grafana Infinity DS (visits, paths, LCP/INP/CLS p75)
             -> same time range as CloudWatch Requests (correlation, not a join)

VPS (not started) - Traefik TLS
  analytics.  -> Umami + Postgres     (owned page views; later optional CWV)
  status.     -> Uptime Kuma          (HTTP checks from outside the laptop)
  grafana.    -> Grafana              (CloudWatch + Infinity GraphQL + Prometheus + Loki + Umami)
  errors.     -> GlitchTip            (optional, Phase 6)
  Prometheus + node_exporter + Traefik metrics

Phase 0 / 0b as-built (shipped)

Laptop (127.0.0.1 only)
  grafana :3000
    ├── CloudWatch DS     (IAM keys, defaultRegion us-east-1)
    ├── Loki DS           (compose network only)
    └── Infinity DS       (CF_API_TOKEN -> api.cloudflare.com GraphQL)
  loki                 (filesystem, ~14d retention, no auth)
  alloy                (parse CloudFront TSV .gz -> Loki)
  cf-log-sync          (aws s3 sync every 300s)

Provisioned dashboards
  CloudFront health          CloudWatch + Loki
  CloudFront traffic         Loki
  AWS Billing                CloudWatch AWS/Billing
  Cloudflare Web Analytics   Infinity GraphQL + CloudWatch overlay

AWS
  CloudFront x 6  ->  CloudWatch
                  ->  s3://cf-access-logs.paulserban.eu  (60-day lifecycle)
  AWS/Billing EstimatedCharges  (us-east-1; billing alerts enabled once)
  IAM user observability-hub-readonly
    CloudWatch GetMetric* + CloudFront List/Get + S3 GetObject on the log bucket
    (covers AWS/Billing; no extra IAM for the cost dashboard)

Not in Compose
  docker run clickhouse/clickhouse-server clickhouse-local
    s3() table function -> saved SQL in infrastructure/local/clickhouse/queries/

Ownership is a first-class architecture decision, not a README footnote:

A clone of the hub repo cannot enable logging. That is correct: logging is a property of the edge, not of the dashboard.

Approach

As architect and lead implementer I treated observability the way I treat hosting and CI: seams first, spend second, orchestration last.

Platform questions before tool shopping

The original notes were a tool list (Umami vs Plausible vs Cloudflare Web Analytics, UptimeRobot vs Kuma, Sentry vs GlitchTip). That is how you accidentally buy four SaaS free tiers that do not compose. I inverted it: what questions must this platform answer, then which signal is allowed to answer them.

If a tool cannot be placed on that table, it does not enter the compose file.

Validate before you rent

Phase 0 is the design spike the console-first hosting case study already argued for: prove the contract on real traffic before freezing spend. CloudFront logging is pennies of S3. CloudWatch basic metrics are free. Docker on the laptop is free. A VPS is not free, and Lightsail's $5 bundle will not hold this stack anyway.

Right-size the runtime

Spikes killed three fashionable defaults:

• Prometheus as the first tool - pull-based metrics need a scrape target. A static origin has none. Grafana's native CloudWatch data source answers CDN health without an exporter. Prometheus earns its keep in Phase 5 for the VPS and Traefik, not for the sites.
• Kubernetes - one node, one replica of everything. k3s would tax ~700MB-1GB before a container starts and roughly double VPS cost. ADR-001 accepts Compose and names the revisit trigger (actual multi-node need, or a deliberate homelab/content goal - not "it looks good on a CV").
• Athena - pay-per-query and a Glue catalog for a volume I can scan from the laptop. ADR-002 picks clickhouse-local via Docker; no binary install, no catalog, transferable SQL.

Traefik over Caddy is the other proxy call: label-based Docker routing, no shared Caddyfile to edit per phase, and it matches the local Traefik mesh already used for the four apps. Consistency of operations beats a slightly simpler ACME story.

Two query paths on purpose

Grafana does not speak ClickHouse. Interactive access-log dashboards are Loki. Exact unique IPs, top IPs, and edge POPs are ClickHouse CLI. That looks like indecision. It is cardinality and operational weight:

• Loki labels are only job, site, status, result_type. Path, IP, UA, referrer stay in the log line (logfmt). Promoting ip to a label would explode the index the first week a crawler showed up.
• A persistent ClickHouse server on a 2-4GB VPS duplicates Loki and eats RAM (~1GB baseline) for no new Grafana panel I need in Phase 0.
• User-Agent is left out of Alloy's logfmt (spaces and = break naive parsing). UA analysis is ClickHouse or it does not happen. That is a documented hole, not a surprise.

Stack (Phase 0, real)

Phased delivery

Each phase has an objective, decisions, an observation contract, extractable metrics, an outcome, and the limits I would put in a design review. Do not start the next VPS phase until the current gate is met. Phase 0 has met its gate. Phase 0b is live in the same Grafana - it did not wait on Phase 1.

Phase 0 - Local AWS log aggregation (done)

Objective. Prove CDN/traffic visibility from data AWS already has, on one machine, before renting anything. What shipped. CloudFront standard logging on all six prod distributions into cf-access-logs.paulserban.eu (60-day lifecycle). Read-only IAM via Terraform. Local Compose: Grafana + Loki + Alloy + cf-log-sync. Four provisioned dashboards. Seven saved ClickHouse queries behind ./scripts/query.sh. Makefile targets matching the portfolio repo's compose_up habit.

Decisions.

• Native CloudWatch Grafana DS, not YACE. Phase 0 has no Prometheus. An exporter would be a service to run so that I could scrape a translation of an API Grafana already speaks.
• Billing is a CloudWatch namespace, not Cost Explorer. AWS/Billing EstimatedCharges already lands in us-east-1 once Receive CloudWatch Billing Alerts is on. The existing GetMetricData IAM covers it. CUR + Athena would be a second data path for a question the estimate already answers: "is this month still cheap?"
• Loki practiced early. The written plan parked logs in Phase 6. Grafana still needed a log UI once Athena was rejected and ClickHouse was CLI-only. Pulling Phase 6's ship path forward locally is non-linear on the Gantt chart and correct on the architecture.
• Poll S3, do not stream. cf-log-sync is a shell while true; aws s3 sync; sleep 300. It is crude. It is also the right cost model: standard logs already lag minutes to hours; a 5-minute sync does not make them worse in a way a visitor would notice.
• IAM user + access keys on the laptop. Fine for a bind-to-localhost stack. The wrong long-term pattern on a VPS (Lightsail instance profiles are weak; this will get revisited in Phase 1, not papered over).
• Local Terraform state. One operator, one laptop. Same named trade-off as the CI/CD case study.

What you can actually observe.

• Whether each distribution is taking traffic, erroring, or shipping bytes - coarse, per-distribution, not per-path
• Month-to-date estimated AWS charges, total and by service (CloudFront, S3, Route 53, CloudWatch, …) - estimate, ~6h cadence, resets each billing cycle
• Cache hit ratio and result types (Hit / Miss / Error / RefreshHit / …) from logs, which CloudWatch will not give you at URI granularity
• Top content vs asset paths, bytes by path, internal vs external referrer hosts, direct vs referred
• 404s over time, top client IPs in Grafana (from logfmt, high-cardinality but not labelled)
• Exact unique IPs, top IPs with path diversity, edge POP codes - CLI only

Metrics you can extract today.

Live check after logging was enabled: top paths populated; roughly 84% 200s already visible. That is a real number from production objects, not a dashboard screenshot of fixture data.

Outcome. Exit gate met: local Grafana answers "is CloudFront serving cleanly this week"; saved SQL answers "top paths and error rate, last 7 days"; billing answers "is this month still cheap" without opening Cost Explorer. All reproduce from a clean laptop with keys in .env. Hosting spend: $0.

Limits I will not dress up.

• This is not real-time. Standard access logs lag. The sync adds up to five more minutes. If I needed seconds, I would pay for real-time logs - I do not.
• Unique IPs are not unique humans. Carrier NAT, IPv6 privacy addresses, office egress, and bots all inflate or collapse that count. Treat it as a CDN forensic, not a marketing KPI.
• CloudWatch cannot tell you /portfolio/prj--…/ is hot. If the health dashboard looks green and a slug is 404ing, you need the traffic dashboard. That split is the product.
• EstimatedCharges is not the bill. No tax, credits, reservations, or per-resource/tag split. SEARCH labels need AWS dynamic labels (${PROP('Dim.ServiceName')}), not Grafana {{ServiceName}} templates - those stay literal on CloudWatch SEARCH.
• Grafana is admin/admin in .env.example, no SSO, no alerting, bound to localhost. You still have to look. Phase 0 does not page anyone.
• Alloy drops CloudFront #Version/#Fields headers and does not index UA. Incomplete parse is a choice; it is also a footgun if someone greps Loki for browsers.
• I practiced Phase 6 before Phase 1. The plan document is slightly behind the repo. The ADRs and README are the source of truth - that is how this should work, and it is also drift I have to maintain.

Phase 0b - Cloudflare Web Analytics in Grafana (live)

Objective. Page views, referrers, and coarse Core Web Vitals from real browsers, on the four HTML production hostnames, without renting a VPS - and in the same Grafana that already shows CloudFront, not a second bookmark.

This is the original napkin item I first treated as "still a third party, wait for Umami". That was the wrong gate. Visitor analytics is a script tag. Coupling it to Lightsail would have left a product question unanswered for a purity goal Phase 0 already broke (AWS has the IPs). ADR-003 records the reversal. The how-to lives in the hub repo: 02 - adding-cloudflare-web-analytics.md.

ADR-003 originally said there was no API worth wiring and the dashboard would stay in Cloudflare. That line aged badly. Cloudflare's GraphQL Analytics API exposes account-level RUM nodes (rumPageloadEventsAdaptiveGroups, rumWebVitalsEventsAdaptiveGroups). The consequence in the ADR is now: Grafana can read the aggregates; it still cannot join a beacon to a CloudFront request.

Decisions.

• Manual snippet, not orange-cloud auto-inject. The sites are CloudFront. Cloudflare cannot rewrite HTML at the edge. Pretending otherwise is a silent no-op. Zone HTTP analytics (httpRequestsAdaptiveGroups) would be empty for the same reason - these hostnames are not proxied. RUM is account-scoped, not zone-scoped.
• One token per hostname in the apps; requestHost in Grafana. Same isolation as one distribution per surface. Sharing a beacon token would mix portfolio and quiz. Filtering GraphQL by the JS snippet token as siteTag returns [] - they are different IDs. Hostnames are the stable join key with CloudFront panels.
• Production only. Tokens as GitHub Environment variables on production. Local Traefik, DEV Pages, TEST, STAGE stay unset so fixture traffic never lands in the series.
• Quiz gets spa: true. TanStack Router uses the History API. Hash routing is unsupported by the beacon - do not switch.
• Infinity, not a custom exporter. Grafana talks to https://api.cloudflare.com/client/v4/graphql through the Infinity datasource (Bearer token, Account Analytics: Read). Pin 3.7.1 on Grafana 11.5.2: Infinity 4.x externalizes react/jsx-runtime and needs Grafana ≥11.6.11. Unpinned GF_INSTALL_PLUGINS silently installed 4.0.0 and the datasource UI 404'd.
• Correlate, do not join. A dashboard row puts CloudWatch Requests (all six distributions) beside Cloudflare page views (four beacon hosts) on the same time range. Edge hits include assets, bots, and health checks. Page views are browsers that ran JS. If those series match, the pipeline is wrong.
• Umami is not cancelled. Phase 2 is still how we own rows. After a week of overlap, keep or drop Cloudflare - that is a revisit trigger, not a forever vendor.
• Phase 3 still shrinks. Grafana now draws LCP / INP / CLS p75 from the same beacon. A second web-vitals pipeline is dual-instrumentation until we need owned histograms or non-Chromium coverage.

What you can actually observe. Visits and page views for browsers that execute the snippet. Top paths, referrers, browser/OS, country. Core Web Vitals p75 with Google's good / needs-improvement / poor bands. Quiz client-route changes, if spa: true is set. Side-by-side CDN vs human traffic for the last 7 days.

Metrics.

Live check after wiring GraphQL: unfiltered last-7-day pageloads on the account were on the order of ~800 page views / ~750 visits, concentrated on blog and apex. That is production beacon traffic in Grafana, not a screenshot of Cloudflare's UI.

Outcome (gate). One real production page view visible per HTML hostname in Grafana; zero from local.* / test.* / stage.*. Snippets ship from the apps; the local stack queries them. assets and news-data have no beacon - they stay CDN-only, which is correct.

Limits I will not dress up.

• Data leaves the AWS account. Cookie-less is not "no third party". Aggregates only; minutes of ingest lag; RUM datasets are US-only under data localization.
• Under-counts privacy-hardened browsers and anything that never ran JS. Will disagree with access logs (assets, bots, 5xx). The correlation row is there to make that disagreement visible, not to hide it.
• CWV is Chromium-first. Safari/Firefox are incomplete. Treat GraphQL LCP/INP as µs or 960ms becomes "16 minutes" in Grafana. -1 is "no interaction sample", not a duration.
• Tokens are in public HTML. Treat them as identifiers, not secrets, but still do not commit them. The Grafana secret is CF_API_TOKEN, not the snippet token.
• If a CSP is added at CloudFront later, allowlist static.cloudflareinsights.com and cloudflareinsights.com or the beacon dies silently.
• This does not replace Phase 0. Turn off CloudFront logging after the beacon and you have lost CDN truth.

Phase 1 - VPS and platform foundation (not started)

Objective. Host the pieces that must be public and always-on: uptime from outside my network, later Grafana and Umami. Visitor analytics no longer waits on this box - that is Phase 0b.

Decisions (designed, not applied).

• Lightsail over Hetzner/Contabo/Oracle free as the default, because the rest of the platform is already AWS: one bill, no cross-account IAM, Terraform aws_lightsail_* in the same provider. Contabo wins raw €/GB. Oracle Always Free is capacity-constrained and ARM-only. This is account cohesion, not cheapest-possible.
• Do not buy the $5 Lightsail bundle. The plan table's "$5-10" line is a trap. Umami+Postgres+Grafana+Loki needs ~2-3GB; budget 2GB ($12) or 4GB ($24). Under-sizing this is how you learn about swap thrashing instead of telemetry.
• Terraform owns the box, Compose owns the processes. Instance, static IP (Lightsail public IP changes on stop/start), ports 22/80/443, Route 53 records. Containers stay in git. Managing Grafana as a Terraform resource would be the wrong abstraction.
• DNS ownership must be picked once. analytics. / status. / grafana. / errors. A records either live in hub Terraform (pass hosted_zone_id) or stay in the portfolio prod stack. Two roots fighting over paulserban.eu is how you get a Saturday outage.
• Lightsail IAM is weaker than EC2 instance profiles. Phase 0's access keys on a VPS is the lazy path. Call that out at apply time; do not discover it after the box is public.

What you will be able to observe. Almost nothing new about the sites. This phase observes whether the platform can exist: TLS on a placeholder, Compose reproducible from a clone, SSH key-only, firewall tight. The exit gate is https://status.paulserban.eu (or equivalent) with a valid cert - a platform heartbeat, not a product dashboard.

Metrics. Instance up, Traefik ACME success, port policy. Visitor metrics already come from Phase 0b; this gate is still about the box, not about page views.

Limits. A public Grafana/status host is a new attack surface. Basic auth at Traefik is the v0 answer; Authelia is a later hardening phase I already wrote down and should not pretend is in Phase 1. Single AZ, no HA: if Lightsail in eu-central-1 dies, analytics and uptime die together. For a personal site that is acceptable; for a team SLO it would not be.

Phase 2 - Owned page analytics / Umami (not started)

Objective. Hold page-view rows ourselves, still without a cookie banner, across all four domains.

Decisions.

• Umami + Postgres, one instance, one site entry per hostname. Cloudflare Web Analytics is already the third-party bridge (Phase 0b). Umami is data ownership, not "first analytics". Plausible hosted is $9/mo for a problem Compose already solves.
• Same layout include points as the Cloudflare beacon - the three BaseTemplate.astro files and quiz main.tsx. There is no shared layout package; consistency is a checklist, not a magic shared component.
• Postgres now, ClickHouse later. Umami can swap the events table to ClickHouse at higher volume. Personal traffic does not need that.
• JS still runs in the browser. Cookie-less is not zero-cost. Two snippets (Cloudflare + Umami) is worse. After ~7 days of overlap, keep or drop Cloudflare (ADR-003 revisit). Do not run both forever out of cowardice.
• Expect disagreement. Umami vs Cloudflare vs access logs are three definitions of "a visit". Matching 1:1 means the pipeline is wrong.

What you will be able to observe. Sessions and page views for browsers that execute the Umami snippet. Referrers as the client reported them. Per-site breakdowns. Still not bots that skip JS. Still not CloudFront cache hits. Still not 5xx that never reached HTML.

Metrics. Page views, unique visitors (Umami's definition, not uniqExact(c_ip)), top pages, referrers, bounce-ish rates, per-hostname series.

Limits. Client-side analytics under-counts privacy-hardened browsers. It will not replace Phase 0. Anyone who turns off CloudFront logs after Umami is live has misunderstood the architecture. Dual-running with Cloudflare without a kill date is how you collect two dashboards and trust neither.

Phase 3 - Owned RUM / Core Web Vitals (optional, default skip)

Objective. Know P75 LCP / CLS / INP from real visits in a system we own - only if Cloudflare's CWV view (Phase 0b) is not enough.

Default: skip or shrink. Grafana already draws LCP / INP / CLS p75 from Cloudflare GraphQL. Dual-instrumenting CWV so we can say "we own the histogram" is vanity until Cloudflare's view is insufficient.

If revisited (Umami live, Cloudflare dropped, or we need owned histograms / non-Chromium / Grafana overlay):

• web-vitals + navigator.sendBeacon, ingested as Umami custom events - simplest path, no new service.
• Datadog RUM stays skipped. Cloudflare is the cheap RUM we actually turned on. Datadog would be the k8s mistake again.
• Lab scores stay in CI if at all. Field vs lab are different questions.

What you will be able to observe (if this phase runs). Field CWV for visits that completed the beacon, in a store we own. Missing data from bounced/abandoned loads is structural. No server timing.

Metrics. P75 LCP, CLS, INP per hostname, 7-day trend, sample count. Exit gate is a 7-day trend on at least one domain outside both the Cloudflare UI and the GraphQL overlay we already have.

Limits. Umami custom events are a crude histogram. Cloudflare already has element-level debug, and Grafana already has the p75 overlay. If I run this phase just to say "we have RUM in Grafana," I am lying about the requirement - that overlay shipped in Phase 0b.

Phase 4 - Uptime / synthetic monitoring (not started)

Objective. Know within minutes if a hostname is down, checked from outside the laptop.

Decisions.

• Uptime Kuma on the VPS, HTTP (+ optional keyword) monitors for all four public URLs, alert to Telegram/Discord/email.
• Skip Route 53 health checks. They cost money to duplicate a container I am already running.
• One vantage point. This is "up from the VPS region" (planned eu-central-1), not Catchpoint. CloudFront can be fine in Frankfurt and broken at a single POP. Kuma will not see that. Access-log POP queries and CloudWatch 5xx will. Use both; do not pretend Kuma is global synthetics.

What you will be able to observe. Binary up/down, latency from one probe, SSL expiry if configured, alert delivery. Not "the page is slow for users in Brazil". Not "the quiz client route 404s" unless you keyword-check a specific path.

Metrics. Uptime %, probe latency, time-to-alert on a deliberate outage (the actual exit gate). If I cannot produce an alert by breaking a monitor on purpose, the phase is not done - dashboards without pages are decoration.

Limits. Putting the probe on the same VPS as Grafana means a dead VPS looks like "all sites down" and "monitoring down" at once. For personal ops that is acceptable. A lead signing off a paid SLO would put probes off-box. I am not signing off a paid SLO.

Phase 5 - Infra metrics and unified Grafana (not started)

Objective. Migrate the four Phase 0 / 0b dashboards onto the always-on VPS, add VPS health, one URL for "is the box healthy, is the CDN clean, is the bill still cheap, are visits and CWV in range".

Decisions.

• Prometheus + node_exporter for the VPS. This is the first time Prometheus has a legitimate scrape target. Traefik's metrics endpoint joins it. That is the opposite of Phase 0's "do not invent exporters for static files".
• Keep Grafana's CloudWatch DS unless PromQL unification is actually painful. YACE is optional. Phase 0 already proved native CloudWatch works. Adding YACE so that every panel is PromQL is aesthetic unless I am alerting from Prom rules across CDN + node.
• Umami Postgres as a Grafana data source for owned traffic overlays - possible, not mandatory. Cloudflare GraphQL already covers the bridge. Do not build a second analytics UI in Grafana if Umami's own UI is enough.

What you will be able to observe. VPS CPU/mem/disk, Traefik request/latency, CloudFront health and estimated spend (same as Phase 0, now always on), Cloudflare visits / CWV p75 already in Grafana from 0b. Still not a trace. Still not unique humans. Phase 3 only adds owned histograms.

Metrics. node_*, Traefik entrypoint/router series, existing CloudWatch KPIs, optional SQL against Umami. Exit gate is one Grafana URL, not five bookmarks.

Limits. Always-on Grafana is now a public (or Traefik-auth'd) surface. Provisioned dashboards must travel with the repo; click-ops panels on the VPS will drift. Retention on Prometheus needs an explicit answer or the disk fills. I have not picked a remote_write/Grafana Cloud hybrid - and I should not "just add Grafana Cloud" to feel enterprise.

Phase 6 - Log aggregation and error tracking (optional, highest effort)

Objective. Centralize logs and capture client JS errors when something actually breaks.

Decisions.

• Most likely phase to skip. Phases 0-5 already cover traffic, RUM, uptime, infra. Build this on the VPS only if log-level debugging is a real pain.
• The parse pipeline is already written. Alloy config, Loki labels, dashboards - Phase 0 did the hard part locally. VPS work is shipping + Traefik JSON access logs, not invention.
• GlitchTip only if Astro islands / the quiz PWA throw enough client errors to care. A static blog with almost no JS does not need a Sentry-compatible service. The quiz PWA is the plausible consumer. One DSN per app, not a single dumped bucket.

What you will be able to observe. LogQL over CloudFront + Traefik from anywhere with Grafana access. Deliberate JS errors in GlitchTip within a minute (exit gate, if deployed). Still no distributed traces.

Metrics. Same access-log metrics as Phase 0, plus Traefik status/latency from logs, plus GlitchTip event rate / affected users if present.

Limits. Loki on a small VPS with CloudFront log volume is the first thing that will OOM if retention and label cardinality are sloppy. The Phase 0 label discipline (filename dropped, ip never a label) is load-bearing - it is not style. Skipping this phase is a valid architectural outcome, not a failure to finish the roadmap.

Design decisions that mattered

• Gates over a big-bang compose file. A 12-service docker-compose.yml on day one would have hidden that CloudWatch already answered half the CDN questions.
• Compose over k3s. Single-node, single-replica, cost-sensitive. ADR-001 is the lead-engineer artifact: alternatives table, resource tax, revisit trigger.
• clickhouse-local over Athena. Same $0 constraint, less AWS surface, real SQL, CLI-shaped. ADR-002. Grafana stays on Loki.
• Two repos, two owners. Logging config is an edge concern; dashboards are an observability concern. Crossing that line is how you get circular Terraform and "just apply both".
• Cardinality as a first-class constraint. Loki labels bounded; high-cardinality forensics in ClickHouse. This is the same instinct as not putting user-id on Prometheus labels at work - applied at personal scale on purpose.
• Traefik for label routing, matching local app-dev, so Phase 1 does not invent a second proxy story.
• Cloudflare Web Analytics now, Umami later - and Grafana reads the aggregates. Visitor analytics is a script tag. Waiting for a VPS was the wrong gate (ADR-003). The first ADR draft said there was no API; GraphQL rum* nodes exist. Filter by requestHost, not the snippet token. Pin Infinity to the Grafana you actually run.
• Cost is a telemetry question. AWS/Billing is already in the CloudWatch DS we provisioned for CloudFront. CUR is a later honesty about invoices, not a prerequisite for "is this month still cheap."
• Trade-offs named up front. Long-lived IAM keys on the laptop; local TF state; log lag; no HA; no traces; Phase 6 optional; Lightsail IAM ugliness deferred to Phase 1; Cloudflare as a third party for $0 visitor/RUM; estimated charges ≠ invoice; LCP in microseconds; Umami JS cost vs ownership; one-probe uptime. A case study that hides those is a brochure.

What shipped

Phase 0 is done. Phase 0b is live in the same local Grafana:

• CloudFront access logging on six prod distributions into a shared bucket (in the portfolio Terraform, 60-day lifecycle)
• Terraform IAM user observability-hub-readonly (CloudWatch metrics including billing, CloudFront list, S3 log-bucket read)
• Local Compose: Grafana 11.5, Loki 3, Alloy, cf-log-sync; Infinity 3.7.1 pinned for Cloudflare GraphQL
• Dashboards: CloudFront health (CloudWatch KPIs + Loki cache/error/miss ops), CloudFront traffic (paths, referrers, forensics), AWS Billing (MTD EstimatedCharges + per-service SEARCH), Cloudflare Web Analytics (visits, paths, CWV p75, CDN correlation row)
• clickhouse-local query pack: top-paths, status-breakdown, cache-hit-ratio, referrers, unique-ips, top-ips, edge-pops
• ADRs for orchestration, query engine, and Cloudflare Web Analytics (GraphQL consequence updated); spikes that killed k8s-first, Prometheus-first, and Athena-first
• How-to for Phase 0b: four tokens, layout insertion points, prod-only CI, quiz SPA flag; Grafana filters by hostname, not snippet token
• make compose_up / compose_down / logs - operable, not a pile of undocumented containers

Phases 1-6 exist as a gated plan. Phase 3 is optional: Grafana already has Cloudflare CWV p75. I would rather show a finished Phase 0, a Grafana-backed 0b, and a contract than a half-installed VPS.

Outcome

The platform has an observability contract that matches how the sites are actually built: static at the edge, logs in S3, metrics in CloudWatch, visitor beacons as a $0 Cloudflare bridge read back into Grafana, questions split by what each signal can tell the truth about. Phase 0 de-risked the CDN half and put estimated spend next to health. Phase 0b refuses to wait on a VPS for "is anyone reading this" - and refuses a second vendor tab for the answer.

What a lead should see is scope control: reject k8s and Prometheus-as-identity, split ownership from the serving stack, reverse a purity call when it delayed a real question (ADR-003), correct the ADR when "no API" turned out to be GraphQL, and write down what you will not observe. The remaining phases are sequenced so each one adds a question the previous phase cannot answer - not another logo on a slide.

Part of prj--personal-portfolio--v3

This hub is the telemetry counterpart to the monorepo's serving and delivery seams:

• AWS multi-subdomain hosting - the CloudFront x 6 topology whose logs, metrics, and estimated spend this stack reads
• Multi-env CI/CD & Terraform delivery - prod is where logging is enabled; other envs are not the observability target
• Content pipeline SSG - four HTML surfaces from one artifact, plus JSON/media CDNs; this stack watches those six hostnames
• Local HTTPS Traefik + Docker Compose - the proxy pattern Phase 1 reuses on the VPS
• Spaced-repetition quiz PWA - the SPA most likely to justify GlitchTip later