Problem
My data was spread across a dozen apps. Recovery and sleep in Whoop, runs in Strava, meetings in Google Calendar, tasks in Things and Asana, time in Toggl, code in GitHub, notes in Obsidian, and screen time locked inside macOS. Each app answers questions about itself. None of them answers "does a late night on my laptop show up in tomorrow's recovery?"
I wanted one place that holds all of it, in a shape I can query with SQL, and that any agent or app can read without its own integration to every source.
Constraints
- Some sources only exist on my Mac. Screen Time, Things and the Obsidian vault have no API. They're read from local files, so the collector has to run on the laptop.
- The laptop sleeps. Anything reading from it over Tailscale only worked while it was awake, which is rarely when I pull my phone out.
- Sources don't agree on change. Some have a cursor or sync token. Others, like Things, only hand back the full current list, with no record of what was deleted.
- Whoop rotates its refresh token on every use. Two processes polling it would fight over the token and lock each other out.
- Agents get read access only. Writes, multiple statements,
PRAGMAandATTACHare all rejected before a query runs. - The site and my devices hold different keys. Revoking the website's access must not break my own.
Architecture
Connectors
14 across 11 sources
Collector
Bun + launchd on the Mac
life.sqlite
system of record
Edge replica
Worker + D1, /v1
Readers
MCP, Lumen, CLI, this site
Every connector maps its source into one of three shapes: a sample (a reading at a point in time, like a recovery score), an interval (a span, like a workout or an hour in an app) or an entity (something with a lifecycle, like a task or a calendar event). The raw payload is kept verbatim next to it, so I can re-run normalization later.
Connectors come in two modes. Incremental ones return what changed since a cursor. Snapshot ones return the full live set, and the engine tombstones anything that was there last time and isn't now.
A sync runs in two phases. Fetches run concurrently and hold no lock. Writes are serialized, one transaction per connector, and skip any row whose payload hash hasn't changed.
The MCP server exposes three tools over stdio: describe_schema, query_sql and sync_status.
The read API started as Hono and ArkType on the Mac. In August I moved it to a Cloudflare Worker on Effect's HttpApi, deployed with Alchemy, over a D1 replica. D1 is SQLite, so the same migrations apply unchanged. The collector pushes deltas after each run. Rows are upserted on id, so a re-sent batch is a no-op and an interrupted push just repeats.
What shipped
- Collector. 14 connectors over 11 sources: Whoop (recovery, sleep, workouts, day strain), Strava, Google Calendar, Things, Asana, Toggl, GitHub, Spotify, Obsidian, Screen Time and weather. It runs as a launchd agent, with backups, an off-device copy and staleness alerts.
- MCP server. Read-only SQL over the whole store, capped at 1,000 rows.
- Edge API.
/v1resource and aggregate endpoints on Cloudflare, with OpenAPI docs, separate read, replicate and ingest keys, and Cloudflare's rate limiter. A parity harness compares it against the local Hono API: 37 of 37 responses identical, auth rejections included. Weather is now polled by a Worker cron: Open-Meteo has no history, so every hour the laptop slept was lost. - Lumen. A Tauri menu-bar app that wraps a React dashboard as a sidecar, with today's recovery in the tray. It also has a local "Ask your data" view that turns questions into SQL with qwen2.5-coder on Ollama, and a
lifeCLI. - This site. The Whoop numbers on einargudni.com come from the edge API, on their own read key.
The test count is 262 test and it calls across 49 files, counted from the source.
What I'd do differently
- Make
synced_atmean "last changed" from the start. Things reports every open task on every tick, so the column first meant "last seen". Once the replica selected rows by it, every push would have sent the whole table. Fixing it meant an upsert that only writes when a field differs, plus restamping tombstones. - Poll each source in exactly one place. I planned to move all the HTTP connectors to Cloudflare cron. The data said only weather needed it. Everything else catches up from a cursor or a snapshot, and two Whoop pollers would break each other's refresh token.
- Mark where each row came from before replicating both ways. A log line or weather reading pulled from the edge would be pushed straight back. An
origincolumn on each row stops the loop. - Test the refusals, not only the successes.
/v1/sourcesreturned a 500 instead of a 401 and still passed the parity harness, because the harness only sent authenticated requests. - Don't trust whatever is listening on a port. The desktop app used to accept anything on :4321 as its dashboard. It now supervises its own sidecar, and the sidecar exits when Lumen does.