Personal Data Dashboard
Aggregate personal data from multiple APIs into a self-hosted dashboard with sync, charts, and privacy.
Phase Navigation
Design & Data Sources
Choose data sources, design the sync pipeline, and pick the chart library.
Why
A personal dashboard's value is in data freshness and source coverage. Connecting 5 sources (bank, fitness, calendar, GitHub, screen time) with a unified sync schedule gives a view no single app provides. Choosing the wrong chart library (heavy D3 bundle vs lightweight uPlot) kills load performance on mobile.
When
Before building charts. Spend 1 week listing sources, testing their APIs, and designing the unified data model.
How
1. List and prioritize data sources
- Tier 1 (daily): bank transactions, step count, screen time
- Tier 2 (hourly): GitHub commits, calendar events, weather
- Tier 3 (weekly): email stats, music listening, sleep summary
- Start with 2-3 Tier 1 sources for the MVP
2. Test each source's API
- Auth: OAuth (Strava, Google), API key (GitHub), screen scraping (bank if no API)
- Rate limits: note per-source limits (Strava 100/15min, GitHub 5000/h)
- Pagination and incremental sync: support ?since=lastSync cursor
3. Design the unified data model
- Every source maps to a common event: { source, timestamp, metric, value, meta }
- bank -> { metric: "spend", value: -42.50, meta: { merchant, category } }
- fitness -> { metric: "steps", value: 8234, meta: {} }
- Store in SQLite with an index on (source, timestamp)
4. Choose the chart library
- uPlot: 40KB, fastest for time-series, no interactivity beyond zoom
- ECharts: 800KB but full-featured (maps, heatmaps, interactions)
- Recharts: React-native, 200KB, good for composable dashboards
- Pick by bundle budget, not by feature count
5. Design the sync schedule
- Cron per source at its natural cadence (bank: 6h, fitness: 1h)
- Stagger jobs to avoid hitting all APIs at once
- Mark each source with lastSync and nextSync for the UI
Example
// Unified event model
interface Event {
source: "bank" | "fitness" | "github" | "calendar" | "screentime";
timestamp: number; // unix ms
metric: string; // "spend" | "steps" | "commits" | "focus_minutes"
value: number;
meta: Record<string, string | number>;
}
// SQLite schema
CREATE TABLE events (
source TEXT NOT NULL,
timestamp INTEGER NOT NULL,
metric TEXT NOT NULL,
value REAL NOT NULL,
meta TEXT, -- JSON
UNIQUE(source, timestamp, metric)
);
CREATE INDEX idx_events_query ON events(source, timestamp);
// Sync schedule (staggered)
cron: [
{ source: "fitness", every: "1h", offset: "0m" },
{ source: "github", every: "1h", offset: "15m" },
{ source: "bank", every: "6h", offset: "30m" },
{ source: "screentime", every: "6h", offset: "45m" },
];
Common Problems
- ⚠Connecting 10 sources at once — each has its own auth quirks, the MVP never ships; start with 2-3 Tier 1 sources
- ⚠Heavy chart library on mobile — ECharts 800KB bundle takes 3s to load on a phone, users abandon before the first chart renders
- ⚠No incremental sync — every run fetches all history, hitting rate limits and wasting the API budget
Best Practices
- ✓Start with 2-3 high-value Tier 1 sources (bank + fitness) and add one source per week — ship the MVP in week 2
- ✓Pick the chart library by bundle budget (uPlot < 50KB for mobile) not feature count — load time determines usage
- ✓Support incremental sync (?since=lastSync) for every source to respect rate limits and keep sync fast
Resources for this stage
Tools
Templates
Implementation & Sync
Build the sync jobs, data store, and reactive charts with loading and stale states.
Why
Sync is the hard part — APIs rate-limit, tokens expire, and partial syncs leave stale data. Without idempotent incremental sync and clear 'last updated' indicators, the dashboard shows wrong data silently and the user loses trust in every number.
When
After the data model is designed. Build sync + store + charts, then scheduling and stale-data alerts.
How
1. Build per-source sync adapters
- Each adapter: { auth, fetch(since): Event[], rateLimitInfo }
- Handle token refresh (OAuth) inside the adapter
- Return rate limit headers so the scheduler can back off
2. Make sync idempotent and incremental
- Upsert on (source, timestamp, metric) — re-running a sync is safe
- Pass the last synced timestamp as the cursor: fetch(since = lastSync)
- Cap each run at 1000 events; paginate if the source returns more
3. Handle failures gracefully
- On 401: refresh token, retry once, then mark source as "auth_expired"
- On 429: read Retry-After header, reschedule the job, back off exponentially
- On 5xx: retry 3x with jitter, then mark source as "degraded" (show stale data with a warning)
4. Build reactive charts
- Query the store with a time range; subscribe to store changes for live updates
- Show "last updated 2m ago" per source — stale indicator builds trust
- Loading skeleton while the first sync runs; error state if a source is degraded
5. Add the sync scheduler
- BullMQ + Redis: one queue per source, repeat job at the source's cadence
- Stagger offsets so all sources do not fire at the same second
- Dashboard: show next sync time and a "sync now" button per source
Example
// Idempotent incremental sync (Strava adapter)
async function syncStrava(since: number) {
const token = await refreshIfExpired("strava");
const activities = await fetch(
`https://api.strava.com/athlete/activities?after=${since}`,
{ headers: { Authorization: `Bearer ${token}` } }
).then(r => r.json());
const events = activities.map(a => ({
source: "fitness", timestamp: a.start_date * 1000,
metric: "steps", value: a.steps, meta: { type: a.type },
}));
// Upsert — re-running is safe (UNIQUE constraint)
await db.events.upsertMany(events, ["source", "timestamp", "metric"]);
return { synced: events.length, lastSync: Math.max(...events.map(e => e.timestamp)) };
}
// Stale indicator in the UI
function SourceBadge({ source }) {
const lastSync = useLastSync(source); // reactive
const stale = Date.now() - lastSync > 2 * 3600_000; // 2h
return <span className={stale ? "stale" : "fresh"}>
{stale ? "⚠️" : "✓"} updated {timeAgo(lastSync)}
</span>;
}
Common Problems
- ⚠Non-idempotent sync — a retried job inserts duplicate events, charts show double counts
- ⚠No stale indicator — a source silently fails and shows 3-day-old data, user trusts the wrong number
- ⚠All sources sync at the same cron tick — 5 APIs hit simultaneously, rate limits trip and the sync fails
Best Practices
- ✓Upsert on (source, timestamp, metric) so retried syncs are idempotent — duplicates are the most common dashboard bug
- ✓Show 'last updated X ago' per source and flag stale (> 2x expected interval) — trust requires transparency about freshness
- ✓Stagger sync offsets per source so they never fire simultaneously — respects rate limits and smooths load
Deployment & Privacy
Deploy as a self-hosted app or local-first site with encrypted storage for sensitive personal data.
Why
Personal dashboards aggregate sensitive data (bank, health, location). A multi-tenant cloud host is a breach target and a liability; self-hosted (Fly.io single-instance) or local-first (SQLite + sync) keeps data on your infrastructure. Encryption at rest is non-negotiable for financial and health metrics.
When
When the dashboard works with live data. Deploy self-hosted before connecting sensitive sources (bank, health).
How
1. Choose the deployment model
- Self-hosted VPS (Hetzner $5/mo): full control, you own the data
- Fly.io single-instance: easy deploy, scales-to-one, data on attached volume
- Local-first (Tauri app + SQLite): data never leaves the machine, sync via the user's cloud
2. Encrypt sensitive data at rest
- Use SQLCipher (SQLite + AES-256) for the events table
- Encryption key from the user's master password (argon2id KDF)
- Non-sensitive metadata (source names, sync times) can stay unencrypted for fast queries
3. Secure the OAuth tokens
- Store refresh tokens encrypted in the DB, never in env files or plaintext
- Use a per-deployment secret (openssl rand 32) as the encryption key
- Rotate the key on a schedule; re-encrypt tokens on rotation
4. Lock down the instance
- Put Cloudflare in front: WAF, rate limit, and TLS
- Auth: single-user Basic auth or a session cookie with a strong password
- No public signup — this is a personal app, not a SaaS
5. Backup and disaster recovery
- Nightly: encrypt the SQLite file and upload to Backblaze B2 or S3
- Test restore monthly: download, decrypt, and run the app against the backup
- Document the restore procedure so a lost VFS is a 30-minute recovery, not a data loss event
Example
// SQLCipher: encrypted SQLite
import Database from "better-sqlite3";
const db = new Database("data.db");
db.pragma(`key = "${process.env.DB_ENCRYPTION_KEY}"`); // AES-256
// events table is now encrypted at rest
// Fly.io deploy (single instance, attached volume)
# fly.toml
app = "my-dashboard"
[mounts]
source = "dashboard_data"
destination = "/data"
# data persists on the volume across deploys
// Nightly encrypted backup (cron)
const dump = db.backup("/data/snapshot.db");
const encrypted = await encrypt(dump, process.env.BACKUP_KEY);
await b2.upload("dashboard-backups/snapshot-" + date() + ".enc", encrypted);
// Cloudflare in front: WAF + rate limit + TLS
// Auth: single-user session cookie (no public signup)
app.use(session({ secret: process.env.SESSION_SECRET }));
Common Problems
- ⚠Hosting on a multi-tenant cloud with no encryption — a breach exposes every user's bank and health data
- ⚠OAuth refresh tokens in plaintext env files — a leaked .env gives full access to the user's bank API
- ⚠No tested backup — the VPS disk dies and 2 years of personal data is gone with no recovery path
Best Practices
- ✓Use SQLCipher (AES-256) for the events table and derive the key from the user's master password — sensitive data is encrypted at rest
- ✓Store OAuth refresh tokens encrypted in the DB, never in env files — a leaked config must not grant API access
- ✓Test the backup restore monthly — an untested backup is a hope, not a recovery plan
Resources for this stage
Tools
Templates
Full Developer Lifecycle
A 14-stage developer roadmap from idea to maintenance — visualize dependencies and track progress.
Next Step