# Deployment Runbook
<!-- Platform: Fly.io | App: 5ml-agenticai-v1 | Region: iad -->
<!-- last updated: 2026-03-25 -->

---

## Pre-Deploy Checklist

Run through this before every `fly deploy`:

### Code
- [ ] `cd frontend && npm run build` — build must pass clean
- [ ] No uncommitted changes (`git status` clean)
- [ ] On correct branch, up to date with remote

### Environment Variables
- [ ] All required `.env` vars are set in Fly secrets (`fly secrets list`)
- [ ] New env vars added to both `.env.example` and Fly secrets
- [ ] `DATABASE_URL` points to correct DB (not local)

### Ephemeral Filesystem — Critical
- [ ] `use-cases/tedx-xinyi/api/.media-metadata-seed.json` committed with latest CDN URLs
- [ ] `frontend/public/recruitai/` static assets committed (not generated at runtime)
- [ ] No runtime file writes expected without CDN fallback configured

### Database
- [ ] New DB tables/columns tested against production schema
- [ ] No destructive migrations without backup

---

## Deploy Command

```sh
# Standard deploy
fly deploy

# Check logs after deploy
fly logs

# Check app status
fly status
```

---

## Post-Deploy Verification

```sh
# 1. Health check
curl https://5ml-agenticai-v1.fly.dev/health

# 2. Service health (all integrations)
curl https://5ml-agenticai-v1.fly.dev/api/health/services

# 3. DB status
curl https://5ml-agenticai-v1.fly.dev/api/admin/db-status
```

Expected responses:
- `/health` → `{ "status": "ok" }`
- `/api/health/services` → list of services with status
- `/api/admin/db-status` → `{ "connected": true }`

---

## Rollback

```sh
# List recent releases
fly releases

# Roll back to previous version
fly deploy --image <previous-image-ref>
```

---

## Fly.dev Ephemeral Filesystem — How It Works

**The problem:** Fly machines restart on deploy, sleep, or crash. All files written at runtime are wiped.

**Affected files:**
- `frontend/public/tedx-xinyi/.media-metadata.json` — generated image metadata
- Any file written by API handlers at runtime

**Current mitigations:**

| Mitigation | File | How |
|-----------|------|-----|
| CDN fallback | `index.js` | If local file missing, 302 redirect to `publicUrl` in metadata |
| Seed file | `use-cases/tedx-xinyi/api/.media-metadata-seed.json` | Restored on startup if `.media-metadata.json` missing |
| mmdbfiles backup | `index.js` | After every metadata save, JSON backed up to CDN (debounced 5s) |

**After generating new images:** Always commit the updated seed file:
```sh
git add use-cases/tedx-xinyi/api/.media-metadata-seed.json
git commit -m "Update TEDx media seed with new CDN URLs"
```

**Recovery if metadata lost:**
```sh
# Export current metadata (if machine still running)
GET /api/tedx-xinyi/metadata-export

# Restore from backup
POST /api/tedx-xinyi/metadata-import
```

**Permanent fix:** Add `[[mounts]]` in `fly.toml` to persist `/app/frontend/public/tedx-xinyi/`.

---

## Environment Variables Reference

| Variable | Required | Purpose |
|----------|----------|---------|
| `DATABASE_URL` | Yes | PostgreSQL connection string |
| `ANTHROPIC_API_KEY` | Yes | Claude models |
| `DEEPSEEK_API_KEY` | Yes | Primary LLM |
| `PERPLEXITY_API_KEY` | Yes | Research agent |
| `OPENAI_API_KEY` | Optional | Embeddings fallback |
| `MINIMAX_API_KEY` | Yes | Image/media generation |
| `RECAPTCHA_SECRET_KEY` | Yes | Form protection |
| `GITHUB_WEBHOOK_SECRET` | Yes | CI/CD webhook |
| `DROPBOX_ACCESS_TOKEN` | Optional | Receipt processing |
| `SMTP_HOST/PORT/USER/PASS` | Optional | Radiance email |
| `META_ACCESS_TOKEN` | Optional | Ads dashboard |
| `GOOGLE_ADS_*` | Optional | Google Ads integration |

---

## Scaling & Limits

```toml
# fly.toml current settings
[http_service.concurrency]
  hard_limit = 250
  soft_limit = 200

[http_service]
  min_machines_running = 1
  auto_stop_machines = "stop"   # machines sleep when idle
  auto_start_machines = true
```

Health check: `GET /health` (grace: 120s, interval: 30s, timeout: 15s)