# 5ML Agentic AI Platform --- ## Project Workflow - Before working: check `docs/` files. Run code-researcher only if docs don't have enough context. - After changes: `cd frontend && npm run build`, update the relevant `docs/` file, then commit and push directly - Current branch: `claude/add-claude-md-file-dtLHa` - Communication: results first, bullet points, TC+EN mix ok, code with comments ### docs/ — What to update - Changed DB table in `db.js` → `docs/DATABASE_SCHEMA.md` - Changed API endpoint in `index.js` → `docs/CODEBASE_MAP.md` - Changed agent/service → `docs/AGENTIC_STRUCTURE.md` --- ## Project Structure - **Backend**: `index.js` — Express API server (single file, ~2200 lines) - **Frontend**: `frontend/` — Next.js 14, App Router, TypeScript, Tailwind CSS - **Database**: PostgreSQL with pgvector for embeddings (**Fly Postgres only — no local DB or local API keys available**) - **Services**: `services/` — deepseek, rag-service, schedule-registry - **Knowledge**: `knowledge/` — VectorStore, EmbeddingService, SemanticSearch, connectors ## 7-Layer Agentic AI Architecture ``` ┌─────────────────────────────────────────────────────────┐ │ Layer 7 │ FRONTEND / UI │ │ │ Next.js 14, 60+ components, dashboard tabs │ │ │ frontend/app/, frontend/components/ │ ├─────────────────────────────────────────────────────────┤ │ Layer 6 │ API GATEWAY │ │ │ 120+ REST endpoints, auth, middleware │ │ │ index.js │ ├─────────────────────────────────────────────────────────┤ │ Layer 5 │ ORCHESTRATION │ │ │ Multi-agent coordinators, workflow managers │ │ │ agents/csoOrchestrator.js │ │ │ agents/smeGrowthOrchestrator.js │ ├─────────────────────────────────────────────────────────┤ │ Layer 4 │ AGENTS │ │ │ 20+ specialized AI agents │ │ │ agents/ → see docs/AGENTIC_STRUCTURE.md │ ├─────────────────────────────────────────────────────────┤ │ Layer 3 │ SERVICES │ │ │ LLM clients, schedulers, enrichment │ │ │ services/ (deepseek, perplexity, minimax, │ │ │ rag-service, scheduler, encryption...) │ ├─────────────────────────────────────────────────────────┤ │ Layer 2 │ KNOWLEDGE │ │ │ RAG, vector embeddings, connectors │ │ │ knowledge/ (VectorStore, EmbeddingService, │ │ │ SemanticSearch, Notion/Web/PDF/Email) │ ├─────────────────────────────────────────────────────────┤ │ Layer 1 │ DATA │ │ │ PostgreSQL + pgvector, 55+ tables │ │ │ db.js → see docs/DATABASE_SCHEMA.md │ └─────────────────────────────────────────────────────────┘ ``` **Agent list:** `agents/` directory or `docs/AGENTIC_STRUCTURE.md` **When adding a new agent:** place in `agents/`, wire endpoint in `index.js` (Layer 6), update `docs/AGENTIC_STRUCTURE.md` ## Build & Verify **Always use `npm`, not `bun` or `yarn`.** ```sh # 1. Frontend build (ALWAYS run before committing frontend changes) cd frontend && npm run build # 2. Backend start npm start ``` **Verification rule**: Never commit without a successful `npm run build` first. If the build fails, fix it before committing. **Environment**: All DB operations (seeding, image generation, API calls requiring Postgres) must run on the deployed Fly.dev instance. There is no local PostgreSQL, no local `DATABASE_URL`, and no local API keys (Gemini, DeepSeek, etc.). The local dev environment is frontend build + code editing only. To seed data or trigger image generation, either SSH into Fly or use the deployed web UI. ## Key Files - Dashboard tabs: `frontend/app/dashboard/page.tsx` - Cost/stats data: `/stats` endpoint in `index.js` (search `useCases:`) - Cron scheduling: `services/schedule-registry.js` - Workflow chat: `/api/workflow-chat` endpoint in `index.js` - Health checks: `/api/health/services` and `/api/health/services/:id` in `index.js` ## UI / Dark Theme Conventions - Cards: `dark:bg-slate-800/60` — NEVER `dark:bg-slate-800` (too harsh) - Subtle backgrounds: `dark:bg-white/[0.03]` or `dark:bg-white/[0.04]` - Hover states: `dark:hover:bg-white/[0.02]` — NEVER solid hover colors like `dark:hover:bg-slate-750` - Table row borders: `dark:border-slate-700/50` — not full opacity - Formula/code boxes: `dark:bg-white/[0.02]` ## Component Patterns - **No auto-fetch on mount**: Dashboard components must NOT call `fetch()` in `useEffect` on load. Let users trigger via button click (e.g., API health tab). - **Dashboard tabs over separate pages**: New features go as tabs in `frontend/app/dashboard/page.tsx`, not standalone pages. - **Canvas pan/zoom**: Use CSS `transform: translate() scale()` — never `overflow-auto` for pannable canvases. - **Node interactions**: Use `onMouseDown={e => e.stopPropagation()}` on interactive elements inside pannable canvases. ## Model Routing - **Primary**: DeepSeek Reasoner ($0.14/$0.28 per 1M tokens) — most agent tasks - **Fallback**: Claude Haiku ($0.25/$1.25 per 1M) — simple classification/extraction - **Research**: Perplexity Sonar ($3.00/$15.00 per 1M) — web search tasks - **Vision/complex only**: Claude Sonnet ($3.00/$15.00 per 1M) ## TEDxXinyi Admin / Media Library ### Fly.dev Ephemeral Filesystem — Critical Fly.dev machines have **ephemeral filesystems**. All files written at runtime (generated images, `.media-metadata.json`) are **wiped on every machine restart, deploy, or sleep/wake cycle**. `fly.toml` has NO volume mounts. **How it manifests**: Images appear in the admin media library after generation, but disappear "after a while" when the Fly machine restarts. **Current mitigations**: 1. **CDN fallback** (`index.js`): If a `/tedx-xinyi/*` image file is missing locally, middleware checks `.media-metadata.json` for a `publicUrl` and 302-redirects to mmdbfiles CDN 2. **Seed file** (`use-cases/tedx-xinyi/api/.media-metadata-seed.json`): Committed to git. On startup, if `.media-metadata.json` is missing, metadata is restored from this seed 3. **mmdbfiles backup**: After every metadata save, the JSON is backed up to mmdbfiles (debounced 5s) 4. **Admin not-missing logic**: Images with a CDN URL in metadata are shown as available even if the local file is gone **After generating images**: Always commit the updated `.media-metadata-seed.json` to git so CDN URLs survive deploys. **Recovery endpoints**: - `GET /api/tedx-xinyi/metadata-export` — download current metadata JSON - `POST /api/tedx-xinyi/metadata-import` — restore metadata from JSON (merges with existing) **Permanent fix**: Add a Fly persistent volume (`[[mounts]]` in `fly.toml`) to persist `/app/frontend/public/tedx-xinyi/`. ### Media Library Key Files - Routes: `use-cases/tedx-xinyi/api/routes.js` - Metadata: `frontend/public/tedx-xinyi/.media-metadata.json` (runtime, ephemeral) - Seed: `use-cases/tedx-xinyi/api/.media-metadata-seed.json` (git, persistent) - CDN fallback: `index.js` (after `/tedx-xinyi` static middleware) ## Known Mistakes to Avoid - Marketing strategy was set to 5 runs/day — should be 1 run/day ($3.60/mo not $18/mo) - API health tab auto-fetched on every tab switch — expensive and unnecessary - Cost analysis was a separate page — should be a dashboard tab - Dark hover backgrounds used `dark:bg-slate-750` — too harsh, use subtle opacity values - `overflow-auto` on workflow canvas prevented proper pan/zoom — use transform-based approach - **TEDx media metadata on ephemeral Fly filesystem** — stored in `.media-metadata.json` which is wiped on restart. Always commit seed file and use CDN URLs as source of truth - **Modifying TSX source files at runtime** has no effect — Next.js is pre-built at deploy time. Use CDN redirect fallback in Express middleware instead - **CRITICAL: Python `open(file, 'w')` truncates immediately** — opening a file with `'w'` mode wipes it before any write happens. If the subsequent `f.write()` throws (e.g. `UnicodeEncodeError` from surrogate characters when decoding `\u` escapes in JSX text), the file is left **empty with no recovery**. Always: (1) write to a temp file first, then `os.rename()`, or (2) build the full content string in memory and verify before opening for write. Never transform-and-overwrite in one step. - **`\u` escapes in JSX text nodes render literally** — `

\u5065\u5eb7

` renders as the literal string `\u5065\u5eb7` in the browser, NOT as Chinese characters. Unicode escapes are only interpreted inside JavaScript string literals (e.g. `{'健康'}` or `{'\u5065\u5eb7'}`). Always use actual Unicode characters directly in JSX text content. - **Agent-generated files with unicode escapes** — when delegating file generation to a subagent, always verify the output file contains real Unicode characters (not `\uXXXX` sequences) before building. Run `grep -c '\\u[0-9a-fA-F]\{4\}' file` — if count > 0, decode them with Python before the Next.js build. - **Real-ESRGAN ncnn-vulkan on CPU-only VMs (Fly.dev)**: ncnn's `create_gpu_instance()` explicitly skips `VK_PHYSICAL_DEVICE_TYPE_CPU` devices in its Vulkan enumeration loop. This means `mesa-vulkan-drivers` (Lavapipe/llvmpipe) is detected but immediately rejected as "invalid gpu device" — `-g -1` and `-g 0` both fail. **Solution**: use the Python `realesrgan` package + PyTorch CPU (`torch --index-url https://download.pytorch.org/whl/cpu`). PyTorch has zero Vulkan dependency and runs genuine AI upscaling on CPU. Do not add `mesa-vulkan-drivers` or attempt ncnn Vulkan workarounds for CPU-only environments.