Tracker/Code/TODO.md

TODO

Phase 0: Infrastructure (DONE)

• Azure VMs created and configured (jt-ub, jt-win)
• Ubuntu VM provisioned (Docker, Go, Python, rclone, zsh, dotfiles)
• Directory structures on both VMs (William's layout)
• Sample document generator (7 generators, Azure Claude content pools, CJK fonts)
• Project restructure: Tracker/Setup.bat + Tracker/Code/{Sync,Tools,Tracker}
• Windows rclone sync setup (Setup.bat, Win-Setup.ps1, sync.ps1, config.ini)
  • rclone auto-download + self-contained install in _LLM/Code/Sync/rclone/
  • Password-based SFTP auth (demo), config.ini for server details
  • .sync_ignore generated on setup (excludes_LLM/, dotfiles, temp/lock files)
  • 5-minute scheduled task (JingTian-Sync)
  • Pull-first-then-push: bidirectional files pulled with --update, excluded from push
  • Bidirectional file list in config.ini (e.g. Admin/Tracker.xlsx)
  • Daily log rotation in _LLM/Code/Sync/logs/
• Ubuntu VM SSH password auth enabled (Azure cloud-init was blocking it)
• Both sync directions tested and working
• Old JingTian-Rclone repo archived on Gitea

Phase 1: Foundation

• Init Go module
• SQLite schema + migrations (documents, extractions, processing_log)
• Config loading (YAML)
• Docker Compose (Docling, Ollama, Python Tools, Go server)

Phase 2: Pipeline Core

• Docling HTTP client
• Ollama HTTP client + structured extraction prompt
• Tools service (FastAPI + openpyxl Excel endpoints)
• File watcher (inotify on Ubuntu, 2-min stability check via SHA256 hash)
• State tracker integration (new/changed/skip logic)
• Pipeline orchestrator (read latest Tracker before writing, merge not overwrite)

Phase 3: Integration + Testing

• End-to-end test (drop file on Windows → syncs to Ubuntu → pipeline → Tracker updates → syncs back)
• Smoke test with real PDFs + generated samples
• README documentation

Future

• Key-based SFTP auth (replace password for production)
• Windows Go service (replace PowerShell scheduled task)
• File hash (SHA256) change detection on Windows side
• WhatsApp deadline notifications
• Processing retry logic
• Multi-client support
• Smarter bidirectional merge (cell-level, not file-level)