MangoPig
77 lines
3.0 KiB
Markdown
77 lines
3.0 KiB
Markdown
# TODO
|
|
|
|
## Phase 0: Infrastructure (DONE)
|
|
|
|
- [x] Azure VMs created and configured (jt-ub, jt-win)
|
|
- [x] Ubuntu VM provisioned (Docker, Go, Python, rclone, zsh, dotfiles)
|
|
- [x] Directory structures on both VMs (William's layout)
|
|
- [x] Sample document generator (7 generators, Azure Claude content pools, CJK fonts)
|
|
- [x] Project restructure: Tracker/Setup.bat + Tracker/Code/{Sync,Tools,Tracker}
|
|
- [x] Windows rclone sync setup (Setup.bat, Win-Setup.ps1, sync.ps1, config.ini)
|
|
- rclone auto-download + self-contained install in _LLM/Code/Sync/rclone/
|
|
- Password-based SFTP auth (demo), config.ini for server details
|
|
- .sync_ignore generated on setup (excludes_LLM/, dotfiles, temp/lock files)
|
|
- 5-minute scheduled task (JingTian-Sync)
|
|
- Pull-first-then-push: bidirectional files pulled with --update, excluded from push
|
|
- Bidirectional file list in config.ini (e.g. Admin/Tracker.xlsx)
|
|
- Daily log rotation in _LLM/Code/Sync/logs/
|
|
- [x] Ubuntu VM SSH password auth enabled (Azure cloud-init was blocking it)
|
|
- [x] Both sync directions tested and working
|
|
- [x] Old JingTian-Rclone repo archived on Gitea
|
|
|
|
## Phase 1: Services (Services-First Approach)
|
|
|
|
Rationale: De-risk unknowns (Docling OCR quality, Ollama CPU inference speed) before building orchestrator.
|
|
|
|
### 1a. Docker Compose + Docling
|
|
|
|
- [ ] Docker Compose base (Docling service)
|
|
- [ ] Test Docling API with sample PDFs (curl)
|
|
- [ ] Validate OCR quality on real PDFs + generated samples
|
|
|
|
### 1b. Ollama
|
|
|
|
- [ ] Add Ollama to Docker Compose (Qwen3-1.7B, CPU mode)
|
|
- [ ] Test structured extraction prompt (curl)
|
|
- [ ] Measure inference time per document
|
|
- [ ] Validate extraction accuracy (deadline, doc type, etc.)
|
|
|
|
### 1c. Python Tools Service
|
|
|
|
- [ ] FastAPI + openpyxl service
|
|
- [ ] Endpoints: read tracker, write/update rows, get column schema
|
|
- [ ] Test with sample Tracker.xlsx
|
|
|
|
### 1d. Manual Integration Test
|
|
|
|
- [ ] Chain test: PDF → Docling → Ollama → Tools → Excel updated
|
|
- [ ] Document any issues / adjustments needed
|
|
|
|
## Phase 2: Go Orchestrator
|
|
|
|
- [ ] Init Go module in Tracker/
|
|
- [ ] Config loading (YAML)
|
|
- [ ] SQLite schema + migrations (documents, extractions, processing_log)
|
|
- [ ] Docling HTTP client
|
|
- [ ] Ollama HTTP client
|
|
- [ ] Tools service HTTP client
|
|
- [ ] File watcher (inotify on Ubuntu, 2-min stability check via SHA256 hash)
|
|
- [ ] State tracker integration (new/changed/skip logic)
|
|
- [ ] Pipeline orchestrator (read latest Tracker before writing, merge not overwrite)
|
|
|
|
## Phase 3: Integration + Testing
|
|
|
|
- [ ] End-to-end test (drop file on Windows → syncs to Ubuntu → pipeline → Tracker updates → syncs back)
|
|
- [ ] Smoke test with real PDFs + generated samples
|
|
- [ ] README documentation
|
|
|
|
## Future
|
|
|
|
- [ ] Key-based SFTP auth (replace password for production)
|
|
- [ ] Windows Go service (replace PowerShell scheduled task)
|
|
- [ ] File hash (SHA256) change detection on Windows side
|
|
- [ ] WhatsApp deadline notifications
|
|
- [ ] Processing retry logic
|
|
- [ ] Multi-client support
|
|
- [ ] Smarter bidirectional merge (cell-level, not file-level)
|