# Menu Extraction OCR Strategy

## Current OpenAI-Centric Flow
- `MenuExtractionService.extract_menu` normalizes languages, builds a structured prompt, batches images against estimated token budgets, and consolidates categories before responding to `/menu/extract` (`app/services/menu_extraction/service.py:33`, `app/services/menu_extraction/routes.py:30`).
- Each batch sends the prompt plus image `data:` URLs as multi-modal content blocks to `openai_client.json_completion`, enforcing the `MENU_JSON_SCHEMA` contract so GPT already returns a validated `MenuPayload` (`app/services/menu_extraction/service.py:116-155`, `app/services/menu_extraction/service.py:230-294`).
- Configuration keeps latency and cost predictable (token limits, max images per batch, per-key failover, rate-limit recovery) and allows translation + structuring to happen in one API call (`app/services/menu_extraction/config.py:6-68`, `app/shared/clients/openai_client.py:10-166`).
- Strengths: single-hop OCR + translation + reasoning, low engineering overhead, multilingual accuracy (Cyrillic, Latin) explicitly prompted (`app/services/menu_extraction/service.py:332-356`).
- Risks: dollar cost tied to images × tokens, dependency on external availability & auth, rate-limit exposure, limited control over deterministic outputs (mitigated via schema but not perfect), data residency concerns.

## Local OCR-First Alternative
Implementing an on-prem/edge OCR stack would require a multi-stage pipeline rather than the current single OpenAI call:
1. **Pre-processing:** deskewing, denoising, splitting menu photos into text-friendly crops (OpenCV or `imgaug`).
2. **Text detection & recognition:** models like PaddleOCR (server-friendly, multilingual), Tesseract (lighter but lower accuracy on stylized fonts), or open-source TrOCR/DocTR hybrids for higher fidelity; these output bounding boxes + raw text.
3. **Layout understanding:** grouping lines into menu sections using heuristics or transformers such as LayoutLMv3/Donut. Needed because GPT currently infers categories directly from the image.
4. **Semantic parsing + translation:** convert detected snippets into `MenuCategory` & `MenuItem` objects, possibly calling the existing translation/allergen services for language conversion instead of GPT doing it inline.
5. **Validation:** run JSON Schema validation locally—today GPT already produces compliant JSON, so the burden would shift to our code.

This path reduces OpenAI dependence but adds engineering + infra scope: GPU provisioning for real-time OCR, custom language models for Cyrillic, asynchronous job orchestration, monitoring for each stage, and security hardening for model weights. Expect higher initial latency (pipeline stages) but predictable per-request costs once hardware is amortized.

## Trade-off Summary
| Dimension | OpenAI Vision (current) | Local OCR Stack |
| --- | --- | --- |
| **Accuracy & completeness** | GPT-4o-style models jointly reason over layout + text, so they capture section hierarchy, prices, notes, and can fill gaps using context; translation clause ensures coherent target language output. | Pure OCR needs extra logic to infer hierarchy; errors compound (mis-detected wording → bad translation). Achieving GPT-level menu understanding requires layout transformers + custom parsers. |
| **Languages & scripts** | Prompt enforces support for Cyrillic/Latin mixes and translation to target locales out of the box (`app/services/menu_extraction/service.py:332-356`). | Need language detection + translation pipeline; PaddleOCR multilingual models help but still weaker on stylized fonts, and translation must defer to another service. |
| **Latency & throughput** | Single network hop bounded by OpenAI latency; batching (`_chunk_images`) manages token ceilings but long menus may need multiple calls (`app/services/menu_extraction/service.py:163-211`). Rate-limit handling/failover already coded. | Local inference removes external hop but introduces multiple sequential stages; GPU throughput can be tuned, yet CPU-only deployments likely slower. Requires our own concurrency limits and queueing. |
| **Cost** | Pay-by-token/image; `openai_client` estimates spend per request for observability (`app/shared/clients/openai_client.py:82-165`). Scaling traffic linearly increases OpEx. | Fixed CapEx/OpEx (hardware, model hosting). After provisioning, marginal cost per request is near-zero, but maintenance (updates, faults) adds ongoing effort. |
| **Operational control** | Minimal infra, but external outages or policy changes are outside our control; API keys must be rotated and rate-limits managed. | Full control over deployment, data residency, and customization (fine-tuning, prompt rules baked into models), but with added DevOps/ML expertise and monitoring obligations. |
| **Security & compliance** | Data leaves environment; depends on OpenAI regional compliance and encryption of uploaded menus. | Data stays inside VPC or even on-prem, supporting stricter privacy requirements (e.g., EU-only processing). Must secure model artifacts ourselves. |
| **Feature velocity** | Adjusting prompt/schema instantly updates behavior; new GPT models unlocked by config changes. | Rolling out improvements requires ML experimentation, dataset labeling, and redeployment cycles. |

## Recommendation & Next Steps
1. **Short term:** Retain OpenAI flow for highest accuracy while instrumenting richer metrics (image count, token spend) per restaurant batch so we know real costs before migrating.
2. **Spike local prototype:** Build a proof-of-concept using PaddleOCR + LayoutParser on a curated menu set to quantify precision/recall vs GPT outputs. Reuse existing translation service for multilingual output to match `MenuPayload`.
3. **Hybrid option:** Consider a detector/reader locally to extract raw text and fall back to GPT only for layout reasoning + translation when confidence is low, reducing token consumption without rewriting the entire pipeline.
4. **Infra planning:** If local path becomes viable, plan for GPU-enabled worker (e.g., A10G on-demand) and stream uploads through the current FastAPI route, swapping `_process_batch` with OCR pipeline while keeping response schema identical so API consumers stay unaffected.