Shopify โ Shop.com AI agent + eval framework
Personalization and conversational AI for tens of millions of users.
- Built a two-step agentic flow generating personalized conversation starters referencing product imagery, with a custom LLM-as-judge eval pipeline gating rollouts.
- Distilled the production LLM into Qwen 4B (open weights), shipping >$4M projected annual savings; extending to process product imagery directly (vision integration).
- Designed the offline eval framework that gates agent rollouts: automated LLM judges, benchmark suites, observability for agent reasoning loops and tool selection.