Abstract

This talk covers how Yelp has incorporated large language models into its search engine over the past four years, focusing on query understanding — the first stage of the search pipeline. I will walk through three production LLM systems, examining how the problem framing for each evolved as models became more capable. The talk also covers practical lessons in LLM evaluation and production deployment strategies such as fine-tuning cascades and caching at scale. A recurring theme is that impactful improvements came from rethinking the problem framing, not just the prompt.

About the Speaker

John Hawksley is a Principal Engineer at Yelp and the technical lead for the core search engine, where he has led the integration of LLMs across the search stack over the past four years. He has a strong interest in the intersection of quantitative and qualitative analysis. Earlier in his career, he held a diversity of roles including working as a venture capital associate, and studied mathematics, computer science, and economics at MIT. In his spare time, he is a crossword puzzle constructor — another area where quantitative and qualitative analysis intersect!