Member of Technical Staff (Data Scientist, Evals)
The vacancy is well-defined in tasks and requirements but lacks compensation details and company links.
Check Match — Just drop your CV
See your fit for Member of Technical Staff (Data Scientist, Evals) in seconds.
Overview
Join Perplexity as a Data Scientist to enhance answer quality using specialized evaluations in a remote, full-time role. Perplexity serves tens of millions of users daily with reliable, high-quality answers grounded in an LLM-first search engine and our specialized data sources. We aim to use the latest models as they are released, but the intelligence frontier is a jagged one, and popular benchmarks do not effectively cover our use cases. In this role, you will build specialized evals to improve answer quality across Perplexity, covering search-based LLM answers and other scenarios popular with our users.
What you'll do
- •Architect and maintain automated evaluation pipelines to assess answer quality across Perplexity's products, ensuring high standards for accuracy and helpfulness
- •Design evaluation sets and methods specifically to measure the impact of tool calls (particularly web search retrieval) on the final answer's quality
- •Develop VLM-based solutions to programmatically evaluate how final answers render visually across different platforms and devices
- •Continuously review public benchmarks and academic evaluations for their applicability to the Perplexity product, adapting and incorporating them into our regular performance measurements
- •Operate within a small, high-impact team where your evaluation metrics directly shape product changes, collaborating closely with technical leadership to measure and improve Answer Quality