Principal Data Scientist
About this role
About the team
At Zillow Group, our mission is to help more people get home. The CoreTech Org delivers on this mission by building the foundational platforms, data, and developer capabilities that power every consumer, partner, and employee experience at Zillow.You will work most closely with the Connections Platform (CP) — the engine that turns consumer interest into a connection with the right partner, owning the conversational experience, the lead funnel and follow-up workflows, and the shared agent services that sit underneath both. Increasingly, that experience is powered by LLM-based and agentic systems: agents that qualify leads, summarize and extract intelligence from conversations for partner teams, and drive proactive outreach. Beyond CP, you will be a measurement partner for the broader CoreTech Org — the foundational platforms every consumer, partner, and employee experience at Zillow is built on — wherever a hard data science problem needs an owner.
These are platforms with hundreds of touchpoints and dozens of dependent teams — and they have historically been among the hardest places at Zillow to measure well. CP itself is mid-transformation, shifting from NLP to LLM and agent-based systems, with agents increasingly built and relied on in production but no rigorous way to evaluate them yet. That combination — high dependency, hard-to-measure, and changing fast — is exactly where rigorous measurement creates the most leverage.
About the role
We are seeking a Principal Data Scientist to be a measurement and impact partner in CoreTech. This is a senior individual-contributor role embedded with the Data & Infrastructure Platform team and who will partner tightly with Product leadership.
You will own four outcomes: (1) make the impact of platform work legible — quantifying what shipped, what moved, and what didn’t, in terms leaders can act on; (2) define the metrics that matter for CP and the other platforms you take on — what belongs on a progress dashboard, what is a leading vs. lagging indicator, and what is vanity; (3) build the evaluation layer CP is missing — standing up rigorous, repeatable evals for its LLM and agentic systems (including LLM-as-a-judge) so the team can tell whether an agent actually works, not just that it shipped; and (4) operationalize and instrument those metrics and evals in partnership with engineering — getting the events emitted, the pipelines stable, the dashboards live, and the data trusted.
The defining challenge of this role is twofold. The first is measuring causal impact: CP has live, expensive, executive-facing questions that hinge on making a credible impact claim when a clean A/B test isn’t available — conversational and agent-driven flows that span a lead’s journey across sessions and channels, and changes that roll out to everyone at once. The second is establishing anevaluation feedback loop: much of CP now runs on LLM-based and agentic systems that were built by engineers with no rigorous way to measure whether they actually work. You will own how we evaluate these systems — defining the signals that matter, standing up eval and experimentation frameworks (including LLM-as-a-judge), and separating activity and adoption from genuine quality and outcomes. The same question recurs across CoreTech in many different areas, and you will need to help prove whether new features or infrastructure actually pay off, rather than trusting activity or adoption as proof. Owning these questions is the heart of the job.
This is a high-leverage, cross-Zillow role: your work will set the measurement bar for platforms every product team at Zillow depends on, and your point of view will directly shape how Data & Infrastructure Platform OKRs are written and reviewed at the executive level.
You will:
- Define and operationalize the measurement strategy for the Connections Platform and the broader Data & Infrastructure Platform team — owning the throughline from raw events to executive dashboards and causal inference attribution.
- Own the evaluation of CP’s LLM-based and agentic systems: design the evals, define the quality signals, and build the frameworks (including LLM-as-a-judge and using tooling such as Braintrust) that tell us whether an agent is actually working.
- Own the causal inference and experimentation program for these teams: design and analyze A/B tests where you can, and where you can’t, build the quasi-experimental causal inference design (switchbacks, difference-in-difference, synthetic control, interrupted time-series) that produces a defensible estimate of impact. Know which method the situation calls for, what assumptions it rests on, and make the call for the best method to answer each hard question.
- Be the credible authority on “what to measure” and “what to experiment on” for these teams — pushing back on vanity metrics, identifying the leading indicators we’ve been missing, and knowing which signals actually reflect the customer experience.
- Own the impact dimension that platform measurement usually lacks — connecting adoption of CP’s agents and other platform investments to business and revenue outcomes, not just activity, speed, and quality signals.
- Partner with engineering ICs to instrument the metrics: events, schemas, and pipelines needed to get quality measurements. Engineers will wire up the engine; you make the data infrastructure for these surfaces something teams can trust, and you shape what gets emitted in the first place.
- Translate complex measurement realities into clear narratives for VP and C-level partners, including the trade-offs they need to consider.
- Set the bar for how data science is practiced inside platform teams at Zillow , treating the platform itself as a product whose adoption, quality, and impact get measured as rigorously as any consumer product (Platform-as-a-Product measurement).
- Help decipher the impact of product changes that are multi-dimensional at once – touching consumers, partners, and developers, and carrying real business and revenue stakes.
Who you are
- Senior data scientist with internal-customer and/or platform measurement experience. 10+ years in Data Science, UX Research, or Applied Science, including meaningful time measuring products where the customer is a developer, an internal team, or a platform. You have led the measurement of platform products — where adoption, partner teams, and downstream outcomes all matter — and you know why that’s harder, not easier.
- Fluent in evaluating complex user-facing systems. You have designed evals for products where quality is hard to pin down – using the right tool for the job, from labeled test sets and human review to LLM-as-a-judge. You can separate adoption and activity from genuine quality, turning “the experience feels better” into a defensible number.
- Causal-inference and experimentation-fluent. You have owned the design and analysis of A/B tests and causal inference methodology in production — and you’ve expertly made the call between which method is the most appropriate to answer a hard question.
- Metric architect, not dashboard operator. You have deep experience designing primary metrics, KPIs, OKR metric definitions, and leading and lagging indicator systems from scratch. You can tell the difference between a measurement problem and an instrumentation problem, and you fix the right one first.
- Engineering-credible operator. You can sit with a staff engineer and design the event taxonomy, debug the methodology, and push back on the schema — not hand a spec over the wall. You write your own Python and SQL, and you’ve shipped instrumentation others can build on – repeatable, reliable, and documented enough that another team could adopt it – not just consumed what someone else produced.
- Executive translator. You can land a measurement narrative with a VP or Senior Director without losing rigor — and you know which assumptions to surface up and which to keep in the appendix.
-
Builder-mentality individual contributor. You write your own code, model your own data, and prototype your own evals and dashboards. You raise the bar that the team’s daily work is rigorous and repeatable, and you make the ICs around you better by doing it visibly.
Preferred qualifications:
PhD and 4+ years, MS and 6+ years, or BS and 8+ years of experience in Statistics, Econometrics, Data Science, User Experience Research, Machine Learning, Operations Research, or a related quantitative field.
- Direct experience measuring complex, production user-facing systems – for example, non-deterministic ones such as conversational, LLM-based, or agentic products (chat and voice assistants, lead or conversation funnels, agent-driven workflows).
- Fluency with modern evaluation methods and the judgment to know which signals to trust – approaches like LLM-as-a-judge and offline/online eval harnesses, or adjacent evaluation experience.
- Experience measuring the impact of AI or agentic tooling on real outcomes — separating adoption and activity from genuine productivity or quality gain, and reasoning about ROI against spend.
- Experience operating inside a Platform-as-a-Product model with internal teams as customers.
- Fluency with quasi-experimental and causal inference methods for when randomization isn’t possible — difference-in-difference, synthetic control, interrupted time-series, and switchback or interleaved designs — including the discipline to state assumptions explicitly and be clear about what you can and cannot measure.
Get to know us
At Zillow, we’re reimagining how people move—through the real estate market and through their careers. As the most-visited real estate platform in the U.S., we help people navigate buying, selling, financing and renting with greater ease and confidence. Whether you're working in tech, sales, operations, or design, you’ll be part of a company reshaping an industry and helping more people make home a reality.
How we work is a key part of that transformation. Through Cloud HQ, our strategic approach to distributed work, most employees have the flexibility to work from wherever they’re most productive—enabling us to move fast, stay connected and deliver on our people promise.
Zillow is honored to be recognized among the best workplaces in the U.S. Zillow was named one of FORTUNE 100 Best Companies to Work For® in 2026, and included on TIME’s America’s Best Companies 2026 list, reflecting our commitment to creating an innovative, inclusive, and engaging culture where employees are empowered to grow.
No matter where you sit in the organization, your work will help drive innovation, support our customers, and move the industry—and your career—forward, together.
Zillow Group is an equal opportunity employer committed to fostering an inclusive, innovative environment with the best employees. We are committed to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity or Veteran status. If you have a disability or special need that requires accommodation, please contact your recruiter directly.
Qualified applicants with arrest or conviction records will be considered for employment in accordance with applicable state and local law.
Los Angeles County applicants: Job duties for this position include: work safely and cooperatively with other employees, supervisors, and staff; adhere to standards of excellence despite stressful conditions; communicate effectively and respectfully with employees, supervisors, and staff to ensure exceptional customer service; and follow all federal, state, and local laws and Company policies. Criminal history may have a direct, adverse, and negative relationship with some of the material job duties of this position. These include the duties and responsibilities listed above, as well as the abilities to adhere to company policies, exercise sound judgment, effectively manage stress and work safely and respectfully with others, exhibit trustworthiness and professionalism, and safeguard business operations and the Company’s reputation. Pursuant to the Los Angeles County Fair Chance Ordinance, we will consider for employment qualified applicants with arrest and conviction records.
Originally posted on Himalayas
