The 131st episode of Datacast is my conversation with Krishna Gade, the founder and CEO of Fiddler AI, an AI Observability startup that helps AI-forward organizations build trusted AI solutions and connect model outcomes to business KPIs.
Our wide-ranging conversation touches on his research on document clustering in grad school, his early career working on the Bing’s Search Engine at Microsoft, his time as an engineering leader at Twitter and Pinterest scaling their data engineering, his experience at Facebook building the News Feed ranking platform, the founding story of Fiddler AI and the Model Performance Management framework, model governance for modern enterprises, lessons learned from hiring/finding design partners/fundraising, and much more.
Please enjoy my conversation with Krishna!
Show Notes
(02:20) Krishna described his academic experience getting an MS in Computer Science from the University of Minnesota - where he developed efficient tools for text document clustering and pattern discovery.
(05:36) Krishna recalled his 5.5 years at Microsoft working on Bing's search engine.
(08:32) Krishna talked about the challenges of competing against Google Search.
(10:22) Krishna shared the high-level technical and operational challenges encountered during the development and scaling phase of Twitter Search.
(14:55) Krishna revealed vital lessons from building critical data infrastructure at Twitter.
(17:54) Krishna touched on his time at Pinterest as the head of data engineering - leading a team working on all things data from analytics, experimentation, logging, and infrastructure.
(20:05) Krishna reviewed the design and implementation of real-time analytics, ETL-as-a-Service, and an A/B testing platform at Pinterest.
(24:40) Krishna unpacked the major ML model performance issues while running Facebook's feed ranking platform.
(28:18) Krishna distilled lessons learned about algorithmic governance from Facebook.
(31:38) Krishna provided leadership lessons from building teams that create scalable platforms and delightful consumer products on Twitter, Pinterest, and Facebook.
(33:19) Krishna shared the founding story of Fiddler AI, whose mission is to build trust into AI.
(37:56) Krishna unpacked the key challenges and tools in his 2019 article "AI needs a new developer stack."
(40:49) Krishna discussed the evolution of MLOps over the past 4 years.
(42:48) Krishna explained the benefits of using the Model Performance Management (MPM) framework to address enterprise MLOps challenges.
(47:01) Krishna gave a brief overview of capabilities within Fiddler's MPM platform, such as model monitoring, explainable AI, analytics, and fairness.
(50:28) Krishna highlighted research efforts inside Fiddler concerning explainability, drift metric calculation, and fairness.
(53:17) Krishna discussed the challenges with monitoring for NLP and Computer Vision models.
(57:18) Krishna zoomed in on Fiddler's approach to model governance for the modern enterprise.
(01:02:24) Krishna distilled valuable lessons learned to attract the right people who are excited about Fiddler's mission and aligned with Fiddler's culture.
(01:06:08) Krishna reflected on the evolution of Fiddler's company culture.
(01:09:19) Krishna shared the challenges of finding the early design partners and defining a new category of Responsible AI.
(01:12:23) Krishna gave fundraising advice to founders who are seeking the right investors for their startups.
(01:14:45) Closing segment.
Krishna's Contact Info
Fiddler's Resources
Mentioned Content
People
Goku Mohamandas (Made With ML and Anyscale)
Krishnaram Kenthapadi (Chief AI Officer & Chief Scientist at Fiddler)
Books
"The Hard Thing About Hard Things" (Ben Horowitz)
"The Five Dysfunctions of A Team" (Patrick Lencioni)
Notes
My conversation with Krishna was recorded more than a year ago. Since then, I'd recommend checking out these Fiddler's resources:
Strategic investments in Fiddler by Alteryx Ventures, Mozilla Ventures, Dentsu Ventures, and Scale Asia Ventures.
Fiddler introduces an end-to-end workflow for robust Generative AI back in May 2023.
Krishna's thought leadership on LLMOps and the missing link in Generative AI.
About the show
Datacast features long-form, in-depth conversations with practitioners and researchers in the data community to walk through their professional journeys and unpack the lessons learned along the way. I invite guests coming from a wide range of career paths — from scientists and analysts to founders and investors — to analyze the case for using data in the real world and extract their mental models (“the WHY and the HOW”) behind their pursuits. Hopefully, these conversations can serve as valuable tools for early-stage data professionals as they navigate their own careers in the exciting data universe.
Datacast is produced and edited by James Le. For inquiries about sponsoring the podcast, email khanhle.1013@gmail.com.
Subscribe by searching for Datacast wherever you get podcasts, or click one of the links below:
If you’re new, see the podcast homepage for the most recent episodes to listen to, or browse the full guest list.
Key Takeaways
Here are the highlights from my conversation with Krishna:
On His Educational Background
Like many Indian students who come to America, I completed my undergraduate studies in the late nineties and then pursued graduate school in the U.S. One area that particularly interested me was data mining.
During the late nineties and early 2000s, significant research was conducted in association rule mining. One of the seminal papers in this field focused on analyzing transaction data from Walmart and uncovering exciting patterns.
I vividly remember one of the most intriguing discoveries from that research, which involved a humorous example. The researchers found a strong correlation between purchases of beer and diapers, which initially puzzled them. They wondered why such a high correlation existed between these two items. It turned out that many fathers, on their way back home, would stop by Walmart to buy beer for themselves and diapers for their children. This finding was quite fascinating.
As a result, Walmart rearranged its aisles based on this insight to help customers find items more efficiently. These techniques have many applications, especially in web and text mining. I focused extensively on document clustering, which was crucial in the early stages of search engines like Google, Yahoo, and Bing. Teams were crawling the web and striving to make sense of the vast amount of textual data available. I dedicated a significant amount of time to developing clustering algorithms that could effectively analyze these large text datasets.
The University of Minnesota has an excellent computer science program. I was fortunate to work with two professors who were pioneers in data mining and graph partitioning: Dr. Vipin Kumar, who is still at the university, and George Karypis, who is currently on hiatus and working at Amazon. They conducted groundbreaking research in bioinformatics, data mining, text mining, and climate modeling. Working with them gave me valuable opportunities to collaborate, publish papers, and gain experience in these fields. Overall, it was an incredible experience.
On Working on Bing's search engine at Microsoft
After working in grad school on data mining-related areas, I wanted to work on building a web search system. Microsoft had a small team of 25 people dedicated to this project, which was like a startup within the company. The goal was to create a search engine to compete with Google.
Joining this team was an exciting opportunity for me as we started from scratch. We were all new and figuring things out together. During that time, everything was new. Microsoft used Linux servers for their data centers, including Hotmail. Bing, one of the first software systems built on clusters of Windows 2003, was also part of this infrastructure. Eventually, this infrastructure was repurposed for Azure.
Working on Bing, I was part of the search quality team, which allowed me to contribute to groundbreaking work. We focused on building page rank and graph-based algorithms to determine the importance of web pages. Additionally, I had the opportunity to work on Bing's first autocomplete feature, which provided quick suggestions for users when they used the search box.
We also worked on innovative projects, such as returning search traffic to Bing. We had many MSN pages and properties and wanted to run contextual ads similar to AdSense. We called it SearchSense, where we displayed relevant ads for search queries on Bing. For example, if you were on an MSN page related to Michael Jordan, you could see a query that would bring you back to Bing search.
Working on search engines allowed me to explore various aspects of computer science, including distributed systems, machine learning, and algorithm development. It was a valuable experience that laid the foundation for my career. I spent around five to six years at Bing and cherish the knowledge and skills I gained.
On Competing Against Google
Google's biggest advantage over Bing was the amount of data they had. This was because they already had a toolbar launched on various browsers, allowing them to collect a large amount of long-tail query data.
In the case of search engines, the more you use them, the better they can become because they can gather more data. At Bing, we developed highly sophisticated algorithms, including one of the first neural networks for a large-scale use case.
Before that, we also productized an algorithm called RankNet, a two-layer neural network used to score search results on web pages. We had a team of Microsoft researchers working on these problems. However, Google had better data due to their larger amount of long-tail query data. This made it challenging for Bing to match or surpass Google regarding search quality.
We constantly compared our performance to Google using a search quality metric called NDCG (Normalized Discounted Cumulative Gain). We aimed to see how Bing performed compared to Google for the same queries.
On Scaling Twitter Search
Listen to this episode with a 7-day free trial
Subscribe to Datacast to listen to this post and get 7 days of free access to the full post archives.
