Show Notes
(01:29) Jeremiah reflected on his academic interest studying Statistics and Economics at Harvard.
(05:33) Jeremiah recalled his four years as a market risk manager at King Street Capital Management.
(07:18) Jeremiah explained how the training in risk management has made a huge impact in his career as a startup founder.
(09:48) Jeremiah then founded his own consultancy Lowin Data Company that designed and built ML systems for time series data.
(12:38) Jeremiah mentioned his fascination with the rapid growth of machine learning in the past decade.
(15:54) Jeremiah talked about his contribution to the Apache Airflow project and lessons learned about open-source development/governance.
(21:48) Jeremiah unpacked the notion of negative engineering and shared the story behind the inception of Prefect.
(27:24) Jeremiah dissected Prefect Core, the open-source framework that is stocked with all the necessary components for designing, building, testing, and running powerful data applications.
(32:45) Jeremiah went over the advanced enterprise features of Prefect Cloud that complement users of Prefect Core.
(36:04) Jeremiah discussed Prefect's product strategy (read his blog post "Toward Dataflow Automation," which distinguishes the difference between what a company makes and what a company sells).
(40:44) Jeremiah explained how Prefect users can take advantage of the hybrid execution model.
(47:08) Jeremiah walked through Prefect Server and Prefect UI that enable users to run parts of Prefect Cloud locally.
(50:27) Jeremiah talked about how his team has gradually open-sourced the Prefect platform.
(51:38) Jeremiah explained how Prefect settles into a "success-based pricing" model, where the cost is based entirely on the number of tasks users run successfully each month.
(54:15) Jeremiah shared how to nurture a highly active community of open-source contributors to Prefect Core.
(58:23) Jeremiah unpacked Prefect's hiring strategy, which emphasizes the importance of hiring a team diverse in thoughts, backgrounds, makeups, and experiences (read this fantastic guide to building a high-performance team on Prefect's website).
(01:07:02) Jeremiah shared fundraising advice for founders currently seeking the right investors for their startups.
(01:11:53) Jeremiah unpacked the two key pillars central to Prefect’s hyper-adoption within the data world: expansion and product.
(01:14:09) Closing segment.
Jeremiah's Contact Info
Prefect's Resources
GitHub | Slack | Documentation | Twitter | Meetup
The Prefect Guide to Building A High-Performance Team (April 2021)
Mentioned Content
Articles
"Positive and Negative Engineering" (Oct 2018)
"The Golden Spike" (Jan 2019)
"Prefect is Open-Source!" (March 2019)
"Towards Dataflow Automation" (June 2019)
"The Prefect Hybrid Model" (Feb 2020)
"Project Earth" (March 2020)
"Open-Sourcing The Prefect Platform" (March 2020)
"Your Code Will Fail (But That's Okay)" (May 2020)
"Liftoff: Prefect's Series A" (Feb 2021)
"Escape Velocity: Prefect's Series B" (June 2021)
Talks and Podcasts
"Invest Like The Best" (Jan 2017)
"Task Failed Successfully" (PyData DC 2018)
"Software Engineering Daily" (April 2020)
"The OSS Startup Podcast" (Nov 2021)
"The Sequel Show" (Jan 2022)
People
Vicki Boykis (ML Engineer at Tumblr, Newsletter Writer of Normcore Tech)
Chris Riccomini (Software Engineer at WePay, Contributor of Airflow, Investor/Advisor at Prefect)
Justin Gage (Newsletter Writer of Technically)
Books
"Creativity Inc." (by Ed Cadmull)
"The Hitchhiker's Guide to the Galaxy" (by Douglas Adams, Eoin Colfer, and Thomas Tidholm)
"Shoe Dog" (by Phil Knight)
Notes
My conversation with Jeremiah was recorded back in July 2021. Since then, many things have happened at Prefect:
The releases of Prefect Orion and Prefect Radar as part of the product roadmap
The announcement of Prefect's Premier Partnership Program for trusted partners
The introduction of Prefect Discourse for data engineers
The latest drop of Prefect 2.0!
About the show
Datacast features long-form, in-depth conversations with practitioners and researchers in the data community to walk through their professional journeys and unpack the lessons learned along the way. I invite guests coming from a wide range of career paths — from scientists and analysts to founders and investors — to analyze the case for using data in the real world and extract their mental models (“the WHY and the HOW”) behind their pursuits. Hopefully, these conversations can serve as valuable tools for early-stage data professionals as they navigate their own careers in the exciting data universe.
Datacast is produced and edited by James Le. Get in touch with feedback or guest suggestions by emailing khanhle.1013@gmail.com.
Subscribe by searching for Datacast wherever you get podcasts or click one of the links below:
If you’re new, see the podcast homepage for the most recent episodes to listen to, or browse the full guest list.
Episode 86: Risk Management, Open-Source Governance, and Negative Engineering with Jeremiah Lowin