Talk
Many data pipelines and automation processes start as simple Python scripts scheduled via cron. Over time, they evolve into fragile systems with ad-hoc retries, inconsistent environments, unclear dependencies, and limited observability. This talk demonstrates how to transition from that state to a robust, production-grade workflow using Prefect, with a focus on in-house deployments and internal data platforms.
The talk will start with a minimal Python script and progressively refactor it into a Prefect flow, introducing task decomposition, retries, logging, and parameterization. It will then shift toward aspects often overlooked in introductory materials but critical in real systems: configuration, deployment, and environment management.
Specifically, the talk will cover how to define deployments using YAML-based configuration, manage variables and parameters, and structure projects to support multiple environments (development, staging, production). It will demonstrate how to separate code from configuration, enabling reproducibility and maintainability across teams.
A key advantage highlighted in the talk is Prefect’s cross-platform compatibility, including support for Windows-based environments. Unlike traditional orchestration tools that often require Linux-based infrastructure, Prefect enables teams to develop, test, and run workflows directly on local machines. This lowers the barrier to entry while maintaining consistency with production environments.
This capability makes Prefect particularly well-suited for in-house data platforms, where rapid onboarding, reduced infrastructure overhead, and automation of internal processes without complex setup are critical.
A key part of the session will focus on using Prefect in self-hosted, in-house environments, where teams rely on internal infrastructure rather than fully managed cloud solutions. It will also discuss deployment patterns, trade-offs, and practical considerations for running Prefect within existing ecosystems.
The session will also demonstrate:
- Local development and testing workflows
- Deployment to self-hosted Prefect servers or Prefect Cloud
- Monitoring and observability through the Prefect UI
The session is practical and implementation-focused. Attendees will leave with:
- A solid working example
- Reusable patterns for organizing workflows (.env and config files for easy setup)
- A clear understanding of how to operationalize Python-based pipelines in real-world, in-house environments
About the Speaker
I am a Data Science Team Lead at Viva Armenia, where I lead a team of data scientists and analysts to design and deploy machine learning solutions for real-world business problems. My work focuses on building predictive systems for customer segmentation, churn management, recommendation engines, and revenue optimization, with a strong emphasis on translating data into actionable business impact. More about my work can be found on LinkedIn and GitHub.
In parallel, I am an Adjunct Lecturer at the American University of Armenia, teaching Marketing Analytics to data science students. My teaching combines academic rigor with industry practice, covering topics such as A/B testing, customer analytics, and data-driven product development.
My experience spans data science, data engineering, and backend development, with hands-on expertise in building production-grade data pipelines and automation systems using Python. I am particularly interested in designing scalable, maintainable data platforms that support efficient decision-making and operational workflows.