Senior Data Engineer

We are looking for a highly technical, hands-on Senior Data Engineer to drive the evolution of our data delivery platform while leading a team of data engineers. This is a player-coach role: you will architect our next-generation data stack, build proofs-of-concept (POCs), and prove out new technologies before we commit to them at scale — and you will also manage and grow a team of engineers who report directly to you. You will partner closely with engineering leadership to shape the technical direction of our pipelines while staying hands-on in the systems yourself We are deliberately moving away from a locked-in, vendor-heavy stack (e.g., Azure Databricks) toward a flexible, largely open-source architecture that keeps our options open. We also expect our engineers to work in a modern, AI-assisted way — using AI coding tools and prompt-based workflows to move faster without compromising quality. You should be energized by evaluating tools, building POCs, and making pragmatic, evidence-based decisions about what we adopt next. This is a build-and-prove role — you are expected to write code, design schemas, profile queries, and get into the details to understand the "how" and "why" behind every pipeline, while also mentoring your team to do the same. Requirements Team Leadership & People Management: Lead, manage, and grow a distributed team of data engineers (across North America and India) who report directly to you. AI-Assisted Development: Champion the use of AI coding assistants (e.g., GitHub Copilot, Cursor, Claude) and prompt-based development to accelerate prototyping, code generation, refactoring, testing, and documentation. Establish best practices, guardrails, and review standards so the team uses these tools effectively and safely. Hands-On POCs & Prototyping: Personally build proofs-of-concept to validate new tools and patterns (e.g., ClickHouse, dbt, open-source orchestration and pipeline frameworks) before broader rollout, and translate the results into clear recommendations. Tech Stack Strategy & Architecture: Help shape the technical vision for our ETL and data platform. Evaluate, prototype, and recommend the technologies that will carry us forward, balancing flexibility, cost, performance, and avoiding vendor lock-in. Platform Modernization: Drive the hands-on migration away from Azure Databricks toward a more open, flexible stack, minimizing disruption to our high-volume daily data delivery. Build & Own Pipelines: Design, build, and optimize robust ETL pipelines that move and transform millions of daily data points reliably and efficiently. Deep System Integration: Master all systems upstream and downstream of your area, from ingestion through to client-facing platforms, to ensure seamless, high-quality, end-to-end data delivery. Operational Excellence: Establish testing frameworks, QA plans, observability, and CI/CD practices that guarantee data integrity at scale. Technical Leadership Through Influence: Set technical standards and raise the bar across a distributed team (North America and India) through design guidance, code reviews, and mentorship — leading by expertise rather than direct people management. Domain Mastery: Rapidly learn the nuances of the Advertising and Market Research domain to translate business needs into robust technical requirements. What You'll Bring (Qualifications) Experience: 8+ years in data engineering / ETL, with a strong track record as a hands-on engineer who has architected and built data platforms (Principal-level candidates will bring deeper architectural and cross-team impact). AI-Assisted Engineering: Hands-on experience using AI coding tools (e.g., GitHub Copilot, Cursor, Claude, or similar) and strong prompt-based development skills, with good judgment about where these tools help and where human review is essential. Core Databases: Expert-level SQL and RDBMS skills with deep, hands-on experience in SQL Server and Postgres (schema design, performance tuning, complex query optimization, root-cause analysis). Modern & Open-Source Stack: Hands-on experience with technologies such as ClickHouse, dbt, and open-source data pipeline / orchestration tools (e.g., Airflow, Dagster, or similar), with the judgment to choose the right tool for the job. Cloud: Strong, hands-on AWS experience is required, as we are standardizing on AWS as we move off Azure Databricks. POC & Evaluation Mindset: Demonstrated ability to independently prototype, benchmark, and evaluate new technologies and make pragmatic adoption decisions. Programming: Strong proficiency in Python (or similar) for building and automating data pipelines. Education: Bachelor's or Master's degree in Computer Science, Engineering, or a related field. Bonus Points (Nice to Haves) Domain Expertise: Previous experience in Advertising, Media, or Market Research. Infrastructure Knowledge: Familiarity with containerization (Docker, Kubernetes) and infrastructure-as-code. Streaming & Real-Time: Experience with