Data Platform Engineering Team Leader

1.4 billion users. 950,000 requests per second. 150TB of fresh data every day. 800M metrics every minute . That's the scale Keystone AI's Data Platform Engineering team operates at — and we're looking for a Software Development Team Leader to help drive what comes next. You'll lead a team of 4–6 strong engineers building the core data platform that powers Keystone AI’s reporting, billing, ML training, experimentation, revenue optimization, and business decision-making — with data that is reliable, fresh, scalable, and easy to use. To thrive in this role, you'll need: 5+ years leading and managing software development teams 8+ years of engineering background in Java or an equivalent object-oriented language — enough technical depth to lead architecture discussions and review code with authority Experience developing large-scale distributed systems and a solid understanding of production infrastructure Deep grounding in CS fundamentals: object-oriented design, data structures, and concurrent/multi-threaded programming Deep expertise in Kafka and Spark Highly proficient with Linux and Kubernetes in production Experience with both SQL and NoSQL systems Demonstrated ability to collaborate across teams and drive alignment Strong problem-solving and critical-thinking skills B.Sc. in Computer Science or equivalent experience Bonus points if you have: Experience with dbt and/or Airflow Experience with StarRocks production deployments Experience with Druid production deployments Knowledge of Ansible, Puppet, or similar configuration systems A track record of leveraging AI agents to accelerate engineering work Familiarity with how ML training pipelines consume data, to partner effectively with algo teams How you'll make an impact: Set technical direction for the team — the architecture, the roadmap, and the bets we make on emerging tech (Iceberg, StarRocks, GPU Spark, dbt, AI tooling) Own the production health and reliability of the main Keystone AI data pipeline — the system behind reports, ML model training, billing, and other business-critical processes Grow your engineers: coach, mentor, run 1:1s, give honest feedback, and create the conditions for senior engineers to do their best work Prioritize ruthlessly across competing demands from algo, product, finance, and infrastructure stakeholders — and say no when you need to Drive cross-team alignment: partner with peer team leaders, group managers, and principal engineers to shape how the platform evolves across R&D Upscale your team with AI: introduce agent-based workflows, measure their impact on velocity, and make your team a model for how engineering at Keystone AI changes in the AI era Evolve our Spark SQL platform (60,000+ jobs/day) and the core framework code used across R&D — making the call on what gets built, refactored, or retired Stay hands-on (~30–40% of your time) — enough to make sound technical decisions, review designs and code with authority, and unblock the team when it matters Represent the team externally — in design reviews, postmortems, hiring, and technical forums Directly shape how billions of people discover the internet Here are some of the things engineers in our group have done — or are doing right now: Pioneered deployment of Spark SQL workloads on GPU with RAPIDS Delivered a real-time pipeline reports product serving all our customers on top of Druid Actively migrating a proprietary ETL system to dbt-core Pioneered usage of Iceberg (and gave conference talks about it) — yes, we manage it ourselves Productionizing our StarRocks deployments Building pipelines on top of Spark Streaming, Iceberg, and dbt Extending data solutions to help algo engineers iterate faster …and many more Our Tech Stack Java, Spark, Kafka, Hadoop, Cassandra, Vertica, StarRocks, MySQL, HDFS, BigQuery, Docker, Linux, Kubernetes, Prometheus, Grafana, Airflow, dbt, Redis, Druid This is us! You'll work with: The people who wrote the first lines of code at Keystone AI (and many, many more lines after that) Engineers deeply familiar with the internals of Kafka, Cassandra, Airflow, and Spark — including committing back upstream Multiple internal Keystone AI hackathon winners The designers and implementers of an off-heap, zero-GC custom Kafka Producer that cumulatively sends ~100B messages a day Why Keystone AI? Adam Singolda, Keystone AI Founder and CEO, says: “ You can copy anything from another business but you can’t copy a company’s culture. ” If you ask Keystone AIrs what they love about working here, they’ll tell you that they’ve been empowered to realize their full potential while growing and learning from and with smart and talented people. They’ll also share more about: Well-being: Enjoy comprehensive benefits (health, etc.), a fully stocked kitchen, and location-specific perks (gym partnerships, parking). Flexibility: We offer a hybrid work schedule with 3 days in-office with an option to come in more often if desired. Work with some of the biggest names: We work with some of the biggest names in t