Data Engineering Team Lead

🚢 Discover Keystone AI : Keystone AI is a unified brand born from the integration of Keystone AI and Ocean Technologies Group. Owned by Lloyd’s Register, an organisation with more than 260 years of trust, integrity and leadership at sea, Keystone AI combines the agility and ambition of a fast-moving innovator with the strength and stability of one of the world’s most trusted maritime institutions. At the heart of Keystone AI is a portfolio unlike any other in maritime. A comprehensive, integrated portfolio built on years of expertise, trusted by thousands of maritime professionals around the world. 🎯 Our Mission: Our mission is clear. In the race to zero emissions, our research, advisory and technical expertise and industry-firsts are supporting a safe, sustainable maritime energy transition. Today we are a leading provider of classification and compliance services to the marine and offshore industries, helping our clients design, construct and operate their assets to accepted levels of safety and environmental compliance. 🔍 Why Join Keystone AI Crew? Legacy & Innovation: We were created more than 260 years ago as the world’s first marine classification society to improve and set standards for the safety of ships. Global Impact: Our digital solutions are relied upon by more than 30,000 vessels, following the acquisition of Keystone AI in 2022 and Ocean Technologies Group in 2024. Product Offering: Covering five proven product areas - learning, fleet operations, compliance, voyage planning and performance management - supporting its customers from ship to shore, from training and people operations, to voyage compliance and optimisation. 🧭 Navigating the position: Data Engineering Team Lead As Data Engineering Team Lead at Keystone AI you will lead a team of data engineers designing, developing and operating the ingestion, transformation and modelling pipelines that power our analytics platform. You will lead by example, driving engineering excellence, ensuring delivery of high-quality, scalable data products, and fostering a collaborative and innovative team culture. Working closely with other Team Leaders, Architects and Product, you will help shape technical direction, streamline delivery processes, and ensure alignment with business goals. This is a fully remote, Philippines-based role. 🚢 Your Voyage Ahead: Lead a team of data engineers, ensuring alignment on goals, quality and delivery timelines. Mentor and coach team members to support their technical and professional growth. Drive engineering excellence by promoting best practices in coding, architecture, testing and observability. Plan and manage team capacity, sprints and milestones to ensure predictable delivery. Own the design, evolution and operation of ingestion and transformation pipelines on Apache Airflow and the analytical serving layer on Apache Druid. Make architectural calls on concurrency, partitioning, memory sizing and cost — including JVM heap and direct-memory tuning on the Druid cluster. Collaborate closely with DevOps on the Kubernetes / EKS platform that hosts our Druid and Airflow workloads. Ensure robust data validation, reconciliation and verification so that reporting is trustworthy. Collaborate with other Team Leaders, Development Managers, Architects and Product Owners to align engineering execution with business objectives. Contribute to the evolution of development processes, CI/CD pipelines and DevOps practices. Foster a culture of continuous improvement, innovation and knowledge sharing. 🚢 Recommended to bring on board: 10+ years of commercial experience delivering Data & Analytics solutions. 5+ years leading a Data Engineering or BI team with a solid grasp of Agile methodologies (Scrum / Kanban). Strong hands-on expertise in Apache Airflow — production DAG authoring in Python; hooks, sensors, XCom, callbacks, retries; dynamic task mapping. Production-grade Python — type hints, packaging, pytest; comfortable reading and reviewing other people's DAGs. Apache Druid (ingestion-side) — index_parallel specs, transformSpec / dimensionsSpec / granularitySpec, tuningConfig, supervisors, task lifecycle, segment management; familiarity with Coordinator / Overlord / Broker / Historical / MiddleManager roles. Strong PostgreSQL — query tuning, JSONB, window functions, indexes, EXPLAIN / EXPLAIN ANALYZE. SQL fluency across dialects (Postgres, T-SQL, Oracle); comfortable optimising queries. AWS data services — S3 (Druid deep storage), EMR, Secrets Manager, IAM, VPC fundamentals; EC2 sizing for memory-bound workloads. Production debugging instincts — reading YARN container logs, tracing failure from Airflow → EMR step → Spark driver → Python traceback. Exceptional communication, teamwork, attention to detail, organisational and leadership skills. Ability to inspire, mentor and lead by example. Nice to have JVM tuning — heap (Xms / Xmx), direct memory, GC choice (G1, ZGC), reading GC logs; distinguishing JVM OOM from cgroup OOMKilled. Kubernetes operations