Middle Data Engineer (6 months' engagement)

5895N-iX is looking for a Middle Data engineer for 6 months' engagement. You will act as the independent hands-on engineer and the technical owner of data readiness. The role combines pipeline and query engineering with investigation of source, mapping, hierarchy, and validation issues; coordination of market and provider validation cycles; reconciliation of outputs; and delivery of technically ready datasets to downstream consumers.The Data Engineering team builds and maintains pipelines, while mappings and configurations determine cell/model scope; Snowflake extraction jobs run SQL through Databricks, with Blob Storage, validation, Medallion transformations, Power BI review, and Gold delivery to Ekimetrics. The operating model also requires end-to-end triage across MDFA, CDF, WPP, Data Foundation, Redmill, and market stakeholders rather than treating every discrepancy as a coding defect.Key responsibilities:Independently design, develop, maintain, test, deploy, and optimize ingestion, validation, transformation, aggregation, data-quality, anomaly-detection, and extraction workflowsDevelop and tune Python/PySpark packages, SQL extraction queries, Databricks jobs, ADF pipelines, Delta tables, and source-to-target contractsOwn technical readiness for new cells and refreshes: clarify filters and expected scope, implement or update queries, review mappings/configuration, reconcile outputs, and confirm readiness for business validationInvestigate source, NCID, taxonomy, hierarchy, mapping, naming-convention, aggregation, missing-week, outlier, and performance issues across CDF, MDFA/PFME, APIs, manual files, and specific sourcesCoordinate technical validation cycles with Product, CMIA/markets, WPP/MDFA, CDF, other data providers, and Ekimetrics; convert reported discrepancies into actionable owners and technical evidenceDrive complex incident resolution across pipelines, data contracts, access, service principals, secrets, networking, Power BI refreshes, and downstream deliveryReview pull requests and test evidence; guide the Junior engineer and delegate scoped engineering/support work without becoming a people managerMaintain architecture documentation, interface contracts, repository documentation, runbooks, deployment procedures, and support/escalation guidanceRecommend practical automation, reliability, performance, and maintainability improvements while respecting platform standards and business-validation ownership.Must-have technical competencies:5–6 years’ professional engineering experience with independent production ownershipStrong Python, PySpark, and SQL, including complex transformations, query optimization, reusable packages, debugging, tests, and reconciliationStrong ETL/ELT and batch-pipeline engineering across relational warehouses, APIs, object storage, and file-based ingestionExperience with a cloud data lake/lakehouse, Medallion patterns, schema/interface contracts, data-quality controls, orchestration, observability, and incident recoverySolid Git engineering practices: branching, pull requests, reviews, automated tests, deployment controls, and documentationProven ability to translate business/data requirements into filters, mappings, transformations, validation rules, and operational workflowsStakeholder-facing problem solving: explain discrepancies, challenge incomplete requirements, establish technical owners, and drive issues to closure.Nice-to-have:Strong preference for Azure Data Factory, Azure Databricks, Databricks Jobs/API, Delta Lake, Snowflake, Azure Blob Storage/Data Lake, Power BI, GitHub, and Azure Key VaultValuable experience with Pandera or equivalent schema-validation frameworks, Streamlit, Managed Identity/service principals, Azure Communication Services, Managed VNet/Private Endpoints, and Dev/Prod release practicesExperience with PFME/media data, syndicated sales, marketing hierarchies, MMM inputs, or multi-market data onboarding is preferred but can be learned.We offer:Flexible working format - remote, office-based or flexibleA competitive salary and good compensation packagePersonalized career growthProfessional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)Active tech communities with regular knowledge sharingEducation reimbursementMemorable anniversary presentsCorporate events and team buildingsOther location-specific benefits