Principal Architect - Databricks

Meridian Softworks is a fast-growing advanced analytics consulting firm. Our consultants bring deep expertise in Data Science, Machine Learning and AI. We are the trusted analytics partner for several Fortune 100 companies, enabling them to generate business value from data. Our business value and leadership has been recognized by various market research firms, including Forrester and Gartner. We are looking for top-notch talent as we continue to build the best analytics global consulting team in the world. Job Summary We are seeking a highly skilled Principal Data Architect to design and implement our next-generation data ecosystem. You will lead the technical strategy for end-to-end data management, data warehousing, and IoT ingestion pipelines. The ideal candidate brings mastery in cloud data platforms (Azure or AWS) and advanced data governance. Key Responsibilities Data Architecture: Design Databricks-driven enterprise data warehousing (EDW), lakehouse, and business intelligence (BI) systems. IoT & Stream Ingestion: Build scalable pipelines using Azure Hub, Device SDK, DPS, and Stream Analytics. Data Management: Implement information lifecycle management (ILM) from data ingestion to archiving. Data Strategy: Define, build, and enforce global data governance and master data management practices. Dataset Creation: Extract and manipulate unstructured data into connected datasets for advanced analytics. Requirements Core Cloud Data Platform (Must Have) Databricks: Strong proficiency in building optimized spark pipelines, delta lakes, and lakehouses. Cloud Ecosystem (Option A: Azure Suite) IoT Infrastructure: Azure Hub, Device SDK, Device Provisioning Services (DPS), and Device Twins. Compute & Storage: Azure Functions, Cosmos DB, and Azure Stream Analytics. Cloud Ecosystem (Option B: AWS Alternative) IoT Infrastructure: AWS IoT Core, IoT Device Management, and IoT Device Defender. Compute & Storage: AWS Lambda, DynamoDB, Amazon Kinesis, and EMR. Data Engineering & Governance BI & EDW: Deep expertise in enterprise data warehousing and business intelligence tools. Data Modeling: Proven ability to manage both structured and unstructured data assets. Qualifications & Soft Skills Communication: Ability to translate complex technical concepts clearly to diverse technical teams. Collaboration: Demonstrated teamwork skills with a proven track record in cross-functional environments. Experience: Extensive background in end-to-end data management principles and data lifecycle execution. Good to have-Experience in architecting and building Azure IoT platforms/apps with good exposure to containerization and server less architecture. Benefits Significant career development opportunities exist as the company grows. The position offers a unique opportunity to be part of a small, challenging, and entrepreneurial environment, with a high degree of individual responsibility. Meridian Softworks provides equal employment opportunities to applicants and employees without regard to race, color, religion, age, sex, sexual orientation, gender identity/expression, pregnancy, national origin, ancestry, marital status, protected veteran status, disability status, or any other basis as protected by federal, state, or local law. Originally posted on Himalayas