Principal Data & AI Platform Architect – Azure Databricks

Reporting to the Senior Director of Data & AI, the Senior Data Architect is a hands-on technical leader responsible for the architecture, engineering, governance, and optimization of UVVC’s enterprise data platform. This role will define technical standards and design scalable data pipelines, medallion data products, semantic models, and governed analytics solutions that enable trusted self-service analytics and AI-driven insights across UVVC. Responsibilities Data Pipeline Development & Orchestration Design, build, and optimize batch and streaming ETL/ELT pipelines and reusable ingestion frameworks using Azure Data Factory and Databricks across APIs, databases, SaaS platforms, and internal systems. Build scalable Delta Lake transformation frameworks using medallion architecture, Spark, and SQL. Implement CI/CD, parameterization, triggers, and pipeline automation best practices. Azure Data Platform Engineering Architect, manage, and optimize enterprise data environments across Azure Data Lake Storage Gen2 (ADLS Gen2), Azure SQL, and Databricks, including serverless and classic compute strategies, cost governance, and workload isolation strategies. Implement DataOps practices including testing, version control, monitoring, and documentation. Unity Catalog, Security & Data Governance Design and administer the enterprise Unity Catalog structure, including catalogs, schemas, external locations, storage credentials, groups, service principals, and ownership models. Implement least-privilege access, governed tags, attribute-based access-control policies, row-level filters, and column-level masking for PHI, PII, financial, and other sensitive information. Establish standards for data classification, lineage, auditability, stewardship, retention, certification, and access reviews. Partner with Security, Compliance, Privacy, and business data owners to ensure data solutions align with HIPAA and organizational security requirements. Govern tables, views, volumes, functions, metric views, dashboards, models, and Genie Agents through Unity Catalog. AI/BI, Semantic Models & Genie Agents Design and develop Databricks AI/BI Dashboards and domain-specific Genie Agents for clinical, operational, financial, RCM, marketing, and executive use cases. Configure trusted datasets, joins, business terminology, instructions, example queries, dimensions, measures, synonyms, and approved KPI definitions. Develop and govern Unity Catalog metric views and semantic definitions to ensure consistent reporting across Databricks AI/BI and Power BI. Establish testing and monitoring processes for Genie Agent accuracy, data grounding, security, explainability, performance, and user adoption. Ensure that AI-generated results respect Unity Catalog permissions and approved business definitions. Databricks Architecture & Platform Ownership Design and implement enterprise-grade Databricks Lakehouse Medallion architecture (Bronze, Silver, Gold layers). Define and enforce data engineering standards, naming conventions, and architectural patterns across all pipelines. Lead the architecture of Delta Lake design patterns, including partitioning, optimization, and data lifecycle management. Establish scalable serverless and classic compute strategies, job orchestration frameworks, and workspace organization. Evaluate and implement new Databricks capabilities and ensure alignment with enterprise data strategy . Cross-Functional Collaboration Work closely with clinical, sales, marketing, finance, RCM, operations, HR, and IT teams to understand business needs. Provide technical guidance on data engineering patterns and platform capabilities. Clearly communicate progress, risks, and technical decisions to data stakeholders and leadership. Data Modeling and Enterprise Metrics Develop conceptual, logical, dimensional, and physical data models supporting clinical, operational, financial, marketing, RCM, and workforce analytics. Establish conformed dimensions, master and reference…