Lead Engineer, Big Data - Databricks in New York at Jobgether

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Lead Engineer, Big Data - Databricks based in United States.

This is a senior technical leadership role focused on building enterprise-scale data and analytics solutions in a healthcare environment.
You will lead the design and delivery of cloud-based data architectures supporting complex Medicare and Medicaid operations.
The role combines hands-on engineering with strategic partnership, translating business challenges into scalable, data-driven solutions.
You’ll work extensively with Azure Databricks, Data Factory, Synapse, Data Lake, and machine learning services.
A strong emphasis is placed on AI-first development, real-time data processing, automation, data quality, and advanced analytics.
You will also mentor engineers, influence technical direction, and collaborate with business and executive stakeholders across a highly matrixed organization.
This opportunity is ideal for an experienced data engineer who wants to connect modern cloud technology with meaningful healthcare outcomes.

• Lead end-to-end data engineering initiatives as a Forward Deployed Engineer, taking ownership of technical solutions and mentoring team members in business-focused problem solving.
• Provide technical leadership and subject matter expertise across Azure Databricks, Azure Synapse, Azure Data Factory, and Azure Data Lake Storage.
• Design and implement scalable, high-performance data architectures for large healthcare datasets, with an emphasis on reliability and maintainability.
• Champion AI-first development practices, including AI-assisted coding workflows and Azure Machine Learning solutions for predictive healthcare analytics.
• Develop business intelligence solutions using tools such as Power BI and Tableau, integrated with the broader Azure data ecosystem.
• Analyze business processes and identify opportunities to apply Azure data, AI, and platform services to improve Medicare and Medicaid operations.
• Build reusable data-quality frameworks and automated validation processes using Azure DevOps and quality-assurance automation tools.
• Lead integration initiatives connecting the Azure data platform with low-code and enterprise platforms such as Pega and Salesforce.
• Translate business requirements into technical architecture, detailed designs, and production-ready implementations for healthcare analytics use cases.
• Design analytical data models that enable self-service reporting for clinical, financial, and operational stakeholders.
• Implement batch and real-time data pipelines supporting claims processing, quality measurement, and population health analytics.
• Partner with business leaders to identify, evaluate, and prioritize high-value data opportunities.
• Lead technical issue resolution and coordinate with cloud support, infrastructure, and external vendor teams when required.
• Support project planning, scheduling, and delivery using Agile methodologies.
• Maintain compliance with HIPAA requirements and applicable healthcare data governance standards.
• Train, mentor, and support new and existing team members across Azure data technologies and Forward Deployed Engineering practices.

• At least 6 years of experience in data management or data engineering across cloud data platforms, or an equivalent combination of relevant education and experience.
• Bachelor's degree in information technology, computer science, statistics, business, healthcare, engineering, or a related discipline.
• Strong hands-on experience with Microsoft Azure data services, particularly Azure Databricks, Azure Data Factory, Azure Synapse, Azure Data Lake Storage, and Azure Machine Learning.
• Demonstrated experience implementing AI-assisted development practices and machine learning solutions.
• Strong proficiency in Python, PySpark, Scala, and SQL, together with experience in automated quality assurance and data validation.
• Experience designing and implementing real-time streaming architectures using technologies such as Kafka, Azure Event Hubs, or Spark Streaming.
• Experience integrating data platforms with enterprise or low-code systems such as Pega and Salesforce.
• Healthcare payer experience, particularly with Medicare/Medicaid claims, eligibility, HEDIS or Stars quality metrics, and regulatory reporting, is highly valued.
• Solid understanding of HIPAA requirements and healthcare data governance frameworks.
• Experience with Azure DevOps CI/CD pipelines and Infrastructure as Code tools such as Terraform or ARM templates.
• Strong critical-thinking, analytical, and problem-solving abilities with a focus on measurable business outcomes.
• Ability to mentor engineers and promote an ownership-oriented, business-focused engineering mindset.
• Strong communication skills, with the ability to explain complex technical concepts and translate business requirements for technical and executive audiences.
• Demonstrated ability to collaborate effectively across teams in a complex, highly matrixed environment.
• Experience with Microsoft 365/O365, Power BI, and Azure DevOps.
• Azure Data Engineer certification and additional healthcare industry experience are preferred.

• Annual salary range of $96,325.57–$208,705.40 USD, with actual compensation determined by factors such as geographic location, experience, education, and skill level.
• Competitive overall compensation package.
• Comprehensive employee benefits package.
• Full-time remote work opportunity within the United States.
• Opportunity to work with modern Azure cloud, Databricks, AI, machine learning, and big data technologies.
• Opportunity to lead high-impact healthcare data initiatives supporting clinical, financial, operational, and population health outcomes.
• Strong opportunities for technical leadership, mentoring, and professional development.
• Collaborative environment involving engineering, business, healthcare, infrastructure, and executive stakeholders.