Databricks Lead Data Engineer in San Diego, California at K2 Insurance Services, LLC

Databricks Lead Data Engineer

Full Time

Remote - San Diego, CA, US

Salary Range: $200,000 To $260,000 Annually

Databricks Lead Data Engineer – K2 Insurance Services

K2 Insurance Services, LLC seeks a full-time Lead Databricks Data Engineer to spearhead our data engineering initiatives, focusing on designing, building, and optimizing scalable data solutions using Databricks. As a leader, you will supervise a team of data engineers, collaborate with cross-functional stakeholders, and define best practices to unlock the full potential of our data.

K2 Insurance Services offers the opportunity to join an established company in growth mode. Our compensation program includes competitive pay; bonus plan; medical, dental, and vision insurance; paid time-off in year of hire; and 401(k) with employer match.

Salary Range: $200,000 To $260,000 USD per year

Key Responsibilities:

• Leadership & Strategy:
  1. Supervise and mentor a team of data engineers in implementing Databricks-based ETL/ELT solutions.
  2. Define and drive the data engineering strategy, ensuring alignment with business goals.
  3. IT Product Owner role for organizational data warehouse project.
• Databricks Expertise:
  1. Design and develop scalable data pipelines and ETL/ELT processes using Databricks.
  2. Optimize and tune Spark jobs for performance and efficiency.
  3. Develop and enforce best practices for Databricks cluster management and data security.
• Data Architecture:
  1. Build and maintain robust data models to support analytics and reporting needs.
  2. Integrate Databricks with various data sources (cloud storage, databases, APIs).
  3. Implement Delta Lake for reliable, scalable, and performant data lakes.
• Collaboration:
  1. Work closely with data scientists, analysts, and stakeholders to deliver actionable insights.
  2. Act as the technical liaison between the data engineering team and other stakeholders.
• Innovation & Optimization:
  1. Stay updated on the latest Databricks and Azure features to drive continuous improvements.
  2. Automate repetitive processes and streamline data workflows.

Qualifications:

• Proven experience (10+ years) in data engineering, with 2+ years as a team supervisor or equivalent role.
• Expertise in Databricks, including Spark, Delta Lake, Delta Live Tables, and MLFlow.
• Proficiency in Python, SQL, and at least one cloud platform (Azure, AWS, or GCP).
• Hands-on experience with data lakes, data warehouses, and big data technologies.
• Strong understanding of CI/CD pipelines and infrastructure as code (IaC).
• Excellent problem-solving and communication skills.
• Prior insurance industry experience is a strong plus.

Preferred Skills:

• Familiarity with Databricks Unity Catalog and Lakehouse architecture.
• Experience with workflow orchestration tools.
• Knowledge of machine learning pipelines and integrations.
• Certifications in Databricks, Azure, AWS, or GCP.
• Previous IT product owner role.