Senior Data Engineer (GCP • Python • Iceberg • Delta Lake • Kafka • Snowflake • Databricks) in New York at Jobgether

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Data Engineer (GCP • Python • Iceberg • Delta Lake • Kafka • Snowflake • Databricks) based in United States.

This is a senior-level data engineering opportunity focused on building modern, scalable lakehouse architectures in a cloud-native GCP environment.
You will design and implement reusable data-sharing and ingestion solutions that connect data products across BigQuery, Snowflake, and Databricks.
The role combines hands-on development in Python and Spark with advanced work across Apache Iceberg, Delta Lake, Iceberg UniForm, Delta Sharing, and Kafka-based CDC.
You will help establish reliable, governed data access patterns while enabling zero-copy data consumption across multiple platforms.
Your work will span ingestion, metadata, lineage, authorization, semantic models, and production troubleshooting across the data stack.
You will collaborate closely with cross-functional teams to deliver end-to-end capabilities while maintaining strong standards for scalability, quality, and maintainability.
This role is well suited to an experienced data engineer who enjoys solving complex distributed data challenges and shaping modern enterprise data infrastructure.

• Design and implement the Delta Lake write layer using Iceberg UniForm to maintain dual Delta and Iceberg metadata and enable data consumption across multiple platforms without data conversion.
• Build scalable GCS-to-BigQuery ingestion pipelines for structured operational datasets.
• Develop robust data engineering solutions using Python and Apache Spark.
• Implement Kafka-based change data capture (CDC) patterns for real-time and near-real-time data ingestion.
• Design reusable, modular data-sharing adapters and components that support scalable lakehouse integrations.
• Develop data lineage, dependency tracking, and metadata capabilities to improve data governance and operational visibility.
• Configure Snowflake Horizon external tables to enable secure, zero-copy access to shared data.
• Implement and certify Delta Sharing endpoints to support data consumption by Databricks users and workloads.
• Build governed data access components, including role-based access control (RBAC), connector registry entries, and tenant-scoped authorization.
• Align semantic-layer models with LookML definitions and established KPI catalogs.
• Collaborate with cross-functional engineering and business teams to deliver complete, production-ready data features.
• Troubleshoot complex issues across ingestion, storage, metadata, sharing, integration, and application layers while contributing to code quality and engineering best practices.

• 6+ years of proven enterprise-level data engineering experience with strong hands-on expertise in Python and Apache Spark.
• Advanced experience working with Google Cloud Platform (GCP), particularly BigQuery.
• Strong practical knowledge of Apache Iceberg, Delta Lake, and Iceberg UniForm.
• Experience implementing Delta Sharing and building Kafka-based CDC and event-driven data pipelines.
• Hands-on experience with Snowflake Horizon Catalog and Databricks Unity Catalog.
• Solid understanding of modern data lake and lakehouse architectures, including ingestion pipeline design and distributed data processing.
• Experience designing scalable, reusable, and maintainable data integration and data-sharing solutions.
• Strong understanding of data lineage, metadata, governance, access controls, and dependency management.
• Ability to troubleshoot complex technical issues across multiple layers of a modern data platform.
• Excellent written and verbal communication skills, with the ability to collaborate effectively across multiple technical and business teams.
• Strong problem-solving skills and the ability to work independently while contributing effectively to a collaborative engineering environment.
• Experience in enterprise SaaS, media, or advertising technology environments is a plus.
• Active experience using AI-assisted software development tools is preferred, with Claude Code experience considered an advantage.

• Competitive compensation package.
• 100% remote work within the United States.
• Full-time employment opportunity.
• Medical, dental, and vision insurance.
• Pet insurance.
• Paid Time Off (PTO).
• 401(k) retirement plan.
• Opportunity to work on modern cloud data technologies including GCP, BigQuery, Snowflake, Databricks, Iceberg, Delta Lake, and Kafka.
• Opportunity to contribute to sophisticated lakehouse and data-sharing architectures with significant technical ownership.
• Collaborative environment focused on high-quality software engineering and modern development practices.