Principal Data Engineer in New York at Jobgether

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Principal Data Engineer based in United States.

This role offers the opportunity to lead the design and evolution of scalable, reliable data platforms within a complex enterprise environment. You will shape data architecture, lead technical decisions, and build robust pipelines and ETL solutions that support critical business and reporting needs. The position combines hands-on engineering with technical leadership, mentoring, and cross-functional collaboration. You will work across cloud, data engineering, application, infrastructure, and network teams to deliver secure and high-performing solutions. The role also involves guiding projects through the full software development lifecycle, from requirements and design through testing and production deployment. This is a senior technical position suited to an experienced data engineer who enjoys solving complex data challenges and improving enterprise-scale platforms.

• Design scalable and reliable data architectures, leading complex technical decisions related to data pipelines, ETL processes, data migration, performance, scalability, and platform reliability.
• Review technical implementations delivered by engineering team members and external resources, providing guidance on engineering standards and best practices.
• Analyze existing systems and identify opportunities to improve data processing, architecture, performance, and reliability.
• Create high-level technical designs covering ETL processes, data dictionaries, metadata, file layouts, data flows, validation, transformation, and loading requirements.
• Gather and analyze business and application requirements, identify gaps or inconsistencies, and translate them into effective technical solutions.
• Plan and execute Agile sprints while coordinating development, testing, deployment, and other phases of the software development lifecycle.
• Develop technical documentation for parallel jobs, batch processing, sequencers, and data migration activities.
• Participate in architecture, code, and design reviews with network, infrastructure, and solutions engineering teams to ensure alignment with enterprise standards.
• Lead and mentor junior engineers, promoting strong engineering practices and supporting technical growth across the team.

Requirements:

• Bachelor’s degree or foreign equivalent in Computer Science, Electronic Engineering, or a related field, followed by at least 7 years of progressive experience as a Senior Data Engineer, IT Consultant, or in a related role.
• At least 7 years of experience building data pipelines, data warehouse solutions, and frameworks supporting end-user reporting requirements.
• Strong experience gathering requirements, creating software designs, and developing efficient solutions within an Agile software development lifecycle.
• Proven ability to build data pipelines using diverse sources, including flat files, databases, APIs, and streaming applications.
• Hands-on experience building ETL and data warehousing solutions with IBM DataStage, Talend, and ODI.
• Strong experience designing data enrichment frameworks and processing large datasets using technologies such as Spark, Python, Talend, DataStage, ODI, Unix, PL/SQL, Teradata, AWS, and Java.
• Experience designing, developing, and automating ETL processes for data validation and reconciliation across multiple source systems.
• Proficiency with programming and scripting languages including Python, Unix, Java, JavaScript, C, and C++, including experience retrieving data from REST APIs and working with security keys.
• Extensive experience with AWS services, including S3, Glue, Athena, Lake Formation, Lambda, EC2, RDS, EMR, Redshift, and IAM, particularly for data migration and cloud-based processing.
• Experience working with cloud environments including AWS, Azure, and GCP and integrating cloud-based data with downstream systems.
• Experience developing and integrating APIs using REST web services.
• Familiarity with CI/CD tools such as Bitbucket, GitHub, and Jenkins.
• Strong technical leadership, communication, problem-solving, mentoring, and cross-functional collaboration skills.
• Ability to work remotely from any location within the United States.

Benefits:

• 100% remote work permitted from any US location.
• Full-time schedule of 40 hours per week.
• Additional compensation and benefits details were not specified in the source job description.