Sr Data Engineer - TS/SCI w Poly - Herndon / Chantilly in Chantilly, Virginia at SpatialGIS LLC

Are you a highly experienced Data Engineer passionate about transforming diverse data into actionable insights? SpatialGIS Services is seeking a Senior Data Engineer to develop innovative tools, code, and services for comprehensive data engineering activities. You'll be at the forefront of moving structured and unstructured data, executing ingestion processes, and developing code to format data for exploration and analysis.

You'll analyze source data formats, collaborate with Data Scientists and partners to define optimal formats and transformations, and build solutions for both one-time and ongoing data extraction. This role involves developing new ETL processes, performing operations and maintenance on existing code, and ensuring robust documentation for all projects, including mappings and usage guides. Join our team and help us unlock the full potential of our data!

Key Responsibilities:
Design and optimize Data Pipelines using cutting-edge tools such as Spark, Apache Iceberg, Trino, OpenSearch, EMR cloud services, NiFi, and Kubernetes containers.

Ensure the pedigree and provenance of data are meticulously maintained to protect data access.

Clean and preprocess data effectively to enable access for advanced analytics.

Collaborate with enterprise working groups to advance and standardize data practices.

Work closely with the engineering team, data stewards, and mission partners to derive actionable value from data holdings.

Partner with software engineers to update, configure, and maintain data services based on evolving requirements.

Ensure data quality by collaborating with testing and data quality teams to enhance the standardization of data conditioning pipelines.

Adapt to various types and formats of data, integrating new data processing platforms in collaboration with development teams.

Required Skills:
Minimum of 10+ years' experience with:

Active TS/SCI w/Polygraph security clearance is required.

Data lifecycle engineering.

Development and maintenance of extract, transform, and load (ETL) tools and services.

Cloud and on-prem data storage and processing solutions.

Python, SQL, Spark, and other data engineering programming.

COTS and open-source data engineering tools such as ElasticSearch and NiFi.

Processing data within the Agile Lifecycle.

Preferred Skills/Experience:
Experience using AI/LLMs to accelerate the processing and transformation of data.