Lead/Senior Inference Engineer in New York at Jobgether
Job Description
This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Lead/Senior Inference Engineer based in United States .
The Lead/Senior Inference Engineer will build and operate the production inference infrastructure powering advanced large language models and AI-driven professional applications. This hands-on engineering role focuses on developing a reliable, scalable model-serving platform while optimizing performance, latency, and infrastructure costs. Working within a newly established engineering team, you will help shape its technical direction, engineering practices, and operational standards from the ground up. You will collaborate closely with model training, evaluation, research, and product teams to bring new model releases into production. The role offers significant autonomy and the opportunity to influence infrastructure strategy, reliability, and international platform expansion in a high-stakes, compliance-focused environment.
AccountabilitiesBuild, maintain, and operate production-grade model-serving infrastructure supporting large-scale AI workloads.
Implement and optimize inference techniques, including batching, caching, quantization, and comparable approaches, to improve latency and reduce operating costs without compromising model quality.
Strengthen platform observability, monitoring, and operational visibility as traffic volumes and deployment requirements increase.
Develop the operational and compliance foundations required for production growth, including controls and evidence supporting the ISO 27001 compliance path.
Establish cost-tracking and attribution mechanisms that support financial planning, as well as GPU-capacity contingency plans that help prevent service interruptions and unnecessary parallel infrastructure costs.
Evaluate and document trade-offs between cost, latency, and reliability, communicating technical recommendations to engineering leadership and partner teams.
Work directly with model training and evaluation teams to integrate new model releases into the serving infrastructure.
Help establish engineering practices and tools, including on-call processes, incident reviews, onboarding, and operational documentation.
Collaborate with research, product, and engineering stakeholders to ensure the serving platform evolves alongside model capabilities and product requirements.
Contribute to the team's technical strategy and mentor other engineers as the function grows.
Hands-on infrastructure engineering experience with Kubernetes or comparable technologies, including ownership of production systems beyond prototyping.
Experience operating and maintaining model services across different workloads, including peak traffic, scaling, disaster recovery, and service reliability.
Direct experience with model-serving infrastructure and/or inference optimization for large language models, at a scale appropriate to previous roles.
Strong background in large-scale distributed systems, particularly production scalability, availability, and reliability.
Proven ability to troubleshoot distributed systems and resolve production incidents under real traffic conditions.
Demonstrated experience delivering software end to end, from technical design and implementation through deployment and ongoing operations.
Ability to establish and uphold engineering best practices while adapting to evolving product and business requirements.
Strong autonomy, technical judgment, and ownership, with the ability to operate as an early lead engineer and mentor colleagues as the team expands.
Preferred qualifications:
Experience with LLM-serving technologies such as vLLM, TensorRT-LLM, Triton, SGLang, or similar frameworks.
Exposure to GPU-cluster capacity planning and/or multi-region or multi-cloud deployment environments.
Experience working in regulated or compliance-intensive environments, including ISO 27001, SOC 2, or comparable standards.
Familiarity with systems programming languages such as Rust or Go.
Knowledge of systems-level performance optimization, including rate limiting, request routing, and identifying bottlenecks across networking, hardware, and lower-level software.
Competitive compensation: CAD $140,000–$175,000 annual base salary for Ontario, Canada.
Potential eligibility for an annual performance bonus based on individual and organizational results.
Comprehensive benefits and wellbeing programs, including retirement savings, flexible vacation, and employee incentive programs.
Flexible work arrangements, including the possibility of working from anywhere for up to eight weeks per year.
Two company-wide mental health days annually and access to the Headspace app.
Learning and career development opportunities, including skills development and tuition reimbursement.
Resources supporting mental, physical, and financial wellbeing.
Two paid volunteer days per year, along with opportunities to participate in community projects and environmental and social impact initiatives.
Inclusive workplace culture focused on collaboration, professional growth, flexibility, and work-life balance.
Scan to Apply
Just scan this QR code to apply from your phone.
- Fast
- Easy
- Apply on the go