Machine Learning Engineer in Brazil, Indiana at Jobgether

This position is posted by Jobgether on behalf of a partner company. We are currently looking for a Machine Learning Engineer in Brazil.

This role is part of an AI Quality and Governance team focused on ensuring the reliability, performance, and responsible use of internal Machine Learning and Generative AI systems. The engineer will help design and implement advanced evaluation frameworks that measure model quality across key dimensions such as accuracy, relevance, latency, cost, and token usage. Working in a highly technical and fast-evolving environment, this position combines hands-on engineering with cutting-edge experimentation in LLM and agent-based systems. The role plays a critical part in building observability and guardrails for scalable AI platforms used in real-world applications. It requires strong analytical thinking and deep technical expertise in ML systems. This is an opportunity to work at the forefront of GenAI evaluation and production-grade AI engineering.

• Design and develop scalable systems that support ML and GenAI development with strong observability and reliability.
• Build and improve evaluation frameworks for ML and LLM-based systems, measuring relevance, accuracy, latency, cost, and token consumption.
• Design and execute LLM behavioral evaluations, including hallucination detection, grounding assessment, and multi-turn interaction testing.
• Apply advanced evaluation methodologies such as golden datasets, LLM-as-judge frameworks, and multi-trial metrics like pass@K.
• Develop and maintain APIs and services that expose AI models and evaluation pipelines for internal use.
• Contribute to ML infrastructure including deployment pipelines, monitoring systems, and performance optimization strategies.
• Write clean, scalable Python code and develop automated testing frameworks (unit, integration, and functional tests).
• Debug production issues related to ML performance, evaluation inconsistencies, and system reliability.
• Collaborate with cross-functional teams including product managers, engineers, and data teams to deliver end-to-end AI solutions.
• Support production systems and participate in on-call rotations when required, while exploring new GenAI evaluation techniques.

Requirements:

• 4+ years of experience in Machine Learning Engineering or an advanced degree in a related field.
• Strong proficiency in Python (primary language), with Java as a plus.
• Hands-on experience with AWS or equivalent cloud platforms.
• Solid understanding of the ML lifecycle, including model deployment, data pipelines, and CI/CD processes.
• Experience with Generative AI and LLM-based application development.
• Familiarity with ML/LLM evaluation frameworks such as RAGAS, DeepEval, or similar tools.
• Experience building or working with observability and monitoring systems for ML/GenAI applications.
• Strong understanding of evaluation metrics including accuracy, relevance, latency, and cost optimization.
• Experience with Docker and Kubernetes is preferred.
• Strong foundation in data structures, algorithms, and distributed systems.
• Experience working in Agile environments and debugging production systems.
• Basic understanding of APIs, HTTP protocols, and databases.
• Awareness of Responsible AI, AI governance, or standards such as ISO 42001 is a plus.
• Strong communication skills and ability to collaborate effectively across multidisciplinary teams.

Benefits:

• 100% remote work flexibility.
• Competitive salary in USD.
• Independent contractor agreement.
• Short-term contract structure.
• 2 weeks of paid time off (PTO).
• Holidays aligned with the US calendar.
• Full-time engagement with CST timezone alignment.
• Opportunity to work on advanced GenAI systems and evaluation frameworks.