Machine Learning Engineer in at Revelation Pharma LLC

Overview

The Machine Learning Engineer owns the intelligence layer that makes Evergreen's agents clinically meaningful rather than simple rule executors. This person designs, trains, validates, and monitors the ML models that power the Markov state transition logic, clinical decision support, patient risk stratification, adherence prediction, and dosage optimization. They work at the intersection of clinical domain knowledge and applied ML — building models that must be explainable to providers, safe for patients, and auditable by regulators. As the architecture matures, this role expands into the predictive analytics layer (Amazon Forecast DeepAR+ for churn risk, adherence forecasting) and the evaluation frameworks that determine whether agent recommendations meet clinical safety thresholds. This is not a research role — it is an applied ML engineering role where every model must ship, every prediction must be defensible, and every failure mode must be anticipated.

Job Activities and Responsibilities: The responsibilities listed below are subject to change.

• Development and tuning of agentic workflows: routing logic, tool use, context management, and prompt/config iteration
• Evaluation frameworks that measure agent quality against real outcomes, not just spot checks
• Production monitoring for agent behavior, including failure modes and escalation paths

• Build the evaluation and validation framework for all agent-driven clinical recommendations — including safety guardrails, confidence thresholds, and human-in-the-loop escalation triggers for the Clinical Protocol Agent
• Develop patient risk stratification models for adherence prediction, adverse event likelihood, dosage titration optimization, and churn/dropout risk using clinical, behavioral, and engagement signals
• Implement and manage the predictive analytics pipeline on AWS — Amazon Forecast (DeepAR+) for time-series clinical predictions, S3 Vectors for embedding-based patient similarity and retrieval, and Bedrock for agent inference
• Design and build the RAG architecture that grounds agent responses in clinical protocols, formulary data
• Own model lifecycle management — training pipelines, feature stores, model versioning, A/B testing, drift detection, and retraining triggers in production
• Build explainability layers for clinical recommendations
• Collaborate with the clinical team to translate clinical protocols and pharmacy domain knowledge into model features, training labels, and validation criteria
• Establish model monitoring and alerting — prediction quality dashboards, distribution shift detection, and automated alerts when model performance degrades below clinical safety thresholds
• Work with the DevOps/AgentOps teammate to ensure all ML decisions are logged, reproducible, and auditable for regulatory review

Required Technical Skills

(Staff-Level Depth Expected)

• Applied ML / Statistical Modeling (Expert): Deep experience with classification, regression, time-series forecasting, and probabilistic graphical models (Markov chains, HMMs, Bayesian networks). Must have shipped ML models to production — not just notebooks. Strong foundation in experimental design, A/B testing methodology, and statistical significance in clinical contexts.
• NLP & LLM Engineering (Expert): Production experience with LLM-based systems — prompt engineering, fine-tuning, RAG architecture design, embedding models, and output evaluation. Must understand hallucination mitigation, grounding techniques, and how to constrain LLM outputs to clinically safe boundaries. Experience with Bedrock (Claude, Titan) preferred; equivalent depth with OpenAI, Vertex AI, or Azure OpenAI acceptable.
• AWS ML Services (Strong): Hands-on experience with at least 3 of: SageMaker (training, endpoints, pipelines), Bedrock (agents, knowledge bases, model invocation), Amazon Forecast (DeepAR+, predictor optimization), S3 Vectors, Comprehend Medical, or HealthLake analytics. Must be comfortable building end-to-end ML pipelines on AWS.
• Feature Engineering & Data Pipelines (Strong): Experience building feature stores and training data pipelines from healthcare data sources. Must understand FHIR resource structures well enough to extract clinically meaningful features from Patient, Observation, Condition, MedicationRequest, and related resources. Experience with data normalization challenges (free-text to structured, terminology mapping) is critical.
• Model Evaluation & Safety (Strong): Experience designing evaluation frameworks for high-stakes predictions — clinical decision support, medical device software, or similarly regulated domains. Must understand sensitivity/specificity tradeoffs in clinical contexts, how to set appropriate confidence thresholds, and when a model should defer to human judgment.
• MLOps & Production ML (Strong): Model versioning (MLflow, SageMaker Model Registry, or equivalent), automated retraining pipelines, drift detection, shadow deployments, and canary rollouts for model updates. Must have experience monitoring models in production beyond accuracy metrics — latency, cost, fairness, and distributional stability.

Required Experience

• Strong coding skills, with production experience in Python
• Experience building and tuning agentic systems: multi-agent orchestration, tool use, retrieval-augmented generation, or LLM-based workflows
• Experience shipping ML or AI features to production, not just prototypes
• Comfort working with evaluation frameworks: building test sets, running evals, and iterating on prompts, tools, or configuration based on results

Preferred Qualifications

• Experience with cloud-based ML/AI training and deployment infrastructure
• Experience in healthcare, clinical, or another regulated domain
• Experience designing guardrails or safety checks for AI systems making consequential decisions

Physical Requirements:

• Proficient in using a computer and related equipment, including printers and fax machines.
• Ability to sit or stand for extended periods.
• Effective communication skills via telephone and email.
• Capable of lifting up to 40 pounds as needed.

Benefits: