Business Domain Expert — AI Model Evaluation Pilot (Admin, Marketing, HR, Accounting) in Canada Creek, Nova Scotia at Jobgether

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Business Domain Expert — AI Model Evaluation Pilot (Admin, Marketing, HR, Accounting) based in Canada.

This is a short-term, structured opportunity for experienced business professionals to help evaluate the quality of next-generation AI models.
You will complete one realistic professional task based on your domain expertise and then assess five AI-generated attempts at that same task.
Your professional judgment will serve as the benchmark for evaluating accuracy, reasoning, clarity, and practical relevance.
You will provide evidence-based scores and detailed rationales that can directly influence future AI model development.
The study is designed for practitioners who understand how work is actually performed in real business environments.
It offers a focused way to contribute your expertise to AI research without requiring a long-term commitment.
The engagement is fully remote, flexible, and expected to require approximately 8–12 hours of your time.

• Sign a confidentiality agreement before receiving project materials and ensure all study information is handled appropriately.
• Complete one realistic, domain-specific professional task using the source materials provided.
• Review five AI-generated responses to the same task and rank them from 1 to 5 with no ties.
• Assign each model response a score from 0–100 based on the quality of the work and your professional judgment.
• Write clear, specific, evidence-based rationales that reference relevant portions of each response to support your evaluation.
• Assess the accuracy, reasoning, practical applicability, and overall quality of the model-generated work within your professional domain.
• Provide structured feedback on the task itself and the overall study design to help improve future evaluation processes.
• Complete all work independently without using LLMs or other AI assistants during task completion, evaluation, scoring, or rationale writing.

• 5+ years of hands-on professional experience in Human Resources or Accounting.
• Current or recent professional practice in your field, with sufficient experience to evaluate work according to real-world professional standards.
• Strong ability to assess professional work critically and distinguish high-quality reasoning from flawed, incomplete, or unsupported conclusions.
• Excellent written communication skills, with the ability to produce concise, specific, and evidence-grounded explanations for your judgments.
• Strong attention to detail, analytical thinking, and professional judgment.
• Ability to work reliably within a short turnaround period and complete a structured evaluation independently.
• Comfortable working without a predefined rubric or “golden answer,” using your own professional expertise as the primary basis for evaluation.
• Must be legally able to participate as an independent contractor in Canada.
• Must not have previously worked on Project Alchemy in any capacity or accessed its source-world data, as prior exposure would invalidate the study.
• Candidates requiring H-1B or STEM OPT sponsorship are not eligible for this engagement.

• Fully remote, independent-contractor engagement that can be completed on your own schedule.
• Estimated commitment of 8–12 hours for the one-time evaluation study.
• Opportunity to contribute directly to the development and evaluation of advanced AI systems.
• Apply your professional expertise to realistic business scenarios without a long-term production commitment.
• Competitive payment for completed work, with weekly payments via Stripe or Wise.
• Flexible project structure; engagements may be extended, shortened, or concluded early depending on project needs and performance.
• No access to confidential or proprietary information belonging to your current or former employers, clients, or institutions.
• Opportunity to earn up to $400 per successful referral, subject to applicable restrictions.