Project Overview:

Join a growing community of professionals advancing the next wave of AI. As an AI Trainer, you’ll play a hands-on role by analyzing and providing feedback on data to improve LLM performance, helping ensure that the next generation of AI technology is accurate and trustworthy.

We are seeking a skilled Power Platform Governance Architect to work as a project consultant in our AI Labor Marketplace. This is not a full-time employment position — you will be engaged as an expert project consultant on a contract basis.

Location: U.S.-based experts only

Engagement: Part-time, project-based expert evaluation work

Work Type: Remote

Project Summary:

Design and produce structured evaluation datasets to train and benchmark an AI Advisor Agent operating on Microsoft Power Platform tenant-state data. The consultant will define evaluation schemas, scoring logic, and governance-aligned validation standards, and create approximately 400 structured JSON-based evaluation test cases across Metrics, Insights, Action Plan Step, and Action Plan Generation components.

Consultant Engagement Terms:

This is a project-based consultant role. Consultants will be paid on a per-project basis; hourly rates are estimates based on anticipated completion time. Consultants control their own schedule, provide their own tools, and may simultaneously provide services to other vendors/employers (subject to those vendors’ allowances).

Responsibilities:

Contributors will:

• Design structured tenant-state schemas for evaluation purposes
• Define evaluation task structure, including expected outputs and scoring criteria
• Develop governance-realistic synthetic Power Platform scenarios
• Author structured JSON inputs and expected outputs
• Create scoring rubrics and validation logic for agent evaluation
• Ensure logical correctness, schema integrity, and compliance alignment
• Produce ~400 gold-standard evaluation test cases over 12 weeks
• Document assumptions and edge-case handling standards

Expected Outcomes:

• A defined evaluation framework and scoring methodology
• A structured tenant-state schema for agent evaluation
• ~400 validated evaluation test cases
• Documented scoring and validation standards aligned to governance best practices

Qualifications:

• 6–10+ years of experience as a Power Platform Administrator or Architect
• Deep experience with tenant governance, DLP, environments, connectors, and ALM
• Hands-on experience with CoE Toolkit or tenant telemetry analysis
• Strong proficiency authoring and validating structured JSON
• Prior experience designing QA frameworks, evaluation rubrics, or scoring guidelines
• Ability to reason at enterprise governance depth

• Mid-Senior level
• Contract
• Software Development

Similar jobs