Alerts
Posted:
Rex.zone is seeking a Remote AI/ML Data Annotation Engineer to support advanced AI and machine learning training workflows across the United States. This role focuses on creating high-quality annotation processes, improving training data quality, and supporting evaluation programs that enhance the performance of large language models (LLMs).
The ideal candidate will have experience with AI/ML data operations, annotation workflows, RLHF-style preference data, and quality evaluation processes. The successful applicant will work closely with engineering teams to develop scalable labeling systems, improve model performance, and ensure reliable training data outcomes.
Key Responsibilities:
• Build and maintain high-quality annotation workflows for AI/ML training projects.
• Develop labeling schemas, annotation guidelines, and ambiguity-resolution processes.
• Monitor training data quality metrics, error patterns, and continuous improvement plans.
• Design and manage QA evaluation programs, including review queues, spot checks, gold tasks, and agreement analysis.
• Perform RLHF preference labeling and ranking to support AI model alignment and performance.
• Conduct prompt evaluations and regression testing to measure model improvements.
• Support NLP tasks including named entity recognition and intent classification.
• Perform computer vision annotation tasks such as bounding boxes and segmentation when required.
• Execute content safety labeling to support responsible AI development.
• Collaborate with engineering teams to integrate annotation tools and datasets into LLM training pipelines.
Requirements:
• 3+ years of experience in data operations, ML operations, evaluation, or annotation engineering.
• Strong understanding of NLP, LLM evaluation, and human feedback signals such as RLHF.
• Experience with QA evaluation methods, sampling strategies, and disagreement analysis.
• Familiarity with annotation tools and workflow automation.
• Strong technical writing skills for creating clear guidelines and repeatable processes.
• Excellent analytical, problem-solving, and communication skills.
Strong knowledge of:
• AI/ML Data Annotation
• Large Language Model (LLM) Evaluation
• RLHF & Human Feedback Systems
• NLP Tasks & Model Testing
• QA Evaluation Programs
• Annotation Tools & Workflow Automation
• Data Quality Management
This tool helps you see how closely your CV matches a job description. It also gives you simple suggestions on what to improve so you have a better chance of getting shortlisted.
