Software Engineer – LLM Evaluation & Repository Validation

We are looking for software engineers to validate real-world repositories used to evaluate large language models. You will triage issues, set up reproducible environments, assess test quality, and check how well models fix real bugs.

Key Responsibilities
- Review and triage issues in open-source GitHub projects.
- Prepare repositories for evaluation, including containerisation and environment automation.
- Assess how thorough and well written the unit tests are.
- Edit and run code locally to see how LLMs perform on bug-fixing tasks.
- Help pick repositories and issues that are hard for LLMs.
- Write up findings clearly for researchers and reviewers.
- Follow project guidelines, take part in calibration and review sessions, and incorporate feedback to keep quality consistent.

Required Qualifications
- 3+ years of software engineering experience.
- Strong experience in at least one of Python, C++, C#, Go, Rust or Ruby.
- Proficiency with Git, Docker and basic pipeline setup.
- Ability to navigate complex code bases and run real-world projects locally.
- Fluent written English.
- Reliable, detail-oriented, and able to work independently and meet deadlines in a remote setting.

Preferred Qualifications
- Prior LLM research or evaluation projects.
- Experience building or testing developer tools or automation agents.
- Open-source contribution or evaluation experience.

Rights, Confidentiality and Data Handling
- Submit only work that you created yourself or that you have the right to share.
- Do not submit material covered by an employer, client or third-party confidentiality agreement, or material you have no right to share.
- Do not submit export-controlled, military or defence-related technical data.
- Remove credentials, secrets and personal data from anything you submit.
- Rex may ask for evidence of ownership or licence, may verify submissions, and may reject or remove any submission.

Legal and Compliance Notice
- All content you submit must be lawful, and you must have the right to share it.
- Do not submit confidential or proprietary material belonging to an employer, client or third party, or personal data without a lawful basis.
- Rex may verify identity, credentials, ownership and licensing; may reject or remove any submission; and may report unlawful submissions to the relevant authorities.
- Engagement and payment are subject to successful verification and to applicable sanctions, export-control, tax and employment laws.
- The contractor agreement will set out ownership and licence terms for the work you deliver (to be confirmed by Rex Legal).