NLAI Evaluation & Agent Systems Engineer
SyntonyRemote
Global catastrophic risksAI safetySoftware engineeringResearch
Why this is on the board
Anyone who wants to know whether an AI system has become safer or riskier after an update needs a reliable way to measure it. Here you build test set-ups that other researchers and developers use to record and compare that behaviour. So you don't answer the safety question yourself; you build the measuring instrument that determines how well others can.
This board is in beta and this is our judgement. If you think we have it wrong, tell us. Give feedback · How we reason about it
About the role
Build evaluation software for AI models and agents, designing tests and measuring behavioural changes. - Develop evaluation harnesses for model behaviour, tool use, and multi-step agent tasks. - Record inputs, configurations, tool calls, and observations in structured traces. - Build replay and regression tests comparing system behaviour before and after changes. - Requires experience with evaluation software, model integrations, or agent systems. - Requires understanding of experiment design, n…
Want to understand this first?
Start with our free course: about 100 minutes in six short units, at your own pace, with no sign-up. To go further, apply for the EA Fellowship: five weeks of reading and discussion in a small group. Also free, but selective.