NLML Researcher, Post-Training and Safety
Compassion Aligned Machine LearningRemote
Global catastrophic risksAI safetyResearchSoftware engineering
Why this is on the board
If you teach a language model certain values, do they hold up when someone tries to mislead it or trains the model further? What you discover about this using open models is published openly, so that other researchers and AI companies can check it and adopt it, even if it turns out that a common approach falls short.
This board is in beta and this is our judgement. If you think we have it wrong, tell us. Give feedback · How we reason about it
About the role
Run LLM training experiments on value instillation through mid-training data and post-training RL, then evaluate whether those values persist. - Design and execute RL post-training (RLHF, DPO, GRPO) and mid-training synthetic data experiments on 12B-scale open-weight models. - Build evals to measure value persistence under adversarial prompting, distribution shift, and training-induced degradation like reward hacking. - Write up results for arXiv and LessWrong; flag broken findings and disagr…
More about Compassion Aligned Machine LearningWhy this problemGlossary
Want to understand this first?
Start with our free course: about 100 minutes in six short units, at your own pace, with no sign-up. To go further, apply for the EA Fellowship: five weeks of reading and discussion in a small group. Also free, but selective.