Human-in-the-Loop Machine Learning

University of Amsterdam
Masters of AI: Semester 1, Block 2, 2023
Course Coordinators: Eric Nalisnick and Mohammad Aliannejadi
Teaching Assistants: Rajeev Verma, Dharmesh Tailor, Alexander Timans

Machine learning (ML) is being deployed in increasingly consequential tasks, such as healthcare and autonomous driving. For the foreseeable future, successfully deploying ML in such settings will require close collaboration and integration with humans, whether they be users, designers, engineers, policy-makers, etc. This course will look at how humans can be incorporated into the foundations of ML in a principled way. The course will be broken down (unequally) into three parts: demonstration, collaboration, and oversight. Demonstration is about how machines can learn from 'observing' humans---such as learning to drive a car from data collected while humans drive. In this setting, the human is assumed to be strictly better than the machine and so the primary goal is to transmit the human's knowledge and abilities into the ML model. The second part, collaboration, is about when humans and models are near equals in performance but not in abilities. A relevant setting is AI-assisted healthcare: perhaps a human radiologist and ML model are good at diagnosising different kinds of diseases. Thus we will look at methodologies that allow machines to `ask for help' when they are either unconfident in their own performance and/or think the human can better carry out the task. Lastly, the course will close with the setting in which machines are strictly better at a task than humans are, but we still wish to monitor them to ensure safety and alignment with our goals (oversight).

Schedule and Topics

Dates Topics Readings
Part I: Demonstration
31.10.23 Logistics, introduction, building prior knowledge into models
  1. Do we still need models or just more data and compute? by Welling
  2. Prior Probabilities by Jaynes
02.11.23 Crowdsourcing, Dawid-Skene model
  1. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm by Dawid & Skene
  2. Learning From Crowds by Raykar et al.
  3. Who Said What: Modeling Individual Labelers Improves Classification by Guan et al.
07.11.23 Active learning
  1. Active Learning Literature Survey by Settles
  2. Active Learning from Crowds by Yan et al.
09.11.23 Practical on crowdsourcing platforms
14.11.23 Imitation learning: overview of reinforcement learning (RL), behavior cloning, distribution matching
  1. Lecture 1: Introduction to RL by Silver
  2. TO DO by
  3. Generative Adversarial Imitation Learning by Ho & Ermon
16.11.23 Imitation learning continued: inverse RL, RL from human feedback, direct preference optimization
Part II: Collaboration
21.11.23 Introduction to human-AI collaboration ('hybrid intelligence'), combining human and machine decisions
23.11.23 Learning to reject, learning to defer
05.12.23 AI-assisted decision making, multi-agent RL, collaborative inverse RL
07.12.23 Large language models: prompting and in-context learning as demonstration and collaboration
Part III: Oversight
12.12.23 AI alignment, off-switch game
14.12.23 Iterated amplification, course summary