Ryunosuke Amo, a postdoc in the Uchida lab, has been honored with the Young Investigator Award from the Japan Neuroscience Society. The yearly award recognizes five early career researchers making major contributions to the neuroscientific field.
Amo was selected for this award for his work on dopamine’s role during reward-based learning, and its relationship to a machine learning theory known as temporal difference learning. His work is giving us a better understanding of how we learn to form expectations based on experience and how that affects our decision making.
“We in the Uchida lab are interested in how we make decisions and how we learn from the environment,” says Amo. “My interest is what kind of algorithm and neurocircuitry is used to produce what is known as prediction error signal in dopamine neurons.”
Amo studies this through the process of learning the association between cues and rewards—something commonly referred to as a Pavlovian conditioning, named for the classic experiment that showed dogs beginning to salivate at the sound of a bell which they had learned indicated a treat was coming.
His work connects this biological experience to the computer science concept called temporal difference (TD) learning. In this machine learning theory–which was first proposed in the 1980s–a program experiences something akin to Pavlovian conditioning; it uses the differences in expectation of reward versus actual receipt of a reward to assess the value expectation of a given moment. This process allows researchers to find the optimal parameters that would result in more accurate predictions or better end-point decisions.
“The link between temporal difference (TD) learning in computer science and dopamine activity in neuroscience has been proposed based on dopamine activity after the completion of cue-reward association learning,” says Amo. “The temporal difference learning model predicts a gradual temporal shift of prediction error signal (which is used as a teaching signal in TD learning). However, this had not been observed in animal or dopamine neurons. There have also been debates to deny that temporal difference learning accounts for dopamine neuron activity by proposing alternative models which often lack temporal shift of the error signal.”
Amo’s work has shown that during Pavlovian conditioning, the timing of dopamine activation gradually shifts from the time of receiving the reward to the time of receiving the cue that predicts the reward (a bell chime, in the case of Pavlov’s dogs). Eventually dopamine activation happens most often at the cue onset.
This affects how subjects learn about expectation and make decisions about their next actions, to maximize the likelihood of reward. The experience is key in learning the value of environmental information, which is critical for future decisions.
And the whole biological process mirrors the same phenomenon seen in TD learning, a theory in machine learning.
“This is a breakthrough link to connect between those theories and the actual brain,” says Amo. “We found this dopamine response shows exactly same type of activity pattern as the surprise signal predicted by TD learning theory.”
Amo has spent the last seven years working in the Uchida lab on dopamine signaling. Before that, he completed his PhD at the RIKEN Institute in Japan, where he began studying developmental biology in zebrafish before becoming interested in the nervous system. There, he found the neurocircuitry of dopamine reactions to be a strong model for studying decision making, and has been focusing on that area since.
His interest in nature extends back into his childhood in Japan. He loved insects and bugs, and collected them from the area outside of his apartment building.
“Next to my room, there was also another boy who was the same year as me. We always were catching insects together and we put a lot of cages of insects in front of our rooms. So I always have had interest in animals and science,” he says. “And that friend is now a researcher of cancer.”
Now, in Harvard, he spends his time outside the lab with his family, exploring Cambridge and Harvard. His young son is a particular fan of Harvard’s Natural History Museum, and has also been a frequent visitor to his father’s lab.
Amo’s postdoc is coming to an end, and he is about to go on the job market. He looks forward to continuing looking into the questions of dopamine actions in his next position.
“Ryu is an joyful, invaluable member of our research team,” says Mitsuko Watabe-Uchida, Research Fellow in the Uchida lab and one of Amo’s supervisors. “Equipped with a strong technical background, he continues to challenge himself to address unsolved questions, and has shown tremendous growth over years. There is no doubt that he will be a wonderful scientist and PI in the future.”
