Publications

At the beginning of summer 2023, a wave of research works applied Large Language Models to sequential decision-making. This caused both excitement and confusion in me, about which I wrote about in a blog post that was pivotal for me. Seriously, I wanted to know whether there was something really interesting behind the hype. When Martin joined Meta as an intern, during many days of intense brainstorming, we enumerated the possible ways to use LLMs for decision-making. I got pretty convinced that extracting a reward function from them was the most promising of all. In that particularly long, rainy and confusing summer in Montreal, we pushed ourselves out of our comfort zone and witnessed the potential of LLMs for creating AI agents.

Zhixuan was initially curious to explore the interaction between the resetting mechanisms we have been leveraging in our previous work and ensembling methods for deep reinforcement learning. In the end, in a piece of work on the empirical science of neural networks for reinforcement learning, we discovered surprising phenomena about the interaction among ensembles, data collection and representation learning.

Motivated by new discoveries about the empirical science of deep reinforcement learning, we started discussing techniques for discovering other phenomena and advancing our understanding of neural network-based agents. After some attempts and after building an appropriate experimental framework, we came to the conclusion that the lens of the return landscape was a good one for our goal. Funnily enough, we found some new interesting phenomena right when we stopped searching for them. I learned a lot about how to do understanding-oriented science.

As the paradigm of increasing performance by scaling the amount of computation was being established in the rest of the machine learning community, we were looking for a way to generalize this to reinforcement learning. Guided by some preliminary evidence we had shown in the primacy bias paper, we thought a way to do it was to increase the amount of updates per environment interaction. I had fun doing some research in which the main goal was not to develop a totally new method or to show good performance really, but to go deep with analyses and to try to empirically understand the implications of different aspects of a complex system.

Evgenii had shared some interesting results on how resetting parameters in neural networks was sometimes giving unexpected performance gains. In a fun scientific sprint, we tried to understand how general the improvements provided by resets were and where they were coming from. Through this project, I understood the huge power of deeply collaborative research and of intuition-guided empirical science.

I moved to Milan in March 2020, one week before the very first Covid lockdown started. I didn't leave my apartment for several weeks, and helped out as an intern with a theory-oriented project. Enduring the lockdown and the pandemic was a life-changing challenge, for me as for almost everybody else.

While living with a daily commute between the wonderful lake city of Como and Switzerland, I tried a bunch of mostly theoretical ideas during my time at NNAISENSE. We found out at some point with my host Wojciech that the ideas I had been thinking about for my master thesis were actually generalizable to actor-critic methods: it implied a simple theory-backed deep reinforcement learning algorithm that yielded good results out-of-the-box. I have established in that occasion, synthesizing my previous experience with what I learned from Wojciech, the core of what would have been my research taste in subsequent years.

Michele, a flatmate at that time, told me that he was collaborating with a chemist on creating a platform for the analysis of experimental data. I decided to help, with the goal of learning about cross-disciplinary collaborations. We spent with Michele several evenings building software together in the student residence we were living in. We learned a lot and celebrated small successes with cheap grocery store cake slices.

At the beginning of my Master's research work, I really had to learn the hard way how to precisely formalize problems and think in math, since I had realized my drawings of boxes on a whiteboard were not enough anymore to express my scientific self. We had in mind the general goal to learn a model of the dynamics that was tailored to its use in reinforcement learning. We ponderer about what that meant exactly, and got inspired by the ideas of Amir-massoud Farahmand on decision-aware model learning. I spent several months staring at a whiteboard and thinking about math for most of my time.

I wanted to have a first experience with scientific research, to learn what it was and to see whether it was fun for me. Concetto told me they were working on video generation and I was very excited to help them. It's probably the first time I've realized you can actually do science as a job for real, when you put enough effort in it. I brainstormed about research ideas, learned how to draw big boxes on a whiteboard, and trained neural networks for the first time. I guess I liked it enough to decide to continue on that path.