Voice over
Welcome to Beyond Lab Walls, a podcast from the Salk Institute. Join hosts Isabella Davis and Nicole Mlynaryk on a journey behind the scenes of the renowned research institute in San Diego, California. We’re taking you inside the lab to hear the latest discoveries in cutting-edge neuroscience, plant biology, cancer, aging, and more. Explore the fascinating world of science while listening to the stories of the brilliant minds behind it. Here at Salk, we’re unlocking the secrets of life itself and sharing them beyond lab walls.

Nicole
Hi, all. I’m Nicole, and I’m excited to be here today with Salk Fellow Talmo Pereira. He uses artificial intelligence to help biologists understand the brain, plants, disease, and more. So, I’d like to start out the podcast just kind of getting to know you a little bit more and where you’re from. So where are you originally from, Talmo?

Talmo
Yeah, so, I was born and raised in Brazil. We grew up on the rural outskirts of a town called Campinas, and I moved to the U.S. when I was 16. But prior to that, my mom immigrated to the U.S. in hopes of finding work and supporting our family back home, during which time I was raised by my grandma and my paraplegic uncle until we had enough money and the opportunity to move to the U.S. explicitly to pursue education as a way to escape the socio-economic immobility that we found ourselves in Brazil.

Nicole
Wow. That’s a long and very formative time in your life to not have your mother physically around you. Was that difficult for you? I mean, how did you get through that period?

Talmo
Yeah, it was challenging. One of the ways that I ended up coping during that time period, especially in middle school, was by not being very popular and spending a lot of time on the internet and on my computer, largely playing video games. But then, in order to get an unfair advantage in my video games, I started learning how to code, and developed little scripts and bots to automate my gameplay, which I found more interesting at some point than playing the games itself. That later on really paid off. It turned out that those skills are actually marketable.

Nicole
Yeah, absolutely. Were other kids doing that in your town or was this just something that you took up on your own?

Talmo
No, it was really a weird confluence of factors. The game is called Tibia. It’s got all sorts of skeletal patterns inside the game. It’s a role-playing game where you just have to, you know, kill baddies and level up and gain gold and get stronger. But it was especially popular in Brazil, Sweden, and Poland for some reason. So, I ended up having a lot of Swedish and Polish friends in middle school, which no one thought was weird, I guess, at the time. But some of those were really into the coding scene and so a couple of them kind of got me into this sort of coding/video-game-hacking scene where I started to just pick up from tutorials, all the basics and just iterated through MSN Messenger and, you know, old school bulletin boards.

Nicole
And so, while you were growing up and developing these coding skills, what was your mom doing in the U.S.?

Talmo
During the separation, she was working, you know, three to four jobs, constantly cleaning, delivering newspapers, taking care of the elderly, babysitting. So, I was very motivated when we moved here to really make it all worthwhile by giving it my all towards my education. And so, you know, in our view, like the version of the American dream where an immigrant comes and is successful is to become a doctor.

I was really fortunate to end up at the University of Maryland, Baltimore County, as a Meyerhoff Scholar. The Meyerhoff Scholars Program is one of the most successful programs in the nation for placing underrepresented minorities in advanced degree programs in STEM.

I started off as a biochemistry major. Again, I’m going to do the whole MD thing, but I was convinced that then I wanted to do research. So, I decided I was going to do an MD/PhD and started pursuing research opportunities when I was in undergrad. I did one at the Broad Institute of Harvard right after my freshman year and learned very conclusively that my strengths definitely do not lie in bench work. I must have spent multiple-fold what my stipend was just in and antibodies and other molecular biology reagents simply because I would always mess up one little step of the procedure or I would forget which well was the control versus the experimental sample and so forth.

But the following summer is when things kind of came full circle with all of my coding experience where I decided to try out computational neuroscience. So, then the next summer I went I went back to MIT to work with Sebastian Sun, who was essentially trying to predict what the connectivity of the brain is by slicing it up and imaging it. And that just involved a lot of coding, a lot of technical challenges associated with reconstructing what the brain looked like.

And I totally fell in love with it, because that was really my first opportunity to really properly apply all those skills that I learned way back when to a serious scientific research problem. And I realized that that’s definitely where my strengths lie. Certainly not on the bench, though. But I also found that I really enjoyed it. So, I kept kind of exploring that throughout the rest of my research experiences, both back at the University of Maryland and the following summer when I was at Caltech working with David Anderson on quantifying animal behavior.

That ended up being a super formative experience because that one summer entirely shaped the trajectory of my career. I set up the research questions. It allowed me to learn what the general methods were, but it also really gave me something even more valuable, which was a vision.

We had a very concrete endpoint, like we’re looking at all these animals, we want to understand how their brain works in the context of how they’re moving. If you have pets you probably have experienced this, but you just see all these little patterns in their body language and over time it becomes very clear and apparent that they’re not just reacting. They’re definitely following very complex programs that must be controlled by their brain.

I was looking around and all the top scientists around me then, and in large proportion still today, were analyzing behavior by essentially doing what people were doing hundreds of years before, which is sitting, watching, and just mainly writing down when and for how long the animal does X, Y, or Z.

And surely there must be a better way, right? That ended up being the motivating question, both of my graduate school career, as well as one of the core missions of my lab here at Salk.

Nicole
I have to ask, did you ever consider becoming a software engineer or going into computer science just given that you’d been developing all these skills and it’s an industry that can be highly lucrative.

Talmo
Yes. Yeah, I very much considered industry, especially having experience with both software engineering and a form of artificial intelligence called deep learning. And I did kind of explore that route a little bit.

In graduate school I took a few months off to go intern at Google AI and got the experience of working as a full-time research scientist at a top industry research lab, doing essentially the same kind of computer science, at least, work that I was doing before. The applications weren’t the same. We weren’t trying to solve any biological questions, but the computer science and the math really were basically indistinguishable.

I did have to make a decision at some point when I was finishing my PhD as to whether I was going to go back to industry and follow up on that experience or go down the academic track, which is where I am now.

And it was a really hard decision, right? Obviously, we don’t as a family have a lot of money, so it’s always been a big priority for me to be able to support my mom and my family. And I definitely had to have an earnest conversation with my mom.

But I’ll be honest, I think, Salk was not only very appealing, but they kind of caught me early. Before I was able to finish my PhD, my plan was really to apply widely to both the industry and academic positions such as this. Kay Tye ended up recruiting me to apply to this position. It was basically at the top of my list in terms of where I would want to go if I went down the academic route. And so, when I got the offer to come to Salk and saw how open-minded they were about providing the kinds of resources that computer scientists normally don’t get in academic settings—namely providing large investments in computing infrastructure, which typically you only find in industry labs—I was like, well, I guess I’ll have enough to do my research and to do it at a top level. So, it’s really just about the money. And ultimately we made a decision that, you know, it was enough money.

Nicole
It seems like that Meyerhoff Scholars program was very important in establishing that foundation for your future studies and in eventually ending up at Salk. So, can you tell us a little bit more about that experience?

Talmo
Yeah, yeah, definitely. So, the Meyerhoff Scholars Program was created by Freeman Hrabowski, who was a civil rights activist, mathematician, and then president of the university.

I was really, really fortunate that when I was at the end of applying to colleges, my high school counselor basically was like, “You need to do this and I filled out a nomination form for you to do it, so you’re doing it.” It was like, okay, I’ve never heard of this thing, but I’m really afraid to say no to her.

It was really important that we do these summer programs. Being able to go to those places really set me up because I was able to get letters for graduate school from those places. I was working with the top labs and had the opportunity to do the top-level research.

Nicole
Yeah, that’s so great. And I guess I’m wondering, have you felt like you’ve been able to pay that experience forward at all at Salk in being able to mentor other students?

Talmo
Yeah, one of the things that I’m really excited about this year, and this is really owing to the Office of Diversity Equity Inclusion, is that we’re about to have our first undergraduate research fellowship summer program, and I’ll get to be for the first time a host PI for one of those summer programs. I’m very excited about that opportunity because it’s really the thing that shaped my career and I’m very eager to be able to give back now.

Nicole
While I’m sure that’ll be an amazing experience for the trainees that get to work with you.

Voice over
If you’re enjoying this episode of Beyond Lab Walls, be sure to check out our other channels at Salk.edu. There you can join our new exclusive media channel, Salk Streaming, where you will find interviews with our scientists, videos on our recent studies and public lectures by our world-renowned professors. You can also explore our award-winning magazine, Inside Salk, and join our monthly newsletter to stay up to date on the world within these walls.

Nicole
Now, you’re a Salk Fellow here at the Institute. What exactly does it mean to be a Salk Fellow?

Talmo
So fellows are a really—it’s a very unconventional type of position. There’s only really a handful that exist in the U.S. It’s a little bit different than the traditional academic trajectory, where normally you do your PhD, then you do your post doc, then you apply for a tenure-track assistant professor positions. What these fellows programs allow you to do is to basically skip the postdoc step and kind of get you straight into something in between where you’ll have functional independence. And what that means is that you get space. You get money for a startup. You get the ability to hire people. But the advantages really are, for this kind of structure of a position, is that you really get to start your independent career straight out of grad school.

One of the key achievements that really made this thing work was the fact that I knew what I wanted to do before even entering grad school, and then I basically executed that and came out with, essentially, a tool that was the culmination of my PhD work and that was quite successful and could support an entire research program around it.

Nicole
Well, you mentioned the software that you developed in graduate school. Can you tell us a little bit about that?

Talmo
Yeah. So for my PhD, I went to Princeton, where I got my doctorate in neuroscience. My core objective was really just to quantify the entire repertoire of fruit fly courtship. Essentially, I wanted to map out the entire Drosophila (vinegar fly) courtship ritual. All the little body language, all their little steps, and all their dance and their song.

That took me on an extended detour into computer science, and specifically a field of artificial intelligence called deep learning, where at the time it was just starting to be shown that it could be used to solve lots of problems in computer vision and essentially, you know, be able to extract information from images and videos. And that included the task of pose tracking, also known as motion capture.

Picture a motion capture suit, just like you’ve seen maybe on behind-the-scenes things for Hollywood. They have all these little dots and all these balls, essentially, that actors have to wear so that they can capture their motion to then animate, you know, characters like Shrek or the characters in Avatar and so forth.

I realized in seeing one, of those, like, wait—we could use this for the animals. We can’t force them to wear a suit. I cannot put a motion capture on my fruit flies. But this whole deep learning thing had just shown that you could do that without having any markers, without putting any dots on them. All it required, really, is getting the algorithm to work.

And so I basically adapted the approach that had worked for the human pose tracking problem and made a few modifications to have it work well in lab animals and proceeded to demonstrate that this would work in a very general sense across basically any species.

Nicole
Yeah, that’s a great idea. And just as you’re explaining it, it almost seems somewhat obvious, like, why hasn’t this been done before?

Talmo
Yeah, that’s a great question. I feel like all the most awesome ideas kind of, you know, have that quality to them, right? Where it’s obvious and it’s surprising no one’s done it before. There’s a confluence of factors that, you know, happen to land me in just the right spot to be able to not only recognize that this was a technology that was going to be possible and applicable here, but also have the opportunities to see it through. To actually execute on it.

Nicole
Yeah. And the skill to do it too.

Talmo
Right, and the background, the technical background to be able to actually implement it. It happened really fast, I think, for those exact reasons. We realized that, yeah, this is a really good idea and it’s surprising that no one’s done it yet. But it really was just the confluence of recognizing that this is a problem and realizing that there was a technology which we could base a solution around.

Nicole
Yeah, that’s amazing. And so how have you brought that into your current lab or perhaps adapted it for any other uses?

Talmo
One of the next things we wanted to do was to extend the algorithm to work with multiple animals, because the original version was really kind of constrained to work on a single-animal-basis. And the multi-animal case actually turned out to be a much, much larger problem. It wasn’t as simple as just kind of extending it to handle, you know, two instead of one.

It was actually a very large knowledge gap because we now need to have our artificial intelligence, our AI, reason not only about where body parts are in an image, but also how to connect them and track them over time. It essentially has to learn what is a body or what is a limb in a much more high-level sense. That’s just a harder problem.

We solved that and we also published that as part of a method called SLEAP. People have just picked it up and use it for all sorts of things, some of which have now bloomed into collaborations now that I have my own lab at Salk.

Just highlighting a few—one of the ones that we thought was really surprising was that some folks have picked it up and started using it to track plant motion, both in timelapse and in other forms of imaging. And yeah, I mean, why not really? All forms of life are able to move at different timescales. And the way we designed it not only makes it easy to use, but easy to try out with new kinds of data.

So, we now have a whole collaboration ongoing in my lab with Wolfgang Busch’s lab and the Harnessing Plant Initiative with the goal of phenotyping plant root systems using SLEAP. The idea is to essentially track the location of all the root tips and all the branch points so that we can pull out things like, how often are they branching or how deep and how massive are the roots, in order to support other efforts to then associate those qualities with particular genetic programs that then can be used to create seeds that improve carbon sequestration and curb climate change. And I don’t know anything about plant biology. I’ve been learning a lot, but I’m super happy to be involved in this and to be able to make a contribution to this field.

And then another one that kind of extends beyond my original neuroscience aims is cancer biology, where we’re now just starting a collaboration with Dani Engle and Christian Metallo to essentially extract body language markers of the progression of pancreatic cancer.

We believe that because pancreatic cancer affects the metabolism and can then subsequently change how the animal chooses to eat, the structure of their feeding, in probably very specific ways, with a more sensitive method like this where we can track them 24/7, our hope is that we’ll be able to a) correlate the changes in their body language to changes in their metabolism, and b) hopefully be able to predict whether they’re going to eventually develop cancer much earlier on, using only video data from which we extract their body posture and their body language. If they’re eating less or at shorter intervals or not eating at all, these are the kinds of things that we’ll be able to measure, as well as all their other behaviors, like whether they’re in pain or exhibiting antisocial behaviors, and other markers that we don’t have an explicit test for but that are obvious predictors because of the systemic nature of a disease like cancer.

Nicole
That’s so fascinating.

Now, remind me, how are you using these techniques to study how humans experience art? Because I know you had a recent collaboration where you were looking at that.

Talmo
We’re working with the Los Angeles County Museum of Art (LACMA), as well as Tom Albright and Sergei Gepshtein here at Salk, on a collaboration to quantify and model a museum-exhibit-goers’ behavior as they traverse a particular exhibit at the LACMA. And so we set up a bunch of cameras. These are a bunch of security cameras all along the ceiling of this exhibit, essentially tracking the movements of all the people that kind of go through, walk through that exhibit.

And by having a digital reconstruction of the exhibit, we can now infer things like where people are looking, and what they’re seeing, and what kinds of visual properties about the exhibit—both in terms of the spatial configuration, the layout of the exhibit, as well as the arrangement of the pieces, the color schemes, and even subtle manipulations like the lighting or other properties—How does that affect how they engage with the exhibit? Do they spend more time in particular places, does that change their trajectory, and can we predict that based on our reconstruction of both their behavior and their experience?

Nicole
This is so cool. It sounds like you have so many interesting projects spanning all these different areas of science. It makes me wonder, just out of everything that you’re doing as a scientist, what would you say is your favorite part?

Talmo
Definitely one of my favorites is the ability to mentor and support my students and trainees. I find that super rewarding.

Scientifically, one of the things I’m really excited about is the fact that these AI methods are really kind of—they’re not slowing down, right? You can see from these large language models like ChatGPT and so forth, that this type of approach, this technology, is only going to get bigger and more capable. And so I’m very excited about basically infusing that in all areas of the lab, including one project that I’m particularly excited about.

Our goal is, now that we’ve been able to capture behavior, to now be able to simulate it. Our objective as part of this project is essentially to create digital avatars of our animals, like 3D-rendered mice, as if they were in video games, and train them using AI to basically play this video game where they get more points for imitating the body language of their real animals, and now use it as an anchor point to study the brain by now, essentially, constraining the AI to have the same kind of structure and connections that we think the real brain does. So now we’re creating a virtual brain that controls a virtual animal and must move and behave like real animals do. That allows us to then generate predictions about what the brain is doing that we can go and test and use, if it’s wrong, to further constrain the model and allow it to make increasingly more accurate predictions.

I’m really excited about this framework because it’s really not a project. It’s a new research direction, it’s a new way to do neuroscience. And I think that this general formula is going to really permeate across all fields of science because you can do this with any type of biological system using an AI to model it explicitly. I think that one’s going to be a real game changer, and I’m really excited to see what kind of discoveries we can make using that type of approach.

But more broadly, I’m really just excited to be able to contribute to all the amazing world-class science that’s going on here at Salk and to see the technology that we develop be used for all of these super cool applications.

Nicole
Yeah, that’s so cool. I did want to ask a follow up question. So, with this new technology that you’re working on, I mean, theoretically years and years down the line, would this help scientists study different systems in the body, like possibly without even using an animal model and instead just using this technology? I assume that, you know, once that’s learning, you can speed it up and you could test hypotheses really quickly.

Talmo
I think that’s the real promise of this thing. But I think you absolutely nailed it. It is exactly that. It’s the ability to generate testable hypotheses and then iterate on it by being able to fold experimental results back into it to continuously improve it.

Nicole
Well, we’re definitely looking forward to seeing what comes next.

In the meantime, I just want to thank you so much for coming on the podcast today. It was a pleasure speaking with you and learning about your path to Salk and everything you’ve accomplished so far and everything you have in the works.

Talmo
Yeah. Thank you so much for having me. I’m super excited about this.

Voice over
Beyond Lab Walls is a production of the Salk Office of Communications. To hear the latest science stories coming out of Salk, subscribe to our podcast and visit Salk.edu to join our new exclusive media channel, Salk Streaming. There, you’ll find interviews with our scientists, videos on our recent studies, and public lectures by our world-renowned professors. You can also explore our award-winning magazine, Inside Salk, and join our monthly newsletter to stay up to date on the world within these walls.