Skip to content

John Lam's Blog

2022-05-06

I've started a podcast!🔗

A couple of decades too late, I've decided to start a podcast. I've had the good fortune to have met many interesting people in my life, and I wish I had written down or recorded more of what I learned from them. In the spirit the best time to start a podcast was 20 years ago and the next best time is right now, I'm going to try and make up for lost time.

For those arriving in the future, here's a link to the main page of my podcast. I need to find a place for it on this blog as well.

I started my experiment by interviewing people on my team. Here's my conversation with Austin Henley, former Assistant Professor at the University of Tennessee and the newest member of the PROSE (Program Synthesis) team at Microsoft.

Summary🔗

In this conversation, Austin and I discuss such topics as:

  • His recently discovered love of LEGO as an adult
  • How bouldering helps him strengthen his problem solving skills
  • The differences and similarities between working as a professor and as a Program Manager at Microsoft
  • The effectiveness of using a 'flipped classroom' teaching technique
  • How Visual Studio and Visual Basic 6 got him started in programming
  • How to use feedback loops in recommendation systems to generate superior results
  • What we can do to improve tools for software developers by better understanding what their intentions are

Transcript🔗

Click here for an interactive transcript.

John: Today on the podcast, we're going to be talking with Austin Henley, the newest member of our team. He just recently joined us a little over a month ago from the university of Tennessee, where he was a professor of computer science. At Microsoft, Austin's going to be working on the PROSE team, our program synthesis team.

And in this, conversation, we're going to be learning a bit more about Austin's background, what got him to come to Microsoft and some of the things that he finds particularly interesting, about his past life as a professor. So you got recently to number one on hacker news, with this post: what a $500,000 grant proposal looks like.

And that was for an NSF career grant. It was also this thing that kind of made it pretty likely that you were going to go get tenure as a professor, but yet you gave all of that up to come work here at Microsoft. Why did you do that?

Austin: Well, that's the $500,000 question. So, you know, there's some history here.

I had a couple of these NSF grants, the national science foundation it's who funds most of the basic research. And you're right. It was looking good for me to get tenure my faculty position. But for the last few years, there's just been this bug in my head about, not that this isn't actually my goal.

Even though it was what I set out to do, there was more to it. I'd already had a relationship with Microsoft before: I interned with them. I spent a summer, visiting, as a professor. And so it morphed over time that I wasn't happy doing what I was doing, this race to get tenure.

And so an opportunity arose. And I couldn't say no. I mean, it goes back to when I got into programming, like in fifth, sixth grade that I wanted to work at Microsoft. And so I couldn't pass up the opportunity. Right.

John: So you're always like the Microsoft fanboy, growing up

Austin: a little bit.

So I started with visual basic six. I used to have to pirate visual studio, 6.0, back in the day, this was probably around 2001. And every time my dad would find it on our home computer, he would delete it. Cause he didn't know what it was and it would take up, I don't know, 12MB. Yeah. It was huge.

And we had this tiny hard drive and so he would delete it. And so every couple of weeks I'd have to re-download it. So I got started and was a bit of a fan boy and that just kind of grew through college and grad school.

John: When you were applying for this grant and what kind of led you to come to Microsoft where they roughly the same things where they do different things, like, could you describe a little bit more about what that was?

Austin: Sure. So the research that I was doing at UT is all about developer tools. So they use like I'm on the human factors side. So I would observe developers, I'd put them in a lab. I would, watch them code, try to understand what their problems are, what their workflow is. And so the grant was actually about how can we build tools to help, not just students with learning, in coding task.

So, maybe they were trying to understand how for loops work. But how can we also apply that to professional developers? So there's a lot of learning that happens while you're actively coding. And so that's what my grant was about: can we help learning process through self explanation?

Kind of like rubber duck debugging. So Microsoft, in the PROSE team, they had a project that was aligned with this. They're working on an intelligent tutoring system, for like intro to CS and intro to data science students. And so it's a very much overlapped with what the team was doing.

John: So now that you've transitioned out of the academic life and into whatever corporate professional life that we have here, what's the biggest difference that you've seen so far, between these two worlds and also what's similar between these two worlds.

Austin: Yeah, there's actually a lot of similarities that I was kinda surprised about. The ability to just figure out what needs to be done, what needs to be done next, and then present that on the fly to diff very different audiences is something that I had to do a lot in academia. And now it's something that I do every day and different different meetings with different partner teams and management just trying to paint the picture of what we're doing and where we plan to go.

That's just constant. Something that's very different is just the pace. So the job is a lot more similar than I thought, but the pace is so radically. Whereas in, in my previous position, if I wanted to work on something, you kind of go off and you think about it for a couple of weeks maybe work with your graduate students to get a prototype over the next couple months and brainstorm and sketch out what's the design space for the problem.

And what's the solution space look like, do some related work and just kind of explore. But we do this. Daytime frame in the team or within a few days we sketch out these plans and figure out what to do. So the pace is the biggest one. What

John: do you think is like the thing that's most responsible for this difference in pace, like I think you, and I've spent some time talking about Tim Urban the wait, but why thing? And he likes talking about procrastination as a thing that he does a lot. And I think we all see that in ourselves and in his, the funny thing that he likes to bring up for that is for deadline based things this panic monster thing wakes up and and governs your life and drives you to complete it.

Do you feel that, you know, I don't want to try and put the words in your mouth, but do you think that it's kind of the difference between, being your own boss in academia versus now where you have lots of bosses, it seems. Is there more panic, monster driven work going on

Austin: here?

Yeah, for sure. And I'm trying to find the balance there because I think there's so much benefit to just having uninterrupted deep thought on a problem that you're working on. And so how to fit that in to my current schedule, which is just ridden with, deadlines and other people, depending on me.

And I, of course don't want to block other people, you know? So how can I work in those sessions of just, how can I get two full days to think about something without interruption while also not dropping the ball on all these other projects? Another thing that I think makes this different is that it's so much more collaborative of a workspace.

And academia, I had my, like a group of six students. I'd worked with them a couple of times a week, but for the most part, we're a bit isolated in how we work. But at Microsoft, I'm in meetings every day with different teams and each team, has I, I'm making promises about, I can help with this, or there's some expectation that they'll get me something.

And so that puts a lot of pressure to basic, basically have deadlines every week at the very least for showing up with deliverables.

John: Did you find that you collaborative much with other colleagues in academia or was it perhaps more competitive? In, in that space? Like how was that?

Austin: So I. Early on. I took a very personal stance that I wasn't going to be competitive with my research. And so I would put everything out there. I didn't treat it as a competition for resources, even though resources were very limited. I would put my ideas out there on my blog and on Twitter. And I did a little bit of collaborating, but again the pace and rhythm of those collaborations at the very most is once a week and more likely every other week.

And so the expectation, which is, was just not there to have like concrete deliverables constantly. But yeah, other people do take that to competitive routes. But I stayed at, stayed away from it. And it, I think it works out better to just be open and collaborative.

John: Right. Cause I think there's a lot of. A lot that we can certainly learn from from being kinder and gentler to each other. If that's the motivation, right? Other times certainly you've heard plenty of horror stories in graduate schools about, supervisors pitting students against each other on the same project and things like that as well.

How did you decide to become a professor in the first place and what kind of led you down that route?

Austin: Yeah at some point I, so I got into programming pretty, pretty early just by accident and fell in love with just coding and really what got me into it was like compilers and programming languages and.

I remember telling people at high school that I just wanted to be an expert in that space. I just want heads down, just know everything I can about coding. I'm like, I want to be the expert that people go to. And so when you start looking into what kind of jobs that can be academia came up a lot, having getting a PhD in a topic means that you were given some years of freedom to just really drill into a topic.

And so I didn't know anyone that had gone to grad school. And so I actually, I applied and was a master's student at first and my PhD advisor is probably what really sealed the deal for me, just working with him was a great experience. And I learned a lot from him. And that made me stick around for the PhD.

But there was still a question of, should I just go do industry research? Should I go be a software engineer? Should I go be a PM? Or should I go the faculty route? And I was in grad school for something like seven years. So I had plenty of time to think about this and bounced all around and did lots of internships.

I did, I think five internships. And so I was actually leaning towards the industry route. But at the very last minute, in my final year of graduate school, my mentors found out that I wasn't applying to faculty positions because I, and what I had really said was, I don't think I have a chance.

It's very competitive, very finite resources number of positions. And the interview process is not like industry positions. It really only happens once a year. So around October, November, everyone applies to the same person. University might hire one person each year, if that and it's, it takes months to, do a phone call.

Then you go out and do a full one or two day interview where people just drill you or grill you about your research. And you give a job talk as it's called in front of the entire department and they ask you hard questions. And some people like to be mean about it and try to trip you up in front of everybody and show you that they're smarter.

Yup. Yeah. They're smarter than you, right. They know your topic better. And so I didn't want to do that. That just sounds very stressful. But some great mentors of mine. They said, everyone's already expecting you to do this Austin. You've done great work. You can be successful if you try.

So after that I did some like soul seeking and decided to go that path. And it took, I think, three weeks of full-time effort for me just to do the applications to get my resume ready. Your statement, your research statement, teaching statement, all of the materials letters of reference. It took me about three weeks and I applied and it turns out I, I got a surprising number of interviews and the first interview was so stressful.

It really was stressful, but after that one, they were also easy. I like, I think my body just had so much stress that it gave up and stopped fighting it. And I was just like having fun, enjoying flying around the country, giving these presentations. And they went really well. And I got some good offers that I couldn't refuse.

So one of them was at the University of Tennessee. And I actually had a plane ticket to fly out to Microsoft for a software engineering position interview. And that was the hard one to say no to because I had accepted the faculty position

John: Did you know, where at Microsoft, that engineering position was for?

Austin: Yes. It would have been working on the Python team for VS Code.

Yep. I also had another the team that I interned with also invited me out. I believe it was back then. It was the Tools for Software Engineering team TSC. I think they've rebranded to one Engineering System now. Yeah.

John: So then you wound up in Tennessee and how long did you stay at Tennessee?

Austin: Three and a half years, but I grew up in the state of Tennessee and I went to graduate school in the state of Tennessee. So there's a lot of, there's a lot of ties here.

John: There's a lot of Tennessee in you.

Austin: Yeah. Too much. Probably.

John: So then when you became a professor what was like the folks who read, like how did you pick or choose what you worked on and was that told to you?

Was that something that you had to figure out for yourself and how did you decide on those topics?

Austin: Nobody trains you how to be a professor, really. Graduate school is supposed to train you to be an independent researcher, but there's a lot of other parts of the job, right.

That, that have nothing to do with research, such as teaching, administration, service, mentorship. When I showed up, I had no idea what to do. I was shown an empty office and it was a very nice office and they gave me the keys and that was it. There was no starting manual. No one tells you what to work on.

No one tells you how to spend your time. And that was, it was probably what I spent my first semester on is just learning where to put my time. And then I evolved it every semester after that. And yeah your point about, where do I, what research problems do I work on? It's really hard too, because some people continue their dissertation research, but there's also some expectancy that you're going to grow beyond.

And be more independent. So breaking away from your prior work and your research advisor from graduate school. And so I'll tell you what I did. This was I'm very deadline driven. My first week there was a deadline for a grant, a national science foundation grant that you can only apply for in your first two years.

And so I was like, okay, this is like my one opportunity to get a grant. And like my first two years it's slightly less competitive. And I can't pass up my first opportunity to apply for it because after like your second year, you can't get it anymore. And so I was like, well, I guess I'll just do nothing, but write this grant proposal.

And I actually started it the week before I started the position. But you know what you have to have a topic to write on. And so I. Just turned off my phone for a week or two. I didn't go, I didn't leave my apartment. And I just thought about what is the coolest tool that I can dream up and I'm going to write my grant proposal on that.

That's what I like to do. I like to work on like just weird ideas for developer tools. And that's what I did and I wrote the proposal and it was funded.

John: Oh, fantastic. And then, so you started working on that you attracted some graduate students and you also had to teach there too, right? Was that something that you were just told you're teaching this class? Or how did that work?

Austin: Yeah, so my department was there. They're very reasonable. And so they worked with me to find what classes are good matches, but also the department needs. And so they brought me in knowing that they have a huge need for software engineering. So like introduction to software engineering, junior, senior level standard class that every, almost every university has.

That really just teaches like a a development process. So they already know how to code but how to work in a team for a group project and build an application. And yeah, so that I underestimated greatly how much time preparing for that takes each lecture would take me. I am, because I would track it something like eight to 12 hours per lecture.

And that's me lecturing for about like 40 minutes. And then I would try to weave in like an activity each class so that my students, don't go to sleep. I get tired of hearing myself talk too, so I don't want to put them through the misery. And yeah, so two days a week I did nothing but make lectures.

The great thing is that I use those lectures every fall after that.

John: Yeah, there was this huge like startup cost and then it started paying dividends afterwards. Yeah. So, do you miss the teaching part of it?

Austin: I really enjoyed the teaching. My classes kept getting bigger and bigger. And with COVID everything went online and teaching online I don't miss it all. I do miss teaching, but it's really the small classes. So once you get beyond 30 students, it's, it doesn't feel like it's teaching anymore. It feels like I'm talking and they're eyes staring back at me. So I was constantly struggling with how to make a large classroom of 60 or 120 students more interactive and I never succeeded there.

But I do, I love teaching the smaller classes and being able to interact with the students. No.

John: Did you take inspiration from the big, like performance art classes, like CS 50 and things like that? Or did you try that? Did you try pulling that off?

Austin: I would try to find different pieces that I, because like I tried doing a flipped classroom for instance, where me and the TAs it's a flipped classroom is where you do the homework in the classroom and they go off on their own and watch the lecture or read like some reading assignment outside of the class, because, and there's some benefit there where they're getting the one-on-one interaction from the instructor or one dominion erection on what's actually hard for them, which is Hey I'm trying to code this.

I can't figure out why I'm getting this error instead of sending them home to do that. And then I can't help them figure out that Python bug that they have. So I tried things like flipped classrooms.

John: How did that work? Was that effective? Cause I, only know this kind of academically when I saw Sal Khan of Khan academy fame talk about this in some thing that he did many years ago with I think it was public schools in the valley where he was living at the time tried this, they would watch Khan academy videos and that was the homework and then the schoolwork or the homework was done inside of the classroom.

Was that effective? Did that work well?

Austin: I think it really depends on the classrooms set up and I didn't have experience doing it so I may not have done it very well. So something that, the immediate thing that I noticed is that for some students it works great. They have no problem asking for help.

They have no problem asking for help in front of other students. There is a large population of the students that I would have my classes that maybe they found it like stressful to have to perform in class. And they see their peers around them like finishing the first step of the project, but they're still stuck on it and they don't want to ask for help because it calls attention to them that there.

So I've read about some ways to try to get around that. I think you can break it up into like small groups and get them a bit scattered in, in the class. But that's what I noticed that a lot of students didn't want to put themselves out there. They didn't want to engage so much.

So what I started doing was just small pieces. Instead of doing your entire homework and class, let's get you started. Let's get you first pass that first barrier. And I would do live coding. So I think with me doing live coding and I'm not very great at live coding, so I would make lots of mistakes and they would see that, and that would help them at least get past the initial barriers.

John: It's amazing like how much how effective that can be, when, because most of the time you see the finished product, right? You see the commit in the repo. You don't see all this work that led to the commit. And one thing like, like Jeremy Howard of fast AI fame, he actually livestreamed a lot of his creation of the fast AI library.

And he does this crazy programming style where he writes all the code and Jupyter notebooks and. But that was actually quite good for his class. So his class was turn, could tune in to watch him code up the library that they were using. And he got a lot of very helpful feedback from seeing that as well.

And I always wondered when I see on YouTube there's certainly a set of folks that live stream coding and it is cool at the same time where you can see some of these things and you just see how someone that's really experienced goes about it. And you'll see that, oh my God, they make the same mistakes I make.

At the same time they're Googling for things and that's okay. And things like that. So let's go back a little bit to Visual Studio and and being a child in that kind of childlike wonder of computers what was there like a problem that kind of attracted you to this thing?

Or was there something that you wanted to create that kind of drove you when you when you were younger?

Austin: Yes. Yes, there were, there was something that drove me towards it. So I think it was in a fifth grade, maybe the summer between fifth and sixth grade, I discovered how to use the internet basically. And we had a home computer with Windows 98 on it.

And I was into video games and anime as many, young kids are. And I found these communities online of people that were probably in high school making these fan webpages for their favorite video game or their favorite TV show and with they had forums. And I S so I started engaging in these forums and realized that these people making these websites weren't that much older than me and they weren't high quality websites at all.

Like they. It was before blogging was really a thing. They would just put their ideas out there. They would put tutorials out there and fan art and just discussion boards. And I, as every kid, when you see something you like, you want to try to emulate it or create it yourself. And so I was like, okay, if another kid can make a website I'm going to learn how to make a website.

And that's how I got started with, I think it was like geo cities for hosting and trying to learn HTML. And I actually had a hard time learning, even HTML. I thought it was so cumbersome to write HTML. And that's where I really got sucked into computers is I wanted to make my own version of HTML my own markup language.

And you can probably see where I'm going next. I found some people that had enough knowledge to tell me, oh, you have to write a compiler to do that. I had no idea what a compiler was. I'd never coded outside of HTML, but I found a book at the library on compilers and I couldn't make no sense of it. I not for many years remember which book

It was the dragon book.

Of course. Yes. Yeah.

I believe the second edition.

John: And how old were you at this time?

Austin: Maybe 12. 12. Okay.

John: Yeah. Yeah. Dragon book of 12. That'd be pretty challenging.

Austin: Yeah. I mean, it's challenging for a college student. Right? But it taught me that I need to learn like a procedural programming language, something.

So I, I first started with basic my dad who I believe had just recently graduated from college, had taking, taken a basic programming course in college and he had the textbooks still and he gave it to me. So I think it was like Q basic or maybe GW basic. I don't remember which. And so I spent maybe a couple of months on that before finding Visual Basic 6

John: Did that change your life? Like what was that like to go from .BAS files? Probably in a text editor and all that, right. To like this visual thing.

Austin: Okay. Yeah. So, it was overwhelming at first for sure. But Visual, Basic 6 I think is what really sucked me in with the form editor.

I still miss that form editor to this day. Because I could just whip up a simple, like something that just gets the job done and isn't a command line. A mystery black box but this like visual application that I can hand off to anyone that's not technical and be like here, this solves your problem.

It was amazing. And then years later I got into C plus, which I really struggled with. And then later C# probably around the time I got into college. But I've always been in that Visual Studio ecosystem.

John: And it seems so sad that, back in the nineties, we certainly call these things, the RAD environments Rapid Application Development environments and we don't really have anything like that today.

It seems like, there's just so much complexity in almost anything that you do. Oh yeah. You can build a UI. Just learn, React, like as if that was just something that you just knock off in an afternoon while it was just some Java script and some CSS is fine. You'll learn that stuff too.

And it just seems like the concept count for doing something. Like putting a button on there and making it do something it's just enormous today with the stacks that we're using, as opposed to Visual Basic, it's a thing. You drop a button on it. There's a thing that looks like a button that you drag and drop onto the form and you double click on it and off you go.

And it seems yeah, we're so far away from that today. And that, worry about that. Like what so now that you're into doing kind of dev tools and making people more productive, what do you feel is the kind of the big problem here? What's the next kind of moonshot thing that we need to try and do in in, in tooling?

Is it making the professional programmers more productive? Is it making the students learn things faster or better? What do you feel? It is.

Austin: Oh, that's another one of these big questions. So something that I'll some of the research that I was doing. And as it related to the grants that I know we talked about earlier is that so much knowledge is stuck in different programmers heads and yet to complete a task, you need that information.

So you need, and I'll give you an example. Let's say you're a programmer working on a project at Microsoft, and you're making some changes that ideally you would need to know the rationale as to why that portion of code was designed the way it was in the first place. And also what are the cascading effects of making changes to that location?

Right. And so you'll do things like you'll look at what code interacts with that code or changing who calls that, that function or interacts with that class upstream and downstream. You'll you might even look at the code history and go back in time and look at what it used to, but there's still so much information that you're not getting out.

And so what that, that last project of mine was getting at is how can we facilitate the knowledge transfer between people in a more efficient way. You can't always just walk over to someone's desk and ask them a question, even though that's a very common thing to do. It's also very common to go back and look at old code reviews, but it's tedious.

And so I wanted to find out how tools can help us not replace it, but help us get that information or archive it and share it. And detect when someone needs it, because you don't always know you don't know what you don't know. And so if you're making a change, you might not realize the design decisions that went into it in the first place until it's too late.

John: How do you think you could solve that problem then? Today with a magic wand, how would you imagine you can marshal whatever resources you need to, to marshal to make this thing happen? What would you need to do? Like or what would you do?

Austin: Yeah, so I was looking at it from three different perspectives.

I was looking at it for students, you know, they're just trying to understand concepts. But we can't look into their head and understand what their current mental model of that concept or of their code is, but we can look for the symptoms. And an example was something that we call code smell detection. If they're writing code, it works.

But maybe it just looks funny. It smells bad. Or it, it often is indicative of a bug then we could potentially intervene. And so that's, this is what my first research project as a professor was, if we can detect that they're writing code that has some code smell, say a loop that will only ever iterate zero times or one times .

That doesn't make sense. It'll compile, it'll run, but no, like you shouldn't be using a loop for that. Instead of showing them a compiler warning or a compiler error, use that as an opportunity to pull information out of their head, ask them a question. And so we built a system that would say, how many times do you think this loop will iterate?

And we didn't want to disrupt them while they're thinking or working. And so we use some prior work on these negotiable interruptions as they're called to try to get them, to give us the answer. And so if they enter an answer, that's something like, I think it'll run 10 to 100 times.

So it was like a multiple choice thing. It was like a click thing.

Yeah, we we played with different ways of multiple choice or like a range, so they could enter two numbers X to Y. So for the loop, it was X to Y and then we could use some static analysis just to see if it's possible. And if it's not possible, then that's an opportunity to help them again, don't just yell at them and tell them what the answer is, but rather, sorry, my dog.

Okay. But yeah, so that's an opportunity to try to help them overcome that misconception. And so that was my very first project at UT and later on for the career proposal, which is the, this big, like before you go up for tenure grant proposal, I wanted to generalize it. So how can we take that same idea of this inquisitive feedback loop, asking someone a question and trying to get that information and help them use it more efficiently.

How can we apply it to professionals? And so I looked at onboarding and code documentation. So if they're writing code that has some smells or is indicative of being complicated or changing a region of code that might correspond with a prior bugs, meaning that there's been a lot of code churn, there's been a lot of code changes over time in that area.

Maybe we should ask them some questions, some structured questions to try to tease out the design rationale and then document it such that it can be shared later for onboarding purposes or code review purposes. So I don't think that's a solution, but that's how I was going about this problem of trying to help people share the knowledge that's stuck away in our heads.

John: And so when that would happen, then what would be the artifact that would capture these insights that they would then in turn like that, that somebody else could glean from?

Austin: Yeah, I didn't get to that. So like the operationalization of what do we do with that information?

What form does it go to? That, that was an open question that I didn't get to. And it's a hard one because you don't just, we already have so much documentation that all these documents these documents that we share within our teams or between teams that after a week becomes outdated and then we never use it again, but it's still floating around in our One Drive. Right. So that's an open question.

John: The good live streamers, that code, they're already doing something like this, right. They're trying to explain or vocalize what they're doing at the time that they're doing to try and make the stream somewhat interesting. Because otherwise, yeah, it could be pretty boring.

You're just seeing so many type stuff on, on, on a screen. And so there might be something there but again, you're almost like relying on the human to vocalize it, but what might be really interesting here is if we captured a time series of them just typing stuff and their interactions, there might be a way to infer things that they're doing by watching the time series of the changes to the code.

Again, right. We're just in crazy, speculation mode here. But it feels like there's it for me.

Austin: Yeah, there's information there. And there's also, so what my grant was, or my the research projects we're trying to build on was this idea that you're getting at, which I believe psychologists called monitoring or self-regulation.

So your kind of just having attention to what you were doing, you're trying to be intentional and think it through it, think through it step-by-step and that process of self explanation has shown to have lots of benefits. So if you can get a student to self, explain what they're working on, like a math formula or programming, they have better outcomes than a student who you don't ask to do that.

And the explanation itself often isn't that useful, but the process of doing it. Gets them better outcomes in some way, it gets complicated. But it's the process of doing that self explanation. And so I was trying to leverage semi-intelligent software tools to help the person do that, but also to help them know when and where to do it.

Because when you're working in code, you can put your attention in all sorts of different places. And so if we can have a system that calls out the high value, low effort regions, where you should put your attention, then it can that was the wind that we were going for. I see. Like, but what you're saying, you were saying that we should use their process to potentially help others as well.

Right, right.

John: Yes, exactly. Yeah. And I, and it also feels Again, back to what kind of latent information is there that we're just not taking advantage of. And this kind of idea about vocalizing verbalizing, or you said self regulating behavior. I remember the story and I've been trying to fact check it online and it doesn't exist.

So take it with appropriate grain of salt. This was verbally told to me by someone else, but they called it the Stanford Teddy bear experiment. And what it was, legend has it that outside of the TA office at Stanford computer science sat in a chair with a Teddy bear. And the idea was, or the rule was that you couldn't go ask a question to a TA until you first asked. the Teddy bear. What they found was the volume of questions that would come in from students. And the quality of the questions would number one increase, but also that they would just go away. They would start talking to Teddy bear and about a minute or two into it, they go, oh, and then they walk away.

That's amazing.

Austin: That's amazing. Yeah. That's amazing. I need to read about that.

John: I'm trying to find this, believe me, I've wanted someone to write this down, but I still haven't been able to find, but someone told me this verbally and I just can't remember who.

Austin: Yeah. I mean, and it makes sense. I had a lot of students that when they would come ask for help, it was obvious that they didn't know how to articulate their question.

And so if you can help them prep just before going into those office hours to help them articulate their question. And oftentimes that just requires. Trying to ask a question. So having that Teddy bear outside I, I can see that being a lot of help and I think it could help now with some of these meetings that I go into I try to do this where I'll write down the questions that I want to ask when I go into the meeting.

And it, a lot of times I realized that I don't know how to ask the question and I, it's much more fuzzy than I thought it was going to be. And so I think that's something that we could all probably practice more.

John: Yeah. I've also noticed that when I've been tempted to go ask an expert about some thing that I find that as I'm writing the email, I never send it right. Because I figure out the answer before I ever have to hit the send button.

And so it's almost the same thing, right? There's this thing. The act of just explaining the problem to someone else, even if the someone else has absolutely nothing like my email client can give the answer. But the other thing nowadays, I think with things like Codex and the thing in copilot our kind of commercial implementation of Codex, the Open AI thing is it generates code based on prompts.

And so perhaps, this idea of just prompting, right? Like explaining to the system, we're trying to describe the thing that you're trying to do to the system and having the system give you code, even if the code isn't quite right, or but it could help just that, that, that feedback loop.

Austin: Yeah. And that's what we're working on now. And in the PROSE team is that feedback loop of what we call programming by example. So you show us some input and output example and we try to generate code and we try to help that feedback loop of trying to solve that problem, whether it be for Excel or visual studio in different domains.

John: Yeah. I think one of the challenges that the PROSE had in the past and that y'all are like, on the path to fixing these days is that PROSE would never in the past and in features like FlashFill for Excel, it would just do the thing for you. It wouldn't tell you how it did it. You would just look at it as huh, it did it. I think it seems right. But you wouldn't know how it did that. So I think that since since the team has put their energy into having it, so that PROSE would synthesize human readable or understandable code, in a programming language, say like Python that all of a sudden it got much better right now you can look at that and go, oh, I see what you're doing.

And or if nothing else, it would give you more confidence in the answer.

Austin: Okay. Yeah. That's been a big research area for any recommendation system is how do you convey some level of meaning of why are you, why are, why is the user seeing what they're seeing? Because there is some amount of trust required for a user to understand what's being recommended to them.

And this could be anything like recommending movies or music. And you could see this on a, like a store it'll say based on what you've bought previously or based on what other people like you have have purchased. We're going to recommend this for you, right? So we're actively doing some studies now with different pros integrations on helping the user understand why they're seeing those suggestions, but also making them good suggestions.

Like you mentioned, people care about readability. So if we generate a formula for somebody, it needs to be readable. It's pretty easy to generate really bad code but making it performance or readable based on their intentions that's a really hard thing and that's what we're working on now.

John: Very cool. Very cool. So again, like maybe just shifting gears a little bit else to, to other things as you're like going through your career academically, could you share, like people that you looked up to or influences in your life that kind of changed or shaped the path that you took to get to where you are today?

Austin: Sure, definitely. So I've had some great mentors and collaborators along the way. So the first one was my PhD advisor. So Scott Fleming at the university of Memphis he introduced me to this idea of human computer interaction. I had no idea what that was. I used to be a compilers nerd, and that's what I cared about was programming language design and how compilers work.

And I had no empathy for users which I shouldn't admit because that's the opposite of what my job is now. I thought that if you have a problem with software it's user error and it's because you don't know enough, you didn't try. Right. You just need to go get smarter. And he, I remember it.

He told me to go read this book, The Design of Everyday Things, and that's still my favorite like book related to work at all, is this design of everyday things. I had all my students read it and it just immediately clicked for me that if I have a problem with software or hardware, it's not me.

Like we shouldn't blame the user. It's a design fail. Why did we not prevent that error in the first place? Why did we not understand our user in the first place so that we could make a better experience for them? So Scott Fleming was my first really big influence in my career. Someone that I really aspired to follow was robbed, align at Microsoft research.

So when I got into grad school, all the best papers were coming out of Microsoft research. And so that's that reinforced this idea of I'm. I need to get some experience at Microsoft. And so people like robbed at line Chris bird Zimmerman Nachi Andy bagel these were the people that I was reading their papers, and I was trying to meet at conferences.

I had no connections with them, but I would like you would stack them in a hallway. Yeah. And had the most awkward, like, hello, I'm a second year grad student. I love your papers. And then I run away in shame.

Did you have a chance to talk to Robson?

Yes. I spoke with him yesterday. And yeah, and he's great.

He we set up a re recurring meeting time, just so that I can hear more about what he's working on and it overall overlaps a lot with what I'm doing now. So it's a small world. Yeah. Rosalind,

John: my favorite people to talk to here at the company. And we always have these fun conversations some way when we get a chance to meet up last time was in person.

But of course we haven't been doing much of that recently, so, yeah. Cool. So you do some other things like, outside of work and things like that as well. I know in a previous chat, you mentioned that you also do rock climbing and I was wondering, how does rock climbing does it have any relationship to engineering and the mindset that you bring to that?

Or like, how do you find these two things overlap.

Austin: Yeah. So that's a funny question to ask about the overlap of, I don't know if I've thought about it. So I got into climbing in college because my university had a brand new climbing gym. And I had never thought about climbing. I'm terrified of Heights.

And I went and I tried it with some friends and I got up four or five feet off the ground and said, I'm never doing this again. It just, but I kept torturing myself. I would go back every couple of weeks and try and get a little bit higher and everybody would make fun of me cause. It was not a difficult like route as we call them.

It was very easy. I was just a chicken and couldn't overcome my fears, ended up getting a job there a year later. And now I ha I built a client home climbing gym in my bonus room. My, my entire bonus room is a climate wall area. And so the overlap, so it's funny that you ask.

So I do a type of climbing called bouldering, which is no rope. You normally if you're doing it outside you'll bring these crash pads with you these small foam pads that will prevent you from getting too hurt. And you don't go very high. 10, 15 feet usually is the limit.

But we call them problems. They're bouldering problems. And so it's a sequence of moves and they can get quite difficult, but it's all about your body positioning. At first you think it's all about like just raw strength, but you'll find that rock climbers, usually aren't, very buff people.

They look like they're quite skinny usually. They're not the most muscular people in, and you'll quickly find that it really is this, I guess you could say physics problem of trying to understand how the twist and turn your body and make these moves. So, you learn a lot of lessons very quickly about how to stand on your toe on your toes how to balance your body, because if you're pulling in one direction, are you going to swing off the wall?

So how to stabilize your body and make these nice clean movements as you're climbing and to avoid injury as well. And yeah, so it's like I said, we call them problems and it's a lot of fun. You sit there and you really try to visualize your body going up these sequence of movements. It's a lot of fun. Yeah.

John: So there's like planning and stuff that you have to do ahead of time as well. Do you study these things and plan your route?

Austin: Yes. Yeah. So once you start getting into the area, the difficulties that are challenging for yourself and there's grading systems for this. And so everybody has knows where they're at that climbs you, we call this the beta which is the sequence of moves that you have to make, and not everyone will have the same beta for the same problem.

And a taller person might climate very differently than I will. And so you can look for inspiration from others, but your own style and your own strengths really change how you climb. And yeah, so you'll plan it out. You'll work on it. Step-by-step so you don't necessarily, it's not like you just jump on and say, I'm going to complete this, you're you'll say, okay, I'm going to work on the bottom region. I'm going to figure out how to just do the first two or three moves in the sequence, and then maybe you'll work on the next two or three and then the next two or three, and then you say, okay, now I have to link them together. And that's a whole nother step of linking those sequences of moves together.

And it changes once you do that. And so you might have to go back and start should my left hand be there or should my right hand be there. And in what order

John: so do you find that you have like when you're at the limits of your ability. At that time.

And I guess you fall and stuff. Like how do you overcome the thing there's this problem you're trying to solve, but you can't quite do it. Like how do you think about different alternatives or options and things for solving?

Austin: Yeah. So there's a few strategies.

The most naive, maybe not the best one is you just need some caffeine. And taking a break. So just taking a break from climbing for a couple of days, maybe even a week, and just getting away from the problem, you might think about it a little bit, like when I'm taking a shower or something and you go, you come back to it for.

You climb so differently in that problem, or, that one move, cause there's usually a crux, the hardest part of the problem that was so challenging for you, you'll come back a week later and you're just like, you didn't even have to think about it. You just, you move past it. And sometimes it's just it just hits you.

And I don't know why but you'll get hung up on those moves. And then a week later they're not even an a problem. And then the other way is just to watch other people climate. So try to understand how they do it. And it's the funniest thing. Like when I go to a climbing gym, I always watch the little kids because the kids climbing are often way better climbers than I am.

Even they make it look so effortless. They're like strength to weight ratio is amazing and I will just go mimic them. So sometimes you're overthinking the problem. And

John: So it's interesting, right? Bringing it back to kind of computer science things, is that oftentimes you have those shower moments, you're sitting there, you're away from the problem. And all of a sudden the solution is just appears to you. Because you're, you've been processing it in the background, but I think that the difference here is this physicality thing, right? So maybe, maybe there's a, something that you're not flexible enough.

You're not like there's some, there's something that's keeping you from accomplishing the goal.

Austin: And so that's one of the things that I enjoy about the climbing is that If I'm not getting better, technically I can get better through cardio or flexibility or different strength exercises.

And so even when I feel like I'm plateauing, there's other areas that I can still improve it feels like I have more chances to get better at it. Sometimes with coding, it's like, I don't know the alternative to get around this problem, like how to get better. Cause it's, I just need to get better at programming.

Right. But maybe I need to treat it the same way and work on, those tangental aspects that, that have some overlap or perhaps

John: this same kind of verbalization thing that you, that we were just talking about earlier. Is that like, when you're observing someone in a gym, I'm guessing they're not talking to themselves, well, I'm going to go do this thing and I'm going to grab this thing over here and go swing around and do this other thing.

You just observe it. Yeah, it would also be interesting, if you could get inside their head or maybe someone that's, I guess there must be people that live stream or record, or do YouTube videos of all these things where they talk it through. But those kinds of things like around here, I've been I built myself over the holidays and F1 driving simulator and I've been trying to get faster and what's challenging is when you're watching good streamers do this stuff, it's really not obvious what they're doing.

And they're not really talking you through the thought process. So again, just getting that extra little bit of insight I think you can do a using all of them. It looks like they're starting to brake here, but you have to almost go frame by frame through the video to see what they're doing.

That's different than what you're doing.

Austin: Yeah, in research, we do a think aloud or a talk about study, where we ask people to verbalize what they're thinking as they're doing like a programming problem. Similar to a programming interview, but what they are saying doesn't always match. So you also have to be careful.

So what people will verbalize them, what they think is important might be different. They don't do it intentionally, it's the subconscious thing. Do you

John: have a good example of some, like a case where you ran into that, that you could share?

Austin: Well, I'll say that when people really are struggling when their cognitive load is high, the last thing that they want to care about usually is trying to communicate with you.

Or to articulate their weight words just right. And you can practice this. So do like a math problem in your head and try to talk out loud. And I mean, like a really hard problem I don't know, start doing some three or four digit addition and keep going. So add three numbers to three numbers and then three numbers and three numbers, and keep that in your head.

While also trying to verbalize your strategy. You'll probably be very good at verbalizing the numbers, but you're not going to be able to tell me your strategy for, okay. I'm trying to think. Remember it in blocks of two. I don't want individual digits in my head. I want to do start grouping them in twos or threes.

I don't think you'll be able to tell me that. So it, there's also some issues and this like psychologists and sociologists will use this method in lab studies, but it will change people's behavior when you ask them to verbalize this information, because now they're doing more regulation of their thought process. So it can still be useful, but also you have to remember that you're changing their behavior by asking them to do it's like a Heisenberg

John: problem, right?

It's the, there, because they're, you're now overloading their cognitive abilities by asking them to do this other high level thing on top of this high level thing that that they might do something completely differently. Had they then had they not been verbalizing at the same time.

Very interesting. Yeah.

Austin: That's a good thing. I had an engineer once get really mad at me because I kept asking him, please continue thinking out loud. I had to remind him every couple minutes of going after going silent. And he got visibly angry during the loud. Yeah. Yeah. Yeah.

John: Wow. Very interesting.

So we're approaching the end of our time here. Let's let's wrap things up, like with what is what's something recently that that you have purchased that has given you a disproportionate amount of joy? Doesn't have to be something you bought just like anything, right?

Like something you've got some object or thing that has given you more choice than you would've expected it to have given you

Austin: So now that I'm an employee at Microsoft, I get the perks pricks plus benefit. And I saw on Twitter, someone say that you can get Lego reimbursed, like Lego, the kids toy of building models. And so I have actually been buying multiple times. I've been buying and that they now let go now makes these adult models where they're more complicated and they're nostalgic.

And so I, for example, I just built the original Nintendo entertaining. System the NDS from the eighties in Lego form. And it even came with a little TV. It came with the super Mario cartridge. And it works like you can put the cartridges in it. The TV actually has a carousel built into it, so you can crank it and you can watch Mario jump and in the level moves.

And

John: That's what I bought her or was that, is that like Lego or something that's

Austin: making it is a mechanical Lego on, so it's like a conveyor belt. And as you are cranking it, it turns the conveyor belt. And so it's a continuous, Mario level made up of all these little, very small tiles, and so the TV is the casing around the conveyor by so that you can only see the view into a small region. And it's like the, it looks like the first Mario level. So I did that a week ago and that was some great, just thinking time where I can mindlessly follow the instructions, building the Lego, find that enjoyable and either just sit there and absolute silence and like meditate on my day or what I want to work on next, or listen to a podcast.

And so I think that's my new, like favorite after work, like cool-down activity is doing Lego as an adult.

John: That's awesome. Yeah. One of these times when you get out here, there's there's brick con in in Seattle. That happens every year. I think it's in the fall. And so you can just walk in and there's you see all the Lego nerds or there's lots of ligaments and the things that they've built and there's some absolutely amazing and fantastic things. Remind me to share with you

some of my Brick Con pictures I've taken over the years. Cause I think there's this. Yeah. I think he really enjoyed that as well.

Austin: Does that mean you build Lego?

John: Certainly. Yes. Like we've spent a lot of time as a kid, I spent a lot of time building Lego and I've got kids here.

They've got Lego sets. We've certainly spent a lot of time going to build building Lego here. Something I don't do as often as I would like to I think that in some ways as a kid I remember that it was a lot of fun. Coming home from school, right? That was the thing, at lunchtime, get to go home.

I can feel my Legos. And and that was just, it was fun. Like it was like, we didn't have these specialized sets and stuff back then. It was just like the generic Lego things. And you just have to imagine how to build different things out of what you had available. And that was just, that was a lot of fun.

Just the kind of creativity from just going to building things and imagining what you could build out of these things. Yeah. Yeah. Yeah. So we were, the kind of like the sound of just rummaging through your box of Legos. That's a very, like a visceral sensation from childhood.

Austin: Oh yeah. Or one of your parents stepping on one of them.

John: Very cool. Well, thank you so much. This was a ton of fun. I certainly had a lot of fun talking about this and hope you you enjoyed this experience as

Austin: well. I did. Thank you so much for all the good questions.

John: I hope you enjoyed this inaugural episode of our podcast. Stay tuned for future episodes, where we're going to be interviewing other members of our team.