Skip to content

John Lam's Blog

2022-05-08

Episode 2: Joe Binder🔗

Summary🔗

Joe Binder is an 18 year veteran of Microsoft and Developer Division, with nearly all that time spent as an individual contributor. Joe's my boss today, and we look back on his time at Microsoft and the many lessons learned along the way.

This was a fun conversation about many things, including:

  • Explaining what a Program Manager in Developer Division does to three different audiences: his kids, a new college grad, and an industry veteran
  • Why he thinks he gets more mileage in his day job out of his English major than his Computer Science major
  • How someone can spend three years working on memcpy
  • What working with large partners like Windows, Azure, and Office really entails and how to be effective in that role
  • Why he really wanted to come to the Pacific Northwest 18 years ago and why he decided to stay
  • What it was like listening to customer feedback in the early days of the Visual Basic .NET team
  • What we should have built instead of Lightswitch
  • How his super power of not taking being yelled at personally contributed to his career success
  • The balance and joy that he gets from living near Puget Sound and his fishing kayak
  • Why the Azure CLI needed to be rewritten

Transcript🔗

Click here for an interactive transcript.

John: So you've been at Microsoft, working in Developer Division for like over 18 years.

Joe: I have, yeah

John: And almost all of that time as an individual contributor.

Joe: Yes.

John: And recently became a boss, like person here at Microsoft and you're in full disclosure. You're in fact, my boss.

So this could be a career limiting, move on my part. But what I thought, I like to do is focus on that time when you were in individual contributor on, so in other words, most of your time here and I thought an interesting way of jumping into that conversation is to have you try cause both you and I are our fathers, we have kids.

And I think one of the challenges that we often have is trying to explain what we actually do in our jobs to our children. And, so what I thought would make this more interesting.

While talking about being an individual contributor is if you could explain, imagine that I was a child here, and you're going to explain to me what you do in your day job.

And then imagine that I'm a college age person. And I was thinking about coming to Microsoft, like, how would you describe what you do to that person? And then finally imagine you're actually like me a grumpy old veteran person and you're in, I've worked somewhere else, not in Developer Division. How would you explain what you do to me?

Yeah, that's a very provocative question. I do have this discussion with my kids all the time. I don't know I've ever given them a satisfactory answer, but let me try. I think of my job here as a PM at Microsoft, as building things for users, with computers. And if you think about a phone, you tap on the app that you're going to launch. Somebody has to decide what that app does and what it looks like and how it behaves. And that's a big part of my job is trying to figure out what technology should we be building to really make users lives better or more fun. And every project I've worked on has distilled down into that.

But how do you know what you need to make? What that person who is using that thing needs?

Joe: Yeah. I think that's actually, the thing that I love most about being a PM is learning that is figuring out, okay, John, I just built this app. I want to watch you use it. I want to see how you interact with it and learn where are you struggling?

Where are you looking at it? This is not helpful. And. Then also looking at data too, to see what else are people using? What are they familiar with? How can I take some of those idioms that maybe you're used to double tapping?

John: But Joe, what's an idiom?

Joe: Yes. That's a very, I'm actually an English major by trade.

So I use arcane words. Sometimes I think what are the interactions with technology that you really familiar with? So if you're used to swiping left maybe my app that I'm building, should really take advantage of swiping left because that's something that your brain already knows how to do.

And so I should design whatever task I'm enabling for you as a user to really optimize for what you already know.

And by doing so you're learning more about me, right? As as the user of the thing that you want.

Yeah. That is definitely the most interesting thing. I would say that a lot of my work as a PM has started from what would I want.

And it's always really fascinating to me for products that I've worked on for a long time, how much the product pivots away from what I wanted. And I think for me, that's the fun part is like realizing that there's so many user needs out there that are so different from mine that I need to go learn.

It keeps us honest. It keeps it's fun. I think.

John: That's super. So let's switch gears a little bit and now pretend that I'm this college age person and I'm thinking about working at Microsoft, I just went to a career fair and see the logo and lots of fancy stuff about the company.

How would you describe what you do to me?

Joe: Yeah,

John: I took computer science, let's say.

Joe: Yeah. So for computer science student, I would describe what I do as building tools to make developers lives easier. And specifically as a PM, it's really figuring out what are the, what's the workflow that the developer that developers go through today.

If you think about you're sitting in your computer science class, you get an assignment, you got to go back to your dorm room or whatever and open up a code editor. You got to write some code, test it, see if it's actually working. What's hard about that? How could we make that even better? How could we make the process of learning technology even easier?

That's really what my job is about. And because I build tools for developers. I also have an excuse to write code a lot, to really put myself in the shoes of our users and work like they work and see, does this feel natural to me?

But it's different than being a programmer in the traditional sense where I'm spending day in and day out, actually writing the code for the tools that we built.

Instead I'm writing code as our users would write it. I'm writing code to understand how does the tool we build work.

John: So actually using the thing and in reality, as opposed to like theoretically.

Joe: Yes, exactly

Yeah. I to have some real projects that I'm working on, either at home or things, online courses or whatever that I'm taking, that I have to use the stuff that I'm actually building because that's helps me understand.

The strengths and weaknesses, or what's hard, or what's inefficient about the tools that we build is when I actually have to get something done, whether it's a plant watering system for my backyard, when I go on vacation, like I've got a goal in mind and it can't be academic. I really need the water to turn on when the humidity goes down.

John: Yeah. And I think that but it seems to me as a college student, that's a bit different than, proving my professors algorithms in some paper that he published. Like how does my education in computer science prepare me for what a job would be like as a PM.

Joe: I like to think of PM-ing as applied problem solving. I think about computer science as a form of applied problem solving. I always wondered when I was in school, Why do I have to take physics and calculus and all these things that realistically I've never really used in a computer science ish field. And it was all to help me understand analytical problem solving and how to break down problems.

And I think that the thing that you carry forward from your academic the proof for XYZ or whatever, write an algorithm to, traverse all cities in the United States, it's helping you understand how to solve problems in a deterministic way. And I think for PMs, we do the very same thing.

It's just the problems that we solve are actually user problems more often than not. And those are not as well structured as the academic problems that we often get, and that's probably the biggest difference is the ambiguity is significantly. higher. The uncertainty of whether or not we've actually solved the problem is similarly higher. But the core approach, the sort of mental model that you use in school around, how do you write an algorithm to solve a problem is really similar to how you work as a PM. You have to break down the problem into individual steps and figure out the constraints in which you need to solve that problem.

If we're we building an app for a phone that has limited compute as an example of a constraint, are we working on something for users who have difficulty seeing or hearing those are all constraints that we need to think about that, that you don't really have those constraints typically in an academic setting, but in a PMs world, that's part and parcel for what

John: And you also mentioned that you were an English major in college and. Do you think that was that helpful or harmful transition into working in this very technical field?

Joe: I think about that a lot because I got a degree in English and computer science concurrently, and I sorta wagered with myself at the time, which of these is actually going to be more useful.

And if I'm honest, I have enough technological curiosity that I think even if I didn't have a computer science degree, I would still be coding today simply because I find it interesting. It's a really unique way of creating things and solving problems that I think I'd always be drawn to. But the English degree, I think actually is probably more useful because being able to make sense of really dense information is just critical to a PM's job.

And it's also really rare, I think. if I think of all the other students in your computer science class, many of them probably would find it exhausting to go write a paper describing to an eight year old, what a piece of software does. Whereas I actually find that kind of energizing and I think it's really fun to figure out how could I describe this in simple and accurate terms?

That's something that I learned from working in literature and I applied pretty much every day. Like I think as a PM, you have to be excellent at written communication, certainly as well as verbal communication. And that's also the thing that I think helps propel PM's in their career is always trying to get better at how can I help people understand my point of view, the problem that I'm solving.

John: Yeah. I think that when I was in college, I took an international relations history class because I had to, it was one of those things that you you had to take. And that looked as interesting as anything else. And I remember finding my papers from that class in some box at home and I was flipping through it.

And then, cause I remember when I got the papers back, I was like, ah, I gotta B+ that's fine. Throw it in the pile. I'm like, I don't really care about this class, but what I, for whatever reason, I started reading through the professor's comments on my paper and I realized, wow, like all the stuff that I got as feedback in graduate school from my thesis advisor, were exactly the same criticisms as this undergrad history professor gave me. And I just really wished that I had communicated better or that I had learned the lesson at the time. I was supposed to be learning the lessons that, Hey, I don't care.

Yeah.

Joe: It's fascinating. I have a funny story about that one. I was just starting grad school and we had a small research grant, which paid for a semester or two of school. And it was from Sun Microsystems at the time Sun was all in on Java and then. the research grant was basically to build a test bed for this networking library that they were building in Java.

And as part of going through that exercise, of writing this sort of mundane test library, I ended up rewriting the documentation for the library because it was really poorly written. It was really hard to understand. And so we shared that back with Sun at the time. And they said, this is great. Forget about the test harness why don't you go rewrite all documentation?

And it was financially very lucrative for me. But it also opened my eyes to

John: so they paid even more over above the grant

Joe: exactly Yes. And it was really eye opening for me, just how critical it is to be able to capture and clarify exactly the work that we do. And I still hold that near and dear to everything that I do now.

Like whether it's an email I write or something that I work on, it's if somebody's really going to be able to walk away with this with a clear understanding of what I'm trying to say. So, yeah, I think overall English has won out as the as the more useful,

John: the truly useful. Yeah. No, that, that's not surprising. It's funny. I've talked to some of the new hires on our team and one of the things looking back, so I have no idea why I took computer science, it's like none of this stuff, I don't see myself using any of this stuff. And so why did I need to know it in the first place outside of what you said, right?

Which is I needed to figure out I needed some place where I would learn problem solving skills and computer science is probably as good of a domain as any

Joe: Absolutely

Yeah. I think it's also interesting for, I, think about this a lot, just given how many developers we serve now that don't have computer science degrees.

And so in theory, it hasn't been super useful, but practically speaking, there's certain code. If you will, certain jargon, certain even biases that we have. For folks who don't have a computer science background that I've been really reflecting on how do we break those down? So that really folks who are good problem solvers can be great PMs anywhere.

John: Yeah. That's what attracted you to the position?

Absolutely. Yeah. Yeah.

So now shifting gears one more time. So imagine that I'm now coming from somewhere else, let's say in a different company, even and you're trying to explain to me what a PM is in Dev Div Developer at Microsoft.

Joe: Yeah. This is probably the hardest of the three in my mind because PMs Dev Div are quite different from other areas of the company. Certainly areas in the industry because Dev Div has a unique role in the company that I've not really seen anywhere else, which is that we are really are responsible for building developer experiences, developer tools, APIs, SDKs.

Chances are good if you've used a Microsoft product as a developer, you've used a product from Dev Div and what's interesting the role as a PM here is that we rarely actually own the platforms that we support from a experience standpoint. And that is both really cool because it actually enables us to work on lots of different products.

Like I've worked on Azure, I've worked on Office, have worked on Windows, all kinds of from the same chair, if you will. And at the same time, it makes our job really hard because somehow we have to both partner with those teams to really help them understand what are the primitives that they need to go build to help developers understand their platform and be successful.

And also we have to build things on top of that to actually make it accessible, right? There's some, you can only make a, micro-controller so accessible to a developer at some point you need tools. And and I think a lot of,

A lot of what do as PMs in Dev Div is figuring out what's that ideal developer experience?

Where can we actually add the most value for developers to make sure that when a developer comes to a Microsoft platform, they have a really productive experience. That's part of our brand as a company. And I think we've come a long ways from the Steve Balmer jumping up stage developers, but the spirit of that still holds true.

I think. We have developers who come and continue to use our platform because of the tools. And so in Dev Div we hold that really high. I think whether it's VS Code or Visual Studio or Azure platform services like App Service or Functions, productivity is probably single most important aspect of the products build.

John: When I think about what you said earlier about how our tools and div are really at the service of a platform owner, like an Azure, like an Office, like a Windows and how we don't own those things. Those by PMs over in those parts of the company. How would I think about coming here if I've been steeped in product management through this idea that my job is to grow usage or revenue or some other growth aspect of the product itself.

Now what do I do once I get here?

Joe: Yeah. I would say that the single most important thing, if I were coming to this job or Dev Div in general is to build great relationships first and foremost, because everything we do is team sport, right? If I think about I'm going to go work on Windows tools or Azure tools, it's really shared success with the platform teams. And so building relationships to really understand like deeply amongst the platform team I'm working with, what scenarios are you really trying to enable and make sure I really understand. that because, as I've reflected on my own career, a lot of times when I've had like friction with some of these teams or just haven't jelled it's because I haven't really understood what scenarios most important to them. And definitely say that's the first thing is to really internalize, okay. Before we talk about usage or revenue, what's the core scenario you're driving for and what's the core outcome for that scenario? Sometimes it is something like revenue, where we might be looking at a small number of customers who are just going to pay us a lot of money, or we might be looking at, we're trying to garner usage from millions and millions of developers, understanding what's the business outcome.

What's the scenario. And what are the relationships I need to, build in order to understand that's definitely like the first thing I always find I need to do when I'm working on a new area.

John: So is that understanding the growth needs of the partner team and then figuring out what Dev Div's role is in trying to help them achieve success?

Joe: Absolutely. I think we often, I used to struggle with the notion of being in service to platform teams. And I don't really anymore because I really do think that our job is to multiply the impact of the platforms we support, whether it's Azure Windows or so on. So yeah, certainly understanding what does growth mean for them?

How do they want to grow is the most important question that we need to answer. And certainly in Dev Div, I work on VS Code as well, which also has its own standalone growth objectives. So we do have some things that we own ourselves.

But I would say that for most products, understanding what the platform's Growth objective is the single most important thing I do first.

John: And then from that, all of the, what kind of follows from understanding the why of the partner team.

Joe: Yeah Yeah

John: Excellent. So then switching gears again a little bit here what I thought we would talk a little bit more about is a little bit more about you and how you came to the company. So again you're at RIT you're an English computer science double major. What kind of drove you to come to Microsoft in the first place?

Joe: Yeah. Interesting times. So I had worked ,so RIT where I went to school Rochester Institute of Technology. They have mandatory co-op program, like many tech schools these days.

And that was actually super cool because it gave me a really good sense for the type of work I wanted to do and the type of work I didn't want to do. And I had worked as a sys admin.

I worked at a company called EMC that a pretty low level, like kernel level development.

Exactly. They're still going strong. And I realized after I worked for them for couple of years during college, even, when I was in school. And the main thing that I realized was that I loved the hardcore problem solving and like the technical excellence, but really missed the user interaction aspect of the job and being just a developer.

I worked on for the entire tenure for three years working there. I worked essentially on memcpy operations. And I learned a lot about performance optimization. I was the only person in the entire team that didn't have a PhD in some operating system- like field. And I realized it actually at it came to a head during 9/ 11 which was like the height of when I was trying to decide, what am I going to do after undergrad?

that I realized that I didn't want to just live in a cave, so to speak work down in the depths. So I really wanted serve users. That's what always drove me to get computer science in the first place. And I wasn't quite sure if I wanted teaching and education because I definitely enjoy that aspect of it.

And I teaching in grad school

Or if I wanted to actually just totally change gears and leave my comfort zone. And I actually chose where I wanted to live before I chose where I wanted to work.

So I had lived in upstate New York for quite a while, eight, nine years. I lived in Boston previously and all of them have a super intense, very edgy environment and culture work culture, the tech culture there at the time, I don't know if it still is super intense. Everybody works 80 hours a week drives 80 miles an hour to work like it's very edgy,

Very confrontational And and I needed a change of pace. And I also love my real passion in life is outdoor activities. So I love to Mountaineer.

I love to fish. I love to whitewater raft and it, the Northeast is not really.

that, is exactly

John: It's

Joe: Yeah

John: is missing.

Joe: Totally. So I decided that I wanted to move to the west coast. I didn't want to move to California. So somewhere in the Northwest.

and it just so happened that Microsoft came to campus.

I sorta joked because I had never, I like literally never had a Microsoft products after

John: Windows 95?

Joe: I Like I think we at school one year we had a Windows 95 or 98, like a bench of computers. But otherwise I had not really been exposed to Windows, anything, and I was not super high on Microsoft generally, but I showed up as a joke to my friends, and I'm going to go to this Microsoft interview and then they will pay for your your travel to come out and stuff.

And then so long story short. It worked out. I really planned it as, this is a sort of cliched story, but I really plan to just come out for a year or two and then decide what I really want to do. When I grew up and 18 years later here, I am

John: You thought you'd come out here, get paid to relocate, check it out and then find your real calling.

Joe: Yeah How many years do I have to work here before so I don't need to pay back my relocation fee.

John: So I have to about the memcpy thing. How can you spend three years working on memcpy?

Joe: Yeah, it's interesting. So at the time Intel had all these hardware instructions for like specific scenarios for memcpy and it's like EMC.

I worked in the file server division, and our job was we basically sold giant file servers. So this is the time when you would have a giant rack of hard drives and their big claim to fame was that they had, you could actually buy a quarter terabyte rack at the time with lots of cache in memory cache stuff.

It was a fascinating computer science problem because tiny performance improvements had huge business impact. So our primary customers were credit card companies, air, traffic control, large governments. And so if you tell a credit card company that, you can increase the number of transactions, like financial transactions per second, that they can, it's like millions of dollars per day. And we had a number high value scenarios. One of them was transactional scenarios where literally we had all of these different situations where we try and use hardware instructions to actually perform the operation instead of like higher level C things like that.

was a case where it always really hard to detect in which fork to take, because you often know what was in cache, what wasn't so that,

John: Were you tuning memcpy for specific. This is the L2 cache size on this CPU

Joe: precisely Yeah. Because they sold standardized hardware, so we could actually make a lot of assumptions about, and they had a fork of the file system and everything. It was very optimized for their scenario.

Interestingly, I also, what led me to leave there was one of the other scenarios that we sold, as a, as like a sales playbook was build machines and build clusters for like really large monolithic software.

And I spent three months debugging a intermittent bug in when the machines were configured as a cluster for a build cluster. We had this intermittent storage loss problem. It was super uncommon, but it turns out like you can't really sell something that keeps your data most of the time.

John: For credit card companies.

Joe: Yeah, I spent three months working on that and it turned out to be a bug and Linux and, but, it was I remember fixing the bug and feeling no sense of gratification whatsoever. It was just like, you know

John: Why did I have to do this?

Joe: I gave somebody a reason not to leave but it's that was the straw that broke the camel's back, so to speak. And I realized that I needed to do something different.

John: So it convinced you no more being a software engineer working on memcopy at the very least.

Joe: Exactly Yeah. And kinda thought maybe moving up the stack.

But I also realized that I am just energized by serving users. Like I really like this interaction it's like being a pro, a an instructor is that seeing light bulbs go off for folks is like a very energizing thing for me. And so I realized that working on something where I had to interact with users and really solve their problems was what I needed.

John: So then you arrive at Microsoft. What was your first thing here that you had to do?

Joe: Yeah, this is going to be an embarrassing couple of minutes here. So I had worked on file system stuff at the time. And so my first at the time .NET was coming out with version two. And it was still we're at this point of indecision, I guess you could say in terms of, we have all these Visual Basic customers who basically put Windows on the map. The ease with which you could build apps.

John: I was of them.

Yeah. It is just incredible how empowering Visual Basic as a tool for users and certainly for the company as well. And so the team that I joined was the Visual Basic team and our real challenge was. like how do we help this huge population of VB classic, non .NET users adopt .NET Because that's where all the innovation is going and overwhelmingly the canonical user of Visual Basic really struggled with .NET because it was written for computer science types. It's one of the first examples that I worked on was, the file API is for VB.NET at time. And I remember going to a user study on reading a file.

And I remember okay, here's System.IO. You got to read a file stream. And I'm like, yeah, this seems fine. It seems like these higher level API. is, And we're really, I'm used to living like down an inodes and stuff like that. So like System.IO was like, whoa, this is super easy. And. All the users were like, I have no idea what this is.

I just need to read the text this file into a string so that I can present it on a text box. Like how do I do that? And that was super illuminating for me. And that kind of became my first job was we worked on this, this set of wrapper, APIs called the "my" namespace. And it was loosely modeled after in Windows, My Documents and all that sort of stuff.

Joe: So we tried to take the mental model of the user and manifest that in a set of APIs that were meant to be easier APIs. So an example would be instead of getting a file reader or a file stream, you could just call a read text method and it would just return a string that had the text from the just Let me pass you the path to the file I want to read and return me as string. That's really what I want.

Yeah. And I worked on that. For quite a while. What was really there was, this was a kind of good, my first lesson in a way, like I learned a lot as a PM in terms of how to do user studies, how to really understand user needs and expectations, because we iterated on those APIs for over a year.

It's just, it's so hard to get the conceptual model right and I gained a deep appreciation for that because I think I, that was the first time I really realized that the target audience we had, they weren't trying to read and write files. They were accountants or they were trying to solve some much bigger problem.

And this was just the tool that they were using in order to solve it. And that light bulb went off for me. And it really energized me in a way cause I'm like, okay, cool. Like I, I'm never going to stop until you are successful. So after the VB product, we worked on the my namespace It never really took off. And in fact, VB.NET at the time as it was affectionately referred. We also were at,

point in time where, a C# was really the brand of .NET and C# were indivisible and we had to really make a choice at some point, of like, how deeply are we investing in VB?

At the time we had a huge VB team that was working in concert. Amanda my boss was the in charge of the compiler and the language for VB.

It was we had a lot of investment in that space because it was so critical to the company. We could not lose those customers. Of course we did lose a lot of them,

John: yeah, exactly.

Those customers, I still remember in Microsoft keynotes, the Professional Developers Conference there'd be these VB zealots, let's call them right. Sitting in the front row. And they would count the number of times that you had mentioned VB and the number of times you had mentioned C# and they would say Microsoft spent three times more times talking about C# than VB.

We feel not heard or unseen. So how was that?

Joe: So I learned a couple things from that. So absolutely. I remember the very same thing, of like really having to be intentional almost to the point of being unauthentic about parity between languages and things like that.

What I found was a couple of things. One was that I realized for the first time that for a lot of folks, those folks who were sitting in the front room of the front row, VB was a huge part of their identity.

John: Yes.

Joe: And we weren't messing with a product. We were messing with their identity and that never dawned on me until, because for me personally, like my job is just my job. It doesn't define me.

John: You were too young, you didn't have history.

Joe: Exactly. Whereas these folks said written seven, eight editions of, business applications with Visual Basic and all these other things. And so I came to understand a couple things from that. One was for me to just personally really reflect on how much of my identity I wanted to be based in my job, because I saw how crushing it was for them to have something intrinsic to who they were outside their control and going in a direction they didn't want. The other thing that I found really striking was we had these zealots as you refer them, like really passionate folks and they occupied most of the soundbites in the room. And I came to the realization as a PM that a lot of what they were saying, wasn't really representative of what users who are day in and day out using this stuff needed and were struggling with.

And that was a super important learning for me because it's always I think as a PM, there's always a email coming in every day, a tweet or whatever, There's some really emboldened customer who tends to have an outsized influence on the team. And so that kind of taught me the checks and balances of these are the most vocal people.

They aren't necessarily the most important folks. And they also aren't necessarily representative of the overall.

John: The squeaky wheel gets the oil.

Joe: Absolutely

John: And was there a lot of that back then on that team?

Joe: Yeah, this is a funny story. I told you that, I worked in this intense east coast environment where people would, you'd go to a code review and it was common to get yelled at.

That was like a common practice. And it was about my first week on the job I had to present to just the zealots. We had a private, I forget what we used to call it.

But it was over in building 20, right across the street from where we are now.

John: The SDRs?

Joe: Yes, software design reviews.

Thank you. Yes. And had to present this, my namespace thing that I had been working on for a total of five days, I think at the time. And it was awful. It was just awful. It was like, designed by committee. A hundred times over where I just got yelled at for a good two hours people I had never met had some deep seated frustration and really frustration was displaced.

They weren't really frustrated with me or with the product, with this sentiment.

But I also realized that was a bit of a superpower for me. It's pretty, I realized like afterwards I was like, yeah, that's just another day. But I realized that can be a skill that I can lean on.

Sometimes I don't really get ruffled very easily.

John: So just distancing yourself from the people checking yup, still got a job.

Joe: Yeah Exactly

John: Didn't get fired!

Joe: Yes. Yeah.

John: So then moving on from there. What led you to your next role at the company? Like how did you transition? How did you choose?

Joe: Yeah, it was interesting because I mentioned that VB was on the down swing and what we realized was that, or this is my interpretation of what we realized. We were missing the mark. We didn't really have product market fit effectively with VB.NET.

So what me, Jay Schmelzer, John Rivard, both of them still in Dev Div today and a few others.

We started just prototyping what we thought was actually what we would serve, the folks that we were talking to the boots on the ground, so to speak, who were using VB back in the day to get their job done, what they really needed, which was like what we would call forms over data. So really simple visual forms talking to a database, creating, transactions, querying data.

John: We called it RAD tools or rapid Application Development tools back in the 90s.

Joe: Exactly Yes We prototyped for awhile. This is when I went back into a little bit more of a dev mode. I did a lot of the prototyping myself.

John Rivard did a lot of it as well. There were a bunch of folks from the data team that owned like SQL Server and all that stuff that we all just had a skunk projects. And we ultimately chose to launch that as a product. It was called Lightswitch. yes And it failed pretty epically and but

it was probably the most important thing in my career to to happen because I had never really worked on something from the up and had to be accountable for figuring out something like product market fit. And I realized about a year or two in that I didn't really have this, the, tool set I needed to figure out whether or not we were building the product we really needed to build.

John: So that was your zero to one experience.

Joe: Yes

John: And then, we do like in the annals of history, there was this thing called Lightswitch that existed at some point, it doesn't anymore. What was it that kept you from achieving product market fit?

Joe: There's an interesting cultural backdrop to the company at this point. So this is during the this is pre Satya and I would say the organizational lines in the company were a lot more pronounced at the time.

There are frequently these kinds of zero to one ideas, just like I am the folks who work across the company on similar things, we put our heads together.

Let's try it. That was definitely not the case at the time. And we ran into a couple things. Like one was the classic large company problem of, if you think you're solving a useful problem, there's probably four or five other teams in the company who are also solving that problem. So Lightswitch definitely had that.

So Access was also going through this journey of their own of like Access had been a super popular tool. It really was the model of Access didn't scale at all. Because you. You used to build these forms over data apps, same thing, but you would deploy the database with your app and

John: It was in a single file, the MDB file.

Joe: Yeah.

There was FoxPro at the time and all these other things. And so everyone was starting to think about cloud type of stuff and trying to make these pivots Power Apps at the time was like in its infancy. So we, ha we were fighting against a lot of folks trying to figure out product market fit in the similar space and rightfully you know, Power, the Power Platform, essentially won.

And I think rightfully so they're actually, because we, what I learned from that was that having this tool that was neither a pro dev tool, nor a citizen developer no code tool, something in the middle. That there aren't too many of those users that actually exist. And that was like something I, I didn't really internalize.

It was like, I can go find all these people. Anytime we went to run a focus group, I could always find them. But like in the overall population, they were overshadowed by, effectively the users that Power Apps serves today.

John: I see. So the low code users that aren't typing a for loop or a simplified for loop.

Joe: Yeah. Yeah. So that was one of the main reasons effectively, we were building a product for a user that did exist, but didn't exist in mass. And we're also building a product that looked from somebody outside the building. It looked a lot like what other teams in the company were building.

And I think at the time the leadership team, rightfully, nipped that in the bud. So as not to create a big product positioning snafu.

John: So if you could go back in time, to, to Joe of that era, what advice would you give younger Joe?

Joe: Oh, that's a great question. The best advice I got this advice about two years after Lightswitch and I wish I had it before.

And the advice came from Cindy Alvarez. She wrote the lean customer development book. And I had the opportunity to go to Yammer at the time to just watch how they did PM there. So Microsoft had just bought Yammer Yes exactly.

Yeah.

And I learned a ton like how PMs work there and I remember sitting down with Cindy, like working on like a similar type of thing. Cause we were trying to figure out okay, that didn't work out. What should we do? Because clearly there's still like unmet needs and she just looked at me and she said, Joe, you have to leave the building.

And it's a profound thing that's always stuck with me that I realized that all of the PM work that I was doing and user understanding and market analysis was in the Microsoft bubble. It was essentially so heavily biased by Redmond Washington that it was totally irrelevant to a lot of the rest of the world. And so I think the one that one bit of guidance of like leaving the building, leaving my bubble, leaving my comfort zone of what the problem domain.

I know the users. I know like we, having worked in the industry for almost 20 years now, I have a long list of folks I can call up and oh, I need somebody in finTech who's doing I know five people But I didn't leave my own network. And so I had a raging case of confirmation bias basically from not leaving the building and in Cindy's words.

John: Yeah. So you're in your ivory tower, your network were people similar to yourself. And so you had a really hard time really empathizing really with the needs of these users that you really weren't satisfying.

Joe: Absolutely. And interestingly that you'll this one will come close to your heart.

So at the time I had started to really realize that Lightswitch was just on a path that wasn't a path we should really continue down. And I think all of us were starting to realize it without really talking about it .

And I got a random side project to do a demo of AWS and Ruby on Rails. And and it was for our VP of DevDiv at the time, his name was Soma, and I needed to do something a little different anyways. So I just needed to like re-energize myself. And I was like, okay, cool. I'll read all about this Ruby on Rails thing. And then I realized, I was like, this is actually what we probably should have built.

And because it was code centric, it took this, convention based model and it didn't try and be magic, but it did try to be helpful and smart defaults and all these things. But I think the other lesson beyond leaving the building it's just be willing to adapt and willing as I take in new information, realize that like the worst thing I can do is, continue to invest in something that isn't worth investing in and be willing to just pivot or adapt or, just own. that, Hey, you know what this isn't going in, the direction that I thought it was going to go, we should

John: Fail fast and iterate as quickly as possible as opposed to we're going to build this perfect thing we're to get enough research so we can build the perfect thing then we're going to go ship it .

Joe: Exactly. Yeah.

John: Yeah. Because certainly seems like that helps you get to a place where you start getting feedback from real people, as opposed to your cherry picked.

Joe: Absolutely. Because the Rails thing really highlighted that for me, it was that a lot of people who were VB customers had migrated to Rails.

And, also, like I remember interviewing in a customer interview since somebody who was on the VB team, who now worked at a Rails shop, and I remember like, how did this happen? Like he used to be one of us know and And, it was really fascinating because it's I we, he was working at a startup He's we started this startup, we just needed something up and running quick and, and it highlighted for me that there was this whole other community that I wasn't learning from.

Cause I wasn't talking to, and that, that whole leave the building thing, leaving the comfort zone. I realized that's probably the most important way to de-risk what I'm working on is if I'm actually talking to folks who aren't using my products, then I'm going to learn as much, if not more than folks who are.

John: Yeah, absolutely. Because there's some reason why people left to go do some other thing. And if you go through life kind of being ignorant to those reasons then you're really not going to learn. It's really hard to grow that way.

Joe: Yeah.

John: And the echo chamber here is strong in Redmond.

And cause it could reminds me of a long time ago with the I still remember the very early Ruby confs. This is circuit 2005, -06 before I came here and what was interesting about Ruby was really more that it attracted a certain kind of person, especially in those days.

Rubyconf back then was a single track 200 person max conference. And when I look back at the people who were in the room physically in that room and the kind of value creation that they created both around Ruby itself, but mostly let's be honest around Rails is astounding, right?

We had the GitHub folks. I actually wasn't quite the GitHub folks but the people before they even thought of creating GitHub were in the room, which was super interesting. Shopify Toby was there. He gave a talk at that conference and it was just like, ah, I'm just trying sell snowboards online. And this Rails thing was awesome for doing that.

The Twitter folks were there.

It was almost like the beauty or the elegance or the kind of the aesthetic perhaps of the language. It brought people in. That alone I don't think would have given you those pragmatic people that you saw.

But I think that it was really DHH and Rails that and in particular, the demo the Rails demo is just this incredible demo is I'm just going to type some stuff with the command line and boom, I've got this whole application and people go, oh, how do I get more of that?

Joe: Yeah. I think the other aspect of this, I remember seeing some demos of real Rails apps at the time.

John: It wasn't just a toy

Joe: Yeah. And I think it was User Voice or some of these other web platforms. And it stood out to me that the thing that I love most about my job is actually seeing what people create with technology. And seeing to me, it's we often think about as PMs, like, what should we talk about at conferences and whatnot?

And the thing I realized from all of the open source communities I've engaged with over the years is that the things that really create the energy is sharing. This is what I built, and this what I learned. And that's such a powerful thing. That I think it's also a bit of a now it's in my sort of toolbox If I can always looking for, if I'm working on something, who's actually building something cool with it?

It's a kind of a nebulous signal to use, but it's an important one. Like you kinda know it when you see it and if there's a vacuum and you haven't seen somebody, who's like wow, I built something cool with. this. That's probably a tell and I want to drill into that, but yeah,

John: Tim O'Reilly built an entire empire, O'Reilly publishing around what he called the alpha geeks or what are the alpha geeks looking at right now, because that's, that, that is an indicator where the industry is likely to go.

Joe: Yeah. And I think it's also I worked after Lightswitch I worked on Azure stuff pretty much until present day and that story has played out time and time again, in terms of container orchestration, CI systems, et cetera, et cetera. It's not really the best that wins often.

It's the one that folks have found the most useful and who are willing to stand on a podium, so to speak and share how useful it's been and contribute back to it because, oh my gosh, we rely on this. Now we need it to be better.

That playbook is super it just continues to repeat itself.

And I think Microsoft has done a better job of recognizing that over the last several years, but at the time, way back in the early 2000s , that was not that was not really in our wheelhouse.

John: Yeah. And so you live through that transition, the transition from this very kind of like inward-looking echo chamber-y kind of company that I still remember cause he brought up Soma.

Before I had to give a talk at OSCON the O'Reilly open-source conference, we wanted to announce that we were going to contribute some tests back to RSpec or Ruby spec, which was this test suite that kind of, because there was a proliferation of Rubys at the time and we needed, and there was no spec, right?

The language, the spec was the implementation. So we needed something that would validate whether or not something was Ruby or not. And all we want to do is contribute some tests to this Ruby spec suite and I had to get Soma's permission, which finally came at midnight , the night before I was going to give my talk at 8:00 AM in the morning.

Oh, that's cutting it a bit close folks. And and that, that was the environment, right? Like we're contributing tests to some tests suite and I needed senior vice president approval in order to go get that done. So how is it from your vantage point like that transition point from, this kind of super inward looking company to now the number one open source contributor, I think out there in the world.

Joe: Yeah. It's it's really interesting. I did have a front row seat to some of that because it's at the, contributing back to RSpec like the precursor to that for us was when I was working on Lightswitch, we had a web client and it required legal approval and VP approval to use jQuery.

And jQuery obviously at the time, it was like, you really couldn't build a website without jQuery, practically speaking. And it was hilarious, but I think over the, after Lightswitch my focus started to be on non .NET languages and customers on Azure. I pivoted there. I came back from parental leave and I was like, told my boss. I was like, I just need to work on something different, throw me on something different and I'll go. And and it was interesting because if you, at the time we actually had a separate organization. MS OpenTech

John: Sam Ramji's thing.

Joe: Yes exactly. It was like a subsidiary of Microsoft to work on open source.

And I had come back just as that was dissolved and it was really interesting because at the time we were trying to get tools like Jenkins, Ansible, and so on to work well with Azure. And there was still a lot of friction for a long time of if we got Ansible to run well, is that going to erode Powershell's share on Azure and things like that?

And I learned a lot from that because I also found that sometimes the open source things ... it seems like at the time folks viewed them in an almost binary way. They were either amazing and perfect. And we should just drop everything and use them, or they were the bane of our existence. And, we should wipe them off the face of the planet or whatever.

And what I realized is that, as I because I had this learning now, that it's I need to see customers building something. Like when I was working on the Jenkins stuff or the Ansible stuff or Terraform in the early days of Terraform, it's the first thing I wanted to do is I want to really get to know every customer who's using this so that I can learn from them.

That was like my big lesson learned. And it gave me a lot of insight into ... Nothing, there is no panacea in technology, everything has limitations and drawbacks. And I think it took us a few years as an organization to recognize that open source had pros and cons just like everything else.

And so can we actually I think now we've definitely gotten to the point where we're super objective about it. We understand the critical role it plays. But working through those kind of growing pains was definitely challenging. I remember being I was working on Jenkins stuff as well as we had chosen to rewrite the Azure CLI because we had this really old node CLI which most customers failed to actually install successfully.

Cause getting node on your box is super hard. It was really slow and it was also a lot like the PowerShell interface. It just didn't feel like a POSIX style tool.

And I remember going to various teams in Azure, describing Hey, we really need to build a real command line tool for our Linux audience.

And yeah, for a while, I was yelled at pretty pretty frequently purple in the face, very senior folks. And I learned a lot from that, but I think ...

John: So your earlier thick skin thing really helped you.

Joe: It definitely a super power of yes, I'm always willing to be yelled at because honestly, like I've always found that I've learned from those those moments a ton.

Like I remember having a discussion with Jeffrey Snover, the creator PowerShell and he was really upset. And it took me to why he was so upset to realize that actually there were a lot of this plan that I had put together that I really hadn't thought through that mattered a to him and I, it was one of those times when I I really because I ended up having dinner and we went to a shared dinner and he happened to be there.

And and it this light dawned for me, because a, fact that I hadn't really or defensive meant that I could now sit across the table from him at dinner, without it being super awkward. I also learned a ton and it really taught me that like in a company like Microsoft, even as big as it is, you always, people are in your network forever.

And so how you respond in those moments, it's going to be the lasting memory that person has of you. And. And it was just a good reminder because I was pretty close to flipping the bit on that one. I'm like, you know what, I don't really need to put up with this. And but overcoming that I think was a good, it was a big milestone for me I'm just like realizing that wow, like actually we ended up collaborating on something the very next month, which was like the porting PowerShell to be cross-platform.

So they're building on dotnet core. And I was like, hey, I sit in the same room as the dot net team, let's work on this together. And it really worked well, but it's one of those things that I think, that was like the next milestone in my career of just learning how to be patient and recognizing that people, the the sort of permanence of one's network in a company like this.

John: And also that ... The one thing that I've learned I think is that people aren't, I don't know, mean or difficult or whatever it is because it's what they are, who they are, but the rather it's almost always a mismatch in priorities. Your priorities are different than Jeffrey's priorities.

And because of that, you look at the world through the lens of your priorities.

Joe: Absolutely Yeah.

I think a mismatch in priorities, and then a different point of view in terms of what requirements we're actually trying. What's our objective that we're really going after, because sometimes I think I've assumed a shared objective when there really isn't one.

John: Right

Joe: And it's no, actually, like we just have different objectives and it's okay, cool. Once I understand then it's actually much easier to engage.

John: Yeah, one thing that, again, that's always really challenging is assuming that you're on the same page when you're not actually on the same page.

Joe: Yeah. Yeah. I think the times when I have not been at my best are the times where I've had those whoa, what just happened there moments it's because I made assumptions that I didn't actually test about where we were as, two teams working together or something, even two people. That's that's a lesson I I continue to learn from I think it's not always it's not always roses, but I think figuring a better system to like always check assumptions about what problems do you think we're solving?

What problem do you think you're solving? What are you driving? What are we, what's our outcome having that explicitly up front, just save so many problems.

John: Yeah. Because in the absence of that, everybody just makes up their own story in their own mind.

And they think, yeah we're on the same wavelength here

Joe: Yeah. Yeah And then they get upset when something different happens.

John: It's somebody once said to me a long time ago it's impossible to over-communicate in a professional or in a work setting. That's just hard won experience from things like this. And I certainly seen in the past when people have a completely different idea about something that I thought I had explained super clearly.

Joe: Yeah.

John: And and that's really hard. So again, like we've few interesting things that you've done in your career. Was there something looking back over the entirety of it that you wish that you had said yes to, but you didn't at the time?

Joe: Yeah. That's a tough one. I have to say I don't... I am usually a yes- and type of person.

John: Improv.

Joe: The examples of things I wish I said yes to I, I'm not sure I'm really not. I think the only thing that I actually wish I said yes to are things that happened outside opportunities I had outside of work that I didn't say yes to because I committed. to work.

And I definitely, I think those are the most profound things for me, where like at the end of the day, my impact at work is not predicated on whether or not I'm here this week or next week or whatever.

And so. it's hard For sometimes to recognize that

John: The place is going to fall apart if I'm not here.

Joe: Yeah, exactly. Yeah. I think those are probably the biggest examples. Yeah.

Yeah.

John: You rarely find anyone at the end of their life looking back and I'm like, oh, I wish I've worked that Saturday. Yeah on whenever that was Yeah. Yeah.

So again, looking back at everything, what's the thing, what's something that you're particularly proud of as an accomplishment? Not necessarily because it had huge impact or anything but something that really changed your trajectory.

Joe: Yeah. The thing that changed my trajectory, I think was this is a longer answer. Most of my role, the first 10 years of my career, like working on VB, working on Lightswitch, that kind of stuff. I worked in a pretty isolated team. And I didn't really know how to collaborate cross team.

And I think a switch flipped for me when I realized that everybody that I'm working with also has a life. They also have emotions, they have feelings that they are just trying to get their job done and I can play a role in making their lives better that to actually help them in some material way.

Okay. I tend to spend a lot of time with Kusto and telemetry and stuff. It's you know what, maybe I can spend 20 minutes helping you write that query for whatever. Realizing that the sort of human aspect of the work environment and that I did have a role in like lifting people up that actually changed my career trajectory in a pretty profound way, because it meant that I was no longer entering problems trying to be like, I'm trying to get this outcome.

Like I needed a decision made and I'm going to go like beat this person over the head until the decision is made or give them all the information, like machine gun the information to them until there's no other answer but yes.

When I realized that at the end of the day, what gives me the most joy from my job is actually seeing other people thrive and playing a role in that. Then I realized that I was actually doing a much better job myself because it was a natural filter, like in my career, as I've grown, it's no longer possible to have significant impact as one person.

A lot of times I find that if I take, if I just pause and say okay, like how can I lift up the people around me so that we all have impact that we have?

That's been like a game changer for me. It's it's made me more successful. It's made me happier as well. Cause it turns out that's a much, much better way to live.

John: Yeah. And when I think think back to, folks coming in from college oftentimes they've spent their entire life just doing things on their own.

I have to write this test. I have to go do this project or this assignment and every now and then there's some group project or something that's thrown in there. But but it's, but now all of a sudden they're in this different world and they working on their own because they think I need to get credit for this thing that I'm working on. So I can get the "A" and I can't do that with other because then could be their work or something. And so overcoming that, it's you said it took you 10 years to get to that stage.

Joe: I think there's a certain irony of being an individual contributor as a PM.

The irony is that you have to be really great at working with others, because everything that, as an individual contributor, like everything that you do is influence without authority, and so people have to want to work with you. And and so that's one aspect of the individual contributor career path.

I would not have been able to continue growing in my career if I hadn't learned that lesson of actually, this is really a team sport, I'm here to actually lift others up because it's actually, they're the ones who are largely going to do the work that I'm trying to make sense of here. And I've actually found that as I've shifted to a manager that has served me super well because it's actually possible to go down the manager path and just own your own team and not really have to work with others, but that can be limiting to.

John: Yeah. And so now you're in this role where your job really is to help grow your team and help those individuals succeed. In addition to the people that perhaps you talk less to, now that you're your manager person that you used to talk to in as an individual contributor person.

Joe: Yeah. Yeah. It's it's it's a really fun journey, it's kinda, I made the switch largely to be honest, because I realized that actually that was what brought me more joy than like the hardcore technical stuff.

As I enjoyed the technical stuff, but after 20 years or so going from memcopy similar types of like down in the weeds things, it was actually the helping the folks around me grow that was actually what was bringing me the most joy. And that's why I made the explicit switch.

John: Yeah. I remember some sage words from an old boss of mine was software engineering is a social science. That was Steven Sinofsky and that, that really stuck with me. Cause that's the hard stuff, in lots of years of observing Microsoft I have no doubt about our ability to execute on some technical thing. That's not hard it's executing on the right technical thing, that's the really hard part. And that's where all the squishy messy people's stuff shows up.

Joe: Absolutely Yeah. It's like how can you bring out the very best in everybody so that you more commonly do find the most important thing to work on.

John: So to wrap things up, I thought I'd just ask a couple of fun questions. Like what's something that you purchased or got, or acquired were given to you recently that brought you like an outsized amount of joy that you would have never predicted.

Joe: Yeah. I have to say, so I, I love to fish. Fishing is like always been a big thing. I grew up working on a farm and stuff, so I've always been outside and during COVID I bought a kayak specifically for fishing and I have friends who have boats I have a whitewater raft and other things

But the kayak has been just like this great source of joy because I can throw it on my car. I live 10 minutes from Puget Sound. Be on the water at 5 and still make it to work by 8:30 no problem. That has just been a really awesome.

John: So what's the Zen of fishing for you?

Joe: Yeah, it's it's really about observation for me. So I like to take in all of the signals, so like I fly fish a lot and It's like understanding what the fish are doing, understanding what insects are hatching, what temperature do the insects hatch at, that sort of stuff I think is it's really fascinating for me because I have to immerse myself in it. And I think leaving work, leaving, the stresses and challenges of being a parent and all that stuff for a period of time and immersing myself to just observe everything is that's the Zen.

John: So fishing is like a solo activity for you. It's not a ...

Joe: I have, I one good friend that we usually go together, but yeah, we it, there's not a lot of talking a lot.

John: And so you're out there, you're in your... what's a fishing kayak what's different from a regular kayak?

Joe: main difference is that they have pedals instead of using a paddle. And the reason for that is so that you can fish with your hands while you're moving.

John: Oh, so you can move forward by like my, like a bicycle pedal or, yeah. Interesting

Joe: They're a little bit stable also than a sea kayak, which, are very long and skinny, but also very tippy.

John: Very tippy.

Yeah.

Joe: Yeah

John: So how does that work. Is it like they're outriggers or is it ...

Joe: There are no outriggers, they're just bit wider.

John: And then what's a a story that you've shared with others and in particular, and what was it about that story that kind of made it compelling to you?

That story could be anything. It could be a book, could be a movie. It could be ...

Joe: A story that I have heard or a story that I have told?

John: Either one.

Joe: I think that. I have a significant recency bias with this. I love to read, I probably read two or three books a week. And I recently read a book called The Golden Spruce. It's a non-fiction book and it's about this huge Sitka Spruce in the Queen Charlotte Islands that has for reasons that nobody knows has golden needles.

And it's about, this seemingly deranged, perhaps not extremist who cuts down the tree. It's like a thousand year old tree. And what struck me about that story is that it talks a lot about the Haida native population. It talks a lot about logging environmental ism. And what struck me about the story that I think was really powerful was that It presented a really genuine, but diverse set of points of view in a really compelling way. Often I find like whether it's at work or in literature or documentary or whatever, that there's always some heavy bias, that's hard to unravel. And what I appreciated about that story was that there is no clear answer for why this tree existed or whether it should have been cut, whether cutting it down, actually achieved the objective, which was to highlight the hypocrisy of logging in the area and things like that because they were highlighting how wonderful this tree is while they were ravaging the rest of the Queen Charlotte Islands, basically.

What struck me about this story was the really diverse points of view in the sense of conflict that's unresolved, because I find that a lot of times stories want to have a resolution, I told you as an English major. There's always some resolution. In life there's a resolution.

And I always gravitate towards stories that are unresolved.

John: So the story left a lot of tension in the air? But at least it exposed you to the tension as opposed to just watch, sweep it under the rug...

Joe: Yeah. It's one of those books that I'm like, I still don't know how I feel about this. Who, do I believe in that the sort of antagonist point of view and that sort of thing. Yeah that one stays with me.

John: Wow. On that note I think I'd like to thank you for the conversation. This is a super fun for me and hopefully folks got something out of this as well.

Joe: Well thanks, That was super fun. I I appreciate you doing this and it's always fun to chat and get to know one another, on a more human and personal level than just talking about shop all time.

And we're actually in the same room doing this right now. And unlike, and one of the impetuses for this was just all the isolation that we all felt sitting in our bedrooms or wherever ...

totally

John: ...talking to each other.

Joe: Yeah. It's I have to remind myself how long the last couple of years have been ...

John: It's been a couple of years

Joe: Reconnecting in whatever form, whether it's through things like this or in person or whatever it's been really awesome.

Long overdue. Yeah. Yeah.

2022-05-06

I've started a podcast!🔗

A couple of decades too late, I've decided to start a podcast. I've had the good fortune to have met many interesting people in my life, and I wish I had written down or recorded more of what I learned from them. In the spirit the best time to start a podcast was 20 years ago and the next best time is right now, I'm going to try and make up for lost time.

For those arriving in the future, here's a link to the main page of my podcast. I need to find a place for it on this blog as well.

I started my experiment by interviewing people on my team. Here's my conversation with Austin Henley, former Assistant Professor at the University of Tennessee and the newest member of the PROSE (Program Synthesis) team at Microsoft.

Summary🔗

In this conversation, Austin and I discuss such topics as:

  • His recently discovered love of LEGO as an adult
  • How bouldering helps him strengthen his problem solving skills
  • The differences and similarities between working as a professor and as a Program Manager at Microsoft
  • The effectiveness of using a 'flipped classroom' teaching technique
  • How Visual Studio and Visual Basic 6 got him started in programming
  • How to use feedback loops in recommendation systems to generate superior results
  • What we can do to improve tools for software developers by better understanding what their intentions are

Transcript🔗

Click here for an interactive transcript.

John: Today on the podcast, we're going to be talking with Austin Henley, the newest member of our team. He just recently joined us a little over a month ago from the university of Tennessee, where he was a professor of computer science. At Microsoft, Austin's going to be working on the PROSE team, our program synthesis team.

And in this, conversation, we're going to be learning a bit more about Austin's background, what got him to come to Microsoft and some of the things that he finds particularly interesting, about his past life as a professor. So you got recently to number one on hacker news, with this post: what a $500,000 grant proposal looks like.

And that was for an NSF career grant. It was also this thing that kind of made it pretty likely that you were going to go get tenure as a professor, but yet you gave all of that up to come work here at Microsoft. Why did you do that?

Austin: Well, that's the $500,000 question. So, you know, there's some history here.

I had a couple of these NSF grants, the national science foundation it's who funds most of the basic research. And you're right. It was looking good for me to get tenure my faculty position. But for the last few years, there's just been this bug in my head about, not that this isn't actually my goal.

Even though it was what I set out to do, there was more to it. I'd already had a relationship with Microsoft before: I interned with them. I spent a summer, visiting, as a professor. And so it morphed over time that I wasn't happy doing what I was doing, this race to get tenure.

And so an opportunity arose. And I couldn't say no. I mean, it goes back to when I got into programming, like in fifth, sixth grade that I wanted to work at Microsoft. And so I couldn't pass up the opportunity. Right.

John: So you're always like the Microsoft fanboy, growing up

Austin: a little bit.

So I started with visual basic six. I used to have to pirate visual studio, 6.0, back in the day, this was probably around 2001. And every time my dad would find it on our home computer, he would delete it. Cause he didn't know what it was and it would take up, I don't know, 12MB. Yeah. It was huge.

And we had this tiny hard drive and so he would delete it. And so every couple of weeks I'd have to re-download it. So I got started and was a bit of a fan boy and that just kind of grew through college and grad school.

John: When you were applying for this grant and what kind of led you to come to Microsoft where they roughly the same things where they do different things, like, could you describe a little bit more about what that was?

Austin: Sure. So the research that I was doing at UT is all about developer tools. So they use like I'm on the human factors side. So I would observe developers, I'd put them in a lab. I would, watch them code, try to understand what their problems are, what their workflow is. And so the grant was actually about how can we build tools to help, not just students with learning, in coding task.

So, maybe they were trying to understand how for loops work. But how can we also apply that to professional developers? So there's a lot of learning that happens while you're actively coding. And so that's what my grant was about: can we help learning process through self explanation?

Kind of like rubber duck debugging. So Microsoft, in the PROSE team, they had a project that was aligned with this. They're working on an intelligent tutoring system, for like intro to CS and intro to data science students. And so it's a very much overlapped with what the team was doing.

John: So now that you've transitioned out of the academic life and into whatever corporate professional life that we have here, what's the biggest difference that you've seen so far, between these two worlds and also what's similar between these two worlds.

Austin: Yeah, there's actually a lot of similarities that I was kinda surprised about. The ability to just figure out what needs to be done, what needs to be done next, and then present that on the fly to diff very different audiences is something that I had to do a lot in academia. And now it's something that I do every day and different different meetings with different partner teams and management just trying to paint the picture of what we're doing and where we plan to go.

That's just constant. Something that's very different is just the pace. So the job is a lot more similar than I thought, but the pace is so radically. Whereas in, in my previous position, if I wanted to work on something, you kind of go off and you think about it for a couple of weeks maybe work with your graduate students to get a prototype over the next couple months and brainstorm and sketch out what's the design space for the problem.

And what's the solution space look like, do some related work and just kind of explore. But we do this. Daytime frame in the team or within a few days we sketch out these plans and figure out what to do. So the pace is the biggest one. What

John: do you think is like the thing that's most responsible for this difference in pace, like I think you, and I've spent some time talking about Tim Urban the wait, but why thing? And he likes talking about procrastination as a thing that he does a lot. And I think we all see that in ourselves and in his, the funny thing that he likes to bring up for that is for deadline based things this panic monster thing wakes up and and governs your life and drives you to complete it.

Do you feel that, you know, I don't want to try and put the words in your mouth, but do you think that it's kind of the difference between, being your own boss in academia versus now where you have lots of bosses, it seems. Is there more panic, monster driven work going on

Austin: here?

Yeah, for sure. And I'm trying to find the balance there because I think there's so much benefit to just having uninterrupted deep thought on a problem that you're working on. And so how to fit that in to my current schedule, which is just ridden with, deadlines and other people, depending on me.

And I, of course don't want to block other people, you know? So how can I work in those sessions of just, how can I get two full days to think about something without interruption while also not dropping the ball on all these other projects? Another thing that I think makes this different is that it's so much more collaborative of a workspace.

And academia, I had my, like a group of six students. I'd worked with them a couple of times a week, but for the most part, we're a bit isolated in how we work. But at Microsoft, I'm in meetings every day with different teams and each team, has I, I'm making promises about, I can help with this, or there's some expectation that they'll get me something.

And so that puts a lot of pressure to basic, basically have deadlines every week at the very least for showing up with deliverables.

John: Did you find that you collaborative much with other colleagues in academia or was it perhaps more competitive? In, in that space? Like how was that?

Austin: So I. Early on. I took a very personal stance that I wasn't going to be competitive with my research. And so I would put everything out there. I didn't treat it as a competition for resources, even though resources were very limited. I would put my ideas out there on my blog and on Twitter. And I did a little bit of collaborating, but again the pace and rhythm of those collaborations at the very most is once a week and more likely every other week.

And so the expectation, which is, was just not there to have like concrete deliverables constantly. But yeah, other people do take that to competitive routes. But I stayed at, stayed away from it. And it, I think it works out better to just be open and collaborative.

John: Right. Cause I think there's a lot of. A lot that we can certainly learn from from being kinder and gentler to each other. If that's the motivation, right? Other times certainly you've heard plenty of horror stories in graduate schools about, supervisors pitting students against each other on the same project and things like that as well.

How did you decide to become a professor in the first place and what kind of led you down that route?

Austin: Yeah at some point I, so I got into programming pretty, pretty early just by accident and fell in love with just coding and really what got me into it was like compilers and programming languages and.

I remember telling people at high school that I just wanted to be an expert in that space. I just want heads down, just know everything I can about coding. I'm like, I want to be the expert that people go to. And so when you start looking into what kind of jobs that can be academia came up a lot, having getting a PhD in a topic means that you were given some years of freedom to just really drill into a topic.

And so I didn't know anyone that had gone to grad school. And so I actually, I applied and was a master's student at first and my PhD advisor is probably what really sealed the deal for me, just working with him was a great experience. And I learned a lot from him. And that made me stick around for the PhD.

But there was still a question of, should I just go do industry research? Should I go be a software engineer? Should I go be a PM? Or should I go the faculty route? And I was in grad school for something like seven years. So I had plenty of time to think about this and bounced all around and did lots of internships.

I did, I think five internships. And so I was actually leaning towards the industry route. But at the very last minute, in my final year of graduate school, my mentors found out that I wasn't applying to faculty positions because I, and what I had really said was, I don't think I have a chance.

It's very competitive, very finite resources number of positions. And the interview process is not like industry positions. It really only happens once a year. So around October, November, everyone applies to the same person. University might hire one person each year, if that and it's, it takes months to, do a phone call.

Then you go out and do a full one or two day interview where people just drill you or grill you about your research. And you give a job talk as it's called in front of the entire department and they ask you hard questions. And some people like to be mean about it and try to trip you up in front of everybody and show you that they're smarter.

Yup. Yeah. They're smarter than you, right. They know your topic better. And so I didn't want to do that. That just sounds very stressful. But some great mentors of mine. They said, everyone's already expecting you to do this Austin. You've done great work. You can be successful if you try.

So after that I did some like soul seeking and decided to go that path. And it took, I think, three weeks of full-time effort for me just to do the applications to get my resume ready. Your statement, your research statement, teaching statement, all of the materials letters of reference. It took me about three weeks and I applied and it turns out I, I got a surprising number of interviews and the first interview was so stressful.

It really was stressful, but after that one, they were also easy. I like, I think my body just had so much stress that it gave up and stopped fighting it. And I was just like having fun, enjoying flying around the country, giving these presentations. And they went really well. And I got some good offers that I couldn't refuse.

So one of them was at the University of Tennessee. And I actually had a plane ticket to fly out to Microsoft for a software engineering position interview. And that was the hard one to say no to because I had accepted the faculty position

John: Did you know, where at Microsoft, that engineering position was for?

Austin: Yes. It would have been working on the Python team for VS Code.

Yep. I also had another the team that I interned with also invited me out. I believe it was back then. It was the Tools for Software Engineering team TSC. I think they've rebranded to one Engineering System now. Yeah.

John: So then you wound up in Tennessee and how long did you stay at Tennessee?

Austin: Three and a half years, but I grew up in the state of Tennessee and I went to graduate school in the state of Tennessee. So there's a lot of, there's a lot of ties here.

John: There's a lot of Tennessee in you.

Austin: Yeah. Too much. Probably.

John: So then when you became a professor what was like the folks who read, like how did you pick or choose what you worked on and was that told to you?

Was that something that you had to figure out for yourself and how did you decide on those topics?

Austin: Nobody trains you how to be a professor, really. Graduate school is supposed to train you to be an independent researcher, but there's a lot of other parts of the job, right.

That, that have nothing to do with research, such as teaching, administration, service, mentorship. When I showed up, I had no idea what to do. I was shown an empty office and it was a very nice office and they gave me the keys and that was it. There was no starting manual. No one tells you what to work on.

No one tells you how to spend your time. And that was, it was probably what I spent my first semester on is just learning where to put my time. And then I evolved it every semester after that. And yeah your point about, where do I, what research problems do I work on? It's really hard too, because some people continue their dissertation research, but there's also some expectancy that you're going to grow beyond.

And be more independent. So breaking away from your prior work and your research advisor from graduate school. And so I'll tell you what I did. This was I'm very deadline driven. My first week there was a deadline for a grant, a national science foundation grant that you can only apply for in your first two years.

And so I was like, okay, this is like my one opportunity to get a grant. And like my first two years it's slightly less competitive. And I can't pass up my first opportunity to apply for it because after like your second year, you can't get it anymore. And so I was like, well, I guess I'll just do nothing, but write this grant proposal.

And I actually started it the week before I started the position. But you know what you have to have a topic to write on. And so I. Just turned off my phone for a week or two. I didn't go, I didn't leave my apartment. And I just thought about what is the coolest tool that I can dream up and I'm going to write my grant proposal on that.

That's what I like to do. I like to work on like just weird ideas for developer tools. And that's what I did and I wrote the proposal and it was funded.

John: Oh, fantastic. And then, so you started working on that you attracted some graduate students and you also had to teach there too, right? Was that something that you were just told you're teaching this class? Or how did that work?

Austin: Yeah, so my department was there. They're very reasonable. And so they worked with me to find what classes are good matches, but also the department needs. And so they brought me in knowing that they have a huge need for software engineering. So like introduction to software engineering, junior, senior level standard class that every, almost every university has.

That really just teaches like a a development process. So they already know how to code but how to work in a team for a group project and build an application. And yeah, so that I underestimated greatly how much time preparing for that takes each lecture would take me. I am, because I would track it something like eight to 12 hours per lecture.

And that's me lecturing for about like 40 minutes. And then I would try to weave in like an activity each class so that my students, don't go to sleep. I get tired of hearing myself talk too, so I don't want to put them through the misery. And yeah, so two days a week I did nothing but make lectures.

The great thing is that I use those lectures every fall after that.

John: Yeah, there was this huge like startup cost and then it started paying dividends afterwards. Yeah. So, do you miss the teaching part of it?

Austin: I really enjoyed the teaching. My classes kept getting bigger and bigger. And with COVID everything went online and teaching online I don't miss it all. I do miss teaching, but it's really the small classes. So once you get beyond 30 students, it's, it doesn't feel like it's teaching anymore. It feels like I'm talking and they're eyes staring back at me. So I was constantly struggling with how to make a large classroom of 60 or 120 students more interactive and I never succeeded there.

But I do, I love teaching the smaller classes and being able to interact with the students. No.

John: Did you take inspiration from the big, like performance art classes, like CS 50 and things like that? Or did you try that? Did you try pulling that off?

Austin: I would try to find different pieces that I, because like I tried doing a flipped classroom for instance, where me and the TAs it's a flipped classroom is where you do the homework in the classroom and they go off on their own and watch the lecture or read like some reading assignment outside of the class, because, and there's some benefit there where they're getting the one-on-one interaction from the instructor or one dominion erection on what's actually hard for them, which is Hey I'm trying to code this.

I can't figure out why I'm getting this error instead of sending them home to do that. And then I can't help them figure out that Python bug that they have. So I tried things like flipped classrooms.

John: How did that work? Was that effective? Cause I, only know this kind of academically when I saw Sal Khan of Khan academy fame talk about this in some thing that he did many years ago with I think it was public schools in the valley where he was living at the time tried this, they would watch Khan academy videos and that was the homework and then the schoolwork or the homework was done inside of the classroom.

Was that effective? Did that work well?

Austin: I think it really depends on the classrooms set up and I didn't have experience doing it so I may not have done it very well. So something that, the immediate thing that I noticed is that for some students it works great. They have no problem asking for help.

They have no problem asking for help in front of other students. There is a large population of the students that I would have my classes that maybe they found it like stressful to have to perform in class. And they see their peers around them like finishing the first step of the project, but they're still stuck on it and they don't want to ask for help because it calls attention to them that there.

So I've read about some ways to try to get around that. I think you can break it up into like small groups and get them a bit scattered in, in the class. But that's what I noticed that a lot of students didn't want to put themselves out there. They didn't want to engage so much.

So what I started doing was just small pieces. Instead of doing your entire homework and class, let's get you started. Let's get you first pass that first barrier. And I would do live coding. So I think with me doing live coding and I'm not very great at live coding, so I would make lots of mistakes and they would see that, and that would help them at least get past the initial barriers.

John: It's amazing like how much how effective that can be, when, because most of the time you see the finished product, right? You see the commit in the repo. You don't see all this work that led to the commit. And one thing like, like Jeremy Howard of fast AI fame, he actually livestreamed a lot of his creation of the fast AI library.

And he does this crazy programming style where he writes all the code and Jupyter notebooks and. But that was actually quite good for his class. So his class was turn, could tune in to watch him code up the library that they were using. And he got a lot of very helpful feedback from seeing that as well.

And I always wondered when I see on YouTube there's certainly a set of folks that live stream coding and it is cool at the same time where you can see some of these things and you just see how someone that's really experienced goes about it. And you'll see that, oh my God, they make the same mistakes I make.

At the same time they're Googling for things and that's okay. And things like that. So let's go back a little bit to Visual Studio and and being a child in that kind of childlike wonder of computers what was there like a problem that kind of attracted you to this thing?

Or was there something that you wanted to create that kind of drove you when you when you were younger?

Austin: Yes. Yes, there were, there was something that drove me towards it. So I think it was in a fifth grade, maybe the summer between fifth and sixth grade, I discovered how to use the internet basically. And we had a home computer with Windows 98 on it.

And I was into video games and anime as many, young kids are. And I found these communities online of people that were probably in high school making these fan webpages for their favorite video game or their favorite TV show and with they had forums. And I S so I started engaging in these forums and realized that these people making these websites weren't that much older than me and they weren't high quality websites at all.

Like they. It was before blogging was really a thing. They would just put their ideas out there. They would put tutorials out there and fan art and just discussion boards. And I, as every kid, when you see something you like, you want to try to emulate it or create it yourself. And so I was like, okay, if another kid can make a website I'm going to learn how to make a website.

And that's how I got started with, I think it was like geo cities for hosting and trying to learn HTML. And I actually had a hard time learning, even HTML. I thought it was so cumbersome to write HTML. And that's where I really got sucked into computers is I wanted to make my own version of HTML my own markup language.

And you can probably see where I'm going next. I found some people that had enough knowledge to tell me, oh, you have to write a compiler to do that. I had no idea what a compiler was. I'd never coded outside of HTML, but I found a book at the library on compilers and I couldn't make no sense of it. I not for many years remember which book

It was the dragon book.

Of course. Yes. Yeah.

I believe the second edition.

John: And how old were you at this time?

Austin: Maybe 12. 12. Okay.

John: Yeah. Yeah. Dragon book of 12. That'd be pretty challenging.

Austin: Yeah. I mean, it's challenging for a college student. Right? But it taught me that I need to learn like a procedural programming language, something.

So I, I first started with basic my dad who I believe had just recently graduated from college, had taking, taken a basic programming course in college and he had the textbooks still and he gave it to me. So I think it was like Q basic or maybe GW basic. I don't remember which. And so I spent maybe a couple of months on that before finding Visual Basic 6

John: Did that change your life? Like what was that like to go from .BAS files? Probably in a text editor and all that, right. To like this visual thing.

Austin: Okay. Yeah. So, it was overwhelming at first for sure. But Visual, Basic 6 I think is what really sucked me in with the form editor.

I still miss that form editor to this day. Because I could just whip up a simple, like something that just gets the job done and isn't a command line. A mystery black box but this like visual application that I can hand off to anyone that's not technical and be like here, this solves your problem.

It was amazing. And then years later I got into C plus, which I really struggled with. And then later C# probably around the time I got into college. But I've always been in that Visual Studio ecosystem.

John: And it seems so sad that, back in the nineties, we certainly call these things, the RAD environments Rapid Application Development environments and we don't really have anything like that today.

It seems like, there's just so much complexity in almost anything that you do. Oh yeah. You can build a UI. Just learn, React, like as if that was just something that you just knock off in an afternoon while it was just some Java script and some CSS is fine. You'll learn that stuff too.

And it just seems like the concept count for doing something. Like putting a button on there and making it do something it's just enormous today with the stacks that we're using, as opposed to Visual Basic, it's a thing. You drop a button on it. There's a thing that looks like a button that you drag and drop onto the form and you double click on it and off you go.

And it seems yeah, we're so far away from that today. And that, worry about that. Like what so now that you're into doing kind of dev tools and making people more productive, what do you feel is the kind of the big problem here? What's the next kind of moonshot thing that we need to try and do in in, in tooling?

Is it making the professional programmers more productive? Is it making the students learn things faster or better? What do you feel? It is.

Austin: Oh, that's another one of these big questions. So something that I'll some of the research that I was doing. And as it related to the grants that I know we talked about earlier is that so much knowledge is stuck in different programmers heads and yet to complete a task, you need that information.

So you need, and I'll give you an example. Let's say you're a programmer working on a project at Microsoft, and you're making some changes that ideally you would need to know the rationale as to why that portion of code was designed the way it was in the first place. And also what are the cascading effects of making changes to that location?

Right. And so you'll do things like you'll look at what code interacts with that code or changing who calls that, that function or interacts with that class upstream and downstream. You'll you might even look at the code history and go back in time and look at what it used to, but there's still so much information that you're not getting out.

And so what that, that last project of mine was getting at is how can we facilitate the knowledge transfer between people in a more efficient way. You can't always just walk over to someone's desk and ask them a question, even though that's a very common thing to do. It's also very common to go back and look at old code reviews, but it's tedious.

And so I wanted to find out how tools can help us not replace it, but help us get that information or archive it and share it. And detect when someone needs it, because you don't always know you don't know what you don't know. And so if you're making a change, you might not realize the design decisions that went into it in the first place until it's too late.

John: How do you think you could solve that problem then? Today with a magic wand, how would you imagine you can marshal whatever resources you need to, to marshal to make this thing happen? What would you need to do? Like or what would you do?

Austin: Yeah, so I was looking at it from three different perspectives.

I was looking at it for students, you know, they're just trying to understand concepts. But we can't look into their head and understand what their current mental model of that concept or of their code is, but we can look for the symptoms. And an example was something that we call code smell detection. If they're writing code, it works.

But maybe it just looks funny. It smells bad. Or it, it often is indicative of a bug then we could potentially intervene. And so that's, this is what my first research project as a professor was, if we can detect that they're writing code that has some code smell, say a loop that will only ever iterate zero times or one times .

That doesn't make sense. It'll compile, it'll run, but no, like you shouldn't be using a loop for that. Instead of showing them a compiler warning or a compiler error, use that as an opportunity to pull information out of their head, ask them a question. And so we built a system that would say, how many times do you think this loop will iterate?

And we didn't want to disrupt them while they're thinking or working. And so we use some prior work on these negotiable interruptions as they're called to try to get them, to give us the answer. And so if they enter an answer, that's something like, I think it'll run 10 to 100 times.

So it was like a multiple choice thing. It was like a click thing.

Yeah, we we played with different ways of multiple choice or like a range, so they could enter two numbers X to Y. So for the loop, it was X to Y and then we could use some static analysis just to see if it's possible. And if it's not possible, then that's an opportunity to help them again, don't just yell at them and tell them what the answer is, but rather, sorry, my dog.

Okay. But yeah, so that's an opportunity to try to help them overcome that misconception. And so that was my very first project at UT and later on for the career proposal, which is the, this big, like before you go up for tenure grant proposal, I wanted to generalize it. So how can we take that same idea of this inquisitive feedback loop, asking someone a question and trying to get that information and help them use it more efficiently.

How can we apply it to professionals? And so I looked at onboarding and code documentation. So if they're writing code that has some smells or is indicative of being complicated or changing a region of code that might correspond with a prior bugs, meaning that there's been a lot of code churn, there's been a lot of code changes over time in that area.

Maybe we should ask them some questions, some structured questions to try to tease out the design rationale and then document it such that it can be shared later for onboarding purposes or code review purposes. So I don't think that's a solution, but that's how I was going about this problem of trying to help people share the knowledge that's stuck away in our heads.

John: And so when that would happen, then what would be the artifact that would capture these insights that they would then in turn like that, that somebody else could glean from?

Austin: Yeah, I didn't get to that. So like the operationalization of what do we do with that information?

What form does it go to? That, that was an open question that I didn't get to. And it's a hard one because you don't just, we already have so much documentation that all these documents these documents that we share within our teams or between teams that after a week becomes outdated and then we never use it again, but it's still floating around in our One Drive. Right. So that's an open question.

John: The good live streamers, that code, they're already doing something like this, right. They're trying to explain or vocalize what they're doing at the time that they're doing to try and make the stream somewhat interesting. Because otherwise, yeah, it could be pretty boring.

You're just seeing so many type stuff on, on, on a screen. And so there might be something there but again, you're almost like relying on the human to vocalize it, but what might be really interesting here is if we captured a time series of them just typing stuff and their interactions, there might be a way to infer things that they're doing by watching the time series of the changes to the code.

Again, right. We're just in crazy, speculation mode here. But it feels like there's it for me.

Austin: Yeah, there's information there. And there's also, so what my grant was, or my the research projects we're trying to build on was this idea that you're getting at, which I believe psychologists called monitoring or self-regulation.

So your kind of just having attention to what you were doing, you're trying to be intentional and think it through it, think through it step-by-step and that process of self explanation has shown to have lots of benefits. So if you can get a student to self, explain what they're working on, like a math formula or programming, they have better outcomes than a student who you don't ask to do that.

And the explanation itself often isn't that useful, but the process of doing it. Gets them better outcomes in some way, it gets complicated. But it's the process of doing that self explanation. And so I was trying to leverage semi-intelligent software tools to help the person do that, but also to help them know when and where to do it.

Because when you're working in code, you can put your attention in all sorts of different places. And so if we can have a system that calls out the high value, low effort regions, where you should put your attention, then it can that was the wind that we were going for. I see. Like, but what you're saying, you were saying that we should use their process to potentially help others as well.

Right, right.

John: Yes, exactly. Yeah. And I, and it also feels Again, back to what kind of latent information is there that we're just not taking advantage of. And this kind of idea about vocalizing verbalizing, or you said self regulating behavior. I remember the story and I've been trying to fact check it online and it doesn't exist.

So take it with appropriate grain of salt. This was verbally told to me by someone else, but they called it the Stanford Teddy bear experiment. And what it was, legend has it that outside of the TA office at Stanford computer science sat in a chair with a Teddy bear. And the idea was, or the rule was that you couldn't go ask a question to a TA until you first asked. the Teddy bear. What they found was the volume of questions that would come in from students. And the quality of the questions would number one increase, but also that they would just go away. They would start talking to Teddy bear and about a minute or two into it, they go, oh, and then they walk away.

That's amazing.

Austin: That's amazing. Yeah. That's amazing. I need to read about that.

John: I'm trying to find this, believe me, I've wanted someone to write this down, but I still haven't been able to find, but someone told me this verbally and I just can't remember who.

Austin: Yeah. I mean, and it makes sense. I had a lot of students that when they would come ask for help, it was obvious that they didn't know how to articulate their question.

And so if you can help them prep just before going into those office hours to help them articulate their question. And oftentimes that just requires. Trying to ask a question. So having that Teddy bear outside I, I can see that being a lot of help and I think it could help now with some of these meetings that I go into I try to do this where I'll write down the questions that I want to ask when I go into the meeting.

And it, a lot of times I realized that I don't know how to ask the question and I, it's much more fuzzy than I thought it was going to be. And so I think that's something that we could all probably practice more.

John: Yeah. I've also noticed that when I've been tempted to go ask an expert about some thing that I find that as I'm writing the email, I never send it right. Because I figure out the answer before I ever have to hit the send button.

And so it's almost the same thing, right? There's this thing. The act of just explaining the problem to someone else, even if the someone else has absolutely nothing like my email client can give the answer. But the other thing nowadays, I think with things like Codex and the thing in copilot our kind of commercial implementation of Codex, the Open AI thing is it generates code based on prompts.

And so perhaps, this idea of just prompting, right? Like explaining to the system, we're trying to describe the thing that you're trying to do to the system and having the system give you code, even if the code isn't quite right, or but it could help just that, that, that feedback loop.

Austin: Yeah. And that's what we're working on now. And in the PROSE team is that feedback loop of what we call programming by example. So you show us some input and output example and we try to generate code and we try to help that feedback loop of trying to solve that problem, whether it be for Excel or visual studio in different domains.

John: Yeah. I think one of the challenges that the PROSE had in the past and that y'all are like, on the path to fixing these days is that PROSE would never in the past and in features like FlashFill for Excel, it would just do the thing for you. It wouldn't tell you how it did it. You would just look at it as huh, it did it. I think it seems right. But you wouldn't know how it did that. So I think that since since the team has put their energy into having it, so that PROSE would synthesize human readable or understandable code, in a programming language, say like Python that all of a sudden it got much better right now you can look at that and go, oh, I see what you're doing.

And or if nothing else, it would give you more confidence in the answer.

Austin: Okay. Yeah. That's been a big research area for any recommendation system is how do you convey some level of meaning of why are you, why are, why is the user seeing what they're seeing? Because there is some amount of trust required for a user to understand what's being recommended to them.

And this could be anything like recommending movies or music. And you could see this on a, like a store it'll say based on what you've bought previously or based on what other people like you have have purchased. We're going to recommend this for you, right? So we're actively doing some studies now with different pros integrations on helping the user understand why they're seeing those suggestions, but also making them good suggestions.

Like you mentioned, people care about readability. So if we generate a formula for somebody, it needs to be readable. It's pretty easy to generate really bad code but making it performance or readable based on their intentions that's a really hard thing and that's what we're working on now.

John: Very cool. Very cool. So again, like maybe just shifting gears a little bit else to, to other things as you're like going through your career academically, could you share, like people that you looked up to or influences in your life that kind of changed or shaped the path that you took to get to where you are today?

Austin: Sure, definitely. So I've had some great mentors and collaborators along the way. So the first one was my PhD advisor. So Scott Fleming at the university of Memphis he introduced me to this idea of human computer interaction. I had no idea what that was. I used to be a compilers nerd, and that's what I cared about was programming language design and how compilers work.

And I had no empathy for users which I shouldn't admit because that's the opposite of what my job is now. I thought that if you have a problem with software it's user error and it's because you don't know enough, you didn't try. Right. You just need to go get smarter. And he, I remember it.

He told me to go read this book, The Design of Everyday Things, and that's still my favorite like book related to work at all, is this design of everyday things. I had all my students read it and it just immediately clicked for me that if I have a problem with software or hardware, it's not me.

Like we shouldn't blame the user. It's a design fail. Why did we not prevent that error in the first place? Why did we not understand our user in the first place so that we could make a better experience for them? So Scott Fleming was my first really big influence in my career. Someone that I really aspired to follow was robbed, align at Microsoft research.

So when I got into grad school, all the best papers were coming out of Microsoft research. And so that's that reinforced this idea of I'm. I need to get some experience at Microsoft. And so people like robbed at line Chris bird Zimmerman Nachi Andy bagel these were the people that I was reading their papers, and I was trying to meet at conferences.

I had no connections with them, but I would like you would stack them in a hallway. Yeah. And had the most awkward, like, hello, I'm a second year grad student. I love your papers. And then I run away in shame.

Did you have a chance to talk to Robson?

Yes. I spoke with him yesterday. And yeah, and he's great.

He we set up a re recurring meeting time, just so that I can hear more about what he's working on and it overall overlaps a lot with what I'm doing now. So it's a small world. Yeah. Rosalind,

John: my favorite people to talk to here at the company. And we always have these fun conversations some way when we get a chance to meet up last time was in person.

But of course we haven't been doing much of that recently, so, yeah. Cool. So you do some other things like, outside of work and things like that as well. I know in a previous chat, you mentioned that you also do rock climbing and I was wondering, how does rock climbing does it have any relationship to engineering and the mindset that you bring to that?

Or like, how do you find these two things overlap.

Austin: Yeah. So that's a funny question to ask about the overlap of, I don't know if I've thought about it. So I got into climbing in college because my university had a brand new climbing gym. And I had never thought about climbing. I'm terrified of Heights.

And I went and I tried it with some friends and I got up four or five feet off the ground and said, I'm never doing this again. It just, but I kept torturing myself. I would go back every couple of weeks and try and get a little bit higher and everybody would make fun of me cause. It was not a difficult like route as we call them.

It was very easy. I was just a chicken and couldn't overcome my fears, ended up getting a job there a year later. And now I ha I built a client home climbing gym in my bonus room. My, my entire bonus room is a climate wall area. And so the overlap, so it's funny that you ask.

So I do a type of climbing called bouldering, which is no rope. You normally if you're doing it outside you'll bring these crash pads with you these small foam pads that will prevent you from getting too hurt. And you don't go very high. 10, 15 feet usually is the limit.

But we call them problems. They're bouldering problems. And so it's a sequence of moves and they can get quite difficult, but it's all about your body positioning. At first you think it's all about like just raw strength, but you'll find that rock climbers, usually aren't, very buff people.

They look like they're quite skinny usually. They're not the most muscular people in, and you'll quickly find that it really is this, I guess you could say physics problem of trying to understand how the twist and turn your body and make these moves. So, you learn a lot of lessons very quickly about how to stand on your toe on your toes how to balance your body, because if you're pulling in one direction, are you going to swing off the wall?

So how to stabilize your body and make these nice clean movements as you're climbing and to avoid injury as well. And yeah, so it's like I said, we call them problems and it's a lot of fun. You sit there and you really try to visualize your body going up these sequence of movements. It's a lot of fun. Yeah.

John: So there's like planning and stuff that you have to do ahead of time as well. Do you study these things and plan your route?

Austin: Yes. Yeah. So once you start getting into the area, the difficulties that are challenging for yourself and there's grading systems for this. And so everybody has knows where they're at that climbs you, we call this the beta which is the sequence of moves that you have to make, and not everyone will have the same beta for the same problem.

And a taller person might climate very differently than I will. And so you can look for inspiration from others, but your own style and your own strengths really change how you climb. And yeah, so you'll plan it out. You'll work on it. Step-by-step so you don't necessarily, it's not like you just jump on and say, I'm going to complete this, you're you'll say, okay, I'm going to work on the bottom region. I'm going to figure out how to just do the first two or three moves in the sequence, and then maybe you'll work on the next two or three and then the next two or three, and then you say, okay, now I have to link them together. And that's a whole nother step of linking those sequences of moves together.

And it changes once you do that. And so you might have to go back and start should my left hand be there or should my right hand be there. And in what order

John: so do you find that you have like when you're at the limits of your ability. At that time.

And I guess you fall and stuff. Like how do you overcome the thing there's this problem you're trying to solve, but you can't quite do it. Like how do you think about different alternatives or options and things for solving?

Austin: Yeah. So there's a few strategies.

The most naive, maybe not the best one is you just need some caffeine. And taking a break. So just taking a break from climbing for a couple of days, maybe even a week, and just getting away from the problem, you might think about it a little bit, like when I'm taking a shower or something and you go, you come back to it for.

You climb so differently in that problem, or, that one move, cause there's usually a crux, the hardest part of the problem that was so challenging for you, you'll come back a week later and you're just like, you didn't even have to think about it. You just, you move past it. And sometimes it's just it just hits you.

And I don't know why but you'll get hung up on those moves. And then a week later they're not even an a problem. And then the other way is just to watch other people climate. So try to understand how they do it. And it's the funniest thing. Like when I go to a climbing gym, I always watch the little kids because the kids climbing are often way better climbers than I am.

Even they make it look so effortless. They're like strength to weight ratio is amazing and I will just go mimic them. So sometimes you're overthinking the problem. And

John: So it's interesting, right? Bringing it back to kind of computer science things, is that oftentimes you have those shower moments, you're sitting there, you're away from the problem. And all of a sudden the solution is just appears to you. Because you're, you've been processing it in the background, but I think that the difference here is this physicality thing, right? So maybe, maybe there's a, something that you're not flexible enough.

You're not like there's some, there's something that's keeping you from accomplishing the goal.

Austin: And so that's one of the things that I enjoy about the climbing is that If I'm not getting better, technically I can get better through cardio or flexibility or different strength exercises.

And so even when I feel like I'm plateauing, there's other areas that I can still improve it feels like I have more chances to get better at it. Sometimes with coding, it's like, I don't know the alternative to get around this problem, like how to get better. Cause it's, I just need to get better at programming.

Right. But maybe I need to treat it the same way and work on, those tangental aspects that, that have some overlap or perhaps

John: this same kind of verbalization thing that you, that we were just talking about earlier. Is that like, when you're observing someone in a gym, I'm guessing they're not talking to themselves, well, I'm going to go do this thing and I'm going to grab this thing over here and go swing around and do this other thing.

You just observe it. Yeah, it would also be interesting, if you could get inside their head or maybe someone that's, I guess there must be people that live stream or record, or do YouTube videos of all these things where they talk it through. But those kinds of things like around here, I've been I built myself over the holidays and F1 driving simulator and I've been trying to get faster and what's challenging is when you're watching good streamers do this stuff, it's really not obvious what they're doing.

And they're not really talking you through the thought process. So again, just getting that extra little bit of insight I think you can do a using all of them. It looks like they're starting to brake here, but you have to almost go frame by frame through the video to see what they're doing.

That's different than what you're doing.

Austin: Yeah, in research, we do a think aloud or a talk about study, where we ask people to verbalize what they're thinking as they're doing like a programming problem. Similar to a programming interview, but what they are saying doesn't always match. So you also have to be careful.

So what people will verbalize them, what they think is important might be different. They don't do it intentionally, it's the subconscious thing. Do you

John: have a good example of some, like a case where you ran into that, that you could share?

Austin: Well, I'll say that when people really are struggling when their cognitive load is high, the last thing that they want to care about usually is trying to communicate with you.

Or to articulate their weight words just right. And you can practice this. So do like a math problem in your head and try to talk out loud. And I mean, like a really hard problem I don't know, start doing some three or four digit addition and keep going. So add three numbers to three numbers and then three numbers and three numbers, and keep that in your head.

While also trying to verbalize your strategy. You'll probably be very good at verbalizing the numbers, but you're not going to be able to tell me your strategy for, okay. I'm trying to think. Remember it in blocks of two. I don't want individual digits in my head. I want to do start grouping them in twos or threes.

I don't think you'll be able to tell me that. So it, there's also some issues and this like psychologists and sociologists will use this method in lab studies, but it will change people's behavior when you ask them to verbalize this information, because now they're doing more regulation of their thought process. So it can still be useful, but also you have to remember that you're changing their behavior by asking them to do it's like a Heisenberg

John: problem, right?

It's the, there, because they're, you're now overloading their cognitive abilities by asking them to do this other high level thing on top of this high level thing that that they might do something completely differently. Had they then had they not been verbalizing at the same time.

Very interesting. Yeah.

Austin: That's a good thing. I had an engineer once get really mad at me because I kept asking him, please continue thinking out loud. I had to remind him every couple minutes of going after going silent. And he got visibly angry during the loud. Yeah. Yeah. Yeah.

John: Wow. Very interesting.

So we're approaching the end of our time here. Let's let's wrap things up, like with what is what's something recently that that you have purchased that has given you a disproportionate amount of joy? Doesn't have to be something you bought just like anything, right?

Like something you've got some object or thing that has given you more choice than you would've expected it to have given you

Austin: So now that I'm an employee at Microsoft, I get the perks pricks plus benefit. And I saw on Twitter, someone say that you can get Lego reimbursed, like Lego, the kids toy of building models. And so I have actually been buying multiple times. I've been buying and that they now let go now makes these adult models where they're more complicated and they're nostalgic.

And so I, for example, I just built the original Nintendo entertaining. System the NDS from the eighties in Lego form. And it even came with a little TV. It came with the super Mario cartridge. And it works like you can put the cartridges in it. The TV actually has a carousel built into it, so you can crank it and you can watch Mario jump and in the level moves.

And

John: That's what I bought her or was that, is that like Lego or something that's

Austin: making it is a mechanical Lego on, so it's like a conveyor belt. And as you are cranking it, it turns the conveyor belt. And so it's a continuous, Mario level made up of all these little, very small tiles, and so the TV is the casing around the conveyor by so that you can only see the view into a small region. And it's like the, it looks like the first Mario level. So I did that a week ago and that was some great, just thinking time where I can mindlessly follow the instructions, building the Lego, find that enjoyable and either just sit there and absolute silence and like meditate on my day or what I want to work on next, or listen to a podcast.

And so I think that's my new, like favorite after work, like cool-down activity is doing Lego as an adult.

John: That's awesome. Yeah. One of these times when you get out here, there's there's brick con in in Seattle. That happens every year. I think it's in the fall. And so you can just walk in and there's you see all the Lego nerds or there's lots of ligaments and the things that they've built and there's some absolutely amazing and fantastic things. Remind me to share with you

some of my Brick Con pictures I've taken over the years. Cause I think there's this. Yeah. I think he really enjoyed that as well.

Austin: Does that mean you build Lego?

John: Certainly. Yes. Like we've spent a lot of time as a kid, I spent a lot of time building Lego and I've got kids here.

They've got Lego sets. We've certainly spent a lot of time going to build building Lego here. Something I don't do as often as I would like to I think that in some ways as a kid I remember that it was a lot of fun. Coming home from school, right? That was the thing, at lunchtime, get to go home.

I can feel my Legos. And and that was just, it was fun. Like it was like, we didn't have these specialized sets and stuff back then. It was just like the generic Lego things. And you just have to imagine how to build different things out of what you had available. And that was just, that was a lot of fun.

Just the kind of creativity from just going to building things and imagining what you could build out of these things. Yeah. Yeah. Yeah. So we were, the kind of like the sound of just rummaging through your box of Legos. That's a very, like a visceral sensation from childhood.

Austin: Oh yeah. Or one of your parents stepping on one of them.

John: Very cool. Well, thank you so much. This was a ton of fun. I certainly had a lot of fun talking about this and hope you you enjoyed this experience as

Austin: well. I did. Thank you so much for all the good questions.

John: I hope you enjoyed this inaugural episode of our podcast. Stay tuned for future episodes, where we're going to be interviewing other members of our team.

2022-02-17

I'm traveling and just got some time to myself to write down a bunch of stuff that I've learned listening to podcasts during over 1000 miles driving in a car.

Types of Procrastination🔗

From the wisdom of Tim Urban:

  • Type 1 procrastination is what you do when there is a deadline present. The panic monster will eventually awaken as when the deadline is imminent which will chase away the instant gratification monkey.

  • Type 2 procrastination is what you do when there isn't a deadline present. This type of procrastination is really, really bad because the panic monster will never awaken to snap you back to your senses. There really isn't a good solution to this kind of procrastination.

Types of Fun🔗

From the wisdom of Kate Courtney:

  • Type 1 fun is fun while you are doing it.
  • Type 2 fun is fun when you are finished doing it.
  • Type 3 fun is never fun

Tim Urban and long form writing🔗

The Tim Ferris interview with Tim Urban reveals a bit about how Tim (Urban) learns things. His long-form posts are him teaching himself what he's learned about a topic in hopes that it will help past Tim learn that topic more effectively. He really is just writing for himself:

Yeah. So, it’s pretty simple for me. First of all, I do this kind of weird thing where I assume that my audience is I picture like a stadium full of me. So, it’s narcistic fantasy. No, but it’s just – I’m just writing for – I’m writing the exact post that I would be thrilled to get.

Some tidbits:

  • On a scale of 1 to 10 where 10 is a world class expert, he starts at a 2 and continues learning until he gets to about a 5 or a 6. Then he writes the post. This sometimes takes him 3 hours, sometimes 3 days, sometimes 3 months(!)

  • He starts with general sources of information like Wikipedia on a topic. He then starts writing down questions (which can be expressed as concepts) and opens a new browser window for each question. Then he CTRL+clicks the Google results for each question. Then he sits down to read. He doesn't try to filter at this point - he just reads and starts forming an impression for each question.

Elon Musk is a Nerdy David Goggins🔗

After listening to back-to-back interviews and books during my long drives, it occurred to me that they are both intense, driven men that have overcome many obstacles during their lives. Different things for sure, but things that many would think are impossible. Some of the things that they did look stupid to outsiders, but I'm convinced that they share the same mindset when it comes to overcoming obstacles in their lives.

Here's an interesting example:

  • David Goggins ran a 24 hour race and completing 101 miles in ~18 hours with zero specific race preparation
  • Elon Musk started a rocket company and burned through nearly his entire PayPal fortune in doing so. A friend even sent him a video compilation of failed rocket launches to try and convince him to not start SpaceX in the first place.

While the impact of those things are quite different, I think the meta impact of those things are quite similar. They both could inspire others to realize that things are not as impossible as what others think - you are capable of more than what you are capable of. They will also inspire different kinds of people as well. I don't think anyone will use SpaceX as motivation to become a Navy SEAL any more than I think anyone will use Goggins' accomplishments to create the next SpaceX.

I highly recommend listening to the Audible version of Can't Hurt Me. In it, you'll hear the book as narrated by his writer, Adam Skolnick. But at key moments in the story, they will cut away to a "podcast format" where David gives additional commentary and context to the story at that point in time. It's a really effective format that I wish others would copy in their audiobooks.

2022-01-30

Joe Rogan, Robert Malone and the history of mRNA vaccines🔗

The current controversy-de-jour is Joe Rogan's interviews with vaccine skeptics. Today, Joe posted an Instagram post where he highlights his controversial interviews with Peter McCullough and Robert Malone.

It's worth listening to.

At first I had no idea who Robert Malone was, so tonight I went looking. It's highly likely that mRNA vaccines will recognized in this year's Nobel Prize awards. As background in the runup to the awards, this article in Nature does a great job of outlining the history of the development of mRNA vaccines. In this article I learned more about the development of mRNA vaccines and Robert Malone's role in it.

Apparently he was one of the first people to observe protein transcription following in vitro insertion of mRNA encapsulated inside of liposomes. Here's an excerpt of his graduate student lab notebook from the day of that observation:

Many other scientists are highlighted, including Katalin Karikó, who pioneered modifying mRNA using pseudouridine to suppress immunological response towards it. Pieter Cullis who made key contributions towards developing lipid nanoparticles that are an essential piece of technology that enables safe delivery of mRNA into human cells.

This diagram from the Nature review paper on lipid nanoparticles shows the parallel development of mRNA and lipid nanoparticle technology that resulted in the creation of the vaccines that we enjoy today, and the times when pseudouridine and lipid nanoparticles (and many other discoveries) entered the picture:

So what does all of this have to do with Joe Rogan again? He was accused of bringing contervailing opinions onto his show and giving them a platform. However, it's worth remembering that scientific progress does not progress in the idealized straight line seen in the diagram above. There are many twists and turns, and occasional reversals of course. Sometimes what was accepted as dogma at one point in the past winds up being known to be false today.

There's a story that Ingmar Hoerr tells about the time when he presented his early mouse data on direct injection of mRNA:

there was a Nobel prizewinner standing up in the first row saying, ‘This is completely shit what you’re telling us here — completely shit’.

In fact, this entire field was woefully underfunded precisely because it was outside mainstream science in the 2000s. What if funding for this field never happened?

The solution to people saying things that are contrary to the mainstream belief is not to censor them. Rather, we should combat that by, as Ben Thompson says, using more free speech. There's a lovely quote from Ben's article that I'm reproducing below that cites John Stuart Mill's On Liberty (from 1859):

If all mankind minus one, were of one opinion, and only one person were of the contrary opinion, mankind would be no more justified in silencing that one person, than he, if he had the power, would be justified in silencing mankind. Were an opinion a personal possession of no value except to the owner; if to be obstructed in the enjoyment of it were simply a private injury, it would make some difference whether the injury was inflicted only on a few persons or on many. But the peculiar evil of silencing the expression of an opinion is, that it is robbing the human race; posterity as well as the existing generation; those who dissent from the opinion, still more than those who hold it. If the opinion is right, they are deprived of the opportunity of exchanging error for truth: if wrong, they lose, what is almost as great a benefit, the clearer perception and livelier impression of truth, produced by its collision with error.

Joe admits as much in his Instagram post. He says he will try to ensure that other opinions are presented on his show in close proximity to the "controversial opinions". And even if he doesn't, Joe Rogan is hardly the only platform on the Internet.

2022-01-17

The Charisma Machine🔗

The Charisma Machine by Morgan Ames chronicles the failure of the One Laptop Per Child project. A telling quote from the book's website:

Drawing on a seven-month study of a model OLPC project in Paraguay, this book reveals that the laptops were not only frustrating to use, easy to break, and hard to repair, they were designed for “technically precocious boys”—idealized younger versions of the developers themselves—rather than the diverse range of children who might actually use them.

It's pretty clear that the leaders of the project fell in love with an idea. One where the solution they were creating matched the problems that the leaders personally had - a desire for a machine like OLPC when they were children. They assumed, incorrectly, that all children in developing countries had the same problem that their solution was designed for. Ames describes this as nostalgic design.

So instead of validating the problem, they went ahead and burned hundreds of millions of dollars in pursuit of a solution to a problem that apparently did not exist. Technology is not the solution to all of our problems, especially in education. But perhaps more interesting for those of us who work in product development is understanding why Negroponte was wrong? What could he have done differently in retrospect? Could the many failures encountered in the field have been discovered sooner? This quote captures Negroponte's hubris:

The days of pilot projects are over. When people say, ‘Well, we’d like to do three or four thousand in our country to see how it works.’ Screw you. Go to the back of the line and someone else will do it, and then when you figure out that this works, you can join as well.

Developing and deploying the solution as quickly as possible was clearly the end-goal here, not understanding the needs of the customer:

We’ll take tablets and drop them out of helicopters into villages that have no electricity and school, then go back a year later and see if the kids can read.

Could they have shipped a better product that fit the needs of their customers if they had only spent some time trying to understand their customers first? One can only imagine what might have been had they pursued a different path, guided by the real needs of their customers. A fantastic case study of what not to do when building a product.

2022-01-13

Multi-Agent Hide and Seek🔗

This is a great visual demonstration of the progress of learning in self- supervised learning via reinforcement learning. Adding human-like behaviors to the simulation make it far more engaging to me!

The accompanying Open AI blog post does a great job of annotating the different skills learned along the way. The reward function is simple:

Agents are given a team-based reward; hiders are given a reward of +1 if all hiders are hidden and -1 if any hider is seen by a seeker. Seekers are given the opposite reward, -1 if all hiders are hidden and +1 otherwise.

From this simple reward function, many strategies emerge and many surprising "hacks" where the agents exploit loopholes in the simulation. The section on Surprising Behaviors shows these clearly.

But given the extremely large number of episodes that lead to the behaviors, I wonder how they were able to see these behaviors?

UPDATE: I just discovered the Two Minute Papers channel on YouTube and they do a great job of summarizing the paper in this video. Subscribed.

2022-01-11

The Ultimate Home Videoconferencing Setup🔗

Nearly 2 years ago, COVID forced many of us learn a new skill: working online. During this time I've been in 1000s of online meetings. This post chronicles my journey to where I am today and what I've learned along the way in my journey to this:

At first, I started like many other folks by using the built-in camera on my laptop (a Surface Laptop 3). While that was serviceable, it wasn't particularly pleasant to use. The ergonomics are terrible unless you use a laptop stand and an external keyboard and mouse to elevate the display so that you're looking at it vs. down at it. I highly recommend the Roost stand if you're going to go this approach:

While that solves the problem of bad ergonomics, it doesn't solve the problem of poor quality video that is endemic to the webcams that are found in all laptops. Lots of people tried to upgrade to an external webcam at the start of working online. The most recommended webcam was a Logitech C920x which was nearly impossible to order and commanded a steep premiums from scalpers who quickly snapped up all available supply.

One

I've given this presentation quite a few times to folks around Microsoft and hey've found it helpful.

2022-01-10

The origins of yak shaving🔗

OK so I find it amusing that, in the process of hunting down a reference for the algorithm that Donald Knuth used to reward bugs in TAOCP, I found the earliest documented mention of the term yak shaving by Carlin Vieri:

From: "Jeremy H. Brown"
To: [email protected]
Subject: GSB: 5:30pm, 7ai playroom
Date: Fri, 11 Feb 2000 16:00:56 -0500 (EST)

"Yak shaving."  Our very own Carlin Vieri invented the term, and yet
it has not caught on within the lab.  This is a shame, because it
describes all too well what I find myself doing all too often.


You see, yak shaving is what you are doing when you're doing some
stupid, fiddly little task that bears no obvious relationship to what
you're supposed to be working on, but yet a chain of twelve causal
relations links what you're doing to the original meta-task.


Here's an example:


"I was working on my thesis and realized I needed a reference.  I'd
seen a post on comp.arch recently that cited a paper, so I fired up
gnus.  While I was searching the for the post, I came across another
post whose MIME encoding screwed up my ancient version of gnus, so I
stopped and downloaded the latest version of gnus.  


"Unfortunately, the new version of gnus didn't work with emacs 18, so
I downloaded and built emacs 20.  Of course, then I had to install
updated versions of a half-dozen other packages to keep other users
from hurting me.  When I finally tried to use the new gnus, it kept
crapping out on my old configuration.  And that's why I'm deep in the
gnus info pages and my .emacs file -- and yet it's all part of working
on my thesis."


And that, my friends, is yak shaving.  (Not that this particular
example happened to me recently or anything.)  


Another example of yak shaving might be "I need to ask  a
question about our group project.  <Person>'s hard to track down.  But
I know <person> will be at GSB tonight, so I'll go there too."  You
see, in this case GSB isn't socializing or slacking off, it's part of
your work -- hanging out there, you'll be connected to your project
via the process of yak shaving!


Now, having been introduced to the term and the concept of yak
shaving, and also having had it pointed out to you that attending GSB
is actually an important part of your research, you are morally bound
to come to GSB tonight and discuss the finer points of yak shaving,
and in particular the subtle distinctions between some forms of "yak
shaving" and "flimsy rationalizations".


****************** G I R L  S C O U T  B E N E F I T ******************
******************        5:30pm  7AI Playroom       ******************

2022-01-09

Donald Knuth as the patron saint of yak shaving🔗

Once upon a time, there was Donald Knuth. He's a well-known Professor, Computer Scientist, and Author of TAOCP, his life's work. Every day he worked on TAOCP as a digital recluse. While progress sometimes slowed down (3 volumes were completed by 1999), he does have a schedule for completing it.

His work was not un-noticed. TAOCP was named one of the twelve physical science books of the century in 1999 by American Scientist, alongside other notable works such as Dirac on quantum mechanics, Einstein on relativity, Mandelbrot on fractals. And so he kept working until one day in 2015, someone noticed that there were lots of other things that Knuth created along the way in support of his life's work.

The activities that led to the creation of these other things are sometimes called yak shaving:

Yak shaving refers to a task, that leads you to perform another related task and so on, and so on — all distracting you from your original goal. This is sometimes called “going down the rabbit hole.

The first time I heard of this term was at a RubyConf many years ago. Not surprisingly, Knuth's behavior was noticed by a member of the Ruby community: Florian Gilcher who penned this wonderful post on Knuth's yak shaving activities. Knuth's most notable accomplishment outside of TAOCP is TeX, a typesetting language that he invented to typeset TAOCP. So if you consider TeX the original yak shave, it led to all these other yak shaves that are lovingly chronicled in Florian's post:

  • Write your own programming language to implement TeX in (WEB)
  • Invent a new programming paradigm (literate programming - a predecessor to today's RMarkdown and Jupyter) for WEB
  • Invent a new algorithm for text layout
  • Design a font (Computer Modern)
  • Invent a new authoring tool for fonts (METAFONT)
  • Implement a new device-independent layout engine (DVI)

The HN post also had this wonderful comment from David Fuchs who told a story about even more yak shaving by Knuth(!):

This and more. The Sail version of TeX is mentioned, but everything else had a usable prototype, too: An early version of Metafont in Sail, an early version of Web (called Doc, though I don’t recall what the names of the Tangle and Weave equivalents were), and an early version of Computer Modern fonts in the early Metafont language. Though fully documented and available via anonymous FTP on the Arpanet, all of these prototypes were experimental proofs-of-concept and were completely discarded, along with the languages they implemented, with the “real” versions rewritten from scratch by Knuth (each first by hand on legal pads, then typed in and debugged as a whole.)

And you missed one more obscure but instructive example: To get the camera-ready copy for ACP Vol 2, Knuth purchased an Alphatype CRS phototypesetter. And, unhappy with the manufacturer’s firmware, he rewrote the 8080 code that drives the thing. Eight simultaneous levels of interrupts coming from various subsystem: the horizontal and vertical step-motors that you had to accelerate just right, while keeping synchronized with four S100 boards that generated slices of each character in realtime from a proprietary outline format, to be flashed into the lens just as it was passing the right spot on the photo paper. (Four identical boards, since you had to fill the second while the first one did its processing; times two for two characters that might overlap due to kerning). Oh, and you had to handle memory management of the font character data since there wasn’t enough RAM for even one job’s worth (see our joint paper).

Fun times. I did the driver on the mainframe side, so got to be there for the intense debugging sessions that used only the CRS’s 4x4 hex keypad and 12-character dot-matrix display.

Thanks for the DVI shout-out, btw.

I was fortunate enough to meet Knuth at useR 2016 when it was held on the campus of Stanford University. At the end of his session on literate programming, he rode away on a bicycle carrying an enormous stack of books precariously perched on one hand. Keep in mind he was 77 years old at the time! Sadly the only photo I have of Knuth from his talk was this one where, apparently, even the great Donald Knuth needs a bit of help to project from his computer!

That trip marked the first time that I rode a mountain bike. I couldn't have picked a better place to do this on the amazing Arastradero Preserve:

2022-01-07

GPT-3 and long form question answer by OpenAI🔗

Note to myself: today Dickie Bush reminded me of the Pixar Storytelling framework. I'm going to try rewriting this post that way to see if it's any better at telling the story.

Today's paper is WebGPT: Browser-assisted question-answering with human feedbac. When I think about thow we can build a bicycle for the mind, I've been wondering how machine learning can help and mused about as much while wondering if it is possible to create a Tim (Urban)-In-A-Box. More important than summarizing or finding information is getting you to read that summary.

There are a number of things that I learned in this paper. First is that there is a high quality dataset, ELI5 sourced from the popular sub-reddit that contains a list of questions and answers "explained like I am 5 years old".

I learned about crows

Pixar version🔗

Once upon a time, the Internet was a place where I would manually do research, often times on the toilet. Every day I would click on links, often times on social media, and read post after post after post, slowly building up a mental model of the things I am researching.

But if I rely on beer mode activities to drive my research, it's going to be pretty random. And since there isn't anything that tries to tie together the things that I have seen on a particular topic, my poor brain is left to fend for itself. So it sees input that looks something like:

  • GPT-3 paper
  • GPT-3 funny meme
  • Picture of cat
  • Shitpost from friend on Twitter
  • OMG COVID will kill us all
  • Spiderman spoiler
  • Introduction to Transformers
  • Transformers: Dark side of the mooon trailer
  • Attention is all you need paper

But one day, I thought: why don't I get my machines to go do the heavy beer mode activity of sifting through the Internet and return me something that I could read without distractions, preferrably written in a Tim Urban style? Just how far-fetched is this idea? Turns out, we're getting closer by the day.

On December 16, OpenAI announced WebGPT, which is a fine-tuned version of GPT-3 that answers open-ended questions by searching for and summarizing what it finds online. Unlike GPT-3, WebGPT augments what is already known in its language model with results obtained by searching the web using the Bing search API.

The results from the model are pretty interesting. When posed the question: "How do neural networks work?", WebGPT responded with:

In its most basic form, a neural network has two layers: an input layer and an output layer[3]. The output layer is the component of the neural network that makes predictions[3]. In a feedforward network, information flows through the network in the following way: patterns of information are fed into the network via the input units, which trigger the layers of hidden units, and these in turn arrive at the output units[1]. The network learns by a feedback process called backpropagation, which involves comparing the output a network produces with the output it was meant to produce, and using the difference between them to modify the weights of the connections between the units in the network, working from the output units through the hidden units to the input units, going backward[2][4]. Over time, backpropagation causes the network to learn, reducing the difference between actual and intended output to the point where the two exactly coincide, so the network figures things out exactly as it should[2].

This is pretty reasonable! It even generated references:

  1. How neural networks work - A simple introduction (www.explainthatstuff.com)
  2. How neural networks work - A simple introduction (www.explainthatstuff.com)
  3. How Do Neural Networks Really Work? | Nick McCullum (nickmccullum.com)
  4. How Do Neural Networks Really Work? | Nick McCullum (nickmccullum.com)

The linked paper provides a lot of additional details. First, the experimental environment (used by humans recruited via UpWork) is a text-based web browser. Here's a screenshot of what it looks like:

text based browser used by webgpt humans

The text based browser was designed to be used by human participants, and did a lot that a conventional browser does not typically do to make it easier to work with the text. It uses a number of existing libraries, including:

to transform the text into a format that was suitable to feed into the model. The browser was used to collect two additional pieces of data from the study participants: 1) demonstrations of humans using search to answer questions, and 2) comparisons between two model-generated answers to the same question.

The model itself was trained to answer questions from the ELI5 dataset that was gathered from questions and answers in the "Explain Like I'm Five" subreddit.

The results from this study are encouraging - the model's answers are preferred 56% of the time. When comparing answers to the highest-voted answer from the ELI5 dataset, the model's answers were preferred 69% of the time. This is not human-level performance yet, but it's getting there.

Because of this, Because of this,

Until finally the glorious day arrives when I can ask a question like:

Have Tim Urban write an explainer for What is Neuralink and why do we need it?

And I get an answer like this amazing 38,000 word explainer for my question. Or if I'm more in the mood for something more succinct:

Give me a Wikipedia summary for What is Neuralink and why do we need it?

But will it make any difference? Will it actually make people smarter? Think better? Will this insulate us from the endless distraction that is the internet?

Maybe what we need is an assistant that filters the web for us but in a way that doesn't reinforce existing biases? I should write about that some other time.

Brian Kernighan's memoir🔗

TIL that UNIX: A History and a Memoir was written by Brian Kernighan and available on Kindle as an e-book.