Enabling Great Teams to Solve Tough Problems in a Culture of Passion and Innovation

Transcript

Shane Hastie: Hey folks, before we get into today's podcast, I wanted to let you know about our upcoming QCon Software Development Conferences. We will be back in person at QCon London this April 4 to 6, and online with the QCon Plus, May 10 to 20. Join the world's most innovative senior software engineers across multiple domains in person and online as they share their real world implementation of emerging trends and practices. You can learn more about the events at qconferences.com. We hope to see you there.

Good day folks, this is Shane Hastie for the InfoQ Engineering Culture Podcast. Today I'm sitting down with Adam Sypniewski who is the CTO of Deepgram. Adam, welcome, thanks for taking the time to talk to us.

Adam Sypniewski: I'm very pleased to be here. Thank you so much for having me, Shane.

Shane Hastie: Probably a good starting point is who's Adam.

Introductions [00:56]

Adam Sypniewski: I actually have a non-computer science engineering background. I have a PhD in Experimental Dark Energy in Astrophysics. So, I spent my time in the mountains looking up at distant galaxies and trying to figure out what does their distribution in the sky say about the history and the evolution and future of the universe. And I applied a lot of AI tooling and machine learning to try to constrain that problem and understand our universe. Once I finished Academia, I definitely had such a passion for working in the computer science space. In fact, I got my QBasic Compiler back when I was 11 and bought my first Compaq desktop with paper route money when I was 13. And so I have always had that passion, and I see physics and I have the passion and I wanted to go into industry and continue moving forward in AI and machine learning driven space. I did that for a few years in DoD contracting for the United States government, and then I moved into Deepgram where we do speach recognition.

Shane Hastie: Tell us a little bit about Deepgram.

Adam Sypniewski: Deepgram, we are a startup, about six years old now, that focuses on speech recognition. If you look at the world, you see that there's this evolution of interfaces with machines where you have punch cards and keyboards and mice and touch screens. What's next? What's the next way that people are going to interact with machines? I think people can start to see the vision. It's very natant, but you can see the vision of voice there. You can see people trying to work with Google Home and Siri and voice assistance. You see terrible IVR experiences like calling customer support and getting automated voices. But it's there. People are trying to break into that product market.

Adam Sypniewski: At Deepgram, we believe that the product market for voice is waiting to explode if it has the right technological underpinnings. Or put another way, speech is the next screen. And when you look at the technology that could possibly power that underlying technological layer, you find existing offerings to be lacking in scale. They can't meet that scale demand. So Deepgram, we threw everything out and we said, "How would we build speech recognition today versus the pipelines of the '90s?" And low and behold, it works very well. We have an incredibly scalable system where we are ready to tackle that voice market.

Shane Hastie: Let's talk about the environment for development at Deepgram. You've mentioned a couple of things to me in our conversation earlier, one, that it's very flat and that you do things that are somewhat counterintuitive, even down to the programming language you've chosen. So let's start there. Why Rust?

Exploring technology choices – why Rust? [03:25]

Adam Sypniewski: Okay. Why Rust? It's a great question. When we picked it up about four years ago now as our primary engineering backend language, it was not a very popular language. It was a very small community still at the time. But what we've found was that we had built speech recognition pipelines. We had built them in Python at first. Python is very natural if you're kind of in this AI researchy space and playing with data munging about. But we kept running into issues trying to make the system run reliably. And the reason for those issues was because working in our space, you have to keep a lot of state in your head.

This is one of the most fundamentally true things about engineering, is that you get in flow and you're thinking about your problem and you spent the last couple hours working in it, you really know what's going on, you have all the different variables in your mind. And of course, there's somebody who would knock at the door at that moment. Poof! It's gone. And that's a horrible place to be in as an engineer. You have to build that space back up and it takes time. In fact, it takes so much time, it takes so much effort that it's so easy for even the best engineers to forget things at a time. This is how you end up with silly things like null pointer exceptions. Of course you should have handled that, but you didn't. And in our space, we have a lot of complexity.

We threw out Python, because Rust, it's compiler and it's language specification actually hold a lot of those invariance for you. So you don't have to sit there and think about every little thing. The compiler has your back. It's going to say, "No, you can't do that because there is a potential for a data race down the line," or "You can't do that because it violates thread safety," or "You can't do that because you might have a double free if you do." Now, the engineer is suddenly free of a lot of the boring burdens of engineering. You can actually focus on the fun problems, the more fundamental problems. And the more time you give an engineer to focus on the real problem, of course, that's the further your business blooms, the further your product grows because they're spending their time more effectively, more efficiently. And so that switch to Rust, it was an abrupt change. I can't say it was pain free, but it was well worth the effort, and the team at this point loves it and could not imagine going back to anything else.

Shane Hastie: So this touches something for me that we recently had a discussion in the Trends Report podcast that all of the InfoQ Editors did. We were talking about getting to higher and higher levels of abstraction in code. We were getting into some of the no code stuff. Is that what's happening here, that the compiler is doing that for you?

Adam Sypniewski: I would say that it is a form of that. It is a form. It's not the same. All analogies are wrong. Some analogies are useful, right?

Shane Hastie: Yes.

Getting to higher and higher levels of abstraction [05:56]

Adam Sypniewski: So the idea, and you're talking about higher and higher levels of abstraction, no code. What do they do that they do? They fundamentally let you focus on your business problem and not on the procedural problem that you need to solve. And that's the advantage of having abstraction layers. It's why we build them. Even in code, we build abstraction layers because we don't want to repeat ourselves in code. We don't want to have to manage the same state in 10 different places. We want to be able to focus on the deep problem.

So Rust is still a systems kind of language. Yeah, you can build video games with it too, which is awesome, but a systems language like C, C++ are so you can ask reasonably how much abstraction can you really get out of that. And so it's just a different kind of abstraction. Now, the compiler is telling you, "No, no, look at the world differently. This is the way that computers actually work. You're thinking about it wrong if you want to write in C because you're going to make these mistakes. Let's apply a layer of abstraction that we will enforce at the compiler level and let you up around that level." So it's still a systems language, but it is still a type of abstraction that lets you focus on your business problem and less on juggling and state keeping.

Shane Hastie: I hear the passion in your voice. How do you convey that passion or find people with that passion?

Make your environment so interesting that great people find you, rather than you need to find them [07:09]

Adam Sypniewski: It's hard, not because they aren't passionate people out there, but people have different passions. We need to find the right set of them. I would say at Deepgram, we try very hard to set up an incredibly high bar for our hires. So that's one thing that you can do. If you set a very high bar, you're going to find people who, well, have what it takes. What does that mean though? Does it mean that they all have three years of Rust experience? No, it's not that. I don't care if they've been in Rust. I don't care if they've been in AI or machine learning. I really don't mind. It's really the kind of people that we're looking for are people who have that excitement. So it's less about us finding those people and about those people hopefully finding us, finding what we do interesting and exciting to them and they can hopefully self-select.

In fact, a lot of the time when I talk to these people and say we use Rust, they'll say, "Really? I've always wanted to learn that." You can hear the passion in their voice. We're looking for the kind of people who have that passion innately and they're just looking for creative outlets to apply it. Maybe their previous job didn't let them have that kind of autonomy that they need. Maybe they just haven't had the time with everything else they've had to do. I don't know. But for us, we view this as in a deeply creative problem fundamentally. So at Deepgram, it's all about showing people that what we do is wonderful, is exciting. We'll let them use the creativity and their intelligence and bring that passion to us. And then at that point, it's our job to culture it, to nurture it, to let that passion grow and to give it the outlets that we've been promising.

Shane Hastie: So how do you do that? In terms of an engineering team working together, how do you, one, keep them going in the same direction, but also give that, what you've said, autonomy and power empowerment?

Adam Sypniewski: It's a great question. If I knew all the right answers, I could write a book, but I'm still learning myself. I would say that showing people what we're trying to do, showing the engineers, "Hey, here's our core piece, here's how it works. Here's how the pieces come together. Here's a whole bunch of problems that we see right now." These ones I think are really important.

Inspiring people to do great work [09:09]

Adam Sypniewski: These ones are less important. "What would you like to work on?" And you have to talk to the engineers frequently, weekly. It could be briefly, 10 minutes. That's fine. That's plenty. Ask, "What do you love to do? What do you love to do at Deepgram? What do you see? You know the problems we have, which ones do you want to focus on?" And sometimes you can say, "Yeah, great. Go pick any of those and go do it." And sometimes you can say, "Yeah, go work on some of those, but let's make sure we don't spend more than four days on that. We need to come back to this." But always having a conversation around it, like, "What tasks do you want to work on? What tasks do excite you?"

It can't just be task-oriented too, because smart people can see a bigger picture. Smart, talented engineers with that passion are going to see other issues in the system that you haven't been thinking about or haven't been prioritizing. They might say, "Hey, our metrics gathering has been a real pain for us to use lately because of this, that, and the other. I really, really wish we had something different." Oh, do you? What? Would you like to fix it? Would you like to learn how to do that? And oftentimes the answer's yes for two reasons. One, because it fixes the pain point they see, and two, they've never had to do that before. "How do I upgrade that metric server? What goes wrong there? What goes right? How do you do that?" Again, people with passion are so excited to learn. And by letting them feel pains and telling them, "You can fix those pains. That's okay. We encouraged that," allows them to feel a sense of ownership.

You can also work with them to do professional development. I had an engineer come up to me and say, "Hey, I really, really want to be better at SQL. I know I don't use it that much, but how do I get better at it?" I'm like, "Well, here's some good resources. Why don't you just spend some time writing a bunch of SQL code? Oh, okay. Great. Go do that." If it means that in the background you're spending a few hours a day or a week learning SQL, great, do it because it's going to pay off and triplicate or more later down the line when you run into that problem with SQL and now you've got somebody who's passionate about it who's been able and just raring to go on a real world problem now.

Shane Hastie: How do you constrain that to the right problems?

Creating semi-autonomous pods of people and giving them sets of problems to address [11:10]

Adam Sypniewski: I think the best way to do it is to have, I wouldn't say self-organization, not entirely, but to have project like pods of people. Engineers, maybe with a researcher or two, maybe with a product person or two, and take that group and say, "Here is the set of things that you need to solve. Go off in that direction and tell us how to solve those." And this other group of people, "Here's another big set of problems." That gives them an overarching envelope for them to think about like, "Oh, this is the space that I need to be in now." And then they can start to feel out creativity within that space. So it's really about setting constraints. It's about saying, "This is the set of problems. Not those other ones. Not right now. This is the set of problems. How can you get excited about that set?"

So if you, as a leader, as a manager can identify 2, 3, 4... It depends on how big your organization is. At Deepgram, it's typically two or three problems is how many we can tackle at once. What are the two or three problems that you can divide the entire team into? They're working closely with their colleagues, and let them be creative.

So an example might be, "Hey, we need to know how to do load balancing better in this situation. Can you guys go solve that load balancing problem?" I do not, as a manager, want to say, "This is the load balancing software you should use. These are the algorithm for load balancing you should be using." I don't even want to say things like, "Your response times must be this or less." I want them to be smart and to learn those things. I will review them with them hopefully a couple times a week, but they're going to learn this. They're going to figure this out. If you give them kind of an open-ended problem in a particular direction, they're like a liquid or a gas. They'll expand to fill their container. The question is just make sure you have containers. Don't say "Here's the universe of all things. Here's your container," but expand.

Shane Hastie: But the other side of that passion is the danger of burnout. How do we keep these teams healthy?

Keeping teams healthy and avoiding burnout [13:07]

Adam Sypniewski: That's also a really good question. And you're right, it's very important. What I tended to find works really well is to make sure that they are focused on things they love. If you're not focused on things you love, it's much easier to get burnt out. If you can focus on things you do enjoy, then oftentimes burnout comes down the line once those first few weeks or whatever start to wear off. So making sure that they're excited about what they're working on. And ask them during those weekly touch points, those 5, 10, 15 minute touch points, "Do you like what you're working on? Is there something else you want to work on? What is the next project you want to work on after this?" Always being aware of their emotional temperature, what they're passionate about, and making sure that you can give them that time.

The other one is be incredibly flexible, allowing them rather to be incredibly flexible with their time. So at Deepgram, we have an unlimited PTO policy. This is sometimes controversial in the industry. People will say like, "Oh, I don't know how much I'm allowed to take." It's awkward conversation. We try very carefully to encourage it to be a very open conversation or to even encourage people to take PTO, like, "Wow, you guys did a great job. Why don't you to take the rest of the week off?" Or if people will just say, "Hey, I've done a lot of work. I just need to take a break." Great, no problem. So being very supportive of that, letting people be open and express when they are starting to feel stressed and immediately letting them be able to vent that or get that out of their system by being able to take time away is good.

At Deepgram, we also operate in a number of time zones around the world. And so we're already kind of aware that some people are available at certain times and not at others, which means honestly that if you want as an engineer to work three hours in the morning and then take a nice long three hour lunch, go for a walk, play with the dog, come back, work later into the evening or however you want to organize your life, that's okay. Don't have this strong expectation that people must always be around at every time. Let them find their times. That will also help them naturally keep their stress levels lower because they don't feel like they have to get everything done in this wall of time in front of them every day.

Shane Hastie: Obviously, the last year and a half, we've all been remote. Is Deepgram a consciously and deliberately remote environment?

Being a remote-first organisation [15:22]

Adam Sypniewski: Deepgram has been since it's very beginning. I'm actually Deepgram's first employee. And our co-founder was in San Francisco and I stayed in Ann Arbor, Michigan, and that was fine. Right from the beginning, we had that separation. For a while, we tended to hire up in San Francisco and slowly we had people move out of San Francisco or we hired one or two people remotely. But really from the beginning we've had an attitude of, "Good people are all over the place. Let's find the good people, the people with passion who want to work on this wherever they might be."

This isn't to say there isn't an advantage to having an office. There can be, especially if you have a very passionate team who want to work together. Engineers, I find, and my experience as an engineer, I love doing whiteboarding sessions, problem solving sessions, draw on the board and think about it. It's so much more visceral in experience than waving your hands on a Zoom call. I think that an office environment is something that is good to have, but not something that should be required. It is a perk. Like many other perks, there are benefits that an employer might offer. So in the case of Deepgram, kind of come full cycle, we've always been remote. We're always been open to remote, but we also offer that office environment with people who want to come in and work with like-minded individuals on exciting problems.

Shane Hastie: One of the other things we touched on in our conversation earlier was the attitude to build versus buy.

Build rather than buy [16:40]

Adam Sypniewski: Yes, in the case of Deepgram, this comes in a number of different places. One would be our decision to stay out of the cloud as much as possible rather than fully invest in it. Another could be building and running our own tools in-house rather than having somebody else serve them. And I think that this has been incredibly valuable for a number of reasons.

In the case of working in the cloud, we actually have some real hardware constraints at play here with our GPUs. GPUs are not cost effective in the cloud. And really, GPUs are funny little devices and you learn an awful lot from having to deal with the day to day issues that they come up with. Having a kernel panic over such and such a GPU, or having it drop off the PCI bus. These are frustrating experiences, but the more you see them, the more you understand how the system is working, how to solve that problem, how to get around it, if you're going to be running services in-house rather than paying for hosted versions of them, you learn sysadmin work, you learn DevOps in a much more low pressure environment than your production services.

For example, if you want to serve an internal dashboard, in the case of Deepgram, like, "Here's how many hours of audio we're processing in the last five seconds." Those are very interesting. And if you host those in-house instead of putting them somewhere up in the cloud where you have cloud analytics and BI tools, well then, your engineers have to learn things like front end work and deploying those services. And what does high availability mean in that case? And if you screw it up, uh-oh, that internal dashboard goes down. Customers don't care. People might internally say, "Hey, is this down for anybody else?" But it's low pressure. You move on.

So people have a fairly friendly environment in which to hone their skills and learn the skills they need to deploy the production system, to maintain the production system. And so I think there's a lot of advantages around building rather than buying because you deeply own the problem space, you understand the problem space. Now, there are trade offs. You don't, especially as a startup, want to overinvest in things that are not in your wheelhouse, that are not part of your value proposition because they can be serious distractions. And so as managers, the question is, when does it become a distraction versus when is this actually an on-the-job learning opportunity for my engineers? And it takes finesse to figure that out and practice. But being openminded to it, I think, is a valuable attribute, a valuable skill for managers.

Shane Hastie: What about the build versus buy in terms of people's skill and talent? You mentioned you use Rust.

Adam Sypniewski: Mm-hmm (affirmative).

Shane Hastie: Not many programmers are coming out with, "Oh, I've got three, five years of Rust experience."

Look for people with funky, strange, interesting, eclectic backgrounds [19:19]

Adam Sypniewski: That's true. Yeah, in our case, we are incredibly open to people's technical background. In fact, we have very few computer scientists at Deepgram. Not because we have anything against them, it's just that we're just so open with the backgrounds that people have. For myself, I've never taken a single computer science class in my life, but I've written so much code at Deepgram. And I love the funky, strange, interesting, eclectic backgrounds. In my mind, it's about, "Do you have the passion and love for your job, for high performance computing world in general, that if we dropped you here, would you be excited to figure out how to solve the problems? You'd be excited to orient yourself to the space." There is certainly a skill bar, "You must be this tall to ride,” sort of thing, but plenty of senior engineers can be above that skill bar. And if you add the excitement on, I'm not worried about it.

If you have never programmed in Rust, that's fine. If you've been in the space for five, 10 years, it's not a problem. A new programming language? You'll know it in weeks. Don't worry about it. Don't sweat it. Never worked in Ubuntu Linux system before? Don't worry about it. You're going to figure it out. It's a lot like all the other Unixes out there. We are very open to letting people stumble. And I don't mean that in a negative way. I mean, yes, there's going to be a learning curve for them, but be patient with them. Because we know that these are smart people. We know that when we hired them, that we're buying their excitement and their passion, and we just need to let it flow into Deepgram's problems.

So let them figure out how to do it, helping them, providing good tribal knowledge. So when they ask questions like, "Hey, I was trying to compile this and I got this error. I don't understand why. Can someone explain it?" encourage good question asking, but then the teams are like, "Oh yeah, I've seen this plenty of times over here and over here. And man, I had to struggle a long time before I figured out it was this." Letting them do that, at Deepgram we call it thermalizing or socializing. Coming to this shared mental state with each other is a wonderful, good thing. Everybody's got different talents. There's going to be times when people who have been programming in Rust for many, many years say, "How do I do this in dotnet? I've got a customer asking about it." I'm like, "Oh, now it's the dotnet guy's time to shine."

Just not worrying about the letter of the law, like, "You must have this many years of experience in these languages and you must know these technology stacks," that's not important. It's important that you find the people who know how to do that. I think a good analogy here is honestly around especially higher education. It's less about the thing you learn and more about how do you learn to think? How do you learn to learn? And that's the same thing. We're looking for people who know how to learn and are excited about it. And that pays off, because as long as you have these sorts of people, you know that they will adapt to whatever problem you'd find for them to solve. If you hire people because they check the boxes, the moment you show them a problem that doesn't check the box, they can more easily stumble. So find people who learn to learn.

Shane Hastie: You mentioned your own background in astrophysics. What are some of the other backgrounds, just out of curiosity, some of the people on the team?

Encouraging diversity and creativity [22:13]

Adam Sypniewski: Let's see. So we have physics, mechanical engineering, doing shaped charge design, so shaped explosives for mining. There's another physicist. We have, I think, two Naval engineers designing boats and ships. We do have some more traditional masters in AI. We have bachelors in CS. We have a master of fine arts. It's kind of all over the board. Some of these people have programmed, some of these people haven't. It's really an exciting bunch of people. Oh, we also have music, cello performance.

Shane Hastie: How do you mesh that diversity?

Adam Sypniewski: Honestly, it self assembles. When you find people who love learning, they're just excited to interact with each other. Playing cello, making cheese, fermenting mead. So many things. These interests kind of come together. It's very, very natural. There's almost no inertial force that the manager needs to apply to make it happen. Really, they just are curious. They ask questions without second guessing or doubting their own questions or their own self confidence. And the team is just happy to assimilate all these different backgrounds. Because every different person knows, is aware of the fact that they have greater strengths or bigger weaknesses than others because the backgrounds are all over the place, and they more or less just fit together naturally. There's nothing artificial about it. So I would say that I don't have to do anything. I don't think anybody does. If you find the right people and you're very careful about looking for that passion that they'll have, it's a recipe for success even if they don't all have that shared background.

Shane Hastie: So as a manager in this environment, what do you do?

The role of the leader in this environment [23:52]

Adam Sypniewski: Yeah, yeah, it's a good question. And it has shifted over time. At this point as CTO, I am normally focused on company strategy and vision for our technology. But that said, I'm on our pager duty rotation too. So I go and I am interested in the problems that come up and I am sitting in team meetings throughout the week, trying to understand the problems that they're facing, trying to see if there's ways that I can help. If you hire good people, don't ever shut them down. I mean, I will tell them like, "Hey, here's my opinion. But if you ignore me, that's fine. I'm not going lose sleep over it. You don't have to listen to me. That's okay." So it's around I think giving advice, but framing it in the right way, but then also doing my best to encourage upward growth for some of the other engineers.

Professional growth in multiple pathways [24:39]

Adam Sypniewski: And I will say this is an interesting professional development problem because you'll end up with two kinds of people. There are people who want upward mobility, "Hey, I'm a software engineer now, but I want to be a team lead. And then I want be manager. And then I want to be the director and I want to be a VP someday." Great. But a lot of engineers like engineering and they say, "I don't want to do anything else. I don't want to be a manager. Because where would I program? Where would that come in? What would I'd be doing? It wouldn't be the thing I love." To some degree, that can come as a fear of the unknown. You don't know if you'll like them managing either. You'd simply know that you like the programming. But talking to them can help with that to understand what it is that they want and don't want, or are afraid of or want to avoid.

But even then, you can find good engineers and you can help them grow as team leads, as tech leads. And it becomes a problem of setting expectations for them. "Here's what tech leadership means at Deepgram: That these sorts of products get produced, that you talk to the team this often, that you talk back to me this cadence, but you still have that autonomy and you will still be programming with them. I just want you to be able to guide them and shepherd them." And this becomes a lot of my job too with the team, is identifying people that we don't have to promote. I mean, not because I wouldn't because they don't want to be, but somebody who wants to organize their engineering knowledge in a different way, in a more codified way or a curated way than the day to day engineer. And so building the team becomes an important part of my time. Recruiting becomes an important part of my time.

Shane Hastie: Adam, thanks very much. Some really interesting points and areas there. If people want to continue the conversation, where do they find you?

Adam Sypniewski: I'm adam@deepgram.com.

Shane Hastie: Thanks so much.

Adam Sypniewski: Thank you, Shane.

Mentioned

• QCon London
• Deepgram
• Adam on email