Ann Lewis Discusses the Political Tech Landscape, MoveOn’s Architecture, and Scaling Challenges

00:05 Introductions

00:05 Charles Humble Hello, and welcome to the InfoQ Podcast. I'm Charles Humble, one of the co-hosts of the show. And this week, I'm talking to Ann Lewis. Ann is the CTO for the political nonprofit MoveOn. She's a technical leader, architect and active coder with 15 years of experience in software engineering and software management, with a particular focus on distributed systems and scalability.

00:29 Charles Humble: So in this podcast, we'll explore the political tech landscape, MoveOn's architecture, some of its scaling challenges, and also, the role of a CTO. Ann, welcome to the InfoQ Podcast.

00:40 Ann Lewis: Thanks, Charles. Thanks for having me on.

00:43 Could give us an idea of the role that political tech plays and what the landscape looks like in general.

00:43 Charles Humble: I think in some ways, political tech is a bit like something like say online advertising. It's one of those fields that's obviously hugely important, but it isn't really talked about very much, even by people who work in tech, unless they happen to work in that particular field. So I thought a good place to start would be maybe you could give us an idea of the role that political tech plays and what the landscape looks like in general.

01:08 Ann Lewis: Political tech is all about reaching people and trying to get them to pay attention to something and take an action. So in election years, it's all about reminding people that they need to register to vote, in the US, reminding people they need to vote and telling them about where there polling place is and doing all kinds of engagement mechanisms to get people interested in and excited about values and issues that ultimately drive them to vote.

01:29 Ann Lewis: In non-election years, typically political tech is focused on advocacy and organizing. So advocacy means shining a media spotlight on a particular set of issues and showing that there's a significant amount of collective or grassroots support for a particular issue or policy agenda item. And sometimes we do this with signing petitions, sometimes we do this with showing up to protests, when it's not a pandemic, or just trying to find ways that people can collectively show their power to advocate for something that they really care about.

01:59 And what's the political tech vendor ecosystem like? 

01:59 Charles Humble: And what's the political tech vendor ecosystem like? I'm imagining that it's probably, because it's quite a niche area, I'm imagining that the vendor ecosystem itself is also quite small. Is it?

02:09 Ann Lewis: The funding for political tech in the United States is cyclic. So you have bursts six months before elections and there are elections every two years in the US. Every four years, there's a presidential election. And then in between, there's a midterm election.

02:22 Ann Lewis: So typically, venture capital or self-funded efforts will spin up at the beginning of an election year and create a whole bunch of new tools. Some of them hopefully will launch in time to be used for the election, but in some cases we see a lot of startups fail to get ready in time for the election timeline.

02:37 Ann Lewis: And the market cap of political tech is relatively small compared to the rest of tech in the US so there's typically not quite enough funding to create enough organic competition to have multiple offerings per type of system that you might use. So what we typically see is that systems with a wider moat, like a CRM, where all of your user databases and where all of your engagement mechanisms sink into, there are just a few offerings in that space, but there's 1,000,001 canvassing apps. Every single time there's an election, we see a few new canvassing app startups, and they're kind of more peripheral. They're definitely useful when you're going around and knocking on doors to remind people to vote.

03:14 Ann Lewis: But typically, we see a lot of little apps that try to serve a single purpose, and then organizations and tech teams spend a lot of time trying to unify these apps together into a coherent system. And that involves a lot of defining of data bridges, or sometimes negotiating with vendors for APIs. So that's a constant challenge, that the ecosystem is relatively small market cap, and so has mixed offerings.

03:35 Ann Lewis: So for nonprofits, the way that we engage in that ecosystem, is whenever possible, we try to buy a product off the shelf versus building it. But when we need to, we do have the capacity to build missing architectural components. We also try to work together with a lot of other nonprofits, and in some cases, vendors, to try and figure out what is the space missing and who can fill that space, who can create that new product, or create a new political tech startup so that we all have the ability to do one particular thing that we know is going to be critical to our success in the next election year.

04:03 Could you give us a high level sense of how MoveOn's distributed systems work from an architectural perspective and how that fits into the work that the organization does?

04:03 Charles Humble: Could you give us a sort of high level sense of how MoveOn's distributed systems work from an architectural perspective and how that fits into the work that the organization does?

04:16 Ann Lewis: So since a lot of the work that we do involves reaching people and communicating with people, the core part of our distributed systems ecosystem is our user database. And we also have a political CRM, that is where our user database lives and that's where a lot of our engagement mechanisms like targeted emailing, SMS, contact and outreach, where that master user list is stored. That's also where fundraising happens and other sets of data like survey management and event management live as well.

04:44 Ann Lewis: But in addition to that political CRM, which for us as our distributed systems system of record, that's where all of the rights and updates and information about users gets synced into, we also have about 30 other systems that play different roles in our ecosystem. And some of them are systems we've developed in house and some of them are systems that we use that are third-party off the shelf tools. And then there are many data bridges that sync data back and forth between these systems, most of which were not built to inter-operate and ultimately sync data, and in particular writes, back to that core political CRM, that is our system of record.

05:15 Since 2015, you've also invested in a substantial data warehousing infrastructure. So can you talk about that?

05:15 Charles Humble: And since 2015, you've also invested in a substantial data warehousing infrastructure. So can you talk about that?

05:22 Ann Lewis: All our data that flows into the political CRM gets ETLed over to a data warehouse. And at this point, we have 15 other data pipelines that sync data into the data warehouse. We use Amazon Redshift for this. And one nice thing about Redshift is you can write queries that span different schemas so you can ask any question about all of the data that you have and answer it with dataset.

05:42 Charles Humble: How much data do you have?

05:44 Ann Lewis: Tens of terabytes of data. And at this point, we're shipping about a billion rows a month from all of our different data sources into the data warehouse.

05:51 Do you hit regular database scaling limits?

05:51 Charles Humble: Honestly, I find that quite surprising. I would not have expected that you would have that kind of volume of data given what you're doing. Do you hit regular database scaling limits?

06:03 Ann Lewis: Constantly. Yeah, that's one of our big scaling challenges. So the architecture in 2015, when I started at MoveOn, was focused mostly on a couple of MySQL databases that were really scaled up to their limits. And so we still have a handful of MySQL databases and Postgres databases that back different systems that we use, but e-tailing those different data sources into a Redshift data warehouse allows us to do different things with different data sources. So when we need to scale up the read volume of requests that go into a particular relational database, you can add read replicas. Scaling up write volume is always difficult and often involves creating queues or figuring out how to control and throttle bursts of writes into your system.

06:42 Ann Lewis: But one big insight that helped us scale our architecture was thinking about what different types of queries needed and where to run those queries on different types of databases that were built for that kind of query. So computationally expensive queries that need to reconcile tens of millions of rows together, that's a use case that belongs in the data warehouse type of database. And then reading and writing based on user actions, that typically belongs in our relational database architecture.

07:07 Ann Lewis: And then the ETLing between those two systems is its own other hard problem. And we have typically used different ETL vendors to make those pipelines happen so that we can make all this work on a nonprofit budget.

07:18 If you need to spin up more read replicas, is that something that you are able to do without downtime?

07:18 Charles Humble: If you need to spin up more read replicas, is that something that you are able to do without downtime?

07:25 Ann Lewis: Yeah, we typically can. That's a scaling operation that the RDS ecosystem makes relatively straightforward. It's also possible to increase the system capacity of your primary database as well, but that typically does involve downtime so we have to carefully plan that.

07:47 Charles Humble: Right. Yes. And presumably, if you're having to update hardware characteristics and that kind of thing, again, I would presume that that typically will involve some downtime.

07:49 Ann Lewis: Yeah. That's like increasing the CPU in your primary database machine or increasing network IO, for example.

07:56 I’m wondering how you go about binding vendors to specific SLA requirements that you might have because you don't have a great deal of leverage to use.

07:56 Charles Humble: Something that I'm curious about, since you use a lot of third party tools and you explained that because the funding rounds are cyclical for political tech, the vendor ecosystem is quite small. And so I'm wondering how you go about binding vendors to specific SLA requirements that you might have because you don't have a great deal of leverage to use.

08:17 Ann Lewis: That's a constant challenge, figuring out how to build relationships or get leverage in some way with vendors. Ultimately, we all have the same goals. Vendors want to build products that political tech organizations like MoveOn can use, but in some cases, depending on the startup and how it's run, sometimes there's not enough product management happening to deeply connect with users in this space to understand what actually needs to be built. We see a lot of tech based startups where tech people are building tech that's cool for tech people and there's a gap between what feels like it's the next cool new thing and what are organizers actually going to use.

08:51 Ann Lewis: So I spend a lot of time building relationships with the tech leads at different vendor companies and talking not just about what should exist in the ecosystem, but also how to build it and how to test it. And I think ultimately the thing that makes a new political tech startup successful or not is not how well they build the system and software, they have to build it to spec, it has to scale, but how many months do they allow for testing, especially testing with non-technical organizers, who do the majority of the actual work in the space. The longer they allow for that testing and feedback period from users, the more likely they are to be successful.

09:24 How did your textbanking system Spoke come into being and how did you get to open source it?

09:24 Charles Humble: Changing tack a little bit. I know you have a certain amount of software that you've built in house that you've opened sourced, including Spoke, which is your text banking system. Could you talk a little bit about that? How did Spoke come into being and how did you get to open source it?

09:40 Ann Lewis: Well, we've been committed to open source as a value for the last five years. And so when we discovered that we were building the same data bridges over and over again, to connect systems together, we realized, well, if we've built it, there's no reason for anyone else to go out and rebuild this. So we started out by building Python libraries that helped get data into and out of different systems. Some of these libraries were actually picked up by the vendors who were creating the software that we were bridging data from. And that's a big success. It's sort of helpful to show what the community needs by building it.

10:09 Ann Lewis: In 2017, we were members of the Progressive Coders Network, that grew out of Coders for Sanders in 2016. What emerged in 2016 was this new engagement tactic called peer to peer text banking and that's where volunteers will text other people and ask them, "Have you registered to vote? Do you know where your polling place is?" Etcetera. Or, "Do you care about this particular issue? Yes or no."

10:30 Ann Lewis: It's a great volunteer activity because volunteers can do it from their phones, wherever they are, and just a few minutes of time spent has a big impact. It's pretty quick to send off a text message. So this has been scaling up as a use case.

10:42 Ann Lewis: And most of what political tech is, is the logistics to manage a bunch of people doing a bunch of actions at scale. It's all the logistics of if you have 1,000 volunteers, who's doing which things, who's managing the volunteers, who's answering questions, etcetera. So that's where the software layer, that's mostly what it's doing.

10:57 Ann Lewis: So from the Progressive Coders Network, Saikat Chakrabarti had created a peer to peer text banking engine and was trying to figure out what to do with it. And basically, he made an agreement with MoveOn to hand the software to us and we'd finished productionizing certain parts of it. Then we would make it open source so that the community could have it and benefit from it.

11:14 Ann Lewis: So we've been building off of that for the last couple of years, and now we have a viable open source community with tens of active developers, hundreds of organizations using Spoke, and it's filling this nice gap in the ecosystem where to do peer to peer texts banking there were a few for-profit vendors, but for companies that are small and don't want to get locked into a contract or organizations that are large and perhaps want to have more control over the scaling characteristics or drive down the cost of texting since you have to pay per message sent, Spoke is a great solution for this hybrid combination of nonprofit orgs.

11:45 Charles Humble: And now that it is open source, what's MoveOn's ongoing involvement with the project?

11:51 Ann Lewis: We sponsor it and that we have a developer and a half working on it all the time and a community manager who's handling pull requests and answering questions with the community and the rest of it is the community. So that's been an exciting success story for us.

12:04 Ann Lewis: One moment that I remember proudly is discovering, after the fact, in the presidential primaries last year, that three presidential campaigns had used Spoke because it's about 70% cheaper than a for-profit alternative, if you want to invest in the resources to have a developer manage it, to send lots of text messages at scale.

12:22 Ann Lewis: The fact that three presidential campaigns, so we can't talk to them because of compliance and communication firewall boundaries in the US. We have a lot of election rules. So if there's a campaign that's doing campaign stuff, and you're not officially coordinated with that campaign, you can't talk to them at all, including you can't help them set up their development environment.

12:39 Ann Lewis: So they couldn't talk to us, we couldn't talk to them, but there was enough documentation that volunteers had created for them to set up and run these systems at scale for their own purposes. And I think the Warren campaign ended up writing this great medium article talking about how they saved millions of dollars by being able to use this system and control it in the ways that they needed and drive down the cost of texting.

12:59 Ann Lewis: So that's fantastic. It means that we invested just enough in the ecosystem to allow the ecosystem itself to grow so different organizations can use Spoke for their needs. Many, many developers contribute code back to it. And there are even a few consulting businesses that have been built off of the open source software itself.

13:15 Ann Lewis: So tiny miniature version of some larger open source projects, but we're very proud of how that ecosystem has evolved.

13:23 If I choose to run Spoke, what infrastructure can I run it on?

13:23 Charles Humble: That's really cool. So if I choose to run Spoke, what infrastructure can I run it on?

13:27 Ann Lewis: You can deploy it to whatever hardware as a service system that you are comfortable using. We have lots of template documentation for the AWS ecosystem because that's what we use. So you can deploy it to dedicated EC2 machines, if that's what you prefer. You can also deploy it to Amazon's version of containerization, which is called Amazon Lambda.

13:45 Ann Lewis: We've done a lot of scale tests on the version of Spoke that we run internally and so we understand where the scaling bottlenecks are and also have some advice on how to plan for scale and mitigate some scaling considerations.

13:56 What's the messaging API for it?

13:56 Charles Humble: What's the messaging API for it?

13:59 Ann Lewis: It sends messages via Twilio. So Twilio, Plivo and a few other providers have basically created, with lawyers and technologists somehow, this API layer around the telecom industry and the rest of the world uses WhatsApp, but in the US for some reason, we're still all on SMS, the worst of the messaging platforms. But since that's where people are, the challenge of political tech is to meet people where they are.

14:20 Ann Lewis: So for a long time, it was difficult to build telecom apps because the telecom industries didn't release their own APIs. But just in the last five or six years, we've seen this new ecosystem of API providers that have grown up around telecom that now make it possible to make a phone call or send a text message through their APIs. So that's one of the core infrastructure building blocks that Spoke is built on.

14:40 What are some of the more complex challenges that Spoke is able to solve?

14:40 Charles Humble: What are some of the more complex challenges that Spoke is able to solve?

14:45 Ann Lewis: The vast majority of the complexity with Spoke has to do with audience modeling and management. Because you have to pay per message sent, you're deeply incentivized to be as efficient as possible in who you try to contact. You don't want to contact everyone. You don't want to contact people who don't want to hear from you.

15:00 Ann Lewis: So this part of it that lives in Spoke is you can upload custom audiences that then you can tell your volunteers to text, but internally, our analytics team spends a lot of time figuring out how to model audiences to figure out who actually wants to hear from us.

15:12 And what are the main scaling bottleneck challenges that you have?

15:12 Charles Humble: And what are the main scaling bottleneck challenges that you have? Do you need to use caching, for example?

15:18 Ann Lewis: Caching is not our main scaling bottleneck, even though it typically would be for most web based services, because we spend so much time doing writes. So the volunteers are sending text messages, we're getting responses back. Every time that happens, we have to do writes to our database.

15:31 Ann Lewis: There are a few layers of caching that are built in just like in the form of memorization, at the database model level. But typically, our biggest scaling consideration is managing bursts of writes, because if you have a campaign and you want to text, I don't know, 700,000 people, you tell the volunteers send out these 700,000 messages.

15:49 Ann Lewis: And then there's what we call an action curve, which is the time at which the messages actually get sent out and the requests come in. So sometimes we'll see unexpected bursts of simultaneous write attempts, when a bunch of messages go out and come in at the same time. And so we have a queue that sort of collects up all these writes and then delivers them to the database at a throttled rate that we can control. But that's typically where we run into scaling issues, is making sure all of our database writes can get in.

16:11 Charles Humble: And how do you manage things like authorization and access control? Is that all delegated out to a third party service?

16:18 Ann Lewis: We delegate that to Auth0. So we use Auth0 as a managed authentication provider.

16:23 What advice would you give to someone who is new to the role of CTO in terms of time management?

16:23 Charles Humble: Yeah. Sensible. Lets maybe step up a level and talk a little bit about your move into a CTO role.  I believe I’m right in saying that MoveOn was your first CTO role.  And when I was researching this podcast I came across a wonderful “Tech Lead Survival Guide” blog post that you’d published on Medium which describes some of the challenges that go along with the job.

16:47 Charles Humble: I’ve made a similar career move myself as a CTO and then also as the editor-in-chief of InfoQ which I was doing until the Spring of 2020, and the blog post really resonated with me.  So I thought it would be interesting to talk about that transition.

17:01 Charles Humble: And one of the common challenges I think for most people when they step into a CTO role or another senior leadership role is actually how to prioritise time when you’ve got 100s and 100s of emails arriving every day. So what advice  would you give to someone who is new to the role in terms of time management?

17:21 Ann Lewis: Well, I think the advice to consider it the hardest problem to solve is great. It absolutely was the biggest challenge that I faced. When you're growing in your career as a technologist, typically you're getting deeper and deeper into a particular area of expertise. Maybe you accumulate a few different areas of expertise, but you typically have a lot of time to focus on your work. And when you have wins or successes, they're typically because you solved a very complicated specific problem.

17:45 Ann Lewis: Being a CTO is kind of the opposite of that. You're going very broad instead of very deep. You're going broad over technical topics that you're more or less familiar with, but it's all about figuring out how to not miss the critical opportunities versus understanding one thing really deeply. And that's a difficult intellectual transition to make.

18:00 Ann Lewis: So I think that the biggest piece of advice I would give someone stepping into the role is that your unit of productivity as a CTO is not lines of code written, it's decisions made. And as we all know, making decisions is hard and most human beings aren't very good at it. But like any skill, it's something that you can practice and get feedback on and get better at over time.

18:17 Charles Humble: Related to that, I certainly found time management was a huge challenge.

18:23 Ann Lewis: Definitely.

18:23 Charles Humble: Just having time to think, that's probably the biggest.

18:29 Ann Lewis: Yeah.

18:30 Charles Humble: You've got so many pulls on your time that actually, like carving out time that you can sit there and work through a problem, a sort of more gnarly issue, is tricky. Was that something you found and were there sort of particular techniques maybe that you've found have been really useful in terms of time management?

18:46 Ann Lewis: That was definitely a big challenge. Every single day there's more to read, more to engage in, more decisions to make, more things to learn about, more new projects and efforts to get involved in than you can possibly do. So you're constantly prioritizing, and this can be uncomfortable, especially if you find everything interesting and you want to be involved in everything.

19:04 Ann Lewis: So I found two books to be especially helpful. One was Getting Things Done and the other was Cal Newport's Deep Work. So Getting Things Done, so if you think about your attention as a computer, maybe you have multiple processes that are running at the same time. There's some limit on the total number of simultaneous processes that you can have running at any given time. Same is true for what Getting Things Done calls open loops, just reading your email for a couple of hours, you'll accumulate a whole bunch of open loops, which are questions that you need to answer, decisions you need to make, things you might need to learn more about, problems that you might need to track and assess over time or intervene in.

19:39 Ann Lewis: And all of these things, once you read them and become aware of them, are active open loops in your head. So the idea behind Getting Things Done is that you should write down all of your open loops, be aware of how many you can actually handle realistically at any given time. Don't add more open loops unless you're able to. And then in terms of tracking, for each open loop, track what's the next step that you need to take. And that minimizes the switching costs. Again, if your brain was computer. If you were backgrounding a process and foregrounding it again, you want to minimize the time it takes for you to get back engaged in that open loop after you finish a decision related to another open loop.

20:12 Ann Lewis: So just a simple practice of tracking. I just have a text file that's my open loops at any given time. And I note for each open loop, what it is, how to get back to the context that it's in. So sometimes, for email, that's just copying and pasting the email subject line so that I can search for it more efficiently, even if I have 1,000 unread emails, because I haven't checked my email in three days. And then the next action that I need to take or that I'm waiting on someone else to take. That helps a whole lot, in terms of just kind of keeping a lot of different ideas and processes moving.

20:39 Ann Lewis: And then when you're doing that kind of work, you're operating at what Cal Newport would call shallow work. You're just sort of like engaging with things that take a few minutes here and there, but it's also important to remember that the thing that will make you most impactful at your job is your ability to deeply focus on something really hard that only you can do.

20:56 Ann Lewis: I've heard this advice from a couple of different management coaches phrased as only do what only you can do. So when you get to be a CTO or a tech lead, there's a lot of work that you could do, that's interesting and familiar because you used to do it, like software engineering work or the work that your team is supposed to be working on. But the more of that comfortable previous role work that you hold onto, the more you avoid doing the important work that only you have the breadth and perspective to be able to do or the authorization to be able to do.

21:24 Ann Lewis: And you also end up kind of like micromanaging your team. So if you pick up too many dev tasks as a tech lead, that'll be weird for your team, because you're not a peer to them. There's a hierarchy in the org chart and there's power that's associated with that.

21:36 Ann Lewis: So what you choose to pay attention to impacts what everyone else thinks you think is important. And as a manager or tech lead or director or C-level executive, you're just going to have less time. So you should try and minimize the work that other people who work for you can do, to give them a real opportunity to own things to lead, but also to avoid bottle-necking and then free up your time to focus on work that only you can do.

21:56 Ann Lewis: So the advice of deep work is once you figure out what's the most important highest value work that you can be doing, you need to give yourself enough time to actually focus on that, to make progress. So Newport's advice is to put blocks of focus time on your calendar, like 60 or 90 minutes at a time, to give yourself the opportunity to go really deep into something really important and make meaningful progress on it.

22:19 Ann Lewis: It can be very easy when you get overwhelmed with too many things to do, to think, Oh, well, I just don't have time for that. I don't have time to focus on something for 60 minutes. But if you stay very, very broad, while you can keep a lot of little decisions moving, you avoid the most important big decisions that only you can make.

22:35 Ann Lewis: So Cal Newport recommends creating work blocks and also ritualizing the process of getting into like focus mode, getting into it and getting out of it again. And his recommendation is if you're able to get at least five focus blocks on your calendar a week, you'll be doing enough significant high value work to be able to keep your whole area of expertise or division moving in the right direction.

22:53 Did you experiment with any of the sort of GTD tools at all?

22:53 Charles Humble: That's really good advice, actually. Funny enough, both of those books I've also read and found incredibly helpful. Did you experiment with any of the sort of GTD tools at all, Omnifocus and things like that, or do you just stick to a test based system?

23:08 Ann Lewis: A little bit. I don't think any one of those is better or worse than the other. Because the US political world is so focused with the written world, I'm often an email, I'm often just slinging around paragraphs of texts. So I find it easier to work in a Google doc or even just notepad, just to kind of keep everything free form and organize together as blocks of text. But I do know people who use some of the GTD tools and love them. So I would just recommend experimenting with them and figuring out what's most comfortable for you.

23:35 Charles Humble: We talked a little bit about the sort of whole business of problem solving and this thing that as a developer, you got very deep into a problem. And over time, you have a pretty good understanding of what your system does and how it behaves. And you have a sort of mental model, which is not actually right, but it's generally good enough that you can make decisions. I think one of the things that's really hard as a senior leader is you often have to make quite high stakes decisions with little, or in some cases, no real knowledge or data to go on. How do you manage that? How do you find that?

24:10 Ann Lewis: It's very difficult. It's something that I've had to do from the first month of my job here. I think, in general, the way I think about it is I try to do a cost benefit analysis with every decision. I try to be clear and transparent about my decision making methodology, what I'm considering, what I chose and why, in case anybody's interested or wants to talk to me about it. And then I keep track of all the decisions, above a certain level, that I've made. And every quarter I evaluate them and look back and say, was that the right decision? Did that work out? Would I have made a different decision now? And if so, what do I know now that I didn't know then?

24:42 Ann Lewis: This is just yet another text file of the, I don't know, the 20 text files that make up Ann's brain dot text. And that feedback step, in particular, has been the most helpful and impactful thing.

24:53 Something that's also, I think, quite challenging is finding technical mentors that can help.

24:53 Charles Humble: Something that's also, I think, quite challenging when you get into more senior roles, like to a tech lead level, is finding technical mentors that can help.

25:02 Ann Lewis: When you get up to a tech lead level, all of a sudden you'll never have a boss who can really be a technical coach for you, in most cases, because your boss is typically a non-technical broad organizational leader, so they can give you advice on, are you working on the right things or are you missing anything? Or how are the people who are reporting to you showing up in other teams in the organization? But they're never going to tell you was that choice of email provider right or wrong. That's something that you have to then start doing for yourself.

25:26 Ann Lewis: So that I found initially to be pretty intimidating, but then as I got more in the habit of it, I now find it to be really reassuring. It makes you less dependent on other people to understand if you're doing the right things and if you should course correct, but it does require being brutally honest with yourself.

25:40 Ann Lewis: So I think incentivizing yourself in some way to make more decisions or to be biased towards acting or deciding versus waiting, that's a critical thing. And then giving yourself that feedback loop at the end of some period of time that's meaningful, what are all the decisions I made and which ones of them worked out? That's a helpful thing as well.

25:58 Ann Lewis: The big thing to avoid is spending way too much time making any one decision because that's you functionally bottlenecking all of the work that you do. And that's something that a lot of new leaders do because they don't want to make a mistake. They don't want to get anything wrong. But as a leader, everyone is paying attention to what you're paying attention to. So if you're not making any decisions at all, and you're spending a lot of time in analysis paralysis, it looks like you're not paying attention to anything that's meaningful or important that other people are trying to escalate up to your level.

26:25 Ann Lewis: So there are a bunch of reasons why, but just trying to make sure that you make more decisions over time or make a significant number of decisions over time, that's important. And then you just have to acknowledge that like with any new nebulous skillset, like software estimation, for example, most developers are notoriously bad at this and especially new developers. The way that you get better at it is just doing it over and over and over again and then understanding when were you right and when you were you wrong. The same goes with executive level decision making.

26:52 During your time as CTO at MoveOn were there decisions that either worked out better than you expected or maybe didn't work out as well as you expected.

26:52 Charles Humble: And given that you've been the CTO at MoveOn for, I think it's five years, which is a relatively long time to stay in that position. I was wondering if there were particular decisions that you took maybe early on, maybe in terms of, I don't know, frameworks or upgrades or particular migrations that you've been able to see all the way through, that either worked out better than you expected or maybe didn't work out as well as you expected.

27:17 Ann Lewis: In the first month or so at MoveOn, I had to make decisions about what percentage of legacy code to sunset and I erred on the side of being aggressive. Sunset more to create more opportunities for innovation for my new team. I think that was a good call. I think I could have spent more time interviewing the stakeholders who were involved in the initial build-out of the older generation of tools to understand which use cases would come up again in the future versus are a thing of the past. But in general, trying to make sure that the surface area of projects that my team has to maintain is small enough to allow for opportunities for innovation. That's a key thing.

27:52 Ann Lewis: I think early adopting a new at the time data warehousing product was really good. I think that has paid off over time and we've just consistently built on that infrastructure and it's helped MoveOn grow to three times the level, in terms of engagement, fundraising, staff, types of projects we can work on, in 2020 compared to 2015.

28:09 Ann Lewis: I created a few new roles that are tech people who interface in a hands-on embedded way with the rest of the org and we call it tech support, but it's really more like technical thought partnership. I think that was a very good idea. I wish I had done that sooner.

28:21 Ann Lewis: I think in terms of decisions that I would not make again, we had a couple of legacy platforms, some of which we sunset and migrated to new vendor systems and one legacy platform in particular that we wanted to keep in house and we tried a couple of times to rebuild with our small in-house team. That ended up not being the right call because the complexity of the legacy system was absolutely incredible. And we didn't have, at the time, a product management capacity in-house to be able to understand what is in this system, what do we need to keep and what can we let go of?

28:51 Ann Lewis: And so we tried a couple of times to replicate exactly the previous systems functionality, but in a new language and framework. And that was a mistake because there is this constant challenge with product management that whenever you ask people, can I take away this thing that you have, even if they didn't know they had that thing and they haven't used it in four years, everyone will say, "No, that's my thing. You can't take away that feature. It's mine." So we needed to develop a product management capacity to think more strategically about which features are we going to have now, as opposed to making a laundry list of every possible thing, and then trying to replicate it.

29:20 How do you balance keeping an eye on the tech landscape and day to day operations?

29:20 Charles Humble: One of the unique challenges of technical leadership is that you have to keep your eye on the sometimes rapidly evolving tech landscape in which you're operating, as well as giving enough time and attention to day-to-day operations, day-to-day tasks. We've touched on this a little bit already, but I'm interested to hear bit more from you in terms of how you go about managing that sort of dichotomy.

29:45 Ann Lewis: A constant challenge. And it's difficult to figure out how to get just enough exposure to new ideas and new platforms and techniques without spending too much time on it and then avoiding the big important decisions.

29:57 Ann Lewis: There are a couple of techniques that have worked well for me over the last five years here. One of them is regularly scheduled pair programming with folks on my team. So it only has to be like a few hours a week with a few rotating sets of individuals. It's a great way to spend extra time paying attention to especially more junior members of the team, just have some like quality tech mentoring time. And then I usually ask folks just show up and work with me on whatever it is you're working on today. I'm just interested and I want to learn more about what you're doing and how you're doing it.

30:25 Ann Lewis: And so if it's an area of expertise where I know more than they do, then it can be mentoring time and I can maybe give some advice on like how to use a debugger or how to figure out how these systems are connected together. But if it's an area where they know more than I do, then I can show up and be like, well, you more than I do about this, teach me this, and then I can appreciate their knowledge and expertise. And so that's also like a positive confidence building thing for the folks involved.

30:49 Ann Lewis: We also, since we're a remote environment, we have lots of carefully scripted ways of running meetings to make sure that every bit of time that we spend on Zoom, since everyone gets Zoom fatigue very quickly, is spent in a meaningful way. So one aspect to our tech team meetings is that if somebody has a pull request that is really big and scary and everyone else in the team has been avoiding it because it seems too complicated. If that pull request has been open for more than two days, we ask the person who filed it to show up to the team meeting and do what's called a talking pull request.

31:19 Ann Lewis: So the way that we run that is that person will start talking through, "Well, here's what I'm trying to do and here's what problem I'm solving and here are the parts of my implementation," but they pause every five minutes and randomly call on someone in the audience and say, "Repeat back to me what I just described." So the repeat back part of this is great because it keeps everyone in the audience actively listening and it forces each individual person, including myself, to articulate here's what I understand and here are my questions. So then the process of doing knowledge transfer is made more efficient because you only spend time explaining what the person is missing.

31:50 Ann Lewis: And typically, we're able to get through relatively complex 500 line code review in about half an hour or less. And the rest of the team has been engaged in the process of knowledge transfer. So that's a great way where I found it sort of like get a slice of information about how systems are working.

32:04 Ann Lewis: I also typically put myself on call when we have major scaling issues or scaling events, either internally to our systems or externally, just because there's a what we call high energy moment and everyone in the US is paying attention to something that involves our systems. So we might get like suddenly - surprise - 5 million views to something and then have to solve that problem.

32:22 Ann Lewis: I'm personally very interested in distributed system scaling issues, because it's my area of expertise, but also, that's another great way of sort of like putting myself on the hook to support my team getting through a scaling event, but also us learning together, where are our scaling bottlenecks? How do we debug and assess the situation? How do we mitigate the problem?

32:40 Ann Lewis: One other thing that I like to do, it's a little more scary, is what's called the Richard Feynman method of learning. So Richard Feynman, the famous physics professor, whenever he wanted to learn something new and was avoiding doing the reading for it, he would sign himself up to teach that thing. And when he signed himself up to teach that topic, he'd have no idea what was involved in the topic, but the support performance aspect of doing public speaking and teaching would force him to learn enough of it to not only explain the concept, but answer all of the unexpected questions.

33:06 Ann Lewis: If you can handle the anxiety involved in it, I find that to be very helpful too. Where I'll sign myself up to teach a topic to a collection of nonprofit organizations all solving the same tech problem and that'll sort of force me to like get all the nuts and bolts together of my understanding of the landscape to be able to answer any hard questions folks might have.

33:23 Charles Humble: That's wonderful advice. I've read quite a lot of Feynman's papers and lectures and things, but I've never actually come across that particular anecdote, but it's absolutely true. There's nothing like explaining something to other people to really test your understanding of it. So, yes, I guess if you can handle that level of anxiety, that's a wonderful suggestion.

33:41 Ann Lewis: Ann Lewis, thank you very much indeed, for joining me this week on the InfoQ Podcast.