Building Effective Developer Communities

Transcript

Shane Hastie: Good day folks. This is Shane Hastie for the InfoQ Engineering Culture podcast. Today, I'm sitting down across the miles with Robin Purohit of Peritus.ai. Robin, welcome, thanks for taking the time to talk to us today.

Robin Purohit: Thanks so much for having, Shane.

Shane Hastie: So, a good starting point, I suppose, is, who's Robin?

Introductions [00:56]

Robin Purohit: Well, I'm a Canadian born tech executive, made my way to Silicon Valley over 20 years ago. And, had a career primarily in the software industry as a growth exec and big company exec. But now, I'm running AI startup Peritus.ai.

Shane Hastie: And, the reason we're talking today is about developer communities. So, what's going on with developer communities?

Robin Purohit: Yeah. So this is the entire focus of our company. We're essentially a machine learning platform for community led growth. So you think of any developer-led business, or developer-first business, they're all trying to engage developers on their buying cycle from discovery, to evaluation, to ultimately production role success. And, we know developers go to a myriad of communities to get experts from people that they view they can trust to get unblocked on that journey. And so, what we've been doing is we've been analyzing, well, what the heck's going on in all these places? Which ones are good, which ones are not, and how do we help companies do that better?

Shane Hastie: What makes a good community?

What makes a good developer community? [02:03]

Robin Purohit: Well, first and foremost, it's a lot of engagement from everybody helping each other. So, that's a cultural thing that you see, particularly, in open source industries, but also increasing now in the API industry. So, when there's a gathering place where people feel like they can safely ask a question and get help from the each other, it creates a virtuous circle of more and more people wanting to be helpful. And those are the ones that we see are the ones that are most successful over time.

Shane Hastie: I love the idea of that virtuous circle, but a lot of these communities have a pretty nasty reputation for treating new people fiercely at least.

Intimidation and nastiness hurt community growth [02:43]

Robin Purohit: Yes. There are huge intimidation factors. That's one of the, I'd say unfortunate and bizarre things about communities. Some are very open and friendly, but what I've heard is that Discord has become the hot one. A hot communication channel. Is turning nasty. It's almost like Twitter for gamers and coders. And so, people are starting to shy away from that as a place to actually learn something. Unfortunately, Stack Overflow, as amazingly successful as that platform is, depending on the moderator, which is nominated by the community, it can get a hostile tone. So, we talked to the Stack Overflow folks, what they'll say is that, there's probably 70, 80% of people who only ever view something there, because they're so afraid to actually ask a question. Yeah. So, I think that has to change. And I think that's why people go to other places as well, to go and more freely share and discuss things.

Shane Hastie: You mentioned, you've done research into this, and benchmarks, and so forth. What were you finding in that research?

Stats from research on community engagement and value [03:42]

Robin Purohit: It was a little surprising to me to be honest. So, we benchmarked probably 50 or 60 vendor communities, ranging from pure open source to enterprise tech vendors, to see what was going on. And, in general we found is that, only 30% of the questions were answered ever. And only 20% of questions were answered in the first 24 hours, but it's really important to somebody. God forbid they're having some outage problem and they need help, but even just, if you're doing your work, you want a quick answer and just get unblocked to move on. Yeah. The other interesting thing, there was a huge variance by communities on how many people were actually contributing to answers. So we looked at how many people are actually answering 80% of the questions, right? So on average, it turned out, magically 20% of the people are answering 80% of the questions. It's the Pareto principle, but the data actually shows that.

The awesome ones, like the Ingenics one is really awesome, is the flip, 80% of the people are actually answering questions. So I think, something to be said about the culture of that Ingenics community of really wanting to be helpful to each other.

Shane Hastie: So, if I'm trying to build, lead, guide a community, whether it's within my own organization or in a bigger group, how do I make sure that I get this safety and maintain that culture?

The importance of great moderators [05:05]

Robin Purohit: Part of it is making sure you nominate more staff great moderators. Now, there's an art form to being encouraging, but also staying disciplined and the right things being discussed on the right channel or part of the community. So, the people part of it is always the most important, because they're the ones that are going to create that culture every day. The second is just awesome content. I think the more content you have that is easily usable by the community, in those discussions and findable, the better, the more you are updating content that's addressing the issue of the day, the more valuable that community gets. And, quite frankly, most of it is about peer to peer help, right? So, if you're trying to help somebody else, you want to be able to make sure you have the latest information, so that you can put it in your answer, your recommendation to somebody. So, these are all things that create that virtuous circle, at least in my view.

Shane Hastie: And, you mentioned some of the stuff about what to avoid, how do we manage, stamp on, avoid the hard stuff, the bad stuff?

Robin Purohit: We have a technology answer to that problem that we provide. But I think in general, if you're a moderator you want to be monitoring for negative sentiment. Right? Assuming you pick a good moderator. The moderator's also going to be looking at things to make sure the tone stays constructive and helpful. And, there's one of the little pieces of machine learning magic we provide, is we actually help you classify groups of conversations that have negative sentiment or brewing frustration. And so, you can mobilize on them, with either a personal intervention or getting these people unblocked with a new piece of content or something. So, that's one way. So, I think, it's a very manual exercise. And I think, what we hear is that, because there's been an explosion of number of ways that communities can gather, like it's GitHub, yes. It's Stack Overflow, yes. But it's also vendor Discourse forums, usually, or Lithium forums. It's Reddit and subreddits, they're specialized in a topic, is a ton of Slack on their.

So, I've talked to some of our customers and they have 10 different community channels. So the challenge of having good moderators that have the time to monitor all these different places, even just for tone of conversation and being helpful is tough enough, but then making sure people are actually getting the help they need is even more challenging, because there's not that many technical people that are experts. Right?

Shane Hastie: Yeah. What I'm hearing is moderation is a more than full-time role.

Robin Purohit: Exactly. Yeah. And unfortunately, it's also the most short staffed role. You've talked to all these managers DevRel teams is that, really running their efforts on a shoestring budget. And it's unfortunate, but that's the way it is. So, they need more automated ways to find these things and scale their impact.

Shane Hastie: That does lead us to, how does AI help?

Ways that AI can support moderation [07:54]

Robin Purohit: Yeah. So what is AI good at? So, this is my first AI startup. Although, I've advised a few in my career. AI is really good at solving problems where there's a heck of a lot of data and a lot of interaction with the data. So, one of the reasons my instinct was to attack this market with a AI solution was, over 26 million developers, probably more, but that's the official number that's bandied about, communicating on hundreds and hundreds of forums, and thousands of slack channels, and subreddit groups. So, there's so much data to look at to apply machine learning, to define things like sentiment, similar questions, use cutting edge, natural language processing to actually help synthesize and put answers back in people's hands quickly. So, this is an ideal problem actually for machine learning and natural language processing.

Shane Hastie: And, through your organization through the work that you've done, how have you set this up? How have you made it happen?

Robin Purohit: Yeah. So, first foundationally, to crack this problem, we have to build a machine learning engine that can ingest all of these conversation along with the content for developers into a big engine in the cloud. So, where we spend a lot of our energy, the early days is getting a data pipeline that knows how to talk to all these different forms, public and private, and different document types, and bring them all into our cloud engine, and make sense of them. And, once we have that there, then we constantly train it based on how we see interactions from developers and users on that data. So, that's foundational. And then, you want to build something useful on top of that. Data's good, but it's necessary, but is not the answer.

Robin Purohit: There's two things that we found are really valuable. One is, you want to help the businesses analyze this data. So, who's getting stuck where and why, and how do you remedy that? How are your metrics trending? So, is the level of engagement going up? Level of people helping each other going up or down? And on which of these various communities is that happening? And then, the second thing we do is we just help automate self-service. There's so many questions that are asked over and over again in different places, maybe in a different way, slightly different way, but very similar. So that's a huge opportunity for machine learning to help. So those are the two prior ways, analytics for the businesses and self-service with the developers, and make that experience common across all those communities.

Shane Hastie: And how do you avoid the pitfalls of AI of the bias that's built in, I would imagine in this case, the one solution repeated over and over or those things.

Avoiding bias and ranking respondents [10:30]

Robin Purohit: Yeah. I mean the good news... Because of the problem that we're trying to solve, we don't have to struggle with the ethical challenge of racial bias or gender bias. We do look a lot at who's authored content and who's authored responses, because over time we build profiles of, "Hey, this individual has shown that he or she, whatever pronoun they choose to use, how good are they at addressing people's questions on this topic?" So, we develop a ranking and a score of those. And then, we use that to actually amplify the best recommendations we could provide back to people. So, it's very much a meritocracy, the way we apply AI to this problem set, that takes all of those other ethical things out of the way. Yeah. So, that's at least one angle.

I think then you look at it more holistically, what we try to engage in these developer communities... Most businesses have understood that the developers are going to make their own decisions, right? This is an influence game, not a selling and marketing game. So, the more insights they have on where people are getting stuck, what they love, how to amplify all of that, it is an influence marketing engine for developers is what all these companies really need, it's not another CRM. It's more a beneficial version of Instagram for the developer at large.

Shane Hastie: You mentioned, the myriad of channels. How do we start to make sense of that?

Making sense of the many channels used [11:55]

Robin Purohit: I think the cat's out of the bag, Shane. I think, given how organic these things work, every year we'll have another explosive communication channel, that people will gravitate to. I think, the unfortunate reality is, it's always going to be many, and people may shift to one or the other, depending on the experience they get. So typically, when we see an open source company, if you go back to even API companies, they all start in GitHub, right? They post their stuff on GitHub and get engagement in their first batch of users. And then, if it's successful, something pops up on Stack Overflow for their product, or usually in the form of tags associated with their product. And then, as things grow, they want to start communicating much more real time with people. And so, Slack starts to get used a lot. And so, it just builds, and builds, and builds.

And so, I think we're stuck with many ,and every time I turned around... I was so shocked this year, that all of a sudden, Reddit has become an explosive area for technical conversations. When we started this journey a year and a half ago, we very rarely heard that. So, I'm sure next year it'll be another one. And so, the problem we try to help solve is, okay, if you're a developer, how the heck do you find where to go? Do you need to care? Or could you bring all of that to wherever you're asking your question, on that day, in that moment? And if you're a business, how do you engage at scale across all these places? Some of which you might own, as your sponsored properties, but many of which, be public discussion boards or forums that you don't own, or don't moderate.

Shane Hastie: Getting into moderation and curation. So, AI helping, but how much active curation should be done on content in these communities?

Active curation to make content useful[013:34]

Robin Purohit: Yeah, I think there's a huge opportunity, and certainly something we're focused on, making the content useful. The only thing that matters in the developer journey, is the content useful? Because this audience, probably many which are listening to this podcast, have no time for marketing, or things that hide the facts they need to get the job done. So, the good thing is that you can measure usefulness really easy, right? Because if people see something, again, they refer to it in posts of their own. It gets more downloads. There's lots of metrics that measure usefulness. And, using more advanced the AI techniques, you can detect questions that are cropping up or trending that clearly are not being addressed by content. Because people are looking at content and they're moving on or ask the question over and over again.

The journey and experience needs to adapt to the user needs[14:21]

Robin Purohit:  So you can measure it and you can detect gaps pretty quickly. And ultimately, you can find out whether you've solved that problem, if the trend lines start to go down. And again, it's all about the effectiveness of the content. The thing I also want to say is that, I've said the word journey a couple times. It is probably very familiar to your audience is that, there's many different kinds of questions people have, right? So, in the beginning you're trying to just discover what options do I have? Who supplies what? Then you are evaluating, and comparing, and contrasting. Then you're actually actively prototyping and kicking the tires usually with a combination of dozens of other technologies. And then, you get to build, deploy, and scale. And so, nature of the questions and the help you need will vary depending on where you are on that journey.

Shane Hastie: And again, thinking from the point of view of the person trying to build, support, and maintain this type of community, and to access it, how do I hear the weak signals?

Robin Purohit: What do you mean by weak signals?

Shane Hastie: The things that are coming down the line that are not there yet, but they're starting to bubble up, the what's next?

Identifying patterns early [15:28]

Robin Purohit: Yeah. So, I'll just give you an example of what we've built to try to be helpful. But, it's something called hotspoting, which is very well understood in the support universe. So, hotspoting is analyzing a chunk of data to understand where is there a pattern that is emerging, like a heat map or a weather map. And so, you're always looking for those early indicators, say there's something that is trending up with a lot of frustrating tone or sentiment, and a lot of people that are engaged in this conversation. So, we actually try to identify that in real-time across all of these community channels of conversation. And then, allow people to mine down through that and say, "Okay, what's really happening? Who's saying what? Who's participating?" And by the way, "Who should be participating?" As I said before, we try to focus on who are the people that are good at these topics.

So because of that, we try to show is that, "Hey, here's 20 people who should be engaged and only three are. So you might want to nudge these folks, whether they work for you or just are active in the community to get on this one. So we can prevent this from boiling over."

Shane Hastie: Slack is the community we all love to hate. There is so many Slack channels today. I know that you said you've been doing some stuff around that. Tell us a little bit about it.

Overcoming Slack sprawl [16:45]

Robin Purohit: Just talk a little bit about the chaos. So, I call it Slack sprawl. It's essentially free to start a new slack channel or workspace. Even in one workspace, the channels pop up in my own company, basically run our business on slack and I can't keep track of the ones... So, I think we've all seen that problem. The other problem and interesting nuance about slack is, it's not accessible to Google Search, right? So, what is the first thing you do with your looking for help. You go to Google type a phrase or keywords. You're not going to see those Slack archives on Google. So, if something good has been said, it's lost knowledge. And then, even within Slack, this search of going back through conversation threads is very, very weak, right? So, we've seen us as an ideal opportunity to be helpful and apply, again, our machine learning techniques, because particular in open source, we've seen huge, huge increase in the use of Slack over the last two years, in this new remote work reality that we all live in, as well as for our DevOps teams, right?

Robin Purohit: And so, we apply our capabilities to make all the Slack archives searchable, but also do that across all of the relevant Slack channels of workspaces we can find. So, we combine that with all the other stuff we have from community forums and content. And now, we have a constantly refreshing engine that incorporates all the knowledge that's being created in Slack to make it useful. So, we released a couple months ago to build on all of that, as an AI assistant for Slack that purely targeted developers and open source communities. So basically, when you ask a question, you can ask our bot and it'll retrieve, "Here's five answers that look like they're very relevant. Regardless of where they came from. And by the way, here's three people. Now look, they're very qualified to help you and how you can find them." They might be on this Slack channel. It might be in another Slack or on Stack Overflow. And that's caused a lot of interest, particularly among the open source community. So, we're excited to see the response.

Shane Hastie: What are the trends that you're seeing? Where are we headed with this whole community support, engagement, and so forth? Because as you mentioned, there's been a huge surge over the last couple of years, but I think there's more coming.

Trends in community support [18:56]

Robin Purohit: Yeah, and I think it's very big and booming and is only going to get bigger. So, there's this great analysis that Tyler Jewell from Dell Technology Capital has published. So, he has characterized the size of the developer first economy, and he's characterized a $40 billion, growing at 20% year over year. So, this is commercial open source, APIs, dev platforms, dev tools. And, I think, if you look at the startups that are getting huge evaluation and growth right now, the vast majority of them are taking a developer first approach one way or another. And SaaS companies, every SAS CEO I know is opening up their platform, right? Salesforce did that first, but now everybody is doing it. So I think this area's going to explode and with explosion will come more conversations and more places.

What we think is missing is some way to make sense of this all, so you can run your business at scale. And I would say that, what this industry doesn't need is another CRM, right? Because, that's a workflow-oriented approach, that's really largely intended to top down selling. Given this is such a bottoms up, data driven, and high volume activity. We think there's room for something new to be created, that's based on machine learning and all this data, that can help people automate disengagement, developers, wherever they are on the journey from discovery all the way to monetization. So, that's our vision. And, I think the nice thing we're playing into a lot of tailwinds of greater growing opportunity.

Shane Hastie: Some really interesting points and good observations there, Robin. If people want to continue the conversation, where do they find you?

Robin Purohit: You can find me on LinkedIn, I respond to pretty quickly. Just find me at Robin Purohit. If you try me on Twitter, I might be a little slow, unfortunately, but you could find me on Twitter @RobinPurohit1. And you can definitely check out our website at peritus.ai. We have all sorts of resources and free downloads available.

Shane Hastie: Thank you so much.

Robin Purohit: It's been great to talk to you, Shane.

Mentioned

• Peritus.ai
• Tyler Jewell - Size of the Developer Marketplace Analysis
• Robin on LinkedIn and Twitter