Transcript
Schneider: I'm Christi Schneider. I am a software engineer at Square. I'm going to talk to you about building extensible software. This will be high level. We'll talk about building software from the ground up. The power points you should be thinking about extensibility, and how to plan to make your software extensible.
What is Extensibility in Software?
Let's start with the basics. What is extensibility in software? Basically, it is the ability to extend out of the box functionality. You have some product that you build, and you'd like partners to play in your ecosystem. You want them to be able to use your data or show their data in your software. This often requires a partner ecosystem. You can see here, a screenshot of the Square developer site where you can go to find documentation and APIs that work with the Square platform. You may not realize it if you haven't talked about the concept of extensibility, specifically, but most software engineers have extended some piece of software during their careers. It's pretty rare that you'll be working on completely standalone software that doesn't integrate with any other systems.
When to Consider Extensibility in the Software Build
When should you be thinking about extensibility when you're building software? This shouldn't be an afterthought. A lot of the things you might think of as checklist items, like the things I've mentioned here are often thought of towards the end. When you say, no, I need internationalization so that this can be used in more than just the U.S. Those things that really touch many parts of the system for the extensibility can really be difficult to finish at the end of your project. You really should be thinking about all of these from the beginning.
You might have heard the term mobile-first or API-first. This is used by people who have experienced an issue in the past, like they suddenly needed to make a non-mobile website, mobile. They say what we really should have done here was we should have planned for mobile at the beginning, so next time, we're going to build something, we're going to do it mobile-first. This is a fallacy. All of these things are things that you need to be thinking about throughout the design and implementation of what you're building. Really, no piece of the software is more important than any other piece. You need to do all of these things. Some of them are more important than others, but you should be at least thinking about them at every point during the software lifecycle.
Examples
Let's talk through some examples. This really basic one is interfaces and abstract classes. This is a very basic concept of object oriented programming. Here, you've got a Java example of an InputStream, the basic shape of an InputStream, you read, and flush, and close. Each of those implementations might have a different way of doing each of those things. Google Chrome extensions. These extensions can be installed on top of your Chrome browser and do things within websites you visit. Social media share buttons are a type of extensibility. This is taken from the sessions page on the QCon website. Here, you've got buttons that let you share to Twitter or Facebook, and those will open a modal or open the website that will let you create a tweet or a Facebook post. Those are both provided by Twitter and Facebook as part of their API in order to extend their websites. A Myspace profile page from 2008. This is Lana Del Rey's old Myspace. Here you can see a music player that's embedded. This is a Flash Player. There are lots of different things you could embed in your Myspace page and customize. Then you have iPhone and Android apps. You might not think of this as extensibility, but it really is. On an iPhone, Apple has a platform for developers to grade apps that tie into the phone's hardware. Basically, any app that you install is extending the functionality of your phone beyond its out of box functionality. Voice skills. All of the skills that you can write and add to your voice assistants are extensions. They do lots of things like tell you what the weather is, for example, that you can install applications just like you can on a phone that extend the functionality of that device.
Levels of Extensibility
Let's imagine another very common scenario where somebody who maybe works in an office has to download a CSV from one piece of software and upload it into another piece of software, and they have to do this every day. This is all manual. They literally go and click download. They download a CSV file. They go to the other place, and they click upload and upload that file. That's processed by the second system, and something happens with that data. The way that extensibility can help a little bit here, let's imagine that we get an IT intern for the summer. The office manager says, "I have to do this thing every day. It takes up all my time. Could you help me make this better?" That IT intern would add a button that pushes that CSV file from one system to the other. Instead of the office manager having to go into both websites every day and download and upload, they just have to click a button.
The next level, you might get some automation. Maybe this happens on a nightly process. All of these things are ways to make the office manager's life better, but it's really still not perfect. There's lag time between when the data comes into system A and shows up in system B. Other than that, might be to use webhooks, and whenever an event occurs in system A like a record is added or updated, then some process picks up that change immediately and puts it into system B. If system A or system B were very extensible, you might not even need to go log into both of them. What this could look like is rather than having to log into system A to track one thing or system B to track another thing, you might just go to system A and it shows the information from system B, or vice versa. You no longer have to use two systems, you can just use one, even though the data is actually coming from both of them. In this case, you might also use that webhooks example to get the data from one system to the other as far as your extensibility in addition to webhooks. Maybe you don't really need to go log into both systems, and you can just go to one place to manage everything you do.
Types of Extensibility
We can talk about the different types of extensibility. There's actually two different kinds of data model extensibility. First, you can have an entirely new data model. Let's say I have this data model. These partners have the ability to add new data models, those could be supported in some cases, or data models that conform to a specific shape. Then you have fields on data models. In this case, these two properties, salesforce_id and is_awesome are coming from other systems. Things you need to consider here are how you're actually going to represent this in your database. How are you going to deal with daemon conflict? What if you want to add an is_awesome flag in the future, are you going to break everybody's data that is already set up within this awesome flag? Is it going to conflict and overwrite all those values? That's something you need to think about before you release publicly. If you're going to add the ability to make custom fields on a data model, you do need to think about how those will actually be managed. You have API extensibility. I'm sure all of you have used an API at some point. This is a screenshot of the Square Payments API. Very often this is going to be a REST API with CRUD operations.
Then there's UI extensibility. This is a screenshot of the QCon Plus website. Here you can see they've embedded a live feed of tweets that tag QCon. This is something provided by Twitter, where you can embed a feed, then it will automatically pull tweets onto your page, and they still update live. That was having another system's data show up on your website. The other way is having your data show up on that first party website. This is from a Salesforce tutorial called Recipes. This shows you how to create UI inside of Salesforce. This Hello tab is entirely custom. You could build out a, for example, Square tab that has Square information that would allow someone who's using Salesforce as their system of record, to not have to leave Salesforce in order to do everything they need to do for their jobs.
Another type of extensibility is event streams. A very common implementation is webhooks, where your system will subscribe to events in another system that you want to be updated about when that event occurs in that other system. They will call an endpoint that you own and that let you know that something has happened. Then you can trigger a process based on that. Event streams are less common, but are used in some particular use cases. Event streams, you generally need like a live connection for, whereas webhooks can be asynchronous. You could use serverless architecture for webhooks, whereas you couldn't for event streams.
Other Considerations
There's a lot of things that you need to consider when thinking about extensibility in your product. A big one is backwards compatibility. Once you have a public API, or you have something public that anybody is using, it's a faux pas to break existing integrations. I'm sure many of you have encountered backwards breaking changes in APIs or libraries that you're using. You know how it can cause issues. You also want to make it not super difficult for you to upgrade. You want to add tests so that you'll know if you broke backwards compatibility. You'll want to make it so it's not a huge process every time you want to upgrade, and that you really don't have to think about it if possible. Try not to paint yourself into a corner. Similarly, maintainability. If you have this public platform, you need to keep it up to date. You need to think about when new software comes out, for example GraphQL, if you're going to go implement new things in that. Often, this gets forgotten about by product managers, because it's not that visible. People can't just go click around in it necessarily. Any APIs, security is a major consideration. You also don't want nefarious users to be able to take advantage of something that you're providing in order to make better software for your customers, which is really the end goal of all of this.
You need to know pricing. Is it going to be free to use your API? If not, what is going to be people who want to use it versus just not using it versus going to a competitor? If you do make it free, how are you going to get it to pay for itself? This isn't necessarily something that engineers need to be doing like a business analysis for, but it is something you should make sure to talk about with your team. Just consider at the beginning, especially if you're like a lead engineer or an architect, if your product manager hasn't mentioned how this will make money. That's something to bring up and ask about.
Privacy, it's similar to security, but you need to think about if you're storing PII in your fields, or you're allowing partners to enter PII. How you're treating that data, if it's getting exposed where it shouldn't be, or clogged, or anything. Just like any first party data, you also need to be careful with third party data. If those third parties get any data that has PII in it, something you might need to think about here is how it will work when you get a data deletion request. If I contact you, and I say I want you to delete me. I want you to delete all the data associated with me, and you do that, does that mean that any data that your partners have pulled previously is also deleted? If not, do I know that I need to go talk to them? Are you going to go talk to them? How do we make sure that my data isn't somewhere that I don't know about, and that I don't expect it to be? Reporting is a hard thing in software in general. It's even harder when you think about third party data.
Reporting is both ways here. Can you use some third party reporting software for your first party data? Say I want to view my data in Tableau, can I pull that data out and visualize it? Then for third party data, does their data show up in your first party reports, or can it be included optionally. Say, there's some third party integration that has a field that is extremely important to one vertical that's not included in the first parties, but that's something that basically all customers in that vertical need on all of their reports. Is it possible for that to show up, or will that third party that includes that field for them, need to recreate all of your reports, but just with this one extra field added?
How will you support this API? If something goes wrong with any of the platform that you build, that is publicly available, how will that be supported? Do you have the support people that you need, and are they trained properly in order to support this platform versus that they see an error message on a website? Additionally, if you have third party integrations that are showing up in your system, how will support determine if the problem is in your system or in that third party system, and direct people to the appropriate place. Something that would be pretty bad is if there's a major issue, like a very important page fails to load site, and the customer is trying to access that. Calls your support. Your support goes back and forth with them finally saying, "It's this third party integration. You need to call them." You really should think about how that interaction will work ahead of time. Try to minimize that back and forth, and then redirecting them, saying, it's not our fault. It's not really a great customer experience. Just consider those when you're talking to support, when you're figuring out a support plan for your platform.
Developer support is also a consideration if you have a public platform. You're going to need to have a way for developers to see the documentation for your platform, and find out how to use your system. Very often there will be developer forums, or idea banks, or bug filing systems like Jira. Tying into that, there's the partner ecosystem apps or any tools that developers might use to interact with your platform. The better these are, the happier your developers are going to be. If you have a very mature partner ecosystem and a very good app store, that's going to make people more likely to use your platform. If they're missing some of these tools, or they're just not as good as they could be, that will make people not want to use your platform. You can also consider no code or low code integrations. Think about things like, if this, then that, where you don't need to write any code in order to tie into an API.
Something that people often don't consider until later is that the extensibility you're building for third parties might enable your first party developers to ship functionality faster and easier. A lot of companies, they basically want first parties and third parties to build things in the same way. One of the companies I worked at, custom code was built in exactly the same way as first party code. All the tools that we use internally, we would also provide in our SDK. This meant that all of the tools that our third parties were able to use were pretty good, because they were the ones that we needed the T's to do our jobs. This is one way to do dogfooding, where as soon as you build these tools, you have testers built in. If there's ever a time where a first party developer needs to Slack message you to ask how to use something, or if they need to go read code that's not open source, that's a potential problem and should be addressed in your documentation.
Software Lifecycle
When should you think about extensibility? As I've been saying, extensibility along with all the other checklist items that I mentioned earlier, are really things that you need to think about throughout the software lifecycle. Discovery and requirements gathering. When you're first gathering the requirements for your software, and talking to your potential customers that you should also be talking to the potential partners. What are the other pieces of software that those potential customers use? Is there a leader throughout that industry? Do they have an API? How do they play in the partner ecosystem? Can you establish a partnership with them early on? How can you integrate your systems together? Are there any parts that are very important to a particular vertical that you know you're not going to be able to build for that you're expecting a partner to build for? Are you enabling partners to build for that? Do you have an ecosystem that partners will want to build on? Can you find a partner to build up that piece so that you are unable to build fully?
When you're designing and architecting your system, you need to think about extensibility there, both UI design and architectural design, of course. In UI design you need to think about if third parties are going to be able to add elements to your UI, how will that show? Then, when architecting your system, if you are building something like the Facebook event feed. If that feed is architected to only pull from an internal database that has five different event types in it, how is that going to work if you want some other event types to show up in there? If you want to integrate with Ticketmaster and have Ticketmaster events show in your timeline. Are they going to be able to add those events straight to your database, or do you want to consider a different architecture that better supports pulling from multiple sources, as long as they follow the same shape. When you're writing the code that your partners will tie in to, making sure that you're doing the same due diligence you do with your UI components, as you do with your developer platform.
When you release, you need to consider how you're going to release changes to the developer platform. How do you get news out to your developers that are working on your platform? When you release, are your third party developers ready for this? How do they know when a release is coming, or what was changed in the release? Do you need to release a new version of your API and have an upgrade plan? Does everyone get over-the-air updates? Do your APIs match with a specific version of the software that's installed on user machines where some calls will only work on certain versions of the software? Then, iteration, often forgotten in software. You do need to iterate on your releases. You need to gather feedback from your users, in this case, other developers, and see if any changes are needed. See if there are any bugs that you need to fix. See if there's something that people really want that you're not supporting today. Then go back and actually implement those changes.
Dog Adoption Exercise - Requirements
We're going to go through an exercise in extensibility. We're going to talk about how you might build a dog adoption website. Your first requirement, obviously, is that potential adopters can view your adoptable dogs. They're going to want to filter on things like location, and size, and breed. You need to get those dogs in there somehow. Another major requirement is that shelters can list their adoptable dogs. They can fill in all that information that people want to filter on, as well as uploading photos, adding a story, and recording how much it's going to cost to adopt that dog. Then they will want to be able to mark the dog as adopted. These shelters may want some type of analytics. They may want to see which dogs are getting a lot of clicks, which dogs get the shortest time to application or a completed adoption, and maybe drive their behavior based on those analytics so that they can get dogs adopted more quickly.
Dog Adoption Exercise - Extensibility
With this dog adoption website, it's pretty basic right now, there are some ways that this could be extended, so that more functionality is available for your website. Maybe some people want to, in their news every morning, hear about new adoptable dogs near them, cat adoption. If you're building software that lets you list and adopt dogs, it seems like it'd be relatively straightforward to allow other types of animals on this platform. If you didn't consider that other types of animals might be available at some point when you began building the software, this might be really hard to retrofit later. That could be something you'd want to think about from the beginning and build it out so that it's generic, so that you have this animal shape with cat or dog specific fields in a child object or in a related table, or something like that. You might want to be able to show DNA test results on your dog data model through your API and on a webpage. If there are any rescues that do DNA testing, they might want to just be able to embed this nice, beautiful UI into the dog's web page so that that's visible to anyone who's trying to look at dogs. Maybe this could automatically tie into that breed data that I mentioned earlier. Where as soon as the dog's DNA test results come back, it updates the dog data to replace all the breeds that were on the dog's data before with the breeds from the DNA test results, so that people who are looking for particular breeds can find dogs that would be a good match for them.
A lot of dog fosters will create an Instagram account, or a hashtag for the dogs they're fostering in order to help get them adopted. You might want to embed an Instagram feed for a particular account or hashtag on the dog's webpage. This will let potential adopters see the dog as they live and play throughout the day. You might want to trigger an event to occur when something happens in your system. In this case, when a dog is adopted, make a Facebook post. You can also do things like integrate with Apple Pay to pay for adoption fees. There are lots of different payment integrations out there, and most of them have some extension that you can embed within your website in order to accept their payment methods. Shelters are always trying to get their animals adopted more quickly, so that they can pick up new animals in order to get those adopted. They might want to do some A/B testing to see what gets their dogs adopted faster. Do dogs with an Instagram feed embedded get adopted more quickly than dogs without one?
Questions and Answers
Betts: There are a lot of concepts, I think people who don't work with extensibility, or don't think about it or think it's something you can apply later on, may not realize just how broad the category is. Is this something that you've just started doing at Square, or you've been working in this space for a while?
Schneider: Yes, I have been working in this space for a while. I have a lot of industry experience here. I worked on an extensible product for most of my career at my former employer, your current employer, Blackbaud, a couple different products there. That gave me a ton of experience, especially working directly with the developers who worked on the platform and seeing their needs. Then I worked on the extensibility team there and then more recently at Square. Working at Square has been good to see a different perspective, see what things are the same and what things are different. Then I have quite a bit of experience with working with other platforms personally, because I worked on many hackathons. Then, I also worked on the innovation team at Blackbaud as well.
Betts: I know Square is one that a lot of companies interact specifically with their API. They're there for payments and stuff like that. It's foundational to the Square business model. I think one of your other examples was Salesforce, which I think is up there as canonical. They exist as a way for people to extend and integrate. In fact, I know more people who work on modules that add on to Salesforce, or I think SharePoint is another one. Most of us aren't working on a Salesforce or SharePoint. How do you consider this for, we want to start doing some extensibility. How do you get a company that's much smaller to just get started, or is it something you just put off and delay and come back to later?
Schneider: I think when you look at companies like Salesforce or Square, those considered extensibility in their initial product. They knew they would never be able to build for all the verticals and niche users that they would want to support. They planned for having a great developer platform and attracting partners that could build out those pieces that they can support. I don't think that is the right decision for many companies. It definitely requires a lot of upfront investment. The things I went over in my presentation are just things you could do. I don't think every developer platform should have all of those things. I think if I was building out a platform where I thought that I would need extensibility, I'd probably do some market research. Figure out exactly what people are looking for, and just start with that very small, minimal extensibility and build from there. Because you don't want to make a huge investment on something that people aren't going to use, and then you're going to deprecate and the 10 users you did get are going to be very upset and move to a different platform.
Betts: I can see it being one of those tradeoffs, you don't want to go too far in one direction versus the other. To your dog adoption example, someone says, we should make an app for adopting dogs. The developers probably are going to be like, great, I'll go create the dog adoption service. Do they think that I should create this pet adoption service, it's more generic, or is it some anything adoption service? It's a different way of thinking. Who is involved in making the decision of saying we need to start doing an extensibility first mindset for all of our development and planning?
Schneider: I think it's really part of everyone on the team's job to think about that. I think, really, it's going to fall to the lead developer and architect, if those are different people, to make sure that product management is considering extensibility when they're writing features for the backlog. If you know that you're going to need some extensibility but you don't see that represented in your backlog, you need to speak up and make sure that's in there. That when you're writing the technical details for stories, you have written in the parts that enable extensibility.
Betts: A lot of the examples you gave were public facing APIs, app stores, things like that. Is extensibility something you should be thinking about if you're only developing an internal application?
Schneider: If you're a very small company, probably not something that you might want to invest the time and money into. If you are a company that has a lot of engineers or thinks they will have an external platform one day, you might want to invest the resources to build out internal APIs and tools that your internal developers can use. Then that will make creating this external developer platform easier in the future, and make it so all your internal APIs follow certain standards. You don't need to do a ton of work adding documentation or making sure all your endpoints are named the same thing, or whatever.
Betts: I think that's a good lead into the next session, which is about standards and creating a standardized specifications driven lifecycle. It is, how do you get those standards defined? If you start using them internally, it's that dogfooding process that all of our developers have to use the API, it makes the API better. You can also start enforcing some rules and some checks and some governance around making sure that all of our APIs across all of our microservices, whatever we're doing, start looking somewhat the same.
Schneider: With everything in software, it's always a tradeoff. You don't want to have everyone spending 10% of their time on fixing dumb little things in their APIs. You also don't want to have to do a year's worth of work when you want to make these public.
Betts: Would you consider extensibility the same as plugin capabilities? Is this just a naming convention? Is a plugin just another form of extensibility?
Schneider: Yes, I think so. I think that's similar to Chrome extensions, and other of the examples I mentioned. Plugins are a type of extensibility. I think the word plugin can mean different things in different places as well. Flash was a plugin for Chrome. I think that plugins are certainly part of extensibility.
Betts: You had a slide that had other considerations that just kept adding on. I think it's almost daunting. It made me think, accessibility. There's a few of the other lesser ilities that get thrown on. You're like, "We don't have time for that. We'll come back and put it on later." Then you realize you didn't think about it up front, you're going to run into a big hell of problems. I think just versioning backwards compatibility is probably the thing most people that develop an API run into first if they have customers. What are some of the other ones that you would say you need to be thinking about this before you just say, we're going to make our product extensible?
Schneider: Probably the main point you should take away from this presentation is that extensibility is not something you can just tack on at the end. You do need to be thinking about lots of things all the way through. A lot of those things are more of a general consideration when architecting something versus something that maybe a junior engineer would do. Which is like, when I'm building out a feature, what are the core requirements for this feature? What buttons need to be where and what needs to happen here? It's something that you really are thinking about when you're thinking about higher level of software. The most important, certainly backwards compatibility because if you break backwards compatibility, you're going to break all of your existing implementations from your partners, and they're never going to want to work on your platform. Your clients won't be happy either because their existing integrations that they depend on won't be working. I think probably the next probably equally important is the developer platform. You have to have a public facing website to explain how to use your platform. You can't just be like, I don't know, we have an API.
Betts: Figured out.
Schneider: Yes, basically. There's many tools you can use to create that, that are very easy to use and do a lot of things out of the box. If you use OpenAPI, there's a bunch of tools that will consumer your OpenAPI and make it pretty. Even Postman can do that. Take advantage of tooling that's out there. I think having somebody who is experienced with the tools that are available or knows how to do the research to find what tools would be good for your solution is pretty important.
Betts: Drawing the line, this is if you focus so much on extensibility, do you get too far away from your core features? If you only do your core features, are you making it too hard to implement extensibility later and add it on? The way I interpreted that is, are there things you should watch out for that make it almost impossible to make your product extensible without a complete refactor and redesign?
Schneider: I think we've probably all seen software that's almost impossible to extend without rewriting. I think this is where you see software engineering as an art rather than as a science. There's no one thing that's going to make it hard to make something extensible later. It's something that you're going to learn over time as you use other APIs and as you build your own. It's ok to make mistakes. A really good way to make mistakes is to make a beta, and then if you break things, to avoid an upset, so definitely recommend doing alphas and betas if possible.
See more presentations with transcripts
