Declarative APIs in an Imperative World

Transcript

Thurium: I'm Tilde [Thurium]. My pronouns are they/them and I'm a developer evangelist at Twilio. I'm happy that QCon invited me to speak, even though apparently they value engineers over evangelists. Anyway, I'm not here to talk to you about Twilio or anything related. I'm here to talk to you today about using declarative APIs in an Imperative World. First, I'll give you some background on what the difference between imperative and declarative APIs are and why you'd want or need to integrate them.

When I wrote this talk earlier this year, I was a software engineer working at GitHub, working on the Atom editor. I might say things like "my team," etc., when I really mean my former team. I'm going to show you some nitty-gritty details about how my team at GitHub integrated React, which is a declarative UI framework [inaudible 00:01:00] imperative APIs of the Atom text editor. Then finally I'll talk about the rise of declarative UI programming for the web and what we can learn from the history of languages and frameworks to make the future better for all of us.

Declarative Programming

What even is declarative programming? Declarative programming is done with declarations, rather than statements. What the hell does that even mean? It's not the greatest of definitions. How about declarative programming expresses the logic of a computation without describing its control flow? That's still not really doing it for me.

I came across this blog post from a guy named Tyler McGinnis, and that was where I found the clearest explanation, which is that declarative programming is about what you do, not how you do it. It's maybe a little bit better. I'm going to ask you a question and I'd like you to think of both imperative and declarative answers. Does anybody know what this thing is? That's not the declarative and imperative question. At the Exploratorium, there is a light art sculpture by an artist named Leo Villareal. He's the same artist that did the Bay Lights on the Bay Bridge and this is super cool. It's got these really beautiful patterns. It's not that far from here.

The question that I actually want you to think of declarative and imperative answers for are how do you get from the Exploratorium to here or from here to the Exploratorium? The imperative answer would be, "Turn right on Drumm Street. In 0.2 miles, turn right onto Washington Street," because turn by turn driving directions are imperative AF. Now the declarativeness, the declarative answer would be to pretend that this is the interface to a ride sharing app. Ride sharing can be thought of as a declarative framework because you merely specify the address and then the turn by turn directions are abstracted away from you, which brings me to my next point, which is that all declarative approaches have an underlying imperative implementation.

Computers at some level are still pretty stupid and they need clear and exact directions on what to do. Declarativeness is just another layer of abstraction. Can anybody tell me what are some declarative programming paradigms that you might already be using?

Participant 1: SQL.

Thurium: Yes, SQL is definitely pretty declarative, because when you're like, select star from table, you don't have to specify like what partition that table lives on or anything like that. Anybody else got a declarative paradigm?

Participant 2: React.

Thurium: Yes, React is a declarative framework. I'm going to be talking a whole lot about that in a few minutes. You're absolutely right. Well, we've got pure functioning programming languages such as Erlang and Haskell. We've got configuration management languages like Puppet and, of course, we have good old fashioned regular expressions. HTML is declarative as well.

What are the advantages? One big advantage of declarative programming in particular is reducing a whole class of bugs known as state management bugs. I first became a software engineer in 2013, and I got a job at Pinterest and I was so excited. Then the first real meaty task that I got was to rewrite the Digital Millennium Copyright Act form. Funsies. Not funsies. I learned a whole lot about both state management bugs and requirements gathering from lawyers, but for the purposes of today, let's consider a much simpler interface. This is a list of names that you can add and remove things from. This list exists on every website ever that's like, "Please, invite all your friends."

Let's look at an imperative implementation of this code, which I forked from a code pen originally by Alberto Gimeno. Here it takes 13 lines of code to sync the UI with the state when you add a new thing to the list. Thirteen lines of code just for that. Now, granted we're using Vanilla Dom APIs here, but using jQuery doesn't even help you all that much, but worse than the verbosity is that this is fragile AF. There's so many ways it can go wrong. If, for example, you're adding two people with the same name and then you forgot to assign the unique data ID attribute to each item, you're going to have a bad time. This is just a demo; if you think about how much more complex production code is than demo code, it only gets worse from there. You might have some state that's mutated by different components outside of the the module that you're just currently looking at.

Now, let's look at the same implementation in React. React is a declarative UI framework and it gives you an explicit object that represents a state, which makes your code more concise and easier to read and reason about. Then the code for adding and removing people also becomes simpler and less error prone.

There's one more concept, just to kind of explain state to you a little bit, how that works in React. There's one more React thing you need to know to understand this talk, which is props. In React, we have a component hierarchy. Each component keeps track of its own state in that little cloud inside of it, and parent components can pass data down to child components, but child components cannot do the same thing. Child components can't pass data upwards. The data only flows one way and this data is called props.

React is far from being the only declarative UI framework that's popular right now. Angular and Vue are also great. This talk is only about React because it's what I know the best, but please let's not get into framework wars because I really don't like that.

Engineering, just like life, is all about making trade-offs. In order to do a good job of making trade-offs, you need to know what the good parts are and what the bad parts are. What are the bad parts of declarative frameworks? I mentioned earlier that declarativeness is merely another layer of abstraction. What do we know about abstractions?

The artist Jenny Holzer would say that abstractions are a form of decadence. I kind of agree with that. We're living these bourgeoisie lives now where you don't have to care about the size of a pointer anymore. Jenny Holzer jokes aside, all abstractions leak and if you want to do something that a framework doesn't give you an affordance for, you're kind of hosed. Another problem with declarativeness is that it is frequently accomplished via a domain specific language. This is JSX, which is how you write your templates in React and domain specific languages are another language that takes up space on your cognitive stack. I know that JSX in particular is confusing to beginners because I've tried to teach it and it looks like HTML, but it's not HTML, and it's confusing. Also, even for people who've been at this for a while, there's little differences that trip you up. Raise your hand if you have been caught up on the whole class name spelling difference thing. Yes, see? It's not just me, it's a thing.

Integration

Now you understand a little bit more about the difference between declarative and imperative APIs. Let's talk about why you might want to integrate them, and also, how to do that. What kinds of imperative APIs exist in the wild that you might want to integrate with? There's video APIs. Because I work at Twillio, I'm contractually obligated to mention telephone APIs. Just kidding. They don't actually make me say that, but it is really cool. Let me know after this talk if you want to talk about any telephone APIs. There's mapping and GIS applications. There are animation libraries like WebGL and Three.js and there's Atom.

For those of you who haven't heard or might not know about it, Atom is a free open source text editor built on web technologies like HTML, CSS and JavaScript. Atom has an API and it's pretty darn imperative and you might wonder why that is. Atom was originally released in 2013 before the epic rise of declarative APIs. Of course, if you're integrating with an imperative API that your team owns, like we were, you've got to ask yourself some hard questions like, should we just rewrite this API to be a little more declarative? In our case, we decided against it because Atom has this rich, awesome ecosystem of community packages and we'd either have to maintain two versions of the API or force our entire ecosystem to update, which is not cool, so we decided against it.

Speaking of packages and Atom, everything is a package, the tree view and the settings page. You get a package, everybody gets a package. The team I specifically worked on maintained the Atom package for the Git and GitHub integration. It's built into the editor, which is what you see here, so my talk is going to be specifically focused around that.

When integrating React with an existing application, how do you choose where to start? It depends on your use case. If you had an existing fully imperative web app and you wanted to just start sprinkling React into it, you might start at the bottom and make your leaf nodes, like little reusable buttons and stuff react and then move upwards from there. The Atom GitHub integration was a brand new package and so, we took a top down approach. Atom allows you to inject DOM trees into arbitrary places, so we had a root React component and then our component hierarchy extending beneath it, like this. At the leaf nodes, that's where we start integrating with the imperative Atom APIs. We map each API that we care about onto a React component that maintains a declarative imperative boundary.

The first API I'm going to talk about is the Atom text editor, which maps onto the text editor API. Text editor's responsible for any state that's applied to a text editor, so things like where are your cursors positioned? Do you have any blocks of code folded? Do you have soft wrapping enabled? Things like that.

I'm showing you simplified versions of these components for the sake of readability and time. Here we are using Atom's built-in pub/sub library CompositeDisposable to subscribe to a code the events that the imperative code emits. How many of you have used pub/sub before or are familiar with it as an architectural pattern? For folks who haven't, it's basically like you can listen for events that have specific names and then invoke a function or do something in response to those events. Here we're using CompositeDisposable and then the didChangeCursorPosition prop is passed down from the parent component. Every time the cursor changes position, we update and rerender the whole component tree below that.

I'll be the first person to admit that this isn't the most declarative possible approach that I could imagine. It would be more declarative to actually pull the state of the cursors into the component and so that it would know when to rerender itself, but because you can have multiple cursors, the API for that gets pretty clunky and so, to simplify and for readability, we decided to just pull using the Atom API and then update when we needed to. Your code doesn't need to meet some perfectly declarative standard in order to work.

Example number two is the Panel API. Panels are containers on the edges of the editor window, like this pull request review component that was the last big feature that my team shipped before we left. Here we take advantage of ReactDOM.createPortal to start a new component tree somewhere in the DOM, away from our main tree. Then it knows how to render any children that it needs to render.

Here we're using shouldComponentUpdate as a performance optimization. When shouldComponentUpdate is implemented, your component won't update every time it gets new props. In this example, we only want to update if a panel is being hidden or shown. We don't care about those other props. Then in componentDidUpdate, we call the imperative methods to actually show or hide the panel with the Atom API. Don't use ComponentWillReceiveProps for this kind of performance optimization because it's deprecated and eventually it's not going to work in newer versions of React and you're going to have a bad time. An underrated performance hack is to clean up after yourself. When a component is unmounted, we need to make sure to clean up any listeners and then call the Atom API for actually destroying the panel.

One problem that was particularly gnarly to solve was focus management. Honestly, part of my motivation for doing this talk was that I didn't understand the focus management code very well and I wanted an excuse to dive in and really learn it. The solution involved refs. Refs are React's dirty little secret and they allow you to reach in and manipulate the DOM nodes directly imperative style. React's documentation is like, "You think you need refs? Are you sure? Are you sure about that?" The use cases that listed were integrating with an imperative API and doing focus management. Since we were doing both of those things, we figured that we recovered. This is like the boiler plate example from React's docs where you create a ref by calling createRef and then you pass that ref to a DOM element in your JSX, and then to actually focus it, you call Ref.current.focus.

We have a problem though, at least in Atom land, we had a problem, which is that React doesn't guarantee that refs are available until the component has finished mounting, but a component doesn't finish mounting until all its children are mounted. This causes problems when a child needs to consume a DOM node from its parent. Our solution were ref holders. Ref holders allow us to defer operations until the DOM node actually exists and otherwise fail gracefully. In a parent, we set the ref center on our div and then pass the ref holder to the child component.

In the child component, which is what we're looking at here, we use the observe method to defer actions until the DOM node actually exists. Ref holder is an example of what's known as a Maybe monad pattern. If you're giving a talk on functional programming, which this sort of is, you're required by law to explain monads. Don't hate me, I didn't make the rules, but it's ok because monads are probably not as complicated as you maybe have been led to believe. It's basically like a burrito or like a tortilla, like it's a wrapper for thing.

To get just a little bit more technical, use functions to wrap values and monads and then you can even nest these wrappers inside of one another, like some kind of Taco Bell burrito nightmare inception thing. See, I used to be so confused about the whole idea of functional programming because all I heard was that side effects are bad and I should feel bad, but a lot of important computery things like reading from disk, entirely side effects. Then I learned that functional programming is not about avoiding side effects entirely. It's about isolating side effects to specific parts of the code base so that you are aware that they're there and you can, you know, make your code easier to reason about. That's where these monads come in. Values that are wrapped in, for an example, an IO monad say, "Here be side effects. Tread accordingly."

Maybe monad is a special kind of monad and it wraps values that might be null so that you can gracefully avoid null pointer exceptions or cannot destructure undefined or whatever that looks like in your language of choice. Maybe monads are also where the tortilla/burrito metaphor starts to fall apart because an empty tortilla is not going to protect you from anything, least of all the emptiness inside of your soul. I came up with a different metaphor to explain Maybe monads, a social metaphor, if you will.

Recently, I asked out this friend that I had a crush on for a long time and I was, "I'd really like to take you on a date, but if not, it's totally ok and I'll never bring this up again." Saying no can be really hard and with good reason. A lot of people really don't handle rejection well, so I don't blame anybody who's afraid to say no. I wrapped my request for a date in a social Maybe monad, to signal that I would handle it gracefully if her desire to date me was null. Of course, you can't just signal that. Then you actually have to handle the rejection gracefully, which could be a whole nother talk, so we're not even going to worry about that, but anyway, I was, "Call me Maybe monad." She said yes.

Another more JavaScripty way of explaining monads is that a promise lets us write code without worrying about whether our asynchronous data has arrived or not. Similarly, a Maybe monad lets us write code without worrying about whether our data exists or not. This definition of what a monad is is probably not helpful if you're learning Haskell or something, but you can add those additional layers of complexity when you need them. I'm a fan of making things simple to begin with and then adding on from there, so don't ask me about funk doors or category theory.

Atom uses commands to move focus back and forth between elements, which is important for keyboard users or for people who can't use a mouse because of accessibility reasons and just power users that know that a lot of times mousing it wastes time. To understand focus management, I need to explain the commands API of Atom that we're going to integrate with. The command registry lets you associate listener functions with commands in a context sensitive way using CSS selectors, and it makes sense because Atom is built on top of a browser.

The command registry is essentially global because we're using the style of event handling known as event delegation, which is popularized by jQuery. As the event bubbles upward through the DOM, all registered event listeners are invoked in order of specificity which mimics the CSS cascade.

We have this command React.Component which wraps the command registry API and allows you to add new commands. Annoyingly, because we're using the event delegation pattern, the command registry is essentially global. Since as I mentioned before, React has one-way data binding, we needed to pass this component all the way down from our component tree to anywhere we wanted to register listener for a command. As a matter of fact, a lot of the Atom APIs have this problem where they just want a ton of props that need to be drilled down through our entire deep component tree, and at the time that I left the team, we hadn't really found a great solution for this. We tended to spread our props, but in my opinion, that is a code smell because props, the reason the prop types exist is for human readability. You want to make it clear what are the dependencies of this component.

If you're just using the spread operator and passing down the props from the component, you use that human readability. We had some ideas to resolve this, like using the context API that was introduced in React 16.3 and maybe some hooks to make it a little less repetitive. There was some reason that we couldn't use Redux. I don't remember what it was, so don't ask me about that either.

We move focus around by registering Atom commands. In our root React component, which is the code I'm showing here, we have these ref holders that can be passed to child components in order to move the focus around and then we also have an event handler that keeps track of the last focus position. Every time focus changes to an element that's a disincentives component, the event handler fires and this sets this.LastFocus which falls back to a sensible default. Then we pass that ref down to any child components that have focusable elements. All of these child components got to implement a method called remember focus, which inspects the event target and then knows where to restore the focus to. Then, when it's actually time to restore the focus, we call focus imperatively on the proper element.

One important use case for the Git integration is that we wanted users to be able to switch seamlessly back and forth between Git in their editor and Git on the command line because there's some parts of Git on the command line that we didn't have support for, but we wanted all your stage files and stuff to be in the same state, which meant that we had another imperative API to encapsulate, which was Git. The architecture for our Git encapsulation is vaguely like this. We have these model objects which are playing JavaScript classes that represent Git entities such as repositories.

Then we have the model observer, which gives us a consistent imperative kind of API for fetching just the data that we need from our models. I'll show you what this looks like in code in a few minutes. The models emit events and then the model observer does stuff and respond to those events. "Hey, pub/sub, nice to see you again." Then we have observed model, which is a React component and observe model gives us some state where our data lives as well as letting us do find functions to fetch that data.

I know that model observer and observe model is a confusing naming scheme, or at least I found it to be so, but I don't know that I could've come up with anything better because naming things is hard. When we use the observe model component, we specify exactly what data we want from the repository model in our fetch data function. Then observe model also takes a render prop we can use to say, "Here's what our loading state looks like. Here's what our error state looks like. Please render this according to the data that we have."

Historical Context & The Future

Now you have a high level overview of how the Atom editor team integrated React with some imperative APIs that we had to work with. Let's put that knowledge in the context of the bigger picture. If declarative UIs are really so great, why were we ever doing things any other way? Why weren't we using declarative UIs the whole time?

I thought about it and I came up with three reasons. Reason number one, the web did not used to be so darn complex. I'm going to pick on Amazon for a minute, but look at this page. Just look at it, all those buttons and widgets and whatsits. Things have changed so much because we did not use to have full on applications sitting inside the browser, but also state management bugs become more of an issue the more state you have on your page. The more complex your view eye gets, the more you need declarative UI frameworks. In a way, these complexities even come full circle because we are building full on desktop applications on top of Chromium with Electron, which is the framework that powers Atom and it powers Visual Studio code and Slack and a bunch of apps that you're probably already using.

Reason number two is that programming is fashion in the sense of programming is discourse. It's an ongoing conversation and everything is a reaction to things that came before it. First, we had Vanilla JavaScript and then jQuery was a reaction to that that made it easier to add and remove class names, make Ajax calls and that sort of thing. Then React and the rest of the declared UI frameworks were a reaction to jQuery.

Reason number three is that I framed this question incorrectly. Declarative UI frameworks are not as new as I thought. When I asked why weren't we using declarative UI frameworks before, we actually were if we expand the definition of we. This is Qt. It's a desktop UI framework that's been around since 1995 and it powers a bunch of really popular software like Adobe products. Qt has been declarative since 2000. React was open source and blew up in 2013. Again, there's a bunch of factors I've already mentioned that impacted how long it took for declarative UI paradigms to make it to the web, but still I think that this is an innovation that web developers could have picked up on a little faster.

Some questions to reflect on. What else can web UI developers learn from desktop UI developers, from functional program to functional programmers from backend developers? What lessons can we draw from the history of computing so we can stop making the same mistakes and reinventing wheels? I've got to say that it would be a lot easier to answer these questions if we weren't pushing older folks out of the tech industry at such a high rate. Ageism is real. It's real and we're going to talk about it. In particular, ageism is real in Silicon Valley where we are right now, so I'm going to ask you to do a thought experiment with me.

Mark Zuckerberg famously said, "Young people are just smarter," which l know, it's gross. Imagine replacing young people with men. How does that change your feelings about this statement? How might that statement go over differently in the public arena? How does that change everything? Yes, I know, the irony of dragging Zuck in the same talk where I'm praising React is not lost on me. You might not know this, but Silicon Valley's 150 biggest tech companies have faced more accusations of age bias in the past decade than racial or gender bias.

Speaking of racial and gender bias, bias compounds. It's multiplicative, not additive, so older workers who belonged to other underrepresented communities are even more likely to experience discrimination. This is from Stack Overflow's 2019 survey and it says that 75% of professional developers are under age 35 and I know that Stack Overflow doesn't represent everybody that writes code, everybody that identifies as a programmer, but it's still a lot of people and that's a huge problem.

What is the root cause of ageist thinking? Ageism is a problem that's going to impact all of us eventually. I have a sneaking suspicion that part of ageism is rooted in the fear of our body's wearing out and fear of dying, and avoiding thinking about it isn't going to help. I have some news for you and you're not going to like it, but you need to face it. You're going to die someday and you need to make peace with that reality. Silicon Valley is still not really doing this. Alphabet even started an entire company called Calico to solve the problem of death. I mean, haven't those people ever seen an episode of "Black Mirror?" I'm not trying to be all doomsday on you, but I don't think living forever actually sounds that great.

Moving on from convincing you this is a problem, I'm going to talk about what can we do about this? For starters, we can rethink the whole concept of aging, because actually it's relatively new. According to historians Carole Haber and Bill Gretton, 200 years ago, no one thought of older people as a distinct population. A bunch of pseudoscience changed that. Thanks, pseudoscience. In the first half of the 19th century, doctors believed that people ran out of a substance known as vital energy and that is what caused people to get older and die. Obviously, that's not true and was debunked, but still it changed our culture so that we started thinking of older people as a distinct population.

Childhood is also a relatively recent invention. We used to just treat kids like little adults, and I don't think we should start doing that again because child labor laws are actually a good thing. I'm not saying that ignoring ageism is going to make it go away. It's not, just the same way that ignoring racism and sexism doesn't make it go away. Socially constructed things are still real. Money is socially constructed, but you still have to deal with it. What I am saying is that we have changed how we think about aging once, so we have the power to change it again in a better direction.

There's a sick feedback loop where we just, the way that we design and build products for older people reinforces the idea that older people are a problem to be solved. Honestly, it's like the root problem is always capitalism if you dig deep enough. Since technologists are responsible for shaping so many products, we need both age diversity and age inclusion on our teams, so how do we get there from here?

A lot of my next suggestions are like rising tide lifting all boats kind of variety. Even if you are an older person in this room, you have some kind of privilege, because we all have some privilege and we all have some areas where we're not privileged. All of us have the responsibility to pay it forward and do what we can to help other people advocate for change.

The first thing you can do is to use objective hiring criteria. Everybody likes to think they're unbiased. I'm no exception to that, but I was really surprised because when I was an engineer at Pinterest we started using hiring rubrics and I had to write down what good looks like and I was surprised by the amount of bias I had and how often someone would walk in and I would immediately think, "I like this person," and then I would kind of tilt myself towards, wanting to support their answers.

Writing down what good looks like for a particular question before you even get anybody in the room actually prevents you from being quite so biased. That's in theory, what does it look like in practice? Medium open sourced some of their hiring rubric and it's actually a really great article. It's linked to you there. For example, on their data structures question, a no might look like a candidate is unfamiliar with the most common data structures and a yes might look like the candidate understands Big O notation even if they don't refer to it as Big O notation. The whole article is super good. I would definitely recommend checking it out.

Another thing you can do is change your language. You need to work to eliminate phrases like "so easy your grandma could use it" from your vocabulary. Changing language is hard and it's a process, so be kind to yourself because you're going to screw it up and that's ok. Did you know that the whole generational naming thing is also a relatively recent phenomenon? We weren't out there in the 1800s being like, "The big hat generation is so whiny and self-absorbed." Boomers are actually the first generation to be named and I don't think generational naming is actually a good thing, because group psychology or the psychology of group prejudice tells us that as soon as you divide people into groups, even if it's arbitrary, they'll start developing a group identity and again, who benefits the most from us being divided in this way? Marketers, capitalism, but anyway, for the love of God, "Ok Boomer" memes are ageist and they don't belong in the workplace.

Another thing you can do is audit your job postings for bias. There is a company called Textio that uses machine learning to audit the language in your job postings and help you build the most diverse team that you possibly can. Now caveat, I have not used Textio personally, but I have heard good things from people who have.

Another important thing to do is ask. Ask people what you can do to be supportive. I started a conversation on Twitter about, "What are best ways to support older workers in tech?" I got a lot of really insightful answers. My favorite was this, which is to "aim for the same emotional presentation and takeaway in meetings," because if you're laughing and joking with some people but not other people, people notice and humor is a way that we bond in the workplace. It's so easy to use humor to push people out, but let's make sure that we're bringing people in. It's ok to use generational references as long as you explain the joke to other people.

Finally, we in the JavaScript land can stop being so obsessed with shiny new frameworks because I disagree that rewriting your entire code base every couple of years the way that we tend to do is healthy. I don't think this is the cause of ageism, but I think it's a manifestation of the same shortsighted thinking that burns people out and causes them to leave tech.

Conclusions

Let's review what we've learned today and tease out some themes. Declarative UI frameworks are eating the webby world according to NPM's most recent developer survey, but it's not just about growth in usage, it's about love. If you've ever built tools for developers, you know that we are not the easiest group of people to please, but the data shows that React and Vue, people actually love using it, which is no small thing, so hopefully this context is giving you some context as to why people love declarative UI frameworks.

Now you also understand some patterns for what it looks like if you need to integrate React with an imperative API. What I really want to emphasize here is that there is no one size fits all solution because the question fundamentally is, how do I model the state of my particular application in a declarative way? Now you have at least maybe some questions you could start asking yourself.

In case you want to see any of the code that I showed you today in more detail, it is all open source and on GitHub. Finally, now that you've heard some small things you can do to address ageism within yourself and within the tech industry, I'd like you to reflect what action are you going to take to make the future better for all of us?

It takes a village to make a talk and credit where credit is due is super personally important to me. I'd like to thank Alberto Gimenez for the declarative and imperative UI examples, Tyler McGinnis for writing a blog post on declarative and imperative thinking, Evelyn Masa, who came up with the call me Maybe monad joke, and then Katrina Otaco, Joe Furman and Ashi Christian for giving me feedback that shaped this talk.

See more presentations with transcripts