Transcript
Mendez: I'm going to speak a little bit about what AsyncAPI is, and how to build an event-driven architecture. I'm going to do a demo of it. My name is Fran Mendez. I am the creator of AsyncAPI. I'm currently working as a Director of Engineering at Postman.
The Specification
What is AsyncAPI? AsyncAPI started as a specification in the beginning. The purpose was to document and design event-driven microservices, then it went well, so for Internet of Things API, and for streaming APIs in general, and pretty much any system based on messages. I want to make clear that the specification is protocol agnostic. That means that it's not for Kafka. This is not for MQTT. This is not for WebSockets. It's for any protocol. Actually, it has some mechanisms to let you define protocol specific information, whatever protocol you have, even your built in-house protocols.
The Tooling
We also develop a bunch of tools, really a bunch to help you out with this, because we think that a specification without tools is just a document, a piece of paper. We want to provide you also with the best tools possible to work with the spec. Go to the website, asyncapi.com, and you will find out about a lot of tools, like we have Studio, which is right now in beta. Generator, which is capable of generating code and documentation, and a bunch more.
Demo
I'm going to just jump into a demo, and you'll see how I'm using AsyncAPI to build event-driven architectures. The demo is about creating a social network, like a very basic social network. Just for reference, I'm going to show you how it will look like, more or less. This is a frontend that I already created, and looks somehow similar to Twitter. This is the profile of the logged in user. These are some publications, or some tweets, or whatever you want to call it. We have user 1 here, so we're just hard coding the users. We have user 2 here, which is my friend, Lukasz. He can see exactly the same publications as I do. There are only publications from me and him, basically. How did we build just for the reference before I continue? You can see this is failing. Firefox can't establish a connection to the server. Of course, because that's only the frontend. There's no backend yet. That's what we are going to be working on right now. If I click on like, you see that it does nothing on the frontend stuff. It doesn't count.
Let me get back to the diagram. This is more or less the architecture that we're going to be creating right now. We have the frontend already done, so the browser is going to be communicating with a WebSocket server. The WebSocket server is going to be either replying to the browser with some events replying. I don't like to say replying in this context. It's going to be sending and receiving messages from the browser. Some of them are going to be forwarded to an MQTT broker. For this demo, we're going to be using Mosquitto test broker. It's easy and it's ready to use. We have a notifications service here that is going to be listening for specific events, and whenever it's received it's going to send a message to Slack. Just so you can see the full picture, whenever we hit like on one of these buttons, on one of these publications, an event is sent to the WebSocket server. It's forwarded to the MQTT broker, which then reaches the notification service, saying that someone has liked a publication. There's some bit of logic there, depending on if you're liking your own publication or other's publication, it might send or not send a notification to Slack. That is it. I think we have the stage set up.
Let's get started building our backend. Let's start with a WebSocket server. I have the frontend here. We're not going to get into the frontend, this is an XJS app built with React, that I will say a very basic one. For our database, we have this db.json file, very rudimentary, but for the purpose of the demo is more than enough. We have our users here. We have Fran Mendez, Lukasz, and all the posts, and all the likes. Right now there are no likes. Let's get started. Let's create, first of all, an AsyncAPI file, or let's create the backend of the WebSocket server. Let's start with npx create-glee-app, and we're going to call it, server, just to make it easier. Glee is one of our frameworks that works with AsyncAPI very well. It's really well integrated with the specification and uses the specification as a config file. That's it. I don't want to wait a lot for npm to install all the dependencies, continue on another terminal, just so you can see. We also have to build our notifications service, so, notifications-service. I'm going to do these two things in parallel in the meantime. This one is creating the server app and installing Glee and all the dependencies. This one is doing the same for the notifications service. This is the WebSocket server. This is the notifications service. They both are going to have an AsyncAPI definition. Let me check how the server is doing. It's going to take some time.
Next steps are going to be like, once we have our server and our notifications service, the basic project, I'm going to edit the AsyncAPI file that it generates as a demo. We have an AsyncAPI file here, we have a bunch of folders and files. We have an AsyncAPI YAML file here. I'm going to just demo this. I have to switch to the server folder, asyncapi start studio. Studio is one of our latest tools to edit AsyncAPI file. It's an editor that works in the browser. Let me check, why is it not working? It's telling us to convert to the newest version of AsyncAPI, so the latest one is 2.2.0, so I'm going to convert it. It's converted. What is it? Studio is our latest editor. Here we can add AsyncAPI file. Let's just change this. This is the social network server. This is our server. We're going to be listening on URL WebSockets. We cannot listen on 3000 because we have the frontend server on 3000, so let's just use 3001. We have another server, which is the Mosquitto, the MQTT server. I'm going to use this, test.mosquitto.org, protocol.
While I was editing on the browser, this was editing our file here. It was connected through the CLI. When I ran asyncapi start studio, it was precisely connecting the browser and our file system, so I can edit this file anywhere I prefer. Let's continue here. This will be working fine outside this demo. Mosquitto is our MQTT server, I'm using test Mosquitto. MQTT has a peculiarity, and it's required to use the clientId of MQTT, so to connect to Mosquitto we need to provide the clientId. What I'm going to do here is I'm going to use AsyncAPI bindings feature, which allow us to provide MQTT specific details, and clientId. I'm going to use social-network-server. We now have our servers.
Let's have a look at channels. This is the demo, and we're not interested in this. We're going to start by the root channel. Since WebSocket don't support multiple channels by default, unless you're using socket.io, or something like that, my only chance here is that we use the /channel, or we use something like whatever path we prefer. That means that we will have to be connecting that path whenever we connect. We're not going to do this. We're going to just use /, and that's it. This is our server. Our server will allow others, the client, the browser to publish a message to the server. We're going to call this operation onLikeDislike. Here, I'm going to use the message likeOrDislike, and I'm going to get rid of this for now, and I'll come back soon. Now I'm going to use likeOrDislike, and our likeOrDislike message is going to be a JSON object. The properties are going to be type. Type is going to be string, and description of this string is the type.
Since we cannot have multiple channels, we're going to simulate them over the same wire, or over the same channel. We need to know if this message is going to be a like or a dislike, something like that. It can only possibly have two values, either like or dislike. I know it's not strictly correct in English, the term dislike in this context, this will be something like undo like or something like that, or stop liking, if you prefer. Let's leave it like this. I think it's easier. We have one property, which is type, and we're going to have another property, which is the rest of the interesting data. For instance, we're going to have properties. This data object is going to contain something like for instance, what is the postId that has been liked? It's an integer, actually, and ID of the liked post, liked, unliked? Who is the ID of the user who liked or disliked the post? ID of the user who liked the post. We got something here right now. That will be something that's working. Let me open the Studio again. Yes, it looks like it's fine. I'm not going to use this editor for now. I'm just going to leave this view, which is the documentation view.
We instantly started seeing, as we were typing, immediate feedback on this place on the right side of studio, and is the recommendation. We are automatically getting the documentation with really little effort just by writing the AsyncAPI file. The cool thing is that you get samples and all this stuff. That's not only for documentation, and that's something we're doing here. I'm creating this AsyncAPI file, and I get the documentation, but I'm going to get a lot more things. For instance, I'm going to get the code actually, or a lot of validations in the code. The next thing that I would love to explain here is what happens when we like or dislike a publication, a post? If we like a post, the next thing that we want to do on the server is update the count on the database here. We're going to send a message to all the clients subscribed to the WebSocket server, notifying that there has been a change of likes on the likes count, some of the posts. All the clients can be notified, saying, on the frontend, please update your count so it can be reflected. Of course, that's an example. You should not be doing this on real life. That's what we're going to do here.
We're going to publish a message once we get this dislike. This message is going to be likeCountUpdated. Let me just come here, and let me work here, likeCountUpdated. The payload of this message is going to be something like, what do we need here? We need the ID of the post that has been updated and the new like count, the new total count. Let me just start then, type object, properties. This is a WebSocket event, so we need to have a type, which is the same as this one. We have a type field, which is a string, and there's going to be an update of count. We're going to have the data, type object. Properties here are going to be something like, postId, type integer, and description is going to be the ID of the post, and totalCount, which is going to be also an integer. Description is total count of likes for a post, something like this.
We're writing a lot. I think it deserves the effort to do this upfront. We're now saying that whenever the browser sends, and we want to compute something here, like the number of likes for a specific post, and then communicate all the browser instances connected to the server that the post has been liked, so they can update. Let me check before we continue. I think that's fine. It will have been updated here. Yes, it's updated here. We have our documentation here right now that you can publish this likeOrDislike, and you can subscribe to the count updated message with the following information. Here we have an example. We have that.
Another thing that we're going to do here is, once we have likeOrDislike, we're going to send a message to a MQTT topic. We're going to call this topic post/liked. Up here, we can follow the MQTT notation and that's fine, which is the same as AsyncAPI in this case. Since we're going to send that means that someone can subscribe to this message, so that we're going to use subscribe. That is highly confusing and we're going to change this in version 3.
Something that we want to do here is we want to do something like this. This message needs to be, notify the post has been liked, something like that. We need a new message. The new message, I'm going to create it here, notify the post has been liked. Since we have to specify that the post has been liked, the very least thing that we want to know is the postId. We need to know the ID of the post that has been liked and who liked this post. That should work. It validates correctly. You can see it here. People can now subscribe to post/liked event because our server is going to send it. That is here. We have two servers, WebSockets and Mosquitto, and we need to limit which channels exist where. For instance, this last channel doesn't exist on the Mosquitto broker, so we want to limit this to WebSockets. We say that this channel only exists on the WebSocket server. We're going to limit this to Mosquitto. That will be most of it. This is more or less the description of the server.
What happens here is that we started putting operation IDs on things like this, only this one for now. We have a functions server here. It has an onHello, by default, because it was coming with the example that was generated. We're going to rename it to onLikeDislike. It has to match exactly the name that we're giving it here. This function is going to be executed every time we receive a message on this specific channel. Every time we receive a message in the WebSockets channel, the only channel that WebSocket has, we're going to execute this function.
What is this function going to do? This function is going to actually do all this stuff, like we need to compute the number of likes, or save the likes. We need to store it on the database, all this stuff. I'm going to close this for now, since we're not editing the file in Studio anymore. I'm going to install a dependency that is called lowdb. This is a small dependency for working with local databases like this JSON file. One thing that we're going to do is I'm going to start importing it already. I think we don't have to wait for npm to finish again, and lowdb. Here it is. Let me check, how is this? I have the documentation right here. You cannot see it, but db.json. We create our JSON database here. What are we going to do now? The first thing that we want to do is we want to read whatever is on the database just so it's updated, and let's just understand what is inside the data. We have the number of likes there. The number of likes are stored here. This is what we're doing right now. So far, it's not a number, so it's an array of objects with userId and postId, so we're not going to store on plain numbers, we're going to store more information.
We're going to understand what event are we talking about? Is it like or dislike? Another thing that we'll need to know is the ID, and the userId. That is it. I have a few functions prepared here to compute all this stuff. Let me just also copy this function, this snippet that I have here. We need to find out if this like already exists in the database, before we compute it, and why, so we need this. If type of event is like, and the existing like index in the array is -1, it means that it doesn't exist. We're going to store the like on the database. If it's a dislike, and it exists already in the database, we remove it from the database, that's what it's doing. We have to save all the time that's why, using this. What's the total count of likes? It looks something like this. Likes, we need to filter every like by those who are about this specific postId. We need to count all the likes that are from this specific post. Let's get into the beauty of this. Something that we need to know is we're going to broadcast to all the instances, all the WebSocket instances. We got to broadcast that the count has been updated. Something that we want to do now here is we need to say that this is the name of our event. The data is the postId that we were obtaining here. Along with the postId, we're going to send the total count that we calculated here. That will be enough for now. That makes our WebSocket server to write on the database.
Let's see if it works. We're going to start this server, npm run dev. There seems to be a problem. I need to update to the latest Glee. That needs to be updated on the generator, but it's working. That's a good thing. It's complaining that Glee is not supporting version 2.2.0. Yes, indeed, it was created before this version was released. A few days ago I updated it so it supports version 2.2.0. I'm going to install this dependency, and then we continue. Let's check if it works now, npm run dev. You can see that we started Glee. It's running in development mode. It's only using the WebSocket server. This can be configured afterwards here. We're not going to use the Mosquitto server. We're not publishing to Mosquitto yet on the code, so we didn't do anything here yet, to Mosquitto. It's on the AsyncAPI file, but we didn't implement it yet. That's correct. That's fine.
If I go to the frontend, and refresh. If I start the frontend, obviously. We now have the frontend. Let me check. WebSocket client connected to server. Here we are. I'm going to like something just to see if it works. I'm not going to like my own post. I'm going to like the Lukasz one. There is a jump on the frontend, I don't know why. You can see it, it bumped the number. Let's see what will happen. We received a message from the backend saying likes_count_updated, postId 11, totalCount 1. It worked. Let me check on Lukasz side. Did it work? Yes, it did work. Let's also do like here, you can see it here. Perfect. Did it work here? Yes. It's here. We have it. It's working. We have a WebSocket server that is storing our information.
We're missing the last part. We have to communicate to the MQTT broker so it can communicate to the notification service and send to Slack. Let's go quickly to that part. I have something here prepared, so we don't spend more time. I'm going to explain it now. Let me go straight here. Something that we said is whenever we receive a message, a like or a dislike, we got to also update, not only the frontend, but we got to communicate that this event happened. We're going to do this only on the likes. We're not interested in the dislikes for now. Only in case this is a like event, we're going to send a message to post/liked channel, which is the post/liked topic in MQTT, with a payload of postId and userId. It has to go to the Mosquitto server specifically, not to all the servers, just to the Mosquitto one. That part should be done. We are sending the message that it's done, but it's not yet completely, because as you can see on the terminal, it doesn't say that it's connecting to Mosquitto. It's because we're not telling this to use Mosquitto as well. I'm going to update the .env file. I don't know if it caught the update. If not, I'm going to restart it again. Server WebSocket is running, and it's connected to Mosquitto. Perfect. Now it can send the message.
Let's see if it's doing what it's saying. I'm going to like this one. What happened? From WebSockets, we'll receive this event. It was sent to Mosquitto, and it was broadcast to WebSocket. That's all we need. Before we continue, let's go to notifications service and see if it's receiving the event or not. One thing that we need to do here, before we continue, is we need to create the AsyncAPI file for the service, so Notifications Service. Something that we need to get from here is the Mosquitto. This one is only connecting to Mosquitto, nothing about WebSocket. This is going to be notifications service, a different clientId. I hope this is not too long for MQTT. Let's just make it shorter, just in case.
What else? We need to subscribe to the post/liked. Since we're subscribing, it means that someone can publish to this application, and onPostLiked is going to be the name of the operation. For the message, we're not going to duplicate with what we have already. We're going to just leverage what we have on the server. You can do this or you can rely on other registries or GitHub repos for your messages, and for all your specific AsyncAPI stuff and definitions. For now, we're going to just do this. Let me just check that this file is valid. I think it's like this. This file is valid, successfully validated. We now have this. We have the AsyncAPI file, but we don't have the implementation yet. Whenever we receive a post/liked as we said here, we need to send a message to Slack. Always? Not really, not always. Let's just get rid of this, we're not going to send any message, just receive. I'm going to just copy what we have on the other function here to access the database. In a real microservices environment you shouldn't be sharing this database. This is a demo.
Let's just read the database. We need to know that you are liking other person's post. Otherwise, this wouldn't be doing the notifications, if you like your own post. I'm going to just get all the users and posts from the database. I'm going to get from the event payload that we're receiving on the MQTT broker, the postId and the userId, the user who liked the post. I'm going to get who's the owner of this post. I'm going to check from the list of users who's the post owner, and who was the user who liked the post. With all this information, all we got to do right now, is, we need to check if the postOwner.id is exactly the same one as userWhoLiked.id. In this case, we're not going to send any notification, we're going to say, "It's great that you like yourself but I'm not going to send a notification." Of course not. If you're liking other people's post, so, "Sending message to Slack." Now we need to make a call. For making the call, I'm going to just install node-fetch as a dependency. In the meantime, I'm going to just copy and paste the snippet that I have here. We need to access process.env.SLACK_URL. We're going to make a POST call to this SLACK_URL. We can send this information that someone liked your post. That is pretty much what we need to do.
The only thing missing right now is actually the SLACK_URL that is not in the .env file. We're going to add it here. This is not WebSocket, this is Mosquitto, and URL. Npm run dev, on the notifications service. Let's see if it works. ERR_MODULE_NOT_FOUND. We didn't install lowdb here. Something that we're missing here is importing fetch. Since I don't have any help on the code editor, it is not helping me saying that this is not being imported. Fetch is there already. We're going to call SLACK_URL. Let me just clear the database, all the likes. I'm going to remove all the likes. That should be before I continue. We don't have the WebSocket server running. We're going to start it. Connected. Frontend is in place. Notification service is in place. I hope that works now. Let me just refresh. We don't have any likes. I'm going to like my own post. Let me check. You see notification service is saying that it's great that you like yourself, but I'm not going to send a notification. It's working. I'm going to like Lukasz's post. Eventually, it will get it to notification service, sending message to Slack. Let's check Slack. We got our message here, "Hey, Lukasz, Fran Mendez liked your post." That was the demo.
The Initiative
This year, we joined the Linux Foundation. Now the project is in a neutral ground. We are growing super-fast, lately, thanks to Linux Foundation, and all the contributors that we have and the users that we have. Also, would love to clarify that even though this initiative started as an event-driven architectures initiative, and we're still there, we want AsyncAPI in the next five years or so to become the number one specification for defining and developing APIs, any APIs. That means that in the future, you should be able to define REST APIs, GraphQL APIs, gRPC APIs, and event-driven architectures, all with AsyncAPI. Don't get me wrong, we're not trying to reinvent the wheel here. It's by leveraging existing specifications, so you should be able to combine your OpenAPI file inside AsyncAPI and your GraphQL AsyncAPI being referenced from your AsyncAPI file, and so on. That means that you should be able to work with AsyncAPI tooling and with the AsyncAPI document referencing other specs. With a single process, you should be able to do all your API stuff.
As a shout out for all these companies using AsyncAPI in production, there are a bunch of them. Really, AsyncAPI is getting well tested, and being used heavily in production. Give it a try. We're a welcoming initiative and community, so join us. Join our Slack workspace or our GitHub. Even though you might not have much experience with APIs, or with event-driven architectures, we're a welcoming initiative and we welcome everyone to join us.
See more presentations with transcripts
