Ballerina Tutorial: A Programming Language for Integration

Ballerina is an approach to make integration and microservices programming simpler by co-designing a language and a platform to be both agile and integration simple. Ballerina was started 3 years ago by architects from WSO2 as a response to challenges they experienced in building integration flows for EAI, ESB, and workflow products like Apache Synapse and Apache Camel.

What is it like to author integration flows in Ballerina that are deployable as scalable microservices on Kubernetes?

This tutorial will create a service that asynchronously publishes Homer Simpson quotes published from a hosted REST service to your Twitter account. The integration will have a circuit breaker for dealing with the unreliable Homer Simpson service and perform data transformations between payloads.

This is part 2 of a series on Ballerina. The first article, "Ballerina Microservices Programming Language: Introducing the Latest Release and "Ballerina Central", provides an introduction to the language concepts.

Getting Started

Please get Ballerina installed and added to your path. The native platform installers manage the path settings for you.

I recommend that you code with Visual Studio Code as Ballerina offers a language server extension that provides rich recommendations, intellisense, and a debugger.

To interact with the service that you author, curl should be installed.

For the Kubernetes deployment, you’ll need Docker and Kubernetes installed. This tutorial uses Docker Edge as its Kubernetes setup is simple, even for Windows. Verify that kubectl get pods does not have any resources deployed.

The connector to Twitter will require your Twitter API keys and tokens. Set up a Twitter account and grab the four values by setting up a Twitter app. Place the four values into a twitter.toml file.

# Ballerina tutorial config file with Twitter secrets
clientId = ""
clientSecret = ""
accessToken = ""
accessTokenSecret = ""

The Tutorial Sequence

We’ll build our integration with iterative improvements.

1. Part 1: Create a Service with a POST Resource. We’ll accept a string from the client and return it as part of our response.
2. Part 2: Tweet By Invoking a Twitter Connector. We’ll search, discover and install a Twitter connector which will be used to post the string payload from the client to your Twitter feed.
3. Part 3: Circuit Breaker Management of Homer Simpson Quotes From an External Service. We’ll source strings using GET requests to an unreliable external service and then add a circuit breaker to deal with timeouts and error conditions.
4. Part 4: Deploy Service to Kubernetes with Observability & Tracing.

Part 1: Create a Service with a POST Resource

Let's create a service with a single resource that is accessible with a POST invocation. The payload will contain a string that we will return to the caller.

Create a file, homer.bal:

// The `http` package which is part of the standard library 
// in the default distribution. This package contains objects
// annotations, functions, and connectors which are reusable
// in code bodies below with the `http:` namespace reference.
import ballerina/http;

// Add this annotation to the service to change the base path
// from `/hello` to `/`. This annotation came from the 
// package that we imported.
@http:ServiceConfig {
  basePath: "/"
}
service<http:Service> hello bind {port:9090} {
// A `service` represents an API that implements a particular
// protocol and listens against an `endpoint`. In this case,
// the service is named “hello” and bound to an anonymous
// endpoint on port 9090.

  // Add this annotation to the resource to change its path
  // and to only accept POST invocations
  @http:ResourceConfig {
      path: "/",
      methods: ["POST"]
  }
  hi (endpoint caller, http:Request request) {
  // This is a “resource” of the service. Each resource is an
  // invocable API point. An `endpoint` represents a networked
  // location. HTTP service resources all have the invoking
  // client represented as an `endpoint` passed as a parameter.
  // This service is also an `endpoint`, acting as a listener.

      // Extract the payload from the incoming request.
      // getTextPayload() returns a union type of (string | error)
      // `check` is an operator that says, “if the return is
      // a string, then assign it to the variable on the left, otherwise
      // propagate the error up the stack.
      string payload = check request.getTextPayload();
      
      // This is a data structure in the `http` package that
      // is used to preparing a payload to send in return. 
      http:Response res;
      res.setPayload("Hello "+payload+"!\n");


      // Send the payload response to the client that originally
      // invokes our resource. The `->` is special syntax that
      // indicates the invocation is a network-bound call. The
      // `_` syntax means to ignore any response or error.
      _ = caller->respond(res);
  }
}

You can compile and run this service at the same time with the "run" command.

$ ballerina run homer.bal

ballerina: initiating service(s) in 'homer.bal'
ballerina: started HTTP/WS endpoint 0.0.0.0:9090

$ # You can optionally build a linked executable into a .balx and run it separately
$ ballerina build homer.bal
$ ballerina run homer.balx

In a different console run curl to execute the service. The resource would normally be reachable at /hello/hi reflecting the names provided in our code. However, we used annotations to decorate the code and our resource is reachable now at '/'.

$ curl -X POST -d "Ballerina" localhost:9090
Hello Ballerina!

There a few interesting concepts:

• Services are first class constructs that can be bound to different protocols. A service is a type of entrypoint that the compiler will package into a server for deployment. Ballerina has support for other entrypoint types including a console-based main(...) function.
• Ballerina has support for a variety of protocols such as WebSockets, gRPC, and JMS. The signatures of resource methods vary by the protocol the service is bound to. There are a lot of samples in the Ballerina By Example.
• Endpoints are also first class constructs that represent any networked endpoint, whether it’s the service we are authoring, the client that calls us, or services we will remotely invoke. Endpoints are data structures that have initializers and embedded invokable functions specific to the endpoint type.
• Annotations are attachable to various objects. Different annotations attach to different objects such as endpoints, services, or resources. The compiler and build system generate artifacts and alter the behavior of the built service based upon annotations. Third parties and ecosystem vendors can add their own annotations and compiler extensions that act on them.
• Packages are reusable modules of annotations, connectors, data structures, and functions. There are a variety of packages as part of the standard library, but developers can create their own packages and push them into a Ballerina registry. Ballerina Central is a free, shared registry for Ballerina packages.
• Ballerina has a strongly typed type system with a variety of primitives including json, xml, and table. Union and tuple types are part of the language and make network programming simpler by providing a type structure that reflects the reality that many network-received payloads can be in many different forms, each represented by a different type.
• Ballerina’s syntax forcibly makes a distinction between local invocations using dots "." versus networked invocations using arrows "->". Ballerina’s syntax can be inferred to represent a sequence diagram without the developer modifying or annotating the code. Sequence diagrams are how integration specialists communicate the structure of their workflows and developers who write Ballerina syntax are able to get graphical sequence diagrams within VS Code or Ballerina Composer, a provided editor that ships with Ballerina.
• Ballerina compiles into its own byte code and executed within its own VM that contains a custom scheduler. In JVM languages, threads are mapped to classes. However, Ballerina’s internal threading model uses the concept of a worker to represent a unit of work. Workers are 1:1 mapped to threads and Ballerina’s scheduler optimizes thread execution to ensure there is never any blocking behaviors unless asked for by a developer. With arrow syntax "->" Ballerina’s scheduler releases the worker that makes the invocation and when a response comes back from the invocation, Ballerina’s scheduler allocates a different worker to handle the response. While the service’s client blocks during the execution of the networked call, behind the scenes there are two separate workers that are used and the overall system treats it as a non-blocking call and response.

Part 2: Tweet By Invoking a Twitter Connector

Let’s take the payload from the client and tweet it to your Twitter account. We must modify our code to:

1. Import and use a Twitter package which has a connector that provides functions for simple invocations.
2. Pass our personal Twitter secrets from a twitter.toml configuration file to the Twitter connector.
3. Extract any status information from Twitter’s response and place that into the response payload that we provide to our caller.

A Ballerina connector is a code object that allows you to execute actions against an endpoint. Connectors are available within shared modules that can be imported into code. You can discover available packages at Ballerina Central (central.ballerina.io) or on the CLI.

$ ballerina search twitter
Ballerina Central
=================
|NAME             | DESCRIPTION                     | DATE           | VERSION |
|-----------------| --------------------------------| ---------------| --------|
|wso2/twitter     | Connects to Twitter from Ball...| 2018-04-27-Fri | 0.9.10  |

We can download this package on the command line by pulling it. It’s similar to how images are moved from a remote registry to your local machine. Ballerina maintains a home repository which caches any downloaded packages.

$ ballerina pull wso2/twitter

Update your homer.bal:

import ballerina/http;
import wso2/twitter;
import ballerina/config;

// twitter package defines this type of endpoint
// that incorporates the twitter API.
// We need to initialize it with OAuth data from apps.twitter.com.
// Instead of providing this confidential data in the code
// we read it from a toml file.
endpoint twitter:Client tweeter {
  clientId: config:getAsString("clientId"),
  clientSecret: config:getAsString("clientSecret"),
  accessToken: config:getAsString("accessToken"),
  accessTokenSecret: config:getAsString("accessTokenSecret"),
  clientConfig:{}  
};

@http:ServiceConfig {
  basePath: "/"
}
service<http:Service> hello bind {port:9090} {

  @http:ResourceConfig {
      path: "/",
      methods: ["POST"]
  }
  hi (endpoint caller, http:Request request) {
      http:Response res;
      string payload = check request.getTextPayload();

      // transformation of request value on its way to Twitter
      if (!payload.contains("#ballerina")){payload=payload+" #ballerina";}

      twitter:Status st = check tweeter->tweet(payload);

      // transformation on the way out - generate a JSON and pass it back
      json myJson = {
          text: payload,
          id: st.id,
          agent: "ballerina"
      };

      // pass back JSON instead of text
      res.setPayload(myJson);

      _ = caller->respond(res);
  }
}

Go ahead and run it and this time pass a configuration file that has your tokens and secrets:

$ ballerina run --config twitter.toml demo.bal 

You would execute the same curl as before and the response will contain the Twitter status.

$ curl -d "My new tweet" -X POST localhost:9090
{"text":"My new tweet #ballerina","id":978399924428550145,"agent":"ballerina"}

And in your Twitter feed you should see:

There are some new concepts:

• Just like system packages and the standard library, third party packages can be imported and used to provide connectors.
• The endpoint keyword is used to create an object that is initialized with a connector. The type of connection is provided by the twitter:Client object which comes from the imported package. The initializer for this object requires a number of parameters which provide the tokens and secrets which will be used with Twitter. The tweeter object has global scope and can be used within the resource invocation block.
• The Twitter package provides a tweet function that can be executed with a Twitter connection. This is invoked with tweeter->tweet(...) in the resource block.
• JSON and XML are primitives within the language. We are able to use strongly typed primitives to map data from networked payloads into data structures. With an ESB, data transformation of this nature usually requires XPath or another query language.

Part 3: Circuit Breaker Management of Homer Simpson Quotes from an External Service

Let’s source the body of the tweet we send with Homer Simpson quotes provided by another external service with an HTTP REST API that it provides. This service has been implemented unreliably and while most responses are immediate, every so often responses are very slow, perhaps due to an overwhelming amount of traffic. We’ll add a circuit breaker that will prevent our service from calling the Homer Simpson API for a period of time if there are error conditions or it takes too long to response.

Update homer.bal with:

import ballerina/http;
import wso2/twitter;
import ballerina/config;

// This endpoint is the connection to the external service.
// The circuit breaker is a configuration parameter of the connection.
// The circuit breaker will flip with certain error codes or a 500 ms timeout.
// The circuit breaker flips back to original state after 3 seconds.
endpoint http:Client homer {
 url: "http://www.simpsonquotes.xyz",
 circuitBreaker: {
     failureThreshold: 0,
     resetTimeMillis: 3000,
     statusCodes: [500, 501, 502]
 },
 timeoutMillis: 500
};

endpoint twitter:Client tweeter {
 clientId: config:getAsString("clientId"),
 clientSecret: config:getAsString("clientSecret"),
 accessToken: config:getAsString("accessToken"),
 accessTokenSecret: config:getAsString("accessTokenSecret"),
 clientConfig: {} 
};

@http:ServiceConfig {
 basePath: "/"
}
service<http:Service> hello bind {port: 9090} {

 @http:ResourceConfig {
     path: "/",
     methods: ["POST"]
 }
 hi (endpoint caller, http:Request request) {
     http:Response res;
     
     // use var as a shorthand for http:Response | error union type
     // Compiler is smart enough to use the actual type
     var v = homer->get("/quote");

     // match is the way to provide different handling of error vs normal output
     match v {
         http:Response hResp => {

             // if proper http response use our old code
             string payload = check hResp.getTextPayload();
             if (!payload.contains("#ballerina")){payload=payload+" #ballerina";}
             twitter:Status st = check tweeter->tweet(payload);
             json myJson = {
                 text: payload,
                 id: st.id,
                 agent: "ballerina"
             };
             res.setPayload(myJson);
         }
         error err => {
             // this block gets invoked if there is error or if circuit breaker is Open
             res.setPayload("Circuit is open. Invoking default behavior.");
         }
     }
     _ = caller->respond(res);
 }
}

Build the code and run it, but to execute and demonstrate the circuit breaker, you will need to run it multiple times.

$ curl -X POST localhost:9090
{"text":"Marge, don't discourage the boy! Weaseling out of things is important to learn. It's what separates us from the animals! Except the weasel. #ballerina","id":986740441532936192,"agent":"ballerina"}

$ curl -X POST localhost:9090
Circuit is open. Invoking default behavior.

$ curl -X POST localhost:9090
Circuit is open. Invoking default behavior.

$ curl -X POST localhost:9090
{"text":"It’s not easy to juggle a pregnant wife and a troubled child, but somehow I managed to fit in eight hours of TV a day.","id":978405287928348672,"agent":"Ballerina"} 

And yet even more interesting learnings:

• Ballerina has union types, which is a type that can take on different types. This is a good way to represent networking concepts as APIs frequently can send back data in different formats for a single response and that data is best represented as different types.
• Ballerina supports a generic var data type which can have any value assigned to it. The match keyword is a block that will execute sub-blocks based upon the type of the variable provided. This acts as a type of branching logic to have different blocks for each possible type in a union type.
• The circuit breaker is configured as part of the connection to the Homer service. This approach to applying a circuit breaker is one example of how Ballerina uses compile time abstractions for distributed systems primitives.

Part 4: Deploy Service to Kubernetes with Observability and Tracing

Now, what kind of cloud-native programming language would it be without native support for the modern microservices platforms? Annotations can be added to Ballerina source files to trigger build-time tasks that create packages for Docker, Kubernetes or environments that you define. Ballerina’s annotation system is customizable and you can author additional builder extensions that operate against the source code tree or annotations to generate artifacts post-compilation.

Update homer.bal one more time:

import ballerina/http;
import wso2/twitter;
import ballerina/config;

// Add kubernetes package
import ballerinax/kubernetes;

endpoint twitter:Client tw {
  clientId: config:getAsString("clientId"),
  clientSecret: config:getAsString("clientSecret"),
  accessToken: config:getAsString("accessToken"),
  accessTokenSecret: config:getAsString("accessTokenSecret"),
  clientConfig:{}  
};

// Now instead of inline {port:9090} bind we create a separate endpoint
// We need this so we can add a Kubernetes annotation to tell the compiler
// to generate a Kubernetes service to be exposed to the outside world
@kubernetes:Service {
  serviceType: "NodePort",
  name: "ballerina-demo" 
}
endpoint http:Listener listener {
  port: 9090
};

// Instruct the compiler to generate Kubernetes deployment artifacts
// and a Docker image from this Ballerina service
@kubernetes:Deployment {
  image: "demo/ballerina-demo",
  name: "ballerina-demo"
}
// Pass our config file into the image
@kubernetes:ConfigMap {
  ballerinaConf: "twitter.toml"
}
@http:ServiceConfig {
 basePath: "/"
}
service<http:Service> hello bind listener {
  @http:ResourceConfig {
      path: "/",
      methods: ["POST"]
  }
  hi (endpoint caller, http:Request request) {
    // No change to our resource code
  }
}

That is it - let’s go ahead and build it:

$ ballerina build demo.bal
@kubernetes:Service                      - complete 1/1
@kubernetes:ConfigMap                    - complete 1/1
@kubernetes:Docker                       - complete 3/3
@kubernetes:Deployment                   - complete 1/1

It will create a folder called kubernetes with deployment artifacts for your service and the Dockerfile used to create the Docker image:

$ tree
.
├── demo.bal
├── demo.balx
├── kubernetes
│   ├── demo_config_map.yaml
│   ├── demo_deployment.yaml
│   ├── demo_svc.yaml
│   └── docker
│       └── Dockerfile
└── twitter.toml

And you can deploy it to Kubernetes:

$ kubectl apply -f kubernetes/
configmap "hello-ballerina-conf-config-map" created
deployment "ballerina-demo" created
service "ballerina-demo" created

Let’s see if it is running and find which external port is mapped to our internal service running on port 9090. In this example, port 9090 is mapped to 31977 externally:

$ kubectl get pods
NAME                              READY     STATUS    RESTARTS   AGE
ballerina-demo-74b6fb687c-mbrq2   1/1       Running   0          10s

$ kubectl get svc
NAME             TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)  AGE
ballerina-demo   NodePort    10.98.238.0   <none>        9090:31977/TCP  24s
kubernetes       ClusterIP   10.96.0.1     <none>        443/TCP  2d

And we can invoke our service using the external port.

$ curl -d "Tweet from Kubernetes" -X POST  http://localhost:31977
{"text":"Tweet from Kubernetes #ballerina", "id":978399924428550145, "agent":"Ballerina"}

What Else Is In Ballerina?

Ballerina’s language is robust and includes syntax for a wide range of logic and integration concepts.

• Networked Type System. Ballerina has support for arrays, records, maps, tables, union types, optional types, nil lifting, tuples, any types, and type conversion.
• Concurrency. Ballerina’s parallelism happens with workers and your services and functions can spawn workers, execute functions asynchronously, and use conditional fork/join semantics.
• Streaming. In-memory objects that enable message passing and a `forever {}` block that makes it possible to write streaming SQL to process a never ending stream of incoming events.
• Projects. Cross-developer collaboration with dependency management, versioning of packages, build orchestration, and shared package management within a registry.
• Integration Framework. Extensions and constructs to make integration simpler including redirects, query paths, HTTP caching, chunking, mutual SSL, multipart requests and responses, HTTP/s, WebSockets, header-based routing, content-based routing, retries, gRPC, and WebSub.
• Testerina. A provided unit test framework for executing tests against services with guaranteed execution order, mocking, and grouping capabilities.

About the Author

Tyler Jewell is CEO of WSO2, the largest open source integration provider, and a partner at Toba Capital. He founded and ran Codenvy, a cloud DevOps company, which was acquired by Red Hat in 2017. As a venture investor, angel, and board member he's lead $100 million in placements for DevOps companies including WSO2, Cloudant (acquired by IBM), Sauce Labs, Sourcegraph, ZeroTurnaround (acquired by Rogewave), InfoQ, and AppHarbor (acquired by Microsoft). Previously, Tyler was at Oracle, Quest, MySQL, and BEA where he contributed to three books on Java.