Shift Left Approach for API Standardization

There has been a growing trend among the organizations API team to better standardize their API design and development process. With an increased adoption towards micro services, software products being more and more just a bunch of micro-services and third-party APIs mashed together.So, it gets more crucial for us to get API structure in order using the de facto standard like OpenAPI (a.ka. Swagger). To achieve this consistency organizations have started to define their own guidelines for standardizing their API. 

What is API standardization

API design is the creation of an effective interface that allows you to better maintain and implement the API, while enabling consumers to easily use this API.

Consistent API design means, standardizing the design, across all APIs, and the resources they expose, within an organization or team. It is a common blueprint for developers, architects and technical writers to follow, to ensure a consistent brand and experience in the API consumption. Organizations standardize design using Style Guidelines that aim to ensure consistency in the way APIs are designed and implemented. Some popular style guidelines are shared below.

1. Microsoft REST API Guidelines
2. Google API Design Guide

I often refer to this stylebook for developing a consistent API for my side projects while following the industry best practices for API development.

Why Standardization

A clear design methodology ensures that APIs align with the business needs. With more standardized API, there’s less ambiguity, there’s more collaboration, quality is more ensured, and API adoption increases.

Having clear and consistent API design standards is the foundation for a good developer and consumer experience. They let developers and consumers understand your APIs in a fast and effective manner, reduces the learning curve, and enables them to build to a set of guidelines.

API standardization can also improve team collaboration, provide the guiding principles to reduce inaccuracies, delays, and contribute to a reduction in overall development costs. Standards are so important to the success of an API strategy that many technology companies – like Microsoft, Google, and IBM as well as industry organization like SWIFT, TMForum and IATA use and support the OpenAPI Specification (OAS) as their foundational standard for defining RESTful APIs.

Without standardization, individual developers are free to make subjective choices during design. While creativity is something to encourage it quickly can become chaos when not appropriately governed by a style guide.

Organizations cannot ensure quality within their API design and delivery process without standardization. Reinforcing design standards improves the ability to predict successful outcomes and contributes to an organization’s ability to scale their API development at speed while ensuring quality.

Journey into API standardization

It wouldn’t be possible to scale your API design and development processes successfully, or comply with regulatory and industry standards, without a formal process to reinforce standardization. Having an API design style guide provides the “guardrails” needed to let internal and external teams collaborate when building API definitions and re-using assets.

Initially, organizations start publishing their API guidelines internally as PDF or wiki for everyone to reference and processes are put in place to make sure teams are following the design guidelines.One solution to develop consistency is providing a manual review during the API development. 

The API’s are specified in OpenAPI format and maintained in version control, we can follow the same review process for API definitions that we follow for other code artifact. Developers can create pull requests for their API changes and have a colleague provide feedback.This manual process can be an effective way to ensure governance and compliance with the API guideline, but like all manual processes it is subject to human error and not always timely.

Waiting for a colleague to review our API change can result in a slow turnaround that hurts developer productivity, especially when it comes to aspects of the review process that can be automated. This process is also not scalable when organization scales and more developers start to develop API. This is where shifting left with automated API reviews is helpful. It’s always better to get feedback early with the help of some automated tools or linters like we do get for our other artifacts. 

What is Shift Left approach

The term “shift left” refers to a practice in software development in which teams begin testing earlier than ever before and help them to focus on quality, work on problem prevention instead of detection. The goal is to increase quality, shorten long test cycles and reduce the possibility of unpleasant surprises at the end of the development cycle—or, worse, in production.

Open API validators

When it comes to OpenAPI linters, I came across a few linters. These linters convert the API style guidelines into a set of rules and validate against the Open API specification. These linters can provide you an option for customizing the rules as per your organization style guide. One tool which caught my attention was linter called Zally which was written using Kotlin and open sourced by Zalando. OpenAPI style guide validators workflow looks like below.

1. The API standard or style guidelines are expressed as a set of rules. Zalando has one such guideline here.
2. API Written according to OpenAPI specification
3. Linting tools such as Zally, sonarQube, Spectral that can validate that the OpenAPI specification the developer has written is complying with the specification’s rules defined in step 1.

What is Zally

Zally is a minimalistic and easy to use API-linter. Its standard configuration will check your APIs against the rules defined in Zalando’s RESTful Guidelines, but anyone can use it out-of-the-box. It’s written in a more extensible way to allow us to add our own set of rules as well. It also provides the following feature.

• enable/disable rules on the server side based on your needs
• Accepts both json and yaml format for Swagger V2 and OpenAPI V3 specification
• Write and plug your own rules
• Intuitive Web UI that shows implemented rules and result of your spec validation
• Github integration using web hooks which validates your OpenAPI on each pull request and echo back the violation in the comments

Motivation behind Zally Gradle plugin

Though Zally is written in a more extensible and customizable way, I felt that we can still improve zally current validation workflow further to reduce the developer feedback loop. Since Zally lacks the plugins like checkstyle, ktlint, spot bug etc,. Below are a few pain points I experienced when I used Zally. 

• Developers need to host the Zally server either locally or in a remote system to use the CLI tool.
• Developers need to switch context for running CLI tools or some additional work needed to configure the CLI execution as part of the maven/gradle build process with prerequisite of point 1. 
• Using github integration components to validate our API for each pull request increases the feedback loop time.

All these options are increasing the feedback time for the developers with some manual overhead of hosting the Zally server. So I decided to write my own gradle plugin which can be integrated in the local development environment as well as in the CI tool which helps me to validate and extract the validation result in different formats. .

Custom Zally plugin

zally-gradle-plugin is a gradle plugin which is written in kotlin and can be integrated in the build script. The plugin validates the specification against the set of rules and provides the report in both json and html format.

The project includes an example task configuration:

```
// settings.gradle.kts
pluginManagement {
    repositories {
        gradlePluginPortal()
        mavenLocal()
    }
}

// build.gradke.kts
plugins {
    id("io.github.thiyagu06") version "1.0.2-dev"
}

zallyLint {
    inputSpec = File("${projectDir}/docs/petstore-spec.yml")
    reports {
        json {
            enabled = true
            destination = File("${rootDir}/zally/violation.json")
        }
        rules {
            must {
               max = 10
            }
        }
    }
}

//execute task
./gradlew clean zallyLint

```
```
Run ZallyLint task
./gradlew zallyLint
```

With this gradle plugin I’m able to get real time feedback during API development. This enables me to fix issues with the API before getting into a manual review step. The plugin can also be integrated with the CI jobs to check to validate the style guidelines. Because all development teams use the same rules, the organization can provide a more consistent API for their users. The benefits of the approach are outlined below.

The plugin provides an option to enable exporting violation reports into JSON and HTML format. This also provides an easy way to configure rules to define the max number of violations allowed in the spec for each severity level.

The json format can be parsed and exported into any database to calculate the API design compatibility score and build a dashboard to share it to the wider organization for decision making on the API standardization initiatives. The same way HTML reports can be exported to S3 bucket or google cloud storage and hosted as a website to a broader audience.