Porting JQ, Command-Line JSON Processor, to the Browser with WebAssembly - Q&A with Robert Aboukhali

jq, the command-line JSON processor, originally written in C, has recently been ported to WebAssembly, and is now available in a browser JavaScript environment. InfoQ talked with Robert Aboukhalil, bioinformatics software engineer at Invitae, about the challenges of porting existing software to WebAssembly(wasm) and the ensuing benefits for developers.

jq, a command-line JSON processor, self-described as sed for JSON data, allows users to slice, transform and recombine structured data – with a friendly interface. The following extract from the result of a GitHub request:

  {
    "sha": "d25341478381063d1c76e81b3a52e0592a7c997f",
    "commit": {
      "author": {
        "name": "Stephen Dolan",
        "email": "mu@netsoc.tcd.ie",
        "date": "2013-06-22T16:30:59Z"
      },
      "committer": {
        "name": "Stephen Dolan",
        "email": "mu@netsoc.tcd.ie",
        "date": "2013-06-22T16:30:59Z"
      },
      "message": "Merge pull request #162 from stedolan/utf8-fixes\n\nUtf8 fixes. Closes #161",
      "tree": {
        "sha": "6ab697a8dfb5a96e124666bf6d6213822599fb40",
        "url": "https://api.github.com/repos/stedolan/jq/git/trees/6ab697a8dfb5a96e124666bf6d6213822599fb40"
      },
      "url": "https://api.github.com/repos/stedolan/jq/git/commits/d25341478381063d1c76e81b3a52e0592a7c997f",
      "comment_count": 0
    },
    "url": "https://api.github.com/repos/stedolan/jq/commits/d25341478381063d1c76e81b3a52e0592a7c997f",
    "html_url": "https://github.com/stedolan/jq/commit/d25341478381063d1c76e81b3a52e0592a7c997f",
    "comments_url": "https://api.github.com/repos/stedolan/jq/commits/d25341478381063d1c76e81b3a52e0592a7c997f/comments",
    "author": {
      "login": "stedolan",

can be queried as follows:

curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | jq '.[0]' | {message: .commit.message, name: .commit.committer.name}'

to extract the first element of the response, and keep only fields of interest:

{
  "message": "Merge pull request #162 from stedolan/utf8-fixes\n\nUtf8 fixes. Closes #161",
  "name": "Stephen Dolan"
}

To build a jq playground, two approaches are possible. The first approach involves setting up a sandboxed environment on a server that executes queries and returns the result to the user via API calls. This means hosting, securing, and sanitizing user inputs, and the introduction of latency due to round trips to the server. The second approach involve simulating the command line interface in the browser. This means a new JavaScript implementation of the logic of a tool which is already written and battle-tested. Porting to WebAssembly seems to be an attractive middle-ground. Aboukhalil explains:

These two solutions are not ideal because we have to choose between security and a meaningful learning experience. Ideally, we could simply run the command line tool directly in the browser, with no servers and no simulations. Lucky for us, WebAssembly is just the solution we need to achieve that.

The resulting jq playground is available at jqkungfu.com:

Aboukhalil details the multiple steps comprising a port to WebAssembly and shared with a correspondent valuable insights on the gained experience.

InfoQ: How did you come about porting jq to web assembly?

Robert Aboukhalil: I wanted to build a web app that would showcase the power of using WebAssembly to build interactive command-line playgrounds in the browser. By compiling jq from C to WebAssembly, we’re now able to run it in the browser, which is much faster, secure, and more convenient than hosting jq on a server that executes user requests in an isolated environment.

InfoQ: Any other interesting wasm ports you would recommend?

Aboukhalil: A really interesting one is a clone of Doom 3 that was ported to the web with WebAssembly: http://www.continuation-labs.com/projects/d3wasm/. Other popular apps that use WebAssembly include Autocad and Figma.

InfoQ: Tell us about your experience with WebAssembly. What challenges did you meet?

Aboukhalil: I first looked into WebAssembly as a way to speed up a web tool that analyzes DNA sequencing data (I wrote about it here for Smashing Magazine).
The biggest challenge I faced was that although I could clearly see that WebAssembly had a lot of potential, it also had a very steep learning curve because:

1. Most of the support is for lower-level languages like C, C++, and Rust. As a web developer, this was challenging because I hadn’t touched C in a decade!
2. Most of the documentation/tutorials available online seem either too simple (Hello World-level tutorials) or too complex (looking at in depth WebAssembly internals)

InfoQ: What do you recommend for developers seeking to integrate existing tools in the browser?

Aboukhalil: I’m a little biased about this, but I recently wrote a book about WebAssembly (available at http://levelupwasm.com) that I think offers a more practical intro into WebAssembly and dives into building full fledged applications with it.

InfoQ: Best practices, pitfalls?

Aboukhalil: Although WebAssembly is a very powerful tool, it’s not always the right tool for the job. For example, if your app needs a lot of RAM or if you require a lot of communication between the JavaScript and WebAssembly worlds, WebAssembly might not be the right tool for you.

InfoQ: What practical applications do you see for WebAssembly in your field, bioinformatics?

Aboukhalil: Here are a few examples of how WebAssembly can be leveraged in bioinformatics web apps:

1. Visualizing quality metrics in the browser. This is very useful for scientists who aren’t comfortable with using the command line and want a quick preview of their data without having to wait for a file to be uploaded to a server. I wrote a case study about this for Smashing Magazine here: https://www.smashingmagazine.com/2019/04/webassembly-speed-web-app/.
2. Creating exploratory playgrounds for commonly-used bioinformatics tools (e.g. like jqkungfu.com, but for bioinformatics), either as a learning tool or to quickly test the effect of changing parameter values on the fly, within an interactive interface.

The key benefits that WebAssembly provides are code portability and the potential for speeding up web apps. Since a lot of bioinformatics tools are already written in C, C++ or Rust, they are great candidates for porting them to the web with WebAssembly.