eBay Announces ql.io

 We are happy to announce – a declarative, evented, data-retrieval and aggregation gateway for HTTP APIs. Through ql.io, we want to help application developers increase engineering clock speed and improve end user experience. ql.io can reduce the number of lines of code required to call multiple HTTP APIs while simultaneously bringing down network latency and bandwidth usage in certain use cases. ql.io consists of a domain-specific language inspired by SQL and JSON, and a node.js-based runtime to process scripts written in that language.

• Most use cases require accessing multiple APIs – which involves making several network round trips.
• Often those API requests have interdependencies – which requires programmatic orchestration of HTTP requests – making some requests in parallel and some in sequence to satisfy the dependencies and yet keep the overall latency low.
• APIs are not always consistent as they evolve based on the API producers’ needs – which makes code noisier in order to normalize inconsistencies.

As the live demos on ql.io show, ql.io solves these pain-points by providing a framework that hides the complexity of stringing together api's with interdependencies manually or, for that matter, generalizing further, interacting with different subsystems. ql.io provides a functional/compositional SQL like query DSL, similar to LINQ or YQL.

InfoQ had a chance to spend some time with Subbu and try and get a sense for the capabilities and the vision behind the framework. Being careful not to type-cast the solution into a REST or a SOAP camp. The interview is an attempt to take an objective look at how ql.io solves the pain points of developers consuming Web API's and making them productive at consuming/repurposing the API's to create applications and mashups.

InfoQ: There seems to be an implicit need for trusted clients with something like this. Or atleast a way to govern and throttle bad players in the client side ecosystem. In other words, how does one set policies on api consumers; rate-limits, execution time limits to protect against long running queries; and how are these policies set up and enforced?

Subbu: Time limits etc are TBD as we are still figuring out what to limit. Long-running connections don't concern us as much as huge response payloads that strain v8's GC. Furthermore, the assumption is that ql.io instances are running in controlled environments with no ability to execute arbitrary scripts. ql.io is modularized such that only particular application specific URLs are exposed to clients. This allows each deployment to monitor and rate limit each URL using commodity tools that already know how to deal with HTTP

InfoQ: From an SOA perspective; client is driving the server side behavior and flow; which is considered an anti-pattern. Opinions?

Subbu: I would leave the discussion about whether this is a good pattern or bad pattern to the experts. The real problem is that optimizing an interface for every consumer is not practical and practicable. This is often a point of friction between teams. One way to solve this is to let consumers create a new tier of interfaces that are optimized for their usage. That's what ql.io does.

InfoQ: Obviously the investment is coming from ebay which means there is a good case for it; much of it is evident in the teaser post as well as the press release. But I’m curious, if there is some anecdotal caveats/benefits that you’re seeing dog-fooding this framework?

Subbu: This problem is generic and is felt by every client app developer that is building an app that relies on some HTTP API for data. I've seen instances where producer teams disagreed on optimizations that client developers so desperately wanted for the reason that those optimizations pollute their interface. Some of these are certainly valid concerns. ql.io intends to unblock consumer dev teams by providing a way to create those optimizations. What is not yet proven with ql.io is the case where the network is not the bottleneck such as between clients and servers on the same network. We're working on numbers to help guide usage.

InfoQ: Could you comment on some of the other technology stacks in the running? Why node.js stood out in comparison?

Subbu: IO workloads, persistent connections, evented programming style, as discussed in the blog post.

InfoQ: Are there plans on how to the framework plays with hypermedia? For example, one could think of a scenario where the result of the api call is a bunch of json objects and links to other "api's roots/tables" sources driven by hypermedia. i.e. still be able to drive decoupled behaviours on the client and server.

Subbu: This is possible today. We recently built an app that does a scatter gather operation based on URIs returned from one server - the feed has pointers to other machines in a large pool. We want to make it possible for scripts to operate on any part of the representation - headers or body, and links are part of this.

InfoQ: Related to that, and may be, this is a jumping way ahead, but this seems like something that inherently has some meta programming facility needs? In the sense, if the query script surfaces hypermedia in the results, then in order for the api consumer to navigate those links, there would need to be a means for the ql.io library to interpret the hypermedia and be able to transform it to another ql.io script that can be executed in subsequent queries.

Subbu: ql.io is doing it in such a way that it does not know what hypermedia is. How hypertext is surfaced, if at all, is impl specific.

ql.io is being built by eBay's platform engineering group, and is actively managed on github. See AUTHORS.md for the list of contributors. ql.io is released under Apache License, Version 2.0. See eBay Open Source for other project contributed by eBay. Check out ql.io on Github for the source, demos, examples, and docs.