Restify and Mobilize Your Data

Enterprises are striving to make their corporate data available on mobile devices via RESTful interfaces, but building mobile apps and REST APIs takes serious time.

We examine two technologies that significantly reduce time to market. Executable Schema (builds on MDD and Convention over Configuration) creates a RESTful API and multi-table UI from existing schemas - literally in minutes. Declarative Behavior for logic and security bring spreadsheet-like power and simplicity - backed up by a fully programmable JavaScript model.

In this piece, we’ll briefly review existing key technologies, to build on their advances and to benefit from lessons learned in the marketplace. We’ll then provide specifics for Executable Schema and Declarative Behavior.

Current Approaches

The promise of accelerating delivery time is enticing and has spawned some great work over the past decade. First, let’s briefly examine the current approaches to draw some lessons learned.

Model Driven Development

Approaches such as Model Driven Architecture¹ and Model Driven Engineering² have met with some success in constructing partial elements of systems. And while promising, the real opportunity lies in:

• Building executable systems directly, rather than elements (e.g, Beans, Transfer Objects) that require understanding and integration
• Leveraging Cloud/Mobile technology
• Eliminating model/schema redundancy and synchronization complexity

Convention over Configuration: Ruby, Grails…

The notion of generating executable systems has been taken up by popular products such as Ruby, Grails, ROO, etc, that provide screen and code generation from models using Convention Over Configuration³. Here is a simple Grails example:

And while it’s great to build Web Apps quickly, significantly greater value would be achieved if:

• The apps were substantially more complete – for example with Master/Details (show list of Lineitems for Order) updatable grids (update the Lineitems in place) and automatic joins to show product names instead of numbers
• Support for multi-table business logic were provided (update and security business logic is tedious and verbose)
• The apps were deployed onto cloud technology - REST and HTML5/JavaScript
• Code generation issues were avoided – for example regeneration and loss of customizations when the originating input (the data model) changes

Mobile Backends

A new category of Mobile Backend as a Service (MBaaS) is focused on cloud technology, such as REST / JSON. Products like DreamFactory and SlashDB create a complete RESTful API for your SQL data, with convenient table browser User Interfaces.

While it’s certainly a start, imagine the value if you could:

• Extend the table browsers to full, multi-table applications with master/details, navigations, etc.
• Provide for business logic, to ensure updates trigger the the proper computations and validations
• Provisions for Custom Multi-Table Resources, with alias/projection

Proposed Approach

These approaches could be significantly enhanced by introducing notions like 

• Executable Schemas, where starting with the schema as the model, you could create a REST/Cloud service that pre-supplies the common behavioral patterns for Multi-Table UI and API - an instantly executable system. 
• Declarative, business-oriented behavior could provide declarative support for API extensions (such as multi-table resources), logic and security behavior. The real value is delivering these behaviors at a very high level of abstraction, more like business-oriented requirements than code. 
• And fully integrated imperative language support in a standard language could provide for procedural events integral to the model, using a standard language such as JavaScript.

So, the development team can operate and deliver at a near-business level of abstraction, with meaningful increases in speed. They can translate BDD stories into running software to confirm understanding and uncover the next set of behaviors, resulting in dramatically reduced miscommunication since stakeholders are seeing running software and can read the “code.”

Executable Schema

If we are to meet the business demands for instant results, we need to be running right out of the box. That is, just by connecting our system to a schema we should have a completely executable API and Application as described below.

Default Application

It is quite possible to build a richly functional mobile-ready back-office app. The presumption here is that while “front office” apps require a handcrafted UI, there is a large class of data maintenance apps that follow a set of well-known patterns, and that these can be created from the schema.

The sample screens below illustrates these common patterns for a fully functional data maintenance application:

• Search / Filter, with paginated List / Form view, sortable column headers
• Foreign Keys drive Multi-Table application behavior:
  • Master/Detail (Orders of Customer)
  • Drill-down Navigations (Screen 2 – Order with Items, via the magnifying glass or the zoom button at the lower right)
  • Automatic Joins: show Product Name, not Number
• Update support, including updatable grids, lookups (find part), and error handling

The join functionality is interesting, because it is quite common. In the example below, the Product foreign key is a Product Number, not Name - a very common pattern. In the ideal case, the system would recognize this case (numeric foreign keys, e.g., Product Number) and automatically join in a more suitable column (the Name). We would have not only an executable schema, but an instantly useful one as well.

(Click on the image to enlarge it)

(Click on the image to enlarge it)

Default REST API

The architecture of choice for cloud / mobile apps is a RESTful API. Such an API not only provides mobile access, but also provides network capable web services that can be accessed by other systems.

And the mapping from database to REST is straightforward… just by connecting:

• Create a REST Endpoint for each base/view table, and Stored Procedure
• Provide Get/Put/Post/Delete support (familiar CRUD operations), with support for
  • Filtering and sorting
  • Well-known patterns such as pagination, discovery for structure and related data, and optimistic locking

The result is a “flat” relational API. A good start, but we’ll need to extend it as described below.

Declarative Behavior

While we are proposing that the model not replicate the schema, there are clearly requirements to extend it for the key behaviors below.

Rich REST API: Multi-Table Resources, Alias, Projection

Our “flat” API is a good start, but a richer “document oriented” API is required. This requirement is both as a convenient API model for mobile developers and to minimize latency due to multiple queries.  

Such objects are somewhat like relational views, with support for aliasing and projection. However unlike views, they are not flat; they support nested “Sub Resource” objects, such as an Order object with LineItems and Shipment information.

Such “multi-table views” might be defined via point and click or syntactically such as:

Create Resource OrdersResource 
      from Orders (id as orderNumber, customer.name as customerName), 
      SubResource Items as LineItems via <fkName> 
         (name as product.name, qty as qtyOrdered...) 
      SubResource Shipment as Shipents via <FkName> 
         (...) 

Logic: Spreadsheet-Like Reactive

Business Logic is a significant element of most systems. It is tedious to code, consisting of significant amounts of change detection and propagation logic, SQL (with caching for performance), etc.

The prevailing assumption is that such logic is domain specific (true) and that it must therefore be coded in an imperative, procedural language (false). A declarative approach for such logic would be a significant advance in accelerating software delivery and maintenance. And there is such a solution.

Reactive is currently getting attention in handling UI events, like sync model to view. But the most common use is the spreadsheet; cell formulas react to changes in referenced values by recomputing themselves, which of course can chain. While simple enough for non-programmers, spreadsheets provide sufficient power to address non-trivial complexity.

Reactive fits very well for database transaction logic:

1. Assign expressions to database columns, or “virtual columns” (these become part of the model for transaction processing, but are not physically stored)
2. RESTful Put/Post/Delete operations are watched for changes in referenced values, which trigger adjustments to the referencing value
3. Validations are addresse`d as well (balance < creditLimit).

Most interesting transactions are multi-table, so it follows that our expressions be able to reference related data (customer.balance := sum(orders.amount_total where paid = false). This enables the system to take responsibility for not only the change detection, but also persistence (reading / writing related data, caching, optimizations, etc.).  

Note that encapsulating logic to columns results in inherent re-use. In the example above our declarative system watches and responds to changes in the orders’ amount_total or paid flag.

Declarative logic is by definition un-ordered. So the system must deduce the execution order, for example via a dependency graph. This has substantial value in maintenance. Automatic ordering means you add / change logic without worrying about the details of execution order.

By eliminating the dependency management, propagation and SQL handling, meaningful advances in conciseness and readability are realized. Meaningful here means in excess of an order of magnitude, where the following logic - virtually a requirements spec - being directly executable:

lineitem.amount = row.product_price * row.qty_ordered 
lineitem.product_price = copy(row.product.price) 
orders.amount_total = sum(lineitem.amount) 
customer.balance = sum(orders.amount_total where paid = false) 
validate customer as row.balance <= row.credit_limit 

Of course there is no silver bullet. Not all logic can be declarative, so critical provisions are required for proper integration of procedural logic. These are described below.

Security

Another key requirement of most systems is security. Beyond familiar endpoint access, the system must enforce security at the row and column instance level. For example, you might want folks to see Orders only in their own region.

Many systems provide such functionality in views, but that leads to a proliferation of views to define and maintain. A better approach is to encapsulate the security into the table as a role-based permission and ensure it is reused across Multi-Table Resources.

Role SalesRep authorized for Orders 
     Columns(orderDate, …) 
     Rows (region = $user.region) 

The notation $user is meant to suggest mechanisms to associate named values (eg., region) with users that can be as parameters in filter expressions.

Procedural Extensions

Executable Schemas and declarative behavior can provide value, but only if they are integrated with familiar procedural extensions so there are no boundaries.

Server-Side JavaScript

It’s straightforward to use the schema to create a JavaScript row object type for each table, with accessors for columns and related data. Much like Ruby Active Records, these are persistence-aware and logic-aware. They ensure the execution of the proper reactive logic as updates are processed.

In addition, they can publish events you can handle in server-side JavaScript. Events are a familiar model (before insert, before update) and provide full access to JavaScript (send messages, start processes, etc). JavaScript is an excellent language choice since it’s a language that’s part of virtually every system, therefore reasonably familiar to most developers.  

UI Creation

Extensibility must also be addressed on the client side. Default UIs are powerful and useful, but most appropriate for back office applications. For front-office applications there must be provisions to develop any User Interface.

There are two reasonable approaches. First, the Default UI can be forward engineered (code generated) into your favorite platform / IDE, so you can use it as a starting point.

Alternatively, it is reasonable to turn the model inside out and craft the default UI objects as components that can be snapped into a larger app as parameterized iFrames. This retains the model linkages, avoiding the “you own it” implications of code generation.

Custom APIs

Finally, declarative API creation in no way obviates the capacity and need to build custom APIs.

Summary

We have outlined here a realistic approach for:

• Instant creation of a mobile app and a default RESTful API from an existing schema
• Extending the API to define multi-table hierarchical resources
• Adding behavior using declarative “reactive” expressions to govern update logic and row access, with the simplicity and power of a spreadsheet
• Integrating server-side JavaScript for complex behavior

Author’s Note: Espresso Logic has been executing on this vision and now provides many of the elements described above. If you’d like to check it out, we offer a free eval with zero install that you can try on your own database. And we’re always eager to exchange views about how we can improve the technology!

References

¹ The OMG effort

²Model Driven Engineering (see this, this and this)

³Convention Over Configuration

About the Author

Val Huber has over 20 years of experience building technology to accelerate the development and maintenance of business applications. Prior to Espresso, Val was the CTO of Versata; a company that was one of the top 5 IPOs of 2000 and was initially funded by visionaries such as Paul Allen and Hasso Plattner of SAP. At Versata, he led the development of a J2EE development environment for large-scale transactional and workflow applications. Versata supported tens of thousands of concurrent users and was deployed by many Fortune 500 firms including ADP, Equifax, Fidelity, IBM and Sears. Val worked as an Architect at Sybase (now part of SAP) before Versata. He was also the driving force behind Wang’s highly regarded PACE (Professional Application Development Environment). PACE provided a relational DBMS, with a fully automated tool suite for reports and interactive applications. Val holds various patents and has enjoyed considerable commercial success of the systems developed by his teams.