Extensible Effects in JavaScript for Fun and Profit - Q&A with William Heslam

William Heslam, a senior JavaScript developer at Ecotricity, recently presented a talk at Lambda Days 2020 on modeling side effects with extensible effects and the testing benefits that derive from it.

Heslam articulates the talk around a concrete example of a back-end for an application handling todo lists (cf. the Todo MVC benchmark application), with the corresponding API:

// Listen to requests
const update = api.put('/update',
// Write to database
  (request, response) => database.batchWrite(
    'todos',
    request.items.filter(hasTitle)
  )
// Respond to client
  .then(result => response({
    code: 200,
    added: request.items.filter(hasTitle),
    notAdded: request.items.filter(hasNoTitle)
      .map(item => ({ reason: 'NEEDS_TITLE', item }))
  })
))

After quickly reviewing application of standard testing techniques (like mocking and dependency injection) to test the API, Heslam introduces extensible effects to express the API, leveraging the freer monad:

// Listen to requests
const update = api.put('/update')
// Write to database
  .chain(request => database.batchWrite(
    'todos',
    request.items.filter(hasTitle)
  )
// Respond to client
  .chain(result => api.response({
    code: 200,
    added: request.items.filter(hasTitle),
    notAdded: request.items.filter(hasNoTitle)
      .map(item => ({ reason: 'NEEDS_TITLE', item }))
  })
))

The initial api.put returns a data structure generated by the etchRune function from Heslam's extensible effects library:

const api = {
  put: url => etchRune({ type: 'API_PUT', url }),
  response: payload => etchRune({ type: 'API_RESPONSE', payload }),
}

The final update variable is a data structure which effectively encodes a program (like an abstract syntax tree (AST) would) that can be processed in miscellaneous ways by providing an interpreter. Heslam provides the following example of an interpreter which logs step-by-step the execution of the update program:

const result = update.interpret(effect => {
  if(effect.type === 'API_PUT') {
    return State.modify(log => [...log, effect])
      .map(() => ({ items: [
        { title: 'Hungry', desc: 'Buy hummus!' },
        { desc: "Finish lambda days talk..." }
      ] }))
  } else if(effect.type === 'DATABASE_BATCH_WRITE') {
    return State.modify(log => [...log, effect])
      .map(() => 'success!')
  } else if(effect.type === 'API_RESPONSE') {
    return State.modify(log => [...log, effect])
      .map(() => 42)
   }
 })(crocksStateAPI)

Heslam concludes with providing an interpreter which combines random input generation to the generation of execution traces in order to support property-based testing.

InfoQ interviewed Heslam in order to discuss the applicability and benefits of extensible  effects for JavaScript developers.

InfoQ: Can you introduce your experience, and how you came to be interested in functional programming?

William Heslam: I've been writing software for about 10 years, mainly web development with GPU programming as a hobby on the side.

I learnt to program using Java and PHP, and I was very frustrated by how I thought programming had to be done: object orientation, deep inheritance class hierarchies with copious getters and setters.

Luckily I was introduced to functional programming in 2015, with the publishing of ES6. I don't think this was a coincidence, as FP in JS became much more palatable due to concise arrow functions.

That was also around the time I learnt React, which really shows off the power of pure functions and has done a great deal to popularise functional programming among web developers.

Functional programming has given me a vocabulary for thinking and talking about state changes, data structures and models of computation that I simply didn't have before. Best of all, it's really fun!

InfoQ: In your talk, you showcased how an application's API which involves the realization of effects may be implemented with extensible effects, and referred to a extensible effects library you are developing, which is inspired from the freer monad. In layman terms, what is that freer monad and those extensible effects?

Heslam: Extensible Effects, broadly speaking, is the idea that you can separate the 'what' and 'how' in your code.

By representing effects as 'tokens' that hold no intrinsic implementation details, you can write programs that are completely unaware of how they'll eventually interact with their environment.

Later on these effects can be 'interpreted' by converting each token into specific actions of your choice.

These effects could be general, such as 'send network request', or domain specific, like 'log user out' - it's up to you.

For those unfamiliar with monads, you can think of this technique as dependency injection for your software's API calls. You program to an interface, and can provide a different implementation depending on the situation.

Extensible effects are implemented via a Freer monad. This is a nested data structure of an initial effect or value, and a sequence of functions that convert the results of a previous effect into the next.

When applied to an interpreter function that converts effects into the target monad of your choice, it unwraps from the 'inside out' - the first effect is converted into the target monad, which is mapped into the next effect-containing Freer monad. Then rinse and repeat, folding and unfolding like origami until there's just your target monad left.

Freer monads were introduced to fix some issues identified with Free monads, namely to reduce boilerplate when declaring effects and mitigate performance problems. The lack of types and very different execution semantics between Haskell and JS means a lot of this doesn't always carry over, and doesn't change any of the ideas here!

InfoQ: Extensible effects seem, over the last few years, a hot subject topic in many Haskell forums, as Haskell developers regularly compare opinions on the right way to structure programs. That amount of interest in extensible effects is not necessarily found among JavaScript developers -- the right way to structure programs generally devolves into a discussion on front-end frameworks. How would you pitch extensible effects to JavaScript developers (or other developers who do not use a purely functional language)?

Heslam: You're right that languages that let you break the rules will always have less incentive to embrace explicitly modelling side effects; doubly so for dynamically typed languages that don't offer much help navigating complex data structures like nested monads or monad transformers.

On the other hand, modern front end development has really embraced pure functions for app state management, leaving side effects as a missing piece of the puzzle.

A lot of IO in Node and the browser is implemented using Promises, so developers are already familiar with the idea of 'lifting' a computation into a context that may be executed at a later date, or not at all.

Given that, I would pitch extensible effects as a continuation (pardon the pun) of that idea: every API call can be represented as a value you create and pass around, not just the ones that return Promises. The interaction between your program and the outside world is made explicit.

This is useful when abstracting away implementation details. For example, you could make your own 'get' and 'set' storage effects, then interpret them as cookies in one browser or security context, and local storage in another - or even an asynchronous network request.

The fact that some API calls might return a Promise and some don't any longer matters to your program. It doesn't have to stop there: the effects could have artificial delays introduced, or a certain set of effects deliberately ignored, e.g. for simulating read-only storage or missing permissions.

Why not create a 'logging interpreter', so your program logs all its external API calls without having to change your business logic, introducing a form of aspect-oriented programming?

It especially shines when testing code that interacts with third parties.

For example, does your program send text messages? I hope your test harness has stubbed out the right API call - or your continuous integration servers might be inadvertently texting people!

A lot of traditional tools for simulating dependencies make a mockery (again, pardon the pun) of encapsulation as you have to know ahead of time what side effects your code will perform.

This necessitates mutating global variables to carefully swap out API calls before they're invoked, hijacking node's require machinery or painstakingly threading dependencies through every call site.

In contrast, extensible effects deliberately reifies these side effects, transpiling guesswork into confidence.

It encourages reuse of business logic by decoupling it from implementation details and makes testing more convenient and that is why those who are unfamiliar with pure functional programming should consider using extensible effects.

For people already familiar with Promise or Observable-heavy code it will feel like a natural extrapolation of what they're already doing.

InfoQ: What are the advantages of extensible effects over for instance injecting effect handlers (the ReaderT design pattern for instance does something similar to dependency injection)?

Heslam: I haven't used that specific ReaderT design pattern myself. There's apparently a relationship between Freer and ReaderT in Haskell, but it's unclear if this carries over to JS due to the lack of inlining and optimisations.

From my experience, if you're already working with a concrete monad, you might as well wrap it with a ReaderT. You could then provide different implementations of each operation.

The advantage extensible effects has is that you can also defer the choice of monad until your program is about to be executed, which keeps your code fully agnostic about the execution context, not just the operation.

A fun example of this might be a program that samples 3 random numbers to produce a final sum.

By swapping the target monad of the 'get number' effect from an IO or Future of a number, to an array of 5 numbers, the end result of running your program will be an array of 15 values, each one being the result of the 3 X 5 different choices!

This kind of analysis requires a completely different execution context to say, a Promise or an Either, which can only produce one result.

One feature often seen in extensible effects libraries is the ability to interpret each effect individually. If your program has 3 effects, Foo, Bar and Baz, you can interpret Foo and Baz, leaving Bar to be interpreted later - to my knowledge this isn't something you could do with a ReaderT!

I had tried some other techniques and approaches before settling on extensible effects. One I looked at was tagless final, which is a very similar concept. This has been a long running debate for Scala devs, and instructive because Scala and JS are both strict; both languages should have the same preference.

It's faster and has a lot of nice properties, but it also involves passing around your effect interpretations as arguments.

Scala's implicit parameters mitigate the awkwardness of this, but JS doesn't have such syntactic sugar, hence why it didn't appeal to me.

InfoQ: Conversely, what would be the downsides or limitations of extensible effects that you would be aware of? An article by Sandy Maguire presents some limitations in a purely functional language context. Would those limitations apply to a JavaScript context?

Heslam: Some of the criticisms raised in that article - namely performance problems, unhelpful error messages and unmaintained libraries - definitely apply to JS.

There's (usually) no type system, macros or compiler to fill in blanks.

Performance can be a problem for functional JS in general because the lack of immutable guarantees makes it hard for runtimes to safely optimise abstractions and redundant allocations away.

Naming inconsistencies in monad implementations means lots of manual gluing together if you're working with multiple libraries, and very confusing runtime error messages if you miss something!

A knock-on result is that when interpreting, you may have to provide a translation to whatever naming convention your target monad uses - e.g. chain to bind or flatMap etc.

I'm still working on ways to improve this in Runic, my extensible effects library.

The lack of support for higher-order effects, for example interpreting error handling in configurable ways is something I haven't broached yet.

Currently Runic has a fairly fixed concept of error handling, which is similar to Promises and Either. You can't reinterpret its error handling (e.g. to support roll back) as that's decided by your target monad; Runic applies bimap/orElse in linear sequence.

A related concern is concurrency, which is crucial for JavaScript... by default you'd lose this by moving away from raw Promises. This is something higher-order effects could help with!

InfoQ: You mentioned that there are a number of extensible effect libraries written in JavaScript? What would be some of those? What are your design goals for Rune? What features do you want to include and in which timeline?

Heslam: There are actually a bunch of different implementations of extensible effects and free/freer monads:

• eff-javascript: an extensible effect monad
• freer: a free Static Land compatible monad implementation
• monet: powerful abstractions for JavaScript: IO monad
• Functional programming in TypeScript: Free monad
• freeky: free monad collection

A lot of them are experimental or not actively developed, but are worth checking out all the same!

There's also a fantastic blog post by Luke Westby that walks through putting together a simple, working example without getting bogged down in terminology.

My design goals for Runic are succinctness, ease of debugging and convenient integration with the wider FP ecosystem.

It would be naturally faster to inject dependencies as arguments, but this would bloat the code a lot and ruin the joy of JS!

Instead, I'm erring on the side of less syntax, but given JS is so resistant to static analysis, you pay a performance cost for that abstraction.

In a similar vein, another balancing act is encouraging TS type inference whilst allowing for referential transparency.

Type inference often requires locating functions directly where they'll be used - potentially requiring you to write the same inline code multiple times, which is in direct conflict with the conciseness of defining it once and referring to it by name.

Built in integration with existing monadic libraries like RxJS and Crocks will cut down on the amount of boilerplate people have to write themselves, especially if it's compatible with the fantasy-land spec.

I want to get an initial release of Runic out in the next month or so, but getting the ergonomics right will be really important.

InfoQ: In your talk, you use your Rune library and extensible effects to test an example API.  The reasoning is that, once you have a free(r) monadic program, you can append any interpreter of your choice to evaluate that program. One interpreter may perform the effects in the program. For testing purposes, you propose another interpreter which generates random inputs to the program, evaluates the program, traces down the program evaluation, including errors occurring during the evaluation. Did I understand that well? What advantages are derived from handling or simulating errors too in the interpreter?

Heslam: Yep, that's a good summary!

Error handling is an interesting problem. As you alluded to earlier it's a topic that Polysemy, a modern free monad Haskell library, is looking at.

There's a great article by Aaron Levin about categorising types of errors when working with Free monads that recommends encoding "semantic errors" as an Either in an effect's result type.

Runic's name for an effect lifted into a Freer monad is a 'Rune' and it takes a slightly different approach. Instead of effect results returning Either, it bakes Either into its signature: you can map and chain as usual, but it also supports recovering from errors with otherwise/orElse, similar to Folktale's Result and implements bimap and bichain, like Crock's Either.

This seems pretty similar to how Scala's ZIO represents errors too.

It relies on the target monad to be able to represent the left state, so if your target monad is State or Array, it's just going to follow the happy path - your effect implementation would have no way to propagate an error!

If your target monad needs to store state AND represent failure, you might want fp-ts' StateReaderTaskEither.

Being able to simulate errors during testing is really useful for proving the robustness of your code. The reverse is also true - by providing fallbacks and retries for failures in your interpreter, you can effectively hide that complexity from your program, if that's what you want to do.

Unlike languages with strong typing, JS makes it pretty easy to accidentally throw exceptions when you least expect it. Throwing an exception inside a Rune's map or chain function will actually occur inside the context of the target monad during interpretation, so it's up to that monad as to how exceptions are handled.

InfoQ: How does property-based testing work together with the interpreter mechanism?

Heslam: If classic 'example-based' unit testing works by asserting specific things for given inputs (given input A, expect to get back B) then property-based testing revolves around proving that your assertions hold for a given range of inputs.

This range can be essentially infinite e.g. every natural number, or it could be just very large e.g. every permutation of a length 5 ASCII string,

so your test will need to be bounded by a maximum number of attempts.

(For property testing I recommend the brilliant fast-check, which is well maintained and easy to use!)

You specify the space of inputs by combining 'Arbitraries' (random value generators) in a monadic style with map, chain and a host of combinators.

A classic example of property testing is checking if a sort is stable: generate a randomly sized array of random numbers, and sort it once, then sort the sorted result and make sure they're the same. This property should hold for all possible arrays of numbers, so if it doesn't, your property testing library will automatically identify the smallest failing case where that isn't true.

In that situation there's a clear separation between your input value generator, your code under test and the property test itself.

What I realised was that because you can chain Arbitraries together in a monadic style, they can serve as target monads for your interpreter, just like an Array or State monad could.

Now the input value generator and the code under test are essentially one and the same thing.

Your Arbitraries aren't creating random arrays of numbers, they're now generating random combinations of results of your effects. Your program is essentially a chain of Arbitraries.

In my Lambda Days talk, I demonstrated the robustness of a program that processes and stores some information using an unreliable database that may only store a subset of records or fail completely.

Testing this traditionally could be really frustrating as there would be only so many variations I could write tests for before getting bored!

Instead, I showed how to represent database writes as an effect and interpret them into an Arbitrary of total success, partial success or failure.

I then designed a property test that asserted that my code would always retry failed writes until every record had been successfully saved.

Running that property test would try random variations in the 'phase space' of all potentially failing database writes, searching for a potential point where my assertion no longer holds.

You can extend this idea further: by representing reading the system time as an effect, and interpreting it as an Arbitrary (potentially wrapped in a State transformer), it's possible to simulate a system time that randomly jumps about in order to check that exponential backoffs work as intended.

In my talk I showed how to use Runic's ArbitraryStateEither, which adds state and errors to Arbitraries. It's great for storing a log of all produced effects, allowing you to prove that your program obeys certain rules.

For example: never performing the same effect more than 5 times in a row, or never trying to 'log out' without having first 'logged in'.

InfoQ: Aside from your talk, did you have the opportunity to use extensible effects in a real-life program? If that is the case, can you detail the context of such use, and the benefits in that context?

Heslam: This isn't something I've used in a shipped project yet, but it's inspired by problems encountered in a serverless microservice that used AWS' DynamoDB. DynamoDB has intentionally limited read/write capacity, reporting which records it wasn't able to insert with the expectation you'll retry failed operations.

Testing partial unavailability corner cases is pretty difficult, and you need to be careful when differentiating between retryable and non-retryable errors, especially when interfacing with non-idempotent third parties!

I tried creating Arbitraries that generate failing APIs I could swap in with traditional dependency injection, but I found this pretty cumbersome. Worse, the generators had no idea if an API call was going to be used for a given iteration of my tests, so they'd be producing lots of redundant permutations.

Looking for a more parsimonious answer led me to learning about extensible effects, and by interpreting my programs into sequences of Arbitraries, I realised my tests would only generate permutations of failing databases that matter. The program became the permutation!

InfoQ: What you do think can be done to make extensible effects more attractive to JavaScript developers, whether on the front-end or the back-end?

Heslam: A suite of predefined effects for node and browser APIs would help people be productive, sooner. For example a Fetch effect with a production interpreter and test interpreter ready to use.

That would also help document practical examples, which is often missing!

A lot of successful JS libraries have popularised non-trivial concepts like observables, sagas and algebraic effects by relentlessly focussing on practical benefits, embracing JS' quirky semantics and for better or worse, leaving the theory and academic terms for further reading.

This is a good balance as in my opinion most web devs are attracted to (or begrudgingly tolerant of) novelty but maintain a strong pragmatic bent. I think this has probably encouraged greater mainstream adoption than libraries that tacitly assume their audience already has a functional background.

There's strong pressure for libraries to export type definitions, and extensible effects could be significantly more appealing with solid support for TypeScript.

If an effect is a natural boundary between a program and its environment, it's also a natural place to embellish with types - unlike internal implementation details, where annotations can be a burden.

Following on from that, an idea I've been exploring in Runic is the concept of effect type constraints.

A program that interacts with the database might have the following type:

Rune<Write | Read | Delete, Error, boolean>

If you wanted to indicate that a given API endpoint should have Read access, but not Write or Delete access, requiring the program's type to be:

Rune<Read, Error, boolean>

would flag the incompatibility as a type error.

Of course, TS has type assertions and escape hatches so it's more a form of interactive documentation than true security measure!

TS' type system has got a lot better but it's not always straightforward to represent data structures that are natural in Haskell.

Support for higher kinded types would help a lot, but see fp-ts for an excellent attempt to do without!

The main attraction of extensible effects for me is how it side-steps ugly problems with both API mocking and traditional dependency injection that makes testing side effecting code an awkward chore. My aim with Runic is to make that difference as obvious as possible: there is a better way!

About the Interviewee

William Heslam is a full stack web developer and international speaker. He's interested in ways we can simplify and improve software development by borrowing ideas from the exciting realm of functional programming. He enjoys giving talks on a variety of subjects, from sophisticated testing techniques to parallelised biological simulations. Heslam's hobbies include seeking out forgotten paradigms from the 1980s and 3D graphics programming. He also likes earl grey tea, sichuan pepper and sauerkraut, but not at the same time.