Dylan Schiemann on the Evolution of Dojo, Web Components and Emerging Trends in Web Development

Show Notes

What were the problems you were trying to solve when you created the DoJo JavaScript framework?

• 00:45 It was around the same time that GMail and Google Maps came out, and people started to realise that the browser could be a viable platform for creating applications.
• 00:55 There were no frameworks or real structure to be able to do that.
• 01:00 We wanted to make JavaScript a real development environment where you could create software.
• 01:05 That was where DoJo was born - we created the first module system, asynchronous programming, widgets, structure code, optimise building large applications - all those were in DoJo as far back as 2005.

As the usage scaled, did you encounter problems caused by the scale of the adoption?

• 01:30 Every open source project faces this problem of suddenly you're popular, and how do you not drown in the noise of everyone asking the same questions over and over again.
• 01:45 In 2006, Alex Russell (the co-founder of DoJo) and I were invited to speak at 100 conferences in 2006, so we got nothing done.
• 01:55 Generally the challenge of open source software is being able to scale the community and dealing with strong personalities.
• 02:00 We re-wrote the codebase in 2007 when 1.0 was released, and did another pretty big rewrite in 2011, when we rewrote our modules into AMD to support better optimisations.
• 02:20 We found that over time, we had a lot of features, and it was difficult to support everyone and maintain everything that everyone needed.
• 02:30 We never really pushed back past being the second or third most popular framework at the time.
• 02:35 We found that there was bifurcated needs in the community - there were people who were using JQuery and happy with it, and the people who wanted to build larger more complex applications.
• 02:45 DoJo tried to serve both needs, but it was more focussed on the latter.
• 02:55 At the time, most people wanted to learn JavaScript and do things better, and that's the gap that JQuery filled for many years.

How has the JavaScript landscape evolved since DoJo was first released?

• 03:10 At the time we started DoJo, there were no in-browser debuggers.
• 03:15 FireBug came out shortly after we started DoJo, so there was no console in the browser that you could rely on.
• 03:20 There was no ES6 or ES5, Safari wasn't a browser yet, the iPhone and Android phones didn't exist; there was no Node.JS or Electron.
• 03:35 The language and environments were quite different - back then, you were trying to target two browsers: Internet Explorer and Firefox.
• 03:40 A lot has changed - for the better - but also things have become more complex, expectations are higher, using open-source has become the standard.
• 03:50 Using open-source was a new thing for enterprises - which is laughable today, since large enterprises now assume everything is open-source.
• 04:00 It's changed completely - but we've managed to change with the times, and every framework around 2015 had to make a significant re-write to support ES6.
• 04:15 We took the decision to work on top of TypeScript as well, to give us the nice foundation.
• 04:20 We were the first framework of known quantity - even before Angular - to announce the switch to TypeScript.
• 04:25 It took some time to make that transition.

Why did you decide to migrate to TypeScript when you did?

• 04:45 Around 2012 or early 2013, Colin Snowver brought it to my attention and wondered what it would take to rewrite it in TypeScript.
• 05:00 Over about six months, he wore me down (in a nice way!) - he showed me that Microsoft was doing this as a real open source project and that it would save a lot of time in the intent and accuracy of APIs.
• 05:20 One of the things that has been missing from JavaScript is some sort of interface definition language.
• 05:25 You have no concept of interfaces, so it's difficult to write APIs and frameworks, because you don't have a way of describing the surface of that API.
• 05:35 Because it's such a dynamic language, it's really easy to extend that or break that in unexpected ways, and that's bad for a framework for being able to strictly define that framework's APIs.
• 05:50 As you upgrade, you are in danger of breaking APIs or users' expectations.
• 05:55 The move to TypeScript was more about that it supported interfaces, as well as a nice tool for accuracy.
• 06:05 Until Typescript 2.2 or 2.3, it was pretty rough, and we didn't have everything we needed, and we were running into challenges and problems.
• 06:10 By the time it hit 2.6 it was fully ready where we could ship DoJo 2.
• 06:20 We didn't want to extend the pain points of TypeScript onto users of DoJo; we wanted to fix the pain points and have things streamlined.
• 06:30 I think that was one of the reasons that Angular 2 got a lot of negative pushback, because they used TypeScript more for their benefit but the ergonomics for the end user wasn't there yet.
• 06:40 We waited until we felt like it was a nice way of writing applications on top of DoJo that is ready to ship.

What problems are you targeting with modern DoJo 2?

• 06:50 Modern DoJo is focussed on being a small, opinionated, reactive framework.
• 07:00 If you squint your eyes, it resembles React - it supports JSX syntax and components.
• 07:10 It tends to focus on giving you all the pieces you need - batteries included - for building an application.
• 07:15 It has a routing library, a build system based on webpack, it has theming, components, built on TypeScript, a store implementation like Redex but with thoughts about how you would use them in an application.
• 07:40 Our default bundle size is smaller than a react or angular application, but still doing just as much.
• 07:50 It's comparable to svelte or stencil in terms of bundle size.
• 07:55 There are a lot of benefits to having small reference implementations of all the features you might expect.
• 08:00 We have build-time rendering and DoJo blocks - that gives you access to NodeJS on a component level at development and build time.
• 08:10 You can create a static site using markdown content, and generate a fully static site, while writing it as a modern DoJo application.
• 08:20 You can do it on a component level, rather than an application level, like Gatsby or NextJS would do for the React community.
• 08:30 Because we have a fuller but not bloated framework, we're able to do interesting things that help you to build an application more consistently and more efficiently, yet we're able to keep it small.

You spent a while going from DoJo 1 to DoJo 2.

• 08:50 I did my first conference talk about what DoJo 2 might be in 2010.
• 08:55 We started work on it in 2012 or 2013, then re-wrote it from the ground up at least five times.
• 09:00 We finally ended up with DoJo 2 in 2018, so it was a good eight years.

Did the problems you were targeting at the beginning change as you developed it?

• 09:15 A lot of the things we innovated on or were the first to work on became part of the language standard.
• 09:20 Whether it's module loading, or asynchronous programming, or features that became part of CSS or HTML or other things, a lot of the things we worked on we could remove.
• 09:35 Then it was how to create a nice ergonomic API on top of all of this stuff.
• 09:40 We felt that Angular is really complex - part of that was because they did their rewrite a bit too soon, before TypeScript had been fully baked.
• 09:55 They had to do a lot of things for Angular 2 that they would like to undo, but will take some time.
• 10:00 React said they were going to create this small library and let the ecosystem grow around it.
• 10:05 The problem with that, I find, is that it really requires a PhD in JavaScript to be effective with it.
• 10:15 It's easy to understand the concepts of React, but to really architect an efficient application from start to finish generally means copying someone else's application verbatim or spending countless hours deciding which routing library or component system to use.
• 10:35 There's not a lot of great default decisions to be made unless you follow someone who has put them together.
• 10:45 If you ask the React core team, the opinions often change quickly, or new things come along that are drastically different, like the hooks release a year ago.
• 10:55 We feel there's a lot of value thinking how you're actually going to build applications on top of the framework, and taking time to try and get that right, and being able to iterate on that rapidly.
• 11:15 From 2.0 to 6.0, what we've effectively found is that we've removed about a third of the boilerplate that you would use in writing a modern DoJo application.
• 11:25 We've found the same things that people are doing over and over again, and tried to streamline it.

JavaScript frameworks tend to be highly opinionated to full flexibility; where does DoJo fit in that spectrum?

• 11:50 Probably in the perfect middle, I would hope.
• 11:55 We have good defaults that allow you to get things done, but you can easily eject one of those defaults and do your own thing as you see fit.
• 12:05 We've tried to align really closely with how you write ECMAScript and TypeScript code, so we're really aligning with the standards.
• 12:15 Our approach to theming is PostCSS, which is like Babel for CSS - it takes newer CSS features and transpiles them to CSS features that work in all browsers today.
• 12:25 We've avoided the plethora of non-CSS syntax extensions, because we want to stay aligned to the standards rather than having to support this different ecosystem.
• 12:40 One day, when PostCSS isn't the right solution any more, we don't want to have our users having to update their styling logic.
• 12:45 We've tried to align with standards; we use web components quite a bit (you don't have to use them, but it's an option for importing/exporting UI components).
• 12:55 We've looked at how to make it easier to align with the standards, and how do we get out of the way if the developer knows better.

What is the interop story like?

• 13:15 For DoJo 1, we wanted to make sure that we didn't break anyone else.
• 13:20 We wanted to do all the right things, so if you happen to use DoJo and React, or Angular, or JQuery, or MooTools in the same page, it shouldn't be DoJo's fault if something breaks.
• 13:30 We stuck with that today - today's frameworks are a bit better, and tend to be more aligned to the standards.
• 13:40 Every framework effectively has a component system that is reactive in some way.
• 13:45 There are projects like Stencil that are cross-framework web component libraries, which is a nice thing to have.
• 13:50 DoJo works really well with that; it works by default, because we support web components as a target as well as an import.
• 14:05 If you're using ES modules, and you're using promises, and web components, then it starts to feel quite familiar.
• 14:15 If you're small - and DoJo is one of the smallest web frameworks by default - then it's quite easy to co-exist because you're not putting a lot of demands on a user's browser.
• 14:30 We see people creating a Map widget using modern DoJo that gets exported as a web component which can be used inside a React or Angular application.
• 14:40 The DoJo runtime for that application is quite small.

Are there challenges where that breaks down?

• 14:50 Where we see challenges still: web components isn't a perfect specification; there's currently no easy way to share resources between web components.
• 15:00 If you wanted to distribute 100 web components, you'd have to include a DoJo runtime to support each web component, and that doesn't scale.
• 15:10 That's being worked on and being fixed - we just haven't addressed that yet, as we're hoping an equivalent model to ES modules happens for web components in the near future.

How do you suggest people get started in JavaScript, given the explosion in complexity as seen in this Trend Report

• 16:30 I've given half a dozen talks on JavaScript fatigue alone; there's even a talk on JavaScript Fatigue Fatigue, where people are fed up with the JavaScript Fatigue articles.
• 16:45 JavaScript is interesting because there are so many modules and dependencies to learn; not only the framework, but the DOM, CSS, HTML and cross-browser issues.
• 16:55 If you're doing full-stack development, you then have to worry about Node.JS and networking and database structures.
• 17:05 You can find yourself in a life-long list of TODOs to learn things, but you have to start somewhere.
• 17:10 What I recommend is that people start with understanding the ES6 language or TypeScript.
• 17:20 Once you understand the language, then you can pick up a framework.
• 17:25 If you start by picking up a framework, what happens is inevitably over time that framework becomes not the best solution.
• 17:30 The history of frameworks is likely that something else will take its place in the future, or the framework itself completely changes because it has to.
• 17:40 If you have a solid understanding of the language and the fundamentals, and you learn some CSS and HTML and foundational pieces, then it's pretty easy to pick up frameworks and tools that you need.
• 17:55 Most analyses are pretty shallow, comparing two frameworks which doesn't tell you what you need to do.
• 18:00 What we've tried to do with the trend report [https://www.infoq.com/articles/javascript-web-development-trends-2020/] this time around is look not just at front-end or back-end frameworks, but all the areas of JavaScript code and the things that matter.
• 18:15 It's not just like React and Angular are popular, but there are these other things out there that might be useful that are emerging that might be worthwhile to learn.
• 18:25 If you're in the IoT space, you might want to look at these projects.
• 18:30 If you're into machine learning, then you might want to look at these projects.
• 18:35 To write a true detailed analysis would take hundreds of pages.
• 18:40 It's really difficult to evaluate the quality and longevity of code.
• 18:45 I've been doing this for over 20 years, and I don't know what's going to win, or dominate, or last.
• 18:55 I can tell whether something feels right, or natural - and if it doesn't work for you, then it's not going to be right for you even if you force it.
• 19:05 You've got to have that critical analysis skill to say: does this make sense to me, and am I going to be more productive with this?
• 19:15 Finding that right balance for you and knowing that you don't know everything - it's rare that you can build a JavaScript application of any substance as a solo JavaScript developer - you need a team.
• 19:35 The best thing you can do is to make decisions that work for now, but don't rely on them so heavily that you could never change later.
• 19:45 Try to use the framework for what it's intended for, rather than the solution to every problem you ever have, so you can escape and switch to something else as needed.

Why do you think React became so important?

• 20:15 In the history of JavaScript, DoJo/JQuery/Angular/React all happened at just the right time.
• 20:20 It wasn't necessarily that they were the best or first solution; they were solutions that did what they needed to do at the right time in history.
• 20:30 React, in particular, had the luxury of coming out right at the time when we no longer needed to support old versions of Internet Explorer.
• 20:35 Before that, the memory allocation would have required far exceeded what you could have achieved in IE.
• 20:45 If you wanted to do something like React - to be fair, early versions of DoJo did some things that React does - we couldn't do it all the way because the language didn't support it, and the browsers that we needed to target couldn't handle it.
• 21:00 React emerged at just the right time when IE8 no longer mattered, which is why it became popular when it did.
• 21:15 They have continued to do a great job with it: is it perfect? No, it has lots of warts and lots of challenges, but so does anything that's trying to support that many people using it.
• 21:20 Their biggest challenge today is not being able change things very rapidly even if they wanted to, because they'd break their ecosystem - so they have to be very careful and deliberate in what they change.

What's the reactivity story in modern DoJo like?

• 21:35 It's very similar; the components are reactive, it uses a mono-directional data model - it has the concept of property diffing and DOM diffing which we think makes it fairly efficient.
• 21:50 From the outside, it's very similar to how React works.
• 21:55 If you look at the modern frameworks: View, Svelte, Stencil, DoJo, Ember's Octane - all are quite similar.
• 22:00 Angular can be that way, though they still go with a bidirectional model at times, but we've all converged on this concept, along with hundreds of other frameworks that use a virtual DOM.
• 22:20 There's another framework called Alakali (by a former colleague) that believes that DOM diffing is not the right approach - there's some compelling ideas he's working on for that.
• 22:30 So there's the question of whether this is the right answer long term? Not necessarily; there may be a way of managing the DOM differently and efficiently in the future.
• 22:40 Historically, the DOM was the difficult thing to synchronize with and get right.
• 22:45 What we've effectively done is back in 2004 before JavaScript was big in the client, we would on the server, reactively render some HTML, ship it to the client and then use JavaScript to enhance it.
• 22:55 We've effectively moved that whole reactive model from the server to the client; that's what react popularised.
• 23:05 Now the network connection to the server is really just about getting data, persisting data, getting permissions and getting information.
• 23:15 In the past, the browser was treated as this dumb canvas that we would paint and repaint it based on commands from the server.
• 23:20 Now the DOM is this dumb canvas that we paint and repaint from this reactive model that's a rendering engine stored in the browser memory.
• 23:30 We've gone full circle from treat the DOM as a dumb thing, to use JavaScript on the front end to render the DOM in a dumb way, to now we're treating the DOM as a dumb rendering thing that we render as needed.

Where did the idea for the virtual DOM originate?

• 23:50 In many ways, it's inspired by game engine development and complex grid components.
• 23:55 DoJo's data grid component DGrid, the vector graphics implementation, the Bespin editor which became Cloud 9 - those all used virtual DOM based system on a smaller capacity for specific components.
• 24:20 That concept then became possible for a full-page application.

Are there any libraries in the innovator and early adopter stages that you think we should be particularly paying attention to?

• 24:40 Obviously DoJo, but beyond that, Svelte and Stencil get a lot of attention as well.
• 24:45 Svelte is described as a disappearing framework - it's really closely aligned with HTML and uses JavaScript as a framework to render what you want, and the framework vanishes over time.
• 25:00 It has a really nice compiler that removes most of itself.
• 25:05 The argument is that if you build a sufficiently sophisticated complex application, you eventually end up with a bigger framework.
• 25:10 For simple applications or simple sites, it's really nice and small.
• 25:15 Stencil is a web component framework that's described a way to create a cross-framework component library that you could reuse or create your next design system.
• 25:30 There are others, like Preact, which is a less than 4k GZip'd implementation that's nearly the same as react, but with a few less features that are less important.

What drove the decision to move the web components from the early adopter to the early majority in the trend report?

• 26:00 All major browsers now natively support it (we'll pretend that IE doesn't exist any more).
• 26:10 Large companies like Apple, Nike, ESPN and others are deploying web components on their sites.
• 26:20 Apple, when they redid their iTunes implementation, it uses web components for that - it uses 140 web components to represent the UI in iTunes.
• 26:35 If you go to Ionic, the parent company of Stencil and the Ionic framework, and you look at some of their clients, like Amazon; they use web components as well.
• 26:50 You've moved from this thing that people thought was a good idea - it was the tenth version of the specification - this one's finally sticking; not perfect, but it's good enough for a lot of use cases.
• 27:00 It solves the problem of migrating components between frameworks; if you create your components using Stencil, then they might work inside of these frameworks if you want to.

WebAssembly seems to have been talked about a lot recently - can you explain what it is?

• 27:35 It started as subset of asm.js, and it's morphed into a standard for a way to describe a target for a language that can run natively in a browser.
• 27:55 You can think of it as a description of APIs that can be fast and that you can transpile to.
• 28:00 It can be interpreted in a browser and run efficiently.
• 28:05 What's happened is that every language has created a way to transpile from their language to web assembly.
• 28:10 The most common is Rust, and it's really popular in that area because it's quite small when it runs as web assembly code.
• 28:15 Go is quite easy to get working, but it adds a 3Mb runtime to make it work in web assembly, so it's interesting but quite inefficient for that use.
• 28:30 There's AssemblyScript, which is TypeScript transpiling to web assembly, but what we've seen interesting is that game developers have transpiled C based game engines to run in web assembly.
• 28:45 That's where we see a lot of it - fast Fourier libraries and so on being transpiled to web assembly.
• 28:50 A lot of it is about taking things that are difficult to do in the browser efficiently, and making them run faster; or taking long-lived libraries in mature languages and taking them to the web.
• 29:05 It's not perfect, but it's got a lot of promise and there a lot of things it does well; but what's interesting is that the work on it is making the web look how to make JavaScript faster.
• 29:20 Binary AST is a representation of JavaScript source tree as a binary abstract syntax tree, which would in theory speed up the parse time of JavaScript.
• 29:35 Facebook ships 6Mb of JavaScript and the bottleneck of that is the parse time of JavaScript, not the network time.
• 29:40 They've connected every ISP to Facebook directly, so network speed is not their bottleneck any more; it's JavaScript parse time.
• 29:50 A lot of that is to support rendering movies, images, chat, and all of the other features they have.
• 30:00 Binary AST would cut their parse time in half, so there's a big push to try and bring that to the language as well - to take some of the web assembly benefits to JavaScript itself.
• 30:10 There's also a lot of work that's been done over the years by moving non-UI work outside of the main thread in JavaScript, and web assembly is a natural way to do that.
• 30:25 You could also do that with some of the background processes that you can run outside of the main thread in JavaScript.
• 30:30 There are ways in JavaScript that can help you do that, but web assembly also gets those benefits with not competing with the main UI thread in JavaScript.

Do you have any thoughts on how web assembly to evolve in the next 12-18 months?

• 30:45 It's got to the stage where it works, and now it's time to make it really good.
• 30:50 There's a lot of push to doing that - there are some nice articles from Mozilla looking at the state of web assembly and where it needs to go.
• 31:05 I'd expect the focus to be on where the warts are and where things need to be fixed.
• 31:10 A lot of it is how you improve the communication between JavaScript and web assembly, and how you make it so that web assembly doesn't have to communicate through JavaScript if you don't want it to.
• 31:20 How do you handle module loading in web assembly, and other things like that - there's a lot of work going on in that space.
• 31:30 I expect it to continue to improve.
• 31:35 Early on, I thought it was trying to usurp JavaScript, but now I realise that isn't the intent; it's to bring any code to the web.
• 31:40 You can think of web assembly as a means to evolve what would have been a plugin years ago, and to do things that you can't do in JavaScript today, but not as a black box plugin.