Robert Virding: What keeps me interested? I am still working on LFE [Lisp Flavored Erlang] and this socket stuff I had to learn about the web to write the stuff to do it. That’s kept me interested.
2. Was it the same thing for you, Joe, you’ve been working on web sockets?
Joe Armstrong: Well that’s a hobby thing actually because it is fun.
Robert Virding: It is fun, it’s fun to do that, and it’s fun to see how it works, and it’s fun that it’s my first real entries into web programming, which I haven’t really done before. I’ve used the web, but not written stuff for it. And it’s also fun to learn how to do it from Erlang and get it to work. It was interesting to do, to see how things are done, trying to get a feel for what real web programs have to do to get around things, to make pages that look much more interactive than the actual protocols allow you to be. So the web sockets give you that, doing things on top of the normal protocols to the emulate web sockets - that was interesting to see how that works.
Joe Armstrong: Yes, people have always been able to do that. The problem with the browser has been that you haven’t been able to push data to them, a client could pull data from the server, the server can’t push data to the client. And you break the symmetry so it’s just not beautiful. I mean HTTP was implemented on top of sockets and you can’t use the sockets, it’s actually absurd. I mean if anything is wrong with the web sockets it’s the framing protocol; if you took the two bytes out it would just be a proper socket and we know how to program sockets.
Robert Virding: And it just fits so naturally in the way that Erlang works; it’s asynchronous communication. You do it in Erlang, sometimes you send data to another Erlang process sometimes you send data out of the socket, but the Erlang point of view is exactly the same, semantically it’s exactly the same.
Joe Armstrong: I mean the socket is just a pipe: you stick a byte in one end and it pops out at the other end and it’s symmetric so why would anybody stop you from using that? It took like ten years to get back to where it should have been in the beginning, it’s just crazy.
Joe Armstrong: There isn’t a right paradigm, it’s just communication. You send messages on it. It’s nothing new, it’s an idea that is thirty years old, forty years old.
Robert Virding: It’s been around for a long time.
Joe Armstrong: For as long as we had sockets.
Robert Virding: It’s a way of giving you interactivity. I mean most web pages that you go to will have that in some way. It won’t be a static just request, you get a reply and wait for next thing. Things will happen on the page while you are sitting there. And in effect the server is pushing data to the page.
Joe Armstrong: Imagine you want to build an alarm system you’ve got a thousand machines that could generate the alarms, you want to push the alarms to a client, it can’t go around and poll a thousand machines or something like that, it’s absurd.
Robert Virding: It doesn’t fit, no. I think the normal way a lot of web servers are written the frameworks for them don’t fit in that. That’s what made it so easy the Erlang point of view.
Joe Armstrong: But that’s because web servers don’t - I mean the natural way of dealing with that it’s born of one parallel process per socket. And if your programming language or operating system doesn’t commit that, that’s arbitrary limits, I mean if you can only have like a thousand parallel sessions in your machine because the operating system has limits or your programming doesn’t allow that, you have a very unnatural style of programming. Because you have to multiplex on to your thousand processes but you can have a million processes in your machine, and a million sockets, which you can do now. Admit that you need a lot of memory to do that but you can do it in principle. And as soon as you map one process to one socket, it’s just an extremely natural way of programming.
Robert Virding: The funny thing is most of the Erlang web server type packages they do that anyway. They would quite happily spawn a process for every request that comes, every get request that comes in, which does something then sends a reply and dies.
Joe Armstrong: Actually in a way they don’t really spawn a process, they spawn a web server. Every process is actually a web server, so it’s not like having one web server handling a thousand sessions. It’s like having a thousand web servers each one handling one session. It’s how you think about it.
Robert Virding: It’s an Erlang process that doesn’t.
Joe Armstrong: But that process is a web server.
Robert Virding: So taking the step from going from a get to sending back a reply to having something that talks over a socket from the Erlang point of view there is no step there.
Joe Armstrong: It’s a no brainer.
Robert Virding: That is what the packages do. I am always using packages, my demo is a package whistle team, and that’s typical it works in a typical way: every time you get a request, it spawns a new process for it. So if you want to do things like streaming, from the Erlang point of view it’s trivial to do that, because you just have a process, if you are writing a web socket, you do exactly the same way.
Joe Armstrong: And at a certain point in abstraction, you can send an Erlang message into a black box and then out comes a JavaScript message and the mapping is one to one and then you just send the JavaScript to the browser and bingo.
Robert Virding: You are running generic eval. Ok, I was more conservative.
Joe Armstrong: It’s just how to do it.
Robert Virding: Yes, I was slightly embarrassed how easy it was to do it because from the Erlang point it was not a problem to do that.
Joe Armstrong: I think this protocol makes it symmetric again. It should have been, I mean TCP is symmetric. And then you put all these protocols on top of it to make it be not symmetric and then you get work-arounds to make it be symmetric and then you remove the work-arounds and it’s symmetric again, that’s good and you wonder why they bothered to put all that stuff in the way. And then still web sockets don’t give you a socket, they give you a little framing protocol because you might need it.
Robert Virding: It’s a pretty trivial one.
Joe Armstrong: Yes, but why bother?
Robert Virding: Because it’s framing the other way.
Joe Armstrong: Come on, the framing protocol is zero and an FF;, you could just remove that.
Robert Virding: Exactly, it’s a bit strange before I realized there was a frame I was using and I was wondering where those zeroes and FFs were coming from. I said I didn’t put them in there, why are they there for? Now I also realized that it was actually part of the protocol. But it’s just that when you do it in Erlang you actually see how in one sense trivial the difference is. So if you are doing streaming, and you have an Erlang process if you send something it sits and waits and sends something else, it sits and waits and sends something else, which is exactly the same way to do it if you had a web socket. And if you got something to wait for a while after you get request and wait for a while and send back, which is long polling, and just the way Erlang is written anyway.
Joe Armstrong: The latency issue, if you hit completion you hit a single character on your keyboard so now you can send three bytes. Before that you could send like eight hundred bytes of headers and such, which you had to parse before you get to your one byte.
Robert Virding: There is a lot of wrapping around them.
Joe Armstrong: It’s just silly.
Robert Virding: It’s not impossible, no, there is a lot of work in it, of course, doing that.
Joe Armstrong: It depends how you design stuff. There are two ways of designing things, one you start off with a little startup and you got this idea and you start with a single server and you build it for a thousand people or something like that, and then you find out that it’s popular so you get two and if it grows, then you figure out how do we distribute it? And then you get three, and you get four, and it slowly grows, and then you haven’t really thought about it yet. And the other way is you don’t do it that way at all, you design it for ten to the ten users, and you sit down and you go to the computer science and you do your experiments and you design it for ten to the ten users, and then you scale it down for a hundred users.
And if you have done it that way, then as it scales up you are not going to have unexpected problems. The behavior of the system isn’t accidental, it behaves as it behaves because it’s designed to behave that way. So when we build big going systems, we have designed it to scale in a certain way, we’ve designed it to fail in a certain way, and most part it behaves like that. If you don’t design the scalability and the failure right from the very beginning then this will give you a nasty surprise "We can’t scale it. What do we do about this? Oh, we got all this legacy system, and then it’s very difficult to change the architecture." This is a question of re-architecting it.
That’s like doing brain surgery on somebody when you are in a racing car, you do brain surgery in an operating theatre where you’ve planned the operation in advance, but doing it on a patient while you are in a bus is a bad idea. So if you are in this position where you’ve started Twitter and it blows up in your face and you suddenly have got millions of users and then you think "Oh, how can we make it scalable?" it’s too late.
Robert Virding: It’s about architecture design as Joe says, getting it in from the beginning if I am getting a million users, how will I design my system to work? Now I’ve only got a thousand, but if it’s a million how would I design my system to work? And often you’ll find there is a difference in designing a small system and designing an enlarged system, that’s the problem, because they would deal differently; if you are designing a small system, you’ll do it one way because that’s more efficient, but that won’t scale properly.
Joe Armstrong: There are a lot of questions on the Erlang mailing lists, and people ask questions like "How do I make a scalable system? How do I do this?" and my answer is always "Ok, tell me the latencies, tell me how much data, tell me how many requests per second, and tell me the fault tolerance requirements because they are different.
Robert Virding: And how many users.
Joe Armstrong: Yes, sure. The system for a thousand users has a different architecture than a system for ten to the nine users.
Robert Virding: It does, that’s the problem.
Joe Armstrong: The architectures are not the same, they are not continuous they are different.
Robert Virding: Using Erlang will help you with some of the things, the concurrency itself makes part of the problem, sort of helps you with part of the problem, it doesn’t help you with everything. II f you need let’s say distributed databases that’s still a problem you have, you have to solve that problem.
Joe Armstrong: I mean if Erlang can handle a million processes, and you’ve mapped for one to a million users, and you throw ten to the nine users at it, it’s not going to work. What are you going to do about it?
Robert Virding: Then you have two or ten systems, or a hundred systems, how do you get those talk to each other? Well you can, it’s not rocket science but you have to plan the system to do that from the beginning. I am going to use a database which I know will scale and it can scale if I need to, and if you do that from the beginning it’s fine. That’s some use when you are building new distributed databases like Riak and Couch, they are designed to scale, but you do that from the beginning. It’s the same with systems like Twitter. They are probably more successful than they thought they would be or hoped they would be. Erlang won’t help you with everything, it will help you with some of the things, or make it easier to do some of the things, but they won’t do everything.
Joe Armstrong: We should do a micro twitter, pico twitter. One byte; one bit.
Robert Virding: And depending if you want to look at it there is a choice to scale the right or wrong language I suppose it depends on how they put in their architecture really and what they are doing in there now, how the whole architecture works. For example like if people build websites with Ruby it’s fine, but it won’t scale very well so how do you scale? One solution is putting in an Erlang front end and it does all the scaling for you and then you can still use Ruby for the website itself, and all the nice stuff you see on the web page is done in Ruby which might be a better way of doing that. But you got to scale in Erlang and then you made your choice on how you are going to handle scaling. One of the original, the Bluetail product which we made was a scaling frontend that would handle scaling. Both in the backends and servers and in itself so it could scale itself at runtime.
Robert Virding: I am old Lisp buff. For we who we love parentheses. It was fun, I enjoy implementing languages basically and the sort like I do in the support. And I wanted a Lisp that would work interchangeably with Erlang so that you could write stuff in LFE and then you could write stuff in Erlang and they could work together. So that’s the base for it. Apart for the parentheses, these Lisps do have some benefits and such, they usually have very nice macro package, which other languages don’t. And I enjoy doing it really, that’s the main reason for doing it. As far as I know there are no hardcore users of it or they haven’t told me anyway but I know some people have written quite a lot of stuff in Lisp and it seems to work. And there are no known bugs.
Joe Armstrong: They are not that different.
Robert Virding: Not from our point of view. It’s not something I worry about.
Joe Armstrong: Memory and stuff, pointers, sort of moving.
Robert Virding: I mean as I said I did LFE because I thought it would be fun, it has been fun from my point of view that’s why I did it. And it provides some new functionality if you look at it seriously, but not that much actually. It’s just a way of playing with the system, writing applications for this, to implement the language.
Joe Armstrong: It’s a good way to learn a language is to implement it.
Robert Virding: It is, I have no serious thoughts that people are going to start using Erlang because it’s got a Lisp interface on top of it. It’s not going to solve any Erlang syntax problems. If anything it would just make them worse; people don’t see polite apprentices.
Joe Armstrong: No, it’s a silly idea.
Robert Virding: You think this, what do you mean?
Joe Armstrong: Because the virtual machine is good at nothing. I mean it’s equally bad for all languages, slightly better for Java than other machines, if you look at the JVM instructions set and the .Net instructions set, you can’t detect integer overflows and other things. If you look at JErlang because we do exact arithmetic you have to detect the overflows into big numbers if you want. So you have to do the multiplication twice with higher precision and compare and see you got an overflow, because you can’t get a hold of the overflow flag which is in the instructions set. So something like the low level virtual machine is much better. Forget this stuff it was a mistake. It will be a historical parenthesis.
Robert Virding: I don’t think so actually.
Joe Armstrong: Because there’s always legacy. It should be though a historical parenthesis.
Robert Virding: It might survive Java. But I think the basic problem is that you have a virtual machine which is designed for some language, Java or Erlang or whatever it might be. And it will have a set of properties, which are specific to that language.
Joe Armstrong: But their bytecode is stack machines, physical machines have word lengths, they have registers, they have overflow flags and things, so there is a big semantic gap between the JVM and the .NET machine and the real processes. And the Erlang VM hasn’t got that big semantic gap it has a different semantic gap but it can model registers and things and it’s a much better factor. If you want to agree on a machine, a virtual machine, you would go to the low level virtual machine.
Robert Virding: But then you have to do a lot more work, if you are using that, there is a lot of stuff you need to do.
Joe Armstrong: No you don’t. Ok, you don’t use it to do your garbage collection.
Robert Virding: There is a lot of stuff in the JVM and in the BEAM in there. But if you are implementing a language on the BEAM like I did, you either choose to accept the properties and limitations of the BEAM in your language, which I did with LFE, if I want to do something like destructive operations or memory that would be a lot of effort to put into a language because the BEAM doesn’t support it. So you have to do it sensibly, you have to work within the limitations you get, doing something on the JVM it’s a different set of limitations, so if you want to do language and do it proper you have to be aware of those and how you design the language.
Joe Armstrong: I think the problem of the JVM would be not the functional things, so certainly you have the instructions that can do things, that’s fine, it’s the non functional thing. It’s like the properties to the garbage collector it’s the property of failure, the property of isolation, these you can’t do anything about, you live with the garbage collector that you have in the JVM, you can’t do anything about it. If you have a low level virtual machine, you have to implement your own garbage collector. Like doing this thing in Erlang, in Erlang you’ve got the garbage collector, but you’ve got limitations.
Robert Virding: I’ve got limitations; I can’t do anything about it.
Joe Armstrong: The garbage collector you got, you can’t do anything about it. You have to write a new garbage collector.
Robert Virding: I have to write a new garbage collector, but I have got the memory management, for example the BEAM will not support destructive operations on memory it just won’t work.
Joe Armstrong: Quite right.
Robert Virding: Yes, it’s quite right, but it is. So if I want to do something in a language that does that, the BEAM is not a good way to do it. I can do it, but it’s not a good way to do it. And the JVM is better but there you’ve got the fact that you always got destructive operations on memory in the JVM. You can lock it out I think but I don’t know if it’s safe, so then you have that limitation. But the Erlang on the JVM is very good, I’m very impressed with that.
Joe Armstrong: Yes, that’s good.
Robert Virding: And I know Kresten [Krab Thorup, founder of the Erjang open source project] said there are new things coming in Java 8 which would make the implementation much better, if it was just supporting a lot of things he is doing.
Joe Armstrong: But the good thing about the JVM is the brilliant implementation. But I don’t think the abstraction things are very good. You have to distinguish them but they just put so much effort into making it efficient. That’s the only reason why these things are feasible because the implementation is so far so good.
Robert Virding: Like the JIT compiler, not an effort into writing a very good compiler. Yes, it’s impressive stuff definitely. But then again if it’s the right base that’s not the question. I use Erjang actually, it works. I tried it.
Joe Armstrong: I think I tried it once.
10. I heard that you committed some code in there.
Robert Virding: Very trivial code. I just found it was returning wrong error, so I had to go and find it, it was very trivial code actually.
11. Isn’t that like having one only garbage collector that breaks the whole Erlang model?
Joe Armstrong: No, it changes the pragmatics. But the logic is the same it would mean that the program will maybe sequence in a different order, but if they’ve been written in such a way that it matters.
Robert Virding: Then there are different JVMs, there are different memory management schemes for them. It’s the same with everything else there is a lot of effort being put in it they have very good systems.
Joe Armstrong: But the pragmatics will be completely different, because if you imagine running an Erlang here, and a Java program running here gets into a tight loop maybe it’s completely stops your Erlang or something. The pragmatics are going to be totally different. So I wouldn’t like to do it for a critical system, because I wouldn’t be able to control these other things that are going on.
Robert Virding: You have to be careful you would get different behavior, different things happen, when your interface it and how you interface it. But I think it’s interesting.
Joe Armstrong: Oh yes. I think Kresten did it because he’s got a lot of Java and he would like to use some of the Erlang stuff and then there is a rather seamless way of integrating materials, so if you want to call Java from Erlang, then that’s very natural.
Robert Virding: I mean that’s the similar reason why they did Clojure. They wanted a Lisp language granting on the JVM with the limitations and the benefits of using the JVM model and from the Lisp point of view you get some strange features, for example there is no tail recursion which is a limitation to JVM.
Joe Armstrong: I mean it’s not bad nor good, it depends on what you want to do. It’s both.
Robert Virding: See what happens, see who wins in another couple of years. Whatever works.
Joe Armstrong: But again it depends on what you want to do you might need it for that application but not for that one.
Robert Virding: Yes, well that is what you find out if you’re comparing Erjang and the BEAM, that the Erjang is faster for some things and the BEAM is faster for some other things, and what you want it depends on your application. So if you want to do number crunching or calculations you want to do Erjang. because it’s faster, but beam works much better on concurrency and the SMP multi-core things. So it depends what you are after. System agnostic. Well, Erlang applications are usually language agnostic. I think you would find most of the Erlang applications today have some parts written in Erlang and some parts written in another language that’s fine. You use what’s best to save you trouble. Saving having to fit something where it doesn’t fit. If you have a square peg then you have a square hole; if you have a round peg then you have a round hole, if you need both you have both there is no problem with that.
Joe Armstrong: I don’t know, I think certain design decisions you could trace directly back to, like memory limitation. For example, the code change mechanism allows two versions of the module and works as it works. If we were doing it today, we would have multiple versions, multiple modules and garbage collect them. There were very definitely things that were limited by the amount of memory we had on the machines when we made it, and I think a lot of design decisions would change.
Robert Virding: Some would definitely change, some would definitely be the same, I mean the concurrency, not that there is concurrency, but the fact that there is such lightweight concurrency and so lightweight communication that would definitely be the same. Process isolation to allow you to do error handling would definitely be the same it’s just irrespective of other languages details. It was just the right way to go. So that would definitely be the same. If it would look the same, if the syntax would be the same, I don’t know, that would depend really where you come from when you are implementing the language. Part of the reason it looks like Prolog is because we used to do things first on the Prolog system. So you just pick what you’ve got.
Joe Armstrong: Pick, sort of an introspection.
Robert Virding: We’d do a lot more, and probably write a better language.
Joe Armstrong: Absolutely, much better.
Robert Virding: I mean twenty-twenty hindsight, we would know where we were wrong. Some things would look the same and some things would most likely look different depending on exactly what you want. And we’d have another syntax war instead.
Joe Armstrong: I think everything would be higher order, modules and everything.
Robert Virding: It would be much more like this.
Joe Armstrong: We didn’t understand that at the time I think. Maybe we would have types.
Robert Virding: I don’t know about types, optional types.
13. That was almost the question I was about to ask.
Joe Armstrong: We won’t say we were wrong.
Robert Virding: That was strange because you were a Fortran programmer before, and I programmed a lot in C and both Fortran and C have static typing. And after we have done other things, we chose a language with complete dynamic typing - it’s almost violently anti-static typing, it just happened.
Joe Armstrong: Messing around with Smalltalk, that’s where it gets you.
Robert Virding: It was a bit strange in a way when you think about it. Having dynamic typing allows you to have much more dynamic systems though because you can do as you know. Nodes can come and go and they can be sort of anything.
Joe Armstrong: But both Fortran and C type systems are broken in different ways.
Robert Virding: You can type cast it in principle. You can do equivalent in Fortran.
Joe Armstrong: Even more wonderful things than in C.
Robert Virding: Especially old C, it is much more forgiving for things like that.
Joe Armstrong: I think it goes back to the Unix C traditions back to basics and other compiled languages and it remedies other deficiencies in C, I don’t think C++ was an improvement but I do think Go is a definite improvement on C and we’ve got Kernighan and things in the background there and obviously they’ve got wonderful experience on building languages. It’s very nicely engineered and actually when it even came out impressive documentation, and all this stuff that you need. Even when it first came out it has a level of maturity that you would think would actually have been there for many years, so it is very impressive actually.
Robert Virding: And it sort of attacks similar types of problems, as Erlang does, it tries to handle concurrency, communication things like this, which is very nice, that is one of the reasons why Erlang because we thought that this was something fundamental.
Joe Armstrong: But it didn’t handle those things like the code upgrade and the evolution, those kinds of issues, we are interested in language like it evolves without stopping so we were going into a lot of operating system issues at the same time.
Robert Virding: It is nice to know that here people native in other language who see similar types of things like we are doing as fundamental, like concurrency and communication, it’s part of the language. Even if they have a different answer, it’s fundamental image. They are right of course, that is a fundamental property something has to be put in from the beginning to get it right. If they have a different solution, it’s ok, that’s fine.
Robert Virding: We can do bytecode emulator which is one way to do it. Or you can go with what Joe has been doing with the Erlang VM, which gives you a more abstract machine which does that.
Joe Armstrong: You’ve got the language itself that compiles on virtual machine or regular machine, that’s not that difficult, I mean it’s difficult but then you’ve got all the libraries, all the low level, the stuff to talk to the socket, and you’ve got SCTP, TCP and UDP and all this stuff, and that’s done in C, it’s not done in Erlang and it’s a massive amount of code.
Robert Virding: There is more code there than in the actual language part.
Joe Armstrong: I think the language part is sort of, if you want to build a compiler you can use Yak and Lex and tools and you know how to do comfortable decision trees and you know how to write a garbage collector. But you start implementing TCP and UDP and all the things get very messy and you have to dig into APIs and then garbage collectors and things that are cheeky.
Robert Virding: The stuff around the basic language core in Erlang everything looks like syntax and you have a large amount of stuff around them which in one sense part of the language and we have all the built-in functions they are part of the language. They’re probably much messier.
Joe Armstrong: Garbage collection is conceptually simple, and then it becomes more and more complicated, and then you get generational garbage collection which is conceptually simple but much more tricky to implement and then you have different garbage collection for different types of data. I mean in Erlang we have got six different reference counting and then it’s getting complicated. For squeezing that extra performance out of the garbage collector is really difficult.
Robert Virding: Compared to that, the compiler is easy. Actually it’s fun because most people when they implement a language they do the compiler bit but then they’ll have sort of a basic core but they won’t have all the stuff around it which you need to make it work.
Joe Armstrong: The first Erlang was just written in Prolog so you didn’t have to worry about the garbage collector because Prolog did the garbage collection, so the big step when it became its own machine but then you have to implement the garbage collector.
Robert Virding: You need to implement everything.
Joe Armstrong: All this stuff you get for free because nobody thinks about, if you are using Java or Prolog, nobody thinks about the garbage collector it’s just there. Or nobody thinks about bignums; they are just there, they just work.
Robert Virding: They just work, all the stuff around them.
Joe Armstrong: Big dividers, always. Do we know the big division?
Robert Virding: We’d be using some packages I don’t know sorry. Tony’s stuff usually works.
Joe Armstrong: It proved it correct, how do you know? Which division is correct?
Robert Virding: I don’t know.
Joe Armstrong: Maybe one day somebody will say "I’ve divided that number and got here wrong" or maybe you multiplied afterwards you just checked to see if you got it right. That could happen.