I’m Salvatore Sanfilippo and I work at Open Source project Redis as my full time work, thanks to Pivotal sponsoring the project. And currently we are in San Francisco but usually I live in Catania that’s in Sicily, that’s the southern region of Italy and it’s not a very technological place maybe, compared to the bay area of course, there are a few companies but it’s the kind of place where you have to explain very clearly to the people what you do for living.
Tomorrow I will talk about different arguments, different topics, all related to Redis but they have different point of view on the project. I will examine the long term projects, the long term goals of the projects and also the software development aspects because after many years of working at Redis, it’s clear that some approach may work, some approach may not work and also the user base is everyday greater, bigger I mean in terms of number of developers, so we get more pull requests, more issues and the reason to understand how to handle this rising lot of things to do. And also I will talk about how Redis can improve in the future from the point of view of the API, because so far many efforts in the latest years were focused into improving how the server behaves taking the user interface constant. In the next years probably we will focus on how also to improve what we export to the final developer and caching is one of the use cases that will get our deep attention in the course of the next year.
Redis it’s a bit strange project because it is a database in some way, is a cache in some way and it is a messaging system in some other way. So what if it’s true of all these many different aspects of Redis, I think that the reality is that Redis is actually a different object, it’s like cut, strip down a programming language with just a DSL basically, a Domain Specific Language, that can alter data structures and the data structures are optionally durable on disk. So because of this general way it works, you can use it in many different ways, the caching scenario is one obvious one, especially when you need the mutable state to be updated alongside with the database view, instead of the old pattern of caching and invalidation and to recreate from the caching view from scratch every time there is a modification in the object that has its cache.
So I think that there are obvious use cases that are not well covered by Redis, that’s where you have a huge amount of data that is not very latency sensitive, so there are better approaches to serve these use cases, like this traditional disk databases. One good use case is for face data, you have an amount of data that’s not huge but you have a huge amount of requests finding this metadata, this small but very intensive data. Another one is caching where it’s a bit like the same idea, but you have just a view of the working set that’s materialized on Redis, maybe mutable, maybe using lists in order to remember the latest N items that are accessed to much more often compiled to the rest of the data and also people use a lot Redis for messaging, because it can process a very big number of messages per second and because being actually a set of different comments, you can like design your own system on purpose for your needs.
And then there is also they uses a message bus not using lists and blocking operations but more using a publish subscribe to exchange messages between processes, this is more of fire and forget use case. One of these use cases for the late tasks processing, jobs processing, is going to be spun off from Redis in a new Open Source project and we hope to migrate a percentage of Redis users from Redis to the new system, it’s a message broker basically that’s very similar to Redis in the API exported to the final user, but it’s very different in the implementation as distributed system that’s fault tolerant multi master, a lot of things that it’s not possible to have with Redis because they are in conflict with the other aspects of Redis, so we are trying to move away what is good about Redis as a messaging system, outside of it with different reliability characteristics.
4. [...] What is the big thing about NoSQL and Redis in particular?
Rags' full question: Ok, so databases in the traditional sense are not going away, with all this new NoSQL ecosystem which supports unlimited scaling, durability and all that, I think you already talked about what is the good use case for a traditional database versus a NoSQL database, but is it just about the fact that you have a flexible scheme, is about scaling, what is the big thing about NoSQL and Redis in particular?
Many kind of system software, many kind of databases and programming languages, frameworks, are in some way equivalent. Maybe company users are given a set of technologies but they will very well move if forced a different set of technology and the final product will look.very similar and maybe using also similar mode of resources sometimes. So it’s a lot about when is possible to the database solves a given problem much better then another one that is possible to get very huge savings in terms of resources used, but many times the problem, this is a problem you have later in the life of company because the first problem is to get users and when there are little users everything can be executed well in little resources, so I see a lot of Redis at something that is convenient in some way at Scale, like for example at Twitter Scale caching timelines with Redis could be interesting because it could be efficient compared with other systems.
But the reason and other aspect that’s the optimization of the developer time and this is a very, a matter of complexity of the database system and the match between the problem you are trying to solve and the data model of the database. And from this point of view I think because of the Redis data model is so far away compared to older system data models, there are problems that really fit well in the Redis data model and it’s possible to model such problems using less time, less complexity and it’s true the contrary as well, if you are trying to model a relational problem with Redis, probably you are going to do ten times the work needed with MySQL or Postgres or something else.
It doesn’t matter a lot, the only difference between different languages with Redis is that languages with basically a type system that’s very early binding, sometimes can be tricky for the designer of the client library to get the interface right because sometimes a given Redis call, can return different data types. So for example for Java it’s a bit more complex to design a client library compared to what could be for Ruby for example, but otherwise iin the Redis Land, there is the idea of taking the API of Redis exposed to the programming language without much changes. So the experience of using the client library for Java or Ruby or NodeJS is not much far away. The problem is with quality of the implementation of the client library. We see different levels of quality in the landscape of Redis clients that are certain that are more mature, developed with more care that are others that are developed in a more casual way and maybe are not as a good experience as they are, but there are databases projects where the client library are provided with their project, are up to their database developers, but in Redis it’s a completely bazaar model from the point of view of the client library, so it’s basically a completely community driven.
Rags: And that leads me to the next question which is I think you are talking about what are the challenges for Redis the technology and what are the challenges for Redis the community, can you talk a little bit about the community, where do you want to take it and the technology. You talked about that before about some of the challenges there are, but if you can summarize that quickly, that be good.
Sure, from the point of view of the technology, the challenges to now that we have these candidates for Redis Cluster is to put this outside with the help of the community bring it to a maturity state and this will happen it’s already pretty stabile, but in the course of the next months and years we will get at a point where people are very satisfied with the distribution model, with the operations and all the new problems that a distributed approach brings to the table because of course it solves a lot of problems but it creates a lot of problems, and from the point of view of the community because in the latest 6 years we always saw the user base growing and growing and growing and it is somewhat hard to keep focused, development focused, when are so many new users because the more users means that even if you have just 5 percent of users outside of what the database is already covering, so you have 95% of users that are happy with their current feature set and maybe just the 5% are not happy and want more, but this 5% is bigger and bigger as the user base grows so you get more pull requests, more complains and this sometimes can drive you outside the focus. So the idea is that you keep focused even if there are many users and you also find some way to still have something good for the long tail of use cases. For example Lua Scripting is something that allows users to extend a bit, to stretch the Redis API outside what is usually what you could do with it, and there are more features I’d like that it’s possible to add to make everybody happy. 14.36
Rags: That’s great. That brings me to the last question, I know Open Source is being used in the enterprises at unprecedented level, but despite that I think that is a general consensus that not many people are contributing back to Open Source, this is from the enterprise perspective, right? You got started with Open Source, you’ve been doing this for a while, what would be your advice to a developer who is in the enterprise who really wants to contribute back to Open Source but it’s not their daytime job.
I started with Open Source many, many years ago when Open Source was very different compared to what is currently. It was more hobbystic maybe but at the same time the community was more a higher quality of course because less people don’t usually have the effect of being high quality, and when this started it was when the Open Source movement was starting to provide a full stack framework basically for new startups to create things without upfront costs like you had Linux, you had Perl and it was the start of Ruby and Python and it was the start of MySQL and Postgres, so you start to have a lot of pieces to create new things without paying a lot of money, and this create a lot of possibility, for creativity in the landscape of startups.