Yes, currently I work with a company called IMIS out of Vancouver, British Columbia. We develop algorithmic trading systems that run in various stock markets throughout the world, generally running automated strategies or trader system strategies when we invest our own money. I'm currently the Chief Technology Officer and in general, I put a lot of time into our overall infrastructure, in building up our original system.
2. Can you tell us a bit about how you are using Domain-Driven Design and use cases of DDD?
Our main domain is an in-memory model representation of what the stock market is in the current state of and as it changes - that would be our domain. Our strategies all run within this domain, doing various things. The choice to use domain driven design here is actually quite an unusual one. For those who are unfamiliar with the industry I am in, speed is generally king. When you are processing 20,000 messages/second, the language that you use in order to process those messages usually is not considered a high benefit item. Also, generally, you will end up with procedural code, as opposed to object oriented code - it is going to be a lot faster. Your concerns generally focus on things like scalability, low latency, as opposed to the understandability of the code. We've tried to move more towards the bound side of having understandable code where we can make changes quickly.
Yes. I spoke a little about that last year at QCon. I am speaking a bit about it this year at QCon. One of the main differences that we've done from any classic version of domain driven design that I've seen is within our domain all the state changes that occur are explicit Entities themselves. As an example: when volume ought to be traded, we actually have a volume-traded command like to remove volume within our domain. That will actually generate out an event on the other side that will represent that state transitions that occurred within our domain.
This becomes really useful for us because, since we no longer have any concept of persisted state outside of our domain, we only have the actual state transitions. We can take all those state transitions and pass them to other bounded contexts, whereas in other bounded contexts the current state or the snapshot of the current state can be built up and viewed in different ways. Some examples of this happen every day in most systems, where your transactional system looks at data in one way, but your reporting systems tend to look at data in very different ways.
They like to have the data in a denormalized way, as opposed to a 3rd normal form within the database, by actually saving out the transitions to an event stream. What you can do is in your first model you are saving in whatever your normalized form is, but then, on the other side of the event stream you have a second model, which is denormalizing it into the form that it wants to see. This is generally a much better way of dealing with the bridges between bounded contexts as opposed to doing a synchronous call across them. It would be more scalable for you in the long term.
Yes, the biggest one is - I'm sure that most people who've used domain driven design have heard before - to only use domain driven design on appropriate projects. Domain driven design is a hard thing to get done and get done right. It is costly and you are going to end up spending a lot of time on it. Most systems that are out there don't have a cost function that allows for domain driven design to be done. If you are dealing with a project that is being done in less than a year, chances are it's probably not a project you are looking to do domain driven design on.
There are rare exceptions to these rules of thumb, but in general, I see most projects that are being done with domain driven design failing, not because people are doing domain driven design wrong, but because they are doing it on a project they shouldn't have had it done. Another thing I will bring up would be the understanding of the importance of Bounded Context. We don’t talk too much about them in many domain driven design circles, but bounded contexts are one of the very keys of domain driven design. They often come up when you are discussing in terms of language and you end up with a word that has multiple meanings to different people.
As opposed to making that one domain concept, we start getting confused. This will often segregate my domain into these different bounded contexts where my word has a meaning in each one of these contexts and I actually end up modeling it as such. This can greatly clarify my model and, over time, these bounded contexts help to modularize my system. A few other gotchas I'd bring back up would be to avoid setters where possible. This isn't so much domain driven design as it is object oriented programming. Objects have behaviors, not shapes. If I am setting a single field on an object, what is the behavior that is associated with me doing that? Chances are there is some behavior that should be there, but I've put it in the wrong spot - it's probably out in an application service, or in some code that's directly interacting with my object.
You can also run into a lot of problems with validation if you end up using the setters. The classic example: anything that has a circular reference. If you see these in your code, it should be an immediate sign that you need to bring in a behavior. As an example of a circular reference, if I were to have a zip code and state validation on an address - I know this is a conjured-up example, because you've probably introduced an Address value object, but let's imagine that when you change one you validate it against the other. If I already have an object and I'm going to update one of the 2, I'm always going to have a problem. I need to do one of 2 things at that point: I either need to let my object enter an invalid state, where my validation is not being met, then later come back through and say "Oh, are you valid?", or I need to introduce a method to allow me to pass both those values at the same time.
I am absolutely in the camp of keeping your objects valid at all times and therefore, you have to introduce your behavior. A lot of people - and this has actually came up recently in the domain driven design mailing list - say that it's too costly to allow your objects or to not allow your objects to enter an invalid state. Instead, the "is valid" methodology is simpler and easier to deal with. What is often not thought about in these situations is that what I put in it "is valid", I now actually have to call "is valid" as a precondition to any method that is going to be receiving my domain object. So if I have a customer approval service where I pass a customer, the first line of code in my customer approval service is being passed a customer has to be "Is my customer actually a valid customer or not?".
If you are using DDD, this should very quickly become a smell to you, but the big concern isn't even so much that I have to add that code, it's "What if I forgot to add that code?". I end up with invalid data that is slowly starts spreading through my entire system; it's no longer isolated within my one small area. Over time, this bad data just continues going across context, until you just end up with garbage throughout the entire system and you no longer know what is valid and what's not. In terms of best practices to give somebody that was new to domain driven design, I would tell them to always remember to use their language. Language is the most powerful tool in domain driven design; it leads you into a good system. It tells you when your system is bad.
If you have problems that you are talking to domain experts about that they can't come up and describe in your language, you've got a problem with your model. If the domain experts are describing something and they are using words that you don't know, you have a problem with your model, there are missing concepts. The best thing that you can possibly do it to go sit with these domain experts and to even go sit with the end-users. What words are the end-users using? Is there a report that's out there that they use from the data warehouse that has a term that you don't have in your domain? What is that term and how does it relate back to your domain? Getting through these types of language situations at first is kind of a tricky thing, but over time, it gets a lot easier. The biggest part at becoming good at domain driven design is actually learning how to manipulate your language and what those differences in language that you see really mean.
I mentioned before that I personally like dealing with valid objects and I also really like to not have any getters or setters on my domain objects, I prefer to have behaviors. Well, there is a big problem with that and people are going to say "What happens when I've got the customer and I need to display information about it on the screen?" My comment would be "Do you really need your domain object to do that or can we have some customer object, which actually represents a data transfer object to the screen?"
If we separate our query objects - meaning things are asking questions about our data, from our command operations, which would be things that we are using to write to our domain, in other words, our domain has become write-only - it opens up a lot of interesting possibilities for us. One of the big ones is because we have these 2 separate paths coming through, we can scale them separately. They don't actually need to go to the same data store. We could set up a denormalized system that was doing our reporting, versus a normalized system that we were executing our commands with. Then have a pipeline between the 2 systems that ran in some near real-time fashion.
The ability to move the discussions with your domain expert from "Everything is always consistent" to "How long should it be before this is consistent?" is a very powerful tool in terms of writing scalable systems. Other benefits you are going to get - I mentioned before that state changes become explicit - when you follow command query separation, your state change is also becoming explicit, mainly because your commands that you send out from your client are generally in the form of messages. These messages can be looked as my state transitions. Again, I can take these and put these into a pipeline as events on the other side and something can be listening to them, but the biggest thing that all this is going to allow me to do is to truly make my domain behavioral.
I can remove all the getters on my domain objects, I can remove all the setters on my domain objects and I am left with domain objects doing nothing but behaviors, because all they ever do is write. As such, I don't leak concepts like the shape of an object into my domain, so I cannot have an area of code that is dependent upon the shape of a customer - for instance, that a customer has a first name anywhere else in my system. If you follow object-oriented programming, what we are doing here is we are really going down the route of tell and don't ask and we are insuring that we are only going to have these behaviors that are exposed. When we're all said and done, we'll only have behaviors, what we've ended up with is beautiful seams throughout our domain, where we can easily interchange what objects do. I had a customer who had these defined behaviors around.
I defined a very strong seam around that object, I can change the implementation of that quite easily, without too much fear of breaking other things within my domain. In general, I consider these to be good things, perhaps other people don't, but it's a different way of looking at your system. And we start going down these roads, you can definitely see a difference compared to your typical object relational mapping solution, but I would still say this is domain driven design and it definitely enforces the desire or the inter-relation between domain driven design and good object oriented design.
If anybody reads the domain driven design list, these are or should be on the frequently asked questions page. There is a camp in particular DI, who says that everything must be injected. Then there is a camp that says "You don't have to inject anything, you can just have hard coded couplings". The answer is actually both. My rule of thumb, when it comes to things like Dependency Injection, is that coupling is OK, if it's at the same layer.
Things like Repositories should absolutely be injected, but when I am dealing with something that's at the same layer, for me, to couple myself to - let's say - a Service Locator isn't necessarily too bad of a thing. Having Entities that do these behaviors that may talk to their own Repository - their own Repository is a classic example, the other classic example is the Domain Service. It's actually OK. If you are going to avoid using the Service locator, then I will absolutely recommend avoiding it. One of the patterns I use a lot to avoid it is passing in, but if I have a method, let's say a customer.validate() and it ends up meeting a CustomerValidationService, basically I'll end up using a Double Dispatch pattern, or I'll pass in the validation service to it, as opposed to using a service locator injecting it directly into the Entity.
It leads to clear code and it's more intention revealing of what is actually happening there. For AOP, there are quite often cross-cutting concerns that arise in the domain. Transaction management is a great example, auditing especially the service layer. I would not say that AOP is required to do domain driven design, and I would certainly not say that Dependency Injection is required, and I know I am going against the statements of a fairly well-known person here. You can implement the same types of behaviors that you would see through AOP, especially - let's say - the application service layer through other mechanisms like passing commands and using a pipeline with filters.
You end up with these same behaviors after the fact, but it's a bit more explicit in the way that you are doing it and you've avoided a bit of magic. That said, I also wouldn't say not to use AOP. AOP absolutely has its place, but I wouldn't consider it to be a core concern in any way, shape and form. One thing there has been some interesting talk about lately, is when you start getting into true domain concerns that are also crosscutting, asides from obvious ones like authorization, but when you actually have true cross-cutting concerns of in your domain and how you model them in terms of language, personally I found that the best guidance for that is to go back to the original AOP based work. Greg Kiczales actually discussed heavily how aspects and language come together and in terms of AOP that's a much more interesting aspect of it for me.
Not necessarily very large, but very complex. I believe that what you are referring to is what my friend Scott (BlogCoward blogger) came here first with - his DDD Lite, or using DDD as a pattern language. There is absolutely benefit in using DDD as a pattern language. In terms of following some of the rules, I actually think they go a little bit too far, in particular with Aggregate roots and whether or not objects are allowed to be held outside of aggregate roots, whether or not an Aggregate root holds a reference to another aggregate root, whether you're favoring all single directional references or rather using bidirectional references.
I think in most projects, if you go through and use something like a Hibernate with a simple stereo-typical model and you actually use bi-directional support and you use all these nice things it has for you, you are going to end up in a much better place then if you try to do all the narrowing, unless you have complexity that warrants it. In terms of using the patterns, the patterns have validity in or outside of a domain driven design, I mean domain driven design did not make up the Repository pattern that may use it. The Value Object pattern existed before-hand - they are all of benefit outside of domain driven design.
I'm probably going to have to come back to that a little bit - this is a big part of my talk tomorrow as well. When I say mapping to DTOs, I don't mean mapping the domain to DTOs, I mean mapping the underlying data source to DTOs. -It's a lot easier to explain this with a whiteboard -. What we end up with is we have a very simple layer - let's imagine it is a web service - where I can say "Get me a customer". It goes directly to database - I don't care how it does it - and we have some form of shared customer DTO. The data that gives me back does not match up to my structure and my domain, it maps up to how I want to use the data in - let's say - my interface.
It may give me back the 5 pieces of data I need for my underlying screens, all in one go. It has nothing to do with my domain, my domain is write-only. The only time I ever talk to my domain is when I send it a command and say "I want you to go update that data". When you start looking at it from that perspective, the need for the getters really goes away, because that responsibility has been moved off. Moving that responsibility off is what allows us to have our behavior rich domain of true objects as opposed to shapes because we absolutely need to know shapes of these things at some point. We have to put them up on screens, we have to do stuff with that data, but, if we consider that to be a report, as opposed to being a transactional behavior and we actually architecturally segment those 2 responsibilities to 2 separate subsystems, it allows us to much more readily get to this point.
9. It sounds like Eclipse has some support for it.
Yes, Eclipse, ReSharper. Actually I think that IntelliJ IDEA does a pretty good job, too.
10. Are you suggesting that you only convert to the internal domain model when you write stuff?
Absolutely, and it is a very common system architecture in general. If we were to look at a typical domain driven design based architecture, we would have - let's say - an application service sitting there. This application service is going to have probably some query in methods - things like I need to get customer information, so I can put up on the screen, that's why I'm going to have some command methods "I need to change this customer's address, too". All the command-query separation is to recognize that these 2 things exist and to separate them. They both do not need to go to the domain.
Whereas the first one was going directly to a domain for the queries, we are just going to pull that across and we are only going to have our thin little DTOs, which we had before, but now we are going to populate them directly from some data source. It may or may not be the same data source that we write to. There may be 2 data sources and they may or may not be eventually consistent. Even if we start off with this single data source, we know that we have this ability to move towards an eventually consistent architecture layer, which will allow us for a large amount of scalability, if we needed that at a later point. Beyond that we've cleaned up our domain in terms of making a transactional only on the write side now. Our domain ends up cleaner, our DTO layer ends up cleaner and let me just say I would hate to be the junior programmer who has to write all the DTO mapping code - It's a pain! Been there, done that.
But the domain code becomes a lot more interesting because you are only passing through a single direction. What I generally do - I will use a pattern as an event sourcing. I read my DTO's and then I create a physical command message based on the data in my DTO's and I send that message to the domain and then the domain will tell me whether it succeeds or fails. On the back end of this you can then move those same messages or you can have a separate event stream coming out of your domain and /or finish then close the loop.
Yes, I have been writing for some time. I'm currently making some fairly large life moves, which are putting some things on hold, but I should be getting started back with that - probably the next 3 or 4 weeks - and I will be putting up chapters hopefully on my blog soon.
I will not claim to have too much expertise with Naked Objects and I will refrain from any opinion given that I don't have too much expertise on it, but I do know that the Naked Objects just came out with the version 4.0 .NET as well. In terms of other things that I've used, I really can't see how a framework would make things a whole lot better for me, but there are a few that can be fairly useful: in the .NET world, obviously something for DIs, like a Windsor Castle, you always have your tools like code weaving tools, like PostSharp or AspectSharp; one of the more interesting ones that's been seen over the Java side is, I believe the name is Qi4J, which supports many of the different paradigms they are going to be coming through and helps make them easier. The best tool I can say to get is a good template editor because you end up doing a lot of the things over and over again and that being able to write code from a template makes things a lot easier and a lot quicker to be able to do.
I have always believed that the only way to bring up junior developers in any technology - it doesn't matter what it is - is an apprenticeship program. You bring them up from apprentices and they move on, they pair a lot, they get code reviewed a lot, they are put on fake work, just like a real apprentice. You want to learn how to make tables.
Well, you are going to make a lot of tables. Not all of them are going to end up being sold. Sometimes you'll make a leg of a table just for the sake of making a leg on a table. Over time, they will build up the skills and the main way of how to build a lot of skills is by being in other people's code, who did know what they were doing and they will start seeing the good habits of those around them. The place where it fails and it fails all the way back to the pre-industrial age era is when you don't actually have somebody who is a master of their crafts, teaching apprentices.
When you end up with a team full of apprentices, there is not a whole lot that you can do - it's the kind of "The blind leading the blind". If you are in one of those scenarios, I would probably say that your best bet is to actually bring somebody in to start leading. If that doesn't work, I might try to go to books, or to take one of your apprentices - the one that you think is probably the highest - or maybe 2 of them and start trying to send them off the training, so that the rest of the team learns through osmosis. Overall, the key is just to have your most experienced people bringing along your least experienced.
14. Can you talk about managing the state transitions in the domain driven design world?
Yes, one thing that we did was we introduced the concept of treating all of our state transitions within our model in an explicit fashion. We did this for a lot of reasons. The 2 biggest ones are: in the financial area we need real audits and I've seen a lot of auditing systems that worked in different ways, but my question to them was "How do you actually know that your audit is correct?" If you make all of your state transitions explicit and then you act based upon those state transitions, that stream of state transitions is your audit. There can't possibly be anything else since you don't allow any state transitions in any other manner.
Since we also build up this event stream we can replay any version of it and we have our order, we have our timing and we have become the deterministic in systems that are often very difficult to become deterministic. This may also be just the financial industry, but another big reason that we did it was that it eased our testing, so basically, cutting into our domain we have a series of messages. Coming out of our domain and into an event stream we have a series of state transitions. If we were to use a given-when-then scenario: given a set of messages, when this message comes in, then expect my event stream to hold the following.
It aims for an easy state-based testing system. You can take this and you can very easily have your domain experts writing tests because they understand messages coming in and they understand the state transitions that are happening on the way out because they are all part of your Ubiquitous Language. The last and probably the most notorious reason that we did this was scalability. Obviously, dealing with a small black box system like this, with absolutely no needs for persistence and that just takes a stream in and writes a stream out, we can scale that very easily and we can deal with the hard things like I/O of these actual state transitions in a separate way. It also allows us to deal with them at multiple levels.
For instance, our domain in memory needs the lowest latency, that's why we make our real time decisions off of, so it is just listening to the event stream and the transitions are applied in memory. The next bit might be real time or near real time, let's say within 5 seconds, historical trades. They are being written out to a database and we can adjust the SLA to that one. Further down the line we might have hourly historical information that as long as it's there within 3 or 4 hours it's good. Aside from that we might have pricing that's used out on a website some place, that's being updated when we have free CPU cycles. By actually using this event stream model with our explicit state transitions, we can actually stage who gets priority on what. We can actually set up SLA's for which services get which events when.
Yes, we have used quite a bit of the Event Stream Processing. One of the most useful things we've seen for it is when you start having a series of messages, also in all these conversations, you can very easily use an event stream processor to start tracking your SLA's and start alerting you on your SLA's. You can start saying "Here is my average time for this, here is what I'm allowed, here is my maximum, my minimum, send an alert any time they get broken." Having these types of measuring tools in place in a system like this is beyond value. The ability to quickly identify problem areas when they are happening will make your operations people the happiest you'll ever see them.