Now in its ninth consecutive year, QCon London 2015 featured thought-provoking and engaging keynotes from "Eloquent Ruby” author Russ Olsen, Cobalt Advisors managing partner Enyo Kumahor, Principal Software Engineer, Google Infrastructure John Wilkes, and Netflix Insight Engineering Manager Roy Rapoport.
This was the largest London QCon yet with 1,200 team leads, architects, and project managers attending 120 technical sessions across 19 concurrent tracks and 13 in-depth tutorials. Attendees had near instant access to video from nearly all of the sessions.
This article summarizes the key takeaways and highlights from QCon London 2015 as blogged and tweeted by attendees. Over the course of the coming months, InfoQ will be publishing most of the conference sessions online, including video interviews that were recorded by the InfoQ editorial team. The publishing schedule can be found on the QCon London web site. You can also see numerous photos of QCon on Flickr.
- Cluster Management at Google
- Netflix Built Its Own Monitoring System - and Why You Probably Shouldn't
- Software Development Tales from the Continent
- To the Moon
- An Architect’s World View
- Evolutionary Architecture and Micro-services - a Match Enabled by Continuous Delivery
- Implementing Continuous Delivery: Adjusting Your Architecture
- Small Is Beautiful
- Treat Your Code as a Crime Scene
Architectures You've Always Wondered About
- Building a Modern Microservices Architecture at Gilt: the Essentials
- Rebuilding Atlas -- Advertising at Scale at Facebook
- Scaling Uber's Real-time Market Platform
- Service Architectures at Scale: Lessons from Google and eBay
Big Data Frameworks, Architectures, and Data Science
Devops and Continuous Delivery: Code Beyond the Dev Team
- Continuous Delivery: Tools, Collaboration, and Conway's Law
- Delivering Gov.Uk: Devops for the Nation
- Devops and the Need for Speed
- Making Continuous Delivery Work for You: The Songkick Experience
Docker, Containers and Application Portability
- Docker Clustering - Batteries Included
- Docker, Data & Extensions
- Docker vs Paas: May the Best Container Win
- How to Train Your Docker Cloud
- Securing "Platform as a Service" with Docker and Weave
- Cake Driven Development: Engineering at moo.com
- Dream Job? The Vision and Journey to the Company Culture You Want
- The Power of Hope: Getting You from Here to There
- Back to the Future: What Ever Happened to Being Extreme?
- Learning to Become Agile, with Retrospectives
- Progress from "What?" and "So What?" to "Now What?"
- Taking Back Agile
- Why BDD Can Save Agile
- Product Ownership Is a Team Sport
- Product thru the Looking Glass
- The Bad Idea Terminator
- The SensemakerŽ Method
- Microservices Are Too (Conceptually) Big
- Microservices: Software That Fits in Your Head
- No Free Lunch, Indeed: Three Years of Microservices at Soundcloud
- Operating Microservices
- Protocols of Interaction: Best Current Practices
Training
Design & Implementation of Microservices
by James Lewis
Twitter feedback on this training session included:
@fotuzlab: Experimentation is the key #qconlondon #microservices http://t.co/CLUI5HWabz
Java 8 Lambda Expressions & Streams
Twitter feedback on this training session included:
@dsommerville_nw: Impressed with lambdas and method references in #java8 so far - far less boilerplate and MUCH more readable. #qconlondon
Keynotes
Cluster Management at Google
by John Wilkes
Will Hamill attended this keynote:
John Wilkes … got straight into the examples of starting up services on Google’s internal cloud. Beginning with a simple ‘hello world’ service, John then created a cluster configuration request for 10,000 instances.
10,000 was the number picked because this is the default maximum in a ‘cell’, a unit of management of clusters. Only 9993 were actually started as some had failed, or more commonly some machines had been taken down for OS upgrades (a rolling scheduled process), or for various other reasons the exact upper limit was not reached, but closely enough to be reliable and at this scale you start to get an appreciation for how inevitable and continuous failures will be in part of the network. John gave us stats collected that indicate that on a 2000 machine cluster, you can expect to have > 10 crashes per day.
This appreciation leads to having more design discussions about reliability; about what happens if something fails while doing maintenance or upgrades, as resilience and fault tolerance are initially more desirable than focusing design effort on speed - as at these resource levels brute force can make up for speed in the short term. John also repeated the remark that we should “treat servers like cattle, not pets”, as while your development laptop is likely to be treated like a precious snowflake, the machines you deploy upon can be automatically created and destroyed much more easily. When the jobs/services you are developing have to tolerate faults in this way, it means that migrating tasks from one machine to another is dead simple: kill it and start up a new one elsewhere….
All of this deployment management, metering, reallocation, live experimentation and such is only available to the teams because Google has made such an investment in monitoring. John impressed the importance upon the audience: “If you are not monitoring it, it is out of control”. …
As demonstrated during the keynote, everything in Google internally runs on containers. Seeing the upcoming schedule, John said of Docker “we don’t use it internally, as we have our own system, but we really like it”. Google have also published the Kubernetes project, a tool for managing clusters of containerised applications that looks really interesting. Asked about when the utility of Kubernetes kicks in, John replied “If you’re going to do one or two or three containers just use Docker. Kubernetes helps you manage things if you have hundreds.” …
John ended the keynote by summarising with a call for incremental improvement, saying that the likelihood for success and building momentum is much higher than a big-bang project: “roofshot is better than moonshot”. John left us with three points to finish:
- Resilience is more important than performance
- It’s okay to use other people’s stuff, don’t do it all yourself
- Do more monitoring
Twitter feedback on this keynote included:
@Helenislovely: #qconlondon "we want our developers to be productive not jump through permission hoops" sounds good @google
@andyhedges: At google devs can spin up their code on up to 10,000 nodes without permission. They want their devs to be productive #qconlondon
@alblue: So Google's internal system for scheduling is called The Borg and they have a bunch of borglets #QConLondon http://t.co/G3q2jbqY34
@adrianmouat: The reason things look like they work from the outside is because we assume they don't work on the inside John Wilkes (google) #qconlondon
@mylenereiners: John Wilkes (#Google ): of course they do not work. Things break all the time. (...) that's OK. #qconlondon
@rvedotrc: For a service running on 2,000 machines (say, Google Calendar), 10 machine failures per day is normal, and fine – John Wilkes #qconlondon
@rvedotrc: Servers are cattle; you don’t care if you lose one. Your laptop is a pet.– John Wilkes, Google #qconlondon
@rvedotrc: “When Michael Jackson died, we thought it was a denial of service attack.”John Wilkes, Google #qconlondon
@floydmarinescu: Google mixes production apps (gmail) and batch jobs on the same machines for cost efficiency #qconlondon http://t.co/gzm6VPsjh2
@scottwambler: Do you understand all of the trade offs associated with a given strategy? Or just focused on how it affects you? #qconlondon
@rvedotrc: Real data is noisy. Live with it.”– John Wilkes, Google #qconlondon
@rvedotrc: Experiments [on live] are OK, provided you have a good way of stopping them in a hurry and rolling back”– John Wilkes, Google #qconlondon
@andyp1per: Exposing mechanisms [to users] is fragile - John Wilkes #qconlondon
@pzfreo: #qconlondon nice distinction between slo (objectives
@bruntonspall: If you are not monitoring it, it *is* out of control #qconlondon
@hnzekto: John Wilkes: "70% of our resources are spent in application monitoring." @Google Cluster Management #qconlondon
@rvedotrc: Everything at Google runs in a container – including our VMs.”John Wilkes #qconlondon
@pzfreo: Not news but confirmed. Google starts 2 billion containers a week #qconlondon
@ludovicianul: internally we don't use Docker, externally it turns out to be a good thing - John Wilkens, Google. #qconlondon
@csanchez: John Wilkes: "you shouldn't use Kubernetes in production until v1, which should be released in 1-2 months" #qconlondon
Netflix Built Its Own Monitoring System - and Why You Probably Shouldn't
by Roy Rapoport
Pere Villega attended this session:
His talk revolved around the NIH issue that affects many companies. If you have a problem,you should consider some questions first:
- are you the first person to have this issue?
- are you the first to care about it given your constraints: relevance to business, your scale, etc
- are you sure about the above answers?
In some cases you may really be the first and you may need to build your own solution. But most of the time that is not true as solutions, either paid or free, exist.
In the end, NIH is about trust: we don't trust other people's code, their product, their organisation, or that they will take good care of us as a customer. Even past performance of a 3rd party in other domains may move the balance towards building our own solution.
If the decision is to build your own product, that's ok.
Will Hamill attended this keynote:
Roy started with describing Netflix’s culture, which is also aptly detailed in CEO Reed Hasting’s now famous ‘Culture Deck’ which you should definitely read. Netflix optimises their organisation to increase the speed of innovation by fostering a culture of freedom and responsibility. Netflix have an inherent anti-process bias that tends to weed out suboptimal procedures; if it doesn’t work it will be corrected or abandoned.
Roy discussed ‘Not Invented Here’, recommending that when you have a problem to solve in your organisation, you should ask if you are the first to have it. Most times you are not the first to have that issue which means there is good news: there are already things out there to help that you can use….
NIH often boils down to ‘not invented by us, an organisation that we can trust’. A few reasons why we wouldn’t trust them are that we don’t trust the technical credentials; we have been warned away from them by people we do trust; we don’t trust that the other organisation has our best interests in mind (for example, they’re selling us something).
Occasionally, NIH is caused by CV-driven development. Roy argues that this is not always bad, as there is value in learning, there is value in improving the reputation of the company by creating a new product especially if you open-source the result, and value in keeping the developers happy in working on something challenging. …
Roy discussed one method for mitigating this when using OSS; forking the project and merging contributions back into it. Roy also talked about composability of the components within your solution: consider whether you may want to replace any of these with other people’s work in order to enhance the system. This would require a good separation of responsibilities across parts of the system to take advantage of such a plug-and-play approach….
Wrapping up, Roy said that when addressing NIH issues, dig in and find out which reasons are actually important to you. Find out if it’s really a technical decision, and if there is any way you can mitigate the effect while still meeting the needs.
Twitter feedback on this keynote included:
@csanchez: Netflix "we hire really smart engineers and stay the heck out of their way" @royrapoport #qconlondon http://t.co/sAEgdPH0mm
@trisha_gee: Process should be descriptive not prescriptive - it should describe what you already do, not tell you how to do it @royrapoport #qconlondon
@rvedotrc: “You added the config [by convention]. then magic would happen. Not necessarily the magic you wanted though.” @royrapoport #qconlondon
@danielbryantuk: Magic would happen, although often not the magic you wanted @royrapoport on home-grown software solutions at #qconlondon
@scottejames: Eventual consistency as a paradigm can ( should ) be applied to architecture decisions. #qconlondon
@danielbryantuk: No one is going to build a monolith without creating it from a series of components, right? @royrapoport at #qconlondon
@csanchez: Netflix: keep calm and build it yourself @royropoport #qconlondon http://t.co/4Kjg4aJSWO
@danielbryantuk: Allowing developers to build stuff they want can be good for morale and hiring. Just be clear when doing it @royrapoport #qconlondon
@trisha_gee: Making developers happy is a Good Thing. You might want to let them innovate @royrapoport at #qconlondon
@csanchez: Netflix monitoring processes 1.2 billion data points per minute @royrapoport #qconlondon
Software Development Tales from the Continent
by Enyo Kumahor
Will Hamill attended this keynote:
Enyo took to the stage for the closing keynote to give us an insight into some of the different challenges and opportunities that are encountered in Africa in software development. This was a really interesting session, giving a completely different view to how to enhance people’s lives with technology than the standard software company style of talks that filled the rest of the conference.
Enyo described with graphs and maps how Africa is considered a mobile-first and mobile-only continent. Mobile penetration is high with the average person having more than one phone [though it is likely a featurephone] in contrast to a very low uptake of wired broadband internet access, which is typically only prevalent in coastal areas where internet fiber connections join the continent.
Voice content and SMS based services are more popular, forming the majority of traffic on mobile networks, with data use very low. Interestingly, Enyo answered a question from the audience about low smartphone uptake and didn’t give the “smartphones cost a lot up front” answer I think most people (myself included) were expecting. Smartphones cost more initially but cost a lot to operate, as they must be charged every night. Not every person has on-demand access to reliable power, so charging would require going to somewhere a diesel generator is being operated. The cost of fuel has gone up considerably in recent years so this is prohibitively expensive….
Enyo stated that it was most important to use design thinking to get the critical context for how the software was actually to be used in order to solve the real problem. One of the side effects of being a developing continent means that few constraints to new systems already exist in this regard - Enyo elicited an “oooh” from the attendees as she deployed the line “we don’t have legacy code on the continent”!
Twitter feedback on this keynote included:
@peter_pilgrim: Crikey! A massive opportunity in #Africa where it is not uncommon to own 2 mobiles for each network #QConLondon http://t.co/FDio5Wc3Mg
@annashipman: Left pic is the *two* Internet cables servicing the whole of Africa in 2009. Fascinating talk by @enyok #qconlondon http://t.co/A37dWB7L0f
@shanehastie: #qconlondon @enyok Advantage of building software in Africa - no legacy code, all solutions are new
@shanehastie: #qconlondon @enyok Africa has manpower - technology shouldn't replace jobs it should support them.
@shanehastie: #qconlondon @enyok Software needs to enable a zero cost service, no barrier to entry.
@danielbryantuk: Understanding local context is key when developing software. Superb evening keynote about software dev in Africa by @enyok at #qconlondon
To the Moon
by Russ Olsen
Quinten Krijger attended this keynote:
The kick-off key note “To the Moon” was given by Russ Olsen. In a very entertaining and energetic way he reviewed the Apollo project. He started at the 1959, when the Western world was plagued by protests and fear because of the Cold War. At this time, Russia was ahead in the space race and had produced the first picture of the far side of the Moon. The reaction of president Kennedy was to declare the goal of landing on the Moon before 1970. Russ Olsen took a chronological approach though the Apollo project, reflecting often on how difficult this actually was.
Some nice things I will remember from this story.
- The software driving the Eagle that landed was developed by a woman named Margaret Hamilton. Designing the first program that needed to do multiple things at once and should be able to react to unexpected situations, Olsen states that she invented the term “Software Engineering” to describe the field of programming at an unprecedented level of complexity, where human lives were on line.
- Although the space race was itself a part of the Cold War, the side effect of it was a hopeful subject. When the landing actually took place, the sense of exhilaration was enormous.
- Very small faults can have life-threatening implication. As an example: the air lock not being completely vented when the pod was released from the ship that was at that time already orbiting the moon led to about a 0.1% difference in speed. The result was that Armstrong and Aldrin had to literally take last-minute measures to avoid landing on a very unfriendly piece of Moon. A good lesson for programmers!….
Aim for the moon. It’s in the nature of an engineer to do so.
Architecture Improvements
An Architect’s World View
Will Hamill attended this session:
Colin began his talk by outlining the structure of the architectural world view he wanted to describe: values leading to principles which are implemented by practices. The analogy given was how the agile manifesto states core values, and is backed by more specific principles and then implemented with particular practices. So without getting as prescriptive as practices, Colin told us about the values that he thought a conscientious architect should have and the principles they inform.
The values of an architect as Colin describes them are as follows: the customers of the architect are both the business and IT; an architect is interested in the big picture (conceptual integrity, as Fred Brooks puts it); leadership and humility - specifically being an enabler of the system rather than the boss; teamwork and an understanding of the types of people involved in the team and their needs (Myers-Briggs given here as an example of differences between different archetypes of persons); and finally the integrity and consistency to warrant the trust invested in the architect to deliver….
Simplicity is an important principle, ensuring that the models created, documents generated etc are done for a specific audience and purpose rather than for their own sakes. Injecting patterns into solutions adds complexity rather than simplicity - less is more. …
The next principle Colin talked about was just-in-time design. Delayed decisions are made with more knowledge about the situation. Deferring decisions to the last responsible moment allows us to investigate and try and attack the assumptions we would make. …
‘Deliver working solutions’ was Colin’s next principle. …
An architect should also keep learning; using retrospectives, lessons learned sessions and the likes to find out how the architecture or design you proposed actually fared in the real world. Find out what happened when the decisions you made and plans you set were actually carried out by teams, and how the implementation panned out in reality….
Quality should be a main consideration of the work of an architect - planning for testing and verification of the assumptions is important. An architecture that does not consider testability is usually not a good architecture. Know when ‘good enough’ occurs for your work, and try to attain this balancing point in ensuring you have built in enough quality and not over-egged the pudding.
Managing change and complexity was Colin’s final principle. … Expect that your architecture will need to change as we find out more about the problem domain. Don’t try and prevent change or preempt future needs, but create systems where changing your approach or throwing away parts of the architecture can be done if needed.
Twitter feedback on this session included:
@lamb0: Architecture should be simple and grounded in values that people can buy into #qconlondon @ColinGarlick
@piotrbetkier: Any intelligent fool can make things more complex, it takes a genius to do the opposite. At #qconlondon on talk about designing architecture
@marekasf: Antipattern: Ivory Tower Architecture @ColinGarlick #qconlondon
Evolutionary Architecture and Micro-services - a Match Enabled by Continuous Delivery
Will Hamill attended this session:
Microservices … tend to be smaller than SOA services …, smaller, and focused around single business capabilities instead of technologies. Microservices need to be independently deployable as they change at different rates, require little centralised management, and isolate tech choices internally from the other services that depend upon it. Microservices are often described as having smart endpoints but dumb pipes. Another very common factor is the lack of (what I call) the BandwagonDB - the single monster database that All The Data lives in; microservices are often responsible for their own data and sharing access or reporting is done via APIs.
More granular services become smaller and chattier but larger services are more inflexible and can suffer from complexity and coupling, so getting the size appropriate is tricky. The implications of pursuing a microservice based approach are heavily weighted towards the operations side: independently scalable implies great investment into deployment automation and continuous delivery; monitoring for services is crucial; it is impossible to pretend that service failures will not happen; eventual consistency in data needs properly addressed. …
Decomposing the monolith: consider DDD bounded contexts to help split responsibilities and business capabilities. Think about what the consumers for the service need - and if there is no consumer, then perhaps there is no need for that service? Consumer driven contracts for services can also form client tests for the interfaces. …
Evolutionary architecture is derived from evolvability of the system as a first class concern during its design. Tolerate and expect change rather than attempting to predict the future and lock in requirements that don’t exist yet. Being aware of Conway’s Law, we can try and design our teams to reflect how we intend the architecture - in particular arranging teams around the services to create for business capabilities.
Microservices is clearly a hot topic right now, but requires discipline, insight into the problem domain and above all is no silver bullet.
Quinten Krijger attended this session:
A nice concept here was the “Reverse Conway’s Law”. While intuitive and on some occasions actual part of our strategy consultancy, this was the first I heard it defined as such. Conway’s Law in short states that organisation structures will be reflected in the architecture of the software that the organisation develops. For example, a company without a DevOps culture (meaning that operations is a separate team) that tries to implement a microservice architecture, will probably end up with many components that are strongly coupled on an operational level that can’t be deployed separately. The “Reverse Conway’s Law” then, is to create software in the way you would like the organisation to be. You will need to be conscious of the many pitfalls of Conway’s Law itself, but when done correctly this can be a good way to induce organisational changes.
Implementing Continuous Delivery: Adjusting Your Architecture
Will Hamill attended this session:
Rachel began by describing the scenario when she was brought into a client site and given a demand by a customer exec “We want Continuous Delivery”. From working with the client and understanding their environment Rachel’s response was “you can’t have CD” - not a satisfying answer for an exec who wants to get to value. When working with the client and their complex codebase, Rachel came across a lot of the aspects of “you must be this tall to ride” barriers to entry of a microservices architecture and implementing CD. Three of the main things she learned were the implications of Conway’s Law, the importance of keeping things simple, and evolving the architecture….
The ‘big ball of mud’ architecture often results because expediency in releasing the system is focused over a clean and evolvable design. More code is added as more features are rushed out the door, increasing technical debt as no slack exists to preserve or improve quality as we go along. Big coupling problems happen in the codebase and the components in the design are pulled tighter together as this happens, which leads to inflexibility in operation. …
Rachel describes the pain of software architecture as dealing with the tension between striving for low coupling and high cohesion. Attempting to mitigate this on the big ball of mud systems means identifying the seams and interfaces between areas of different behaviour and writing tests around those boundaries to allow us to safely separate the components from each other. Rachel also described the Strangler Pattern to begin replacing parts of the older system and redirecting functionality to newer, cleaner components.
Rachel then went on to discuss the aspects of a microservice architecture that can mitigate against these issues. … Things to watch out for when moving to microservices include distributed transactions and an understanding of the domain (as services should split along domain boundaries and not technical ones). Rachel says that we don’t need to microservice-ify everything; that maturity and competence in continuous delivery is an important pre-requisite, and that automation and close collaboration with operations are very important to help manage the overheads of going from single monolith deployments to deploying, operating and maintaining many services.
Rachel finished off her talk with calling for an appreciation of evolvability in system architectures. You don’t need to design for Google scale now, but you should design for the ability to be changed. Architecture of the system is the things about it that are hard to change. The parts these are usually correspond to -ilities and bigger decisions made that can’t be unmade cheaply or quickly; to identify where these decisions are you need to be talking to the customer about their scale, security, business needs, and future direction. Creating an architecture where components can evolve separately, with less constraint to change, is more important than trying to predict an unknown future. Areas which need to change most often are likely to hide the most complexity. Putting more emphasis on the testability of these areas and treating the testability as a top level requirement of the architecture will result in a higher quality system.
Twitter feedback on this session included:
@paulacwalter: Flexibility of organisation is key for effective design. Otherwise it's very hard to make changes where needed @rachellaycock #qconlondon
@AndrewGorton: Software architecture represents the tension between coupling and cohesion @rachellaycock #qconlondon
@randyshoup: Yesterday's best practice is tomorrow's anti-pattern @rachellaycock #qconlondon
@randyshoup: Hope is not a design pattern @mtnygard via @rachellaycock #qconlondon
Small Is Beautiful
Yan Cui attended this session:
Kevlin has plenty of well-applied, memorable quotes, starting with this one:
Sustainable development is development that meets the needs of the present without compromising the ability of future generations to meet their own needs. - the report of the Brundtland Commission
when applied to software development, my interpretation of it is : “don’t take on more technical debt than you can reasonably pay back in the future in favour of short-term gains”….
On the other extreme of the spectrum, you have people who are so concerned about future needs they end up completely over-engineering their solution to cope with this uncertainty and end up with projects that are delayed or worse, never delivered.
You should think of software as products, not projects. If software are projects then they should have well-defined end state, but most often, software do not have well-defined end state, but rather evolved continuously for as long as it remains desirable and purposeful….
Creativity needs a boundary. Without any boundaries, a painter might be lost if you just ask him to “draw a picture”, and would you create anything more than a “hello, world!” application if asked to just “write a program”? …
Kevlin also pointed out another good point – the more time you spend working on a project, the more the endowment effect kicks in and we become less inclined to change. …
Twitter feedback on this session included:
@JanSabbe: Best way to deal with legacy code? Beer. #qconlondon
@adrianmouat: 'This legacy system is really small and comprehensible' - things people don't say @KevlinHenney at #qconlondon
@alblue: “It hit the deadline — at some considerable speed judging by the fallout” — @KevlinHenney at #QConLondon
@camassey: Software is executable fiction -@KevlinHenney #qconlondon http://t.co/pK9mLdPqca
@camassey: Coding styles, if practiced by enough people, *are your architecture*. @KevlinHenney #qconlondon
@camassey: For any activity, there is an appropriate scale. @KevlinHenney #qconlondon
@camassey: If you are striving for beauty or elegance, *constraints are necessary*. You NEED a boundary. @KevlinHenney #qconlondon
@jgrodziski: Softwares do NOT have economies of scale @KevlinHenney #qconlondon small software is cheaper http://t.co/BxxAv0VuGD
@daverog: Unlike milk, software gets more expensive, per unit, in larger quantities (diseconomies of scale) @KevlinHenney #qconlondon
@camassey: Trees nicely describe a neatly decomposed structure. Except that the real world is complex @KevlinHenney #qconlondon http://t.co/78vg0eefM9
@camassey: We design & staff-up teams at the start of the project. When we are the *most* ignorant about its requirements. @KevlinHenney #qconlondon
Treat Your Code as a Crime Scene
Ben Basson attended this session:
Adam quickly introduced the idea of Geographical Profiling - a criminal investigative method used to help narrow down the likely area where a serial offender may live or work, based on the location where the related crimes were committed. I very much enjoy the idea that bugs are essentially "code crimes" and that we may be able to leverage data in such a way as to zero-in on troublesome areas in this way.
Of course, to do this we can't just look at the code in its current state, we must draw upon revision history and statistics from version control, and then analyse and present this data in a useful manner. Adam introduced a number of potential visualisations, including the intriguing Code City, where lines of code are represented in the height of the generated buildings.
Adam goes on to suggest that the code quality of an individual source file is inversely related to the number of programmers that have worked on a source file (the theory being that more people get involved in troublesome areas as they have to be touched more often), and that while there are lots of measures of complexity, the number of lines is in most cases a pretty good indicator.
The final suggestion from Adam was that in addition to analysing code in this way, it would be interesting to experiment with more proactive warnings or monitoring - letting developers know when they're about to work on particularly complex or commonly edited code (i.e. here be dragons). He also suggested that version control tools could implement Amazon-style recommendations; "other developers that worked on file A also worked on file B", which sounds like a great idea.
Yan Cui attended this session:
Many studies have showed that we spend most of our time making changes or fixing bugs, which always start with understanding what the code does. We should therefore we optimize for that.
A common problem we face in today’s world is that software is produced by many developers across many teams, and no one has a holistic view of how the whole looks.
When it comes to measuring complexity, both lines-of-code and cyclomatic complexity are useful metrics to consider even though neither provide a full picture of what we’re up against. They are useful because they fit nicely with our main constraint as developers — our working memory.
Adam shows us how techniques from forensic psychology can be applied in software, specifically the practice of geographical offender profiling. …
Using tools such as CodeCity you can lay down the geography for your code which reflect their complexity…. Adam also showed how you can track complexity of hot spots over time and use them to project into the future with Complexity Trend analysis….
Temporal Coupling – by analysing your commit history, you can find source files that are changed together in commits to identify dependencies (physical coupling), as well as ‘copy-and-paste’ code (logical coupling)….
By showing the number of commits each developer makes on a source file you can identify the knowledge owners of that part of your codebase. …In the perfect world, all knowledge owners for a component (source files for one project, for instance) would be concentrated within a team, which shows that the responsibility of that component is well defined and aligns with the organizational structure.
Twitter feedback on this session included:
@ignazw: Code as a crime scene. Pretty cool! There's a lot of crime info in your software control system. #qconlondon http://t.co/EQQVEQ7OOt
@willhamill: But the change took ages because DBAs are where Change Requests go to die - @AdamTornhill at #qconlondon
@danielbryantuk: Adding extra software development teams to a project increases communication channels @AdamTornhill at #qconlondon http://t.co/fEnD918NHq
Architectures You've Always Wondered About
Building a Modern Microservices Architecture at Gilt: the Essentials
Will Hamill attended this session:
Yoni described how Gilt made three main architectural changes to their application: moving the application platform to the JVM (primarily Scala) for perceived platform maturity, the stability & concurrency benefits and the garbage collection; refactored the single Postgres database into dedicated data stores for different parts of the application; and began splitting the monolith up by behaviours, which Yoni called entering “the era of macro and micro services”.
Initially splitting the application into a small number of services met the majority of the scaling needs but most of the developer pain was not solved: the new services became almost monolithic due to size and internal complexity and still the codebases had little ‘ownership’ and integration and deploys were painful.
The team at Gilt then doubled-down on the microservices approach, reducing the scope of individual services and empowering the teams as the owners of the service responsible for the deployment and focusing on continuous delivery as a means of streamlining the releases. APIs used for the microservices to communicate with each other were defined by an ‘API design committee’ in each team and documentation generated using Swagger. The front end was decomposed into a larger number of Play and Scala applications responsible for different sections of the website. For example, the search pages, product pages, checkout and so on are all served by different apps….
Data ownership was devolved to the teams operating the services, and each team chose the best solution for storing their data. Managing databases, a schema evolution manager independent from the service code was responsible for DB changes, deploying updates as tar files to be applied to the database. Fix-forward was the approach taken to DB migration, with no rollbacks.
Yoni also described the concept of ‘mid tier microservices’ which exist to aggregate multiple calls to many fine grained services (for example a ‘customer’ mid-tier service aggregating calls across half a dozen or more specific ‘customer account’ or ‘customer profile’ type services) to cache, decorate and collect results needed by other depending services.
Pere Villega attended this session:
Yoni Goldberg, Lead Software Engineer at Gilt, explained the Gilt moved from a Ruby monolith to a Microservices approach. The reason was that Gilt operates a model of flash sales with massive spikes, and adding certain vendor caused cascading errors across the site. To fix the issues they moved to JVM, started what he calls a macro/micro services era and used dedicated data stores.
During the process they realised the new services where semi-monoliths, not fixing all of the issues, so they kept working until they reduced the scope of the services (both for back-end and front-end, they have multiple webapps for UI), which in turn facilitated deployment and rollbacks.
Currently they have 300 services in production, and their data show a very interesting pattern: once they had in place all the right tools such a team could go from creation to deployment of a new service (a basic placeholder) in around 10 minutes, the productivity and the number of services increased a lot….
Something that has facilitated the adoption of Microservices has been a well defined API. Yoni argues that a well defined API solves issues like discoverability, documentation and internal adoption. Tools like Swagger facilitate this.
An issue associated to having so many Microservices that is not mentioned often is that the network ends up acting as a bottleneck, due to the amount of calls generated. Their solution is to create mid-tier Microservices, which are a coarse API that hides multiple small services from the request. Your application just calls an endpoint and that, in turn, does several calls. This reduces the load, even more if you use caching….
Gilt also uses micro-databases, that is, every Microservice has its own database, they are not shared across services. This means the service owns everything: API, data, behaviour. No conflicts between services due to shared databases.
Finally, they don't advise teams of less than 30 people to go into Microservices due to the manpower needed.
Rebuilding Atlas -- Advertising at Scale at Facebook
by Jason McHugh
Twitter feedback on this session included:
@charleshumble: Time spent on mobile in the US surpassed time spent watching TV. Just mobile. #qconlondon
@charleshumble: Facebook Atlas uses Presto extensively. Useful when you need a relational model and can't easily Shard #qconlondon https://t.co/imrxOi9MPn
@ignazw: So Facebook aquired Atlas for 100 M$ ... http://t.co/NO9ZCqguoV via @forbes #qconlondon
Scaling Uber's Real-time Market Platform
by Matt Ranney
Leo Simons attended this session:
My favorite talk of the day is from Matt Ranney, who talks about uber's real time challenges. It's a pretty quirky talk about a pretty quirky architecture. The shape of uber's problem is a bit different from a lot of other big architectures, and so Uber are doing various interesting things really differently. For example, we learned that when failing over a data center, uber stores the active trip data on the driver's phone, and when that phone gets routed to the new data center, it's tasked to re-upload that data to the new data center. This means they get to avoid expensive cross-data center replication for the hot data. We also learn that Uber is being a good open source citizen and open sourcing various interesting bits; I'm definitely going to be studying some of that!
Yan Cui attended this session:
Uber’s services are written in a mixture of Node.js, Python, Java and Go, whilst a whole mix of databases are used – PostgreSQL, Redis, MySQL and Riak….
In order to scale their services, Uber went with an approach of building stateful services using Node.js. In addition, they also introduced a custom RPC protocol called ringpop, which is based on the SWIM paper. Ringpop also runs on its own TChannel multiplexing and framing protocol….
For Uber, availability is of paramount importance, as the cost of switching to competitor is low.
Twitter feedback on this session included:
@randyshoup: Never underestimate the power of developer enthusiasm @mranney @Uber #qconlondon
@csanchez: Uber's dispatch system is written in NodeJS. DBs used: Redis, Postgres, MySQL, Riak,… @mranney #qconlondon
@glynn_bird: Uber talk their AP data layer: "we always favour availability because the user will switch to a competitor if we're down" #qconlondon
@markgibaud: In NodeJS, tchannel [Uber's custom RPC protocol] is 20x faster than HTTP - @mranney at #qconlondon
@csanchez: Uber availability: everything retriable, killable, crash only (no graceful stops) even databases @mranney #qconlondon http://t.co/1ro9il6tLs
@vwggolf3: #qconlondon Uber fails over data centers by using state & data stored in partner phones
@FZammit: Uber using mobile app as a failover mechanism #developers #qconlondon
@randyshoup: @Uber fails over between data centers by having driver apps regularly replay their state. Clever! @mranney #qconlondon
@colmg: Everything is retryable, everything is killable at Uber - @mranney #qconlondon @aolireland
Service Architectures at Scale: Lessons from Google and eBay
by Randy Shoup
Yan Cui attended this session:
At Google, there has never been a top-down design approach to building systems, but rather an evolutionary process using natural selection – services survive by justifying their existence through usage or they are deprecated….
Services are built from bottom-up but you can still end up with clean, clear separation of concerns.
At Google, there are no “architect” roles, nor is there a central approval process for technology decisions. Most technology decisions are made within the team, so they’re empowered to make the decisions that are best for them and their service. …
Even without the presence of a centralized control body, Google proved that it’s still possible to achieved standardization across the organization.
Within Google, communication methods (e.g.. network protocol, data format, structured way of expressing interface, etc.) as well as common infrastructure (source control, monitoring, alerting, etc.) are standardized by encouragement rather than enforcement. …
Whilst the surface areas of services are standardized, the internals of the services are not, leaving developers to choose:
- programming language (C++, Go, Python or Java)
- frameworks
- persistence mechanisms…
If it proves to be successful then it’s extracted out and generalized as a service of its own with a new team formed around it. Many popular services today all started life this way – Gmail, App Engine and BigTable to name a few….
As the owner of a service, your primary focus should be the needs of your clients, and to meet their needs at minimum cost and effort. This includes leveraging common tools, infrastructures and existing service as well as automating as much as possible.
The service owner should have end-to-end ownership, and the mantra should be “You build it, you run it”.
The teams should have autonomy to choose the right technology and be held responsible for the results of those choices.
Twitter feedback on this session included:
@ignazw: Large companies change their architecture regularly #ebay #twitter #amazon #qconlondon http://t.co/DZAjSMPCYY
@AlibertiLuca: #qconlondon why is Google so fast ?? Simple :) http://t.co/ZZIRW9fwTu
@jabley: 'At Google, most technology decisions are made locally rather than globally. Better decisions made in the field.' – @randyshoup #qconlondon
@charleshumble: No architect role at Google. No central approval for technology decisions. eBay did have architecture review board @randyshoup #qconlondon
@grantjforrester: “Standards become standards by being better than the alternative” @randyshoup #qconlondon
@jgrodziski: In a mature service ecosystem, we standardize the arc of the graph, not the nodes #qconlondon @randyshoup
@charleshumble: Every service at Google is either deprecated or not ready yet. Google engineering proverb. @randyshoup #qconlondon
@solsila: On establishing standards: make it really easy to do the right thing and really hard to do the wrong thing. @randyshoup #google #qconlondon
@a_alissa: #qconlondon google does not enforce standards, each team can use any programing language and libs they want#empowering_teams
@djmcglade: Microservice - the word is relatively new, the concept is relatively old @randyshoup #qconlondon
@charleshumble: Teams should be no larger than can be fed by two large pizzas. @randyshoup #qconlondon
@jabley: Good use of economic incentives to align service teams at Google. Charge downstream teams to grow more considerate customers. #qconlondon
@jgrodziski: Risk of code change is nonlinear in the size of the change @randyshoup #qconlondon
@charleshumble: Every code submission is reviewed at Google. #qconlondon
@charleshumble: You can have too much alerting but you can't have too much monitoring @randyshoup #qconlondon
Big Data Frameworks, Architectures, and Data Science
Evolving a Data System
by Simon Metson
Twitter feedback on this session included:
@paulacwalter: Choose a realistic problem, not "We need to fix all of our IT in the next six months. Here's a L20 note. Go do it." @drsm79 #qconlondon
@paulacwalter: It's not rocket science. Identify a problem (this is the hardest part), build a solution, evaluate it and repeat. #qconlondon @drsm79
@paulacwalter: Evolving Data services. The relevant problems are not technical. How are we going to talk to each other and share data? @drsm79 #qconlondon
Devops and Continuous Delivery: Code Beyond the Dev Team
Continuous Delivery: Tools, Collaboration, and Conway's Law
Twitter feedback on this session included:
@dsommerville_nw: Conway's Law (and the Inverse Conway Maneuver) becoming a recurrent theme at #qconlondon
@DevOpsGuys: Bring people with you, appreciate current skills #qconlondon #devops
@DevOpsGuys: Optimise globally across the teams that need to collaborate #qconlondon #devops
@camassey: Silos exist across environments as well as roles. Don't optimise your pipeline for just one environment! @matthewpskelton #qconlondon
@Idris_Ahmed251: More dev teams solves nothing. Adds coupling with people's work, causes merge problems! (Conway's Law) #qconlondon http://t.co/oBPJ70WcQV
@camassey: Conway's law has HUGE implications for org architecture, if you want particular software architectures @matthewpskelton #qconlondon
@marekasf: microservices are like children: they're small, cute and the more the better "@DevOpsGuys #qconlondon @camassey http://t.co/IMRaoVaBeO ”
@julianghionoiu: The organisation's topology should closely resemble the application's architecture. #qconlondon http://t.co/h8aVdjxn7k
@neilisfragile: Hadn't considered that tool choice not only to promote collaboration, but also to dissuade certain interactions #qconlondon @matthewpskelton
Delivering Gov.Uk: Devops for the Nation
by Anna Shipman
Ben Basson attended this session:
The things about the talk that I found interesting were:
1. There is a well-maintained operations manual to help people support the live services, so if someone is on-call and doesn't know a particular area that well, they can draw on a wealth of information - or write that information for the next person once the problem has been investigated and resolved. This is clearly a good idea that all companies should really have in place.
2. Deployment to production is managed by the requirement to have custody of a stuffed toy badger in order to deploy. I don't know if it's an intentional spoof of the Government's policy on culling badgers, but I couldn't help but chuckle slightly at the irony. It seems a little silly, but I can see the merit, especially as…
3. Developers can deploy from their own laptops - a stark contrast to the usual Government process of using dedicated, locked-down machines with direct VPN access to data centers.
Twitter feedback on this session included:
@matthewpskelton: .@annashipman "#DevOps is a *culture* where developers and operations people work together" #qconlondon http://t.co/70FxG0lOz0
@rvedotrc: Allowing the developers to deploy using their own hardware, not locked-down gov hardware, was a big win says @annashipman #qconlondon
@rvedotrc: Heartbleed announced at 10pm, patched by 2am, deployments done from home, just because devs cared - @annashipman #qconlondon
@rvedotrc: “Are you sure the deployment process will work?”“Well, we have done over 1000 of them already”- story from @annashipman #qconlondon
@matthewpskelton: .@annashipman "Technology choices at @gdsteam are *not* top-down" < +1 chosen by team in collaboration #qconlondon
@danielbryantuk: Use what technology you like, as you're going to be supporting it in production paraphrasing @annashipman on GDS DevOps at #qconlondon
@matthewpskelton: .@annashipman "I could not bring @BadgerOfDeploy with me today because that would stop deployments!" #qconlondon http://t.co/pOfdf9bzd7
@matthewpskelton: .@annashipman "The most important tool we have is our Ops Manual" "It's a living document" < +1 #qconlondon
@peter_pilgrim: GOV.UK now puts their operational service manual online on GitHub. "Document everything for people who are new to it." #qconlondon #in
@camassey: How to bring in #DevOps: Document All The Things -@annashipman #qconlondon http://t.co/elPCYQeGCR
@camassey: #DevOps has implications for everything - inc. hiring, leaving, and (obviously) team trust. @annashipman #qconlondon http://t.co/qJBRcRv43E
@phuturespace: #qconlondon. Great talk by @annashipman. Great to see a practical successful application of DevOps.
@matthewpskelton: .@annashipman "I do not see architecture as Command & Control, but instead to help the teams and then get out of their way" #qconlondon
Devops and the Need for Speed
Twitter feedback on this session included:
@lamb0: If you don't engage HR and Finance, then you will fail to adopt devops, it’s a mindset and organisational model #qconlondon @TheOpsMgr
Making Continuous Delivery Work for You: The Songkick Experience
by Amy Phillips
Twitter feedback on this session included:
@rvedotrc: “If you get 4 people to look at code for 2 hours before release, I guarantee you, you *will* find a bug” - @ItJustBroke #qconlondon
@matthewpskelton: .@ItJustBroke "Adding more developers did not make things faster" #qconlondon http://t.co/fSDHEGx2IC
@matthewpskelton: Release processes need flexibility and risk assessment - @ItJustBroke #qconlondon http://t.co/AHgug7UZDs
@rvedotrc: “Features add no value until your users are using them” – the argument for fast turnaround, by @ItJustBroke #qconlondon
@matthewpskelton: .@ItJustBroke "We asked the business to help define the acceptance tests" +1 parallel pipeline stages #qconlondon http://t.co/XDzf4pN0Fn
@rvedotrc: “Limiting our automated Selenium acceptance tests to around 5 minutes, gives us the level of assurance we need” – @ItJustBroke #qconlondon
@rvedotrc: Identify the biggest problem with your process. Fix it. Repeat.Using problems to drive positive change.– @ItJustBroke #qconlondon
Docker, Containers and Application Portability
Docker Clustering - Batteries Included
Will Hamill attended this session:
Docker supports clustering of containers OOTB with Swarm which serves the standard Docker API and allows transparent scaling to multiple hosts. If Swarm isn’t your bag, LibContainer which is also written in Go can be used, and LxC containers are now supported as well.
Service discovery is also provided OOTB with Docker, though it can be configured to use etcd, consul or zookeeper instead. For scheduling, bin packing is provided OOTB and there is also a native option, with Mesos currently on the way.
Jessie then gave a demo of using Docker with Swarm to define clusters of containers and manage them on the CLI. Regular Docker commands for individual container management work with Swarm, and also Swarm adds a number of commands for provisioning clusters, joining containers to clusters and the like with simplicity: docker pull swarm,docker run --rm swarm create and docker run -r swarm join….
Wrapping up, Jessie outlined the future direction of Swarm; rescheduling policies, further backend drivers for OOTB management functionality, support for Mesos, cluster leader elections and more & faster integration with new Docker features.
motwin attended this session:
A new tool in the Docker ecosystem: Swarm, which is a cluster management system for Docker containers.
This is a native clustering system for Docker with:
- native discovery of containers (and optional feature based on either etcd, consul or Zookeeper)
- schedulers (bin-packing and random which are native support and soon Mesos)
- constraints management
- affinity management
Docker, Data & Extensions
by Luke Marsden
motwin attended this session:
Fig alias now Docker Compose enables composition at the host level. For instance, if you have an application deployable on a servlet container that needs a database, you may choose in a microservices approach to use one Docker container for your servlet container and one for your database. But you need to deploy and run these containers in the right order (the database first and then the servlet container), links these two containers to each others, setup the endpoints / ports and so on…
Flocker can be seen as the companion of Fig. In addition of a Fig yaml configuration file, Flocker needs a second yaml file that describes the topology of your Docker containers cluster: you describe on which node each of your Docker container has to be installed. The description of the Docker container being hold by the Fig file.
A second issue addressed by Flocker is the migration of a Docker container from one node to another node. Let’s say you have a database wrapped in a container. To persist the data stored in the database, you can use Docker volumes that enables to persist data outside the container in the filesystem of the host. What happens if, for one reason or another, you wish to migrate the database from one node to another? Flocker does the job: it can migrate a such container from one node to another. …
Another miss of Docker is a plugins / extension mechanism. Right now, it’s hard to glue some tools based on Docker (for instance, Weave and Flocker).Powerstrip may circumvent this issue. It’s an open-source project which aim to rapidly prototype extensions to Docker and enables to glue them. …
What I have learned is also that using Docker volumes leads a coupling between the Docker container and its host. Hence, you can have issues when it handles to migrate a such container to another host.
Docker vs Paas: May the Best Container Win
by Paula Kennedy & Colin Humphreys
Will Hamill attended this session:
This talk was about discussing the difference in needs which may lead you to choose Docker over PaaS - obviously a straight comparison of one versus the other would be illogical, so the two tried to point out the areas where one approach is stronger than the other. PaaS can easily handle multiple application instances and can have autoscaling rules defined; Docker does exactly and only what you configure it to do. PaaS can feature shared services such as health checking of tenant applications, centralised log aggregation, etc; Docker does not seek to provide this and you would need to create it yourself.
Docker is more about the basics - letting you run your application in a lightweight containerised environment and moving or creating new instances of that container rather than value-adding features like PaaS now tends to be. Docker focuses on customisability and control in ways that you cannot control on PaaS. Docker container provisioning is much faster than instantiating a new virtual machine on IaaS.
Colin and Paula argued each other down to an agreement: PaaS is likely to be better for fast iteration of a basic application, and Docker is likely better for control and more specific needs such as database management. Colin recommended that PaaS be considered more for apps following the12 Factor principles, and containers with storage volumes used for stateful micro-services….
Overall I was convinced that the argument comes down again to whether you want to give up control of low level concerns in order to benefit from paying for more hands-off deployments and scalability, and if you can live with the lock-in that PaaS tends to imply. It depends - on your particular environment constraints :).
motwin attended this session:
Colin and Paula agree that there is place for both PAAS and containers:
- if your micro-services require to fit the 12 factors, then a stateless PAAS can be your holy graal
- if your micro-services didn’t require to fit the 12 factors, then Docker containers with volumes management can do the job
How to Train Your Docker Cloud
motwin attended this session:
Clocker is a Docker containers cloud manager that can deploy applications described in the Brooklyn blueprint format. It can deploy the application on containers of several nodes and across multi-hosts. Clocker seems to have lots of features:
- autonomics: scaling policies that can be driven by sensors, cluster resizing
- health room: to ensure resources availability (cpu, memory, etc.)
- container management: with Docker images catalog, support of Docker files, creation of images automatically
- placement and provisioning: on demand, with several possible placements strategies (random, CPU, memory, geography, and so on)
- network management: with network creation, IP pool control, Docker ports forwarding for debug purposes, pluggable network providers (Weave,Kubernetes, libswarm), network virtualization
Securing "Platform as a Service" with Docker and Weave
by David Pollak
Will Hamill attended this session:
David Pollack, the creator of Lift, began his talk about securing PaaS stating he believes that security skills require a different mentality to most developers, and understanding of more granular responsibilities. David said that he wanted to try and hire more replaceable people rather than creating esoteric tech experts (for obvious business reasons) so he preferred more widely understood and adopted technologies for securing his platforms - Docker and IP tables being better collectively understood than JVM Security Manager, in David’s example. David also praised Docker’s ease of use, providing a declarative format for configuration instead of relying upon Perl scripts and raw LxC containers….
One of the problems David had was considering not only layers (a typical approach to both physical and application security) but also isolating the tenants of the PaaS from each other. Tenants’ applications needed to run inside containers on virtual LANs that can talk to each other and shared backend resources but not other tenants. Shared services at the backend may be subject to potential attack, so splitting into read only or write only services can limit surface area and impact.
David addresses these issues in his platform with each tenant application deployed into a Docker container, using Weave to define the tenant-specific subnet and IP tables to secure the access to the rest of the network. Shared data services in an RDBMS use table or column level access controls managed by the RDBMS, and I/O heavy services with well understood security models can also be shared. Credentials for services are isolated to each tenant and not globally visible.
David said that he was happy with Docker’s security as LxC containers are a reasonably well understood technology and the new popularity it meant that there are many eyes looking at it both to exploit and improve it. Finishing his talk, David said that he thinks the move from VMs to containers is as big a shift in approach and utility as the shift from physical machines to VMs; that IP tables still work just fine for network level application isolation; and that a layered approach to isolating risks is still the best approach.
motwin attended this session:
David Pollak distinguishes 5 threat models:
- app to shared services (e.g. credentials)
- app to the world via the network
- app to the host via the code that runs on the host
- app to the host via the network
- app to app via the network
To prevent from these vulnerabilities, Docker provides a quite reasonable isolation from the host, while Weave subnets can isolate tenants (i.e. different apps). As far as iptables, they can secure the rest of network.
At last some takeaways from David Pollak:
- –icc=false in DOCKER_OPTS which means no intercontainer communication except via Weave
- use iptables to control / restrict the bridge traffic to well known ports and public hosts
- partition tenants onto separate Weave subnets
Engineering Culture
Cake Driven Development: Engineering at moo.com
Will Hamill attended this session:
Mike described how a few years ago Moo was facing troubles internally with meeting the needs of the business to release new products to market quickly … the teams were dissolved and reformed into cross functional groups responsible for specific business areas/products. The new teams, called crews, had responsibility for making a new cocktail and hosting an event to welcome each other into the new form of the business … Crews were given specific business-aligned goals for their areas of work, were allowed to create their own workflows and were not forced into homogenising with the rest of the company. Autonomy in how the crews accomplish their goals is a powerful factor of motivation.
Mike then revealed that most crews had stopped doing formal estimation of work items - instead of producing estimates for each item and planning an iteration, the crews moved to a flow based system, doing planning as needed and working to improve the product backlog. The business don’t care that you are not doing detailed estimates for each piece of work when you can prove that you are releasing new working software on a regular and reliable basis. Product managers from the crews were mediated when they clashed by a crew lead representing the overall business goals.
After the reorganisations, the smaller cross functional crews had better decision making as everyone needed to understand how they can release working software was embedded in the teams. The development manager role is being replaced with a platform manager, someone with vision across the teams and who can help balance doing things fast with doing things right. Another role added was the ‘people engineer’, combining HR responsibilities with the tech team lead responsibilities.
In day to day work terms, Mike described the culture in Moo as being focused on teams aligning their releases with a fortnightly release train. … ‘Bug squashing Tuesday’ is set aside for people to tackle defects and improve complex or low quality areas, and people in the crews typically use XP practices such as pair programming, regular retrospectives and collective ownership. … Wrapping up, Mike stated that overall they aim to create a culture which empowers people to be proactive to solve problems.
Dream Job? The Vision and Journey to the Company Culture You Want
by Pete Burden & Helen Walton
Twitter feedback on this session included:
@SalFreudenberg: @Helenislovely #qconlondon I've seen companies ignore ideas because the person is too junior, too external or just from another department..
@portiatung: #qconlondon @Helenislovely @peteburden How systems shape our behaviours and the people we become http://t.co/xF5odzo8vi
@portiatung: #qconlondon @peteburden @Helenislovely "Advocacy 6 times more than Inquiry in organisations"
The Power of Hope: Getting You from Here to There
by Portia Tung
Twitter feedback on this session included:
@shanehastie: #qconlondon @portiatung hope ISN'T - unrealistic optimism, learned optimism, type A mindset, a measure of intelligence or previous achievement
@shanehastie: #qconlondon @portiatung Hope is "the sum of the willpower and waypower to achieve your goals"
@Helenislovely: Way power: mental capacity we call on to find more effective ways of reaching goals. Hope with @portiatung #qconlondon
@shanehastie: #qconlondon @portiatung useful goals need success criteria. Validate the clarity of the goal.
@Helenislovely: Validate your goals. Write in pairs to clarify. #qconlondon
@charleshumble: @portiatung real options. Never commit early unless you know why. #qconlondon
@johannescarlen: Aren't programmers the most hopeful people you know? - Portia Tung #qconlondon
@SalFreudenberg: @portiatung #qconlondon Improve hope by letting go of fear of failure. Whatever the outcome I will have learned.
Evolving Agile
Back to the Future: What Ever Happened to Being Extreme?
Ben Basson attended this session:
Some of the things I found really fascinating included:
- Developers decide what to work on next - they do research with the business and work out shared priorities, so that nobody spends time working on features that provide no business value.
- Mobbing - basically the same idea as paired programming but with more people involved, so a group sit around a large TV and observe and discuss while one person writes code - swapping around every 10 minutes.
- Building 20% learning time into the working velocity - to keep fresh ideas coming in and motivation high.
- Using a developer-on-support rota essentially as a human distraction shield, so the other developers can get on without interruption.
- It turns out that developers at Unruly only write code about 40% of the time, due to the 20% learning time and other responsibilities (research, monitoring, support, etc). As Rachel points out, this is fine.
Sebastian Bruckner attended this session:
Great talk from Rachel remembering of the core principles of Extreme Programming which sometimes might come too short in the today’s agile life. She also gave an interesting insight about how she and her teams were implementing it in the field. Among the known and well adapted aspects of XP she mentioned a method which was new to me.
Mobbing (Mob Programming):
Mobbing is similar to pair programming but with three to five persons instead. The code is on a big TV, one developer is actually programming the others are thinking and discussing. After a fixed time box (e.g. 20 minutes) another one grabs the keyboard, similar to pair programming. – They use mobbing to start difficult or complex stories.
Twitter feedback on this session included:
@Helenislovely: Microsoft windows XP was the death of extreme programming. Name no longer cool. @rachelcdavies #qconlondon
@shanehastie: #qconlondon @rachelcdavies one risk with a pure craftsmanship focus is losing the focus on building software for people
@paulacwalter: Continuous everything, no separate integration and testing phases, do all activities all of the time. @rachelcdavies #qconlondon
@douglastalbot: #qconlondon No point in researching features if you are simply going to do all of them! Just get building @rachelcdavies
@shanehastie: #qconlondon @rachelcdavies When the people who build the product also support it "they don't build dumb things that don't work"
@Helenislovely: Being able to keep learning keeps you fresh. Keeps you happy. This is a very @SparkConf practice from @rachelcdavies #qconlondon
@shanehastie: #qconlondon @rachelcdavies retrospectives: it's essential that teams get together and examine how they are working and adapt
@shanehastie: #qconlondon @rachelcdavies XP is about "if there is something that works, how can we do more of it" Turn the dial up. Experiment and learn
@dsommerville_nw: Truth around the world: developers are always downstairs - so put some "interruptable" devs upstairs [with the business] #qconlondon
@paulacwalter: focus on quick continuous feedback but don't ignore feedback that takes longer to arrive, like customer usage! @rachelcdavies #qconlondon
@shanehastie: #qconlondon @rachelcdavies XP lets us: Deliver value sustainably and build change tolerant systems. Also Mastery & Autonomy,
@metmajer: If you pair with the same person for a long time you really have to like them. @rachelcdavies on #PairProgramming at #qconlondon
@mattwynne: Team at @unrulyco only budget to spend 40% of time developing stories — interesting stat from @racheldavies at #qconlondon
@paulacwalter: Learning time for the team is just part of the service, factored in like holidays and meetings @rachelcdavies #qconlondon
Learning to Become Agile, with Retrospectives
by Ben Linders
Twitter feedback on this session included:
@shanehastie: #qconlondon @BenLinders Evaluate your retrospectives - make sure the team is getting value from the time spent
@shanehastie: #qconlondon @BenLinders in retrospectives the facilitator must be focused on allowing the team to make good decisions for themselves
@shanehastie: #qconlondon @BenLinders Coach role in retrospectives is to support team with the right questions and to support them in making changes
@shanehastie: #qconlondon @BenLinders Yes! the product owner is part of the team - they should be in the retrospective!
@shanehastie: #qconlondon @BenLinders Manager role in retrospectives is to support and empower the team to make changes
@shanehastie: #qconlondon @BenLinders some of the benefits that effective retrospectives can enable in your teams http://t.co/lEAEuNmdjT
@shanehastie: #qconlondon @BenLinders don't allow teams to overwhelm themselves with too many action items - limit the number of actions #retrospective
Progress from "What?" and "So What?" to "Now What?"
Twitter feedback on this session included:
@rvedotrc: Challenge people for rationale and to provide models used for decisions – @lmaccherone #qconlondon
@_yowan_: Every decision that you make is a forecast #qconlondon http://t.co/vzTDcXzYOB
@Helenislovely: Monte Carlo forecasting to build probability distribution. Improvement on #noestimates? @LMaccherone #qconlondon
@shanehastie: #qconlondon @LMaccherone use metrics correctly to change the nature of the forecasting conversation
@Helenislovely: Amazing how often people create stupid metrics because they forget the outcome they wanted them for in the 1st place #qconlondon
@Helenislovely: How to choose correct visualisation. Comparison, trend, forest AND trees @LMaccherone #qconlondon
@Helenislovely: Only 2% of the data collected gets used. Don't improve analytics, bake data use into product #qconlondon
@Helenislovely: Pattern-based decision makers. Issue is that we get the wrong pattern, often filtered by the cognitive bias @LMaccherone #qconlondon
@Helenislovely: Changing habits: direct the rider, motivate the elephant and shape the path #qconlondon
@Helenislovely: We make emotional decisions but we think it's rationally based on data. So true. @LMaccherone #qconlondon
@Helenislovely: Imperfect data may be better than no data. not sure imperfect models better. Think of bad decisions driven by bell curves #qconlondon
Taking Back Agile
by Tim Ottinger & Ruud Wijnands
Twitter feedback on this session included:
@Helenislovely: I recognise all these agile pains. But I think they stem from control cultures and that is hard to fix. @tottinge @RuudWijnands #qconlondon
@shanehastie: #qconlondon @tottinge @RuudWijnands No one else will make change for you - you have to make the change yourself
@shanehastie: #qconlondon @tottinge @RuudWijnands Original intent Programming More Intensely , but that was PMI and was already taken so it became XP
@Helenislovely: 'Death of hope' that resonates for me. Keeping hoping something will change can get in the way of action. #qconlondon
@shanehastie: #qconlondon @tottinge @RuudWijnands Becoming (acknowledge lack of knowledge and build it up) vs Seeming (cannot admit ignorance)
@twicezer0: Don't stockpile pain: root of agile. @tottinge @RuudWijnands #qconlondon
@shanehastie: #qconlondon @tottinge @RuudWijnands remember the beginning of the #AgileManifesto - "we are uncovering" - constant ongoing learning
@shanehastie: #qconlondon @tottinge @RuudWijnands The productive social context of getting things done, done, done! I want this agile back.
@Helenislovely: 'Getting it done, not holding hands and talking about feelings'. XP group hug! @tottinge @RuudWijnands #qconlondon
@Helenislovely: You don't need permission but companies can create blocks. @tottinge @RuudWijnands #qconlondon
@shanehastie: #qconlondon @tottinge @RuudWijnands as a leader give people enough trust to grow into.
@shanehastie: #qconlondon @tottinge @RuudWijnands Get rid of hope (someone else will fix it) and take responsibility to fix it yourself
@shanehastie: #qconlondon @tottinge @RuudWijnands Velocity is not a choice - it is a consequence. Every bug is a decision making flaw.
@shanehastie: #qconlondon @tottinge @RuudWijnands how fast you go today depends entirely on the quality of the code you work on.
@Helenislovely: Bugs are defects in thinking. Nice description. And bugs in culture are what bother me. @tottinge #qconlondon
@V_Formicola: How fast you develop a story depends on the condition in which codebase is…@tottinge #qconlondon
Why BDD Can Save Agile
by Matt Wynne
Ben Basson attended this session:
Matt gets quickly to the point, identifying the common problems faced by software development teams:
- Predictability - is the team delivering on time?
- Communication - are they working together well as a team (including all disciplines, i.e. testers, product owners, developers)?
- Quality - strongly related to the two above - it causes frustration for the team if there are lots of problems or defects.
He goes on to explain that it's possible to counteract these by addressing them directly:
- Small pieces (solve predictability by breaking things up properly).
- Collaboration (communicate and really work with each other)
- Technical discipline (TDD, refactoring)…
Explaining why TDD (Test Driven Development) is important, Matt says that automated tests are essentially warning lights, and whether you add them before or after writing code, you guard against the risk of regression later on when making changes. The crucial thing that this enables you to do is refactoring - which he says is a horrible technical term that means that product owners and customers don't necessarily think it is a necessary practice, when in fact they should be interested as it is a key part of maintaining the health of their software….
Matt concludes by saying that you can't just cheat on agile, you have to demand excellent communication, have excellent collaboration and have excellent code; this is where the agility comes from in agile.
Twitter feedback on this session included:
@shanehastie: #qconlondon #mattwynne How BDD can save Agile, Important point: Scrum != Agile http://t.co/TFtXRaZwVt
@shanehastie: #qconlondon @mattwynne Small pieces, collaboration and technical discipline are frequently missing in many "agile" implementations
@Hylke1982: #BDD / #ATDD helps us with delivering small pieces, collaboration and with technical discipline #qconlondon
@Hylke1982: BDD is a conversation between different roles to define and drive out specificatons in a structured understandable way. #qconlondon
@V_Formicola: “...test after is ok, but if you want to do it right you TDD” @mattwynne #qconlondon
@shanehastie: #qconlondon @mattwynne #BDD Conversations matter because ignorance is the bottleneck in software development
@rvedotrc: “Writing down the list of things you don’t know [business rules, examples, questions] is very helpful” - @mattwynne #qconlondon
@shanehastie: #qconlondon @mattwynne 3 Amigo's workshop - customer, developer, tester spend 20 mins to help understand the needs and express them usefully
@V_Formicola: “Analysing stories as a small group breeds empathy in a team and brings everyone at the same level of understanding” @mattwynne #qconlondon
@shanehastie: #qconlondon @mattwynne there is no excuse for not using a ubiquitous language - just be consistent!
@shanehastie: #qconlondon @mattwynn The bit of TDD that everyone forgets is Refactoring. Refactoring should be a constant activity!
@V_Formicola: Technical discipline….is what is missing in teams which are doing “half-agile” @mattwynne #qconlondon
@shanehastie: #qconlondon @mattwynn You can't stay agile without clean code! Refactor it, and that needs TDD as the warning lights about regression.
@merybere: You will fail unless you are listening to The tests #BDD #qconlondon
@Hylke1982: Product owners should/must demand refactoring to ensure agility #BDD #qconlondon
@shanehastie: #qconlondon @mattwynn To have real agility you need Excellent communication & excellent code
@paulacwalter: A code base without refactoring is like a dirty kitchen. You risk injury and poor hygiene when you trip up #qconlondon @mattwynne
@_yowan_: You can't cheat on Agile practices and expect things to work #qconlondon http://t.co/gz29NahE4W
@V_Formicola: You can’t cheat on Agile...it requires commitment..you need to have great communication...you need to have great code @mattwynne #qconlondon
@markhobson: Great reaffirming talk by @mattwynne. No agile w/o refactoring, no refactor w/o tests. #qconlondon cc/@BlackPepperLtd http://t.co/dqRqcPYgnn
HTML And JS Today
The Business of Front-end Development
Twitter feedback on this session included:
@DevOpsMD: Don't become an expert in one brand of hammer. Become a master carpenter. Develop timeless skills. --Rachel Andrew #qconlondon
@wonderb0lt: You get a lot of stuff for free if you're just doing it well. @rachelandrew #qconlondon
@V_Formicola: @rachelandrew “Progressive enhancement. Start with the core experience. We ship. We iterate.”. Sounds like Agile to me. :) #qconlondon
@nimpedrojo: You can't do everything.You can do something . @rachelandrew in #qconlondon
@DevOpsMD: We don't stop playing because we're old; we grow old because we stop playing -- @rachelandrew #qconlondon
@rajshahuk: How many people actually end up with the 'not invented here' problem and go of and create something new? via @rachelandrew #qconlondon
@rajshahuk: Flip side is that we are afraid to create and become more reliant on frameworks! I think this is more true. @rachelandrew #qconlondon
@dsommerville_nw: Also a huge fan of: Ship the core experience and *then* iterate [via progressive enhancement]; use tools lightly @rachelandrew #qconlondon
Internet of Things
When Arduino Meets Application Server: Love at Second Sight
Twitter feedback on this session included:
@deonvanaarde: IoT: Websphere Liberty app server running on @holly_cummins homemade ball on pcDuino over WiFi... Cool!! #qconlondon http://t.co/2JXbyoi6MO
@techiewatt: IoT track at #qconlondon with demonstration by @holly_cummins of a literally throwable websphere server with sensors! http://t.co/3OBxBmPnbl
@lauracowen: The world's first cuddly, throwable application server, with creator @holly_cummins. Running #WASLiberty #qconlondon http://t.co/FcDuyaASj8
Java - Not Dead Yet
Refactoring to Functional
by Hadi Hariri
motwin attended this session:
Hadi showed how some OO patterns can be turned into a functional style. T Use functions to pass behavior t was the motto and cooking recipe. Thus, it demonstrated how to rewrite a bunch of classes that implement the template patterns into one class. Again, thanks to the “use functions to pass behavior” principle. The Strategy pattern is also a good candidate for being rewritten in a functionalish style: the strategy has only to be encapsulated as function. Elegant code and less code: cool ! And as matter of fact, “Patterns of yesterday can become anti-patterns of today” (which is more or less a citation I don’t remember the author). Another use-case of Hadi is when the dependencies of a class grow. Firstly, it may mean the code smell bad. Secondly, it may mean that we have lots of dependencies just because we need to use dependencies behavior. And as in a functional style, you can “use functions to pass behavior”, … you get the trick, now. In functional programming, as a function can return a function, you can get a pipeline of functions call. That can also contribute to make the code shorted and more readable. Hadi just warned a too long pipeline can be in turn unreadable… so, be wise and encapsulate behaviors in meaningful named functions to avoid a too big chain of functions.
Scala in the Enterprise
Will Hamill attended this session:
Peter started with some simpler Scala examples of pattern matching, reducing boilerplate compared to Java code, collections operations and Futures for asynchronous method processing. HSBC were used as an example of a larger enterprise that has some Scala adoption along with GOV.UKwho use the Play Framework in some places. Peter said that Scala adoption depends on a confident and talented team, and delivering something working was the key to proving viability. …
Scala was then demonstrated for the same types of behaviours as the Java 8 examples. … Peter covered function composition, partial functions, tail recursion, functions returning other functions and standard map reduce type examples. Futures and Promises were also briefly covered, though I think that should be focused a little more given the power of this in Scala compared to Java.
Peter finished his talk by stating that while Java 8 is new, Scala is here already and can be used as a full-fat functional language as well as object-oriented. Java 8 however changes things by making functional paradigms accessible to a much wider and arguably slower-changing audience.
Modern CS in the Real World
Protocols - the Glue for Applications
Will Hamill attended this session:
Torben advocates Erlang for learning how objects should solve problems by communicating with each other, rather than ‘single page programming’ where people learn only to develop with an understanding only of the current class. Torben proclaimed the ‘golden trinity’ of Erlang: fail fast, failure handling, share nothing. Including failure handling as a specific case in your protocol means you should be able to handle failure gracefully. In Java world, failures are not tolerated and unexpected exceptions cause your process to die. In Erlang world, failures are embraced as alternate scenarios and managed.
Torben gave an example of a financial application for a simple stock exchange. Buyers post purchase intentions, sellers post sale intentions, deals happen when the seller price <= buyer price. In Erlang, this will be modeled using one buyer process and one seller process per sale interaction, communicating by sending messages that form the sale protocol. gproc, a process registry, would be used as a pub/sub mechanism for buyers and sellers listen for messages of intent to sell/buy. After price conditions are met, the sale is confirmed with a three-way handshake.
Failure is handled in the message protocol such that when the buyer or seller dies after the initial message of intent (determined by response timeout or monitoring the other process) then the processes can simply restart the interaction. If a party dies after the first part of the handshake, e.g. buyer dies before getting sale complete message after seller closed sale on their side, a restart of the process will the buyer back to the previous state. A supervisor process is commonly used in Erlang to monitor worker processes and handle restarts. Other options for handling failure in the stock exchange are to keep a transaction log per-process in order to easily replay until the last state. Alternatively a central ledger process could be used which tracks completed deals and allocates buyer and seller processes deal IDs so they can link back up when they fail.
Twitter feedback on this session included:
@jimshep: These 2 tools are all you need to build mission-critical systems @LeHoff #QConLondon http://t.co/fQFaA1Y76G
@dthume: Erlang fail fast error handling - "if you don't know what to do, what's the point in living?" @lehoff at #qconlondon
@solsila: If a process makes a call with incorrect data, it deserves to die @LeHoff on protocols #qconlondon
@willhamill: if you call the API with the wrong data, you deserve to die - @LeHoff on process error handling in Erlang #qconlondon
Product Mastery
Product Ownership Is a Team Sport
by Shane Hastie
Twitter feedback on this session included:
@NitinBharti: Product management is a "value management" *team* sport #qconlondon @shanehastie http://t.co/orWdN63iXR
@solsila: Velocity is a measure of work (cost), not value. @shanehastie #qconlondon
@lissijean: Building more is not always better. Recognize when the value flattens out. @shanehastie #qconlondon http://t.co/2xHhcnOZeh
Product thru the Looking Glass
by Chris Matts
Twitter feedback on this session included:
@shanehastie: #qconlondon @PapaChrisMatts The agile test: Deliver quality: Deliver Quality (bugs, pert UX) : Short Iterations (max 1 month)
@lissijean: We don't want a tea bag, we want the value we get from a cup of tea - quenching thirst. #prodmgmt #qconlondon @PapaChrisMatts
@shanehastie: #qconlondon @PapaChrisMatts Product management is the realm of hypothesis - we think that the need exists and meeting them matters
@lissijean: After you've gathered insights you need to manage them. Analyze by value and personas. @PapaChrisMatts #qconlondon
@lissijean: Testing hypotheses is a #Kanban process not scrum. Don't try to shove it into a sprint. @PapaChrisMatts #qconlondon http://t.co/XVLqrwPn5D
@shanehastie: #qconlondon @PapaChrisMatts getting the audience to build UI designs and vote on them in the session. Bad idea terminator works @lissijean
The Bad Idea Terminator
Yan Cui attended this session:
We often start off doing things right – we test and iterate on our ideas before we hit the market, and then we end up with something that people want to use. But then we just keep on building without going back to finding those innovative ideas that people love. …
We can fall into the build trap in a number of ways, including:
- pressure from stakeholders to always release new features
- arbitrary deadlines and failure to respond to change – setting deadlines that are too far out and not being flexible enough to adapt to change
- “building is working” mentality – which doesn’t allow time for us to step back and think if we’re building the right things…
So how do you become the Bad Idea Terminator, i.e. the person that goes and destroys all the bad ideas so we can focus on the good ones? We can start by identifying some common mistakes we make.
Mistake 1 : don’t recognize bias…
Mistake 2 : solutions with no problems - When people suggest new ideas, most of the time they come to the table with solutions. Instead, we need to start with the WHY, and focus on the problem that we’re trying to solve….
Mistake 3 : building without testing - When we get stuck in the build trap we don’t tend to test our assumption, as we tend to commit to one solution too early. Instead, we should solicit many solutions at first, and get people off the fixation on the one idea….
Mistake 4 : no success metrics - Another common mistake is to not set success metrics when we go and do experiments, and we also don’t set success metrics when building new features.
Twitter feedback on this session included:
@paulacwalter: Figuring out what to build is the hard part. Don't get stuck in the "building is working" trap. @lissijean #qconlondon
@shanehastie: #qconlondon @lissijean in software we get stuck in the build trap - just build the next piece, stop and go back to check that still want it
@shanehastie: #qconlondon @lissijean Putting out more features doesn't make your product more attractive, just bloated
@shanehastie: #qconlondon @lissijean The most important part of the product manager role is the ability to say No
@_yowan_: Feature ideas are not your babies inspiring talk by @lissijean #qconlondon http://t.co/Yjemn9rHji
@shanehastie: #qconlondon @lissijean Idea terminator: notice differences among customers and businesses
@quaasteniet: Now The bad idea terminator where @lissijean talks about killing bad ideas even more useful than killing bad code ;) #qconlondon #TOPdesk
@shanehastie: #qconlondon @lissijean Bad idea terminator: Change perspective
@Helenislovely: Building in questions that help evaluate features and therefore truly prioritise. Dealing with ideas @lissijean #qconlondon
@shanehastie: #qconlondon @lissijean Bad idea terminator: Focus on the problem
@shanehastie: #qconlondon @lissijean Bad idea terminator: Is this a problem we can and want to solve?
@shanehastie: #qconlondon @lissijean Bad idea terminator: Consider many solutions
@shanehastie: #qconlondon @lissijean Bad idea terminator: Test the viable solutions very quickly for very low investment
@shanehastie: #qconlondon @lissijean Bad idea terminator: set success metrics when you identify the experiment, then CHECK THE RESULTS
@shanehastie: #qconlondon @lissijean Bad idea terminator: Set goals early (late goals will be adjusted to our bias) and make sure they align with KPIs
@Helenislovely: Thinking of 2 other biases that kill our innovation: loss aversion and attachment to our own creativity. @lissijean #qconlondon
@solsila: The faster you kill the bad ideas, the more time you have for the good ones. Becoming a Bad Ideas Terminator w/ @lissijean #qconlondon
The Sensemaker Method
by Tony Quinlan
Twitter feedback on this session included:
@shanehastie: #qconlondon @tquinlan direct questions never give you the truth - need to get underneath to get meaning
@shanehastie: #qconlondon @tquinlan is your solution actually creating the problem?
@shanehastie: #qconlondon @tquinlan "on average" doesn't help, what you want is specifics to understand actual needs
@shanehastie: #qconlondon @tquinlan collecting stories as people experience the system/environment allows meaning to be exposed
@shanehastie: #qconlondon @tquinlan context and moment matters - people are complex, can't simply extrapolate
@Helenislovely: Meaning is not in the content you're reading. Its the context and intertextuality. @tquinlan #qconlondon
@shanehastie: #qconlondon @tquinlan "red shirt" is an assessment of life expectancy not a fashion statement (#startrek) #context matters
@Helenislovely: Gather smart feedback @tquinlan connects to @LMaccherone point about using data to change the tone of conversation. #qconlondon
@shanehastie: #qconlondon @tquinlan collect stories with the audience's meaning; look for patterns that emerge; come up with activities to improve soltn
Reactive Architecture
Responding Rapidly When You Have 100GB+ Data Sets in Java
by Peter Lawrey
Will Hamill attended this session:
Peter described how he believes that a modern system should be reactive: responsive, resilient and elastic. When your weapon of choice is the JVM, you can process data much faster when you can map your entire data set into memory (given I/O bottlenecks, I’m sure this is true for almost any language). However, what happens when you move into the realms of very large data sets - which in Java land is pretty much anything beyond 32GB?…
In terms of accessing more memory on the JVM, going beyond 32GB on standard compute platforms means you’ll need to jump up to 64-bit address references, which though increasing available memory area also reduces the efficiency of CPU caches due to increased object size. Garbage collection of such larger memory spaces also starts to become a problem, with a concurrent collector being needed to avoid stop-the-world execution pauses.
Peter described how the Azul Zing concurrent collector was an option for tackling this issue up to a given size, as for memory sets of around ~100s GBs their garbage collector will perform with minimal execution impact. A different approach would be to use TerraCotta BigMemory as a memory management layer inside your application, allowing the application to use off-heap memory, though the disadvantage is needing to explicitly build applications against their library so it can’t be injected in to existing applications as a mitigation in the same way using Azul Zing could be.
When addressing bigger data sets of up to 1TB in Java, the NUMA region limit kicks in at around 40 bits of physical memory (40 bits for Ivy Bridge and Sandy Bridge generation Intel CPUs, 48 bits for Haswell generation CPUs). Addressing beyond 40 bits requires using a 48-bit virtual address space, with data paged in on demand. The 48-bit limit then pushes the threshold to 256TB in CentOS, 192 TB in Windows and 128TB in Ubuntu. I can’t wait for someone to be quoted at this point saying “128TB will be enough for anyone!” that we can all look back upon and laugh in 2025 :). Moving further up the orders of magnitude, a 1PB (Petabyte!) memory space can be achieved by mapping the address tables themselves into the main addressable space, in order to achieve paging of the virtual space.
Twitter feedback on this session included:
@peter_pilgrim: Reactive system design to consider if your data size >32GB from @PeterLawrey #qconlondon #in #java #performance http://t.co/2VJvkKs903
@charleshumble: “32 bit i still relevant in IoT. If you have a digital toaster it's unlikely to need more than 32 bits." @PeterLawrey #qconlondon
@Idris_Ahmed251: x86 computers... Your toaster will never need that! #qconlondon @peterlawrey
@charleshumble: Many systems have NUMA regions that are limited to 1TB. @PeterLawrey #qconlondon
@charleshumble: PetaByte JVMs for a use case that needs random access without going across networks @PeterLawrey #qconlondon http://t.co/9PyGDF8ilC
The Art of Protocol Design
Twitter feedback on this session included:
@miguel_f: Don't break user code! Once again: don't break user code! @hintjens #qconlondon
@marcusolsson: Versioning is not an excuse to break contracts. @hintjens #qconlondon
Taming Microservices
Microservices Are Too (Conceptually) Big
by Philip Wills
Pere Villega attended this session:
In this talk, Philip Wills, Senior Software Architect at The Guardian, explains how The Guardian moved from a monolith to Microservices. Currently they release around 40 different services to production each week.
When they were a monolith they started to find some problems. … They also had a concern with limiting the scope of failure. The monolith allowed embedding some microapps into specific places, but they hit performance and coupling issues soon enough. …
So they moved to single responsibility apps, focused on resilience and with limited scope (you can fit it in your head)….
An important point he raised is that they try to avoid sharing libraries across services, due to the amount of contention they cause. They consider this a last resort.
Will Hamill attended this session:
Phil explained that the main reason The Guardian had pursued a microservices architecture was for faster innovation in a high pressure media marketplace. Independent teams were needed for the different functional areas of the platform, and as on the old platform taking something experimental from a Hackday and putting it into production was prohibitively expensive, the organisation wanted the teams to be able to work independently from each other and release rapidly without overhead….
With the ‘micro app’ approach, teams at The Guardian were developing independent products with ownership inside the team. This made removing and replacing parts much easier than the pain caused when they tried to do this with the monolith application. Teams were no longer dependent upon each other for releases and making changes to their interfaces backwards compatible reduced interdependencies. Phil mentioned how it was important for each team to own their own datastore and to prohibit integration via DB so as to maintain the benefit of cleanly defined APIs.
Each application had a single responsibility, and a single key metric that would tell teams about its performance, in terms of that responsibility. Different applications change at different rates, so we want them not to depend on each other, as this would restrict them to the lowest common denominator.
Twitter feedback on this session included:
@stefanoric: #qconlondon Some lessons from The Guardian about microservices: separate teams, limit scope of failures, design for things to die.
@danielbryantuk: Fail-fast for the win in a monolith and microservice integration @philwills at #qconlondon
@danielbryantuk: JSON is a really poor interface mechanism. We're looking at Thrift to provide strong-typing for protocols @philwills at #qconlondon
@stefanoric: #qconlondon The Guardian is considering ditching JSON and go with Thrift
@rvedotrc: “On AWS, you can throw more hardware at it, or turn it off and on again. This solves a surprisingly large amount.” @philwills #qconlondon
@danielbryantuk: Interesting to see the Guardian focusing on 'single responsibility apps'. Find one key metric for each app that measures impact #qconlondon
@adrianmouat: Everything was crashing, but nothing was causing a problem @philwills on the advantages of microservices at #qconlondon
@rajshahuk: Yeah! The Guardian try to avoid shared libraries via @philwills #qconlondon
Microservices: Software That Fits in Your Head
by Dan North
Will Hamill attended this session:
Dan has argued for some time that software itself is not an asset but a liability, so producing more code is less valuable than making the software more effective. The costs (as well documented across the industry) cover not just creating the code, but understanding and maintaining it on an ongoing basis. The biggest problem to software development is the code in the system that nobody knows about, as this is expensive and risky to maintain. The best way to deal with this is to stabilise the offending code or kill it off.
Dan describes two complementary patterns for understandable, maintainable code: ‘short software half-life’ and ‘fits in my head’. …
Short half-life results from the replaceability of discrete components with clear boundaries and defined purposes and responsibilities. …
‘Fits in my head’, a metric originally inspired by the length of a class on screen compared to James Lewis’ head but not generally referring to the ability of a person to understand the conceptual whole of a component at a given level of abstraction, is used to judge whether or not other people on the team can reason about the component with the same context as whoever designed or first implemented it. This is useful as the contextual consistency for the person understanding the component is more important than homogeneity across implementation methods, so that freedom is left to meet the needs inside the component in whichever way is best, but the people making decisions about it will come to similar decisions as the designer/implementer would have. When you have this contextual consistency you can trust that the decisions people make that result in different outcomes have been driven by different needs rather than arbitrarily.
Dan described the approach of splitting these components into services that together fit the business need as a ‘replaceable components’ style (no doubt familiar to anyone who understands the original intent of SOA). To reduce coupling these components should be isolated from each other and should communicate by passing messages through well-defined APIs. Implicit coupling between components can be identified by heavy use of mocks - over-dependence on mocking implies you are too tightly coupled to the behaviours of other components.
Pere Villega attended this session:
We don't care about code, but about the business impact. That is to mean that the code is not an asset, is a cost we assume to obtain the business impact. Writing code, changing code, and understanding code; they are all costs.
As a consequence we want to stabilise our code or, alternatively, kill it fast and replace it by less-costly code. As it happens, the patterns that facilitate this process lead to Microservices.
The first pattern is to have a short software half-life. An application can be long lived, but the code that composes that application may not be. Effective teams tend to have a very short code half-life, in which after a few weeks the code has changed a lot and whole sections have been replaced or moved. This keeps the costs associated to code low.
To facilitate a short half-life we want to write discrete components, with clear boundaries, and clear purpose and responsibility. The boundaries happen at many levels: deployment (containers), design (DDD), etc. A clear purpose reduces uncertainty….
Another pattern is to consider anything that doesn't fit in your head as too big. …
A Microservice can be a type of replaceable component architecture, if you optimise for replaceability and consistency. Don't optimise for size, smaller is not necessarily better, more replaceable is better. And kill code fearlessly.
Twitter feedback on this session included:
@rvedotrc: The goal of software development is NOT to produce software – it’s to sustainably deliver positive business impact @tastapod #qconlondon
@V_Formicola: What is our business about? “Sustainably minimise lead time to business impact.” @tastapod #qconlondon
@trisha_gee: Code is not the asset, code is the cost @tastapod at #qconlondon http://t.co/gqtnarm5xv
@jimshep: productive != effective- @tastapod #QConLondon http://t.co/LGSmbZ3mok
@rvedotrc: Writing code is the annoying time-expensive part that gets in the way of solving problems (paraphrasing @tastapod #qconlondon)
@stefanoric: Code should be stabilized or killed off Dan North #qconlondon
@solsila: Heisenberg effect in code: the issue occurs until you try to observe it! #thisexplainsalot @tastapod #qconlondon
@trisha_gee: An application can have a long life, but the code should have a short half life @tastapod at #qconlondon
@rvedotrc: “If I can’t reason about [a component], I can’t kill it” @tastapod #qconlondon
@paulacwalter: Documentation is valuable. Documenting everything a shocking waste of time. Tricky bit is knowing what to document. @tastapod #qconlondon
@rajshahuk: Killing code is refactoring, it isn't brutal -- @tastapod #qconlondon
@V_Formicola: “I don’t like to look at code that doesn't fit in my head” @tastapod @boicy #qconlondon
@V_Formicola: Familiarity is different than simplicity@tastapod #qconlondon
@AlibertiLuca: #qconlondon Mocking is an anti-pattern @tastapod
@axhixh: I am going to write best code I can that I don't care about. @tastapod #qconlondon
No Free Lunch, Indeed: Three Years of Microservices at Soundcloud
by Phil Calcado
Pere Villega attended this session:
SoundCloud moved from a Sacrificial Architecture to Microservices. … Before you can start with Microservices, you need 3 things:
- rapid provisioning of servers (or containers or vm's)
- basic monitoring
- rapid app deployment with a short turnaround
For provisioning, SoundCloud moved from AWS to their own datacenter in Amsterdam, although they still use S3 and some other Amazon services….
On telemetry they found a similar issue: the tools available at 2011 weren't great. The common tooling was a push model based on StatsD, Graphite and Nagios. Engineers they hired at that time wanted a pull model so they developed Prometheus which works along Icinga to provide better data. When they moved to Microservices their monitoring didn't break. …
Regarding their pipeline at the beginning they had 2 different pipelines: one for build, one for release. Customisation and other factors ended up creating 7 different deployment scripts. Currently they use Docker to run tests when in development. Jenkins takes care of building and packaging the application as a deb package, and they use that for deployment. They have not adopted Docker yet as they don't want a strong coupling.
Will Hamill attended this session:
The three aspects of this that Phil found most important at Soundcloud were rapid environment provisioning, basic monitoring, and rapid application deployment. The first of these, provisioning, looked a little different in 2011 than it did now when Soundcloud were preparing for what they thought would be the “microservices explosion”. With Heroku as the example, usingDoozer for service discovery, LxC containers and the 12 Factor principles, Soundcloud managed to put together a provisioning platform much better than any other complete solution around at the time….
In terms of telemetry and monitoring (another hot topic at QCon this year), Phil described how in 2011 the tooling available was not quite what it is today, and dissatisfaction with some common tools led Soundcloud to build their own system. Interestingly, some of the former Google Site Reliability Engineers that had been hired by Soundcloud advocated this as they missed the detailed monitoring when moving away from the Google platform. Moving from a solution of Statsd, Graphite and Nagios, Soundcloud developed and subsequently open-sourced Prometheus as their metrics & monitoring system, with use of Icinga and PagerDuty for alerting.
Teams at Soundcloud were also reorganised - component teams became feature teams with more vertical responsibility (surprisingly no explicit call out for Conway’s Law).
The second of the three aspects Phil talked about was deployment. Soundcloud moved from a two-pronged delivery pipeline using make and Jenkins where Jenkins ran many sets of tests but did not build the artefacts which would actually be deployed, to a singular pipeline of Docker containerisation for unit/integration tests before commit to source, then Jenkins running a wider set of tests on the latest code, generating packages either for including in an AMI for AWS deployment, or for containerisation to allow developers to run a ‘mini-soundcloud’ for dev/testing purposes.
Twitter feedback on this session included:
@danielbryantuk: People are always migrating to something from Rails @pcalcado on the current #microservice migration trends at #qconlondon
@philwills: .@pcalcado says at @SoundCloudDev their engineers with a javascript background write scala, so not just us at @gdndevelopers #qconlondon
@rvedotrc: “When you have a monolith, and it breaks overnight, you know where the problem is: it’s in the monolith” @pcalcado #qconlondon
@markhneedham: We had Scala people getting very confused about JavaScript... #qconlondon @pcalcado
@tastapod: “Don’t forget companies that open source cloud, monitoring, lservices, etc. got it wrong a bunch of times first!" @pcalcado at #qconlondon
Operating Microservices
Pere Villega attended this session:
His talk applied to Microservices understood as 'vertically aligned stacks that communicate via simple and standard interfaces'. In this context ownership of data matters, you don't want to share data. Teams own the code, people outside the team don't modify that code. Teams also own the runtime, they choose what they want to use, even Erlang in a supposedly JVM shop. …
Microservices are [using] Conway's Law. Management and developers love them because:
- small owned services that can be updated more often
- teams can move fast and break stuff
- teams can own the whole stack
On the other hand infrastructure teams hate them because:
- small owned services that can be updated more often. But changing things breaks things.
- teams can move fast and break stuff. Ops want to be slow and stable, avoid breaking things
- teams can own the whole stack. Ops are not required, thus they lose control over security …
The best starting point it to just run 1 Microservice, by itself, successfully. Microservices in the small. You need to:
- make sure you are able to do it
- automate your infrastructure (use of containers is not necessary)
- create a base image for your service
- deploy it fast, with a time-to-server of a few minutes, not hours …
You need monitoring tools that are easy to hook into. …
You must automate your deployment. It doesn't need to be complex, it can be as simple as executing a self-contained Jetty, but keep a log of the deploys done including the commits that were released. And you must give the keys to the development team, as well as root. And give them pagers too: if they break it, they fix it. But to do that they need full access….
At this point point you have 1 Microservice running. Now repeated to get 10, 100, ... Microservices in the large.
When you have many services running, they are going to fail in more spectacular ways than their equivalent monoliths. They are not complicated (like a car), but complex (like a traffic system) in which the root cause may be distant in time and space. Due to that, diagnosis tools are a must.
Twitter feedback on this session included:
@danielbryantuk: I've never known an ops team who like the phrase 'move fast and break things' @bruntonspall on operational microservices at #qconlondon
@rvedotrc: Ops teams hate microservices because you can change stuff, and change it without them. @bruntonspall #qconlondon
@jabley: lservices as a way of subverting Conway's Law and redesigning your organisation – interesting idea from @bruntonspall #qconlondon
@danielbryantuk: Simple, complicated, complex <- all microservice architectures are complex! @bruntonspall on complexity at #qconlondon
@markhneedham: We're building the new legacy services #qconlondon @bruntonspall http://t.co/piNTvakhmc
@kronk2002de: Operations teams should act more like consultants @bruntonspall #qconlondon
@V_Formicola: The first person pays the price, the next will pave the road, the following will have a road to walk on. @bruntonspall #qconlondon
@danielbryantuk: Look at the 90th, 95th and 99th percentile when dealing with microservice telemetry <- this stuff matters at scale #qconlondon
Protocols of Interaction: Best Current Practices
Will Hamill attended this session:
Todd began by describing some of the issues of service communication that can be observed and sometimes addressed in protocols. Data loss, duplication and reordering as the most common issues tackled and can happen even in protocols portrayed as ‘reliable’. In TCP, for example, connections can be closed or traffic interfered with by proxies.
In request/response synchronous comms, throughput is limited by the round trip time for each communication multiplied by the number of requests. Asynchronous communication can reduce this to closer to the round trip time for a single communication but responses require correlation to original requests and this therefore adds complexity. Ordering of messages is an illusory guarantee, as the compiler, runtime environment and CPU can change ordering in real terms, so ordering is imposed upon events by the protocol….
In scenarios when multiple recipients are sent the same data set, if each recipient requests a retransmit of a different part of the data set, it will cause a retransmit of the entire set even if between them the entire set was accepted - a common problem when distributing loads across horizontally scaled consumers. Solving this is non-trivial, and Todd recommends a combination of patience, and waiting to listen for possible retransmit requests from all recipients. …
For queue management, Todd recommends using a bound total queue length and using back pressure or even dropping messages to ensure minimisation of the queue contents. Extended queue lengths leads to ‘buffer bloat’ and delays between services (large queues between work states being a cause of delays should be known to anyone familiar with queuing or ToC).
Todd concluded by summarising that existing protocols such as TCP, Aeron and SRM, are full of patterns for tackling complicated communications problems, and that we should look to how others have solved these problems when working on our own systems.
Pere Villega attended this session:
Protocol definition matters. Protocols not only define how we format and treat data, they also define how do we interact with something. Protocols of interaction matter much more now that we are embracing Microservices, and they are a great solution for many of the problems raised by them. Internet (in fact, any network) is an hostile environment were data can be lost, duplicated, reordered, etc. And TCP is not a safeguard from those issues. They will happen.
The main takeaway of the talk: Protocols are a rich source of solutions to complicated problems.
Taming Mobile
How We Build Rock-solid Apps and Keep 100M+ Users Happy at Shazam
by Savvas Dalkitsis & Iordanis Giannakakis
Twitter feedback on this session included:
@IsraKaos: Test first, we found is actually easier @iordanis_g @geeky_android Now even in Android!! #qconlondon
@RogerSuffling: #Shazam and how to do rock solid builds. Glad to see testing is central #qconlondon http://t.co/AsDwh2LbEt
@trisha_gee: I actually found the code version of the test easier to understand than the user story! @geeky_android at #qconlondon http://t.co/VkztZDl3aT
@trisha_gee: Unit tests that take 3 or 4 seconds to run are not acceptable @geeky_android at #qconlondon
@trisha_gee: Cognitive load is lower when you don't use a DI framework in Android - @iordanis_g at #qconlondon
@trisha_gee: The power of manual testing comes in when you're trying to do things like check your animations are smooth @iordanis_g at #qconlondon
Tales from Making Mobile Games
Twitter feedback on this session included:
@IsraKaos: With 20 internal teams (2-32) if one fail it's just one. But when 1 succeeds, all 20 can benefit from it @jrirei #qconlondon
@trisha_gee: Feature switching useful not just for A/B testing to see what's effective, but also to turn off features with bugs @jrirei at #qconlondon
@IsraKaos: While we waited for the iOS version to be approved, the Android guys released 8 times in their normal cycle Hehehehehe @jrirei #qconlondon
@JanSabbe: I love Apple, but... I'm starting to love Android even more @jrirei #qconlondon
@ellispritchard: Apple loosing love to Android due to lack of staged roll-out/slow iteration #qconlondon
The Go Language
Infrastructure Built in Go
Will Hamill attended this session:
Jessie began with giving an overview of what Docker is and what it’s for: a runtime for application containers, which are a subset of Linux kernel features such as namespaces, cgroups and pivots. Docker allows you to ‘containerise’ your application and produce a fully static binary containing all dependencies, giving ease of installation and deployment. This can be as basic as scp the container to the target server and bootstrap the binary. Jessie also informed us that Docker support would be coming to Windows, which would bring the lineup to four main platforms (the others being Darwin, BSD Unix and Linux).
Jessie also described a distributed message platform in Go called NSQ which is used to help scale the Docker project in a number of ways. NSQ is used by the build app responsible for listening to GitHub hooks to trigger builds and deployments, used by the Docker master binary build to run on every push to the master branch, and used by the app which automates building and publishing docs from the code. The team at Docker rewrote a Python-based Jenkins plugin in Go to handle the pull requests, which also uses NSQ to perform housekeeping such as checking for signed commits, labels and documentation comments.
The Go language was chosen by the Docker team for a number of reasons: it’s simple, has common useful tools such asfmt, vet, lint, and others for documentation and tests. Some of the issues they found when using Go were in package versioning across the teams and inconsistency in approaches to this. The Go test framework is also still basic and not as fully-featured as those in other languages (like JUnit for example) and so lacks some useful aspects such as setup and teardown step definitions. The Go community is also smaller than that of other languages and as a result is still helpful, friendly and comparatively drama-free.
Twitter feedback on this session included:
@paulacwalter: A container is a magical thing that doesn't really exist in the wild. @frazelledazzell #qconlondon
@csanchez: Go is a neutral language, not like Java. It's the Switzerland of languages @frazelledazzell #qconlondon http://t.co/cD2q3SHsnf
Sponsored Solutions Track
Microservices, Micro Deployments and Devops
Sebastian Bruckner attended this session:
An advanced micro service talk with the focus on creating resilient, changeable services and how to operate them. Especially his anti- or problem patterns in a micro service architecture were very interesting.
- The Gatekeeper – Many outgoing dependencies
- likely not very “micro”
- complex application logic
- very deployment sensitive
- Single Point of Failure – Many incoming dependencies
- central application service
- very fragile to scaling
- user facing impact
- Hub – Many incoming and outgoing dependencies
- the worst thing to happen
- “legacy” migration
- highest deployment risk
Opinions about QCon
Sebastian Bruckner attended the microservices open space:
Every track on the QCon had a slot with an open space. An open space doesn’t have a prepared talk but several slots where the attendees can promote topics they are interested in and want to talk about. The microservice open space was the first and only open space I attended at QCon. The topics were surprisingly advanced, I really enjoyed it. I can only recommend you to try this format.
Impressions expressed on Twitter included:
@stealthness: Day 1 of #QConlondon 2015 and the QEII Conference center had make over since last. Looking very sleek and modern. http://t.co/WMRCpGNvpp
@QuackingPlums: #qconlondon is worth it every year just for this view… http://t.co/dDD5CPj51O
@V_Formicola: Positively impressed by the % of women at #qconlondon, the most I have ever seen at a technology conference! Are things finally changing?
@lauracowen: #qconlondon do some very good food. Nom http://t.co/CK8CSHdxJf
Takeaways
Ben Basson’s takeaways were:
It seems like a lot of teams are struggling to properly implement agile practices, and I'm glad to see that I'm not the only one who has experienced some of these problems over the last few years. I got a lot of food for thought, but what I found great about QCon was the focus not only on ideas, but how to sell them and the real business benefits behind these improvements in working practices.
Takeaways from QCon London included:
@fotuzlab: Building software is not like building a house, it’s like town planning #qconlondon
@_angelos: Observing #qconlondon, I get the feeling much of software engineering is about automating what exists, not inventing what isn’t there.
@Helenislovely: This whole conference is like an index of things I don't know but should. Good coz I like learning but also overwhelming! #qconlondon
@Yann_G: How to keep running the romance w. Agile in a team? Fresh air comes from the outside! Send teams to conferences! #qconlondon
@solsila: Main #qconlondon themes this year? Culture, Microservices, DevOps, Docker... and cats.
@rolios: Firebase, microservices at The guardian and at Soundcloud, protocols, rxjava on Android, Aeron. Last day at #qconlondon was huge. The end!
@bencochez: Almost home after a week at #qconlondon. A truly inspiring experience.
@portiatung: @rkasper #qconlondon Thanks for the great facilitation of the open spaces!
Conclusion
The ninth annual QCon London brought together over 1,100 attendees - including more than 100 speakers – who are disseminating innovation in software development projects across the enterprise. QCon's focus on practitioner-driven content is reflected in the fact that the program committee that selects the talks and speakers is itself comprised of technical practitioners from the software development community.
As well as being notable for its size this QCon represented another milestone. InfoQ.com & Trifork have brought you QCon London in partnership since 2005. Now InfoQ is acquiring Trifork's interest in QCon London. Trifork runs GOTO conference in Chicago, Amsterdam, Copenhagen, Berlin, as well as Scala Days & FlowCon. Going forward, London will host two separately run conferences, reflecting the two companies’ different visions for how to run developer conferences:
- QCon London: 3 days conf, 2 days tutorial - March 7-11 2016 - 1,200 attendees. Run by InfoQ.com who runs QCon San Francisco, New York, Sao Paolo, Rio, Beijing, Shanghai, Tokyo.
- GOTO London: 2 days conf, 2 days tutorial - September 14-17 2015 - 450 attendees. Run by Trifork, who runs GOTO Aarhus, Amsterdam, Chicago, Berlin, Copenhagen, FlowCon, Scala days.
The two companies remain friends and Trifork maintains an equity stake in InfoQ's operating company C4Media Inc. The two conferences will cooperate and co-promote each other for the next few years.
The next English language QCon is New York starting on June 8th, followed by San Francisco on November 16th. QCon London will return on 7-11 March 2016.