Close to 1,000 attendees descended on the Hyatt Embarcadero for the seventh annual QCon San Francisco for one simple reason: to see what makes the ‘big boys’ tick. Data Scientists, Lead Architects, and Process Gurus from companies like Netflix, Facebook, Google, and Twitter shared their successes, failures, and most importantly, lessons learned from building, managing, and growing their large scale systems.
QConSF attendees - Software engineers, Architects, and Project managers from a wide range of industries as well as some prominent Bay-area companies – heard over 100 practitioner speakers present over 100 technical sessions and 19 in-depth tutorials over the course of five days. They also heard keynotes from industry giants like Rich Hickey, Brendan Eich, and Jeff Hammerbacher.
This article summarizes the key takeaways and highlights from QCon San Francisco 2013 as blogged and tweeted by attendees. Over the course of the next 4 months, InfoQ will be publishing most of the conference sessions online, including 19 video interviews that were recorded by the InfoQ editorial team. The publishing schedule can be found on the QCon San Francisco web site.
You can also see numerous QCon photos here.
Resources: @SeenFeed: Highlights of #qconsf based on 2352 tweets and 26 instagrams: http://t.co/FFuCJPdaPb
@SeenFeed: Top users from #qconsf @jezhumble, @qconsf, @dtunkelang, @ddoomen, @xamat: http://t.co/FFuCJPdaPb
- Tutorials
- Keynotes
- Applied Machine Learning and Data Science
- Architecting for the Cloud
- Architectures You've Always Wondered About
- Continuous Delivery
- Online Controlled Experiments: Introduction, Insights, Scaling, and Humbling Statistics by Ronny Kohavi
- From Code to Monkeys: Continuous Delivery at Netflix by Dianne Marsh
- Growing from the Few to the Many: Scaling the Operations Organization at Facebook by Pedro Canahuati
- Test Automation at Google Scale by Jason Leyba
- Demystifying the API Lifecycle
- Engineering Culture - How to Build, Organize and Fuel the Best Engineering Team Possible
- Building a Culture Where Software Projects Get Done by Greg Brockman
- How GitHub (no longer) Works by Zach Holman
- Scaling Engineering Culture at Twitter by Raffi Krikorian
- Evolving culture and values. Understanding the tradeoffs. Growth through failure. The importance of leadership and open communication. by Pedram Keyani
- Hadoop : Beyond Map-Reduce
- Making Virtual and Remote Teams Shine
- Next Generation HTML5 and JavaScript
- (Post)Functional
- Scalability, Availability, and Performance: Putting It All Together
- Taming Mobile
- QCon Open Space Events
- Opinions about QCon
- Takeaways
- Conclusion
Domain Driven Overview by Eric Evans
Twitter feedback on this tutorial included:
@dmysticd I have a new found respect for the complexity of container shipping and airline travel @ericevans0 #ddd #qconsf
@jamshally Modeling is not a virtuous activity. It helps you build better software, more quickly and with fewer bugs -@ericevans0 #qconsf
@jamshally #NOSQL& #DDD a good match. One document per aggregate. @martinfowler article: http://martinfowler.com/bliki/AggregateOrientedDatabase.html ... - @ericevans0 at #qconsf
@jamshally The constant inference of context is a terrible way to design software. Enter the "Bounded Context" - @ericevans0 #qconsf
@dmysticd "Without boundaries you can't make statements that are simple and true... and your software gets complex" @ericevans0 #qconsf
@dmysticd pudding or pie? when integrating software systems there are no 'pleasant' surprises @ericevans0 #dddesign #qconsf
Secrets of Infrastructure as Code - Day 1 of 2 by Jez Humble , Tim Brown
Twitter feedback on this tutorial included:
@wadcom: The job of the change management process is to make sure that nothing ever changes @jezhumble #qconsf
The Culture Engine: Exploring the Social Side of Software Development to Accelerate Team Performance by Amr Elssamadisy , Steve Peha
Dennis Doomen attended this tutorial:
The goal of this workshop was to learn how to build high-performance teams ... what I liked about this workshop is that those guys provided a model that helps us understand the behavior exposed by those very same cultural differences, as well as tools to overcome them.
So what are those impediments? The first one they discussed is the lack of safety. How likely am I going to participate in solving a problem if I don't feel safe enough to speak my mind or take changes, whether it involves a colleague or a supervisor? Of the many tools Amr and Steve shares with us, I liked leveraging your vulnerability. For instance, share your origin story. Where did you come from? What are your goals in life? What choices did you make? Sharing information like that breaks down hierarchies, improves respect and levels the playing field. And if that all feels too soft, consider the foreverness of your current situation. Do you really want to be in this situation forever? Considering that might fuel your sense of safety just enough to do something about it.
The second impediment is lack of respect. How likely am I to participate in solving a problem if you think I'm an idiot? ... An essential mindset when dealing with people is to consider that it is not about how you talk with somebody, but how you see that person. ... Amr claimed that if you don't like somebody, you haven't' talked enough with that person and you should retry. That, and the statement that respect is a gift, was already a key takeaway for me.
The third impediment is lack of intention, or in different words, how likely am I to participate in solving a problem if I don't know what you want? Does that make sense? Because it happened to me many times; getting disappointed or irritated because a co-worker didn't do or behave as I expected, but never realizing that I never told them what I expected in the first place. Suffice to say is being open and clear about what you expect from other people.
The last but not least impediment is lack of ownership. How likely am I to participate in solving a problem if I think it's not my problem? Ownership is a very important phase in a relationship. ... It's really difficult to expect people to take ownership if you don't give them the freedom to make their own choices. Unfortunately there are no easy tools or strategies to overcome this impediment. However, there are ways to work together that increase the chance that somebody does take ownership; agreeing to work by agreement...
Twitter feedback on this tutorial included:
@ddoomen: Software development in teams is all about feelings by @stevepeha #qconsf
@ddoomen: Almost every problem in life can be solved by a Broadway musical by @stevepeha #qconsf
@ddoomen: Just learned that the success of agile processes is accidental because it increases the change that people work like people #qconsf
Design, Composition, and Performance by Rich Hickey
Dennis Doomen attended this keynote:
He elaborated on a great analogy between software developers and professional musicians. He explained how musicians always start with ideas about short melodies, cadences and rhythms and use that to compose a harmonious song. In our profession that would be the equivalent of creating small autonomous components and use those to 'compose' a system 'in harmony'. If those components end up to be too complex, then you should really work hard to take them apart. Just accept that a large system will eventually end up as too complex to grasp completely. Instead, focus on the small parts first and then look at the bigger picture.
But he even went further with that. He explained us that instruments are always created for experienced musicians, simply because of the fact that you're only a novice for a short time. So why do we still optimize our code for beginners? Just like musicians practice every day, rather than making our 'instruments' (our tools) simpler, we should practice hard as well. Definitely a very inspiring talk, in particular due to the many references to the origins of electronic music (which, as a former fan of Jean-Michel Jarre, really resonated with me).
Twitter feedback on this keynote included:
@QConSF: A full house for @richhickey #qconsf keynote. Close to 1,000 practitioners attending the conference this week. pic.twitter.com/CxKkkw5qbC
?@dtunkelang: Good design = separating into things that can be composed, making each component about one thing, composing them to solve a problem #qconsf
@jar349: @richhickey contrasts old school designs (top down) with new school (composable pieces) #qconsf
@dtunkelang: Full orchestration vs. just the melody and chord changes -- great metaphor for variation in specificity of design by @richhickey #qconsf
@stephenzr: Rich hickey: the harmony of designing a system is important. Avoid big complex things; decompose. #qconsf
@mabotelh: Instruments are built for experts not beginners. Developers as players. Beginners play anyway, they make mistakes but they learn. #qconsf
@jezhumble: "Learning requires inefficiency" @richhickey quoting http://m.theatlantic.com/magazine/archive/2013/11/the-great-forgetting/309516/ ... #qconsf
?@dtunkelang: Software designers can learn from electronic instrument designers: build human interfaces on top of machine interfaces - @richhickey #qconsf
@mabotelh: Use design constraints as a way to drive creativity, like in music #qconsf
@jaaju: @richhickey design for machines, humans will adapt #qconsf
?@rschoenrock: As an industry, we spend a crazy amount of time focusing on ourselves. Without constraints, we degenerate into a complex system #qconsf
@jaaju: Fallacy of composition - every part being good does not make the whole necessarily good #qconsf
@benjchristensen: Embrace constraints. Make decisions. Leaving all options open is avoiding design. - @richhickey on designing at #QConSF
?@ddoomen: Great summary RT @janmelander: @richhickey #qconsf pic.twitter.com/SjOv6fT6aq
The Evolution of Machine Learning from Science to Software by Jeff Hammerbacher
Twitter feedback on this keynote included:
@dtunkelang: Oryx machine learning library for Hadoop released by Cloudera today: https://t.co/IvzsMqOehf #qconsf
@neilepalmer: Very excited about Gertrude - open source multi variate experimentation framework from Cloudera #qconsf http://t.co/M3WFMuXaOV
@rundavidrun: Interested in machine learning? At #qconsf first, @hackingdata announces Oryx: does model prediction for you. http://t.co/bSG4VC5LYh
@zhamakd: @hackingdata great talk calling sw engs building ecosystem of machine learning model pipeline, like DevOps for deploy pipeline #QConSF
The Unreasonable Effectiveness of Tuning by Keith Adams
Dennis Doomen attended this session:
The day started with a keynote by Keith Adams, a Facebook founding member. He claimed that only 25% of the success of software can be accounted to brilliant teams, tools, frameworks, disciplines and programming languages. The other 75% is added by tuning the system. I'm not sure I fully agree, but I do see that you don't really have a clear understanding of your system's performance until you put it in production. And that's something we do agree on, because he said that if you don't have users, you're not tuning yet (and you shouldn't). I've noticed myself that benchmarks and heuristics don't really give you enough real data to really make your system shine. On the other hand, I think you're already lucky if you managed to pull of a performance or load test in your project. He also started talking on machine learning, but that's a topic that doesn't work for me (yet).
Twitter feedback on this keynote included:
@_wee_ Tuning makes real users happier, not benchmarks or reviewers. No real users? No tuning. #qconsf #Facebook
@dgkich You must have users before you can start tuning. And tuning may account for 75% of measure of software quality, claims @keithmadams #qconsf
@rundavidrun Keith Adams at #qconsf: "Even back in 2007, performance of #PHP was a big iceberg that USS @Facebook was heading straight towards."
@dtunkelang "The experimental method does not absolve you from your responsibility for thinking"-- Keith Adams http://qconsf.com/keynote/unreasonable-effectiveness-tuning ... #qconsf
The Present and Future of the Web Platform by Brendan Eich
Paul Krill, InfoWorld, attended this keynote:
Developers are moving forward on the next two planned versions of ECMAScript, the official specification underlying JavaScript, said Brendan Eich this week. Plans call for the ECMAScript 6 specification to be done by the end of next year with work in parallel happening on the subsequent ES7 release...
Mozilla's browser support page for ES6 lists numerous programming capabilities planned, including arrow functions and spread operators....
ES7, meanwhile, will have such capabilities as Object.observ, providing a way of observing objects that is not a proxy....
Eich said WebRTC, a specification equipping browsers with real-time communications apps via JavaScript APIs and HTML5, would become a standard.
Dennis Doomen attended this keynote:
The keynote was done by Brendan Eich, the inventor of JavaScript, and dealt with the present and future of web development. However, he didn't hide the fact that he was essentially trying to convince us that the next versions of EcmaScript are going to be feature rich, performing like race cars, and solve all the issues JavaScript opponents have been throwing at it. In fact, looking at the amount of bullet points on his slides, it is as the JavaScript committee has been trying to add every single feature of every other programming language around. I did notice how well some of the things align with Microsoft's TypeScript, and support for the lambda expressions is indeed an awesome little gem. On a side note, he did manage to impress us with something really cool; running Unreal in WebGL (provided you have Firefox 22).
Twitter feedback on this keynote included:
@rundavidrun .@BrendanEich at #qconsf gives us a preview of new features in #ECMAScript 6. Finally, a bound version of 'this'. pic.twitter.com/eCGVicjzGK
@TrevorOke "Ecmascript sounds like a skin disease" Brendan Eich #qconsf
@rundavidrun Part 2 of the list. Proxy, Map, Object.mixin... #JavaScript is becoming uber-awesome. ETA end of 2014. #qconsf pic.twitter.com/9olfTLyiBX
@ronaldharmsen Nice webgl demo with Unreal on stage. Really shows of the power of the HTML5/JavaScript webplatform #qconsf
@ddoomen WOW! Seriously impressed by the frame rates of Unreal running in WebGL. My respect for web technologies has just raised a few bars #qconsf
@rundavidrun Awesome! @BrendanEich at #qconsf demos @EpicGames Sanctuary running perfectly in the browser thanks to emscripten. https://github.com/kripken/emscripten ...
@stephenzr Wo! Web as a platform- Unreal compiled to Asm. Wow! Play doom within another game. Brain melt. #qconsf
@sedovsek We're blown away by 3d games and real-time chats. Not sure how this looks in the eyes of non-web-devs. #qconsf
Applied Machine Learning and Data Science
Data Science for Hire Ed by Gloria Lau
Twitter feedback on this session included:
@dtunkelang Students and recent graduates are fastest growing segment on LinkedIn, which offers guidance based on career outcomes. - @gloriatlau #qconsf
@neilepalmer LinkedIn look at themselves as an app, not a web site. Responsive design good for web sites, not apps #qconsf
@dtunkelang Data standardization key to build data products. Entity resolution more tractable when you add social network features - @gloriatlau #qconsf
@dtunkelang Best way to improve data quality is to not have to: invest in typeahead interface to collect starndardized data upfront -@gloriatlau #qconsf
@dtunkelang LinkedIn built Notable Alumni product by matching name & standardized school data with Wikipedia, then crowdsourced eval-@gloriatlau #qconsf
?@DangerLemming LinkedIn: "Event-driven is good when I/O-bound; process-driven is good when CPU bound." Good approximation. #qconsf
@_wee_ #LinkedIn mobile app uses html5 & node.js with screen based json. One json per screen. #qconsf
Large-Scale Social Recommender Systems at LinkedIn by Mitul Tiwari
Twitter feedback on this session included:
@dtunkelang LinkedIn delivers news recommendation using explore / exploit: optimize predicted CTR vs data to improve predictions - @mitultiwari #qconsf
?@dtunkelang Feature engineering key for LinkedIn's People You May Know, e.g. modeling organization overlap http://www.cs.utexas.edu/~cjhsieh/fp086-hsieh.pdf ... - @mitultiwari #qconsf
?@dtunkelang Here's the CIKM 2012 paper that @mitultiwari is citing about LinkedIn's related searches recommendations: http://www.azreda.org/paper/cikm2012.pdf ... #qconsf
@dtunkelang Lower structural diversity increases invitation rate for LinkedIn People You May Know. Contrast to 2012 Facebook study -@mitultiwari #qconsf
Large Scale Discovery Systems at Google -- YouTube and Google Play by Hrishikesh Aradhye
Twitter feedback on this session included:
@dtunkelang: Learn more about Hrishi's YouTube annotation research from the publications he's posted at http://t.co/eiZ52sRd18 #qconsf
@dtunkelang: Discoverability on Google Play: short/clear/unique title; functional & vivid description; real screenshots; video previews, reviews #qconsf
@dtunkelang: Google Play's typeahead search interface blends navigational instant results (labeled with icons) with suggested search completions. #qconsf
@dtunkelang: Google Play sees 6M+ unique phrases searched monthly. 50% of them are misspelled. -Hrishi Aradhye #qconsf
@dtunkelang: Features to gain / retain users for Google Play apps: design for tablets, helpful web anchors, small footprint, get reviews. #qconsf
From The Lab To The Factory: Building A Production Machine Learning Infrastructure by Josh Wills
Twitter feedback on this session included:
@nopiedra: +1"@randyshoup: "Log Everything: Log files are a data scientist's best friend" @josh_wills #qconsf"
@randyshoup: Data Science needs DevOps for multivariate experimentation, exploring the parameter space @josh_wills #qconsf
@dtunkelang: Useful links about online experimentation from @josh_wills http://t.co/D5PuhEim6v , http://t.co/ivp9cHcqto , http://t.co/JmvePkc93C #qconsf
@dtunkelang: Before you invest in a complex solution, do the simplest thing that could possibly work. - @josh_wills #qconsf" #truth
Architecting Organizations for the Cloud by Sebastian Stadil
Twitter feedback on this session included:
@sklarbodds #qconsf "Infrastructure treats servers like pets, cloud treats them like cattle. If a server malfunctions just shoot it and move on"
Architectures You've Always Wondered About
How Netflix Architects for Survival by Jeremy Edberg
Dennis Doomen attended this session:
The 2nd session of the day was done by a Netflix architect, Jeremy Edberg, and dealt with how they managed to get such a scalable and reliable solution, something you would take for granted when using their services. The big 'secret' is that they always design their systems with the assumption that something will go wrong eventually. He mostly showed us how they build custom tools to monitor all their environments, but definitely impressed with the vast amounts of energy they've invested in making their systems so resilient against failures. And now that I think of it, I noticed that all those companies start to create their own tools after they've grown to a certain scale. Definitely something to remember the next time you're looking for an off-the-shelve tool to solve your problem.
Twitter feedback on this session included:
@jjathman: Highly aligned, loosely coupled teams #qconsf #netflix
@ddoomen: Good design heuristic: "Assume some system will fail at some point" #qconsf
@_wee_ : Netflix's Freedom and Flexibility allow developers to deploy anytime they want with redundant systems and automation #qconsf
Online Controlled Experiments: Introduction, Insights, Scaling, and Humbling Statistics by Ronny Kohavi
Twitter feedback on this session included:
@jezhumble: A/B testing (controlled experiments) are great because they are simple and they prove causation between @ronnyk #qconsf
@jezhumble: 90% of randomized controlled experiments are replicable, only 20% of non-randomized. @ronnyk #qconsf
@jezhumble: Online Controlled Experiments: "stop debating - it's cheaper to get the data" @ronnyk #qconsf
@jezhumble: Twyman's law: "any figure that looks interesting or different is usually wrong" @ronnyk #qconsf http://bit.ly/twymanLaw
@jezhumble: Suggestion for controlled experiments: optimize for customer lifetime value, not immediate short-term revenue @ronnyk #qconsf
?@jezhumble: "At Bing, we run over 250 concurrent experiments" - "in a visit, you're in about 15 experiments" - 5^15 versions of bing @ronny #qconsf
@jezhumble: Problem with online experiments: people cannot accept that data beats intuition - Semmelweis reflex @ronnyk #qconsf http://en.wikipedia.org/wiki/Semmelweis_reflex ...
@gereonk: "The less data, the stronger the opinion" @ronnyk on controlled experiments #qconsf
?@jezhumble : Slides from @ronnyk's awesome talk "Online Controlled Experiments" are up at http://bit.ly/expQCon #qconsf
@jezhumble: Three ways to reduce scope of an online expt: 2 don't bother with cross-browser compatibility @ronnyk #qconsf
@jezhumble: Three ways to reduce scope of an online expt: 3 use the 80/20 rule and don't bother with corner cases @ronnyk #qconsf
@jezhumble: Three ways to reduce scope of an online expt: 4(!) Accept that some experiments will take several months to design and build @ronnyk #qconsf
From Code to Monkeys: Continuous Delivery at Netflix by Dianne Marsh
Twitter feedback on this session included:
@jezhumble: Teams deploy their own code at netflix: culture is based on freedom + responsibility @dmarsh #qconsf
@thatjeffsmith: Run what you wrote, freedom + responsibility #rapid #QconSF
@thatjeffsmith: Expose best practices and features via your tools to devs, don't limit them #QconSF
@dmysticd: Nothing exposes a problem you didn't expect like a problem in production @dmarsh #qconsf
@jezhumble: Netflix created janitor monkey (OSS) to clean up unused resources on Amazon so you don't have to keep track of them. @dmarsh #qconsf
@rundavidrun: Well said! @dmarsh: "You can't have good, solid continuous delivery without good, solid testing at all stages of the automation." #qconsf
@dgkich: Best question following Netflix CD talk: "How much is your monthly AWS bill?"#qconsf
Growing from the Few to the Many: Scaling the Operations Organization at Facebook by Pedro Canahuati
Twitter feedback on this session included:
@andraz: Facebook Ops evolution: Fire, Prioritization, Expansion, Automation #qconsf
@jezhumble: In @facebook, developer commits per month is increasing exponentially @collidr #qconsf
@sedovsek: 30 mins stats from Facebook! Amazed at #qconsf. http://t.co/E0wXmiezWD
@jezhumble: Scaling operations at @facebook required cultural change, including changing hiring and retraining interviewers @collidr #qconsf
@jezhumble: If you were weak in one area of ops, put you in a team that requires that skill and pair you with an expert #qconsf @collidr
@jezhumble: At @facebook, job of operations is not to say "no" - move fast and break things. @collidr #qconsf
@jezhumble: Ops at @facebook is not there to do things that engineers don't find sexy. Have to be involved in design phase of services. @collidr #qconsf
@jezhumble: If engineers think more like ops, they'll build better software. All devs at @facebook now go through on-call @collidr #qconsf
@jezhumble: Operations don't own _anything_ in operations - you build it, you run it. Ops will support you, but not responsible. @collidr #qconsf
@jezhumble: Scaling ops is done through people and culture, not tools and processes. @collidr #qconsf
Test Automation at Google Scale by Jason Leyba
Dennis Doomen attended this session:
One session I did get a lot out of it is the one on how Google has setup their developer workflow. In our current project we've recently started using feature branches to move towards a more stable trunk. But just imagine the surprise when I learned Google has about 10000 developers all checking in on the trunk about 20 times a minute. But after learning their aggressive testing strategy and how they've optimized their automated build-and-test pipeline I've gained a renewed goal to improve ours as well. They don't even have a traditional QA group anymore. Developers are held responsible and they facilitate the testing efforts by injecting a test engineering professional into each team. In fact, test evangelism is a quintessential aspect of the developer workflow. A good example is the Testing On The Toilet practice, a single page of new ideas and experiences that developers are encouraged to read at the toilet. How's that about dedication...
Twitter feedback on this session included:
@jezhumble: : Google does everything off trunk despite 10k devs across 40 offices. @jmleyba #qconsf http://t.co/PpjyQno18D
@TrevorOke: What would the continuous integration cycle time be for 10K developers? Google uses a dependency graph to decide what needs to run. #qconsf
@rbeier: If you broke the build, fix it within 5 minutes or roll it back. Continuous builds at Google. #QConSF
@YellowBrickC: One code trunk, 20+ codechanges/min., 100 million tests/day! Via @jmleyba #qconsf http://t.co/GIP5ts4IEn
@juretriglav: How big can your company be and still change 50% of your codebase every month? The answer: As big as Google. #qconsf
@sperzu @qbaas Some stats for you from #qconsf pic.twitter.com/t7N6Z7oi7X
Demystifying the API Lifecycle
What Makes a Great API? by John Musser
Twitter feedback on this session included:
@dmysticd: A great api understands its audience. @johnmusser #qconsf
@dmysticd: Avoid api fail #qconsf http://t.co/BPZEYcdAgv
Evolution of the Netflix API by Ben Christensen
Twitter feedback on this session included:
@_wee_: Netflix has more than 2 billion web service calls per day, rank #3 in downstream traffic. #qconsf
@constantin10: In < 3 yrs Netflix switched from an API For external developers to one for internal developers #qconsf
@_wee_: Netflix API usage was shifted from 100% external developers a few years ago to >99% internal systems. #qconsf
@rbeier: REST was not working for us. It was optimizing for the wrong thing. We did not have a big external audience for our API. -Netflix #QConSF
The Magic Behind Enterprise Apps: How to expose Reliable, Scalable and Secure Enterprise APIs? by Blake Dournaee
Twitter feedback on this session included:
@dournaee: Attention Marketers: 90% of the attendees at my #API talk at #qconsf said they were actively doing #SOA inside their Enterprise #soanotdead
Engineering Culture - How to Build, Organize and Fuel the Best Engineering Team Possible
Building a Culture Where Software Projects Get Done by Greg Brockman
Twitter feedback on this session included:
@rundavidrun .@thegdb at #qconsf: missed schedules, feature creep, increased bugs, growing complexity: we still don't know how to build software well.
@_wee_ "Code is liability, not asset" "Can't solve complexity by adding more complexity" "Rewrite will always fail" Stripe CTO #qconsf
@dmysticd "You should look at code as shackles" @thegdb #qconsf
?@ronaldharmsen Rewrites will always fail (that doesn't stop people from trying, though) #qconsf
@petesoder "Roll your own solutions to your hardest problems, not your easiest ones." via @thegdb #qconsf
@mabotelh Once a bug triggered, it will keep biting you on a short timeline, no matter how unlikely it seem. #qconsf
@mabotelh No code should be considered completed until someone else is capable of maintaining it. #qconsf
@dougbarlow "Tests aren't for your benefit." Great session about software culture by @thegdb at #qconsf. pic.twitter.com/GzpXc4EsxD
@rundavidrun Technical debt says @thegdb, is necessary to get things done; just make it explicit and manage intelligently. #qconsf pic.twitter.com/Ty3KcJmOJH
@mabotelh "Launch should not be viewed as the end goal." Yes! You need to measure it! #qconsf
@skipper69 great talk by @thegdb "Building a Culture Where Software Projects Get Done". need to slow down the video later... #qconsf
How GitHub (no longer) Works by Zach Holman
Dennis Doomen attended this session:
GitHub's Zach Holman shared the many trials and tribulations of moving from a few people to the 250 they have right now. Some of the things they do to deal with the many small and distributed teams are really interesting. As an example, they try to limit the number of meetings that require in-person contact and instead facilitate collaboration by recording all internal talks (they have a dedicated team for that), and providing always-on video conferencing.
But what really struck me is that they don't have any managers. Everything is based on trust. Definitely something a lot of companies can learn from. If you can get your hands on the slides, make sure you check them out. They contain a lot of fresh ideas, but one that really stood out for me is: "Your product should be cutting edge, not your technology". It really covers the no-nonsense mentality that GitHub is showing.
What's interesting is that GitHub puts a lot of emphasis on their values, even though it has been growing ridiculously. And that's something that resonates well with the stuff Pedram Keyani has been sharing with us in his talk on evolving culture and values. Just like Zach said, you should allow failure as well as define your companies values, even if it means making a tradeoff, and make sure your entire company supports those values. I've never thought about these things, but being part of a company that is growing fast, it's easy to see how much those values can help new employees to understand the culture of the company. In fact, if you're hiring new people, you should be verifying whether that persons aligns well with your values. If you don't, you risk people who are fully disconnected from the company's culture.
Twitter feedback on this session included:
@_wee_ GitHub now has 217 employees, said the 9th employee. Zach is a cool speaker. I learned what Dunbar number is. #qconsf
?@_wee_ #github is 60% remote and no managers. It has async workflow so that they can work remotely together. #qconsf
@_wee_ #github has 153 chat rooms from 217 employees. More rooms improve signal to noise ratio. #qconsf
@_wee_: Some #github teams have Primarily Responsible Person (PRP). Smaller teams usually have 1-2 devs and 1-2 designers #qconsf
@gereonk: Tech Stack is getting smaller @ GitHub "Your product should be cutting edge - not your technology" #qconsf
Scaling Engineering Culture at Twitter by Raffi Krikorian
Dennis Doomen attended this session:
They put a lot of focus on teams and the learning experience. For instance, the teams (5-7 people) get a lot of freedom to execute an assignment in whatever way then want to do it, using whatever process that works for them. They can publicly accept an assignment and then act fully autonomous. Moreover, it's the team itself that is accountable for everything, never the individual person. In fact, bonuses are always granted to the team as well. Only your salary is affected by the individual performance. And if you don't feel comfortable in the team anymore, just apply the "vote with your feet" rule and move to a different team.
Everything else Twitter does is to support those teams. You want to know what the other teams are doing? Just join one of the many Open Beer sessions where teams show each other what they plan to do and when. Or wander through the corridors and look at the posters teams put up to share their plans. You need certain skills? Just talk to the full-time in-house teachers to setup a training (which is then recorded and posted on YouTube for the rest of the world). In fact, Raffi told us that although they do hire a-class developers, they prefer to train the in-house developers instead (which also keeps it interesting for them). In other words, continuous training is part of their values.
So how do they protect the quality of the code? Well, failure is allowed, provided that you learn from that, but hacks are not. Even better, teams are required to spend 10% of their time on removing technical debt. "You don't have technical debt? I don't believe you!" would be Raffi's reaction. Next to that, a global architecture team exists that are there to assist or advice the teams on anything that is required to maintain quality. Raffi told us that Twitter wants to be best company in the world for developers to work for, and I must admit, this does sound like an awesome company that a lot of others can learn from.
Twitter feedback on this session included:
@_wee_: #twitter "Every problem is a scaling problem" #qconsf
@_wee_: #twitter challenges in early days #qconsf http://t.co/uL1SJcZUZf
@petesoder: We've really worked hard to get rid of the 'hero' inside Twitter. @raffi on twitter's eng team organization #qconsf
@rundavidrun: In early @twitter days, @raffi says the whole dev team would swarm on defects like a team of soccer players on a ball in play. #qconsf
@LawrenceByrd: .@raffi of @Twitter at #QConSF - the "Team" is unit of culture/innovation/failure/metrics not the individual. Each team has it's own style.
@cyberelfo: Twitter teams. #QconSF http://t.co/icoAqZkDxW
@_wee_: #twitter "Failure is always an option" #qconsf
@LawrenceByrd: More @raffi @Twitter at #QConSF @QConSF: http://t.co/kh83jRG0jH http://t.co/V2zPxas4Hu http://t.co/Kd6Vmo0afX http://t.co/gyqEvFJVUv
@cyberelfo: Twitter #branchingout is really cool! #QconSF http://t.co/aCkasM8gBk
@ddoomen: Vote with your feet at @twitter sounds like a great self-organizing team practice #qconsf
@DangerLemming: Don't like your team? Leave. (I don't mean quit.) #branchingout #qconsf http://t.co/qQOtJLN8D9
@jamshally: Developers at Twitter can ship any feature to 1% of users without approval!.. Twitter talk at #qconsf
@rundavidrun: .@raffi describes #branchingout program: engineers can freely move to another team and in effect do their own garbage collection. #qconsf
@_wee_: #twitter encourages engineers to open source their work since they believe open source makes they write better code. #qconsf
@_wee_: #twitter every team should spend at least 10% of their time working on technical debt. It's lying if a team does not have any. #qconsf
Evolving culture and values. Understanding the tradeoffs. Growth through failure. The importance of leadership and open communication. by Pedram Keyani
Twitter feedback on this session included:
@softwaryblog: Hard to single out one talk from the Engineering Culture track today, but the honesty and humility of Pedram Keyani was refreshing. #qconsf
Apache Tez : Accelerating Hadoop Query Processing by Bikas Saha Arun Murthy
Twitter feedback on this session included:
@cyberelfo Apache Tez takeaways. #QconSF pic.twitter.com/83zAL5yVkF
Big Data Platform as a Service at Netflix by Jeff Magnusson
Twitter feedback on this session included:
@stonse: @NetflixOSS hadoop library Genie can grant 3 wishes so long as the wishes are hadoop, hive or pig jobs : jeff at #qconsf hadoop track
@wadcom: Great talk from @netflix about reactive REST, Falkor and JSONG (JSON for graphs) at #qconsf http://t.co/rOGEUC5hzb
Making Virtual and Remote Teams Shine
Culture and Happiness in Virtual Teams by Floyd Marinescu
Twitter feedback on this session included:
@samadisy @floydmarinescu A virtual team can only work on stage 4 or stage 5 of @davelogan1 Tribal Leadership. #qconsf
@kathyklotzguest Attributes of great virtual teams? social channels, purpose, shared values, metrics, communications. #QConSF
@samadisy @floydmarinescu Virtual Teams need a water cooler. #qconsf
@roxanneinfoq #qconsf Floyd Marinescu using a social network is essential for building strong team culture.
@samadisy @floydmarinescu After a watercooler, you need a dashboard. InfoQ/QCon uses a simple spreadsheet. #Qconsf
@samadisy @floydmarinescu Dashboards are good for triggering discussions. Self-reporting and motivational. #qconsf
@QConSF Tools for virtual team success - #yammer , scoreboards, standups and culture - #qconsf CEO @floydmarinescu #agile
@samadisy @floydmarinescu Don't use dashboards as a hammer - it will torpedo your effectiveness. #Qconsf
@samadisy @floydmarinescu quoting @DanMezick "everything in life is opt-in", so you need to be a coach more than a manager. #qconsf
?@samadisy @floydmarinescu StandUps are a MUST HAVE. Twice a week in the early morning for timezones. #qconsf
@samadisy @floydmarinescu StandUps are the most effective way to create visibility. #qconsf
@samadisy @floydmarinescu 1. Personal Check-ins. 2. The numbers. 3. Feedback. 4. Stuck points. #qconsf
@dmysticd "Routine can set you free" - Floyd Marinescu in virtual teams track #qconsf
?@samadisy @floydmarinescu SuperTransparency can be achieved on a virtual team that exceeds co-located teams. #QconSF
High Performance Virtual Teams - Oxymoron or Necessity? by Ashley Johnson
Dennis Doomen attended this session:
After lunch I moved to another virtual team related session hosted by Ashley Johnson, a very experienced agile coach which proved to be capable of involving the audience in a unique fashion. He basically give the audience short assignments that you were supposed to complete with your neighbor. Even better, at some point he even managed to get an entire row of seats to work together to complete little task. And that's also the most important message of this track. The only way to turn a group of people in a team is to give them a task to complete. That may sound like an open door, but it's quite a fundamental notion that really helped me understand why our teams are still not real teams.
We practice Kanban and use a feature-driven approach. We have been doing Scrum for 45 3-week sprints and decided to move to Kanban to better support the continuous nature of our work. However, now teams are missing some of the pressure they were used to in Scrum and a clear goal/purpose. With the insights from this session it is clear we really need to look at finding a way to reintroduce an explicit task or assignment.
Another great analogy that made me rethink some of our teams issues is to consider what happens if the wheels of a car are not aligned properly. You don't need a engineering degree in the automotive industry to understand that a slight misalignment is enough to really spoil the driving experience. In our project we've introduced a lot of principles, rules and guidelines, but I think we've failed to really help the teams understand why we are doing this. E.g. what's the architectural vision? Why did we need to introduce so much complexity? What kind of challenges are ahead of us that might affect our technical design? I'm definitely going to give some attention to that once I'm back at the office. Oh, Ashley also recommended another book titled Agile Adoption Patterns.
For the record, the audience agreed that these were important characteristics of the best teams: humility, helping each other, autonomy, tasks, and clear success criteria. Similarly, they also identified some of the worst things a team can suffer from such as: nano management, fear of failure, personal agendas, tasks per person, competition for leadership or an uneven workload.
Synergistic Effects: A mixed remote/inhouse team can be better than the sum of its parts by Dana Caulder
Dennis Doomen attended this session:
Part of the same track was a talk by Dana Caulder on how they managed to overcome some of the cultural and time zone differences when working with teams in the US and Poland. ... In her experience a successful team knows each other well. So a great tip is to always use the time that you have to wait for each other during meetings to small talk on weekend plans, someone's family, their pets or traditions, or to make jokes. It can really increase the cohesion of the teams. Hopefully my colleagues will not look funny at me the next time I'm asking those kinds of questions...
Next Generation HTML5 and JavaScript
Building URL-Driven Web Apps with Ember.js by Tom Dale
Dennis Doomen attended this session:
I also attended a follow-up session on Ember, another JavaScript alternative to Knockout and AngularJs. Tom Dale, the author of Ember did a pretty good job highlighting some of the conventions and elegance he was missing in the other frameworks. Within our current project, we've already decided to go for AngularJs, but maybe we should re-evaluate that decision once again.
Twitter feedback on this session included:
@dgkich @tomdale provides food for thought: phone # interface hasn't changed in hundreds of years. URLs are the phone #'s of the internet. #qconsf
Advanced front-end debugging by Panos Astithas
Twitter feedback on this session included:
@rundavidrun: .@pastith at #qconsf: Firefox includes responsive design mode - see diff screen sizes and USB debug on mobile device - soon for Android too.
@rundavidrun: Slides from @pastith talk at #qconsf showing new Firefox debugging features: http://t.co/lcgwxzuxtV
Managing JavaScript Complexity by Jarrod Overson
Twitter feedback on this session included:
@rundavidrun: At #qconsf @jsoverson demos Plato #JavaScript viz tool: great way to show non-technical people progress of code. https://t.co/7QW8V6b3VY
@paulmccallick: Gain productivity and performance by tuning the use of your stack, not by changing your stack #qconsf
Reactive Javascript at Netflix by Jafar Husain
Dennis Doomen attended this session:
And just to stay on the JavaScript topic, I also attended a talk on Reactive Extensions for JavaScript, performed by Jafar Husain, an extremely fast-speaking Netflix architect. It seems that Netflix is doing an impressive job, because a lot of the talks on scalability and high-performance services were done by Netflix developers. Anyway, I was happy to see that somebody finally managed to find some good use for Rx. What I particularly liked about his approach is to think of events as collections that you can execute operations on. ... Oh, and for those .NET developers that have been using Rx for .NET already, the JavaScript version has practically the same API.
Deconstructing Functional Programming by Gilad Bracha
Twitter feedback on this session included:
@naviscargo: From QConSF: "Once you understand monads, you immediately become incapable of explaining them to anyone else" -- Gilad Bracha #qconsf
Scalability, Availability, and Performance: Putting It All Together
Top 10 - Performance Folklore by Martin Thompson
Twitter feedback on this session included:
@rundavidrun .@mjpt777 at #qconsf: .NET Dictionary is 10x faster than Java HashMap for >2GB data due to memory architecture-friendly sequential table.
@humaknlght If a code function is larger than your hand then you most likely have a performance problem. #perfmatters #qconsf
The Facebook Mobile Release Process by Christian Legnitto
Twitter feedback on this session included:
@_wee_ Features vs quality vs schedule. Pick two. #Facebook chose last two. #qconsf
@_wee_ #Facebook rule #1 we ship on time with Dated-based releases. #qconsf
@_wee_ #Facebook rule#2 Make users no worse off. Rule #3 there's always the next one. If you miss to ship on this release, do next one. #qconsf
@rschoenrock "If someone comes in suggesting that we move a release date... we mock them." At the Facebook release process talk #qconsf
@_wee_ #Facebook automated cut every 4 weeks. 3.5 weeks of stabilization and half week of RC then ship #qconsf
@sedovsek: https://t.co/DIixTe5bd3 i'm just gonna leave this here... #qconsf Facebook release process.
Dennis Doomen attended an Architecture Open Space event:
The slot just before lunch was reserved for an Open Space on architecture. As somebody who really spends a great deal of my time on agile and architecture, I can tell you it was a great experience. First I joined a discussion on agile architecture and how to get the business to understand how important bringing down technical debt is, and why we need time to work on architectural changes. I was glad to hear that this challenge is a universal problem in our profession (and not only mine :-)). The main take away for me was that you should really try to avoid running in stealth mode and spend more energy explaining the business the cost of technical debt.
I couldn't resist proposing a discussion on Event Sourcing and using it to build occasionally connected system. It was fun to explain a group of experienced architects how and why we used events as a unit of synchronization. The big question was whether I would choose this architecture style again, provided similar requirements. But after some contemplating, my answer is 'most definitely yes'. However, I did emphasize the fact that I underestimated the complexity it introduces, even though I've beentalking about this topic for several years now.
Dennis Doomen proposed an Open Space event on feeling responsible:
My last session of the day involved another Open Space discussion. I proposed a topic on how to promote the pride of ownership within the teams, as well as encourage developers to feel more responsible of the code base as well as the product. I was lucky to have both Dana and Ashley in our little group. Ashley provided some background on the psychology behind the responsibility of people and basically told us that it is very hard to make people feel responsible for their choices from the outside. It should come from the person him/herself as part of moving through the stages of a responsibility process. However, generally people feel responsible for the choices that they made themselves or things they've opted in. Obviously that resonates well with Ashley's earlier observation that tasks help transforming groups of people into a team.
Dennis Doomen attended an Open Space event on cultural diversity:
I attended two discussions of which the first tried to identify ways to solve different levels of skills within the teams. We concluded that the more common solutions like brown paper sessions, peer reviews and pair programming can really help, at the end of the day, it's much better to accept the differences and allow room for failures.
The other one was proposed by myself to get some feedback on how to help team members that are afraid to ask questions, or are too much focused on solving a problem without verifying they are actually solving the right problem. To overall consensus is that you should start recognizing the differences, asking specific questions and to try to understand the people in your teams on a personal or private level. One way to do that is to organize a kind of recess outside the confines of the office to meet up with other colleagues and talk about the job, the company or whatever you like. Dana Caulder also pointed out that triggering people to ask two questions about something that cause them confusion might help break through the often seen "Yes, I'm doing fine. I fully understand". Another thing that I'm trying to do myself a bit more often is get back at people after a meeting to see if there were any concerns or questions, especially for those that find it difficult to speak out in a large group.
Opinions on QCon via Twitter included:
@_wee_: What I like the most about attending #qconsf is to learn how other companies evolve their architecture, infrastructure & operations.
@ddoomen: Even though I thought that this first day of #qconsf wasn't going to give me a lot of new insights, looking at the notes proved I was wrong.
@kriswager It is interesting to compare and contrast subjects/tracks at US and European tech conferences #qconsf definitely different maturity levels
@eduardk Beautiful view during break time at #QConSF pic.twitter.com/LmSJoTzhaO
@vserik: Not sure whether I'm happy that #qconsf is over or want 3 more days :)
@flavioribeiro: QCon was awesome. Conferences of this size renew my willingness to learn and code more and more. #qconsf
Twitter takeaways on QCon SF 2013 included:
@rundavidrun: Survey at #qconsf reveals biggest challenge for software developers is effective collaboration across remote teams...not technology!
@jonnekats: It's great to see companies like twitter and github talk so openly about their process and culture #qconsf
@AntonGerbracht: Day 3 of #qconsf over. So many great sessions. Learned tons. So hyped. Thanks infoq for putting this on. Next up tutorials for 2 days.
@tweetprateek: #qconsf 2013 started of slow for me but ended on a high note. Fun time learning
@ddoomen: Great sharing of stories from real agile experiences here at #qconsf
@infynyxx: You shouldn't write concurrent code against business domains - @mjpt777 #qconsf
@fawad: Really enjoyed @mjpt777 's lecture on Lock Free algorithms at #qconsf. Realization of the untapped potential in hardware is profound.
@QConSF: Jafar Husain presents "Reactive #Javascript @Netflix" at #qconsf - 133 attendees; 96% green rating http://t.co/55uTEnUuit
@QConSF: #qconsf honor roll: "Scaling engineering culture at #Twitter" session results: 168 attendees; 97% green. @raffi http://t.co/aNwp8A4NLz
@QConSF: @randyshoup's eBay, Google performance & #scalability war stories #qconsf session results: 99 attendees; 81% green http://t.co/rb2TdpLWet
@QConSF: #qconsf honor roll: @mjpt777's "Top 10 performance myths" - 172 attendees; 91% green rating. #java #concurrency http://t.co/rnLkau3GNE
@QConSF @meteorjs talk by @debergalis a phenomenal hit with #qconsf attendees: 139 attendees; 90% green votes. #javascript
@QConSF Honor roll - Jafar Husain's @ReactiveX #qconsf session results: 133 attendees; 98% green votes. #javascript #html5 pic.twitter.com/s3oqKfoBYR
@QConSF #qconsf honor roll: @andrewfong on the #HybridCloud Architecture @Dropbox results: 105 attendees 86% green votes. pic.twitter.com/dnX4RP0fAl
@QConSF @thegdb on software engineering culture - #qconsf session results: 187 attendees 80% green votes. #agile @stripe
@QConSF #qconsf honor roll - Pedram Keyani's @facebook engineering session results: 133 attendees; 96% green. #agile pic.twitter.com/l6uCxcNv49
@QConSF @petesoder on the "Power Shift in Engineering Recruiting" #qconsf session results: 129 attendees, 91% green. #agile pic.twitter.com/Jq76cpKItv
@QConSF #qconsf honor roll - @tomdale on building robust #javascript apps. Session results: 101 attendees, 96% green votes. pic.twitter.com/HB3wJ4IL8b
Dennis Doomen collected noticeable quotes from QCon SF 2013:
"Can't tune until you have real users." (Stephen Rylander)
"If someone comes in suggesting that we move a release date... we mock them." at the Facebook release process talk (Rich Schoenrock)
"We've really worked hard to get rid of the 'hero' inside Twitter." @raffi on twitter's eng team organization (Pete Soderling)
"Gain productivity and performance by tuning the use of your stack, not by changing your stack" (Paul McCallick)
"Your product should be cutting edge not the technology" in the GitHub talk (Prateek Jain )
"Anyone that wants to write multithreaded code should not be allowed to." so true. (Christopher Frenning)
"No code should be considered completed until someone else is capable of maintaining it." (Mauro Botelho)
"Roll your own solution to your hardest problems, not your easiest ones." (_wee_ )
"Rewrites will always fail (that doesn't stop people from trying, though)" (Ronald Harmsen ?)
"Stability is sexy" (Justin Swartsel)
"The purpose of a change management process is to make sure nothing ever changes" by @tpbrown and @jezhumble (Dave Kichler)
"Once you understand monads, you immediately become incapable of explaining them to anyone else" by Gilad Bracha (Navis )
"Software development in teams is all about feelings" by @stevepeha
"You don't need to know Category Theory, but we mention it to show how cool we are." Gilad Bracha (Rick Warren ?)
"Almost every problem in life can be solved by a Broadway musical" by @stevepeha
Conclusion
The seventh annual QCon San Francisco brought together close to 1,000 attendees and more than 100 speakers in what was the largest ever QCon to be held in the US.
QCon's focus on practitioner-driven content is reflected in the fact that the program committee that selects the talks and speakers is itself comprised of technical practitioners from the software development community. Presentations and interviews from the event will be posted on InfoQ over the coming months.
QConSF 2013 was produced by InfoQ.com. Other upcoming QCons include:
- QCon London Mar 3-7, 2014
- QCon São Paulo Apr 9-10, 2014
- QCon New York Jun 9-13, 2014
- **New: QCon Rio De Janeiro Sept 23-25, 2014
Visit http://qconferences.com/ to see a complete list of our upcoming events.