QCon San Francisco, our our practitioner-driven conference designed for team leads, architects and project managers, was held mid November. With over 650 practitioners in attendance, the conference kicked off with two days of tutorials days presented by over 80 industry experts and authors followed by three conference days with 15 tracks and over 100 speakers covering topics ranging from cross-platform mobile development to big data architectures and NoSQL.
The following takeaways include summaries and blog entries written by editors and practitioner attendees for all keynotes, tracks and sessions along with aggregated twitter feedback during the event. InfoQ will follow this by publishing video-casts of most presentations over the next few months. The publishing schedule can be found on the QCon SF web site.
Table of Contents
- Objects On Trial with Michael Feathers & Brian Foote & Richard P. Gabriel & Joshua Kerievsky & Eliot Miranda & Dave Ungar
- Things I Wish I'd Known by Rod Johnson
- Product Engineering by Mike Lee
Advancing Agile to the Next stage with Lean
- Soup To Nuts - Harnessing Lean To Break Through The Local Optimisation by Russell Miles
- Understanding the Magic of Lean Product Development by Don Reinertsen
- Lean Startup: Why It Rocks Far More Than Agile Development by Joshua Kerievsky
- Retrospectives, Who Needs Them, Anyway? by Aino Vonge Corry
- Architecting Visa for Massive Scale and Continuous Innovation by John Davies
- Max Protect: Scalability and Caching at ESPN.com by Sean Comerford
- Scaling Social Computing - Lessons Learned at Facebook by Robert Johnson
- SimpleGeo: Staying Agile at Scale by Mike Malone
Architecture Case Studies: Cloud
- Netflix in the Global Cloud by Adrian Cockcroft
- On Track For The Cloud: Building A Hybrid Cloud At Canadian Pacific by Stuart Charlton
- Rearchitecting amazon.com for the Cloud by John Rauser
- Beauty is in The Eye of the Beholder by Alex Papadimoulis
- How to Stop Writing Next Year's Unsustainable Piece of Code by Guilherme Silveira
- Software Naturalism - Embracing the Real Behind the Ideal by Michael Feathers
- Sufficient Design: Quality In Sync With Business Context by Joshua Kerievsky
- Who Ever Said Code Was Supposed to be Pretty? by Brian Foote
Cross Platform Mobile Development
- Exploring Composition and Functional Systems in the Cloud by David Pollak
- Webmachine: A Practical Executable Model Of Http by Steve Vinoski
HTML5 & JavaScript, State of the Art
- HTML 5 Design and Development Tooling by Christophe Coenraets & Greg Wilson
- Realtime Web Apps with HTML5 WebSocket and Beyond by Guillermo Rauch
New Languages Changing Your World!
Software Architecture Improvements
- Refactoring to Patterns by Joshua Kerievsky
- Software Quality – You Know It When You See It by Erik Doernenburg
- And It All Went Horribly Wrong: Debugging Production Systems by Bryan Cantrill
- Cloud-Powered Continuous Integration and Deployment by Jinesh Varia
- Reliability Engineering Matters, Except When It Doesn't by Michael T. Nygard
- Uptime in High Volume Messaging Systems — Lessons Learned by Erik Onnen
- Building a Great Web API by Evan Cooke
- Techniques for Scaling the Netflix API in the Cloud by Daniel Jacobson
- Riak Use Cases: Dissecting the Solutions to Hard Problems by Andy Gross
- Stratos - an Open Source Cloud Platform by Paul Fremantle
- Java EE 7: Developing for the Cloud by Tyler Jewell
- Overcoming the Top Four Challenges to Real-Time Performance in Large-Scale, Data-Centric Applications by Tom Lubinski
- Performance and Scalability Panel by Hans Jespersen & Tyler Jewell & Michael Kopp & Tom Lubinski & Floyd Marinescu &Dan Pritchett
- Automating (almost) Everything using Git, Gerrit, Hudson and Mylyn by Ryan Slobojan
- Building an Ecosystem for Hackers by Rich Manalang
- Integrating SaaS Applications with Mule iON by Ken Yagen
Keynotes
Objects On Trial with Michael Feathers & Brian Foote & Richard P. Gabriel & Joshua Kerievsky & Eliot Miranda & Dave Ungar
Jean Lambert summarized this panel session including a picture of the judge and the jurors:
This morning, objects are being put on trial. Before us is the head Judge and a panel of jurors accusing objects of the following serious charges:
- Did no deliver the promise of code reuse, reusable component marketplaces and frameworks that would take the drudgery out of programming
- Inability to communicate intent clearly amongst themselves
- Insisting on intimately encapsulating behaviour and mutable state, and by having paid inadequate attention to concurrent, have left the industry vulnerable to what some have dubbed an ill-conceived "Neo-Functional Renaissance"
- Eschewing static type information forcing implementers to sacrifice performance in the name of linguistic simplicity
- High skill and education required to adequately craft state-of-the-art code to a level that is beyond what the industry is prepared to pay for
In the accused box are the following:
- A Penguin (Tux)
- A coffee mug (Java)
- A UML object (a.k.a. person, corporation, domain, identifier, serializable, cloneable, iterable)
Twitter feedback on this session included:
johnkwaters John K. Waters: Onstage now at #QconSF: "Objects On Trial." They're actually going for a mock trial, complete with wig and robe for the judge.
rbarcia Roland Barcia: According to #QConSF keynote, Objects are on trial for crimes against Computer Science. Java and UML being called out.
gbatcisco Gary Berger: #qconsf Objects "Guilty as Charged"
johnkwaters John K. Waters: Onstage now at #QconSF: "Objects On Trial" we failed to make things simple, says panelist. Complexity should be on trial.
johnkwaters John K. Waters: Onstage now at #QconSF: "Objects On Trial" comparing panelist's decision to get a vasectomy--no such discipline in the OO world!
johnkwaters John K. Waters: Onstage now at #QconSF: "Objects On Trial" we've evolved into "software in the age of sampling."
johnkwaters John K. Waters: Onstage now at #QconSF: "Objects On Trial" reuse was going to make code too cheap to meter, but it didn't work out that way. Copy&paste won
johnkwaters John K. Waters: Onstage now at #QconSF: "Objects On Trial" comparing the Space Shuttle program, which emphasized "reuse" of the craft to code reuse movement
johnkwaters John K. Waters: Onstage now at #QconSF: "Objects On Trial:" 1980s were the golden age of reuse.
robdaemon Rob Roland: Do you swear to tell the truth? The whole truth? So help you, Knuth? #QConSF
Things I Wish I'd Known by Rod Johnson
Jean Lambert reported from this keynote:
So, if your planning to do a start-up, you need to ask yourself: where is the opportunity? Remember that technology must come first! There must be an opportunity for disruption. Where is there a market gap? Did everyone else fail to see something? Did someone else screw up? …
Do NOT start writing code... it's addictive if you're any good at it. …
Once you've validated your business idea, believe it! Vision must matter to you at an emotional level. Align you team on the vision. Unless someone like you cares a whole awful lot, nothing is going to get better. …
Being an entrepreneur means a lot of risk. …
A great team makes a great company; This is what investors are looking for. Building a team is about complementary skills. First, you need to understand yourself and know what things you are good at and what things you suck at. Assemble individuals who possess the required skills and agree on your vision and ambition. Remember that you cannot afford to have different goals. …
Finally, a few mistakes you shouldn't do:
- Don't piss people off, it will come back to bite you
- Don't let customers drive your product roadmap
Twitter feedback on this session included:
rundavidrun David Whittaker: @springrod at #qconsf reminds us that often it's the questions that are complicated; the answers are usually very simple. #DrSeuss
bigballofmud Brian Foote: @springrod warns of the peril of having a Code Monkey on your Back / Green Bar Fever / Addictive qualities of TDD #qconsf no 2 code 1st
jrolstad Josh Rolstad: When beginning a startup: The technology comes first. #qconsf
Product Engineering by Mike Lee
Twitter feedback on this session included:
frubim Felipe Rubim: Funny evening keynote by @bmf - Mike Lee - on Product Engineering: "What's worse than technical debt? Social debt" #qconsf
ramrubio ramrubio: #qconsf mike lee keynote.... new things must be better than old things they replace
adrianco adrian cockcroft: Mike Lee giving #qconsf keynote on product engineering in a Three Amigos outfit, huge sombrero
CalebeSan Calebe Santos: Mike Lee @bmf keynote at #qconsf t.co/qVwujiia
Advancing Agile to the Next stage with Lean
Soup To Nuts - Harnessing Lean To Break Through The Local Optimisation by Russell Miles
Steve Mercier attended this session:
Our software development actions should be rooted in values (Values lead to Beliefs which lead to Behaviors and eventually to Actions) …
Value Statement and Value Assertions as tools
He then suggested that we develop a Value Statement that should answer the why portion of any prescribed change and to then develop the Value Assertions. The idea is to validate the measurable (automatically or not) portion of the value statement (He introduced the concept of Value Tests as tests against the market, the stakeholders - this is done now, but generally outside of IT completely). The bad news is that the toolset for value assertion/feedback is not good but if we find a way to deliver smaller increment of value and measure it, we get the benefits (similar to what Facebook seems to be doing with what they call experiments);
The good news: seeing delivered value (aka progress) around us motivates us (even if it is in another team) - this can be a key to motivate teams. …
Conclusion
We should be more Value-Driven in everything we do (track it, measure it, etc.). Why is Applesuccessful? Could it be because it is essentially value-driven and not technology driven?IT (software development departments) should stop seing itself as a slave of the business but as an important part of the business instead that can use technonlogy to deliver business value... a big change!
Understanding the Magic of Lean Product Development by Don Reinertsen
Steve Mercier attended this session and took extensive notes:
Here are some ideas the speaker considered good enough to be imported from Lean Manufacturing to Software Product Development domain:
- queuing theory: queues sizes vs resources capacities is not linear, but exponential... the more you utilize the capacity, the more the queue size explodes (ex: traffic at rush hour: reducing capacity by one lane out of 4 at rush hour will cause more than 25% delay in the queue time)... To produce an economic output, we should take great care at selecting the right amount of capacity and use the economics of queues; we should measure queues sizes in our product development cycle and search for $ vs excess product development resources with 3 curves, a-total cost, b-cost of excess capacity and c-cost of delay. Why it works so naturally for manufacturing? Because queues are physically visible. In Software Product Development, they are invisible... however, it is really useful to know how much delaying a certain software release actually costs globally.
- Batch size is an very important aspect as well (ex: coffee break with 100 person at the same time vs 5 coffee breaks with 20 persons for each one) - see the economic lot size equation reference (created in 1930s). To select the right batch size, you need to measure. There are huge benefits in small batch testing for example (less debug complexity therefore cheaper debug, less open bugs, therefor cheaper testing, faster cycle time, early feedback therefore lower cost changes, ... in the end, better economics). TBD: a link here with CI small increment theory
- WIP (Work In Process) constraints: very powerful to deal with accumulation in states (ex: maximum 10 items in Coding, ready to test and testing states); See Little's formula that says that waiting time in the queue is function of number of items in the queue and departure rate of the items. A useful tool for this one is the visual WIP boards - those boards makes non physical/invisible items as physical tokens on the boards - the horizontal axis changes from the traditional time axis to an axis on the software state. Another useful tool is a Cumulative Flow Diagram (showing Cumulative quantity vs time). It permits to see arrivals vs departure times in the queues, therefore, both at the same time the queue sizes (by taking on the x slices) and quantities in queues (by taking y axis slices).
- Synchronized cadence: it makes wait time predictable (ex: dedicated support time on each day vs a % of the time of support at an unknown time); Cadence sets an upper bound for wait time and favor less context switching costs.
…
Conclusion
- The importance of Math: The underlying mechanisms behind lean methods can be used in Software Product Development; these methods affect more than one measure of performance so trade-offs are necessary; An economic model gives comparable measures with common units. (See Life Cycle profit impact (ex: one month delay can cost 500K$)).
- We need to calculate more economically and less with intuition! The speaker measured scientifically that any analysis (even poor ones), beats intuition in software development!
Twitter feedback on this session included:
@JoshuaKerievsky: "People are measuring cycle time and not looking at queuing time." @DReinertsen #QconSF #lean
stephen_younge Stephen Younge: I never thought an old guy in a suit could blow my mind! Don Reinersten talk on Lean Product Development at #qconsf was INCREDIBLE.
JoshuaKerievsky Joshua Kerievsky: "The queues in product development are financially invisible." @DReinertsen #QconSF
Agile in the Midlife
Lean Startup: Why It Rocks Far More Than Agile Development by Joshua Kerievsky
Twitter feedback on this session included:
JoshuaKerievsky Joshua Kerievsky: Today at #qconsf in San Francisco, I'll be talking about #leanstartup: how it rocks far more than #agile development.
pascal_collet Pascal Collet: @JoshuaKerievsky presented Lean Startup at #qconsf and got me all excited about the possibilities beyond Agile
scc1976 Sean Comerford: Very interesting talk on Lean Startup by @JoshuaKerievsky at #qconsf.. best session I've attended so far
stephen_younge Stephen Younge: #agile vs. #leanstartup at a glance -- from #qconsf t.co/XsuUSR2l
stephen_younge Stephen Younge: "If you think you're cutting edge for doing Agile in 2011, you're not." -- @JoshuaKerievsky on #leanstartup at #qconsf
timotta Tiago Albineli Motta: We all have to be confortable to debug systems in production #qconsf
jrolstad Josh Rolstad: Lean startup thought: I wonder how many features I've built in the enterprise that are never used once released. #qconsf
bigballofmud Brian Foote: @joshuakerievsky admonishes us to learn to Visualize Waste, so that we can better dispose of it. #leanstartup #qconsf
bigballofmud Brian Foote: @joshuakerievsky contends that #lean and a #leanstatup mentaility are much more than an Anorexic Agile rehash #qconsf
Retrospectives, Who Needs Them, Anyway? by Aino Vonge Corry
Steve Mercier attended this session:
What is a retro? Essentially it is a chance to reflect and learn (historically called postmortem made to learn from failures) …
What happens during retros - the 5 steps
- Set the stage - getting ready (ex: create safety, I'm too busy)
- Gather data - the past (ex: artifacts contest, Timeline)
- Generate insigths - now (ex: fish-bone, 5 whys, patterns and shifts)
- Decide what to do - the future (ex: Making the magic happen, SMART goals)
- Closing the retro - retro sumary (ex: what helped, what hindered, delta)
…
You start the next retros by asking how we are doing with regards to previously set SMART goals.
Don't "sell" retros, instead sell a way of learning how to:
- avoid repeated mistakes
- identify and share success
Take away:
If you leave out a step in retrospectives, you will make problems emerge that you do not understand fully nor find solutions to.
Architecture Case Studies I
Architecting Visa for Massive Scale and Continuous Innovation by John Davies
Rowan Hick attended this session:
Visa, in 18 months, launched their product V.Me, using an Agile development process, and, sitting down ? JRuby on Rails on the front end with Java on the backend. So lets repeat that back to ourselves, one of what must be amongst the largest IT shops in the world, who are probably the most paranoid in the world for security and process etc have started adopting Agile, and using rails. Cue blowing of mind.
Twitter feedback on this session included:
jtdavies John T Davies: This is the first release (at #QCon) of the secret project we've been working on at Visa (V.me) t.co/z87yWAcY Please follow @vmedev
vmedev V.me Developer: @jtdavies and @ekatzin showed the new V.me service LIVE - lots of interest still after #qcon #qconsf #vme pic.twitter.com/b1cAMOll
amoratti Andrés Moratti: Visa is launching t.co/5AIzJPVm payment method #qconsf t.co/lVpmZoKs
rowan_h Rowan Hick: #qconsf seeing Ruby on Rails in a Visa presentation - never would have predicted that a few years ago
Max Protect: Scalability and Caching at ESPN.com by Sean Comerford
Twitter feedback on this session included:
iaflash I, Architect: #iaflash Pretty good slides on the #technical #architecture at #ESPN.com: qconsf.com/dl/qcon-sanfra… #qcon bit.ly/vUF7zC
Scaling Social Computing - Lessons Learned at Facebook by Robert Johnson
Rowan Hick attended this session:
Robert Johnson, at Facebook said something that resonated with me profoundly (paraphrased - didn't write the exact words!) "if we push to production every day, we get 365 chances to test, vs pushing every week we get only 52 chances a year to test". That's not to test things work (ie QA), it's to test that the theory of this new widget/tweak/whatever is actually going to be adopted by your users. For me, that encapsulates the 'Lean Startup' theory on the largest scale world wide, and there was a reason the audience was engrossed listening to Robert speak.
Steve Mercier attended this session and reported:
Lead in: Scaling is about dealing with failures
Facebook culture: Move fast
Problem domain: social data database (lots of distributed data!)
Context: Facebook usage has been continuously growing for the past 7 years, every single week... that gives really no rest to the development teams
Twitter feedback on this session included:
rowan_h: The FB#qconsf presentation got me all interested in Graph databases, and how they could define/allow new ideas in Product development...
pascal_collet Pascal Collet: "goal is to reduce cost of making mistakes not to avoid mistakes" R. Johnson #qconsf
cutshot Mike Fitterman: Handling failures is about trying to keep small problems from turning into big problems. #qconsf
rbarcia Roland Barcia: In Scaling Facebook session at #QConSF, number one architectural principal: "Move Fast". 2 week sprint iteration is too slow.
cutshot Mike Fitterman: Moving fast enables the ability to try and back out (try and fix or remove) #qconsf
JBollinger Jeff Bollinger: Facebook: Use percentiles in monitoring not just averages. Averages often hide problems. #qcon
SimpleGeo: Staying Agile at Scale by Mike Malone
Jean Lambert summarized this session:
Failure is a powerful tool. People learn from failure but unfortunately, the first reflex most of use have is trying to prevent failure. When you think about it, we should be spending more time becoming resilient to failure instead of preventing it; This would be a much more valuable investment of our time.
- Push code all the time - It will get less painful
- Upgrade all the time - It will get less painful
- Write tests all the time - It will get less painful
- Do code reviews all the time - It will get less painful
- Merge branches all the time - It will get less painful
- Fail all the time, it will get less painful
So go ahead, learn from failure and enable change! Just don't forget to make sure people aren't stigmatized or afraid to fail.
Twitter feedback on this session included:
marianocortesi Mariano Cortesi: Beauty is the ultimate defense against complexity @mjmalone #qconsf
JBollinger Jeff Bollinger: Deployments - Fail fast and loud so people can quickly see casual relationships between a change and failure. - @mjmalone #qcon
Architecture Case Studies: Cloud
Netflix in the Global Cloud by Adrian Cockcroft
Jean Lambert attended this session:
Simply put, if you want to do Cloud, don't try to build one, find the best possible infrastructure web services platform in the could out there and use it! …
Why use the public Cloud you ask? Well... Netflix just could build data centres fast enough and has decided to search for a robust and scalable Cloud offering to develop their platform on. The choice was set on Amazon Web Services (AWS). Partnering with AWS has allowed Netflix to be ~100% Cloud based! Some smaller back end bits are still not moved to the Cloud however, they are aggressively pushing to change that. The simple rational behind it all was that AWS has dedicated infrastructure support teams which Netflix did not, so the move was a no brainer.
Twitter feedback on this session included:
mtnygard: My favorite #qconsf talk was definitely @adrianco talking about the Netflix Global Platform. Great detail & experience report.
jrolstad: Still basking in the glow of "Netflix in the Global Cloud" at#qconsf. Good insights on how to use EC2 effectively and reliably.
neuhausler Marcel Neuhausler: #qconsf Interesting talk by Daniel Jacobson about the platform behind the Netflix API. I use the API for new movie 'recommendations' @ntflxd
beyondWeb Michael Floyd: #qconsf Netflix 30 billion requests per month. @InfoQ
JBollinger Jeff Bollinger: AWS not so infinite. Netflix often hits AWS usage limits which requires a manual process to contact Amazon to get more capacity. #qcon
JBollinger Jeff Bollinger: Netflix does connectivity and bandwidth checks at the client level (Wii, PS3, etc..) and selects among dozens of CDN locations. #qcon
JBollinger Jeff Bollinger: Netflix AWS automated scaling is built completely on real-time CPU data. Keeps average server CPU at 50%. #qcon
stephen_younge Stephen Younge: Could this be the most beautiful graph of #qcon? Auto Scaling at Netflix. t.co/fPMis8bN
On Track For The Cloud: Building A Hybrid Cloud At Canadian Pacific by Stuart Charlton
Twitter feedback on this session included:
jinman: Excellent!!#qconsf presentation by@svrc Canadian Pacific Railway Success Story Dev/Test Solution rolled out in 4 months using#aws
Rearchitecting amazon.com for the Cloud by John Rauser
Twitter feedback on this session included:
charleshumble: In Amazon mean-time between production [to] deployments during the week is 11.6 seconds#qconsf
charleshumble: Love this idea from Amazona. Write press release first. If you can't boil it down to a page or so, maybe you shouldn't build it#qconsf
floydmarinescu: Amazon.com has been running exclusively off of AWS since Nov 2010. Awesome talk at#qconsf
JBollinger: Deployments during week days average every 11.6 seconds at Amazon. At peak over 1000 in one hour.#qconsf
mtnygard: Amazon: in May, mean # hosts receiving a deployment was 10,000.#qconsf
Beautiful Code
Beauty is in The Eye of the Beholder by Alex Papadimoulis
Twitter feedback on this session included:
dblockdotorg Daniel Doubrovkine: @apapadimoulis is saying that to prevent ugly code you need to shoot the cowboys and to step back the cleverness in our code #qconsf
How to Stop Writing Next Year's Unsustainable Piece of Code by Guilherme Silveira
Twitter feedback on this session included:
amartinsn Alexandre Martins: @guilhermecaelum did a really good presentation on writing code the right way. #qconsf pic.twitter.com/l2Encmjp
rundavidrun David Whittaker: @guilhermecaelum at #qconsf: "It's not about beauty, make yourself clear. Don't mix concise with clear."
rundavidrun David Whittaker: @guilhermecaelum at #qconsf: "Is it so hard to hit Enter?" Separate lines by control flow. #makeComplexityVisible
jrolstad Josh Rolstad: Beautiful Code: Concise = with the least amount of words, make your code as clear as possible. #qconsf
Software Naturalism - Embracing the Real Behind the Ideal by Michael Feathers
Twitter feedback on this session included:
janm67 Jan Mattsson: @mfeathers applies scientific method to software development and ties in with wabi-sabi. Nice. #qconsf
rundavidrun David Whittaker: @mfeathers at #qconsf : Plot # of commits vs complexity; high/high = runaway region - lots of bugs here but high ROI if refactored.
rundavidrun David Whittaker: @mfeathers at #qconSF : "Make metrics diagrams so simple that you don't need to be technical to see there's a problem."
unclebobmartin Uncle Bob Martin: @mfeathers #qcon (sigh) What do you call people who ship Big Balls of Mud? 99.9% are failures. 0.1% are temporarily millionaires.
mfeathers Michael Feathers: "They have name for people who ship big balls of mud: millionaires" - Brian Foote (@bigballofmud) #qcon
bigballofmud Brian Foote: Code Blindness, @mfeathers says, happens when you have no way to visualize anytihing about what is happening in the code. #qcon
bigballofmud Brian Foote: Anytime you measure something, and attach incentives to it, people will find a way to game the numbers. #qcon
Sufficient Design: Quality In Sync With Business Context by Joshua Kerievsky
Twitter feedback on this session included:
amartinsn Alexandre Martins: Really inspiring talk by @JoshuaKerievsky on simple design! Well done. #qconsf
jdonelan James Donelan: @joshuakerievsky talk on sufficient design and entrepreneurial engineers was the best yet today #qconsf
lorgio_jimenez Lorgio Jimenez: Programming is like mental warfare, areas of your code should be based on ROI #qconsf #SufficientDesign
Who Ever Said Code Was Supposed to be Pretty? by Brian Foote
Twitter feedback on this session included:
rundavidrun David Whittaker: @bigballofmud at #qconsf : "Big Bucket of Glue - the dominant architectural design of the 21st century." #unfortunatelyTrue
mtnygard Michael Nygard: Below average programmers create mud. So do above average programmers. #qconsf
mtnygard Michael Nygard: "Small acts of negligence can really add up." @bigballofmud #qconsf
Big Data and NoSQL
NetApp Case Study byScott Fleming & Kumar Palaniappan
Steve Mercier extensively covered this session:
Agenda
- NetApp big data challenge
- Technology assessment
- Drivers
AutoSupport Family
- catch issues before they become critical
- secure automated call-home service
- system monitoring and non intrusive alerting
- RMA request without customer action
- enables faster incident management
…
Takeaways
- NetApp assessed multiple traditional DB technologies to solve its Big data problem and determine Hadoop was the best fit.
- HW & SW short and long term costs were key in the decision to move to Hadoop on ESeries
Cross Platform Mobile Development
Mobile JavaScript Framework Bake Off! byRoland Barcia & John Bender & David Kaneda
Twitter feedback on this session included:
multiplegeorges georges: js mobile framework bake off was great this morning. went into overtime! #qconsf
Functional Web
Exploring Composition and Functional Systems in the Cloud by David Pollak
Twitter feedback on this session included:
mtnygard Michael Nygard: End user programming in Visicalc, Excel, Hypercard, etc. all got squashed when the web came along and re-empowered IT. @dpp #qconsf
mtnygard Michael Nygard: Sudden shift from @dpp: functional, different approach to cloud computing, and iPad. Being announced here. #qconsf
Webmachine: A Practical Executable Model Of Http by Steve Vinoski
Daniel Doubrovkine attended this session:
I really enjoyed @stevevinoski’s the talk on Webmachine. I’ve never written anything in Erlang. At first I sawWebmachine as a silly framework for declarative RESTful programming. But as the talked got into more complex issues a light bulb lit in my head and I think I totally got it. It’s unlikely that I’ll ever get to use this framework, but I went from being a skeptic to actually learning something.
Twitter feedback on this session included:
argv0 Andy Gross: At @stevevinoski's webmachine talk - the only piece of software cooler than Riak #qconsf pic.twitter.com/3JNDVfzy
HTML5 & JavaScript, State of the Art
HTML5 Design and Development Tooling by Christophe Coenraets & Greg Wilson
Steve Mercier attended this session:
Agenda
- Edge - Motion and Interaction Design Tools
- Webkit contributions - CSS Regions, CSS Shaders
- JQuery Mobile
- PhoneGap
- Flash News
…
2 problems with what was demoed so far
for the first problem, phonegap offers a a builder service in the cloud on EC2 (seebuild.phonegap.com) - all connectable to a Git repository
- multiple IDEs/tools for multiple target platforms (Xcode for IOS, Eclipse for Android, windows, etc.)
- debugging remote html applications when it runs on a device
for the later problem, they developed the Web Inspector Remote tool (called also weinre), similar to web application debuggers such as the one in Chrome, Firebug, etc.)…
Conclusion
Kind of amazing how easy it was to develop animated web content applications and to deliver them to multiple devices!
Realtime Web Apps with HTML5 WebSocket and Beyond by Guillermo Rauch
Twitter feedback on this session included:
ligolnik: Socket.IO is impressive. Now I want it without node.js on the backend.#QconSF
New Languages Changing Your World!
Dart in Depth by Gilad Bracha
Jean Lambert attended this session:
So, what is Dart?
- A simple and unsurprising OO programming language
- Class-based single inheritance with interfaces
- Has optional static types
- Has real lexical scoping
- It is single-threaded
- Has a familiar syntax
Twitter feedback on this session included:
amacarth: Resurrected some Smalltalk, mixed in with Erlang like actors and optional typing to make something very cool: Dart.#qconsf#dart
ChrisHirsch: I think I will be keeping an eye on Dart#qconsf
amacarth: Wondering if DartVM will be part of the Chrome platform?#qconsf
charleshumble: #qconsf Dart has optional types "In other programming languages, a conventional type-checker is a lobotomized theorem prover"
jrolstad: Interesting point about Dart: types are only there for documentation. Your app will still run with mismatched (or no) types.#qconsf
Software Architecture Improvements
Refactoring to Patterns by Joshua Kerievsky
Twitter feedback on this session included:
stephen_younge: @JoshuaKerievsky (3rd from left) illustrates refactoring techniques using human objects at#qconsf.t.co/oRfNbarF
rundavidrun: @joshuakerievsky at#qconsf : to remove a method, delete the body and then refractor inline. Boom! All instances are deleted.
Software Quality – You Know It When You See It by Erik Doernenburg
Jean Lambert reported from this session:
Eric first posed the following question: "How can you see software quality or non-quality?"
Well, as it turns out, there are many ways this can be done. Simply put, the steps are fairly simple and straightforward:
- Collect metrics
- Aggregate data
- Render graphics
Twitter feedback on this session included:
timotta Tiago Albineli Motta: The tools that @erikdoe is showing are making me think about lot of bad examples of code #qconsf
Systems That Never Stop
And It All Went Horribly Wrong: Debugging Production Systems by Bryan Cantrill
Steve Mercier reported on this session:
Do we still need to care about failure?
single component failures still has significant costs (both economic and run-time)... but most dangerously, a single component failure puts the global system in a more vulnerable mode where further failures is more likely to happen... This is a cascading failure - and this is what induces failures in mature and reliable systems. …
Conclusion:
these challenges are real but surmountable - as in some open source systems presented below (MDB, node.js, DTrace, etc.)
Postmortem debugging: MDB
- a debugger in illumos OS (solaris derivative)
- extensible with custom debugger module
- well advanced for native code but much less for dynamic environments such as Java, Python, Ruby, JS, Erlang...
- if components going into infrastructures are developed using those languages it is critical that they support postmortem debugging.
Postmortem debugging: node.js
- not really interesting for non JAVA...
- debugging a dynamic environment requires a high degree of VM specificity in the debugger...
- see all details on dtrace.org/.../nodejs-v8-postmortem-debugging
Debugging transient component failure
- fatal failures, despite its violence, can be root-caused from a single failure
- non fatal failure, it is more difficult to compensate for and debug
Twitter feedback on this session included:
rundavidrun David Whittaker: @bcantrill at #qconsf : most failures unfort don't cause a full stop, instead they're transient failures that slowly wreak havoc over time.
Cloud-Powered Continuous Integration and Deployment by Jinesh Varia
Steve Mercier reported from this session:
Poka yo-ke technique
See here: A poka-yoke is any mechanism in a lean manufacturing process that helps an equipment operator avoid (yokeru) mistakes (poka). Its purpose is to eliminate product defects by preventing, correcting, or drawing attention to human errors as they occur.
Ex: ATM machines with card swipers designed so you keep you card at hand instead of those that "eat" your card offering you a way to possibly forget to take it back when you leave the ATM.
Basic lessons
- Keep absolutely everything in version control (scripts, etc.)
- Commit early, commit often
- Always checkin to trunk, avoid branching
- take responsibility for check ins breaking the build
- automate the build, test, deploy process
- be prepared to stop the mainline when breaking occurs
- only one way to deploy and everybody uses the same way
- be prepared to revert to previous versions
Cloud
VCS systems in the cloud, CI servers in the cloud, distributed build in the cloud
They use Jenkins, Team city, cruise control, bamboo
Twitter feedback on this session included:
mtnygard Michael Nygard: Recommendation from @jinman: put CloudFormation template in version control, create whole stacks same way for test & prod. #qconsf
mtnygard Michael Nygard: But apparently they've got one of every CI tools out there. #qconsf
lorgio_jimenez Lorgio Jimenez: facebook doesn't do development branches. They dark launch everything. Everyday is a new experiment. #qconsf #continuosdeployment
Reliability Engineering Matters, Except When It Doesn't by Michael T. Nygard
Steve Mercier attended this session and reported:
3 Types of failures
- Independent failure: Failure of one unit does not make another unit more likely to fail (excellent! but is that true for software systems? not likely)
- Correlated failure: Failure of one unit makes another unit more likely to fail.
- Common mode failure: Something else, external to the measured system, makes 2 redundant units likely to fail (ex: redundant LEDs in an Apollo capsule that were both subject to overheating)
Important Note for RE when applied to software systems:
Lots of software reliability analysis make the error of assuming a perfect independence between duplicated resources. The 2 last types of failures are way more common.
Twitter feedback on this session included:
bigballofmud Brian Foote: @mtnygard points out that 50%-65% of outages are causes by human error. #qconsf
Uptime in High Volume Messaging Systems — Lessons Learned by Erik Onnen
Steve Mercier attended this session:
Architecture - General principles
- Keep everyone moving in the same direction
- Help discrete teams understand how they interact
- Think in terms of small discrete services
- Continuous capacity planning based on real data
- Avoid local optimization decision making
…
Engineering for automation
3 level of testing (Unit, Functional (with mocks)and Integration). Commits are done to a single main git branch
Engineering for simplicity
They simplified metrics and stats capture and so they can do it everywhere!
They capture latency for Service critical operations and External service invocation
They capture counters for Service critical operations and Services faults
…
Engineering for Continuous Improvement
They use the 5 whys approach
Operations at UA
A team of 5 operations engineers handling more than 100 servers (Mostly bare metal, using EC2 for surge capacity)
Operations for Transparency
They measure absolutely everything, and monitor only the important things
Web APIs
Building a Great Web API by Evan Cooke
Daniel Doubrovkine attended this session:
I watched @emcooke's excellent Web API talk next. Evan came to GA a few weeks ago and gave a short version of the talk from a couch to a few techies. This is exactly the kind of talks I wanted to see more of. The guy had his hands in the code. He succeeded at building an API that people loved. He had priceless advice on each and every slide.
Techniques for Scaling the Netflix API in the Cloud by Daniel Jacobson
Daniel Doubrovkine attended this session:
@daniel_jacobson from Netflix described the recent evolution of the Netflix API. This was mostly about the internal API that thousands of devices use. They chose to create an API variation for every device a longtime ago and are now bringing scripting-like functionality to a middle-tier, server-side. Think of it as a server-side proxy for your API where you execute sequences of operations that are device-specific. Instead of performing chatty operations between a client and a server, you compile and execute them on the proxy.
Why is Java Still Sexy
JVM Performance Optimizations At Twitter's Scale by Attila Szegedi
Steve Mercier summarized this session:
Takeaway: Twitter went from 60 second of Garbage Collection (GC) per hour to 5 seconds per 3.5 days, mainly because of the internal slab allocator….
Twitter's biggest enemy: LatencyWhat are the main latency contributors (from worst to less):
- Garbage collector (GC)
- in process locking and thread scheduling
- I/O
- application inefficient algorithms but only the first one can be addressed by JVM tuning (which happens to be the expertise of the speaker)
…
Profile your applications, especially when 3rd parties are involved abd not well known (ex: primitive SCALA array types eating lots of memory; another ex: Guava's concurrent map eating a lot of resource - especially if you do not need concurrency)….
Writing a new memory manager is not a good idea unless you already optimized your data and your JVM... if you start doing this, it is a strong smell that you are doing something wrong!
Cloud and NoSQL Solutions
Riak Use Cases: Dissecting the Solutions to Hard Problems by Andy Gross
Twitter feedback on this session included:
jrolstad: Got sidetracked tonight with@basho's Riak database. Very enticing, want to see how it actually works for me.#qconsf
Stratos - an Open Source Cloud Platform by Paul Fremantle
Twitter feedback on this session included:
jrolstad Josh Rolstad: The Stratos presentation has good insights into how a PaaS product is built. So many things I take for granted that are hard to do. #qconsf
Performance and Scalability Solutions
Java EE 7: Developing for the Cloud by Tyler Jewell
Twitter feedback on this session included:
rbarcia Roland Barcia: At #qconsf, Java EE v7 theme is Cloud. "Cloud Services, Not Just API's." Standard REST client API, etc...
Performance and Scalability Panel byHans Jespersen & Tyler Jewell & Michael Kopp & Tom Lubinski & Floyd Marinescu &Dan Pritchett
Twitter feedback on this session included:
floydmarinescu floydmarinescu: Cloud is good for scalability because it forces you to design upfront with good patterns #qconsf performance panel
Solution Track Friday
Automating (almost) Everything using Git, Gerrit, Hudson and Mylyn by Ryan Slobojan
Steve Mercier extensively covered this session;
A presentation on how good open source tools can be integrated together to push development efficiency up….
Setting up the server (in that order)
- Install all the softwares and then:
- Clone GIT repository
- Initialize Gerrit
- Register and configure a Gerrit human user
- Register and configure a Build user (the automating user)
- Install Hudkins Git and Gerrit plugins
- Configure standard Hudkins build
- Configure Hudkins Gerrit build (integrate the review within the build)
Setting up the client (contributor - in that order)
- Install EGit, Mylyn Builds (Hudson), and Mylyn Reviews (Gerrit)
- Clone GIT repository (pull)
- Configure Gerrit as remote master
- There is no step 4! Done.
Setting up the client (committer - in that order)
- Configure a Mylyn builds Hudson repository
- Configure a Mylyn review Gerrit repository
- Create a task query for Gerrit
- There is no step 4! Done.
Building an Ecosystem for Hackers byRich Manalang
Steve Mercier attended this session:
The software life-cycle
It goes from ideas, to prototype, to code, to shipping the product and rinse and repeat until your product is bloated and the code begins to stink!…
How can we empower users
- Build opinionated software (if you do not like it, don't buy it - but you are thus building a fence around your product)
- Build a platform they can hack on (you allow you customers to get involved)…
People want to contribute to your product, but you need to give the ways to do it!
Integrating SaaS Applications with Mule iON by Ken Yagen
Steve Mercier attended this session:
The speaker wen through a discussion on APIs explosion, popular ones, the ones he thinks we should be considering in cloud application and how MuleiON can be used to properly decouple an application from all the cool cloud APIs discussed and presented….
Top 10 APIs by usage
- Google Maps
- YouTube
- Flicker
- Amazon eCommerce
- Twilio - Telephony service
- eBay
- Last.fm - Radio service
- Google search
…
iPaaS: Do not clutter your app
Use MuleiON (integration PaaS) for loose coupling your app to this cool stuff
Your killer app --> integration layer (ex: MuliON) --> cool stuff (Twitter, etc. all that we just listed above)
Benefits
- Configuration approach to APIs
- Handled for you (security, session management, streaming, http callbacks)
- consume and invoke multiple services, retries, error handling
Essentially, decouples and keeps your application code clean
Social Events
Twitter feedback on this session included:
charleshumble: Mariachi band after the@bmf party keynote bit.ly/vAriv4#qconsf
vmedev: Great turnout at the Thirsty Bear celebrating V.me by Visa announce. Thanks devs!#qconsf#qcon#vmepic.twitter.com/neDLjbRS
signde: Visa v.me#qconsf party at thirsty bear is really nice. Free drinks and food. They even have paella! Great beers here.
floydmarinescu: Check out last night's Mariachi band after the @bmf party keynote bit.ly/vAriv4
@bmf: Loved the mariachi band march to the Metreon! t.co/htem6uM3 #qcon #qconsf
suziedwards Suzi Edwards: Time for drinks at #qconsf. Feel free to bring a bucket of white wine to me at the #thoughtworks stand. And I'll enter you into our draw!
Opinions about QCon
Daniel Doubrovkine joined QCon SF this year:
What I loved about this conference is that it was mostly technical in nature and that the best speakers were hands-on industry practitioners. I also greatly enjoyed the fact that most attendees were architect-level engineers, which made for well balanced conversations over the frequently supplied drinks.
Steve Mercier shared his impression after the first day of the conference:
Ok, day 1 of the conference is done. Now, what can be said about the experience up until now? Well, for one, I think QCon SF is awesome. The organization of such an event can be difficult as there are many people, many speakers, lots of coordination and a bunch of areas where slip-ups can occur but Trifork and InfoQ have got everything under control. The venue layout minimizes walking distance between presentations and the hotel staff is very efficient and courteous, unlike the cable car operators. So, organization-wise, I have no complaints; We are in very good hands.
Jean Lambert shared his impression after the conference:
The past three days have been quite intense and I'm on the verge of a total information overload. All that aside, the QCon San Francisco International Software Development Conference was in my eyes a great success.
Twitter opinions on QCon SF 2011 included:
jamesy829: The QCon San Francisco Conference was a blast! Time to try new stuff!#qconsf
jrolstad: Relaxing at SFO after #qconsf. Was an intense 3 days - kept us busy from 8am until late at night w/ conference and vendor parties!
argv0: #qconsf winding down. great to see old friends and meet new people. thanks @trifork, @infoq, and @basho !
ddossot:My gutfeel is that #qconsf is a refactoring away to go from good to great. Let's come back next year and see how great it'll be :)
marienborg Andreas Marienborg: And it's a wrap! My first #qconsf ever, and I loved it :)
@adrianco: Biggest problem with#qconsf is wanting to be in more than one talk at the same time. I missed some good stuff today :-(
stephen_younge: Sad to leave#qconsf early but it was probably good that I got out of there before my brain exploded.
Takeaways
Me and my work colleague came here on an information seeking mission and I can now say with great confidence that we have achieved our goal. In fact, we might have collected enough data to sift through for the next couple of weeks which will then package that in a form that is consumable in our work environment. The discussions and idea exchange will definitely continue for us but at a more local level.
Twitter feedback on QCon SF 2011 included:
stephen_younge: Back in Boulder and already experiencing#qconsf withdrawal - but at least I'm finally beginning to assimilate some of the stuff I learned.
stephen_younge: Day Two of#qconsf was awesome and it has me rethinking some#Agile and#Craftsmanship assumptions. Amazing@vmedev party last night!
emcooke: Very impressed with my first#qconsf this year. It was a hybrid of an academic and 'tech' conference with meaty talks
frubim: Flying back home soon. Exciting time at#qconsf 2011. Great discussions and sessions. Can't wait for London and New York next year!!
vmedev V.me Developer: Day 2 of #qcon about 2 wrap up. hard to tweet when I'm so engrossed in the sessions.Sid from Netflix session was interesting #vme #qconsf
platypusguy Andrew Binstock: At excellent #QConSF conference today. Heard @mfeathers @dpp @hlship and @stevevinoski among others. Good line up for one day!
ChrisHirsch ChrisHirsch: Alrighty! Back from #QCon and ready to change (or guide heh) the world at work ;-)
Conclusion
The success of each QCon lies in the fact that each is created virtually from scratch by the small team of experts that make up the program committee. This year’s committee members include Floyd Marinescu, Kresten Krab Thorup, Ryan Slobojan and Aino Vonge Corry. The committee ensures that new tracks will reflect current trends in enterprise software development, and that sessions are both educational and practical. Speakers are handpicked by track hosts who themselves are practitioners. Attendance is limited so that the experience is intimate, allowing attendees to mingle with highly respected speakers, community influencers and thought leaders. The result is a unique conference that is given by practitioners for practitioners.
QCon is co-produced by Trifork (GOTO and other Conferences) and InfoQ’s parent company, C4Media. QCon conferences include QCon London, QCon San Francisco, QCon Beijing, QCon Tokyo, QCon Sao Paolo, and our recently announced QCon New York conference.