At the start of November around 1,300 attendees descended on the Hyatt Regency in San Francisco for the ninth annual QCon in the city and the biggest QCon we’ve held in the US.
QCon SF attendees - software engineers, architects, and project managers from a wide range of industries including some prominent Bay-area companies - attended 94 technical sessions across 6 concurrent tracks, 18 in-depth workshops, facilitated open spaces and, as at all English language QCons, had instant access to all filmed presentations from the event on InfoQ.
The conference featured keynotes from Bill Buxton, Principle Researcher at Microsoft, a joint presentation from Ines Sombra, Engineer at Fastly, & Caitie McCaffrey, Distributed Systems Engineer at Twitter on large-scale distributed systems, Matt Ranney, Chief Systems Architect at Uber and Co-founder of Voxer, and Shuman Ghosemajumder VP Product Management at Shape Security on the evolution of AI based security attacks.
This article summarizes the key takeaways and highlights from QCon San Francisco 2015 as blogged and tweeted by attendees. Over the course of the next 5 months, InfoQ will be publishing most of the conference sessions online, including 10 video interviews that were recorded by the InfoQ editorial team. The publishing schedule can be found on the QCon San Francisco web site.
- Avoiding the Big Crash
- So We Hear You Like Papers
- The Future of Build vs. Buy
- The Imitation Game: The New Frontline of Security
- Beyond Ad-hoc Data Science
- Dino DNA! Health Identity from the Wrist @jawbone
- Personalization in the Pinterest Homefeed
- Takes a Village to Raise a Machine Learning Model
- The Lego Model for Machine Learning Pipelines
- Architecting for Failure Induced by Human Errors
- Building Highly-resilient Systems at Pinterest
- Crossroads of Asynchrony and Graceful Degradation
Architectures You've Always Wondered About
- Alibaba Mobile Infrastructure at "China Scale"
- How Netflix Directs 1/3rd of Internet Traffic
- Scaling Uber
Browser as a Platform (Realizing HTML5)
- Beyond Ad-hoc Automation: To Structured Platforms
- Containers at Netflix - an Evolving Story
- Debugging Microservices in Production
- Inside Yelp's SOA Infrastructure
- Creating Debt-aware Culture
- Culture and the Games People Play
- Nordstrom's 114-year-old Culture in the Tech Era
Taming Distributed Architecture
The Amazing Potential Of .NET Open Source
- Architecture @ Eleme
- JD Internet + Transformation
- Mobile Monetization: Scenario Design & Big Data
- Search Technology on Mobile App @Baidu Mobile
- Technological Innovation at Alibaba
- The Philosophy of Mass Services at Tencent
- Transformation from Call Center, Web to Mobile
- Deconstructing API Security
- Getting Started with Spring Cloud
- Microservices in Continuous Delivery
- What Do Defects Really Cost? More than You Think
Keynotes
Avoiding the Big Crash
by Bill Buxton
Twitter feedback on this keynote included:
@ramrubio: Technology is not moving fast. There are just a lot of things moving slowly. #qconsf keynote @wasbuxton
@charleshumble: Think of Windows 8 on a large display. Actually please don't! Bill Buxton #qconsf
@philip_pfo: The extreme form factors have more in common than the middle. Engineer for the human first then the form factor. #qconsf @wasburton
@charleshumble: The cumulative complexity of a bunch of simple things exceeds are ability to cope with it. Bill Buxton #qconsf
@philip_pfo: The next big thing isn't an app/service/device, it is a change in the interactions of what is already there. #qconsf keynote @wasburton
@charleshumble: Every new product and service must provide great experience and excellent value - it works, flows and has demand. Bill Buxton #qconsf 1-2
@TonyPrintezis: I think your computer should be smarter than your toilet -- @wasbuxton #QConSF
@dberlind: Microsoft principal researcher @wasbuxton @ #QconSF: All $billion industries take 20 years to get to a $billion and are stealth for 1st 15.
@charleshumble: Devices need seamless aggregation and disaggregation, and graceful augmentation and degradation of capabilities. Bill Buxton #qconsf
@jessitron: Never beat yourself up for being human. Recognize it and take necessary precautions. @wasbuxton #qconsf
@NickVido: Who cares if we're connected if there is no sense of community? -The Legendary Bill Buxton #qconsf https://t.co/mcNiKTKYgW
@jitterted: Making things less complex for users is hard, making cross-device usage seamless is going to be really hard. (@wasbuxton keynote) #QConSF
So We Hear You Like Papers
by Ines Sombra & Caitie McCaffrey
Twitter feedback on this keynote included:
@infosec812: Reviewing #CAP theorem reminds us that designing for available systems is hard #qconsf
@craigsmitham: CAP told us to keep working on how to build more correct eventually consistent systems @caitie #qconsf https://t.co/rLKHUssxy7
@craigsmitham: Applications must be aware of and integrally involved in conflict detection and resolution -D. Terry #qconsf https://t.co/IBOyv2IVLF
@jitterted: Actually have to ship stuff, not just sit around and think about concurrency @caitie at #QConSF
@jitterted: Yes: don't recreate the wheel/re-research what's already been done, learn from history. @caitie #QConSF
@bridgetkromhout: If you don't write tests cause you're a "good engineer", don't say this to @randommood or she may punch you. #qconsf https://t.co/fFHVAZyzKu
@randommood: Slides for our keynote "I hear you like papers" with the loveliest @caitie link #qconsf
The Future of Build vs. Buy
by Matt Ranney
Twitter feedback on this keynote included:
@bridgetkromhout: The computing empires of today are also unlikely to endure forever @mranney #qconsf https://t.co/cIDhv1hXLw
@pbochynski: Today you can get millions of users overnight. @mranney #qconsf
@redsneaker: The cloud is a lot like renting mainframes from IBM in the 60's. #QConSF
@orrsella: Airbnb is not in the service discovery business! @mranney on Build v. Buy #QConSF. So good.
@bridgetkromhout: At a certain scale, there's no shelf you can get things from; the shelf is empty. @mranney
@__3ve: We would never make any progress if we didn't reinvent the wheel, says @mranney. #qconsf https://t.co/E33zLb35rT
@redsneaker: The biggest problem in making decisions is people. #QConSF
@sarah_shewell: Build or Buy - we're here because we want to create, not maintain someone else's code. If you're having fun, you'll work better #qconsf
@bridgetkromhout: The pace of our industry means facing our cognitive biases & emotions when making infra decisions. @mranney #qconsf https://t.co/sLTw9PxBIo
@sarah_shewell: If you build something and know how it works you will be the BEST team in the world to fix the problem @mranney #qconsf
@redsneaker: Build and easily support or buy and be at the mercy of the vendor #QConSF
@bridgetkromhout: Infrastructure software is going to be delivered as a service - the ultimate commoditization. @mranney #qconsf
@soumit: @mranney argues against a blind #NIH attitude towards buy over build. #qconsf https://t.co/4kqVWFBEbd
@jon_moore: .@mranney, in his #qconsf keynote: "People expect Netflix to be as reliable as TV." ;)
@jnorthcott: when you build, try to build up the stack and try to build open source. @mranney #qconsf
@ollie_ona: Cloud computing has made hardware a true commodity. @mranney at #qconsf keynote Build vs. Buy
The Imitation Game: The New Frontline of Security
Twitter feedback on this keynote included:
@jbillybytes: $110B/yr in damage done by botnet based attacks, Shuman Ghosemajumder keynote, #qconsf
@orrsella: "The Imitation Game" keynote by Shuman Ghosemajumder: "Sony was hacked so many times we need to say which year we're talking about" #QConSF
@DC_LAN: Every day humanity spends 17 person year solving CAPTCHAs #qconsf
@__3ve: Really wish I could ask all the devs if they've considered incorporating security auditing code when they develop their framework. #qconsf
@DC_LAN: Password is the worst kind of authentication, except for all the other authentication methods. #qconsf
Applied Machine Learning
Beyond Ad-hoc Data Science
by Oscar Boykin
Twitter feedback on this session included:
@BigDataSc: Data scientists see themselves as Tom Cruise in Minority Report, using their hands to move data from side to side - @posco #qconsf
@BigDataSc: Twitter has 100s - 1000s of jobs with complex dependencies - @posco #qconsf
@BigDataSc: This is the typical data job input -> Function -> Output - @posco #qconsf
@BigDataSc: Whether you like it or not, data is functional programming! - @posco #qconsf
@BigDataSc: No one gets into data science because they are excited about Continuous Integration - @posco #qconsf
@BigDataSc: Machine Learning data flows don't have "private modifiers" like OO languages, someone can become a dependency of your flow - @posco #qconsf
@charleshumble: Data dependency has poor encapsulation relative to OO programming so you end up with deep data dependencies. @posco #qconsf
@BigDataSc: Rigor is not a bureaucratic process to make you run slower, it's about communication and will make you run faster - @posco #qconsf
@charleshumble: Twitter scale: O(1000) batch jobs, O(100) streaming jobs. Things break without some rigour. @posco #qconsf
@BigDataSc: Logic always requires some implicit or explicit schema, an explicit one is much better - @posco #qconsf
@BigDataSc: I'm a jack when it comes to SQL - @posco #qconsf
@charleshumble: At Twitter, all data has a schema which can be expressed using Thrift. Written to HDFS/Kafka, MySQL etc. @posco #qconsf
@BigDataSc: Data that is not changing only happens in our Utopian world of Data Hygiene - @posco #qconsf cc @dpatil
@r39132: Twitter's data pipeline only allows optional fields to support fault-tolerant schema evolution @posco #qconsf https://t.co/NY1XjthUjp
@BigDataSc: Today's build tools are not equipped to compare changes between versions, only the last version - @posco #qconsf
@charleshumble: I'm showing a distributed data job that is not word count. Thank me after the talk. @posco #qconsf https://t.co/TMULEWFJMF
@jessitron: Step 1 for reliable data science: all jobs are pure functions.@posco #qconsf
@jessitron: At Twitter, the system refuses to run a job if it hasn't been deployed in over 90 days.via @posco #qconsf #NoLegacy
Dino DNA! Health Identity from the Wrist @jawbone
by Brian Wilt
Twitter feedback on this session included:
@charleshumble: Woman sleep 20 more minutes on average than men across any section of the population. No-one is sure why. @brianwilt #qconsf
@charleshumble: 4 best data products - LinkedIn People You May Know, FB newsfeed, PageRank, and Amazon Related Items - are all about CTR @brianwilt #qconsf
@BigDataSc: The amount of data that comes from a single @JawboneUP3 band is GBs/day - @brianwilt #qconsf
@BigDataSc: It's a miracle if you can run Fourier transforms on a wrist band ... but we do it - @brianwilt #qconsf
@BigDataSc: Tesla could not ship a bicycle as their MVP product, they had to ship a car - @brianwilt #qconsf
@BigDataSc: Jawbone used their wrist bands to monitor heart rate at a company all hands @brianwilt #qconsf Genius!
Personalization in the Pinterest Homefeed
Twitter feedback on this session included:
@BigDataSc: Pinterest Homefeed problems: Generating candidates, scoring and ranking, blending, creating the final feed - @dmitrychechik #qconsf
@BigDataSc: Pinnability - the name of the output of Pinterest's machine learning model - @dmitrychechik #qconsf
@BigDataSc: co-pins of images on Pinterest creates a very rich graph to extract information from - @dmitrychechik #qconsf
@BigDataSc: Origami, Russian posters and Places to visit in San Francisco is the definition of @dmitrychechik as a Pinterest user #qconsf
@BigDataSc: The scale of recommendations of Pinterest is several orders of magnitudes larger than those of Netflix and Pandora - @dmitrychechik #qconsf
@BigDataSc: Pinterest's content is evergreen, an image from years ago might be as relevant today. not true for social networks - @dmitrychechik #qconsf
@charleshumble: Pinterest working with around 1b pins. Many are evergreen. More than movies (Netflix) or Songs (Pandora). @dmitrychechik #qconsf
@BigDataSc: Pretty amusing to see how @Salesforce @Intuit and @Pinterest all have similar modeling frameworks #qconsf
@charleshumble: Pinterest has a DSL for working with end-to-end Machine Learning flow. It looks quite R-like. @dmitrychechik #qconsf
@charleshumble: Pinterest DSL allows same model to be used for off-line training and on-line classifier. @dmitrychechik #qconsf
@charleshumble: Interesting that Pinterest ML models might be different for international users vs. new users. @dmitrychechik #qconsf
@charleshumble: Encapsulate as much of the model process as possible, and separate code from model config. @dmitrychechik #qconsf
Takes a Village to Raise a Machine Learning Model
Twitter feedback on this session included:
@BigDataSc: more smart data is better than purple data - @datariver #qconsf via @bigdatasheep
@BigDataSc: The next frontier of data intensive products is well designed software architerctures - @datariver #qconsf
@BigDataSc: An amazing % of US GDP flows through the @Intuit systems - @datariver #qconsf
@BigDataSc: Awesome way to present personalization - restaurant food choices - @datariver #qconsf
@BigDataSc: Having your data app connect directly to the database without middleware is not a great pattern - @datariver #qconsf
@BigDataSc: If you have many models running in production, you need model versioning - @datariver #qconsf
@BigDataSc: It appears that Intuit has hundreds of A/B tests running on their website at the same time #qconsf
@BigDataSc: Allowing an easy way for non-technical people to run experiments is a great step towards building a data driven culture @datariver #qconsf
@BigDataSc: Storing history is overrated for most data products - @datariver #qconsf
@BigDataSc: Storing history is overrated for most data products - @datariver #qconsf
@BigDataSc: Ask yourself whether there is a business justification to make all of your data products highly available - @datariver #qconsf
@BigDataSc: Common pattern for explanatory data models, use the best model for prediction, but decision trees for your customers - @datariver #qconsf
@BigDataSc: Personalize your personalization platform - @datariver #qconsf
The Lego Model for Machine Learning Pipelines
by Leah McGuire
Twitter feedback on this session included:
@BigDataSc: Spark and Scala is the way to go for production machine learning pipelines - Leah McGuire #qconsf
@BigDataSc: Scala was given to us as an instructions from above, "My boss said, you'll use Scala and you'll love it" - Leah McGuire #qconsf Guilty :)
@BigDataSc: Monolithic machine learning frameworks are not a good fit if you already have parts of it, which is most cases - Leah McGuire #qconsf
@BigDataSc: The generic Machine Learning pipeline - Leah McGuire #qconsf https://t.co/VnNstdfMYw
@BigDataSc: The @salesforce Machine Learning pipeline named Optimus Prime, because it's a collection of bunch of Transformers - Leah McGuire #qconsf
@BigDataSc: We extended Twitter's #algebird aggregators to power our Feature Aggregators - Leah McGuire #qconsf cc @posco
@BigDataSc: The modeling, which is the step everyone thinks about in Machine Learning, ends up being the shortest step - Leah McGuire #qconsf
Architecting for Failure
Architecting for Failure Induced by Human Errors
by Amos Barreto
Twitter feedback on this session included:
@philip_pfo: Make changes with confidence by minimizing downside thru:1. simplicity2. isolation3. process4. anti-fragility@amos_barreto #qconsf
@johnscattergood: Uber creates isolation by not colocating critical and non-critical components on the same host or database #qconsf
@johnscattergood: Enforced peer review are good examples of good process #qconsf https://t.co/6gwKsJ5pTz
@johnscattergood: 99.99% availability means that you have 5 minutes to solve a problem if you have 1 incident a month #qconsf
@johnscattergood: Every production outtage is an opportunity to learn, if you are willing to take it... #qconsf
@johnscattergood: Let's take the downside out of the risk #qconsf https://t.co/h5CgfWMWaq
@agonigberg: OMAR allows Uber to automatically rollback/take action based on anomaly detection. Removes humans from critical path. @amos_barreto #qconsf
Building Highly-resilient Systems at Pinterest
by Yongsheng Wu
Twitter feedback on this session included:
@orrsella: Pinterest at a glance: 10s of 1000s of AWS instances, Scala, Finagle, Go, 100 services, 1M QPs. Wow. #QConSF
@Y2F00: #qconsf @yswu @Pinterest has +100 services written in python, java, go, c++, we gain a lot of performance with @golang
@johnscattergood: Areas on concern for building resilient services at Pinterest #qconsf https://t.co/5oa60cEkfp
@johnscattergood: Pinterest uses ZooKeeper for service registry and dynamic discovery #qconsf
@johnscattergood: Pinterest added McRouter component to delegate to different cache servers and handle failures #qconsf
@johnscattergood: Pinterest fronts databases with data services to provide availability through MySQL slaves in case of MySQL master failure #qconsf
@johnscattergood: Advantages of Async (Reactive) processing #qconsf https://t.co/KqQdJquy4T
Crossroads of Asynchrony and Graceful Degradation
by Nitesh Kant
Twitter feedback on this session included:
@johnscattergood: Synchronous APIs simply do not scale, instead embrace Asynchronous interactions #qconsf
@philip_pfo: Empathetic systems have knowledge about their peers and adapt to those signals. @NiteshKant #qconsf
@johnscattergood: Key aspects of Async systems: lifecycle control, flow control, and function composition #qconsf
@johnscattergood: Async architectures extend to the network layer with multiplexed bi-directional communication #qconsf
@johnscattergood: reactivesocket.io is an approach to asynchronous HTTP request processing including "request leases" #qconsf
@philip_pfo: .@NiteshKant convinced me- async all the things to improve avail and reduce config complexity. #qconsf
Architectures You've Always Wondered About
Alibaba Mobile Infrastructure at "China Scale"
by Zhuoran Zhuang & Zhao Xu
Twitter feedback on this session included:
@orrsella: 80K payment transaction per sec on Alibabas 11.11 sale day #QConSF
@jkuner: Wow! 11.11 shopping festival generated 14.3Billion dollars, 249M online customers, and 68% on mobile! #qconsf
@Serginio89: Shopping.. Chinese scale - Alibaba alone is bigger than all US retailers combined - #qconsf https://t.co/9D7BXWEsH8
@__3ve: Just an example of the scale of China's 11.11 Day versus US's Black Friday... #qconsf #alibaba https://t.co/EuNJFQ7K1W
@r39132: Taobao's 11.11 dwarfs Black Friday. $1.6B made in first 12 mins & 100 countries purchased in 1st min. #qconsf https://t.co/zHyf3NbJY7
How Netflix Directs 1/3rd of Internet Traffic
Twitter feedback on this session included:
@agonigberg: Netflix has 3 main components: device, data plane and control plane. Implemented using hundreds of microservices. #qconsf
@orrsella: We store the entire worlds routing information to optimize video delivery to our viewers How Netflix directs Internet traffic #QConSF
@rmacdonaldsmith: 2 out of 2 talks describe the use of a consistent hash to solve scale problems; Uber and Netflix #qconsf
Scaling Uber
by Matt Ranney
Twitter feedback on this session included:
@johnscattergood: Uber has 1500 engineers. 30% of there workforce! #qconsf
@orrsella: More than half of our engineers have been with the company for less than 5 months Scaling Uber by @mranney #QConSF
@jitterted: No casual Uber app usage, very different usage than Twitter or Facebook. (Matt Ranney #QConSF)
@adidasGuy: #qconsf - - Uber - every year we just barely make it through New Years
@infosec812: Lots of @Uber engineers don't go out on New Years Eve @MattRanney during #QConSF
@johnscattergood: Uber's system has rapidly changed to support its growth, but this speed has created a lot of technical debt that needs to managed #qconsf
@r39132: @Uber is wrestling with 700+ micro services deployed to Production and call graph spaghetti #qconsf
@pbochynski: Should microservices become immutable after initial baking period?@mranney at #qconsf
@johnscattergood: Uber engineering tacitly supports 4 server side languages: JavaScript(node.js), Python, Go, Java #qconsf
@johnscattergood: Uber is moving away from HTTP and JSON for internal communication due to performance #qconsf
@infosec812: @MattRanney "#Thrift is OK, but generated code is bad" - Been there, done that! #qconsf
@chrisTesene: What do you do when http is too slow? Well if you are @uber you write your own rdp protocol #qconsf
@orrsella: We opt for availability over consistency every time we can @mranney #QConSF
@johnscattergood: Uber culture changed to successfully deal with constant chaos #qconsf
Browser as a Platform (Realizing HTML5)
New in ECMAScript 2016 and Beyond
Paul Krill attended this session:
Under a plan to have ECMAScript upgraded each year, the 2016 version would be available June 16 of next year, said Brian Terlson a Microsoft senior program manager serving on ECMA Technical Committee 39, which is developing the specification. But Terlson, speaking at the QCon conference in San Francisco on Monday, cautioned that all feature plans are still tentative.
"Async functions will drastically change the way you write JavaScript code," said Terlson
ECMAScript 2016 also is on track to include a SIMD (single instruction, multiple data) numeric type, which would help exploit data-level parallelism. "It's strictly the ability to process more data and to process it simultaneously in a single thread," said Terlson.
The 2016 specification will also include exponentiation operator capability, which takes a base number and offers the exponent of it. PadLeft and PadRight, meanwhile, would pad out strings to fill out a certain number of characters.
Features under consideration for future ECMAScript editions include class property declarations and decorators, providing a declarative syntax for properties .
An Observables type for modeling push-based data sources such as DOM events, timer intervals, and sockets may be introduced to the standard library in ECMAScript 2017 or beyond. "Observables really help lay the foundation for a new syntax," Terlson said. Value types, such as int64 bignum, decimal, and complex, are under consideration as well.
One capability that could be a long way off, though, is JavaScript types. "It's an active area of investigation," said Terlson, who estimated a time frame of five years before developers can start talking about it. "It's a hard problem -- a very hard problem."
Containers in Practice
Beyond Ad-hoc Automation: To Structured Platforms
Twitter feedback on this session included:
@randommood: If you are building something local to your organization then your expertise becomes local to your organization @bridgetkromhout #qconsf
@jbillybytes: Sometimes you need to think value-add and choose 'boring software' over going the bespoke infrastructure route. #qconsf @bridgetkromhout
@jbillybytes: because you can doesn't mean you should. excellent point. @bridgetkromhout #qconsf
@bcantrill: Containers aren't going to fix your broken culture! - @bridgetkromhout breaking the tough news at #qconsf
@jon_moore: .@bridgetkromhout asks if your platform can offer these things... #qconsf https://t.co/PqmW5u39Yg
@jon_moore: .@bridgetkromhout : "Make the right thing be the easy thing" #qconsf < systems thinking!
Containers at Netflix - an Evolving Story
Twitter feedback on this session included:
@dubmun: Netflix has some great culture ideas including highly aligned, loosely coupled development teams teams. #qconsf
@aiborisov: Freedom and responsibility is a key part of #Netflix Culture by @sangeetan. So many companies miss this! #QConSF
@philip_pfo: OpEx makes changes go live reliably, DevEx makes changes easier to make. Containers improve the dev experience. @sangeetan #qconsf
@agonigberg: Netflix data pipeline team has ~10k containers running at any given time. @sangeetan #qconsf
@agonigberg: Recurring themes for using Docker at Neflix are: infrastructure abstraction, local dev env, automation. @sangeetan #qconsf
Debugging Microservices in Production
Twitter feedback on this session included:
@sangeetan: Think methodically to avoid debugging anti-patterns @bcantrill #qconsf
@ericturcotte: If you haven't seen Silicon Valley, you should. It's a great documentary. @bcantrill #qconsf
@erwin57: Software debugging is a pure distillation of scientific thinking.- Bryan Cantrill #qconsf
@_wsh: My mom used to tell me 'you can't play on that computer for the rest of your life.' Turns out you can! @bcantrill #QConSF
@khoubyari: The first non trivial program had 120 lines and 20 errors! @bcantrill #qconsf https://t.co/CBzfixCI4u. #docker https://t.co/XlXTs3YzRS
@jbillybytes: @bcantrill #qconsf Working software works forever because it is true.
@it_vegard: @bcantrill: Many "non-reproducable" bugs are merely unusual. #qconsf https://t.co/4jfsxVP77m
@randommood: Taxonomy of software failures by @bcantrill #qconsf https://t.co/I1Kvex9Noy
@_wsh: If you draw one less about software engineering from Richard Nixon, it's 'don't cover anything up'. @bcantrill #QConSF
@redsneaker: The difference between my code and your code: My code is a Ferrari. Your code is rustheap. You should be ashamed of your code. #QConSF
@cpswan: The 3 commandments@bcantrill on top form at #qconsf https://t.co/fyWl8zU6o5
@_wsh: Let's not romanticize production. Production is war, and war is hell. It is you & us and we have no one else to blame. @bcantrill #QConSF
@jimternet: Distributed systems almost ALWAYS have to be debugged in production. Production is war. War is hell - bryan cantrill #qconsf
@klavergne: My therapist says its good for me to talk about this. @bcantrill on debugging at #qconsf
@portixol: 'You want the GC to take away your stinky sofa? @bcantrills talk on debugging microservices - Ive never laughed so hard at a talk #qconsf
@dberlind: @Joyent's @bcantrill on what a microservice isn't (in his talk at #QConSF): "If you have a large microservice, then it's not a microservice"
Inside Yelp's SOA Infrastructure
Twitter feedback on this session included:
@dberlind: Yelp's Kyle Anderson (@solarkennedy) in his microservices talk @ #QConSF: "We call it the right-sized service not the microservice"
@ssasalatti: It's not done until it's S?h?i?p?p?e?d? monitored #qconsf
@bcantrill: Mesos out of the box is so un-opinionated that out of the box, it's almost not useful. -- Yelp's @solarkennedy at #qconsf
@sangeetan: Leverage what's out there and put it together to build a solution that works for you PaaSTA @yelp @solarkennedy #qconsf
Culture Reimagined
Creating Debt-aware Culture
by Aaratee Rao
Twitter feedback on this session included:
@WesCravens: 'If you spend a lot on production/maintenance of the software then you are paying interest on technical debt' Aaratee Rao #qconsf
@dubmun: Types of technical debt. #qconsf https://t.co/PSh1QFQwlB
@dubmun: Common symptoms of high tech debt. #qconsf https://t.co/N8jgGWYF6r
@dubmun: Consider separate technical debt backlog. #qconsf
@WesCravens: Senior business management are the least aware of Technical Debt. Aaratee Rao @ #qconsf
@dubmun: Culture needs to recognize that how things are done is as important as getting them done. #qconsf
@dubmun: Introduce a definition of done that encourages cleaning up technical debt. #qconsf https://t.co/CsxDSORdnm
@dubmun: There may come a time, if technical debt is out of hand, to declare technical bankruptcy. #qconsf
Culture and the Games People Play
by Roy Rapoport
Twitter feedback on this session included:
@dubmun: Orgs want an outcome that result from actions. Actions are informed by what people hear and what they see. @royrapoport #qconsf
@jbillybytes: I like the idea of alignment in the cultural context - powerful idea that resonates with people who care about the company. #qconsf
@dubmun: If people see and hear conflicting things, it is difficult to achieve an expected outcome. @royrapoport #qconsf
@dubmun: Number one rule: do what is in Netflix best interests. Sensing a pattern here. @royrapoport #qconsf
@jbillybytes: The Override Bar - params for overruling a proposal: 1) am i certain it's wrong 2) what harm would come by proceeding @royrapoport #qconsf
@dubmun: Set context for whats important and people will make the right decisions. @royrapoport #qconsf
@dubmun: Its not necessary to dictate what needs to be done. @royrapoport #qconsf
@jbillybytes: balance agility and predictability. optimize for agility. be smart about things you must predict. @royrapoport #qconsf
@jbillybytes: ask: are our incentive structures motivating the right behavior? @royrapoport outstanding - this will be the 1st question I ask post #qconsf
@bm2yogi: By constraining the ability of people to make decisions, you constrain their ability to make *good* decisions -@royrapoport #qconsf
@jessitron: I try to hire for insubordination:Be hard to manage, easy to lead.@royrapoport #qconsf
@jessitron: As a technical manager, I don't get to make technical decisions. I get to make priority decisions. @royrapoport #qconsf
@jessitron: Agility and predictability are in opposition.Predictability is really important for surprisingly few things.@royrapoport #qconsf
Nordstroms 114-year-old Culture in the Tech Era
Twitter feedback on this session included:
@fox_felippe: @QConSF #qconsf A well defined culture is not an option! https://t.co/AEHjBJcDcQ
@sarah_shewell: Culture is what attracts, retains, and enables people to do amazing things Sarah Lake Hagen -@NordstromPLab #qconsf https://t.co/cG2abvCkQR
@dubmun: Auditions for the Nordstroms team is a process to ensure skills and culture fit. #qconsf
@dubmun: Nordstroms Tech has walk-up service desk when user can bring tech problems. There is a dashboard with expected wait time. So cool! #qconsf
@fox_felippe: Your innovation relies on your culture pillars! @QConSF #qconsf @NordstromPLab
@slshee: Different divisions may have different culture values, but they have several common ones that binds them together - @thesarahhagan #qconsf
Design Thinking
Design Thinking, Whats in It for Me?
Twitter feedback on this session included:
@sarah_shewell: 1 size fits all never wins -@jeaninespence You have to really understand your customer #qconsf https://t.co/tpXUfhDwc5
Developing Customer Empathy
by Chris Witeck
Twitter feedback on this session included:
@sarah_shewell: Customer empathy mapping is not a one time event. It's a constant cycle for your product. @cwiteck #qconsf #designthinking
@jitterted: Many times what appears to be a technical problem can be reframed as a business/ux problem and better solved at that level. #QConSF
@johnscattergood: To develop empathy: Look below the surface (motivation), look for patterns of behavior, be observant #qconsf
@johnscattergood: Observation in practice #qconsf https://t.co/m0bMd53Xn9
@johnscattergood: Involving the technical team with Customers #qconsf https://t.co/qhT7SLJJHD
@itsjustneil: Don't hire engineers only on technical merit. Hire engineers who are excited to meet and learn about their customers. #qconsf
@jitterted: Invite customers (buyers, users, etc.) to your office, and get them to talk to your development team directly. Not just the execs. #QConSF
@jitterted: To most of your customers, your engineering team is *magic*. You can do that?!? #QConSF
@dubmun: Find every way possible to get engineers in front of customers. #qconsf @stgarrity
@craigsmitham: Instead of modeling domain objects as actors, consider first modeling concurrent activities - emphasizing workflow for improved perf #qconsf
@DC_LAN: Everyone wants to work with super ninja pirate developers, but what I want is engineers that can talk to customers. #qconsf
Modern CS in the Real World
Contracts in Clojure: Settling Types vs. Tests
by Jessica Kerr
Twitter feedback on this session included:
@agonigberg: I'm not going to deploy code to production that I only tested in my repl using printlns! @jessitron #qconsf
@agonigberg: Clojure drove me nuts until I found prismatic schema. @jessitron #qconsf
@jitterted: One test is an anecdote, generative tests are evidence - @jessitron #QConSF
@jitterted: Creating good sample/test data can be hard. Checking them can be harder, hence property tests (Schema tests!). #QConSF (@jessitron)
@jitterted: Its not enough for my service to be great, I need to boost the greatness of your service (@jessitron interpreting @wasbuxton) #QConSF
It Probably Works
Twitter feedback on this session included:
@philip_pfo: The world is not perfect but is probabilistic; our systems should be so too. @tbmcmullen #qconsf
@caitie: A single source of truth in a system is also a single source of failure @tbmcmullen #qconsf
@_wsh: Heh. @tbmcmullen, the problem with RFC 2549 is that dropped packets get really messy. #QConSF
Modern Languages in Practice
Experiences Building InfluxDB in Go
by Paul Dix
Twitter feedback on this session included:
@fawad: Unit and integration tests won't save you. If you're operating at scale, you have to test as scale @pauldix #qconsf
@Y2F00: #qconsf @pauldix : just used the standard tooling with @golang to build @InfluxDB
@Y2F00: #qconsf @pauldix : we have shockingly few dependencies due to standard tooling of @golang @InfluxDB
Go GC: Prioritizing Low Latency and Simplicity
by Rick Hudson
Twitter feedback on this session included:
@charleshumble: A blink of an eye takes about 300ms so doing GC in the blink of an eye isn't that impressive. Rich Hudson, #qconsf
@charleshumble: Go GC has one tuning knob for space vs. time. Quite a contrast to Java. Rick Hudson #qconsf
@charleshumble: GC latency is not a barrier to Go's adoption. Rick Hudson #qconsf.
Org Hacking
Beyond DevOps: How Netflix Bridges the Gap
by Josh Evans
Twitter feedback on this session included:
@petesoder: @Netflix open sources spinnaker, continuous delivery platform - announced by @Ops_Engineering at #qconsf https://t.co/y2eaQrrwV3
@portixol: Just seven engineers built @netflix's Spinnaker - their continuous delivery platform, which is open source as of today #qconsf
@portixol: Production ready checklist @netflix #qconsf @Ops_Engineering https://t.co/CTSrqMRRc9
@oyvindma: There's a new boy in town! Spinnaker from Netflix. Check it out at https://t.co/Pe8z89dkpr #qconsf
Preparing PayPal for Launch
Twitter feedback on this session included:
@johnscattergood: PayPal has 270 scrum teams each about 10-12 members #qconsf
@jkuner: TIL Paypal uses 10MW of power for 53000 hosts (in Q1) - #qconsf
@johnscattergood: PayPal separation from eBay followed an 18 month effort to integrate systems #facepalm #qconsf
@johnscattergood: PayPal separation required 100s of projects to be completed simultaneously #qconsf https://t.co/5vqPNR3sSm
@johnscattergood: PayPal and eBay used a scalable agile process using scrum of scrims to separate the companies #qconsf
@johnscattergood: When starting initiatives, it's vital to start with small quick successes to build momentum #qconsf
Taming Distributed Architecture
Demystifying Stream Processing with Apache Kafka
Twitter feedback on this session included:
@caitie: So true "The Hard Part of Distributed Systems. Coding < Debugging < Operations" @nehanarkhede #qconsf https://t.co/GzuIjCD23n
@agonigberg: In distributed systems coding for ease of operations is more important than for future debugging or coding. @nehanarkhede #qconsf
@SivaKalagarla: New interfaces coming up in Apache Kapka @nehanarkhede #qconsf https://t.co/ZrJZrBfkfk
@mohitvora: @nehanarkhede Happy to hear abt SSL support! Great talk, learned a bunch! #qconsf
Engineering for the Long Game
Twitter feedback on this session included:
@bridgetkromhout: I was there when that ecosystem was born. @shinynew_oz on being at Google for over a decade, with Borg. #qconsf https://t.co/Qu0RL2u5lq
@bridgetkromhout: If things go well, you may end up living with the consequences with your decisions for a long time to come. @shinynew_oz on growth #qconsf
@bridgetkromhout: Software is easy. People are really hard. That truth we don't always acknowledge: realtalk brought to you by @shinynew_oz. #qconsf
@charleshumble: The goal is not 'to not get paged' - the goal is to make sure you're never paged for the same thing twice. @shinynew_oz #qconsf
@charleshumble: If you cant tell if you system is up, it probably isnt. @shinynew_oz #qconsf
@bridgetkromhout: Have to cross tooling boundaries for people to work together? One more barrier you dont need @shinynew_oz #qconsf https://t.co/Pl1pEV8Aeu
@jon_moore: .@shinynew_oz : "Pick *something* and try to get most people on it" #engineeringatscale #qconsf
@bridgetkromhout: Think broadly about costs (including time), not just on your own team. Consider opportunity costs of not taking risks. @shinynew_oz #qconsf
@bridgetkromhout: If you must use a hot new thing, it should only be if nothing else solves your problems. Well-known open source = best. @shinynew_oz #qconsf
@bridgetkromhout: You can throw software at the problem of managing a system at a much greater scale than you can throw people at it. @shinynew_oz #qconsf
@charleshumble: On a large enough distributed system your chances of failure go up to 100%.Resilience is a property of system design. @shinynew_oz #qconsf
@bridgetkromhout: The basic philosophy of a well-designed distributed system is distrust and also verify. @shinynew_oz in her epic #qconsf talk.
@bridgetkromhout: Your goal is not to prevent failures but to handle them effectively. @shinynew_oz dropping real-world Google-scale experience at #qconsf.
@charleshumble: There's a significant velocity drag if you have to talk people. @shinynew_oz #qconsf
@bridgetkromhout: Monoliths are unfashionable these days, but there's also such a thing as too many microservices. @shinynew_oz on consolidation. #qconsf
@bridgetkromhout: At Google, all of our biggest DOS attacks come from ourselves. (Botnet? No, intern.) @shinynew_oz #qconsf https://t.co/JrXRkooqhv
@charleshumble: Move your biggest customer first. When we were rolling out Borg the first customer we moved was web search. @shinynew_oz #qconsf
@jon_moore: Don't let the weeds get higher than the garden. @shinynew_oz #qconsf
@bridgetkromhout: If your goal is to build a racecar, you dont want to spend >10 minutes on a cardboard box. @shinynew_oz #qconsf https://t.co/Lcq9ovrdfJ
@bridgetkromhout: Software is made of people. The hardest problems are always the human interactions. @shinynew_oz #qconsf
@charleshumble: It is not the code that's important. Its the mission. @shinynew_oz #qconsf
@philip_pfo: If your second system doesn't unlock new capabilities, don't do it. The cost will always be higher than estimated. @shinynew_oz #qconsf
Unlearning Performance Myths
Broken Performance Tools
Twitter feedback on this session included:
@bridgetkromhout: Broken performance tools are more dire than youd think: even top may be lying to you. @brendangregg #qconsf https://t.co/LRIcMcHg9D
@bridgetkromhout: All disk metrics are misleading. @brendangregg dropping some realtalk about how your i/o subsystem is complex. #qconsf
@bcantrill: 'Let's trace everything!' Now you have one million problems per second! - @brendangregg at #qconsf
@bridgetkromhout: I've done a lot of benchmark analysis... 100% of benchmarks are wrong. @brendangregg #qconsf
Java vs. C Performance
by Cliff Click
Twitter feedback on this session included:
@TonyPrintezis: #Scala is the C++ of #Java. -- @cliff_click #QConSF
@orrsella: 1/ Interesting insight from Java v. C Performance #QConSF session: HFT companies mostly use Java for quick algo iteration
Taking Java to the next Level
Netty @Apple: Large Scale Deployment/ Connectivity
Twitter feedback on this session included:
@RichardWarburto: Apple running 10s of Petabytes of data/day through over 400,000 netty instances. @normanmaurer #qconsf
@orrsella: Netty at Apple by @normanmaurer: 400,000+ instances, 10s PB/day, 10s M req/sec. Wow. #QconSF
@charleshumble: Please dont roll your own connection pool - take a look at the Finagle one - its hard to get right. @normanmaurer #QconSF
@charleshumble: Rules for the long game: 1. Imagine that you will be very, very successful. 2. Complexity grows over time.#qconsf
The Amazing Potential Of .NET Open Source
Net Machine Learning: F# and Accord.Net
by Alena Hall
Twitter feedback on this session included:
@jbillybytes: some data scientists say they spend 70% of their time preparing the data. interesting!@lenadroid #qconsf
@jbillybytes: f# integrates with the R language - that's really appealing. #qconsf @lenadroid
@jbillybytes: @lenadroid says, "Talk is cheap, show me the code". awesome. #qconsf
@jbillybytes: F# comes with data-source-specific data providers (example: World Bank). Cool! #qconsf @lenadroid
Space, Time, and State
Twitter feedback on this session included:
@iWriteSciFi: You can write UI code that is easy to reason about and predict. Just remove STATE. #qconsf @ammeep
@iWriteSciFi: state + mutation = complexity #qconsf @ammeep
@iWriteSciFi: Events are a snapshot in time. They are one method of changing from a pull to a push paradigm. #qconsf @ammeep
@iWriteSciFi: However, events are limited: they results in a large mental model and cant be combined with other events. #qconsf @ammeep
@iWriteSciFi: Instead, create an immutable core with mutable adapters. Model states, not objects. #qconsf @ammeep
@iWriteSciFi: Write pure functions that do not change state. No side-effects. #qconsf @ammeep
@iWriteSciFi: Reactive extensions = time monads :) #qconsf @ammeep
China Tech Day
Architecture @ Eleme
by Haochuan Guo
Amos Xu attended this session:
Ele.com is the largest food online ordering/delivery service in China. This company started in 2009, and experienced a 10x+ growth in 2014. It is now serving over 3 million orders per day, and during lunch time this could peak at 300 orders per second.
The initial stack was a simple PHP application running on Nginx in a single server. When the load became heavy, more servers were added with an HAProxy server in front of them. This structure was capable of serving up to a few hundreds of thousands of orders per day. Then the 10x+ growth came.
To enhance scalability, Ele.com started with decoupling its PHP application. The single application was broken down to a User Service and an Ordering Service, in which the User Service was re-written using Python. More machines were added and HAProxy servers were put in between these machines.
Business quickly grew to a million orders per day and the structure could barely cope. An F5 gateway was introduced to replace the front-end HAProxy, and HHVM was used to replace FPM in oder to enhance the performance of the PHP application. A large number of caches were used. Heavy and unimportant APIs were downgraded. Queries were optimized. Databases (MySQL and PostgreSQL) were partitioned based on domains.
The Ele.com team is now planning to simplify its architecture by putting all APIs under the management of an API Service Orchestrator, and to remove the legacy code from the stack.
JD Internet + Transformation
by Gang He
Amos Xu attended this session:
Currently, all JD.com applications run on top of their cloud. The cloud infrastructure has an IDC physical layer at the bottom, a software-defined datacenter layer (JDOS) above it, and a container cluster scheduling layer above the software-defined datacenter. JD cloud uses OpenStack and Docker extensively with 100K+ containers running in production,. They also developed their own file system (the Jingdong" file system) for storage.
JD OpenStack uses Docker as a hypervisor (docker virt driver) and uses the Nova scheduler as the Docker scheduler. On the network level, Neutron is used with integrated OpenVSwitch(OVS). The JD team worked on OVS to improve its latency for small network packets by 20%. Glance is used for image management.
Mobile Monetization: Scenario Design & Big Data
by Arther Wu
Amos Xu attended this session:
The Clean Master app has various ad formats. For user engagement without intruding into their behaviour flow, ads are usually placed after user actions as an extension of a behaviour flow (e.g. after a user takes a photo, edits it, and shares it).
Precise audience targeting is crucial in mobile ads, so building user profiles is important. Through data mining of raw data, user insights could be generated (such as Google Play app categories, interest tags, and click preferences). Various algorithms are used for supervised learning and un-supervised learning.
Search Technology on Mobile App @Baidu Mobile
by Chao Han
Amos Xu attended this session:
The architecture of Baidu Mobile is a super client-server application for mobile app users. A client plug-in system is applied to both Web UI and native UI.
A super client-server application extensively re-uses web app concepts in the client UI implementation. The main design concept is a cross-platform template. As such, performance, compatibility and controllability work can easily be shared among different platforms.
Performance utilization focuses on two aspects - the server performance, and customer experience on the client side (display speed for example). In the end four optimization points were used: server page cache, async loader and displayer, request merging, and cache negotiation.
Technological Innovation at Alibaba
by Yuan Qi
Amos Xu attended this session:
Alibaba strengthened its technology teams and initiated various projects, including Aliyun (Alibaba Cloud) starting from Sept 2009 and ODPS (Open Data Processing Service) starting from Apr 2010. As a result, Aliyun and ODPS now carry hundreds of petabytes (PB) of data from dozens of business units. Aliyun is capable of running 5000 nodes in a single cluster, and ODPS is capable of processing 100 PB of data in 6 hours.
There are lots of experiments on applied data science beyond search engines, advertisements and recommendation engines. An AI based customer service robot now serves over 95% of Alipay users, which heavily rely on speech recognition, deep learning, natural language processing and knowledge base construction. The micro-loans service provided to small enterprises requires a risk control model equivalent to those used in the financial industry, but with more agility and flexibility. A shipping insurance service for returned products on the Taobao platform always faces risks of incorrect pricing and fraud, and a dynamic pricing model significantly increased its accuracy.
The Philosophy of Mass Services at Tencent
by Bison Liao
Amos Xu attended this session:
Bison Liao, the Director of Tencent Social Network Group, summarized their philosophy of mass services into four key points.
1. Flexible availability. All features are decoupled and graded according to importance .
2. Comfort notices". Comfort notices are especially important when a service becomes unavailable. A simple "login failed" or similar failure notice would usually cause confusion for users, users might even do multiple retries which further overloads the backend servers.
3. Process crash. All processes need to be monitored, so that in the case of a crash they can be restarted in milliseconds.
4. Overload protection. The system is designed to handle the case when the buffer becomes full. Requests that stay a long time in the buffer are discarded, and number of retries are limited.
Transformation from Call Center, Web to Mobile
by Eric Ye
Amos Xu attended this session:
As the biggest travel service platform in China, Ctrip now serves over 3 million bookings per day, with 72% from mobile phones. Yet in the year 2011, Ctrip only had 25% of bookings online, while the other 75% were from call center.s The Transformatuib from call centers to mobile businesses took a lot of effort.
Major work started in 2012 when the Ctrip website was re-architected, and the UI was re-designed. New architecture encouraged open APIs, and search was optimized. A centralized mobile business unit was built in 2013, but this caused a lot of trouble in 2014 - since many new business units were added in the year, and each business unit was fighting for mobile resources. Lots of work was repetitive and inefficient. The dev infrastructure was also a bottleneck.
As such, a re-organization was initiated in 2014 and mobile resources were decentralized. Mobile app types were reduced to iOS and Android platforms only, as compared to multiple device support (iPhone/iPad/Android phone/Android tablet/Windows phone). Everything was decoupled into Microservices. Mobile apps used to run on its own business modules, so for example the hotel app had its own business modules,as did flight app and other apps. Now all services can be accessed via the data/url bus.
Solutions Track
Deconstructing API Security
Twitter feedback on this session included:
@dberlind: @Akana product VP Ian Goldsmith @ #qconsf "Anything you do from a security perspective is not going to be very useful if it's not automated"
@APIenterprise: 5 common #API weaknesses - and how to deal with them https://t.co/xPiWQui6eL #qconsf
Getting Started with Spring Cloud
by Josh Long
Twitter feedback on this session included:
@bridgetkromhout: Production is the happiest place on earth for us - it's better than Disneyland. @starbuxman on life at @pivotal. (It's true!) #qconsf
@bridgetkromhout: If you're stuck in the distant past... choose WAR. If you're here today with me, in 2015, then please choose JAR. @starbuxman #qconsf
@bridgetkromhout: Remember, the move to microservices is all about smart services, dumb pipes. @starbuxman #qconsf
Microservices in Continuous Delivery
by Ken Mugrage
Twitter feedback on this session included:
@selimekizoglu: The ability to deploy independently is vital. If you don't, don't bother with #microservices and avoid the overhead - @kmugrage @ #qconsf
@itsjustneil: If you have long running feature branches that get deployed every few weeks it's not CI/D. May have good automation just not CI/D #qconsf
@itsjustneil: CD Means: Your team prioritizes keeping the software deployable over new features #qconsf
What Do Defects Really Cost? More than You Think
by Wayne Ariola
Twitter feedback on this session included:
@buildscientist: Attending a session on the cost of software defects only to learn that Sony lost $18B during their 2011 data breach debacle. #qconsf
Opinions about QCon
Twitter feedback on QCon SF 2015 included:
@dberlind: If there's a developer event that's not to be missed, #QconSF may very well be it. @floydmarinescu & his @Qcon team have done a killer job!
@SivaKalagarla: First day #qconsf rocks!!! Good food, great concepts, extraordinary speakers. Good job @QConSF.
@Yauvansuba: #qconsf has been very helpful in accelerating thought process required to solve engineering probs while building professional software
@jbillybytes: #qconsf is empowering engineers to innovate culturally as well as technically. Great holistic approach!
Takeaways
Kelvin Meeks takeaways were:
The confluence of diverse backgrounds, technologies, and business segments represented by a global breadth of attendees - merges into a synthesis of pure enthusiasm that provides one-and-all an opportunity to pause in our day-to-day professional challenges, and lift our heads up for a moment and embrace the realization that there are people doing extraordinary things in our industry - and as their knowledge is willingly shared - new plateaus of achievement are now within our own grasp .
Conclusion
The ninth annual QCon San Francisco brought together 1,300 attendees and more than 100 speakers in what was the largest ever QCon to be held in the US. QCon's focus on practitioner-driven content is reflected in the fact that the program committee that selects the talks and speakers is itself comprised of technical practitioners from the software development community.
Presentations and interviews from the event will be posted on InfoQ over the coming months. QCon SF 2015 was produced by InfoQ.com. Other upcoming QCons include:
- QCon London March 7-11, 2016. This will be our 10th QCon in the city.
- QCon São Paulo Mar 28-Apr 1, 2016.
- QCon New York Jun 13-17, 2016.