The fourth annual QCon New York was the biggest yet, bringing together over 800 team leads, architects, project managers, and engineering directors. Keynotes were given by one of the key drivers of the Lean software development movement Mary Poppendieck, Java champion and engineer Trisha Gee with former NASA researcher Todd Montgomery, and Zudio founder and stand-up comedian Mark Rendle.
In total, over 100 practitioner-speakers presented 71 full-length technical sessions and 18 in-depth tutorials over the course of five days, providing deep insights into real-world architectures and state of the art software development practices from a practitioner’s perspective.
As well as our standard 50 minute talks, this year we added a selection of 21 shorter "mini-talks" to the event covering topics such as algorithms for anti-money laundering, creating a culture of experimentation, real-time distributed event-driven computing at Credit Suisse, and testing.
Videos of most presentations were available to attendees within 24 hours of them being filmed, and over the course of the coming months InfoQ will be publishing these sessions as well as 13 video interviews that were recorded by the InfoQ editorial team. The publishing schedule can be found on the QCon New York web site.
InfoQ reported from the event, and there are also numerous QCon photos on our Flickr page. This article, however, presents a summary of QCon New York as blogged and tweeted by attendees.
- Build an Application on an OpenStack Cloud
- Modern Container Orchestration: Kubernetes, Coreos and More
- Abstraction and Federation - from Microchips to Microservices
- Driven to Distraction - How Are We Driving Development This Week?
- How Did We End up Here?
Applied Data Science and Machine Learning
- Breaking Bad at Netflix: Building Failure as a Service
- Fail Better: Radical Ideas from the Practice of Cloud Computing
- Making Distributed Data Persistent Services Elastic (Without Losing All Your Data)
- Partial Failures in a Microservices Jungle: Survival Tips from Comcast
- Too Big to Fail: Lessons from Google and healthcare.gov
Architectures You've Always Wondered about
- Google Docs - Architecting the World's Largest Real-time Collaboration System and Developer SDK
- Scaling Stack Overflow: Keeping It Vertical by Obsessing over Performance
Continuously Deploying Containers in Production
- 20 Minutes from Ticket to Production. Zero Downtime.
- Easier, Better, Faster, Safer Deployment with Docker and Immutable Containers
- Here Be Dragons: Security Maps of the Container New World
- Orchestrating Containers with Terraform and Consul
- Paasta: Application Delivery at Yelp
Emerging Technologies in Front-end Development
- Dodge Disasters and March to Triumph as a Mentor
- Lessons from Etsy: the Secrets to a Successful Remote Culture
- Nailing Engineer Retention
Fraud Detection and Hack Prevention
- Fighting Fraud, Scammers, and Hackers on the Blockchain
- Real Threat and Real Defenses – Case Study of the Unknown
- Storing Production Secrets at Scale
High Performance Streaming Data
- Distributing a Mobile Team: a Brave New Etsy Chapter
- Mobile-first Architectures
- Powering the Industrial Enterprise: Introducing the Iot Platform as a Service
- Results Driven Development & Pushing the Boundaries of Mobile
Modern Computer Science in the Real World
- A Pragmatic Introduction to Multicore Synchronization
- Consensus Systems for the Skeptical Architect
- Evolving Prolog
Modern Advances in Java Technology
- Microservices and the Art of Taming the Dependency Hell Monster
- Netflix’s Viewing Data Microservices: How We Know Where You Are in House of Cards
- Uberrush and Rebuilding Uber's Dispatching Platform
Training
Build an Application on an OpenStack Cloud
by Dana Bauer & Everett Toews
Rod Hilton attended this workshop:
I especially want to call out Everett Toews for doing an excellent job with his OpenStack half-way workshop. He had a bunch of helpers who got the slower people set up without having to slow down himself, and overall the workshop got a lot accomplished and taught me a lot about OpenStack. I think the session occasionally devolved into "here, copy and paste these commands" lacking explanation, but for the most part the entire workshop was great, and easily a highlight of the conference for me.
Modern Container Orchestration: Kubernetes, Coreos and More
Everett Toews attended this tutorial:
I took a full day tutorial called Modern Container Orchestration: Kubernetes, CoreOS and More by Kelsey Hightower. Great tutorial and very intensive. We deployed Kubernetes "by hand" (using Terraform) on Google Compute Engine (GCE) in order to understand the bits and pieces better before we might use a hosted solution like Google Container Engine. Then we created some apps, ran them on our Kubernetes cluster, and implemented patterns like Canary and Rolling Updates.
Keynotes
Rod Hilton attended the keynotes:
Mary Poppendieck's Microchips to Microservices keynote was fantastic, as was Todd Montgomery and Trisha Gee's less technical but still highly enjoyable "How Did We End Up Here?" talk. I disagreed with a lot of points that Todd and Trisha made, but their talk gave me a lot to think about, which is always fun.
Abstraction and Federation - from Microchips to Microservices
Jeanne Boyarsky attended this keynote:
Large systems don’t deal well with friction. Container ships helped with this. Standard size/shape.
Enterprises have databases. We create deep dependencies across apps because all hit same data. Which means high friction because it is hard to change. Amazon microservices have local data. Trying to decouple.
What is a microservice?
- A microservice must be independently deployable to be useful.
- Small team with end-to-end responsibility. You build it. You monitor it. You fix it.
- No central DB. Extensive automation and monitoring. Canary releasing. Smart versioning services. Double mock contract services.
Companies with very high volumes like Amazon seem to require microservices. Requires strict discipline/operational excellence. Teams need situational awareness. Must know all the time how a service impacts customers and providers.
If you and your provider change rapidly, how do you know a mock returns same value? Create a pact so can check. See pact github project.
Goals: limit risk and lower friction. …
If you have branching, you aren’t doing continuous delivery. Deploy to the mainline always with new features on switches so can turn on when ready.
Some people want to be safe (safety goals) and some want to be experimental (aspirational goals). With safety, failure/setbacks cause more efforts. With aspirational goals causes more effort. For safety, attention is for bad behavior; with aspirational, it is for good. Ops tends to be safety and dev tends to be aspirational. Marriage works well when have both, but must work towards same goal.
Who is responsible? Everyone. “Nobody succeeds unless everyone succeeds.”
Getting to continuous delivery isn’t inherently harder than other tech issues. The main obstacle is organization.
Twitter feedback on this keynote included:
@charleshumble: The real scaling in software happens not through abstraction but through federation. Mary Poppendieck, #qconnewyork
@philip_pfo: h/w scales by miniaturization and abstraction; s/w scales by federation and wide participation. #qconnewyork
@wesreisz: We do something different in the software industry... we share. Great quote from @mpoppendieck at #qconnewyork
@philip_pfo: The tech industry scales and grows so fast because of its unique culture of open sharing @mpoppendieck #qconnewyork
@charleshumble: A microservice is independently deployable or it doesn't count Mary Poppendieck, A lot of duplication, but that's fine. #qconnewyork
@DataMiller: Microservices: Independence means duplication of effort, effort which can be customized to the problem #qconnewyork
@SrikarDoddi: Good slide from QCon that reinforces the Monolith vs. Micro-service architectures #qconnewyork http://t.co/wxvETT9F0y
@DataMiller: Microservices means end to end responsibility, or it doesn't work. #qconnewyork
@danielbryantuk: The only way to scale software is through federation and wide participation, e.g., the Internet @mpoppendieck #qconnewyork
@philip_pfo: Microservices key benefit is low friction via independent deployment, not removing duplication. #qconnewyork http://t.co/6AJMpmBHQn
@jon_moore: .@mpoppendieck: "we used to smash our systems with things called 'releases'" #qconnewyork
@charleshumble: You are not actually doing continuous delivery if you have branching. Mary Poppendieck, #qconnewyork
@danielbryantuk: Poke, don't smash, a complex system like a monolith @mpoppendieck talking about the need for frequent small system releases #qconnewyork
@danielbryantuk: One goal for software delivery - shared responsibility @mpoppendieck at #qconnewyork http://t.co/V36Oml5UbF
@philip_pfo: People are either prevention or promotion focused. Accept this and align goals for #devops success. #qconnewyork http://t.co/CMFmUQNJ69
Driven to Distraction - How Are We Driving Development This Week?
by Mark Rendle
Benoit Lafontaine attended this session (excerpt translated from French):
A very refreshing ending keynote, after 3 days it was welcome, where we did a tour on "xxx driven development". Some examples:
- BDD: Buzzword Driven Design, or how to develop a nodejs-AngularJS-MongoDB-Hadoop application, on a microservice architecture... Only to find out if was just to display the cafeteria menu.
- CDD: Clipboard Driven Design (aka Stack Overflow Driven Design)
- WDD: Wizard Driven Design, the VBA developers know what this is. And to the devs using Rails, Yeoman, or the like: it's not because it's command-line that it's not a wizard ;)
Twitter feedback on this keynote included:
@zimmermatt: Mark Rendle going over [A-Z]DD.He's on Hipster Driven Development.#qconnewyork http://t.co/6i4XTlZYdL
@jeanneboyarsky: ADDD - attention deficit driven development- squirrel! @hellata awesome to see squirrel on screen! #qconnewyork
@nzmark: ADDDD: ADD driven development! More common than we think :) #qconnewyork
@NativeWired: Maybe do LDD? Love Driving Development CDDCulture driven development#qconnewyork #qcondd
@skucherov: Self-Driven Development (SDD) #qcondd #qconnewyork
@radicalcoder: CDD -Caffeine Driven Development - new tendency? #QconNewYork @ New York Marriott At The Brooklyn Bridge https://t.co/NhjIspVR8E
@PhoenixRejoices: SDD: Sugar-high Driven Development #qcondd #qconnewyork
@NativeWired: DDD - Domain Driven Developmen - development based on which .com domain is available ;)@markrendle at #qconnewyork
@NativeWired: IDD - introvert driven development (no-one talks with each other)@markrendle at #qconnewyork #qcondd
@NativeWired: QDD - QCon driven development.All code is inspired by stuff people heard at #qconnewyork #qcondd
@emerleite: BDD - Boss Driven Development - you do what your boss wants you to do, not what you have to do #qconnewyork #qcondd
@emerleite: NDD - Not invented here driven development. Recreate tools, frameworks, libraries, etc. #qconnewyork
@jeanneboyarsky: every program contains at least one bug and can be reduced by one line #qconnewyork
@jeanneboyarsky: refactoring tomorrow involves coming back and wondering how the tests even pass in the first place #qconnewyork
@emerleite: NDD - nonsense driven development - when you're asked to develop nonsense stuff - #qconnewyork #qcondd
@NativeWired: Type of development: WWBD = What Would (uncle) Bob Do? @markrendle talking at #qconnewyork #qcondd /cc @unclebobmartin
@zimmermatt: Testosterone Driven Development... the rise of the bro-grammer!#qconnewyork @markrendle
How Did We End up Here?
by Trisha Gee and Todd Montgomery
Jeanne Boyarsky attended this keynote:
Software process stats
- Agile and iterative have similar project success rate
- Ad-hoc and waterfall are similar
- Agile/iterative is only a little better
- Projects with less than 10 people significantly better than twenty or more
- Different reports measure successful projects differently. 30%- 60%.
- More expensive projects not better; “throwing more $ at projects don’t make it better” …
Hardware
- Mobile makes us think about hardware again – battery and bandwidth. Free lunch on throwing hardware at problem is over
- Hardware has been designed to make bets on locality and predictable for access. Pre-fetching and the like make an order of magnitude difference.
- Bandwidth is increasing. Latency is staying the same.
Diversity
- Testosterone Driven Development
- Increase in women went up with other STEM fields. Then in CS it started declining in 1980. This is when PCs were introduced and marketed to boys.
- Can catch up in a year even if start coding in college.
- Grace Hopper invented COBOL and the compiler
- Margret Hamilton invented async programming via NASA
Twitter feedback on this keynote included:
@charleshumble: Waterfall is less successful then having no process at all. @trisha_gee and @toddlmontgomery #qconnewyork
@dstefanov: Software project success rates #qconnewyork http://t.co/NPhRd45Qw9
@philip_pfo: Software projects - more people, process, money doesn't increase success #qconnewyork keynote @trisha_gee @toddlmontgomery
@skormos: Projects of $1M+ have a higher % chance of failure than smaller budgets. Larger teams, same issue. Mo-money, mo-problems. #qconnewyork
@jeanneboyarsky: Enterprise architect == so good at job that no longer do development -trisha gee #qconnewyork
@danielbryantuk: More people and more cash does not necessarily make a better project... #qconnewyork http://t.co/w7RbcxigR3
@zimmermatt: Secrets of software project success?* Iterative process* Small team <12* Low budget#qconnewyork @toddlmontgomery @trisha_gee
@charleshumble: Technologists ARE part of the business. @trisha_gee and @toddlmontgomery #qconnewyork
@mjbrender: As Technologists, we are fundamental to the business. We need to understand the business and they us.” ~ @trisha_gee
@philip_pfo: MVP is really asking how to get an answer with minimum investment, not an excuse for anemic features #qconnewyork keynote
@kylehayes: The product owner is more than a buffer—they provide a unified and championed vision on behalf of the customer. #qconnewyork
@charleshumble: Shared mutable state should be the most feared words in computing. It should only be used for systems programming. #qconnewyork
@jon_moore: Embrace append-only, single writer, and shared-nothing designs. @toddlmontgomery and @trisha_gee from #qconnewyork
@charleshumble: Agile is the least unsuccessful methodology we have. @trisha_gee and @toddlmontgomery #qconnewyork
@jon_moore: .@toddlmontgomery and @trisha_gee: "embrace binary formats and stop whining..." #qconnewyork
@jon_moore: Synchronous communication is the crystal meth of distributed programming < THIS /via @toddlmontgomery and @trisha_gee @ #qconnewyork
@mjbrender: .@trisha_gee is making me realize the term “exception” is incredibly misleading. Exceptions are far from exceptional. #qconnewyork
@SpykeBytes: Errors need to be first class messages. Don't swallow important information. #qconnewyork #qconnyc #PCTY
@kylehayes: Perhaps abstraction should be thought of as extraction #qconnewyork
@charleshumble: The purpose of abstraction is not to be vague,but to create a new semantic level in which one can be absolutely precise #qconnewyork
@zimmermatt: Think about transformation and flow of data, not code!#qconnewyork @toddlmontgomery @trisha_gee
@zimmermatt: Functional programming is not the answer to multi-core @toddlmontgomery and @trisha_gee assert. #qconnewyork
@charleshumble: As soon as you realize that most people don't know what they are doing, the world makes a lot more sense. @trisha_gee #qconnewyork
Applied Data Science and Machine Learning
Automating Operational Decisions in Real-time @ Netflix
by Chris Sanden
Twitter feedback on this session included:
@philip_pfo: Human effort does not scale - people make squishy decisions @chris_sanden #qconnewyork
@jon_moore: Intriguing > via @chris_sanden : add a control group of old verison of same size as canary group during incremental rollout #qconnewyork
@danielbryantuk: Netflix use machine learning for automated canary release analysis, server outlier detection and alerting #qconnewyork @NetflixOSS
@allmycode: 16% of canary tests result in “do not deploy”. @chris_sanden #qconnewyork
@danielbryantuk: Machine learning models need to be constantly re-evaluated when analysing operational metrics @chris_sanden #qconnewyork
@philip_pfo: Better alert tuning- experts tag anomalies and the system learns and improves. @chris_sanden #qconnewyork http://t.co/B3gxUaAIkU
@allmycode: .@chris_sanden #qconnewyork User may trust your system, but want to understand why a decision was made. Visibility into why is important.
Architecting for Failure
Twitter feedback on this track included:
@philip_pfo: The one who falls & gets up is stronger than the one who never fell Learn about how systems fail & recover this week at #qconnewyork
@danielbryantuk: Key takeaway from #qconnewyork 'architecting for failure' track today is that culture is the most important factor for handling failure
Breaking Bad at Netflix: Building Failure as a Service
Twitter feedback on this session included:
@philip_pfo: When change is a constant in your system then failure is also a constant @deelyle #qconnewyork
@philip_pfo: Failure testing is a form of hormesis- little doses of poison build up immunity @deelyle #qconnewyork
@charleshumble: I start by scoping the failure when doing failure testing in production, starting with a single customer. Me. @deelyle #qconnewyork
@charleshumble: Knowing how the system responds in the face of failure is invaluable because our assumptions are often incorrect. @deelyle #qconnewyork
@danielbryantuk: Very cool to see some failure injection with the FIT tool on the actual live Netflix site! @deelyle at #qconnewyork
@charleshumble: Aggressive failure testing encourages an anti fragility programming culture. @deelyle at #qconnewyork
@SpykeBytes: I break things in production on purpose. @deelyle #qconnewyork #PCTY
@CodeMonkeyShawn: @deelyle I used to think trying to fail in prod was madness. I now think not testing our assumptions is insanity. #qconnewyork @mpmason87
@philip_pfo: If a system fails in prod and no one notices, that's a good thing! @deelyle #qconnewyork http://t.co/eURbz03Fxp
Fail Better: Radical Ideas from the Practice of Cloud Computing
Twitter feedback on this session included:
@philip_pfo: Collective effort always outpaces the lone hero. @yesthattom #qconnewyork http://t.co/CtOYsu47Jh
@philip_pfo: Radical ideas are uncommon not common sense. Embrace failure to fail better. @yesthattom #qconnewyork http://t.co/02urcYvWkV
@zimmermatt: 3 ways to fail better* Cheaper, <reliable hardware* If process is risky, do it a lot* Don't punish ppl 4 outages #qconnewyork @yesthattom
@philip_pfo: Small batches are cheaper and more reliable - release early and often. @yesthattom #qconnewyork http://t.co/57B3ips1As
Making Distributed Data Persistent Services Elastic (Without Losing All Your Data)
by Joe Stein
Twitter feedback on this session included:
@zimmermatt: #mesos is your data center operating system.@allthingshadoop #qconnewyork
@danielbryantuk: You don't have to build a super computer with Mesos, but you can if you want... @allthingshadoop at #qconnewyork
@danielbryantuk: A list of new features for @ApacheMesos that will make constraints and implementing storage easier #qconnewyork http://t.co/UZMFmc79S1
@jon_moore: Here's the "Omega paper" @allthingshadoop just referenced: http://t.co/4RmO4yQGYi #qconnewyork
@jon_moore: Intriguing! > @allthingshadoop: "Once you start using Marathon your need for Puppet or Chef goes to zero" #qconnewyork
Partial Failures in a Microservices Jungle: Survival Tips from Comcast
by Jon Moore
Twitter feedback on this session included:
@mjbrender: “How long are you willing to wait for me to come back?” Bhahaha! @jon_moore walked out of his room to show timeouts. #qconnewyork
@n0r1: @jon_moore @ #QConNewYork: Idempotent operations will save your bacon in the event of network partitions. Preach it!
@philip_pfo: 1/3 Ways to avoid partial failures: Idempotency - repeating with the same input achieves the same outcome @jon_moore #qconnewyork
@philip_pfo: 2/3 Ways to avoid partial failures: Graceful degradation - sometimes you have to let the jungle win@jon_moore #qconnewyork
@philip_pfo: 3/3 Ways to avoid partial failures: It's micro not nano services- recombine some to simplify your life.@jon_moore #qconnewyork
@amunda: Three key ways to handle partial failure in micro services #qconnewyork @jon_moore http://t.co/Dl7JtYUkF6
@n0r1: @jon_moore @ #QConNewYork: Gaps between microservices are each opportunities for partial failures. Possible to overdo “single-purpose” arch.
@philip_pfo: Resilience pattern: draw a criticality boundary and find ways to reduce the critical set.@jon_moore #qconnewyork http://t.co/o67mvtL8Gi
Too Big to Fail: Lessons from Google and healthcare.gov
Jeanne Boyarsky attended this session:
Lessons [from a major network failure]:
- Model ahead of time. Plan for spikes and unexpected events so have spare capacity. Do load testing to know limits and capacity planning.
- Regression analysis. How software and hardware changes over time.
- Network capacity is harder to do capacity planning than web server capacity
- Visualize in real time. Must be able to make decisions on the fly. What effect would X have? Humans can then decide if they *should*.
Lessons [from a satellite cluster failure]:
- Mitigate first.
- Fall back to established protocol – “How to panic effectively”. ICS (Incident Command System) – firefighters have this too. Allows cross-agency collaboration. Centralizes responses.
- When not in firefighting mode, easy to think don’t need
- Created a lighter weight version of ICS. Changed terminology to make sound less hierarchical.
- No penalty for invoking protocol. Err on the side of caution because we don’t know at first of an early warning sign
- Train people before prod support. Give them scenarios
Lessons [from healthcare.gov outage]
- Tech surge team did fix technical things, but it was mostly a cultural problem. Tech surge team brought a sense of urgency
- In five nines (three minutes of downtime a year),the fifth 9 is people. Health care needed to get to the first 9 not the fifth.
- Need a culture of response. Must care if it is down. Monitoring. Reacting when down. Connect engineer’s work to the bottom line.
- Need a culture of responsibility. Must do post mortem or root cause analysis. Need to know how to prevent the following time. Must be blameless
- If people afraid to speak up, you extend outages
- Operational experiences informs system design. Want to avoid getting paged about same thing repeatedly
Twitter feedback on this session included:
@philip_pfo: Resilience to small failures doesn't prevent total outages. (Lesson from the Titanic) @n0r1 #qconnewyork http://t.co/sa8SRi9E0I
@mjbrender: Prevention is ideal, but failure is a fact of life.” ~ @n0r1 #qconnewyork
@jon_moore: Two approaches to reason about your system: (1) modeling; (2) visualize in real time. /via @n0r1 #qconnewyork
@jon_moore: .@n0r1 describing a workflow that interpreted "decommission <empty set>" as "decommission all the things!" #ohnoes #qconnewyork
@jeanneboyarsky: Fall back to established protocol - "How to panic effectively". ICS (incident command system) - firefighters have this too #qconnewyork
@zimmermatt: How to Panic Effectively: Have Incident Command System & lightweight management that fits culture and no penalty for invoking@n0r1 #qconnewyork
@philip_pfo: Handling outages - practice + muscle memory, coordinate efforts, and don't freak out! @n0r1 #qconnewyork http://t.co/WxrmEsSNgK
@jon_moore: .@n0r1: Develop a culture of responsibility. Write (blameless!) post-mortems. Find root causes. Mitigate/improve. #qconnewyork
@camilin87: culture is crucial to scaling & reliability by @n0r1 #qconnewyork
@philip_pfo: Highly available system design incorporates people as a necessary and essential part. @n0r1 #QConNewYork http://t.co/hJ1Z5O3ojO
Architectures You've Always Wondered about
Google Docs - Architecting the World's Largest Real-time Collaboration System and Developer SDK
Jeanne Boyarsky attended this session:
Startup 2WebTechnologies made product XL2Web. …
In 2004/2005, Google approached them and asked what if could drive an emulator through actual web browser client. They made a prototype for IE 6 Service Pack 1 …. It worked and Google acquired the 3-person company. They did server side calculations because they couldn’t figure out how to do everything in JavaScript at the time.
Then, they wondered what would happen if they connected a second web client at the same time to the in memory spreadsheet. Got started with atomic broadcasting (multiple clients receiving events from a single server). Used a hanging GET to the server so the server could always respond to clients. Determined save based on a base state plus a series of mutations until they got to a save point. Then, they realized they needed auto-safe. The side effect was that users could access any revision that ever existed….
Scaling – declare a master server for all writes to each document. If that server goes down, fail over to a different primary. Multiple servers can’t handle non-commutative messages at the same time. Could shard by chapter. … Once your unit of consistency is too large for one server, it’s no longer a unit of consistency.
Twitter feedback on this session included:
@zimmermatt: Micah Lemonik shared that a lot of the great features in Google docs were serendipitous!#qconnewyork
@zimmermatt: gSheets model is commutative, gDocs is not. To bring gSheets collaboration to gDocs they used Operational Transformations.#qconnewyork
@philip_pfo: Mathematical correctness doesn't always achieve what the user expected. UX trumps math. (Micah from Google at #qconnewyork)
@zimmermatt: They use a stateful server for Google Docs and use PAXOS to provide Availability and Commutativity.#qconnewyork
@philip_pfo: When you are building products at scale know your arch constraints. Set a limit of behavior or consistency. Micah from goog #qconnewyork
@zimmermatt: CAP vs Product FeaturesCan't have unlimited collaborators, consistency, AND availability.#qconnewyork http://t.co/F0T2fQILPX
@jeanneboyarsky: Get billions of queries a day. So a one in a billion thing happens 20 times a day -google #qconnewyork
Scaling Stack Overflow: Keeping It Vertical by Obsessing over Performance
Benoit Lafontaine attended this session (excerpt translated from French):
Very interesting session on the Stack Overflow’s architecture, mostly very monolithical based on C# and MSSQL, that corresponds perfectly to their needs (lots of reads, mostly the homepage, little personalized content).
Their architecture is very "simple":
- a load-balancer
- 9 web servers
- a big database
As opposed to lots of people, they deploy this monolith very quickly (~3 minutes), and multiple times per day. Their number of servers is a little over-dimensioned, but that enables them to be cool while doing maintenance, recovering from a server crash, etc.
Their global process to come to such a "boring" architecture is:
- Start from what you know. That's why they have no stack like nodejs, Ruby or Scala
- Measure in live production
- Fix what's slow
Performance is considered a major feature. Their target is to have everything below 30 ms. When it becomes slower, they have a sort of "stop the line", and everything's stops until performance is back to normal again.
Over time, they have externalized some components (like the search in ElasticSearch), or implemented specific components (a new ORM to replace LINQ).
His presentation goes well with the global message from the conference on microservices: start with a monolith and cut where the need starts to arise, a good quote to sum this up: "Extract a service that solves real problems, not imagined ones".
Some of their specifics are interesting:
They have few tests and don't hesitate to test directly in production. Using feature flags, canary releases, combined with their capacity to deliver quickly enables them to quickly fix problems.
Same thing for the performance tests: directly on live production.
Twitter feedback on this session included:
@jon_moore: .@df07: "We view management as a support function." #qconnewyork
@danielbryantuk: An interesting look at the 'boring' architecture of Stackoverflow by @df07 #qconnewyork http://t.co/zgslmoV329
@danielbryantuk: The 'monolith plus' approach of the Stackoverflow team... @df07 #qconnewyork http://t.co/SVH4qotSi3
@danielbryantuk: Building a community allows for a more forgiving user base when rolling out change "we're building this together" #qconnewyork #culture
@jon_moore: .@df07 says Stack Exchange lives or dies by "Performance is a feature" #qconnewyork /cc @sreekotay
@skormos: Start with what you know Stack Overflow used MSSQL's text search "not that good, but got us out the door". Sounds like "MVP"! #qconnewyork
@_chriskucharski: Listening to @df07, at #qconnewyork, talk about scaling and performance tools they have opensourced...look forward to checking them out.
@skormos: StackOverflow's @dc07 talking about debugging performance, one of the items was replacing out-of-the-box ORM. Sounds familiar. #qconnewyork
@jon_moore: .@df07: "Optimize for performance, get scale thrown in." #qconnewyork
@jon_moore: .@df01 suggests we "extract services that solve real problems, not imagined ones" (avoid service-orientation tax). Amen! #qconnewyork
@CGuntur: Start with what you know, measure live and fix the slow. Great talk on architecture at stack overflow. Reads like a poem #qconnewyork!
@danielbryantuk: There is no reason that you can't have a monolith with clear internal interface boundaries... without paying SOA tax @df07 #qconnewyork
@jon_moore: .@martinfowler on microservices, via @df07 at #qconnewyork: "So my primary guideline would be..." http://t.co/C0ph3Ckgaz
@danielbryantuk: Great to see @df07 extolling the virtues of a Monolith with a superb overview of the Stackoverflow approach #qconnewyork
@skormos: Nice to see another technologist say "Don't drink the Kool-aid" for another buzzword tech, this time @dc07 and #microservices #qconnewyork
@danielbryantuk: SOA is not the only way... Great food for thought on the monolith vs microservice debate by @df07 #qconnewyork http://t.co/D9NxwPdfWf
Continuously Deploying Containers in Production
20 Minutes from Ticket to Production. Zero Downtime.
by Paul Payne
Twitter feedback on this session included:
@danielbryantuk: Google never got the Virtual Machine memo... And therefore pressed forward with containers @paulrpayne #qconnewyork http://t.co/fFiFdB9ocT
@fnthawar: @cloudfoundry is ideal for centralized IoT; similar to the @pivotal mobile services @jrdothoughts #qconnewyork http://t.co/6xsi9HIFJu
@danielbryantuk: Reactions to zero-downtime and container-based deployments at Nordstrom by @paulrpayne #qconnewyork http://t.co/N3veUEffwm
@danielbryantuk: Anti-corruption Layers within a microservice architecture... #qconnewyork http://t.co/wYkmJR3Npf
Easier, Better, Faster, Safer Deployment with Docker and Immutable Containers
Jeanne Boyarsky attended this session:
We upgrade by creating a new server and deploy to it. Keep old server around just in case.
We do this to avoid configuration drift between “identical” servers. Caused by provisioning servers at different time or manual operations….
Rolling back is hard because of transitive dependencies. With immutable servers you have old versions of everything so rolling back is easy – switch to the old server. …
How to debug
Workarounds for dealing with debugging tools
- Allow drift but tag as “re-imaging” and schedule for self destruct. This lets you install debug tools or whatever you need.
- Install debug tools on all servers. Have feature switch to enable/disable…
Containers
You can build them from scratch or incremental. Use a simple DSL to break down the build into steps.
A Docker file is a glorified shell script. (looks like DOS it has commands like RUN git clone repo)
Docker recognizes what parts haven’t changed so it doesn’t re-install all the infrastructure for an incremental change such as new app code. But it still gets the benefit of two copies; the old one is still around.
Containers can share directories, logs, backups, metrics, etc. since they are on the same machine.
You can make containers read-only to enforce immutability. Easier security because they can’t install anything. You can make an exception for data volumes and prevent execution in that area.
Here Be Dragons: Security Maps of the Container New World
by Josh Bregman
Jeanne Boyarsky attended this session:
Risks
- Escalation of privileges
- Image integrity
- Unpatched containers
- Any others – see Docker presentation
Need to prevent
- “good” containers calling you accidentally
- “good” containers calling you without your permission
- “bad” containers calling you …
DevOps is about velocity. Security and Risk Management can put on the brakes.
Orchestrating Containers with Terraform and Consul
Twitter feedback on this session included:
@danielbryantuk: The container dream of abstracting the deployment fabric breaks a lot of the assumptions with modern infra provision and config #qconnewyork
@danielbryantuk: Living with legacy in the new container world... And will there be a new disruptive tech? @mitchellh #qconnewyork http://t.co/8jRUqZ6YX8
@danielbryantuk: Current infra provisioning tools, such as Puppet and Chef, are not built to run on container deployment timescales @mitchellh #qconnewyork
@danielbryantuk: Terraform use cases for infrastructure provisioning in cross-vendor datacenters... @mitchellh #qconnewyork http://t.co/Hjm2R4B0v6
@GreatWebGuy: It's too easy to launch infrastructure around your Ops team, if you fight it you're gonna lose @mitchellh #qconnewyork
@danielbryantuk: The killer feature of Terraform is still the infrastructure provisioning planning/preview feature... #qconnewyork http://t.co/ZQ01nG5rKw
@mjbrender: I’m realizing that Terraform from @Hashicorp will be really exciting to my hybrid cloud infrastructure friends. #qconnewyork
@danielbryantuk: Don't let people become the bottleneck when provisioning infrastructure "geniuses tend not to scale..." @mitchellh #qconnewyork
@danielbryantuk: The next version of @hashicorp Consul will support ACLs for a lot more things, such as restricting service discovery @mitchellh #qconnewyork
@jon_moore: .@mitchellh: "Software continues to come out assuming you're running in a single DC [but we're in a multi-DC world]" PREACH. #qconnewyork
@danielbryantuk: Events, execs and watches (& distributed locks) - primitives for orchestration within @hashicorp Consul #qconnewyork http://t.co/0csJn6kuOe
Paasta: Application Delivery at Yelp
by Evan Krall
Twitter feedback on this session included:
@danielbryantuk: Paasta is @YelpEngineering SOA glue... @meatmanek #qconnewyork http://t.co/mH42nQ34qe
@danielbryantuk: The driving forces behind Yelp's Paasta - @ApacheMesos and @mesosphere's Marathon running @docker #qconnewyork http://t.co/5T870nmWCo
@danielbryantuk: Alerting apps like Nagios may be terrible, but we're very familiar with how they're terrible. Newer tool are less battle tested #qconnewyork
@danielbryantuk: .@YelpEngineering's Paasta has no API, and instead the control plane is Git... @meatmanek at #qconnewyork http://t.co/WnsN5tt0yR
Emerging Technologies in Front-end Development
New in ECMAScript 2016, JavaScript's First Yearly Release
Jeanne Boyarsky attended this session:
Functions
- return a single value synchronously with functions
- return many values synchronously with generator functions
- return a single value asynchronously with promises
- return many values asynchronously with observables…
Math.pow(10,2) becomes 10 ** 2
SIMD (single instruction multiple data)
Decorators
Value types
- New primitive types – Int64, Bignum, Decimal, Complex, Rational,SIMD types, TypedArray types
- Factory let int8 = Int8(254)
- Literal suffixes let i64= 0L; let bigNum b = 0n; etc ;
- Primitives are better for serializing across frames
- Still immutable like other primitives
- === compares by value
- Custom primitive types – export let Yard – ValueType(Symbol(“Yard”), Float64)
- Can overload operators on new custom primitives: Reflect.defineOperator(‘+’, f, Yard,Yard)
Twitter feedback on this session included:
@emerleite: New things that are planned to JavaScript #simd #QConNewYork http://t.co/klwEAcTlp2
@emerleite: Finally JavaScript will have decorators #QConNewYork #fb http://t.co/k8g7YOp3zs
@SpykeBytes: Just saw the awesome parts of ECMA2016. Observables, value types, memoization, async and await, and more. From @bterlson #qconnewyork #PCTY
@yfain: Watched the presentation in ES7 at #qconnewyork. #JavaScript goes strong. Half of the ES7 features are already implemented in #dartlang.
Engineer Your Culture
Dodge Disasters and March to Triumph as a Mentor
Jeanne Boyarsky attended this session:
Why to be a good mentor?
- Mandatory for career advancement
- Needed for their careers too. Having a mentor is often the difference in staying in the industry
- Your company needs you to mentor junior engineers because they need senior engineers. Who are hard to hire
- Our industry’s future. The next generation is trained by us, not by university …
Lessons learned by speaker
- Be eager to help. Or at least appear to be. If you look like you are angry when someone asks for help, they will be afraid to ask. Need to seem excited to be interrupted. (or preferably actual be). You don’t want your apprentice to be blocked. He/she isn’t learning anything and you can’t evolve. We overemphasize figuring things out on your own.
- Know what your apprentice doesn’t know. ex: github, pm, working on one thing at a time. Try to anticipate these questions….
- Admit when you don’t know. It models that it is ok to not know things and not be embarrassed when you don’t know. Also it helps teach problem solving.
- Ask an apprentice for help. He can do code review even if he doesn’t find much/anything. Still learning. Still creating good patterns.
Excuses for why not to mentor
- “I’m an introvert” – controlled interactions are easier. Predictable. An apprentice is likely to be an introvert too so understand each other. Introverts listen. Can develop a relationship.
- “I can’t explain what I know” – you can teach by working together; you don’t have to explain to teach. If it can be explained, it probably is in a book somewhere.
- “I’m a bad mentor” – It is a skill that improves with time/practice.
- “I just want to code” – Some people don’t like leading young people. … Making the investment can change how you think of yourself. It is a satisfying feeling when it works.
Twitter feedback on this session included:
@jeanneboyarsky: mentoring works best when you like your mentee. i concur! more trust; more thought; more feedback #qconnewyork
@NativeWired: What changed the time it went well: Expert mentor, small clear goals, guiding visionary present, simpatico.@jessejiryudavis #qconnewyork
@NativeWired: Don't underestimate actually liking each other @jessejiryudavis #qconnewyork
@NativeWired: Lessons learned: Know what your apprentice doesn't know.Admit what you don't know.Teach problem solving @jessejiryudavis #qconnewyork
@NativeWired: We need to mentor to respect the history of craftmanship.This is how our craft survives.@jessejiryudavis #qconnewyork
@NativeWired: Apprentices are investments. There is a lack of senior engineers, so we need to grow them ourselves @jessejiryudavis #qconnewyork
@n0r1: @jessejiryudavis’ response to “I still don’t care abt mentoring; I just want to code”: Our craft survives via 1:1 instruction. #QConNewYork
Lessons from Etsy: the Secrets to a Successful Remote Culture
Jeanne Boyarsky attended this session:
Number one factor for success is critical mass. Having one remote on the team doesn’t work. Having enough makes communication happen in a remote friendly format. Using chat/email/video conferencing rather than in person/physical whiteboards.
Communication
- Chat – like IRC or Slack. They use channels; not just one on one chat like Sametime or Lync. Have #remotes channel. Virtual water cooler.
- Shorter, more frequent interactions build stronger bonds than longer, less frequent ones.
- Etsy is a “reply all” email culture. Use ignore/mute feature so not having to read all.
- A/V – 4 full timers work on A/V. Google hangout wasn’t enough. Switched to Vidio. Remotes type in name of room to join video conference. Remotes never late to meetings so a remote showing on screen reminds the previous meeting to end
- Make it easy. Don’t want resistance from on sites to including remotes. …
Policies
- Remotes can visit any time they want and company pays for. He visits quarterly. Goal: appreciation.
- Local employees aren’t free. Company pays for desk. Should pay for remote to have good space too.
- Responsive IT group.
- Mailed hoodies in advance so everyone got on same day.
- Be mindful of decisions and how they affect remotes. Ex: Friday afternoon beers exclude remotes.
Advice for remotes
- Visit at least once a quarter. Socialize when there. Visiting to talk to people; not to sit in corner and code
- Make sure have proper work environment at home or find a co-working space
- Some people need a dedicated space for work at home
- Don’t forget to go “home” at end of work day. If start at 7am to sync with East Coast, don’t feel bad about ending at 3pm.
- At disadvantage in being heard/seen, so put extra effort into being noticed.
- Support each other; talk to other remotes
- Mixer app – created app to randomly pair people and suggest they talk
Twitter feedback on this session included:
@rodhilton: Open office plans suck for local AND remote employees. Discourages ad-hoc video/voice chat with remotes. #QConNewYork
@n0r1: A good reminder from @bgreenlee not to be discouraged when visits to the mothership aren’t recognizably productive #QConNewYork
Nailing Engineer Retention
by Erran Berger
Jeanne Boyarsky attended this session:
Make talent your #1 priority.
#1 – on day 1, make team a process
#2 – maximize employee engagement
#3 – talk about recognition
- People need to know they are seen and valued – don’t want to feel invisible
- People stay where seen/appreciated
- Receiving feedback is difficult for most people – even if they say “no big deal” still means something
- Needs to be specific, timely and personal. Not just “good job guys”
#4 - be prepared; by the time a person comes to management about leaving it is too late
- Observe – behavior change
- Assess – personal and work related interests. Reasons for engineers to become unhappy include- FUD, friends leaving, lack of opportunity, nobody is listening (never solve problems), lack of recognition.
- Connect – try to discuss/open up. Leave the office; go for a walk. People more comfortable opening up when elsewhere. The goal is to find out root cause. They might not know or be able to articulate it. Don’t argue about how wrong they are about what they are perceiving. Be emotionally supportive. Try to buy time to solve the actual problem.
- Act with urgency – if not highest priority, is it worth retaining the person.
Twitter feedback on this session included:
@NativeWired: How to make talent your number one priority 1. Make them a promise 2. Maximize engineer engagement 3. Be prepared @eberger45 #qconnewyork
@NativeWired: Talk to your people about what they want for the future. Even if it will not be in your company @eberger45 #qconnewyork
@NativeWired: Engage your engineers: Inspire, challenge, recognize#eberger45 #qconnewyork
@NativeWired: Giving sincere feedback. Make it personal, timely, specific..@eberger45 #qconnewyork
Fraud Detection and Hack Prevention
Fighting Fraud, Scammers, and Hackers on the Blockchain
Jeanne Boyarsky attended this session:
Primary risk isn’t with the protocol; it’s with the security on the websites where trading takes place.
Hackers love Bitcoin because they have cash if they can get into it….
Assumptions:
- Passwords will leak
- Emails are compromised
- Users will be phished
- Computers will be left open and unlocked
- There will be social engineering
How address them:
- 2-factor on everything. Login, sending money, changing things. Send SMS message when sending money to communicate on a separate channel.
- Rate limit to minimize danger
- Device verification required. Must authorize to continue
- Added five minute delay on transferring money after changing password to avoid using that token to transfer money
- Optional vault with extra security features – time delayed withdrawals, alerts to two verified emails and confirm from both, SMS notifications, cancel at any time, banner reminders to enable,option for M-of-N management (3/5 people must authenticate)
- Train support really well on social engineering
- Multi-sig vault – only for technical users, key splitting architecture. cold storage as a service. Three keys – user key, coinbase key and shared key that is an encrypted key with a password that only the user knows. Need two of three keys to get access to Bitcoin.
Twitter feedback on this session included:
@jeanneboyarsky: Credit card is like private key. Every time you use your credit card, it's like handing over your private key. #qconnewyork
@jeanneboyarsky: cold storage as a service - need two of three keys #qconnewyork http://t.co/XiTxRpJXNO
@jeanneboyarsky: awesome! coinbase uses chromebooks for support staff so physically can't open a virus laden zip file. great chromebook use #qconnewyork
Real Threat and Real Defenses – Case Study of the Unknown
by Alex Holden
Jeanne Boyarsky attended this session:
Target breach
Started almost a year before actual breach. Learned from bad experience with Black POS with Verifone POS attempted breach in Russia. Then planned for several months. A week before Black Friday did large trial run spending $40 million of their own money. …
Defense 101
Credit card theft on decline because credit card companies see patterns faster and act. For example, Chase flagged Home Depot users quickly and hackers stopped buying the numbers.
Need to understand patterns.
Biggest vectors – viruses, zero day vulnerabilities (Heartbleed, Shellshock), stolen/reused credentials
Quantitative analysis - how much of data is transferred? What is normal? Learn to look at stats. If you usually transfer a 20KB on a page, set a limit of 100KB to know you have a breach.
Honeypot – using a SQL Injection flaw as honeypot in your system allows detecting hacks, rather than a separate system. The idea is that hackers will think it is real and spend time on it. A zero day attack might be revealed via a honeypot. Install an early warning system to alert you. Having fake data that can be identified as yours but looks normal. Not mymail+hack@gmail.com. Hackers know about the + technique.
Auditors can get you in trouble with your boss. Hackers can get your company in trouble and end your career.
Assuming you have been breached already. Look for your data online. Look for a unique identifier. Search for “/logout” on your company’s site to see if spidered and people can use Google cache to navigate….
Treat security issues as risks; not just bugs/defects. A risk has to be addressed or mitigated. A bug can drag on for years.
Twitter feedback on this session included:
@jeanneboyarsky: We use different keys for our house/car/etc, but use same passwords repeatedly. #qconnewyork
@jeanneboyarsky: Learn to look at stats. If usually transfer 20KB on a page, set a limit of 100KB so know have a breach. #qconnewyork
@jeanneboyarsky: like the idea of building an intentionally vulnerable fake feature as early warning about hackers #qconnewyork
@jeanneboyarsky: Auditors can get you in trouble with your boss. Hackers can get your company in trouble and end your career. #qconnewyork
@SpykeBytes: Learn to analyze data flow statistics to detect breaches. Utilize honey pots. Know what hackers want. From Alex Holden. #qconnewyork #PCTY
@jeanneboyarsky: 7% of people in US ages 16+ had credit card/bank acct misused in 2012. single password security. shame on banks w/o 2 factor #qconnewyork
Storing Production Secrets at Scale
by Armon Dadgar
Twitter feedback on this session included:
@danielbryantuk: What's your break glass procedure for security issues? In many cases it's undefined... @armon at #qconnewyork http://t.co/dxSFIya0P6
@danielbryantuk: Hashicorp's Vault can also act as an internal Certificate Authority too... (useful for mutual TLS etc) @armon #qconnewyork
High Performance Streaming Data
Akka Streams: Streaming Data Transformation r la Carte
by Viktor Klang
Jeanne Boyarsky attended this session:
Akka Streams
- Higher order abstraction for concurrency
- Solving the 100% case is impossible and 90% case is hard. Akka focuses on 80% case. Solve most problems
- Immutable, reusable, composable, coordinated, async transformations.
- Flows – one input and one output. Not connected to source or destination. Like a reusable pipe. Goal is to describe the transformation.
Types of transformations
- Linear Time-agnostic –ex: common ones like map
- Linear Time sensitive – ex: dealing with infinite streams
- Linear Rate detached – ex: how deal with buffering
- Linear Async – ex: calling a service
- Non-linear – ex: binary flow (two inputs going to two outputs), custom stages
Sources – publisher, iterator, future, etc
Sinks – subscriber, foreach/fold/onComplete, ignore, head, etc
Fan in/fan out – merge, zip, etc
Output/Input – Http, tcp, InputStreamSource/OutputStreamSink blocking wrappers, etc. …
Push vs pull
- Want to get data across async boundary without blocking back-pressure.
- When pushing data, have to drop if get it too fast.
- “A slow solution is no solution”
- Pull can be slow
- Better: Two way channel – only push when you know you are ready. Know what is needed based on requests. Don’t demand 100 ice creams unless you know what to do with them. Switching between push/pull and batching requests when you know about related ones. Called “dynamic push pull”.
Mobile and IoT at Scale
Distributing a Mobile Team: a Brave New Etsy Chapter
Twitter feedback on this session included:
@fnthawar: We deploy to the web 50x a day, but can't for mobile apps. So we release every 2 weeks @hannahmitt #qconnewyork
@fnthawar: We are all the mobile team #qconnewyork @hannahmitt @Etsy
@fnthawar: You should feel like we can cut the master branch for our mobile apps into prod at any moment @hannahmitt @Etsy #qconnewyork
@fnthawar: Giving unsolicited advice feels weird at first, but can help build great products with your feedback @hannahmitt @Etsy #qconnewyork
@fnthawar: Getting really good at a new thing sucks @hannahmitt @Etsy #qconnewyork
Mobile-first Architectures
Twitter feedback on this session included:
@allmycode: Alexander Stigsen #qconnewyork says do as much of the computing on the mobile device as possible.
@fnthawar: Alexander Stigsen: Endpoint Computing: move computing to the device (it's free!) #qconnewyork @realm
@fnthawar: Alexander Stigsen: Mobile first should mean mobile *device* first design #qconnewyork @realm
@fnthawar: Alexander Stigsen: The web is something annoying you have to use when you don't have an app #qconnewyork @realm
Powering the Industrial Enterprise: Introducing the Iot Platform as a Service
Twitter feedback on this session included:
@fnthawar: IoT requires a new type of platform @jrdothoughts #qconnewyork
@fnthawar: The IoT world is fraught with protocols, devices, frameworks @jrdothoughts #qconnewyork
Results Driven Development & Pushing the Boundaries of Mobile
by Aaron Glazer
Jeanne Boyarsky attended this session:
Building a mobile app is like a Formula One car. Someone else creates the rules. People care about how you perform, not the internals. Results driven means working with all areas (sales, marketing, tech, etc.) to achieve a common goal.
Data on it’s own isn’t useful. Needs focus. Clarity through simplicity. Simplicity alone is not enough….
Do A/B testing to focus on causation instead of correlation…. A/B testing more important on mobile because there are less opportunities to hook user.
A/B Testing Walkthrough
Know your goal.
Setup distribution. 50% baseline, 50% variation.
Segmentation: only show to users meeting target audience.
Results Driven Development
- Everyone must work together – Isolating the mobile team is bad. The engineering team controls app, but not accountable for user retention and other business goals. In results driven, have a cross functional team. Center the team around checkout flow, not the platform.
- Get the right tool for the job
- Ensure accountability is directed properly
- Data gives you information, but needs a goal. Results give you answers.
- Choose a contextual business metric. Hypothesize/test/improve.
Twitter feedback on this session included:
@jeanneboyarsky: Building mobile app like Formula One car. Someone else creates the rules. People care about how perform, not the internals. #qconnewyork
@tinkadoic: In mobile apps, no one cares about what's under the hood, like in formula1. They only care about results! @aaronglazer #qconnewyork
@fnthawar: Too much data = no focus @aaronglazer @taplytics #qconnewyork http://t.co/ZvUju2fJJt
@fnthawar: A/B testing is a way help you find causation from correlation by holding the externalities constant @aaronglazer @taplytics #qconnewyork
@fnthawar: Mobile web is a backup plan @aaronglazer @taplytics #qconnewyork
@fnthawar: Desktop to mobile to watch:each change makes a/b testing 10x more imp. since you have 1/10th the real estate @aaronglazer #qconnewyork
@fnthawar: .@FrankandOak treats everything as an experiment before implementing any new feature" @aaronglazer #qconnewyork
Modern Computer Science in the Real World
A Pragmatic Introduction to Multicore Synchronization
by Samy Bahra
Twitter feedback on this session included:
@charleshumble: Locks are not composable and are susceptible to inversion, livelock, starvation, deadlock etc. Samy Bahra #qconnewyork
@charleshumble: Lock-freedom provides system-wide progress guarantees. Samy Bahra #qconnewyork
Consensus Systems for the Skeptical Architect
Twitter feedback on this session included:
@charleshumble: .@skamille citing the Google Chubby paper at #qconnewyork http://t.co/66kRETuEBI
@jon_moore: .@skamille: DNS is the original service discovery mechanism. #qconnewyork
@zimmermatt: 3 areas of eval to assess if you need a consensus systemWhere is it running?What is it doing?What are we running?@skamille #qconnewyork
@zimmermatt: Service Management Alternatives* load balancer* DNS* database (not highly recommended)@skamille #qconnewyork
@jon_moore: .@skamille: ZooKeeper is written in old crufty Java...but it's production-hardened old crufty Java. +1 :) #qconnewyork
@mjbrender: The algorithms are proven but they're implemented within systems. And systems break in the wild. Great takeaway from @skamille #qconnewyork
@philip_pfo: .@skamille: The bugs will be in the stuff no one uses / the things that happen rarely. #qconnewyork #pioneertax
@charleshumble: Consensus owns your availability if using Zookeeper or other consensus system. @skamille #qconnewyork
Evolving Prolog
Twitter feedback on this session included:
@allmycode: “Harder to code problem into genetics than to just solve the problem without genetic programming” #qconnewyork @mndrix
@allmycode: homoiconic language = a language in which the code and a data structure are the same thing #qconnewyork @mndrix — So, you can *run the data*
@allmycode: Genetic algorithm is trivially parallelizable #qconnewyork @mndrix
Modern Advances in Java Technology
Developing Functional Domain Models with Event Sourcing
Jeanne Boyarsky attended this session:
Core Patterns
- Monolithic architecture – traditional way of building an app, simple to develop/test/deploy/scale. Successful apps grow. Then have millions of line of code in one war file and beyond comprehension. No longer agile. Much fear. Infrequent deployments. Overloads the IDE and container. Can’t scale. Too much coordination and communication required just to deploy. Works really well on small apps because can move incredibly fast.
- Microservice architecture – apply functional decomposition – scale by splitting similar things and/or different things and/or horizontal duplication. Amazon needs 100 microservices to render a page. Smaller/simpler apps. Less jars/classpath hell. Easily and safely experiment with different techs. Opposite of release train. Low risk because small. Drawbacks: more complexity, inter-process communication, partial failure
Communication Patterns
What does the client talk to when services are distributed? API gateway is common to do request routing and API aggregation.
How do services with system communicate? Synchronous or messaging API?
Deployment Patterns
- Multiple Services per host – older approach. Like when servers were like pets and kept them alive forever for efficient resource utilization, fast deployment, poor isolation, risk dependency, version conflicts, can’t constrain resource use. Some benefits, but lots of downsides.
- Single service per VM host – Netflix does this- great isolation/manageability, detailed metrics on resources using, resources constrained, less efficient resource utilization, slower deployment because it requires creating new VMs.
- Service per container host – like Docker images, great isolation/manageability, container encapsulations implementation tech, efficient resource utilization because it does not include the virtualizing OS. Fast deployment. Immature infrastructure for deploying containers
Java 8 in Anger
by Trisha Gee
Jeanne Boyarsky attended this session:
The Java 8 features less known:
- Stub Service uses a Supplier for generating random numbers.
- Compute if absent and method reference:
- map.computeIfAbsent("key", TwitterUser::new);
- Streams, comparators
- map.values().stream()
- .sorted(Comparators.comparing(TwitterUser::getTweets)
- .reversed()).limit(10).collect(Collectors.toList();
- JavaFX
- laterPlatform.runLater(() -> items.setAll(x));
- JavaFX comes with built in animation
- Date/time to get current minute
- LocalDateTime.now().getMinute();
- Loop from x to y
- IntStream.range(0, 10).forEach(this::method);
This could be better or worse than a regular loop in terms of readability or performance.
- Read fileStream
- lines = Files.lines(path);
- Filter:
- lines.filter(s -> !s.equals("Ok"));
- Check on stream:
- lines.peek(s -> method())
- Convert array to stream, map
- functionStream.of(arr).map(String:toLowerCase)
- Join strings with delimiter in between
stream.collection(joining(",")
Twitter feedback on this session included:
@charleshumble: Connecting to Twitter took longer than the rest of application put together @trisha_gee #qconnewyork
Minitalks
Rod Hilton attended the minitalks:
I wish QCon had done a mini-talks session for every track, rather than just the tracks on the first day of the conference, because the mini-talks were great. Again, the video recordings were instrumental here, it was extremely tough to decide which mini-talks to attend, and with the help of video I was able to attend all of them.
Jeanne Boyarsky attended this session:
Deterministic testing in a non-deterministic world
- determinism – everything has a cause and effect
- pseudorandom – algorithm that generates approximately random
Should see LocalDateTime with Clock instance to reproduce results in a program. There is a fixed clock so all operations in program see the exam exact time. Similar to using a random seed for generating numbers.
Hash spreads and probe functions: a choice of performance and consistency
primitive collections faster/less memory than boxed implementations. Uses 56 bytes for each Integer.
- hash spread – function that destroys simple patterns in input data while presuming maximal info about input. Goal is to avoid collisions without spending too much time hashing.
- hash probe – function to determining order of slots that span array. For example a linear probe goes down one slot if collision. A quadratic probe goes down down further if collision
Typesafe config on steroids
Property files are hard to scale. Apache Commons adds typing, but still limits to property file format and limit composition. Spring helps with scaling property file.
Typesafe Config – library used by Play and Akka Standalone project without dependencies so can use in Java. JSON like format called HOCON (human optimized config object notation)
Scopes – library built on top of Typesafe config
Real time distributed event driven computing at Credit Suisse
Credit Suisse produced their own language that they call “data algebra”. Looks like a DSL.
Monoliths to Microservices
Microservices and the Art of Taming the Dependency Hell Monster
Benoit Lafontaine attended this session (excerpt translated from French):
They had developed on a monolithic architecture and that was a good idea: "monolith first, microservices second". Today they have more than 160 microservices. He showed the details of a few services to highlight a few points:
1. Their services are approximately between 5 and 20 methods, without any strict rule
2. Each service is written in a different language (Scala, Java, Ruby) and uses a different storage (HBase, MongoDB, PostgreSQL), depending on the needed performance baseline, and also the year when the service was first implemented.
His advice:
1. The API design must be "first class", that means treated with special care. That also means that it needs to be an "upfront" design. For these reasons, it can be good to use protobuf, thrift, avro or their own tool apidoc.me. But the upfront design must also be in relation with two other points:
a) "monolith first, microservices second", so when you feel the need to detach a microservice, you already have a good idea of how it would work, and you spend some time to think a bit more on the resource schema
b) the API lifecycle: when you're in 0.x, you do what you want, you break stuff all the time. When you're in 1.x, you commit to no breaking change because otherwise you switch to 2.x.
2. API must be coherent system-wide
At first, everyone used to write API in the most intelligent fashion. They got to a point where they had tons of different ways.
Nowadays, they prefer to have coherent API to smart designs for every little case. But looking at the example he gave, we can also say that today they respect more the REST "resource" philosophy.
3. Keep backward AND forward compatibility. That means that a piece of code written on the first version must be able to run with today's data, and that a program written today must be able to run on the first version's data. That also means that no attribute must be removed or deleted, that every new attribute must be either optional or with a default value. You need to remember that before 1.x, you do what you want. …
4. Generate the client libraries... And they need to have zero dependencies. Again, very interesting. That means no dependencies problems. You can have any client version of two services, that will always work. Yes, the developers that at least once used a SOAP client with JAXB on WebSphere know exactly that that means.
Twitter feedback on this session included:
@danielbryantuk: Interesting tour of @gilttech services by @mbryzek at #qconnewyork. The variety of data stores in use is interesting! http://t.co/moWYIw0FI1
@danielbryantuk: The @gilttech guiding software development principle - the open source way... @mbryzek at #qconnewyork http://t.co/bZEwRZOAFC
@danielbryantuk: API design must be first class. In theory a microservice can be thrown away and rebuilt if the contract is good enough... #qconnewyork
@zimmermatt: Design with Forward Compatibility:blowing up if code sees a field not expected, you prevent evolution of a service. @mbryzek #qconnewyork
@zimmermatt: Generating a client. @mbryzek was skeptical at first, but now sees it works. #qconnewyork
Netflix’s Viewing Data Microservices: How We Know Where You Are in House of Cards
by Matt Zimmer
Benoit Lafontaine attended this session (excerpt translated from French):
Matt Zimmer works on the service that tracks what everyone's watching, when, and on which device.
The architecture started out as a monolith, with lots of stateful things. And it's not a problem: it works fine. It scales well... And it could have scaled a few years more like this. So why change?
Even if it scales, it will probably hit some limits, in 3-5-10 years, they don't know exactly, but looking at their stats on the growing number of users and the consumption per user, it can come quicker than expected. So it should be done while there's still time to be able to do it without the rush.
But one of the main reasons is to optimize costs. He showed a stat that indicates the number of active EC2 instances, the stat is flat, they do not use any AWS auto-scaling.
Their migration, that will end in a few weeks time, will have taken a bit more than 18 months.
To minimize risks (cf. the Poppendieck talk), they ran their version 4 in parallel of the v3 and did shadow testing: every request that goes to the production servers is duplicated to the new system and their responses are compared to see if the new system is correct. They also put in place a circuit-breaker to avoid having an impact on users if the new system becomes too slow.
While migrating, they discovered a quite important bug, where the fact that they had an event-based system enabled them to quickly repair by replaying the logs. So their second big advice is that having an event-based system and being able to replay everything since the beginning, has an enormous value in case of a crash.
Twitter feedback on this session included:
@allmycode: .@zimmermatt #qconnewyork Netflix accounts for about 37% of downstream bandwidth in North American countries.
@philip_pfo: Monoliths aren't a bad thing - they are a good starting point. @zimmermatt #qconnewyork
@allmycode: .@zimmermatt #qconnewyork says “System architectures are a throw-away artifact."
@SrikarDoddi: 'Devour the whale a bite at a time' #microservices #qconnewyork http://t.co/tWp1orU2zj
Uberrush and Rebuilding Uber's Dispatching Platform
by Jeff Wolski
Twitter feedback on this session included:
@mjbrender: “Hope is not a strategy at Uber” and "If you’re afraid to kill it, it’s not production ready”Awesome talk by Jeff. #qconnewyork
@philip_pfo: Uber next gen routing- looks Dynamo-inspired for fault tolerant discovery/routing layer. Jeff Wolski #qconnewyork http://t.co/JzcsXvxOfd
Optimizing Yourself
Developing Cultural Intelligence
Jeanne Boyarsky attended this session:
Culture matters because:
- shapes how people work together
- makes companies great
- to blame when people not doing right things anymore
- powerful tool for new solutions
People need to be empowered and responsible. Nobody should be allowed to treat others badly. Culture in tech is creating shared expectations for behavior. When something bad happens, does someone say it is not ok? If not, it becomes ok. Culture is the accumulation of lessons like this.
There is no manual on culture. So learn to recognize, reason about and affect culture around you. In time, you gain the confidence to develop a culture.
Conway’s law – structure of the architecture comes to reflect the organization of the company.
Rules not written down, but we all know about the culture where we work….
Leaders set the culture whether good or bad. If you set the culture, you became the leader.
Twitter feedback on this session included:
@jeanneboyarsky: when something bad happens, does someone say not ok. if not, it becomes ok. culture is the accumulation of lessons like this #qconnewyork
Stress and Depression – the Taboo and What We Can Do about It
Twitter feedback on this session included:
@skormos: We schedule times in our calendar for *not* working, we never set aside time for planning to prepare for events. @NativeWired #qconnewyork
@skormos: Relax your brain: Meditation. Sit in a chair, close eyes, & be aware of how your body touches the chair, the floor #qconnewyork @nativewired
@charleshumble: .@NativeWired talking about the black dog of depression #qconnewyork https://t.co/HgXrp4XIk0
@skormos: Exercise can be counter-productive for depression: if you don't get to the gym, feel like a failure, & sink further. #qconnewyork @nativewired
@skormos: For exercise, it's enough to get out and walk. #qconnewyork @nativewired
@charleshumble: Things not to say to someone with depression:1. Pull yourself together #2. Other people are worse off than you. @NativeWired #qconnewyork
@charleshumble: Things not to say to someone with depression:3. You have no reason to be depressed 4. It's all in your head.@NativeWired #qconnewyork
@charleshumble: We know it's all in our head. The head of a depressed person can be a very scary place. @NativeWired #qconnewyork
@skormos: Grow your ears and open your heart: learn to listen, and not be an engineer trying to fix things for people. #qconnewyork @NativeWired
@charleshumble: Love that the Winnie-the-Pooh stories have a depressed donkey. They include him, and don't try and change him.Via @NativeWired #qconnewyork
@skormos: Hugs are not strictly physical. It's about dedicating your time solely to that person for that moment. #qconnewyork @NativeWired
Reactive Architecture Tactics
Becoming Reactive Without Overreacting
by Pavlo Baron
Jeanne Boyarsky attended this session:
Context is reactive – data received, processed and delivered. Lazy logic. Ready to be called. We’ve been using reactive without noticing.Examples, UNIX pipeline, OS kernel, app container.
Benefits
- Responsive to change. Just have to register/subscribe. Easier to experiment with features
- Resource efficiency. Stay lazy until there is work to do
- Fight or flight – react to changes of behavior
Overcome temptation to go full reactive. For example, you can change the way you are notified about events. Also, don’t go reactive when it doesn’t apply. For example, CRUD for predictable load. Don’t write your own reactive framework. …
Reactive programming can be functional, but doesn’t have to be. Embrace functional programming for laziness, composition and side effect freedom.
Twitter feedback on this session included:
@hbrumleve: Reactive: minimal contracts and expectations between components. @pavlobaron #qconny #qconnewyork http://t.co/KHalff8Z5h
@zimmermatt: Don't buy the Reactive sales pitch..."Complexity will go away by just ignoring it."@pavlobaron #qconnewyork
@zimmermatt: Build on Streams and their Topology to bring explicitness into #reactive@pavlobaron #qconnewyork
@zimmermatt: You need to be prepared for partial processing.Don't bind, instead parse and compensate, ignore, or alert.@pavlobaron #qconnewyork
@zimmermatt: You don't need to adopt #reactive wholesale. Incrementally adopt bits as they make sense for your system.@pavlobaron #qconnewyork
Managing Complexity – Functionally
by Ryan Trinkle
Twitter feedback on this session included:
@skormos: Not having side effects is great for managing complexity #qconnewyork
@skormos: Reactive programming solves the problem of side-effects from callbacks in functional programs. #qconnewyork
@SpykeBytes: Reduce complexity and manage it with reactive functional tactics. Side effect free & input dependent. From Ryan Trinkle #qconnewyork #PCTY
The Art of Software Design
Design vs. Data: Enemies or Friends?
Twitter feedback on this session included:
@charleshumble: Learning through ongoing experimentation is not an excuse for sloppy system design. Mary Poppendieck, #qconnewyork
@danielbryantuk: Get inside the data, like you would the code... or a good book @mpoppendieck at #qconnewyork on analysing data from experiments
@danielbryantuk: A design vision from @mpoppendieck at #qconnewyork on how to work with data http://t.co/F4eVjMkbrD
@danielbryantuk: You must have adaptable business systems and processes, ready and able to use data insights and services @mpoppendieck at #qconnewyork
@jon_moore: via @mpoppendieck : "Real architects spend a lot of time at the building site." #qconnewyork
@jon_moore: . and also: @mpoppendieck: "If you're going to be an architect, you'd better know how to build." #qconnewyork
@fabienjallot: What's the link between Notre Dame & Software Design ? Incremental design that keeps "wholeness" over time. @mpoppendieck #qconnewyork
Maneuverable Architecture
Twitter feedback on this session included:
@fabienjallot: So, maneuverability is the new agility ? @mtnygard #qconnewyork #agile #lean #maneuverability
@DataMiller: On technology choice commandments: "of more value is: 'what is our canonical representation of an item?'" @mtnygard #qconnewyork
@DataMiller: Every [unique set of needs, culminating in a ] script gets a URL. You're not going to run out of them! @mtnygard #qconnewyork
@synodinos: @mtnygard building the case for #maneuverability #disposability and pointing out common arch fallacies #qconnewyork http://t.co/pGb6WuAcxG
@charleshumble: The date/time calculation is the hardest thing in this slide. It's a nightmare. @mtnygard #qconnewyork http://t.co/ZPkOIR5o3b
@GreatWebGuy: Typical IT Architecture is a disaster for maneuverability @mtnygard #sotrue #livingit #qconnewyork
Opinions about QCon
Rod Hilton attended the conference:
I choose QCon for a few reasons. One, the sessions seemed very focused on architecture and higher-level concepts, with very few language/technology talks. This was right up my alley because, while there are some languages and tools I'd like to go deeper on, I think a more significant area for improvement for me is architecture and scalability. We get tons of traffic at my job - more than any other place I've ever worked - so I've had to learn a lot about scalability, and the nature of the work has forced me to really see broad system design differently.
I went to QCon specifically wanting to improve some areas where I was weak, namely containerization, microservices, and reactive programming. I hear a lot of buzz about these things, and they pop up on recent ThoughtWorks Technology Radars, and QCon seemed to have a lot of sessions to offer in these areas. …
I really liked the class of people at QCon, it seemed like mostly seasoned veterans in the world of software engineering. …
I really liked how QCon set up their lunch tables. Lunch is typically when most of the socializing happens, and OSCON for example has tables for people with similar interests to meet and mingle. QCon had a similar setup, but I appreciated that they also had a handful of rectangular tables pushed against the walls on the perimeter of the lunchroom. A clear set of "I don't want to mingle" tables. Good stuff if you hate chit-chat like I do….
I really enjoyed my QCon experience and it's definitely on my radar for a conference to attend again in the future.
Tom Limoncelli attended the conference as a speaker:
I had a great time at QCon New York last week. It was my first time there and my first time speaking too. The audience was engaged and had great questions. I did a book-signing at the Pearson booth and it was fun meeting readers (and future readers) of our books.
Jeanne Boyarsky attended the conference:
The conference in general seem set up well with 25 minutes between talks along with an open space by area at the end of the day (not presentations; discussions). For lunch they have tables designed for discussion – large normal conference tables, 4 people discussion tables and “loner” tables. I also like the intro about usability including the big names on the badge. …
Logistically, I really like that you give feedback by putting a green, yellow or red paper as you walk out the door of the session. Low overhead, low time commitment and asked while you still remember the details.
Aditya Chaturved tweeted about video early access: Just finished viewing the first talk I missed because of parallel schedules. Thank you!!! #qconnewyork
Impressions expressed on Twitter included:
@Dana Peele: #QConNewYork was the best conf I've attended, the quality of the topics/speakers/attendees was 2nd to none. Even the food/coffee was killer.
@Caliph: it's an awesome conference for all serious software developers, the big boys play here. #qconnewyork
Takeaways
Rod Hilton’s takeaways were:
I picked QCon based on the sessions, and I was not disappointed.
The sessions were well-arranged into tracks, as is common with these kinds of conferences. What was somewhat different about QCon, at least from my perspective, was how cohesive sessions were within a track, and how diverse the tracks themselves were. A lot of times tracks are really general, like "Java" or "Agile", or they can be too similar to each other. In QCon's case, all of the tracks themselves were very different, but very specific, like "Fraud Detection and Hack Prevention" and "High Performance Streaming Data". Within a track, all of the talks were very closely related, and it actually made sense to pick a track and stick with it, rather than buffet style picking-and-choosing based on session alone.
The sessions were the perfect length. I've complained before that UberConf's 90 minute sessions can sometimes seem overlong or padded, and that OSCON's 30-minute sessions seemed rushed or abbreviated right when they were getting good, but QCon strikes a great balance at 50 minutes each. This is short enough to prevent topic fatigue, but long enough to go in depth. Speakers usually did a great job of giving a presentation in-line with the topic title and description as well, which is (somewhat surprisingly) rare for tech conferences….
One excellent feature of QCon was that almost all of the talks were published in video form after the sessions were over (usually late at night or the next day). The recording quality was excellent, full video of the speaker and their slides synced up, and actual cameramen that kept the speaker in frame for the whole talk. Audio quality was excellent as well….
I learned a lot at the conference, and was able to gain a lot of insight in the areas where I was hoping to.
Conclusion
QCon's focus on practitioner-driven content is reflected in the fact that the program committee that selects the talks and speakers is itself comprised of technical practitioners from the software development community. QCon New York was produced by InfoQ.com. We will next be in Rio de Janeiro August 24-28 and San Francisco November 16-20 this year. QCon New York will continue to run around June of every year.