InfoQ Homepage Articles Key Takeaway Points and Lessons Learned from QCon New York 2017

Key Takeaway Points and Lessons Learned from QCon New York 2017

Bookmarks

Aug 08, 2017 70 min read

Write for InfoQ

Feed your curiosity. Help 550k+ global
senior developers
each month stay ahead.Get in touch

The sixth annual QCon New York was the biggest yet, bringing together over 1,100 team leads, architects, project managers, and engineering directors - up from last year's record of 940. It was also the first to take place in our new home in Times Square.

During his opening keynote Site Reliability Manager at Google, Matt Sakaguchi presented research from Google on creating effective teams. On day 2, Dmitri Alperovitch, co-founder @CrowdStrike who uncovered the 2016 DNC Hack talked about "lessons learned from fighting nation states in cyber space," and on day 3, Cathy Polinsky, CTO & head of engineering and product at StitchFix, talked about building a company on data.

In total, over 130 practitioner-speakers presented 80 full-length technical sessions, 75 talks, and 17 in-depth tutorials, three keynotes, and 12 open spaces over the course of five days, providing deep insights into real-world architectures and state of the art software development practices from a practitioner’s perspective.

Videos of most presentations were available to attendees within 24 hours of them being filmed, and we have already begun to publish them on the InfoQ site. You can view the publishing schedule on the QCon New York website.

InfoQ also reported from the event, and recorded podcasts with a number of speakers. This article, however, presents a summary of QCon New York as blogged and tweeted by attendees.

Keynotes
Tracks & Talks
Workshops
- Zen of Architecture
Community Night
- Papers We Love
Opinions about QCon
Takeaways
Conclusion

Keynotes

Data as DNA: Building a Company on Data

by Cathy Polinsky

Henrik Warne attended this keynote:

Cathy Polinsky of Stitch Fix gave some interesting examples showing when the combination of humans and machine learning can be helpful. Stitch Fix uses both algorithms and stylists to give personalized clothes recommendations. In one instance, a customer had received many skinny jeans, and was getting tired of them, and told the company so. A human stylist had to override the algorithm recommending even more skinny jeans (since the first few times that had been a success). Conversely, many people in Boston were receiving recommendations for shorts in December. Although it seemed like an algorithm error, it turned out to be correct, since they were all going to Florida on vacation.

Twitter feedback on this keynote included:

@shanehastie: #qconnyc @cathy_polinsky Being a data centric company is about using the data to improve the customer experience

@jeanneboyarsky: 90% of the world's data was created in the past 2 years" - has been true for a number of years because in the age of data #QconNYC

@shanehastie: #qconnyc @cathy_polinsky Define your questions and metrics. If you're optimizing for the wrong metric you solve the wrong problem https://t.co/o6IKO8xLWX

@danielbryantuk: If your data is not visible, it's meaningless. Make it visual, and use it to drive learning @cathy_polinsky… https://t.co/RmXJuUUOav

@charleshumble: Open data does not mean sharing personal data. Strip or mark sensitive data, use aggregates, protect customer data. @cathy_polinsky #qconnyc

@danielbryantuk: Agreed! –> "Guinness is the best beer, as they have been running experiments on the taste for over 100 years"… https://t.co/QmRlbNaWjR

@shanehastie: #qconnyc @cathy_polinsky A-B testing doesn't help when you are evolving a ux - the old way will always be "easier" https://t.co/IZg04h3YT0

@charleshumble: a/b tests don't always work. Remember tabs at Amazon? When you change site navigation you interrupt peoples flow.… https://t.co/TAknDoEPUa

@shanehastie: #qconnyc @cathy_polinsky Some testing advice https://t.co/STOesjPqNz

@jchyip: A / B testing with short term effects may be misleading due to users being used to the control. @cathy_polinsky #qconnyc

@shanehastie: #qconnyc @cathy_polinsky Iterate based on the learning. https://t.co/RRYAdlydcR

@shanehastie: #qconnyc @cathy_polinsky The closer you are to the goal, the more important it is to question the goal-metric https://t.co/Gw2Z13RlqX

@charleshumble: Not all big data is interesting data. Stanford has a class called small data; machine learning on a single machine. @cathy_polinsky #qconnyc

@shanehastie: #qconnyc @cathy_polinsky Personalization is not new https://t.co/9clPOCLnlY

@danielbryantuk: Big data is not always valuable. Sometimes 'small' data is good. Ultimately it's about 'Goldilocks' data… https://t.co/D21IekQQ3B

@shanehastie: #qconnyc @cathy_polinsky Stitchfix personalizes everything and learns constantly through feedback https://t.co/ovwar03rJY

@shanehastie: #qconnyc @cathy_polinsky Personalization depends on feedback loops. Make sure you incentivize giving feedback.

@shanehastie: #qconnyc @cathy_polinsky Compelling self interest drives feedback loops https://t.co/t726WgGq7P

@jeanneboyarsky: Need compelling self interest to give you data. Best way is if used to make experience even better. #QconNYC

@charleshumble: If you ask your customer to give you data you'd better use it. @cathy_polinsk #qconnyc

@shanehastie: #qconnyc @cathy_polinsky Make feedback collection fun - example of turning it into a game https://t.co/WSbzyNKUjp

@shanehastie: #qconnyc @cathy_polinsky No customer will share data with you if they don't trust you. https://t.co/sg3xCUh9xH

@shanehastie: #qconnyc @cathy_polinsky Three lessons on personalization https://t.co/ZVLTsIMw4u

@danielbryantuk: You need data from your clients, but you must be trustworthy, engaging, and actually listen to the responses… https://t.co/VfpzjToxdR

@shanehastie: #qconnyc @cathy_polinsky Humans and machines have different abilities. Use both. https://t.co/yTacsnJfL4

@shanehastie: #qconnyc @cathy_polinsky pairing humans and machines gives better outcomes https://t.co/l1O5ZbnNyY

@jeanneboyarsky: Freestyle chess - Best of computer/human skill. + superior technique for how to use computers. Strategy matters! #QconNYC

@shanehastie: #qconnyc @cathy_polinsky Three Takeaways https://t.co/DCyx2KqJQQ

Lessons Learned from Fighting Nation States in Cyber Space

by Dmitri Alperovitch

Dennis Doomen attended this keynote:

The second day's keynote was hosted by the CTO of CloudStrike, a security firm often involved in investigating hacking attempts by nation states such as China. It was a pretty in-depth discussion on how they and similar government agencies map the behavior of hacking groups. I never really realized this, but it's amazing to see how persistent some of these groups are. I kind of assumed that hackers would find the path of least resistance, but the patience with which they inject malware, lure people into webpages or downloading .LNK files that will install the initial implant is truly scary. I particular awed at the idea how hackers manage to embed an entire web shell into a page which allows them to run arbitrary Windows commands on a host system with elevated administrator rights. My takeaway from this session was that if you're targeted by any of these groups, there's nothing you can do. Unless you have the money to hire a company like CloudStrike of course.

Twitter feedback on this keynote included:

@jeanneboyarsky: Whatever business you think you're in, you're in the security business - hacktivists, money, etc #QconNYC

@jeanneboyarsky: In past, only government entities had to worry about nation state attack. Now commercial entities have to protect IP and info. #QconNYC

@danielbryantuk: The reality is that every business today is coming under attack. We need to change the way we think about security… https://t.co/81WNCqk9VG

@shanehastie: #QConNYC @DAlperovitch providing examples of Nation states actively breaking security and stealing IP. https://t.co/OzLJnahUOS

@jeanneboyarsky: Once you use a cyber-weapon, others can use it. Ex: WannaCry is good example of reuse. #QconNYC

@shanehastie: #QConNYC @DAlperovitch The notion of data as a weapon- any organisation can be taken down by leaking all their internal information https://t.co/hXWrTKHNPo

@shanehastie: #QConNYC @DAlperovitch Adversary world map - over 100 identified operators https://t.co/Q1tl54LIIp

@danielbryantuk: Information hacks and leaks can be especially damaging when fake information is combined with the real data… https://t.co/InmF127dSS

@shanehastie: #QConNYC @DAlperovitch Example of a threat actor. Economic espionage focus https://t.co/tOwiyqLQRm

@jeanneboyarsky: learned new term. a webshell is a short web script to gain remote access to a system & access via a web browser #QconNYC

@shanehastie: #QConNYC @DAlperovitch Timeline of an attack https://t.co/dH2bLZPLxW

@ddoomen: You can say a lot about cyber hackers, but boy these people are persistent (and smart). #qconnyc

@shanehastie: #QConNYC @DAlperovitch example of a cloud provider compromise https://t.co/e7OolNnJxY

@danielbryantuk: In the cloud you should monitor access logs, not just for VMs, but for image snapshots and other data at rest https://t.co/bVPlBJ2tcc

@jeanneboyarsky: why do all that when you could use an open source tool. true for both the good guys and the bad guys... #QconNYC

@michellebrush: In any organization you can send an email that says 'This is malware, do not click,' and 5% will click. from @DAlperovitch #QConNYC

What Google Learned about Creating Effective Teams

by Matt Sakaguchi

Dennis Doomen attended this keynote:

This year's keynote was one of the most inspiring keynotes ever. Former police officer Matt Sakaguchi, now a senior Google manager who's terminally ill, talked about how important it is to be completely yourself at the office. He referred to it as "bringing your whole self to work", and spoke of how he never could while working in the macho culture that existed at the police force. According to Sakaguchi, the prerequisites for this is to want to make an impact, have meaning, and the need for structure and clarity on what is expected from somebody. He particularly emphasized the need for people to have psychological safety. Nothing is more restraining to someone's potential than feeling shame. It's the responsibility of a manager or team lead to provide such an environment and to make sure that no employee calls down on another for asking a "dumb" question. It should be perfectly fine to ask them. In fact, somebody else might be wondering the same, but might be afraid to ask because of that same feeling of shame. A nice example of a manager who did that quite well involved his former SWAT sergeant. Whenever they had an assignment, the sergeant would start the briefing with the statement, "Here's my plan. What's wrong with it?".

Ivan Campos attended this keynote:

Google’s Matt Sakaguchi kicked off the conference with the most powerful keynote I’ve ever attended in over 15 years. The main messages were simple, yet powerful:

Make differences in people’s lives and live life to the fullest.

While the presentation made it clear that life is more important than work, it also posited that you can make an impact at work by improving the effectiveness of teams. Backed by empirical data, Google identified that the foundation of team effectiveness is psychological safety….

His message bears repeating: make differences in people’s lives and live life to the fullest.

Twitter feedback on this keynote included:

@msheldon83: Engineering is easy. People are hard #QConNYC

@gwenshap: Turns out that police investigation background is very useful when fixing broken teams at Google #qconnyc

@shanehastie: #qconnyc @matt_sakaguchi "Im greatful for the cancer because it helped me focus on what matters" make life better for others.

@jeanneboyarsky: Matt is passionate about making work better. "We spend a lot of time at work, we spend a lot of time in teams." #QconNYC

@shanehastie: #qconnyc @matt_sakaguchi Improving the experience of teams can make a difference in people's lives

@shanehastie: #qconnyc @matt_sakaguchi A lot of people in tech can't bring their whole selves to work

@danielbryantuk: Great insight into creating effective teams at @Google by @matt_sakaguchi at #qconnyc https://t.co/RpAGiCCPYh

@shanehastie: #qconnyc @matt_sakaguchi in one image. How a team works matters more than who is on the team https://t.co/OR6H01ENVx

@jeanneboyarsky: why defect count not a good metric: "well, you fixed a lot of bugs because you wrote a lot of bugs" #QconNYC

@shanehastie: #qconnyc @matt_sakaguchi Impact of psychological safety of sales team results https://t.co/Ot80gCrKhc

@charleshumble: Factors for great teams: 1.Psychological safety, 2.dependability, 3.Structure and clarity, 4.Meaning, 5.Impact… https://t.co/nO56mPCn64

@shanehastie: #qconnyc @matt_sakaguchi What makes up Psychological Safety Voice - I'm listened to, I trust and am trusted, we're really together https://t.co/oCvsBifZwl

@danielbryantuk: Teams that feel safe are effective (e.g. beat their sales targets). Psychological safety is *vital*… https://t.co/VYohVGNFMX

@jeanneboyarsky: Inclusion is not just inviting all people to the party, but making sure everyone has a good time. The importance of support #QconNYC

@charleshumble: Set the tome for psychological safety: 1. Frame work as a series of learning problems, not execution problems. @matt_sakaguchi #qconnyc

@charleshumble: Set the tome for psychological safety: 2.Model curiosity and ask more questions. 3. Admit your own fallibility.@matt_sakaguchi #qconnyc

@msheldon83: Be willing to ask "dumb" questions. There are probably others wondering the same thing. #QConNYC

@charleshumble: Parting thoughts - What are the top 3 things you enjoy doing outside of work. Prioritize those things. @matt_sakaguchi #qconnyc

@msheldon83: Every meeting should start with the question: "How can we make this work?" #QConNYC

Tracks & Talks

Architectures You've Always Wondered About

IoT and Edge Compute at Chick-Fil-A

Ryan Harbison attended this session:

At the QCon New York conference held in June 2017, Brian Chambers, Chick-fil-A’s enterprise architect, spoke about how the fast-food company is using Internet of Things (IoT), edge computing, and cloud services technologies to change the way it enables in-restaurant interactions, from customer experiences to remote monitoring and kitchen automation. On the surface, it appears that the fast-food chain is simply utilizing sensors to monitor the temperatures of coolers and freezers as a precaution against the public relations nightmare that accompany E. coli or, more likely in this case, salmonella outbreaks. On the back-end, though, Chick-fil-A is embracing connected products and services, driving its digital transformation to impact operational experiences at individual restaurants. Through its container-based edge compute architecture, IoT platform, and data analytics, Chick-fil-A is differentiating itself from its competitors and proving that IoT is not just limited to industrial applications.

Stephen Steen attended this session:

This architecture is spawning some very well thought-out IoT architectures, such as the one presented by Brian Chambers from Chick-Fil-A. Chambers showed a scalable and secure IoT infrastructure utilizing Docker for edge tools connected to AWS. This architecture has several steps for security, including dynamic client registration for OAuth clients, human authorization of devices, stateless JWT tokens and local persistence/authentication if there are connectivity issues.

Because these architectures are taking hold, the integration of APIs has never been easier or more necessary. From Netflix's streaming architecture to Stitch Fix’s new approach to shopping, they all utilize microservices through APIs to build and scale seamlessly. New features or tools created can add value in minutes compared to the months or years of integration and testing in more closed monolithic architectures.

Scaling Event Sourcing for Netflix Downloads

by Phillipa Avery & Robert Reta

Henrik Warne attended this session:

Phillipa Avery and Robert Reta from Netflix talked about their experience using event sourcing when implementing the download feature. Overall they were quite happy with the result, especially that it was easy to extend when new requirements emerged. They also mentioned how trouble shooting became easier when it was possible to view they entire event stream for a subscriber. I liked this talk because the presenters had been involved with the actual implementation, and because it is both interesting and useful to listen to experience reports (in case you want to implement something similar).

Twitter feedback on this session included:

@kcasella: Changing business requirements require flexibility in systems @PhillipaAvery @rreta04 #QconNYC https://t.co/YqtfB7Y7iP

@kcasella: Basic building blocks for Event Sourcing @PhillipaAvery @rreta04 #QconNYC https://t.co/bJDeaygSN3

@kcasella: Resilience in Event Sourcing for Downloads: "err on the side of customer experience" @PhillipaAvery @rreta04… https://t.co/sycHBhnVGW

@kcasella: Take-aways from Event Sourcing talk by @PhillipaAvery @rreta04 #QconNYC https://t.co/3qK9Pru9q8

Unifying Banks & Blockchains @Coinbase

by Jim Posen

Jeanne Boyarsky attended this session:

In 2016, the Rails app was becoming a problem and the BitCoin logic started to degrade. Created first microservice at that point. Now support multiple currencies. Still maintain monolithic Rails app.

Bitcoin

Definition – Bitcoin is a scarce digital asset and a protocol for transferring the asset over the internet. “Email” is overloaded in two ways as well.
Public transactions ledger
About 30 minutes for transactions to clear – regardless of hours and holidays
Irreversible payments

Coinbase architecture

Uploads batch file daily to the originating depository financial institution (ODFI). Clears thorough ACH operator to receiving depository financial institution (RDFI) and receiver receives
The RDFI has 24 hours to return for insufficient funds
Then the receiver can challenge/return for up to 60 days – important consumer protection, but a challenge
Bitcoin uses gossip protocol where nodes talk to other nodes

Twitter feedback on this session included:

@danielbryantuk: Interesting exploration of the @coinbase architecture at #qconnyc - the idea of polymorphic microservices for different payments is cool https://t.co/L3GxF8zxno

Chaos & Resilience

Ivan Campos attended this track:

NETFLIX, known for their Chaos Monkey tool, provided insight into Chaos Engineering. As defined by principles of chaos, “Chaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production.” As our architectures embrace microservices, testing by means of unit, regression, and integration, will need to be augmented by chaos engineering. The main takeaway from chaos engineering is that the strategy doesn’t cause problems, it reveals them. …
Dropbox also embraces chaos engineering as a means to remove single points of failure. To add credence, they ran through their approach which is currently being applied across three data centers and involves 10k machines.
Coinbase has a 30 day plan which stipulates that no server should live longer than 30 days. Their median fleet age is ~9 days. This means that each and every AWS VM is rebuilt within a month’s time. By rebuilding their entire fleet of servers so often, they are ensuring that their servers are patched frequently to mitigate exposure to published vulnerabilities. Having servers updated and reconfigured via rebuilds is all part of an even larger plan to codify and automate everything. Over the past 12 months, they have launched nearly 150k servers. Security played a major role in the 30 day plan. For Coinbase to remain the most popular way to buy Bitcoin, Ethereum, and Litecoin, they must prioritize security. Coinbase also discussed how they use machine learning for payment fraud detection. The call for an end to SMS-based two-factor authentication was also made. For more details on why you should move away from SMS-based 2FA, check out their medium post.

Choose Your Own Adventure: Chaos Engineering

by Nora Jones

Jeanne Boyarsky attended this session:

Known ways for testing for availability

Unit test
Integration test
Regression Test
Chaos Engineering – much less common. It doesn’t replace need for “traditional” tests

Chaos Engineering

Most people know what Chaos Monkey is; far less know about Chaos Engineering. The former is a tool; the latter is a strategy. The former is mature; the latter is emerging
“Chaos” means different things to different companies. Common thing like experimenting and distributed systems, make systems stronger through experiments.
The goal is to run chaos all the time, not just on deployment…

Introducing chaos

Think about where you are now and the expected response
How many people should know the chaos is intentional? It's helpful to know when running an experiment
Define “normal” system and behavior
Relate chaos to automated tests, SLAs and customer experiments
Start in QA, not Prod. This establishes a baseline
Only run during business day…

Culture and implementation

People need to understand that revealing problems is good (vs causing problems)
Start with opt-in so people have control
Monitoring is important. Use dashboards to communicate
Automatically shut down an experiment if it goes too far astray
Have your incident/Jira/PagerDuty tickets gone down
Don’t forget about your company’s customers. Focus on business goals and not causing customer pain

Twitter feedback on this session included:

@charleshumble: Computers are complicated and they will break. @nora_js #qconnyc

@jeanneboyarsky: Failures will happen anyway. Why are we afraid of that? #QconNYC

@jeanneboyarsky: working on chaos experiments is a quick way to meet your new colleagues; but not a great way #QconNYC

@charleshumble: Socialization tends to be harder then implementation. This was a surprise to me. @nora_js #qconnyc

@rmn: Chaos doesn't cause problems it reveals them - @nora_js #qconnyc

@charleshumble: Strongly consider customer impact when considering chaos engineering. Also, start in QA to atlas establish a benchmark. @nora_js #qconnyc

@charleshumble: Cascading failures often lie dormant for a long time until they are triggered by an unusual set of circumstances. @nora_js #qconnyc

@charleshumble: At Jet we caused a cascading failure (doing chaos testing) but it wasn't the one we'd intended.It took QA down for a week @nora_js #qconnyc

@charleshumble: Netflix testing tool ChAP focused on minimizing blast radius. @nora_js #qconnyc https://t.co/iYVuR4p9Qm

@charleshumble: Netflix ChaP Goal: Chaos all the things and *run all the time.* @nora_js #qconnyc

@charleshumble: Its important to record chaos success stories (especially important during adoption). @nora_js #qconnyc https://t.co/hQwnxrs3qB

@danielbryantuk: Our goal at Netflix is to have all tier one services running chaos testing (ChAP) all of the time @nora_js… https://t.co/Ri2pex1mhS

@danielbryantuk: Great takeaways from @nora_js's #qconnyc talk about introducing chaos testing into a organisation https://t.co/GrVYzNYUYS

State of Chaos Engineering

by Bruce Wong

Twitter feedback on this session included:

@charleshumble: The first thing I leant about systems is that you have to know what is critical and isn't. @bruce_m_wong #qconnyc

@charleshumble: There are two ways to find out what is critical in a microservices environment. Outages and chaos engineering. @bruce_m_wong #qconnyc

@charleshumble: If we don't continually test then everything is in an unknown state. That is more scary to me then chaos testing@bruce_m_wong #qconnyc

@charleshumble: People make mistakes. More people, more mistakes. @bruce_m_wong #qconnyc

@charleshumble: Learn from the mistakes of others, you can't live long enough to make them all yourself.@bruce_m_wong quoting Groucho Marx at #qconnyc

@charleshumble: @bruce_m_wong recommended Gremlin - the company started by @KoltonAndrus - https://t.co/XTUdeGRaCs #qconnyc

Developer Experience: Level up Your Engineering Effectiveness

Dennis Doomen attended this track:

Quite a lot of the talks and open space sessions I attended talked about the development experience, and more specifically about removing friction. Although some of it should be common sense, they all tried to minimize the distance between a good idea and having it run in production.

Don't try to predict the future, but don't take a shortcut….
Don't bother developers with infrastructure work….
Bugs should not be tracked as a separate thing. Just fix them right away or decide to not fix them at all….
Closely tied to that is to keep distractions away from the team by assigning a Red Hot Engineer on rotation. This person handles all incoming requests, is the first responder when builds fail, and keeps anybody else from disturbing the team.
To track the happiness of the team, introduce and update a visual dashboard that shows the teams sentiment on various factors using traffic light….
Don't run your code locally anymore. If you're unsure if something works, write a unit test and learn to trust your tests. …

Production - Designing for Testability

by Michael Bryzek

Henrik Warne attended this session:

Michael Bryzek talks about how Gilt only has their production environment. There are no local, QA, or staging environments. Testing is done via comprehensive automated tests, including tests running in sand-boxed accounts in production. The rationale for this approach stems from the question “How do you know if it is working in production?”. Even if you verify your services in a testing environment, you still need to ensure it is working in production. By focused all testing and monitoring efforts to production, you avoid any issues with maintaining separate testing environments. I liked this talk because it really challenged my assumptions. Can you really do this? But that would mean… . Definitely food for thought.

Twitter feedback on this session included:

@shanehastie: #qconnyc @mbryzek Software quality is hard. https://t.co/xAAaXccWOo

@shanehastie: #qconnyc @mbryzek Ideas around delivering quality software https://t.co/HftHvlF25k

@sangeetan: @mbryzek on verification in production #qconnyc https://t.co/Hivtqnd5gb

@RichardChmielek: .@mbryzek on Production: Designing for Testability Staging environment? Don't need them!#QconNYC https://t.co/LiLewdFcGJ

@shanehastie: #qconnyc @mbryzek True continuous delivery. Is it OK to deploy this code right now? Decision made by the developer.… https://t.co/WaY3D4AozR

@shanehastie: #qconnyc @mbryzek We should say what the desired state is, let the tooling figure out how to get there.

@shanehastie: #qconnyc @mbryzek The wastes from having staging environments https://t.co/5dvesCHbiY

@shanehastie: #qconnyc @mbryzek 30-40% of engineering budgets are spent maintaining staging environments.

@shanehastie: #qconnyc @mbryzek If it works in staging you can't guarantee that it will work in production

@danielbryantuk: I love my staging environment - said no one ever@mbryzek on the need to continuous deliver through to production https://t.co/Rm4tVKmg6j

@shanehastie: #qconnyc @mbryzek Don't run your code locally. Write the test. Write high quality tests and trust your tests. https://t.co/BpjGxnQrUw

@shanehastie: #qconnyc @mbryzek Quality through architecture. https://t.co/GUk0ShyOrQ

@shanehastie: #qconnyc @mbryzek Real examples of testing in production. Does the failure rate increase when we test in prod? https://t.co/mDWGVX5GUO

@shanehastie: #qconnyc @mbryzek Treat all services as external services. https://t.co/pT21mggJ94

@danielbryantuk: At @flood_io every (internal) service is treated as a third party for dev. We run sandbox services/account in production https://t.co/TP9YCQaT1j

@shanehastie: #qconnyc @mbryzek End to end integration tests running in production. If a test fails it's a build failure https://t.co/EFFsOgbToe

@shanehastie: #qconnyc @mbryzek Example : verify that proxy server works as expected https://t.co/7WnA4shuq0

@shanehastie: #qconnyc @mbryzek Things can go wrong, so some important considerations https://t.co/oMIaSh7E3S

@shanehastie: #qconnyc @mbryzek Unexpected benefits- perfect documentation. https://t.co/3Hca9UDYo1

@danielbryantuk: The quest for perfect documentation, with generation of docs and full integration of tests within the CD pipeline https://t.co/NsL4agRm7a

@shanehastie: #qconnyc @mbryzek Tooling matters. Additional examples https://t.co/I6qJA8FyPM

@danielbryantuk: Many devs at Flow write their own SDKs for third party services in order to get high quality mocks from our frameworks https://t.co/mW8z336Y99

@sangeetan: Testing in prod - key takeaways from @mbryzek #qconnyc https://t.co/zDZKGXMeAr

Reasoning about Complex Distributed Systems

by Erich Ess

Twitter feedback on this session included:

@shanehastie: #QConNYC @egerhardess Complex systems are messy and hard to reason about. Presenting some ideas to help think clear… https://t.co/V8hGIGswpS

@shanehastie: #QConNYC @egerhardess Mental models and experiments https://t.co/zump0e4gv1

@shanehastie: #QConNYC @egerhardess Things to build your mental model https://t.co/coV4lthHSF

@shanehastie: #QConNYC @egerhardess Think about the structure and interactions in the system, not the individual decision logic https://t.co/Oa7OOm51X0

@shanehastie: #QConNYC @egerhardess A simple example - make sure you have the big picture view in your mind, without getting lost in the details https://t.co/wajRa3nfkJ

@shanehastie: #QConNYC @egerhardess Use the model to reason about the symptoms you see happening https://t.co/2gjoCpa9nR

@shanehastie: #QConNYC @egerhardess ask "what happens when...", what would you expect to see elsewhere in the system https://t.co/qQaSn2b7BY

@shanehastie: #QConNYC @egerhardess Start with the most likely, simplest explanation- that's probably where the problem sits. Occam's razor

@shanehastie: #QConNYC @egerhardess A more complex example https://t.co/PZRVj4ZSaI

@sangeetan: @egerhardess on using mental models to reason about systems #qconnyc https://t.co/WkZuoicuZf

@shanehastie: #QConNYC @egerhardess Form a hypothesis based on your mental model and observations, then design experiments to validate it https://t.co/z1x0zYJYu9

@shanehastie: #QConNYC @egerhardess Hypothesis validation https://t.co/Ep8C5bATo1

@shanehastie: #QConNYC @egerhardess experiments help build your mental models https://t.co/zKBX8gBeHj

@shanehastie: #QConNYC @egerhardess Design your test data to answer experiment questions https://t.co/a5j11Jpxjl

@shanehastie: #QConNYC @egerhardess wrapping up. https://t.co/fS2lQaav2y

Removing Friction in the Developer Experience

by Adrian Trenaman

Twitter feedback on this session included:

@shanehastie: #qconnyc @adrian_trenaman A great developer experience is about minimizing the distance from an idea and production https://t.co/akrTHkjNYL

@danielbryantuk: We are software development professionals - we don't work for treats @adrian_trenaman on the (un)importance of work-place perks #qconnyc https://t.co/COknXSbLG1

@shanehastie: #qconnyc @adrian_trenaman key things needed - we are an engineering organization, code is the artefact https://t.co/gfRNPGlliq

@shanehastie: #qconnyc @adrian_trenaman Autonomy, mastery & purpose is necessary but not sufficient. Must overcome the friction too https://t.co/zJzKkHxUMz

@shanehastie: #qconnyc @adrian_trenaman Friction #1 Staging or testing environment. https://t.co/XVlFlmXWgw

@danielbryantuk: Every time you create a staging environment a fairy dies. Don't waste energy on a poor representation of prod https://t.co/vud8K5LKMb

@shanehastie: #qconnyc @adrian_trenaman Spaghetti diagram showing the amount of wasted effort & time in the process. https://t.co/jQobdzchJV

@shanehastie: #qconnyc @adrian_trenaman Types of waste - Muda https://t.co/c7Q5nqdZOr

@philip_pfo: To improve DevEx, reduce the multi environment theater - lots of things moving but not getting anything done.… https://t.co/zffFNd8tGA

@sangeetan: Spaghetti diagram showing wasted cycles in software delivery process - example of poor https://t.co/gCpKLq6QfB

@shanehastie: #qconnyc @adrian_trenaman Core idea #1 Test in prod with dark canaries, canaries, rolling release, rollback https://t.co/qjJ4rvFMn8

@danielbryantuk: If all this back and forth to QA and pre-prod isn't benefiting us, then we should test in prod @adrian_trenaman minimizing work https://t.co/Ssv4HY3cTM

@shanehastie: #qconnyc @adrian_trenaman Core idea 2: your teams are startups proving services to other development teams. Make the software easy to access https://t.co/0o004KUe8w

@shanehastie: #qconnyc @adrian_trenaman Core idea 3: exploit multi-tenant design for confident testing in production https://t.co/igaBV0ydww

@shanehastie: #qconnyc @adrian_trenaman To enable this you need to design for testing in production from the beginning

@shanehastie: #qconnyc @adrian_trenaman Core idea 4: give your teams secure, unfettered control of their own infrastructure. https://t.co/iTbhyQf5Wt

@shanehastie: #qconnyc @adrian_trenaman Friction #2 - forced technology choices. Prefer voluntary adoption https://t.co/jEI4OK2EqO

@shanehastie: #qconnyc @adrian_trenaman use a technology radar to discuss tools https://t.co/WZvHHdOnoN

@kcasella: Philosophical note: choose your abstractions & frameworks carefully @adrian_trenaman #qconnyc

@shanehastie: #qconnyc @adrian_trenaman Look for consensus, consider the philosophy https://t.co/1hWHPKBtpw

@shanehastie: #qconnyc @adrian_trenaman Friction: fear of breaking other things https://t.co/Oqd3yj1xWf

@sangeetan: Choose where you want to be on the tech adoption spectrum. There are tradeoffs @adrian_trenaman #qconnyc https://t.co/3eSDtrzLm5

@shanehastie: #qconnyc @adrian_trenaman Evolving towards micro services, safely deploy a single piece without the possibility of breaking anything else https://t.co/Ow5dLsIpyq

@philip_pfo: For tech choices, favor voluntary over forced adoption; developers will migrate towards what works best. https://t.co/9V7ivLafks

@shanehastie: #qconnyc @adrian_trenaman Friction- forced team choices - prefer self selection https://t.co/2J4nd4FT71

@shanehastie: #qconnyc @adrian_trenaman self selection needs a few roles allocated, then pitch and sell https://t.co/AF4j8mv3FP

@shanehastie: #qconnyc @adrian_trenaman Friction: distractions. Reinforce the notion that coding is the primary activity https://t.co/gZPSd0S9JT

@shanehastie: #qconnyc @adrian_trenaman reducing Friction- sacrifice one pattern, reduce & refine meetings https://t.co/LkGYna6e6z

@shanehastie: #qconnyc @adrian_trenaman Measure it - people operations goals. Use team health checks. https://t.co/IMb9bYlZE2

@shanehastie: #qconnyc @adrian_trenaman Summarising https://t.co/GgseHIeOef

@danielbryantuk: Lots of great takeaways on the people side of software development from @adrian_trenaman #qconnyc https://t.co/jQNKotm7MT

Spotify Lessons: Learning to Let Go of Machines

by James Wen

Ivan Campos attended this session:

Spotify discussed how ephemeral service design avoids hard coded host names, but requires centralized logging. Their “Ops in Squads” approach has improved the lives of over 900 developers by allowing them to provision their own machines. They have provisioned over 20k servers.

Twitter feedback on this session included:

@shanehastie: #qconnyc @RochesterinNYC Spotify lessons: letting go of machines starting https://t.co/cLFO20aocR

@shanehastie: #qconnyc @RochesterinNYC Setting the scene https://t.co/aFITxUCWuD

@shanehastie: #qconnyc @RochesterinNYC Key Takeaways https://t.co/YsXYRQXXGQ

@shanehastie: #qconnyc @RochesterinNYC Don't make feature developers worry about their infrastructure https://t.co/WQSitvZXpm

@shanehastie: #qconnyc @RochesterinNYC Example of a task devs have to do which is wasted time https://t.co/BzTwfrslxO

@danielbryantuk: The centralised model to ops wouldn't scale at @Spotify scale. We do 'ops in squads' #qconnyc https://t.co/B6JGEUxKB1

@danielbryantuk: Feature developers should be focused on developing features, not the underlying infrastructure platform… https://t.co/5NaAluz70m

@shanehastie: #qconnyc @RochesterinNYC Tooling built to remove the Developer concerns and allow them to focus on writing Feature code

@sangeetan: immutable infra with docker helped eliminate snowflakes at #spotify @RochesterinNYC #qconnyc ;next step - cloud with GCP!

@sangeetan: Use automation to abstract away infra concerns & reduce hardware related context devs need to be effective @RochesterinNYC #QconNYC

@shanehastie: #qconnyc @RochesterinNYC key learnings from commoditizing developer hardware https://t.co/ivO0qEKTEp

@sangeetan: Things you need to start thinking differently about when moving to cloud @RochesterinNYC #qconnyc https://t.co/0wLBHDLjmu

@shanehastie: #QConNYC @RochesterinNYC Additional learnings from hybrid migration https://t.co/GylyyegBED

@shanehastie: #QConNYC @RochesterinNYC People resist change, even if change is good. https://t.co/qO4PnB1vlP

@danielbryantuk: Interesting learnings from the Spotify journey towards hybrid cloud, from @RochesterinNYC at #qconnyc https://t.co/LZpctg3rLV

@shanehastie: #QConNYC @RochesterinNYC Wrapping up https://t.co/VpUeMx0jTy

Zero to Production-Ready in Minutes

by Tim Bozarth

Twitter feedback on this session included:

@shanehastie: #QConNYC @timbozarth The focus - level up engineering effectiveness https://t.co/kaDrCJp6pz

@ddoomen: So Netflix is consuming 1/3 of the US network traffic.... Wow! #qconnyc

@shanehastie: #QConNYC @timbozarth Netflix current state and very simplified architecture view https://t.co/0bFxvcvdX7

@shanehastie: #QConNYC @timbozarth High availability is crucial for Netflix - winning moments of truth https://t.co/cor9vd15B4

@shanehastie: #QConNYC @timbozarth Challenges in the Netflix environment https://t.co/GqH4fKFsFw

@shanehastie: #QConNYC @timbozarth accepting an existing challenge as "the cost of doing business" is a fallacy- it's probably a tax you're paying

@shanehastie: #QConNYC @timbozarth Goal of the runtime team at Netflix https://t.co/A8lKVw1oQ6

@shanehastie: #QConNYC @timbozarth The story of how it was to build an app https://t.co/QCCR5Cypd2

@shanehastie: #QConNYC @timbozarth time to be able to start building 3 days to 3 weeks.

@shanehastie: #QConNYC @timbozarth Live example of generated code and config to enable starting build in minutes https://t.co/Ai9cVsk4Y6

@shanehastie: #QConNYC @timbozarth Generators enable consistency without restricting creativity. Generating components that developers build on.

@shanehastie: #QConNYC @timbozarth Simple rules for consistency https://t.co/3lXVF5BAbd

@sangeetan: #Netflix service owners responsible for defending consumers against failures @timbozarth #qconnyc

@shanehastie: #QConNYC @timbozarth Service design (simplified) https://t.co/tAomx2ICPF

@shanehastie: #QConNYC @timbozarth Problems with hand written letters libraries https://t.co/EzsidLOYkN

@shanehastie: #QConNYC @timbozarth replacing hand writte n libraries gives big wins gRPC and Proto https://t.co/aZFDCf6f0h

@shanehastie: #QConNYC @timbozarth revised design https://t.co/0aQyE3KqDY

@shanehastie: #QConNYC @timbozarth Interceptors - value and examples https://t.co/JlJ6znhQAz

@sangeetan: Code generation and abstraction simplify comm between services so devs can focus solely on biz logic. @timbozarth #qconnyc

@shanehastie: #QConNYC @timbozarth Before & after https://t.co/PZ9hwc1tJT

@shanehastie: #QConNYC @timbozarth The move from build by hand to open source should be obvious, but it is hard. https://t.co/gNR4HAZvxA

@shanehastie: #QConNYC @timbozarth Inertia is a powerful force and a terrible strategy https://t.co/vyEdujrdIf

@sangeetan: Think about when you've reached the point of diminishing returns with your investments @timbozarth #qconnyc https://t.co/HODQa0whhL

@shanehastie: #QConNYC @timbozarth Principle: favor commodity when it's not our Core competency https://t.co/LmZdSv8XDr

@shanehastie: #QConNYC @timbozarth Evidence that this works and has been done. Summarizing https://t.co/hrKLiQJlZC

High Velocity Dev Teams

Development Metrics You Should Use But Don't

by Cat Swetel

Twitter feedback on this session included:

@shanehastie: #qconnyc @CatSwetel Life is like a box of chocolates, development metrics are like a bowl of fruit loops. Do nothing for your team's health

@jonathansen: Bad dev metrics are like Froot Loops - @CatSwetel #QconNYC

@shanehastie: #qconnyc @CatSwetel Four metrics areas that matter https://t.co/KNF335i0F2

@shanehastie: #qconnyc @CatSwetel For each unit of value, piece of work, need to explicitly measure start and end points.

@shanehastie: #qconnyc @CatSwetel Advanced math - how long does a piece of work stay in process. https://t.co/gd7VbBd4yB

@shanehastie: #qconnyc @CatSwetel exploring time in process over time scatter plot. Look at the trends. https://t.co/yB88uvxj27

@shanehastie: #qconnyc @CatSwetel using time in process over time gives predictability and probability https://t.co/lfRxevQ4PT

@shanehastie: #qconnyc @CatSwetel Same data - different view. How predictably responsive are we? https://t.co/moHjZjYK4Q

@jeanneboyarsky: Weibull distribution - fat toward zero but tail trickles into infinity. Like commute. Usually takes X minutes but then sometimes... #QconNYC

@jonathansen: The price of more certainty is delay of everything else - @CatSwetel #QconNYC

@shanehastie: #qconnyc @CatSwetel Make sure you include all work and available capacity in your predictions. https://t.co/HsinGRrGsC

@shanehastie: #qconnyc @CatSwetel Satir Change Curve in action.Make change and productivity dips. https://t.co/DfUiStXrcf

@shanehastie: #qconnyc @CatSwetel add start & end dates PER state to get more granular. How much touch time vs total time. Value streams

@shanehastie: #qconnyc @CatSwetel identify the inflection points. Tells where to focus improvement efforts https://t.co/eeUIvgtC5r

@MurrMarie: @CatSwetel showing arrival and departure metrics. Development Metrics You Should Use but Don't @qconnewyork… https://t.co/EIxQKRWxPu

@shanehastie: #qconnyc @CatSwetel provide the story around your metrics. Most metrics are currently presented without context.

@jannabrummel: .@CatSwetel about averages in development metrics: "they're not good for anything!" #usepercentiles #orotherstatistics #QconNYC

Machine Learning: from Theory to Practice

by Debbie Madden

Twitter feedback on this session included:

@shanehastie: #QConNYC @debbiemadden200 5 secrets to improve team communication. Team communication is the foundation for everything else https://t.co/xWEG1gmvzt

@shanehastie: #qconnyc @debbiemadden200 Ideas are a dime a dozen. It's the best teams that win, not the best ideas.

@shanehastie: #qconnyc @debbiemadden200 Five key secrets to improving team communication https://t.co/XhtUwp3MUx

@shanehastie: #qconnyc @debbiemadden200 Every organization team goes through alternating periods of calm and chaos. Greiner Curve. https://t.co/SyxmZJCXd0

@shanehastie: #qconnyc @debbiemadden200 Simple ways to tell chaos from calm https://t.co/u0wJ9e7VZ9

@shanehastie: #qconnyc @debbiemadden200 many tech leaders miss the signs of chaos. For clarity - ask your peers

@jeanneboyarsky: Your engineers didn't get dumber. You scaled to the next level of chaos #QconNYC

@shanehastie: #qconnyc @debbiemadden200 Place people in roles they are interested in. Keeping people in the wrong seat results in productivity loss https://t.co/Oa73vyo4Mx

@shanehastie: #qconnyc @debbiemadden200 Help B become A by putting them in the right roles. BC is toxic -get rid of them https://t.co/kU3ePkeTM1

@shanehastie: #qconnyc @debbiemadden200 Most companies and teams fail in the periods of chaos. If you're not sure get an outside coach to help.

@shanehastie: #qconnyc @debbiemadden200 Starting with Why - people align around mission. Quoting Simon Sinek https://t.co/wbsfMBwSwa

@shanehastie: #qconnyc @debbiemadden200 Team productivity is more important than individual productivity https://t.co/YxOz3jeGAj

@jonathansen: Work from home IF IT DOESN'T IMPACT THE TEAM - @debbiemadden200 #QConNYC

@shanehastie: #qconnyc @debbiemadden200 Hold each other to working agreements https://t.co/gNU3iaEn9F

@shanehastie: #qconnyc @debbiemadden200 Team health is based on trust https://t.co/0arAiioK1L

@shanehastie: #qconnyc @debbiemadden200 Two types of trust - Capability and Desire. Understand the difference.

@shanehastie: #qconnyc @debbiemadden200 Healthy conflict matters. Good reference case study https://t.co/cjPTJWbtZw

More Reliable Delivery with Monte Carlo & Mapping

by Conal Scanlon

Twitter feedback on this session included:

@danielbryantuk: Nice!! –> "If you want to replace four hour meetings with 30 lines of code doing Monte Carlo simulations, then come to this talk" #qconnyc

@shanehastie: #QconNYC @cms5109 Current planning situation https://t.co/mw21fo3R8g

@shanehastie: #QconNYC @cms5109 Explaining what/how Monte Carlo analysis process works https://t.co/SJXCwZTuPA

@shanehastie: #QconNYC @cms5109 two important rules for forecasting https://t.co/P2oQhnBQLD

@shanehastie: #QconNYC @cms5109 velocity and points not very useful. Use flow metrics instead https://t.co/XWugXK2V2Q

@shanehastie: #QconNYC @cms5109 Some flow based metrics https://t.co/8FGAOTLwyE

@shanehastie: #QconNYC @cms5109 Need data to work with. Extrapolate into a set of samples using statistical distribution https://t.co/iInb6AowZW

@jonathansen: If @WarrenBuffett walks in the room, the average wealth goes up, but the rest of us don't get any richer. #averagescangoscrew #qconnyc

@shanehastie: #QconNYC @cms5109 Add backlog size and aggregate the results across the sample. https://t.co/zhehuKCJLS

@shanehastie: #QconNYC @cms5109 represent as a cumulative completion curve https://t.co/fqXZ7aXB7S

@shanehastie: #QconNYC @cms5109 How much data do you need - what if you have none to start with. Need some real data to start. https://t.co/rHAj39vA8U

@shanehastie: #QconNYC @cms5109 recap - using Monte Carlo forecasting. https://t.co/lC3gc9bcg3

@shanehastie: #QconNYC @cms5109 need a quality backlog rather than expert guesses. Use a story map https://t.co/9YSqlv1rR9

@shanehastie: #QconNYC @cms5109 quoting @jeffpatton "mile wide, inch deep" when building out your story map https://t.co/XhVQXrKYtC

@shanehastie: #QconNYC @cms5109 Explore the journey not just the cards https://t.co/hGVzuQbAQu

@shanehastie: #QconNYC @cms5109 Story Map partitioned into releases https://t.co/l4FM4hMCFa

Refactoring Organizations - a Netflix Study

by Josh Evans

Twitter feedback on this session included:

@shanehastie: #qconnyc @Ops_Engineering tribalism is a common challenge in organizations which creates walls between teams

@shanehastie: #qconnyc @Ops_Engineering Conway's law applies https://t.co/jW5q1hyEbe

@shanehastie: #qconnyc @Ops_Engineering the premise: Conway's law describes dysfunction

@shanehastie: #qconnyc @Ops_Engineering Selfless leadership - company first, the the team, then yourself

@shanehastie: #qconnyc @Ops_Engineering why refactor? https://t.co/mA1UO6QOsW

@shanehastie: #qconnyc @Ops_Engineering Defining organizational scalability https://t.co/n8ulcox6DV

@shanehastie: #qconnyc @Ops_Engineering Organizational inheritance - you build it, you run it as an instance of culture https://t.co/QzexM9gdm6

@KyleTitus: Engineering priorities at @netflix circa 2010. Sound familiar? #QConNYC https://t.co/6tcmmIYv5y

@jonathansen: Team structures can be monoliths too - @Ops_Engineering #QconNYC

@shanehastie: #qconnyc @Ops_Engineering Context switching and single point of failure risk https://t.co/0r4G7Vs1eR

@shanehastie: #qconnyc @Ops_Engineering Context switching visualised https://t.co/i2VlAGBnGu

@jchyip: Engineer as a library: Embedding engineers into other teams to provide a particular capability. #QconNYC

@danielbryantuk: Great analogies from tech design patterns to organizational patterns to address issues when scaling https://t.co/jrMB9U63Jq

@shanehastie: #qconnyc @Ops_Engineering People are not computers - the metaphor can only extend so far. You need bimodal thinking. https://t.co/6BqkcE7QG6

@shanehastie: #qconnyc @Ops_Engineering Overcoming tribalism. Needs IQ & EQ https://t.co/aiFzCCDXeG

@jannabrummel: .@ops_engineering explaining why context switching sucks by showing what it does to an operating system #qconnyc https://t.co/IFKrL26fj4

@danielbryantuk: Great takeaways from "refactoring organizations" by @Ops_Engineering at #qconnyc -> A prerequisite is freedom and responsibility culture https://t.co/6dZuc7tA2M

What Does Speed Mean in Software Product Delivery?

by Jason Yip

Twitter feedback on this session included:

@jeanneboyarsky: We think of speed as a number. But it's not a quality without a context. Are you fast when you deploy? When at peak load? #QconNYC

@jonathansen: Refined annoyance: the refusal to accept friction of any kind - @jchyip #QconNYC

@michellebrush: Poor approaches delay pain. Good approaches don't. @jchyip #qconnyc

@nzben: Bring the pain, because avoiding pain just means you’ll have more pain later.

Immutable Infrastructures: Orchestration, Serverless, and More

A Series of Unfortunate Container Events @Netflix

by Amit Joshi & Andrew Spyker

Twitter feedback on this session included:

@RichardChmielek: .@aspyker over 1M containers in public cloud in production @netflix #QConNYC https://t.co/PP8eZGUfWh

@danielbryantuk: If you just throw @Docker on @awscloud you simply get slightly less Docker and slightly less AWS. Integration is key https://t.co/oy5YRi4Zmg

@danielbryantuk: There will always be bad actors in a system. Protect against this with input parsing, rate limiting and fault tolerance https://t.co/DkR4hgxfKl

@danielbryantuk: Interesting journey of implementing container security at Netflix. User Namespaces (done right) and AppArmor FTW. https://t.co/tPwC2N5kLm

@sangeetan: Cloud is not perfect yet for containers @aspyker on dealing with rate limiting, bad VMs #QConNYC

@danielbryantuk: Stuff sometimes doesn't work in the cloud. We put health checks on our Titus agents and terminate instance or drain work https://t.co/3FlSgfx2o5

@danielbryantuk: Sometimes a daemon locks up, and we simply can't terminate a job. We run aggressive reconciliations to address this https://t.co/Zt12BWF7r2

@danielbryantuk: You must have kernel expertise on your team if you are providing a container service to your organization… https://t.co/1T1GP3Xbmj

@sangeetan: Nobody escapes chaos monkey! Uncovering failures in Titus container runtime @amit_joshee #qconnyc https://t.co/H65aHPh8F4

@danielbryantuk: Building upon solid open source technology is core to the success with containers at Netflix @amit_joshee #qconnyc https://t.co/apRpwnPptm

@danielbryantuk: Netflix used shadow traffic to test running containers, with prod traffic initially still running on VMs… https://t.co/XNq3Cc5euM

@danielbryantuk: It is vital to focus on core business value provided by the team, even if we are technology-focused @amit_joshee… https://t.co/nhaM6sts9p

@danielbryantuk: Great takeaways by @aspyker and @amit_joshee about running containers at Netflix scale #qconnyc https://t.co/eqdYiaKqh7

Managing Millions of Data Services @Heroku

by Gabriel Enslein

Jeanne Boyarsky attended this session:

AWS S3 failure

February 28, 2017 – AWS S3 outage – pager duty failed to give message
Down for about six hours
Heroku recovered before everyone went to bed (10pm Eastern)
Avoid failure by having failover strategies
It would have taken 35 years to recover if all the tasks had to have been done manually
No Heroku customers lost any data…

Key takeaways

Automate yourself out of regular operations
Have emergency automation in place – scripts, jobs, etc.
Make routine failover strategies
Treat infrastructure as full units
Abstractions have their limits

Twitter feedback on this session included:

@RichardChmielek: Automate yourself out of job... This have been my profile headline on @LinkedIn for a while.Thanks for sanity check Gabriel https://t.co/unStFLW8tn

@danielbryantuk: Automate yourself out of a job, and avoid "I'll *just* do this operation..." to remain fault tolerant at scale https://t.co/C3A5O5GOyP

@jeanneboyarsky: 35 years if doing al 2/28/17 recovery work for Heroku manually #QConNYC

@jeanneboyarsky: Manual fixes can cause more problems than started with. Immutuable infrastrucure enforces the "just" #QconNYC

@sangeetan: Favor consistency,failover strategies,Infra as code & Testing for resilient systems at scale Gabe Enslein #heroku #QConNYC

@danielbryantuk: Don't stick in an ops job you don't like waking at 4am to fix issues. Automate yourself out of the job #qconnyc https://t.co/hzRIaYrSyT

Serverless Platform: Scientific Computation @scale

by Diptanu Choudhury

Twitter feedback on this session included:

@danielbryantuk: Developers need a basic understanding of the underlying infrastructure fabric. For example, the thread model impact https://t.co/4X4ugdYs2T

@sangeetan: @diptanu describing components of custom serverless platform at NASA #qconnyc https://t.co/7W3MwamYiw

@PhillipaAvery: @diptanu answers: 'Serverless from who's perspective?' #qconnewyork #QconNYC https://t.co/FXNNNht8V0

@danielbryantuk: Bin packing in a cluster only works effectively when you have a range of jobs with diverse resource requirements https://t.co/JcOzHZTj2q

@danielbryantuk: I've built several schedulers, and I never sleep well during the creation. Schedulers are *very* difficult to get right https://t.co/aEARXyLMdI

Innovations in Fintech

Digital Assets: 7 Lessons in Securing What's Next

Twitter feedback on this session included:

@danielbryantuk: Remove human SPOF and move fast with consensus engineering, e.g. shamir secret sharing like @HashiCorp Vault” @rwitoff #qconnyc https://t.co/uH30DEKBMq

@danielbryantuk: Do not reconfigure your infrastructure - rebuild it. Regularly rolling immutable infrastructure can mitigate breach effectiveness" #qconnyc https://t.co/dJHJn82TbL

@danielbryantuk: Great point by @rwitoff at #qconnyc on crypto currency-based enterprises - keep the majority of funds offline. Focus security accordingly https://t.co/P7MxtfJcA4

@danielbryantuk: Fantastic advice on building a business around crypto currencies by @rwitoff at #qconnyc https://t.co/QS4MxXMxq5

Java @speed: Get the Most out of Modern Hardware

by Gil Tene

Twitter feedback on this session included:

@danielbryantuk: Fascinating insight into how the Java JIT optimises and deoptimises code, and the impact it causes. @giltene at… https://t.co/NXyfnuKFSo

Java - Propelling the Ecosystem Forward

Fearless AWS Lambdas

by John Chapin

Twitter feedback on this session included:

@rmn: Benchmark your lamda perf over a long period of time #qconnyc https://t.co/N7q7v9fUUK

@danielbryantuk: Interesting insights into performance of @awscloud Lambdas running a Java benchmark, via @johnchapin #qconnyc https://t.co/kqKgLmNiB6

@danielbryantuk: Interesting to learn that AWS Lambda sets the JVM runtime heap size to 85% of memory assigned... @johnchapin… https://t.co/MY5AgXjw3a

@danielbryantuk: The AWS Lambda diet for Java programs great commentary on how to improve start-up time for Java functions, via @johnchapin #qconnyc https://t.co/tAKBk3Jkex

@danielbryantuk: Watch out for DoSing your Cloudwatch service when trying to capture business metrics. Use Cloudwatch Metric Filter… https://t.co/uVXWW9NpV5

@mikebroberts: “Look at this chart! Just look at it!” Says @johnchapin , bringing the #Serverless truth to #QConNYC https://t.co/INgWiAQOXi

Migrating Speedment to Java 9

by Dan Lawesson

Twitter feedback on this session included:

@jeanneboyarsky: I like that he showed the assembly code and explained the relationship to a simple for loop. #QconNYC

The Java Evolution of Eclipse Collections

by Kristen O'Leary

Jeanne Boyarsky attended this session:

Eclipse Collections

Was once GS (Goldman Sachs) Collections
Memory efficient collections framework
Open sourced in 2012…

Collectors

Collectors2 has collectors
Can collect into Bag, ImmutableSet, BiMap, Stack, etc
Have full primitive collections library
Have full set of multi-map (map with multiple values for same key ex: key to list/set)
Have extra APIs like chunk() and zip()…

Java 9

Module system
Internal API Encapsulation
Need to change APIs that use reflection in order to build. Can’t call setAccessible(true) any more
There is a command line argument to ignore reflection errors. Don’t want to impose that on callers

Microservices: Patterns & Practices

Dennis Doomen attended this track:

QCon is not QCon without a decent coverage of microservices, and this year was no different. In 2014, the conference was all about the introduction of microservices and the challenge around deployment and versioning. Since then, numerous tools and products emerged that should make this all a no-brainer. … Fortunately, the common agreement is that microservices should never be a first-class architecture, but are a way to break down the monolith. …

I've noticed an increasing trend to use code generators to generate most of the client and service code based on some formal specification. Those handle the serialization and deserialization concerns, but also integrate reliability measures such as the circuit breaker pattern. And this is where the new kid in town joins the conversation: gRPC. Almost every company that talked about microservices seems to be switching to gRPC and Protobuf as their main communication framework. In particularly, the efficiency of the wire format, its reliance on HTTP/2, the versioning flexibility and gRPC's Interface Definition Language (IDL) are its main arguments.

Shachar Tal attended this track:

My key takeaways around microservices architectures were:

- The key differentiation (that one should pursue) between microservice architecture and “SOA” is the segregation of microservice data into data stores that are not accessed by any means other than published APIs.

- State changes, and its consumption as events, should be treated as a first-class citizens of microservice design, not just query and mutation APIs. These allow the composition of higher-level workflows, as required by the design of other microservices or by the business logic. Allowing versioning around events (produced AND consumed by a microservice) is crucial to evolvable API design….

- For a microservice-based system you need to have a supporting org structure. Among other aspects, this means teams operating and monitoring their own services. In some organizations this happens bottom-up (grassroots growth) and without central direction, resulting in many ways to build, test, ship. This is a great pattern, according to many, as it allows the organization to figure out what’s good for it quickly and effectively since this happens in parallel, and then the best approaches get cross-pollinated into other teams.

- You need to have a TDD mentality and good coverage of tests to be able to move quickly and refactor mercilessly, without fear of breaking things. Make a lot of small changes and ship them; for that you need to have continuous delivery in place to reduce friction.

Managing Data in Microservices

by Randy Shoup

Twitter feedback on this session included:

@charleshumble: We don't have time to do it right. "Do you have time to do it twice?” #qconnyc

@charleshumble: Stitchfix uses TDD & doesn't have a bug tracking system. Bugs are fixed when they come up. Backlog contains features. @randyshoup #qconnyc

@charleshumble: We use Ruby on Rails. Rails *really* wants you to build a monolith. And we *really* didn't at Stitchfix. @randyshoup #qconnyc

@charleshumble: There are lots of definitions of #DevOps. This is mine. End-to-end ownership. I'm done when the service is retired. @randyshoup #qconnyc

@charleshumble: eBay on 5th architecture. Perl--> Monolithic C++ - we were hitting compiler limits.--> Java --> microservices @randyshoup #qconnyc

@philip_pfo: DevOps = team owns service from design to deploy to retirement. Devs are responsible for the full SDLC. @randyshoup #qconnyc

@charleshumble: Twitter - 3rd generation - Rails—> JS/Rails/Scala –> microservices @randyshoup #qconnyc

@charleshumble: First law of distributed object design: Don't distribute your objects - @martinfowler quoted by @randyshoup #qconnyc

@charleshumble: If you don't end up regretting your early technology decisions you probably over engineered. @randyshoup #qconnyc

@philip_pfo: Events (state changes) are a new tier that we should start making a first class part of our architecture. @randyshoup #qconnyc

@charleshumble: Stitchfix has a Monolithic shared DB. Solution is decouple applications/services from a shared DB. @randyshoup #qconnyc

@charleshumble: Microservices - shared data: Single system of record. Every other copy is a read-only, non-authoritative cache.@randyshoup #qconnyc

@charleshumble: In a monolithic database joins are very easy. Spltting the data into services makes joins hard. @randyshoup #qconnyc

@azenker: microservices are nothing more than SOA done right @randyshoup quote in microservices track #qconnyc

Take Two: Evolving Microservice Architectures

by Andrew Hart

Twitter feedback on this session included:

@michellebrush: Things that would seem to be trivial become technically challenging due to lack of standardization. - @andrewfhart #qconnyc

@ZooFood: Evolving an architecture might require changing the organization. - Andrew Hart #qconnyc #Microservices

The Paved PaaS to Microservices at Netflix

by Yunong Xiao

Jeanne Boyarsky attended this session:

Scale. 1K+ microservices. JavaScript front-end; Java back-end.

Client teams own microservices for front end/client devices. Edge API is server based/back end API.

Standardized components

Common set of components – ex: metrics, logging, alerts, dashboard, configuration
You don’t want everyone picking RPC mechanism; invest in one set of tooling
Benefits of standardizing: consistency (ease of troubleshooting/support), leverage (more time for business problems with platform team focused on components), interoperability (easier integration so faster velocity and less cognitive overload), quality (more investment so higher chance of quality), support (greater knowledge base)
“But I’m a Snowflake” – Netflix has culture of freedom and responsibility. This helps with innovation. If it works, re-integrate into ecosystem. Be conscious of talking to the others and the cost to other teams of your choice.

Pre-assembled platform

On starting a new project, there isn’t velocity yet nor stats on reliability.
Putting components into a pre-assembled platform so you can just add business logic. You will be less likely to be missing things like metrics because of pre-assembled platform.
Guarantees standard and consistent metrics. Reduce MTTD (mean time to detect) and MTTR (mean time to recover)
Maintenance vs convenience – easier for platform team to include just basics. Easier for app team to have more included. The solution is layers and favors. Having a base platform and add ons.
Testing – running other team’s microservices tests when upgrading a platform tests the platform upgrade
Design for configuration and hooks
API Semantic versioning (like what Maven versions do)
Use conventional changelog to automatically create changelog

Automation and tooling

Provide CLI for common dev experience – allows scripting and consistent environment each time. Ensures can run locally and in the cloud
Local development fast – use local debugger and curl. Can still test with container so mirrors prod config
Provide first class mocks for components in pre-assembled platform. Facilitate determining where the problem lives. Easier to write reliable automated tests.
Provide facilitate to generate mock data
Need a testing API so different groups can work on mocks. Component teams create mocks against standard APIs
“Production is war” – different levels of experience in ops. Use tools to avoid shooting off own foot. Ex: pipelines for deployment/rollback, automated canary analysis
Dashboards provide a consolidated view for ops. Having platform generate dashboard and alerts standardizes them. Also allows for automate analytics and tooling

Provided warning about not just copying Netflix. PaaS is for certain use cases. Components help regardless.

Twitter feedback on this session included:

@danielbryantuk: Why standardize (or not) with a PaaS? @YunongX discusses the tradeoffs at Netflix during #qconnyc https://t.co/dOoXLEdF9Y

@danielbryantuk: A good argument for providing core infrastructure and platform (with sensible defaults) as a service from @YunongX… https://t.co/dcvh4LUWXh

@danielbryantuk: How should you test your platform? By 'dog fooding' your own services of course... @YunongX from @NetflixOSS at… https://t.co/MRXkUCIrUh

Modern CS in the Real World

Homoiconicity: It Is What It Is

by Stuart Sierra

Henrik Warne attended this session:

Stuart Sierra does a deep-dive into the origins of homoiconicity (code and data represented the same way). The talk started off showing the slides of the presentation as data that could render itself. Next, where examples of languages that are homoiconic to some degree: Clojure, Lisp, Tcl, and TRAC (a programmable text editor from the 1960s). Sierra then traced the idea back through history thanks to some impressive detective work, and finished with some general reasons why homiconicity is useful. Both interesting, very entertaining, and extremely well done.

Next Gen APIs: Designs, Protocols, and Evolution

Beyond REST: Coursera's Journey to GraphQL

Henrik Warne attended this session:

Bryan Kane from Coursera described how they started using GraphQL. Their site was getting slow because of the many separate API calls needed to display a page. To speed up the page loads, they decided to use GraphQL to be able to fetch related data in a single API call. This should be possible in principal, but they ran into problems because the service had not been designed for GraphQL from the beginning. This talk was well presented, but I particularly liked it because it highlighted the problems they encountered as well. Too often talks only focus on clean solutions, instead of also describing the cases that aren’t so easily solved.

Optimizing Yourself

Deep Listening: Creating Conversational Agility

by Brian Branagan

Twitter feedback on this session included:

@shanehastie: #QconNYC @BrianBranagan One third of projects fail because of poor communication and ineffective listening https://t.co/4jHocZXdOf

@jeanneboyarsky: Project failure lingers as a source of agitation even when move on to other projects. Like a low priority background process #QconNYC

@shanehastie: #QconNYC @BrianBranagan Telling a good story of listening and conversations. Good quote https://t.co/Kfst5AUHXZ

@shanehastie: #QconNYC @BrianBranagan Listening is more than hearing the words - pay attention to voice, tone, body language & emotions

@shanehastie: #QconNYC @BrianBranagan 3 Layers of Deep Listening - each layer contributes to the next https://t.co/O8Pz7mI3Qn

@shanehastie: #QconNYC @BrianBranagan layer one of Deep Listening- the body. Layer 2 – emotions. Layer 3 - our speaking

@shanehastie: #QconNYC @BrianBranagan Layer one - the body. https://t.co/VajhCBcxB7

@jeanneboyarsky: Body pattern matches for similar emotional signature. This makes us upset based on past.Predisposes us for certain kinds of action. #QconNYC

@jeanneboyarsky: Four basic emotions: fear, anger, sad, glad - we need more positive emotions! #QconNYC

@shanehastie: #QconNYC @BrianBranagan Layer 2 our emotions 4 basic emotions: Fear, anger, sadness, gladness

@shanehastie: #QconNYC @BrianBranagan Layer 3 - speaking https://t.co/AlE2mvbc9Z

@shanehastie: #QconNYC @BrianBranagan Three types of speaking https://t.co/U5AbDfK7jX

@shanehastie: #QconNYC @BrianBranagan Conversational agility - moving from Me to We https://t.co/tFRWau3FLz

@shanehastie: #QconNYC @BrianBranagan Simple advice for better listening https://t.co/041FJT8A5C

@shanehastie: #QconNYC @BrianBranagan Useful references https://t.co/OJ0BBYtvzw

Getting Old(Er) in Tech: Staying Relevant

by Don Denoncourt

Dennis Doomen attended this session:

Denoncourt shared some of the great stories of people who loved what they did until they died of old age, and emphasized that failure makes you stronger. So keep challenging yourself and keep exercising your brain. If you don't, you'll loose the ability to learn.

Denoncourt had a couple of suggestions of how to do that. First of all, he wanted us to look at what the best in our profession do and follow their lead. For example, they might read code for fun, they might join or start one or more open-source projects, they write blog posts or speak at user groups and conferences. They also discover technologies that other team members are adept at. And finally, they mentor other people. Next to that, Denoncourt wants us to understand our learning style. If you’re an auditory person, class room session, audio books or YouTube videos might be the thing for you. But if you're more like a visual person, books and articles might be better suited. However, if you're a kinesthetic type, doing some tech projects in the evenings is probably the most efficient method to gain new knowledge.

He also suggests to do short bursts of learning while waiting for our unit tests to complete, in between Pomodoros, between projects and while job hunting. He even wants us to do some learning during our commute, during workouts, or, when you are a (grand) parent like Denoncourt, while taking your (grand) kids out in the stroller. And if you're into a certain topic, be sure to apply multi-pass learning by reading more than one article or book on the same subject. And finally, to make sure you don't run out of learning material, stockpile books, blogs, online courses and videos. And don't forget to accumulate posts from web magazines, newsletters, conferences and seminar videos. A great tool to collect all of this is Pocket.

Twitter feedback on this session included:

@shanehastie: #QConNYC @denoncourt Average age statistics https://t.co/WUUIygs0Nv

@shanehastie: #QConNYC @denoncourt Strategies to remain relevant- lifelong learner, don't be a dojo champion - be ready to learn from anyone

@jeanneboyarsky: Cross boundaries - Every time you fail, your brain gets stronger. You might not learn, but still stronger. #QconNYC

@shanehastie: #QconNYC @denoncourt staying relevant - share knowledge freely, cross boundaries, challenge yourself

@shanehastie: #QConNYC @denoncourt Challenge yourself- what is the next thing you need to learn about. https://t.co/19eHKhlroT

@shanehastie: #QConNYC @denoncourt Make sure you get 10 years experience, not one year 10 times https://t.co/dCfXGOjNmY

@jeanneboyarsky: After doing 1 year 10 times, folks often lose the ability to learn #QconNYC

@shanehastie: #QConNYC @denoncourt If you plan on being in the IT industry more than 10 years you need to become a lifelong learner https://t.co/AhCLpiENh6

@shanehastie: #QConNYC @denoncourt "You're only as good as your last two years of accomplishments" Ref Kent Beck on pair programming.

@shanehastie: #QConNYC @denoncourt On learning - know your own learning style.

@shanehastie: #QConNYC @denoncourt We remember the first and last things we study. https://t.co/l0bvjTPLxo

@shanehastie: #QConNYC @denoncourt Find opportunities to "surge" your learning- a blast of concerted learning on a specify topic… https://t.co/O5duV9hMfU

@shanehastie: #QConNYC @denoncourt Ideas on stockpiling learning resources https://t.co/RVcrdt7sHp

@shanehastie: #QConNYC @denoncourt Hack your commute, have a multi-threaded life. https://t.co/Nl66yl1bqB

@shanehastie: #QConNYC @denoncourt On hiring - be upfront about your age. https://t.co/Ywm47962U7

@shanehastie: #QConNYC @denoncourt Be and look fit, don't worry about looking younger https://t.co/gXqW8IbURX

@shanehastie: #QConNYC @denoncourt Having hobbies makes your mind grow

Practical Empathy: Unlock the Super Power

by Pavneet Singh Saund

Twitter feedback on this session included:

@shanehastie: #QConNYC @pavsaund Definition of empathy https://t.co/8Ek0zt7y8D

@shanehastie: #QConNYC @pavsaund Building empathy: Introspection & mindfulness https://t.co/BwaBu0Ua7L

@sangeetan: Before you can show empathy towards others you need to have empathy for yourself. @pavsaund #QConNYC

@shanehastie: #QConNYC @pavsaund Mindfulness defined https://t.co/THV4DFko2g

@shanehastie: #QConNYC @pavsaund aspects of mindfulness https://t.co/PHjRfnrOMv

@shanehastie: #QConNYC @pavsaund Listening to understand https://t.co/UkyXXDbVkQ

@jeanneboyarsky: Don't just do something, listen. As developers, our first reaction is to solve problems. #QconNYC

@sangeetan: Don't just do something; listen! & do so without judgement @pavsaund #QConNYC

@shanehastie: #QConNYC @pavsaund Listen to connect https://t.co/f4aVbJcHDO

@shanehastie: #QConNYC @pavsaund Gain perspective- challenge comfort zone https://t.co/InuwJ6cGM4

@shanehastie: #QConNYC @pavsaund Walk a mile in their shoes. Sit with your users https://t.co/DEzWlrXtA4

@shanehastie: #QConNYC @pavsaund The level of human connection- blame & shame prevents empathy https://t.co/TvMLQLIoJq

@shanehastie: #QConNYC @pavsaund Any step you take towards empathy will make life better for someone around you https://t.co/jB73ugpGU1

@jeanneboyarsky: Empathy is the key to understanding. "Empathy is a skill. You need to actively practice giving & receiving" Empathy is hard #QconNYC

@shanehastie: #QConNYC @pavsaund empathy is a choice, it takes courage to have empathy as a guiding principle

The Effective Remote Developer

by David Copeland

Henrik Warne attended this session:

The whole development team at Stitch Fix is remote, and Copeland has many years’ experience being a remote developer. This talk contained a lot of concrete, practical advice on how to make it work. For example, publicize your working hours so people know when to expect answers. Learn how to screen cast and to use diagramming tools (works better than text for some people). Invest in a good microphone. In video conferences, use nouns instead of pronouns (avoids confusion), and ask to have the camera pointed to the speaker if it isn’t. This was a good “experience talk”, with lots of practical advice that comes from personal experience.

Twitter feedback on this session included:

@jeanneboyarsky: You are remote if you do not often interface face-to-face with the people you work with #QconNYC

@shanehastie: #QConNYC @davetron5000 What does "effective" mean? https://t.co/UnaPeJgPxZ

@shanehastie: #QConNYC @davetron5000 This is not easy - needs upkeep https://t.co/pArp3iiTYB

@shanehastie: #QConNYC @davetron5000 Some of the benefits. https://t.co/0iTG6O64lh

@shanehastie: #QConNYC @davetron5000 To make remote work work you have to build and maintain trust. Trust is the most important factor.

@jeanneboyarsky: the half life of trust is six weeks - it must constantly be replenished #QconNYC

@shanehastie: #QConNYC @davetron5000 Four mindsets to build and maintain trust https://t.co/4Vh1rF4Id7

@shanehastie: #QConNYC @davetron5000 Technology helps - you need a base level of tools. https://t.co/F5ZjTKsirf

@jeanneboyarsky: turn big projects into smaller ones; we tend to turn small projects into big ones #QconNYC

@jeanneboyarsky: nice idea; mental SLA for different types of communication. will scan inbox every X. (but not necessarily respond) #QconNYC

@jeanneboyarsky: Assume good intentions - we are not good at critiquing. remember the reviewer is trying to help even if sounds cold and harsh. #QconNYC

@shanehastie: #QConNYC @davetron5000 Code is the work product of development. Advice on coding https://t.co/eJEgUMCDe8

@jeanneboyarsky: in code reviews, remember to give *specific* affirming feedback and not just "good job" #QconNYC

@shanehastie: #QConNYC @davetron5000 Advice on asynchronous communication https://t.co/FYoresEawG

@shanehastie: #QConNYC @davetron5000 Advice on synchronous communication https://t.co/X3AQC3dffA

@jeanneboyarsky: If in office, people still know who you are and that humanizes you. If remote, you are a producer of emails. "person; not perl script" #QconNYC

The Marriage of Communication and Code

by M. Scott Ford & Andrea Goulet

Dennis Doomen attended this session:

One third of all projects fail because of poor communication and ineffective listening. … And to be clear, failure includes missed deadlines and overrunning budgets, and seems to be pretty traumatic. They also told us that the outcomes we experience, both individually and socially, come from conversations we had, didn't have, which went well and didn't go so well. So being able to become more effective in communication is a pretty important skill to learn.

Ford and Goulet shared a couple of techniques and tips to help you with that. First of all, you should try to see each other's potential before commencing in a fruitful discussion. Just by thinking about that person's possibilities rather than focusing on their bad habits can change the way you approach a person. Next to that, it's useful to understand the speaking types. According to the model Goulet uses, people can be transactional where the interaction is based on asking questions and telling people what needs to be done. People can also be positional where they advocate their opinions from a particular role or position. And finally, some people are transformational in which they share a vision and try to steer the discussion in a direction that aligns with that.

Emotions are part of a face-to-face interaction and can negatively influence your ability to communicate effectively, so it's imperative to transform an agitated response to a state of deep listening. If you do feel agitated, Goulet strongly recommends to pause, feel your feet, watch your breath and to remember what you care about. To help us understand how our body, our emotions and what we are saying work together, she shared a model where each layer contributes to the next. Our body is the first layer and informs us about threats to our safety. It helps us to identify our friends or foes and lets us know how to fit in. It provides us with a sense of reality and helps us to make judgement calls. Our emotions form the 2nd layer and provide us with biological reactions to circumstances that we perceive as fear, anger, sadness or happiness. Our speaking represents the 3rd and last layer and allows us to express our opinions, say what we think what is and what is not factual in our world. It also allows us to ask people to do things for us and gives us the ability to commit ourselves to do things for others.

Pretty abstract ideas, right? Fortunately, they had some simple and actionable tips as well. For instance, instead of stating your frustration as a fact, they advised us to use phrases like "I feel….when you…..because", or to respond to proposals you don't (entirely) agree with using an affirmative statement followed by "and" instead of "but". And apparently we also need to be humble, helpful and immediate (instead of sugar coating things). And when we do have criticism, keep praising people publicly and save that criticism for a private setting.

Twitter feedback on this session included:

@shanehastie: #QconNYC @andreagoulet @mscottford Framework 1: "I" statements. https://t.co/w1PJX8YRZN

@shanehastie: #QconNYC @andreagoulet @mscottford I statement in a code review. https://t.co/T3cvIuPLIr

@shanehastie: #QconNYC @andreagoulet @mscottford Framework 2: Listen, verify, validate. https://t.co/uya2vrJNmW

@shanehastie: #QconNYC @andreagoulet @mscottford Example of Listen , Verify, Validate https://t.co/uQfQ1iLpHm

@shanehastie: #QconNYC @andreagoulet @mscottford Framework 3: Shame vs Guilt (sounds a lot like fixed vs flexible mindset) https://t.co/Baak2bkYrm

@ddoomen: Shame corrodes the very part of us that believes we capable of change #qconnyc

@shanehastie: #QconNYC @andreagoulet @mscottford Blame has an inverse relationship to accountability https://t.co/XudSNaGbWd

@shanehastie: #QconNYC @andreagoulet @mscottford Framework 4: Yes, and ...The rules of Improv. Never say yes, but... https://t.co/7cpkFRz59u

@shanehastie: #QconNYC @andreagoulet @mscottford Allow happy accidents https://t.co/iJ0eSswpvG

@shanehastie: #QconNYC @andreagoulet @mscottford Framework 5: Radical candor. Care personally and able to challenge directly https://t.co/P47e2SI4kJ

@shanehastie: #QconNYC @andreagoulet @mscottford Radical candor - don't be a jerk https://t.co/4rjUKWXetT

@shanehastie: #QconNYC @andreagoulet @mscottford Framework 6: Inception Interruptions(Ref Inception movie) when you go deep you need to surface slowly

@shanehastie: #QconNYC @andreagoulet @mscottford Name the layers and indicate your level of Inception- agree interruptible level. https://t.co/b0DDBsGWIG

@shanehastie: #QconNYC @andreagoulet @mscottford Communication is a skill - it can be learned and is hard to learn. https://t.co/wcXZEoOnO4

Stream Processing at Large

Adopting Stream Processing for Instrumentation

by Sean Cribbs

Twitter feedback on this session included:

@michellebrush: Something is wrong with my super important metric but I think it might be getting better. Art courtesy of @seancribbs #qconnyc https://t.co/EwjlFRReyf

@michellebrush: .@seancribbs sharing how Streaming systems are a good paradigm for monitoring and metrics #qconnyc https://t.co/c4RsfEJrlb

@michellebrush: We spend more time reading our code than writing new code, so we need to adopt Knuth's advice. @seancribbs #qconnyc https://t.co/HbpaTCqEiv

Streaming Microservices: Contracts & Compatibility

by Gwen Shapira

Stephen Steen attended this session:

Everyone across the board said they are either moving to a microservices architecture in the near future, or are in the process of doing so now. This is great because it allows teams to maximize a mix of off-the-shelf APIs (everything becomes an API) and seamlessly integrate into in-house solutions. The challenge is updating up/downstream services with changes in schema. In compatibility testing, these can come back as “ok” when in reality they are broken. As Gwen Shapira put it, schemas should be treated with all the first class citizenship of an API through schema serializers and registries. This allows everyone to check compatibility, both producers and consumers, without breaking the production system later.

Ask Me Anything (AMA)

Dennis Doomen attended this track:

Another interesting trend at QCon was the increased focus on reducing overhead by dropping a separate development, testing and acceptance environments while trying to bring something into production. Many companies have found that those staging environments don't really make their product better and have a lot of drawbacks. They are often perceived as a fragile and expensive bottleneck. And when something fails, it is difficult to understand failure. The problems companies find in those environments are not that critical and never as interesting as the ones that happen in production. In fact, they might even give the wrong incentive, the one where developers rely on some QA engineer to do the real testing work on the test environment, rather than invest in automated testing.

According to their talks, both the Gilt Group and Netflix seem to wholeheartedly support this mindset by working according to a couple of common principles. For starters, teams have end-to-end ownership of the quality and performance of the features they build. In other words, you build it, you run it. Teams have unfettered control to their own infrastructure. They assume continuous delivery in the design process. For instance, by heavily investing in automated testing, employing multi-tenant Canary Testing and making sure there's one way to do something. A nice example that Michael Bryzek of Gilt gave was a bot that would place a real order every few minutes and then cancel it automatically. Teams also act like little start-ups by providing services to other dev teams. This gives them the mentality to try to provide reliable services that are designed to allow delay instead of outage. They may even decide to ship MVPs of their services to quickly help out other teams to conquer a new business opportunity, and then mature their service in successive releases.

Workshops

Zen of Architecture

by Juval Lowy

Twitter feedback on this workshop included:

@ddoomen: Reuse doesn't exist in functional decomposition, so why use that to drive architecture Juval Lowy #qconnyc

@ddoomen: If they can't get the code of the entire system on my laptop in a couple of minutes, that company has already failed Juval Lowy #qconnyc

@ddoomen: In software, the inmates are running the asylum Juval Lowy #qconnyc

@ddoomen: This workshop by @juvallowy is more like stand-up comedy. It's simply hilarious the stupid things we do as software architects #qconnyc

@ddoomen: A quack with a blog is still a quack Juval Lowy #qconnyc

@ddoomen: Functional decomposition never worked. Never, ever. So why are still doing it? Juval Lowy #qconnyc

@ddoomen: The Zen of Architecture according to @juvallowy: Decompose Based On Volatility #QConNYC

@ddoomen: If you're only doing something 2% of the time, you'll never going to be good at it. Don't bother and let somebody else do it #qconnyc

@ddoomen: Microservices will be the largest failure in the history of software architecture Juval Lowy #qconnyc

Community Night

Papers We Love

by Matt Adereth, John Langford & Gwen Shapira

Twitter feedback on this meetup included:

@danielbryantuk: Great "papers we love" talk about stream processing at Facebook by @gwenshap at #qconnyc. Sounds like more papers should be written! :-) https://t.co/nRMY60VBj0

@shriyarora: @gwenshap is giving us #speakergoals at #QConNYC, not in her paper review-which is super-but in how much fun she's having up there!

@danielbryantuk: Events and phrases of data tell a story that a simple metric can't. We often have to apply a narrative to dashboards" @mipsytipsy #qconnyc https://t.co/TpVZAFyc4N

@danielbryantuk: Observability tools from the LAMP stack era are not only showing their age, they are knocking on death's door @mipsytipsy #qconnyc

Opinions about QCon

Henrik Warne’s impressions about QCon were:

Just as when coding, the details matter a lot to the end result for a conference too. At QCon New York, they had paid attention to a lot of the small details. This added up to a really good experience overall:

Excellent food, especially the lunches. Much better than what I have experienced before.
Quick presentations of all the upcoming talks within the tracks before the morning keynotes.
Badges with NFC for green/yellow/red feedback after each session (and staff that made sure everybody had a chance to give feedback).
Badges with easy-to-read names of the participants and companies – helpful for starting conversations.
Site where you select and build your own schedule (but you still see what the other talks are).
Short interviews with the speakers published together with the talk outlines.
A good mix of male and female track leaders and presenters.
Very central location (by Times Square).
Five parallel tracks, so many interesting talks to choose from.
Swag – no useless trinkets or weird-shaped bags, just a functional back pack.
Videos available (for attendees) almost immediately (and will become available to the public in the next few months I believe).
A gong sounded five minutes before the next session started – useful to be reminded.

Impressions expressed on Twitter included:

@danielbryantuk: Loving the new venue for @qconnewyork , and great location in Times Square! #qconnyc https://t.co/N2mjMnpmhm

@matt_sakaguchi: was honored to give the keynote today at #QConNYC. #makeworkbetter @infoq such great hosts. https://t.co/o4CdwSBlzU

@adrian_trenaman: Love the real-time feedback on talks at #QConNYC. Great way to sharpen the game and reinforce a very high standard. Loving today!

@KevinHoffman: Sitting in a rotating restaurant above times square, celebrating delivering my #qconnyc talk on #golang #microservices

@mgroves: It was great hanging out with you #QConNYC! Thank you speakers, sponsors, volunteers, attendees, and organizers for a great conference! https://t.co/dwJZK1uAwL

Takeaways

Brett Uglow’s takeaways were:

Overall I would say that attending QCon NY 2017 was a fantastic experience. I’ve been exposed to lots of new ideas and ways of working that I ordinarily would not have heard of for another 2+ years. Getting rid of staging environments resonated the most with me, as I’ve experienced first-hand how costly they can be while still not preventing production bugs.

Ivan Campos’ takeaways were:

After digesting the QCon New York experience, I distilled the takeaways into three themes—ephemeral, chaotic, and secure.

The rise of serverless architectures, microservices, containers, and cloud computing, have ushered in an expectation that:

The lifespan of servers will be short-lived
The number of connected devices/servers/services will continue to grow
Securing our infrastructure and applications remains as important as ever

Shachar Tal’s rakeaways were:

Stream processing adoption is very much still an open problem. The areas where it is widely understood and successful are often where the data is stupendously simple in structure (e.g. clickstreams, page view data), and so consumption graphs and schema evolution are less of a compounding issue. …

Many organizations are doing their own thing to manage containers in production, especially around scheduling and workload management, rather than leveraging projects like Kubernetes. Some of that is due to the maturity of these projects at adopt-or-build time, but some is due to a common perspective that the body of work they would need to build around these tools to support their specific needs will dwarf the value derived from latching onto existing projects, comparing the learning curve vs. the accelerated value add.

Takeaways from QCon New York included:

@jonathansen: Thanks for making the #qconnyc High Velocity Devs track so great @debbiemadden200 @catswetel @cms5109 @jchyip and @Ops_Engineering!

@ccheng52: Enjoyed listening to the amazing speakers on this track @qconnewyork #qconnyc https://t.co/HYPy3eg8jS

@randyshoup: Overlooking Times Square after the best #qconnyc yet https://t.co/XEm1OckZGu

@legege: #QConNYC is already over. Learned a lot. Very impressed by the quality of presenters, tracks and the overall organization!

@fagerholm: Attended #qconnyc for the first time this week, and will strongly recommend it. Located on Times Square and filled with interesting topics!

@haacked: This was my fifth time as a QCon track host and as always, a wonderful time with thought provoking conversations! #qconnyc

@BrianBranagan: Thank you #QconNYC for Creating WE by hosting a great conference full practical actions to implement cutting-edge technology https://t.co/ZP6WZ9kXCK

@alexnellefson: Loved the learning intersection of intelligences from technical, practical and emotionally natured perspectives presented at #qconnyc. ??

@pavsaund: Bye #qconnyc! Lots of wonderful people and great conversations! Hope to see you again! https://t.co/aY4szRqgZ3

@ccheng52: And that's a wrap. Had a blast volunteering and learning at #qconnyc can't wait till next year!

Conclusion

QCons are produced by InfoQ.com. Our focus on practitioner-driven content is reflected in the fact that the program committee that selects the talks and speakers is itself comprised of technical practitioners from the software development community. .

We will next be in Shanghai, Oct 19 - 21 and then San Francisco Nov 13 - 17 this year. QCon New York will return around June of 2018.

How could we improve? Take the InfoQ reader survey

Each year, we seek feedback from our readers to help us improve InfoQ.
Would you mind spending 2 minutes to share your feedback in our short survey?
Your feedback will directly help us continually evolve how we support you.
The InfoQ Team
Take the survey

InfoQ Software Architects' Newsletter

Login with:

Don't have an InfoQ account?

Key Takeaway Points and Lessons Learned from QCon New York 2017

Write for InfoQ

Related Sponsored Content

Rate this Article

This content is in the QCon Software Development Conference topic

Related Topics:

Related Editorial

Popular across InfoQ

How could we improve? Take the InfoQ reader survey

The InfoQ Newsletter