1. [...]Do you mind briefly introducing yourself to our audience?
Manuel's full question: Hi. I'm Manuel Pais, and I'm here at QCon London 2016 with Aaron Bedra, Chief Security Officer at eligible.com. Thanks for accepting our invitation. Do you mind briefly introducing yourself to our audience?
Sure! So my name's Aaron Bedra. I'm from Chicago in the States. I work for a company called Eligible in New York City, a health care information management company, really kind of changing the way we can get doctors paid.
Property-based testing is really an abstraction on top of traditional unit testing. It's the idea of being able to describe your system as a series of constraints or properties and being able to allow the tools to generate input data versus specifying them. So removing the magic numbers, removing kind of the details and really focusing on what the essence is of what you're trying to test. So it gives you a lot more coverage on different corner cases and different things that you may not think about because the tool is providing the input values versus you kind of making things up as you go.
You very well might. You might find an edge case that way and then maybe write a unit test that specifies that just one edge case and never think about it again. Or you find a regression and maybe you'd do that. This [property-based testing] is more kind of helping you find the things you wouldn't have found or wouldn't know about otherwise.
4. Is this just for unit testing or are there any other applications?
No. The application is really anything you want to test and it really focuses, like I said, on what you want to test, not how to test it. So it helps you with a lot of things. There's certainly the [input] value generation but it can give us details on order of operations changes in the code, figure out if you have race conditions or unsatisfied ideas or things that aren't quite settled properly.
There's no one state of the art thing for security testing. It's kind of a holistic thing. I actually do use these property-based tests in the security space. There's a lot of application there, software security is still a big piece of that. Security of our product in particular is important for Eligible. Property-based tests are important to me as well and for that space.
But it's a holistic approach, right? It's everything from your basic unit tests, to static analysis, to fuzzing, and even the property-based tools can give you that fuzzing idea as well. It's not quite a traditional fuzzer but it is a good way to think about a problem.
Then you've got third parties. People helping you test and people that aren't you that maybe can poke differently or maybe do something that you wouldn't normally do. So you rely on the tools you have, the right software, the tools you create around the process and even the folks you can hire to help you out externally. So you have a combination of everything that really kind of makes it good.
I love them. So preference is really not being the right word. I use them. You use them because it gives you quick feedback. Developers can run them on their laptops when they're writing code, we can run them in CI, we can run them before production. You can make them fast. You run the slower, longer tools in other cases but having that as close to the developer as possible, they're less likely to check in something that can quickly be found. And even when you put it in CI of course, you can have that as well but it really gives developers a chance to get that feedback fast.
7. [...]What do you think are the pros and cons of this kind of approach?
Manuel's full question: That actually leads me to my next question. We've seen some BDD-based security testing frameworks like Gauntlt or BDD-Security coming up. What do you think are the pros and cons of this kind of approach?
So I like Gauntlt. I think it's a good tool. My gripe about that is the cucumber style, really is what Gauntlt really tries to follow. I haven't had a ton of success with that in normal software projects. So I'm always a little hesitant because I've seen that fall apart in non-security focused areas. However, the tool itself is actually really quite useful.
I'm a big fan of what they've done with the tool and I think it's something everybody should think about using. I think there's a lot of advantage to that because you can really describe scenarios and say, “Hey, I want to make sure that this is behind the right authorization or authentication or this behaves in a certain way given these bad parameters.”
So it gives you a nice way to be able to focus on areas that you either aren't sure of or want to make sure you never really have an issue. You can have that as part of your overall smoke testing and other kind of go live testing.
I think it makes it easier to communicate with developers and that's probably where it stops. I think that you're still describing technical scenarios and scenarios where there's a technical application about control, right? So if you're talking to somebody non-technical about how authorization works, it's not going to scale well. So I think talking to developers, yes. It's absolutely a great mechanism but beyond that I don't think it has much merit.
9. [...]How do you see this changing?
Manuel's full question: In general, on this topic of how to get security teams to fit or adapt to today's world when everyone's going Agile, DevOps, Continuous Delivery - so at a faster speed of change when, traditionally, security teams have a different life cycle, typically performing their tasks and their tests at the end of the release. How do you see this changing?
So I don't see it changing. I see it completely starting over. Personally, I won't hire a product security person who can't build software. This is an engineering role. There are traditional security tasks that are not engineering roles but in terms of software security, anybody on a software team that I'm going to work with has to be a software developer and has to be good at it.
They have to be able to build systems, they have to be able to write code, they have to be able to jump into situations where ideas are foreign and understand them quickly. So for me, it's an absolute necessity that these are also engineers. They're also writing software, they're capable, they're respected by the development team, and they really are part of the development team.
So it's not a security team process. It's that the product security team is just another engineering team. That for me is a crucial element to making this all work because then it's part of the development life cycle. Once you have it outside that, you have conflicting forces and they really don't agree and the pressure and tenses are all in the wrong place and usually at the wrong time.
So for me, it's absolutely critical to have developers as part of software security team.
I'm a huge fan of bug bounty programs and responsible disclosure in general. I think that there's a few really important things here. One is having a channel for people to communicate with you that they found a problem. Having an open and safe channel where somebody can report this to you helps you get that information. When you close yourself off, when you make it unavailable, people may not be able to report, or may be scared to report . Making sure there's an open and safe communication channel, that's a huge deal for me.
There's other pieces to it that are good and bad but I think overall is a net positive. I think it lets people understand these bugs in greater detail, how they look like, write reports in the general industry. They can give you information so you can get better.
I'm a fan of paying bounties, I think that work is important. You shouldn't just say thank you, you should reward them, they found something on your site. Even if it's not a ton of money, it's still that sentiment that is important. At Eligible we have a bug bounty program, we have a responsible disclosure program and we do pay when somebody submits legitimate findings.
Manuel: It's actually interesting because at one talk about security here at QCon they were saying that people are normally looking for being in the hall of fame or getting recognition for finding a bug. So when you see people who are more focused on the money then probably something is sketchy there or maybe they're bluffing.
It's entirely possible. Everybody's motivated by something different. In some cases, some of these people who identify security breaches, this is their entire job. This is their livelihood, this is how they make their money. So sometimes they are motivated by how much they can get paid and that is where their paycheck comes from. So you have to understand the situation and where they're coming from. We treat everything equal as it comes in and if it's valid, that's great.
The thing that I think it's important to remember is that each company is responsible for how it wants to run its program. They can choose to accept or not accept certain things. You get to define the rules of engagement. You own the process and so I think that too many companies become beholden to the reporters when really, it's their process to own and to really kind of stick by. So it's important to be very firm about your process once you adapt it and found what works for you.
11. [...]Do you mind explaining a bit more what you mean by that, maybe with some concrete examples?
Manuel's full question: You're also a proponent of dynamic security response systems and I like this quote where you said “We need to rely more on the data that we have to make informed decisions about the intentions of users and respond in kind”. Do you mind explaining a bit more what you mean by that, maybe with some concrete examples?
Yes, absolutely. The data that we have and really, there's tons of it, right? On the web, every single request somebody makes is data for us. We can examine that. We can understand, we can really finger print, foot print, identify behaviors. Security is not a black and white thing where you have a control or don't have a control.
Security is the thing where you look at all the information in front of you and decide whether control should be applied or not but control is not something you apply to every user. A control is something you apply to reduce your risk. So if you have a great user who's not a threat to you, you reduce the controls and let them play. If you have somebody who seems adversary, you increase the controls and hopefully deter them from doing anything bad. That's really what should be going on.
I think my favorite idea change here is the mental model shift from the fortress to the casino. The fortress being keeping everybody out and the casino saying everybody can come in, we're just going to watch you while you're there and if something goes wrong, we'll make sure we respond to it but otherwise, come in and do your thing. And when you think about e-commerce and those kind of things, that applies very well.
Some sites, it doesn't apply to as much. Some sites are restricted to their business partners, they're restricted over API channels, that kind of stuff. Of course it becomes more of a fortress but you still want to watch. The thing is that applying controls based on data is still a very relevant thing.
12. [...]How do you minimize that complexity overhead?
Manuel's full question: One might say that can lead to more complexity to be able to support that kind of dynamic response based on the profile of who is using your application. How do you minimize that complexity overhead?
It is a complexity overhead, absolutely. It's another system. It's more software. Nothing about this is not complicated, or not complex. It actually isn't that complicated when you think about it. It's a pretty simple mechanism but it does add complexity to the system. And it's something you choose to do. If you have real threats, if you have something to loose, if you're big enough, if it makes sense to apply these kinds of controls, this kind of complexity, then you do it.
If you're a static site that has nothing behind it and you're not going to have any loss because of it, you may not want to do this. But if you have more to lose, if you have personal information, financial data, something on behalf of your customers that could be lost, then you want to invest in this kind of complexities because they give you a lot more information.
Manuel: So tradeoff is important.
It's a tradeoff. Like everything, it's a tradeoff.
13. You're also a fan of functional languages?
I am.
Manuel: In your experience, have they been helpful to test more efficiently or do you think it's just a matter of personal choice which language to use?
It's honestly a matter of preference. Any business can be built with any language and there are advantages in some cases, especially in the beginning. But as you go down the road, data over time pretty much shows that it kind of becomes a wash.
I prefer functional languages. I like to program in functional languages but that being said, you can accomplish business goals on anything.
The complexity of the system over time, the way you architect, design, and maintain the system, the way you pay back the technical debt of the system, that's what really slows you down in the end, over a language. That first year, you may get a big boost out of using something that you can go really fast in but in year five, the complexity you created is what kills you, not the language.
So I'm a proponent of functional languages but they're not necessary to do your job. Unless you do research and really focus on improving computer science and changing and evolving techniques. A lot of this testing techniques that we've talked about, property-based testing and those kinds of things, they come from the functional language research. So I think that is absolutely critical to improve the state of everything. Eventually, they all make their way back into other languages.
14. Do you want to share any other projects or interests you have currently?
Yes. Based back on the dynamic security controls, I have a suit of open source tools under the organization called repsheet which is about being able to apply controls to the situation and really kind of improving and creating that environment that allows you to do dynamic response.
It is open source, yes.
Manuel: Can you describe it? What does it do?
Yes. So it's a control framework. It allows you to deploy to all of your web servers and have a shared cache which is a shared view of the world on who needs to be blocked and who needs to be white listed and who should be tracked further or maybe examined further. And that's that dynamic part where you can say “somebody's marked on the repsheet so I want to ask them a couple of questions, I want to change the way the page looks or show a captcha”. Really is that idea of dynamic security.
So there's that for the web servers, there's some command line tools, there's a visualization environment so you can see what’s in the cache and see what that's looking like. It's kind of a series of tools to help you do that.
Manuel: Sounds interesting. Thank you very much, Aaron.