Transcript
Ruth: My name is Mike [Ruth]. I work at Cruise Automation, the self-driving car company. Well, maybe the other self-driving car company, but we're working on really cool stuff. What I want to talk a little bit about today is really canaries, all things canaries. What are they? Where did they come from? Who do they work for? All different types. Then, maybe towards the end, talking a little bit about how you can take these and really put them into anything and make a canary out of any device that might be on your network.
But first selfish plug, I really enjoy working at this place. If you guys are interested in working on similar challenges that of all the topics that really are out here at QCon today - data science, machine learning, security, all that type of stuff, feel free to either sync up with me after or take a look at the slides there.
Canaries & Coal Mines - A History
Who has heard the term "canary in a coal mine" before? All right. I had this plan that no one was going to raise their hands, and I was going to make a joke, like, "Yes, I don't like being told what to do either." But you all raised your hands, so thanks for being a great audience. Perhaps less of you know maybe where the term came from, although, maybe for those of you know who already do, some refresher. The idea with canaries and coal miners is that years ago, decades ago, what would happen was it turns out coal mines are a little dangerous, not only because tunnels collapse, but also because of the sort of lethal gases that are in those mines. So carbon monoxide, in particular, happens to be translucent. You can't actually see it, and also odorless, so you can't smell it. In high enough quantities, it's lethal.
They were many deaths, and what they decided to do was they decided to take an actual canary with them down there. So because canaries are smaller, they happen to have a faster metabolism, faster heart rate, the effects of carbon monoxide, actually, impact them more. Unfortunately, they would die. However, it would actually give coal miners an early warning, a detection, that says, "Hey, this is actually unsafe for us. We need to evacuate." Hopefully, they would be able to survive. So that's kind of where the term "canary in a coal mine" comes from. It's a little gruesome. Kind of irresponsible of them, but I guess it saved them.
The idea of a canary sort of stemmed from that, and we have a lot of different ways to use canaries these days. One example is a warrant canary that you might be familiar with. That's when you have a website of some variety with a bunch of legal texts and at the very bottom, there's a line that says, "We have never served customer data to a government agency," for example. But then one day, perhaps that line goes away, and an astute observer could have looked at that page and said, "Wait a minute. That used to say something there. Maybe that statement is no longer true." As a result, you can identify that perhaps something has changed. So that's one version of a canary.
Climate canary is another one where you have a species of animal that happens to be actually showing the effects of changes in climate a bit more than perhaps other animals in that ecosystem. In doing so, humans or other animals can look and observe those changes and behaviors, and they can actually do something with those observations.
Software Canaries
We have a handful of different types. For example, this might be the most popular example of canaries, is the software canary. This talk isn't going to talk or focus too much on them, but I think it's a pretty good place to start. How does this work? So you have a process. You have an application. It's located some amount of memory on it. Then inevitably, these processes and applications they have inputs, and they have an input buffer, so that's also a located a subsection of that memory.
The way that a canary would work is you would actually place that in a continuous memory space right after the input. If you have this input that's vulnerable to say, a buffer overflow, what would happen is that that input would then, as the name suggests, overflow and the first thing it'll hit is that canary. And so the idea here is that you have a monitor that's constantly looking at that address space of the canary, over and over and over again, and it knows that the canary itself, that value should never change. It should say, "Static," over and over again.
In the event of a buffer overflow, what would happen is that the value inside that canary will eventually change. Then you can identify, "Hey, something happened that should never happen. Let's alert on that and let's do something, like, say, kill the process." So there are a lot of different types of software canaries. That happens to be just one use case. Again, if you're interested, by all means, go and take a look at what software canaries are all about.
We've talked a little bit about where the term canary comes from. We've talked a little bit about software canaries. There are other different types, too, that, I think, more align closely to honeypots. That's when a canary sort of impersonates a device that's on a network. Maybe it's an AD server or maybe it's some sort of SIFT server, or maybe it looks like a desktop of some kind. So the idea here, and it's very similar to honeypots, where you have a device that's impersonating another device with the intention of looking appealing. In this case, it's appealing to an attacker or an individual who probably shouldn't be in that network.
The difference is here, though, honeypots, they tend to be similarly attractive. However, they're isolated. They're sandboxed. They're away from critical assets, despite looking like a critical asset. In doing so, you actually keep them open. You want individuals, attackers, to come in, and the reason for that is you want to see what they're doing. You want to see if they're trying to exfiltrate data. You want to see if they're trying to pivot to other machines. You want to see sort of what code they're running there. And as a result, you can actually take that and then make informed decisions.
Canaries, on the other hand, while they may look appealing, they are also deployed right beside other critical assets in your environment. The idea here is they should not ever be tampered with. In doing so, if they are tampered with or if they are ever attempted to be accessed, they need to alert automatically, or right away. The idea here is that immediate instant response is required as a result. So they're a little different. One's kind of for research. They are intended to be found. The other not intended to be found and immediate action needs to be done.
Hardware Canaries
What does a canary kind of look like? There are a couple different types. We're going to start with hardware canaries here. Like I mentioned, these are physical devices, have some sort of software that's running on them, and they're intended to impersonate something. There are different types that you can use. I've thrown a couple up here. I'm not trying to sell you on any one or the other, but I have the most experience using a lot of Thinkst products, so a lot of examples and demos today might be using their tools. But I'm all for the right tool for the job. If you want to create your own canary service, by all means. My opinion happens to be if you have an open source library and it hits 80% of the use cases that you might be looking at, maybe you spend the time working on that 20% and commit those changes upstream. But, to each their own.
There are a couple of tools. Thinkst is a company that provides a handful of canary services. I like them a lot. One of them is OpenCanary. They have open source variant that allows you to spin up some type of hardware device that looks like a lot of different personalities that can masquerade, like I was saying, as some sort of critical asset on your network. There also happens to be paid-for services. The value there might be, like, you don't have the time or, the resources, but you might have the money. That might get you certain things. That might get you a management UI to handle all of the canaries that you're deploying. They might be able to provide you with an alerting system, so you're really [inaudible 00:08:13], those types of things.
I want to show you really, at a high level, what deploying a hardware canary might look like. I will see if I can run this video real quick. Right now I'm in the Ubuntu container and I'm going to go through the processes on trying to deploy an open canary. I can't see it on here. So I will do the play-by-play up here. It only runs Python. It's all automated. So it's pretty convenient in that regard. If you take a virtual environment and you set up one and then activate it, which we're going to do right about now, then you can start to pull in through PIP or you can build it your own, the OpenCanary Library. You'll notice I already did that because I'm not crazy, and I wanted to make sure that it worked beforehand. But I also had a config lying around that you'll see that I'll delete right here, because we want to create a new one so we can see what's going on.
When you try and run OpenCanary, it's going to say, "Hey, you don't have a config." That's because we just deleted it. So what you can do is they give you a pretty nice template that you can use. And you can copy that template over and then we'll go and take a look at what's actually in there. So the way that OpenCanary works, and this is the way that a lot of different canaries work, so if you're trying to impersonate other devices on your network, you want to be able to have all of those protocols available. If you go and take a look at the configuration file here, you have all these different modules, FTP, HTTP. You can run a web server. You have a lot of different personalities and permutations to provide, and really, you can just be creative with it. The idea here is you want to trick typically a person into wanting to go to whatever this device is if they end up seeing it on the network.
We go and create a unique device ID. We see that FTP is enabled. We're also going to just use a simple use case of writing to a log file here. And then we're going to enable SSH. Really what this OpenCanary is going to be is it's going to be an SSH server with FTP on it. You can tell that there's a lot of different options though. Let's start it now that we have our config. You'll notice that we have a handful of information here. One is that the node ID is that OpenCanary QCon SF demo, so we have a unique ID. We also notice that a couple of modules have been started, the Canary SSH and Canary FTP modules.
Now we'll try and SSH into this device. I didn't run this for the docker file when I really should have, because now you have to get unique and creative with finding out what the IP is. You have to do these weird OC statements. All right, there it is. Also, at this point, SSH just refused to use my connection, and so I panicked a little bit. I thought for a second, "Should I just destroy this whole demo? Should I just, go back and redo it again? What's going on here? No, no, I'll actually persevere." So I persevered through this. I'm like, "Why isn't this working?" So I go and try and check if OpenCanary is still running. I try and start it again. It says, "Hey, yes. That's still running." I think it was really just a weird issue with this container. So I said, "Okay, it's still running. Let's try again." All right, it worked.
The idea here isn't that you actually get SSH access. It's someone who's going to try and hit this device over and over again. You're going to, in some way, prevent them from getting access while still logging the attempts made. So it doesn't matter what the user is. It doesn't matter what the credentials are that's being passed in here. The most important part is, one, someone tried to access it. And two, we've logged it and alerted properly on it.
You'll see I tried three times. It failed, as we expected. So now let's go and take a look at the logs and see sort of we can find in there. We go to this OpenCanary log that was created, not with less because that's not in this, but with them. There's a lot to parse through here. You can go back and look at the recording if you want to take a look at all these logs. But one of the things you can notice is we have the QCon demo ID. We know that we're actually interacting with this Canary device. We know that the source host, where it came from, happens to be the same location because I did it in the same shell. But it also happens to show the destination and what sort of things the individual was trying to log in as. Of course, super secret password is probably not going to be the password of any SSH server, but you should probably try it, just in case. And that's it. I mean, you can do a lot more with OpenCanary, but that just happens to be a quick little example.
Let's get out of here. Because this is all in Python, this is actually pretty portable. You don't need to be dependent on a specific instruction set. So this can exist. I've tried this on Linux VMs. I've tried this in a container. You can put on a Raspberry Pi, which is pretty sweet because it's a small little form factor. I will say, if you are putting them on the Raspberry Pis, you might want to have a conversation with your IT or security department before you start plugging them in kind of all over your corporate network.
A funny story. The Thinkst canaries, the hardware ones that you can actually purchase, they happen to come in a really sleek form factor. So they look a lot like Raspberry Pis. When you plug them into your data center, they're a little too small to sit anywhere in particular. The chances are they might end up on the top of a server rack or hidden behind the back of a server chassis. When your IT folks don't know that you've actually put them in your network, and they round the corner and see what would appear to be a Raspberry Pi plugged into a switch, they tend to panic. Maybe have that conversation with folks if you end up putting these things into your network. I learned that one the hard way.
Virtual Canaries
We kind of already covered virtual canaries. But we can see sort of the same parallels happening with canaries as we see with sort of all infrastructure. At one point in time, we had to deploy canaries onto physical devices. But once you've turned them into an image or an MI or a container, right, they become much more portable. You don't actually need to manually go and configure a canary every single time if you've already got a docker image with it already configured.
Then you can start deploying them into places where you don't have physical access like cloud environments, for example. There are some risks I suppose, and these are the kind of the same challenges that you would see in any sort of pipeline, is that if you want to start uniquely tailoring personalities, but you want to do that in an automated fashion, you probably need some sort of config management to be able to handle that.
Application Canaries
The other type of canary. This is the Application Canary. We've got two use cases here. Use case one is pretty much the same thing. You can take something like an OpenCanary, and you can go in and put a web server on it. You can have it alert if you try and resolve whatever URL it's listening in on. That's effectively the same thing as what we just saw.
Use case two, I think, is a little bit interesting. Instead of going to a website or an application and having that alert you as to someone being there, because maybe that someone should never have been able to get there or there was no good reason for an individual to have ever been able to go there, instead of having the alerts come from going to the application itself, we can actually have the alerts come from actually using a pair of credentials while going there. This actually changes the value of the canary itself, because instead of it being the application, you can now store this string, it's basically just a string, anywhere you want. You can store it wherever you store all your other credentials or all your other secrets. That probably means on a Post-it next to your monitor or on Mac OS notes. Hopefully not. But if you do do that, having canaries actually sit there is probably not a bad idea.
We can see now that we've gone from hardware to virtualizing it, to actually just turning it into a string. I'm going to show you kind of what that looks like, use case number two. Interesting story about this. I was actually randomly pinged on Reddit of all places, and they said, "Hey, I saw you talking about canaries on this forum or wherever. Here's my service. I've actually just created this thing. Would you care to take a look at it?" I said, "Sure." What this person ended up doing, it's called Tamarin. He ended up creating a bunch of different websites and sprinkled them all onto the internet. And so there's really no good reason to ever need to go to one of these things.
However, if you do end up there, you'll notice that they take all different types of personalities. One looked like an enterprise login. One might look like a photo bucket for storing photos. The other might be a place to store your crypto. All of these places that might have actually important information or valuable resources that might be interesting to an attacker, he has all these websites out there. The idea is you create these fake credentials, you store them somewhere, and if that location happens to ever be compromised, they'll know to be able to take those credentials, go to this random site out on the internet and try and use them. In doing so, you'll have some sort of awareness as to, "Hey, that location, that database, or that secrets management store, that's been compromised. Something has used the sets of credentials that should have never been used before."
Let's take a look at kind of what that looks like. Let's see if I do this song and dance again here. I should say that this is a fairly new service. It's a work in progress. One of the biggest critiques I had was, "Hey, aesthetics is actually super important, because if you have a website that kind of looks sketchy, chances are an attacker who has credentials probably isn't going to risk it, right?" There's a mental block that says, "Who's going to store all of their crypto here if it doesn't look like it's a legitimate place." So that was kind of the biggest critique I had of this service. But I suspect that that will get better over time.
How does this work? We have the site here, and this is the main console. I don't know if you can actually see. What we're doing is we're creating these new bait credentials, is how they call them. We have a username, we have a password and then we have a web hook of some kind. We can send it to an email, we can send it via text, which I don't know if I'd want to do that, or you can send it via web hooks, say to Slack or wherever it is that you want it to go. We chose email and then when we go and create these credentials, again, this is in the console, we can see that it's actually associated with this site bitcoino.com right here. So if we go to bitcoino.com, we'll notice the completely inconspicuous place. "The Crypto Concierge" is what it's called.
The idea is, if you try and login here with any sort of credentials, that's going to fail. It's going say, "Hey, you have an incorrect password." But if you go and take these bait credentials, presumably stored in a location that sensitive to you, and you use those, it will go, "Oh, hey, your subscription has actually expired," indicating that perhaps these credentials at one point were valuable, but they can no longer be used. But on our end, we can see if we go into our alerts email, “Hey, that actually triggered an alert.” And it says, "Okay, this is the description of the bait credentials. This is the source IP where it came from. And here's a user data string." So you can kind of get an idea of where this individual was coming from, and what was it that perhaps you need to look for. That's basically at a high level. You can go back and take a look at the incident once you reload this page. But in the sake of time, I'm going to end it there.
Participant 1: Question: I'm just curious how often do you get a take rate or hat you're expected interaction rate, or how do you justify doing it, and how vigorous is that feedback loop, I'm wondering? Is this sort of a makes-you-feel-good measure, or what's the experience like?
Ruth: I will say that they pay for themselves. If you're going for the paid version, and I guess even if you're not, I would say they pay for themselves over and over again, from personal experience anyway. Jared was talking - I don't know if anyone in the room was able to make the talk prior - but he was saying how one of the most important things when trying to identify how to mitigate attacks is how do you know that you've been attacked. How do you identify that? This is actually a really great mechanism as an entry point into, "Someone's in my network. They're doing things that they shouldn't be doing. And they happened to have tripped on or hit a device that they shouldn't have." No one else is hitting it. Then you know as a result, perhaps that incident responses is required. So from that perspective, yes. That might not happen often, if you've put a lot of defense in, a lot of time and effort into your defenses, but if it does happen, you want to know about it right away. And these are very helpful with that.
Canary Tokens
We're back on track. Application canaries we had noticed that we had gone from a hardware device to a virtualized device to what would seem like a string. Now we literally have the option to take these strings and use them as canaries. And so this is kind of a joke slide. But if you take a look at the bottom right corner, these are the folks from Thinkst at Black Hat 2015. They were talking about, "We have this cool idea. It's not really a new concept. But we wanted to create this idea of canary tokens."
And one good example is the City of Agloe. I'm not sure if anyone has heard of the City of Agloe before, but perhaps for any of you enterprising cartographers, the idea here is that I think back in the late 1800s, there was a mapmaker, and he found, I guess, that he was too good. People started copying his maps. He was like, "I'm sick of this." So what he did was he put this city, Agloe, on his maps so that whenever someone tried to copy him, he could say, "No, no. That city does not exist. I created that. And I know now that you have done something that you probably shouldn't be doing." But not to worry. You can put Agloe actually on your maps now because the city exists actually, because of all of the attention that that little trick got. That's an interesting little tidbit.
Anyway, so they were talking about canary tokens back in 2015. A colleague of mine, Karl Molnar, actually presented at Kent Tech West earlier in the year. Ge decided, "Maybe we can do something a little different with this. If we have a string, we can use that as some sort of bait or some sort of alert. Maybe if we inject that as a string into our binaries, that might actually help in the event that someone tries to reverse them. They might see that and then we can do cool interesting tricks as a result." I'm not going to get into too many details there. But if you're interested, definitely go take a look at his talk. I have all the links in the presentation as well.
Here we are today, canaries are strings. They're embedded in different places, embedded, all that stuff. Hopefully, if someone does a canary talk in the future, they'll take this slide and we'll just have slides all the way down, presentation of a presentation of a presentation. Canary tokens. Now that we have these strings, we can make them whatever they want. Are there any services that can actually do that? The answer is yes. If you see a type that you need that doesn't exist, definitely make your own. If you go to Canary Tokens, that's an open source library. PRs are accepted, I'm sure. Generate your own tokens. Use them from the site itself.
Since it's no longer a device, and it's a string, these opportunities grow tremendously. What can we do now with strings, versus that we couldn't do with a device that was on the network? I'm scared to now try and do another demo because it took 20 minutes out of our time here but we'll see. I just won't zoom in. That's what we'll do. This is much better resolution, so I think we should be okay. So this is a canary token screen. We have the option to go and create a handful of different token types. We saw a couple of them on the previous screen, DNS tokens, documents, credentials maybe. We're just going use a simple example of a URL token.
[Inaudible 00:29:45] here is very similar to the ones we've seen before. I'm sure you're seeing a pattern. We set the place where you want the alert to go, if it's ever triggered a unique description, which is important because if you have many of them, you want to uniquely identify the token type to the actual location or where it's being stored. Then when we create this guy, for some reason, this thing wasn't turning green to create it, but it seemed to work anyway. It says, "Hey, you've got the string." Then it suggests some clever ways that you might actually want to go and do this. You could embed it into an image source. You might be able to store it an email, tag it with a saved label, and be like, "Hey, I need this help." You could use C names for this. You can get creative with how it is that you actually want this to be triggered.
What we're going to do, we're just going to actually go right to it. If you were to go there, it's on a single pixel, you can't see it. But the important part is if you get into digital to trigger that, you get an alert. So similar to bait credentials - instead of in a credentials file or with your credentials, you can store it anywhere You can get creative with it, too. In what ways can you force someone by going to a certain area to automatically generate an event, trigger that, and have an alert respond. Here's a bit more of a beautified version of perhaps what we saw earlier. Canary token has token type, source IP, user agent data, all that type of stuff.
Canaries at Scale
This has basically been all a refresher up until this point. If you guys were familiar with canaries before, you may not have necessarily learned anything new. If you have learned something new up until this point, awesome. I'm glad to hear that. So what makes this talk unique is really what can we do now that we have these strings and these virtual devices to really scale that out? And I contend that there are probably four main or three main things that we need to be able to do that. You probably need some sort of library or wrapper for your clients to generate those tokens. You probably need some automation framework of some kind to actually create the tokens themselves. And then, you need to be able to pragmatically embed those into wherever it is that you want them to go.
What would that look like? Maybe something like this. This is a kind of high-level architecture of what it might look like. So step one, is you have this client come in. You say, "Hey, I want a token that will perhaps authenticate to your Lambda GSF Azure variant, whatever it is that you've got going.” Service architecture seems to be a pretty good fit for this. It will go and create- I used the canary tokens as an example, but like I said, you could use whatever it is that you want. They just happen to be an open source variant. They go and create the actual token itself. It sets up the endpoint and then if it's ever used, now that it's been sent back to the client, you can have that trigger like we saw.
I wouldn't suggest taking this and using it, perhaps, in a production environment. This is mostly kind of pseudo code. But you can kind of see, based on this, what that might look like. So Canary Tools is the non-free version of the canary token service. That's the pay-for version by Thinkst. But the idea here is you'd set up a client, you get back some sort of token-type with it, and you'd provide the description and the kind, the type that you want. Then based on that, you would go and hand back whatever it is that you want. So for AWS keys, you would need an access key and you need a secret key. If you wanted to set up a DNS Canary token, there would be hosting of some kind.
That's kind of on the automation side of things. And the wrapper would need to then call into it. It would be like, "Hey, if you're using AWS, if you're using GCP, make sure you set up a client to this serverless function.” Make sure that you generate your event, give it all the information that it needs, receive back that payload. And then, from that payload, you can go and actually parse through it to get a specific token that you wanted. In this case, you might want AWS credentials, and you may want to put that in your AWS credentials file. From GCP, you might want service account credentials, so you want that JSON key. Canary type for DNS, maybe you put that somewhere. You can get creative with it.
That's kind of at a high level what this might look like. I have a demo to show you all of that in action. So this is going to be pretty simple since we saw what these were doing. We have our AWS credentials location. We have this canary automation script that we can run. We can give a description of the token that we want. And then we'll say, "Hey, let's go and create AWS credentials for us please." I'll run this and say, "Hey, now you've got a new pair of credentials." If we go in there and take a look, oh hey, there they are. You take a look at what those are in the credentials canary file. Now you have - dramatic pause - an access key and a secret key. If you were to use those, they would trigger in the same way that we've seen in the past. And similarly, if we don't want AWS credentials, and if we instead want a DNS record, we can get that, too. So this is extensible. All of the different token types, you can build those out.
This would be what it would look like. I had some hurdles to jump. So this is a console that you look like, guys, when you go and create database credentials and when you go and create a DNS. Those are just examples. You saw that there are a lot of different types.
Can We Do Better?
But a client hitting this framework once every so often, that's not really too exciting. That's kind of kind of baseline. So what else can we maybe do, if we have this type of automation, that might be a little better? Maybe we can hook that into a source control. Maybe we can, on certain events, merge this into our CIC pipelines. What that might look like is something like this. This is kind of the end goal of what we kind of want to get with canaries. We no longer have a client, but maybe, perhaps, whenever we're ready to merge a PR, we have a hook from that, for let's say, call it whatever, whatever source [inaudible 00:36:52] you have. You have a hook that goes and actually fetches and creates a token for you. It sets up that endpoint. Then through all your normal CIC processes, you actually use that token and you inject it into the locations that you want them in.
Now you've got tokens distributed into your infrastructure, and now you've got tokens distributed into your Kubernetes environments. Every Kubernetes manifest now has a secret in it, for example. Every VM or cloud instance has AWS and GCP credentials. Every database has a unique canary user in it. So now you have all of these uniquely tailored canaries that just about every device that you have is able to alert on its use. The point here is that those secrets, they should never be used. But they're not actually used for anything. But if someone goes and finds them and then tries to use them, well, they will alert in a certain way and then you'll know that these things are being used incorrectly.
What are the Implications?
What are the implications? We kind of went over them already. You can put tokens on anything. So long as you can trigger that event function, whatever it is, you can get one. Now you have an alertable tampering device on every single one of the things in your environment. It's limited only by your creativity, and perhaps possibly by the fact of actually having a CIC pipeline in the first place.
What are some of the implications on the bad side? So the biggest concern, I would say, is an increased rate of false positives. So if you have tokens everywhere, and someone in there just trips, you put them in the wrong locations, or you kind of configure it in the wrong way, people might just trip over these canaries over and over again. If they do so and you get alerts, you're going to get fatigued by those alerts. So there's sort of some risk involved if you have them in too many places, or if you have them in places that perhaps they shouldn't belong in.
The other problem becomes, hey, where do you store a list of all your canaries? You don't want to go stumbling over them accidentally. However, you also don't want to have a list available that may eventually end up in the wrong hands, and then people can circumvent or sidestep those canaries entirely. If an attacker, for example, has access to your Slack, and your Slack happens to be where you have a bunch of canary notifications or alerts for things like that, that could be a problem. So you have to make sure that the sensitivity and the isolation is there.
The last thing on canary AWS credentials- one thing that should probably be noted, you don't want to get crazy with AWS credential tokens, unfortunately. There was a great article by Rhino Security Labs not too long ago, which showed, hey, people can get actually clever with trying to bypass certain credential canaries for AWS, because not all events go to CloudTrail, and CloudTrail tends to be where you send out SMS and how the alerts come. So just be careful with that credential type is all. And that's it. Thanks.
Questions & Answers
Participant 2: I understand for the SQL connection string, how you trick someone to putting that if they find it. But if you put a password or username or credential combination, how can you trick them or lead them into where to try those credentials? Does that make sense?
Ruth: If it's just a random string sitting somewhere, how do you know where to use that string correctly? I've seen a couple of ways that this is solved. One of them is, the format is pretty obvious. Access key ID, secret key ID, those are probably AWS credentials. A lot of secret stores will have files. You can name the file something. Oh, the contents of inside of it go to the name of the file. The Tamarin application canary use case, if you were to copy them and put them somewhere, it actually stores a string of the URL along with it. So you may need to consider how to do that. But you might want to give subtle nudges or indicators depending on the format, so that they might know where they need to go. Anyone else?
Participant 3: How do you deal with the case where you're distributing these canaries, but then you also have tooling to scan your own sites or containers, and find credentials so that you're not clicking them?
Ruth: I think it's maybe twofold. One is, you learn real quick because all of the canaries start triggering and then you probably want to whitelist your sources. So if you have scanning tools, a lot of people use things like Nessus, Nexpose, these types of scanners. You probably want to whitelist the source from them, so they probably have the same IP every time. If you're using open source variants, you might need to put that functionality into your infrastructure. If you're using paid-for variants, they tend to have whitelisting as a capability.
And the other part is also, but if you have a list of the canaries, perhaps you can get intelligent with where you do and don't look as well. Those are probably the two sides. So from a scanning perspective, maybe you know that a canary exists somewhere, and you don't try and actively trigger it. And from the Canary's automation side of things or from the infrastructure side of things, maybe you know that this is a scanner, and don't worry about it, if it does trigger it.
Participant 4: A lot of your assumptions were that the attacker is somebody from outside unfamiliar with the system. Let's say I'm storing a lot of personally identifiable information and I don't want even somebody inside my organization to have access to those credentials. Do you have any tricks for keeping people unaware that the canary exists?
Ruth: Well, I would actually say that those types of individuals are the number one false positive individuals. They are the ones that are actually causing canaries to trigger probably the most often. In fact, so that means that if you are responsible for doing incident response, chances are you're tracking down a lot of your own employees, being like, "Hey, what are you looking at? Where are you go in there?" And so you need to identify, was this actually a malicious intent, or was it accidental, or just pure curiosity? I think it actually solves that problem in and of itself, simply because those individuals shouldn't be doing that and they take the form of an attacker regardless.
See more presentations with transcripts
