Transcript
Soroosh Khodami: We're here to see, are we ready for the next cybersecurity crisis like Log4Shell or not? That's probably not. Based on my experience in the different companies, different clients, we are not ready yet. How many people remember the Log4Shell or at least heard about it? My story is a horror one, a scary one. I'm quite sure you will think about it.
Let's start with the prologue. Who has ever copied a set of bash commands from the internet? I'm talking about this type of set of bash commands with high votes or from our well-trusted chatbot, Claude, ChatGPT, whatever. We all have done it. When you ask developers in a company, yes, I will not run these commands in terminal blindly. The hackers start laughing at us. I want to tell you something really scary that you are, all of us, one command away from gifting access to a hacker. I'll show you why I'm saying gifting. Our Linux experts, what is this? Anybody can recognize this? It's called reverse shell. What is this? It's actually a really interesting thing. Who has ever connected to another server via SSH? We know that when we want to connect to a server, we run SSH, so there's a terminal connection, we connect and we run the commands. This one is like that.
A victim, like me, runs this bash command and a hacker on another machine will wait for a connection to be received on their machine. What does that mean? I will show you. What will happen? I will show you. I will get into my hacker mode. We have two terminals. One of them, I'm connecting to a server. I bought this server, €5. It's clean Ubuntu, nothing installed on it. I will just connect with SSH that you're all familiar with. It's a live demo. We're connected to a machine and I put the name of the hacker machine on it, so clear.
The second one is my computer. To just see it, I will just do that. I will run pwd, and I will run pwd also here. You see that this one is a machine on the root, and this one is Soroosh machine. I want to run this command that I showed you, the bash command. Before I run this one on the second terminal, or maybe I can just move it up, so clear. This is the second command that will run on the victim machine. I'm going to wait for the connection on port 4242. One command, it's already installed on the Ubuntu. I will run this one and see what happens.
Did you see what happened? On the terminal top, I got the bash. One command. You might say, I'm not going to run that command on my machine. I didn't tell you that you're going to run that command on your machine. How many Java developers do we have here? How many Node.js developers or JavaScript stuff? Python? Do we have any other non-developers? DevOps engineers. I want to show you one scary thing under Java. Maybe I can also show you something under Node. Just imagine, I created a project with a Spring initializer. It's an empty project. Empty. I just created with a Spring initializer. I did only one thing. What was that I did?
In the pom.xml, I added this dependency that you're all familiar with, com.google.code.gson. I added this one. What can go wrong? I just added this dependency. I want to install it. This is my terminal again. I'm in the Spring Boot victim, so pwd again. You see the folder? I just want to run maven install. I will do the nc -lvn again. Maven install. Pretty simple. We are all doing it every day, if you're a Java developer. Let's see what can go wrong. It's running the test. You're hacked. Of course, the hacker will not print it for you. You see what happened on the top.
Again, I've got the terminal access. I'm not sure if you're as scared as me or not. I ran just maven install and it happened. Do you want to also see under npm? I can show it under npm as well. When we want to create an API with Node.js, what library do we need to install? Express. I want to just mkdir, like a fancy-project. I'm in the folder. I'm doing the npm init. I just press enter. I initialized the npm project. Simple. What's next? I will run the clear. I want to run npm install express. Who's a fan of using the latest and greatest packages? You should not be. Latest. I'm running the Express latest. Again, I want to run that fancy nc -lvn command. Let's see what can go wrong. Here we go. I installed Express, the most downloaded npm package in the world. At least it was my intention. I lost interactive terminal access of my machine to someone else. Maybe it is still not clear for you what just happened. I will try to open it a little bit more.
What the huck? I'm in a private network. My laptop is connected to my phone, to my hotspot. My laptop even doesn't have a public IP address. Technically, if a hacker wants to reach to my computer, it's not possible because there is an incoming request and the firewall and router say, buddy, you're not allowed. Incoming connection is blocked by default on all the routers. I'm not a hacker anymore. Let me just speed it up. How I lost my laptop access to this guy, the Black Hat Hacker? I initiated a request from my machine toward the hacker machine. Outgoing request. The firewalls are like, you're in a house.
If you want to enter the house, you need a key. When you're inside, there's a handle, you can just open the door and leave. That's how, by default, default configuration of the routers and firewalls are there. Is it clear? Am I the only one that's scared about these types of things? It is really scary. Welcome to my world. It's something that I'm busy with every day. It's called supply chain attack. It's not a new battlefront in cybersecurity. It's been there. The supply chain attack was there since a thousand years ago in human history. In cybersecurity, supply chain attack started 15 years ago, 20 years ago, with things like Stuxnet. Nowadays, it's become really popular because the hack that I just showed you is pretty simple, and I will show you what it is.
Who am I that I'm talking about these types of things? I'm Soroosh. I'm a developer since the good old dial-up days. I'm currently working as a solution architect in a consultancy called Code Nomads in Amsterdam. My client is Rabobank, so most of the stories that I will share with you are all the researches that I did and all the experiences that I had as an architect in a team that actually works on the platforms that helps team squads to reduce the risk or prevent these types of risk for their day-to-day work.
Act I - Unmasked
Let's go to the first act and unmask what just happened. Before that one, classic cyberattacks. Do you know what is a SQL injection or XSS or something like that? Who has ever tried SQL injection in practice, like really trying? If you haven't tried it by yourself, you should close your laptop, register for hands-on training, and just do it once, and then you will see what is this SQL injection. Theory is not enough. XSS and those in path traversal, you might say, these are simple things. No, that's not true. These things are still gifting. They're really important.
If you haven't tried them in practice, you should not commit any code from that time, from my perspective. This is the experience that I have with 8,000 engineers. What I mean by this is that maybe those SQL injection stuff are a bit old, but they are still gifting based on this report. 2017, we had a report from OWASP, and you can see that some of these items move to a different order because they are trending or are more popular nowadays. This report is for 2021, but I'm not here for those classic attacks. Experts believe that in the new report from OWASP, the supply chain attack will be a new item, and most probably in the top three. Now you might get why I think this topic is really important to be shared with you.
Supply chain attack. I will go a little bit more higher level, and I'll talk about supply chain risk. I think we're more or less familiar with the third-party vulnerabilities. We are using some dependencies in our project, and those dependencies can also have some security issues. We are all familiar with it, so sometimes we need to update them. There are also other types of supply chain issues or risks like dependency confusion, or software supply chain hijacking, or compromised build environment, and counterfeit components. What I demoed to you, the Maven one was the compromised build environment, and the npm one was dependency confusion. I will show you how those things work. You might say that you showed something, looks cool, looks scary, but how many times do these things happen? How serious is it? Look at this. Every year, we have attacks that will make headlines on the news. All of you remember Log4j. You might remember the SolarWinds, all these things started then.
I see a lot of people, DevOps, the people remember what happened to DevOps. Go to 2025, we had a Kubernetes ingress, and also the one from Next.js. Still, 2025, we had a huge attack under npm: not one, not two, a lot, on a project that had 2 billion weekly downloads. This number is not a joke, it's a really big number. You might say, I will fix them. Whenever they're on the headline, we are really quick, we just fix it. Maybe you are fast. After three years of the Log4Shell disaster, 13% of people are still downloading the wrong version, the vulnerable version. It's not a joke. There are some people next to you that are still downloading the wrong version.
Let's go to the dependency confusion. It's the most trending one nowadays, because it's pretty simple. You can read the history about it. The idea is pretty simple. During the Corona time, somebody was a bit bored and found this trick, just imagine that you have a project. I will explain under Maven. You have a project in your company, like a shared library that you want to share with the other folks. They are a really senior team, they are building really good and robust code, and you want to use their library. You put it in your pom.xml, and because they are really good and you want to always use the latest version of their library and you have trust in them and you also have trust in your code, so you have a lot of automated tests. You just say, I will stick to the major version, but whatever is the minor or patch, I want to use the latest one. Is it clear? Seems ok, because automated tests are in there, we have unit tests and everything.
Usually, those dependencies, those libraries are stored on a private repository. Or maybe you have multiple private repositories that your package manager has configured to download the packages from them. When you run Maven install, like what I did, and you have a version range, the Maven or the package manager will become confused that, ok, I'm looking for latest, but there is a package with the same name on one of those repositories, like 1.999.999. If you see it with your eyes, it looks fishy, it looks scary. You say it's wrong. Maven doesn't know these things, so, ok, let's download it. The master asked me to download the latest and greatest. That's what will happen, and you already saw it. You are hacked without explicit print.
The same can happen with npm, and a lot of things are happening every day. If you just search for the supply chain attack on npm, every day, a lot of companies are getting hacked by these types of attacks. Google, Meta, Microsoft, and all the big techs got hacked on the day that these things started. Is it simple? The thing that I showed you, the npm one, I built it, because I realized there are some people that write npm, and it took me half an hour to build a malicious Express.js. The building part is simple. You just need to find a good repository without enough security, you know the rest, you are hacked.
How much risk are we talking about? You think you are only using Spring Boot, you're building a Hello World application, the minimum dependency, but you're actually importing 49 dependencies in your project, and each one of these dependencies are like an attack point, attack vector, that a hacker can just find, there is a library that a developer in Nebraska, thankfully, are maintaining it for 20 years. No one paid a penny to that guy. This guy is on vacation. Maybe I can just do some social engineering hack, or steal his credential, or whatever. If you can manage to hack that small library, like 5-kilobyte library, you will get into one of the other libraries. Don't think that this thing's not going to happen. It already happened. The last time was two weeks ago. I will show you. Also, under npm, so Express, you just install Express, you have 65 dependencies imported to your project, but you only see Express. It still looks scary.
We are in the AI era. A lot of fancy stuff about AI. Can you spot the difference between these two dependencies? Can you see? Who can spot such a thing in the pull request? I cannot. Maybe the guy who found the XE library, he found out that 20 millisecond delay in the connection, maybe that guy can spot this one, but I'm not the one who can catch this difference in the PR. This is what AI can do to you. Still, you might say, why would AI give me a wrong name?
In the Keynote, you remember we can poison the LLMs, we can do a lot of different things to change the behavior of LLM. We can have a fake statistic on npm, so even if you want to manually check the npm to see which one is the real one, the hackers can create a fake number of downloads for the npm packages. They can. This is the new report from Anthropic, that they realized with 250 documents, you can influence a large LLM like Claude or ChatGPT. With 250 documents, it's nothing. It's very dangerous. You can read the rest of the story. Also, on Hugging Face, there are a lot of free to use models, and you think, that's fun, let's use it. Recently they realized that those LLMs can be really harmful. They can do a lot of things. Still, I'm not sure. I still need to read more and more about this, but the AI era is definitely bright, but full of terror. Do you agree with me? We can agree or not, but this is the reality.
I promised something on the abstract, whatever, we have the SQL injection vulnerability on our code. What can go wrong? I'll tell you. Just imagine that you have an application that has the security issue, SQL injection, what can go wrong? Hackers know the things that you maybe don't know really, in a good way. They can just get into your system with SQL injection, and maybe unfortunately you're running a Postgres that has another issue, because you have a dependency on your Postgres.
This Postgres can allow an attacker to run some arbitrary command. Just imagine that Postgres is deployed on a container that the base image also had some vulnerability that allows you to become root on the container. Maybe that container, that pod, is deployed on a Kubernetes that is running on a specific version of the Kubernetes that also allows you to get elevated access and become root. You know what? The hacker is king in the castle now, because of only the SQL injection that you had. You might say it's a very niche or rare attack vector. Maybe this is one of the examples, but the hackers have recipe and automated scripts that they will just try everything. That when they're in, they will try to do many different tricks to see how they can move forward with new things that they can discover.
Act II - Fortress
I'm scared enough. I'm not sure about you. Who's scared like me? Some people are really brave. They're not scared. I talked about the scary part. I unmasked it. The fortress, how we can protect, that's the important part. I have some recommendations. They are really practical recommendations: critical, essential, and advanced. The critical ones that I think you need to pick up today, tomorrow, next week that you go to the office. Essential ones take a month or two. The advanced ones need a bit more strategic planning. The critical ones, naming convention. Try to agree with all the teams in your organization to have a prefix or a naming convention when you're building an artifact, a library, or whatever. It's really important, because you can reserve those namespaces on the public package managers or the public repositories. Like, for example, NuGet, npm, those stuff. It's a really huge difference if you're using your own prefix, and you can control it somehow.
Second, using latest or not? Don't use latest. Always use a specific version, because then you can have a lot of different things to protect you against many of the attacks, like what I just showed you. Express at latest was the problem. The third one, package install check. There are a lot of tricks that you can consider when you're installing a package. For example, for Maven, you can have strict checksum to make sure that, you want to download Spring Boot, but this Spring Boot, is it the real Spring Boot? Is the checksum the same as the one reported on Maven Central? That's important. Also, some enforcer plugin rules that you can specify that I want this package, that package, those things. Package signature verification. It's very important if you can verify this in your package manager. For the npm, just write these ignore scripts, because then you're well protected.
Many times, you do not expect that library to install anything on your machine. Just run this one. For the one that I demoed, with that flag, we could avoid that attack. npm audit really helps you to check the packages that I installed, the packages that are in my Node modules, are the verified ones, are good ones or not. Last but not least, try to use lock-file. I had conversations with a lot of squads recently that, why should we commit the package-lock? They said, no, we should not commit it. It's very wrong if you put the package-lock in gitignore.
Package-lock is actually helping you a lot to prevent these types of attacks, because if you tested something on your local machine, so you have the package-lock, you can push it also to the git repository, and you make sure that your deployment will be exactly the same as you tested on your local machine. Also protects you against a lot of attacks. Last but not least, if you're not using a package, why have you imported or installed it? You saw it. I only imported the Gson, but I was not using it. It was an empty project, but it was still harmful for me. Clean it up. There are two commands that you can use.
The essential ones. I've seen people just randomly copy, paste text from ChatGPT or Stack Overflow, that if you want to fix that error, copy this in your pom.xml or in your settings.xml. Just stick to the Maven Central, for example, or just stick to the npm. Don't start using the unofficial repositories. Second, if you have a proxy repository solution, there are some solutions in the market, like Nexus Repository or JFrog. You can have filters that I do not expect to get any packages with these type of names from that repository. You can put those rules. That's really nice. If you do not have the proxy repository, some of the package managers, they support repository filter. I will show you on the next slide. It's a bit advanced.
Second one, keep the dependencies up to date. Using Renovate or Dependabot is really good. Who is familiar with Renovate? You all should try considering using Renovate or Dependabot because they really help you have version pinning, and at the same time, have automated pull requests to update the dependencies of your project. It's very simple. You can set it up in an hour or two. You can just use it. It's really nice. Enforce immutability for release versions. What does it mean? Just imagine that you're a team that's building libraries. Once you published version 1.1.1, nobody should be able to overwrite that version. When I mean nobody, it means also a hacker.
If you enable immutable version, then the repository manager that you have will not allow you to replace that version. Even if you had a mistake like, we forgot to delete something. Don't. Release a patch version and just write in your change like that, nothing has been changed. Or like Twitter or Spotify, like, we fixed something. You don't need to always explain what you fixed. Security improvement, something like that. Stick to the immutable version because from the security perspective it's really important. Last but not least is my favorite topic, dependency scanning tools. There are a lot of tools in the market that help you to scan what are the dependencies that you're using in your project. It can list all your dependencies. Before I go into the details of the dependency scanning, which I'll explain, I want to share something with you.
One of my colleagues, a really smart woman, she wrote a blog post about the repository filter for Maven and Gradle. It's not easy to do, but this blog post simplified how you can do it and how beneficial it is to use the repository filter. I will go to dependency scanning. I said, with dependency scanning tools, you can scan your project to see how many packages you already installed and imported to your project. Because I showed you, you think you have Spring Boot, but you're actually having 45 dependencies in your project, transitive dependencies and dependencies of dependencies, and npm one. Dependency scanning tool generates a report called SBOM, software bill of material. It's a JSON file. It also has the XML version, but who likes XML? It has a JSON one, I really like it. It contains a lot of information about what you have in your project. It can have information about the application that you have in your project, and all the dependencies where you have downloaded this one, the version, everything, even the licenses. Because you might be using the library that you're not allowed to use. We have some, for example, the PDF generators in the Java ecosystem.
Some of them, if you use in your project, you have to make your project open source. You want to generate PDF, you install the library, but because of the license of that project, you have to make your project open source. You're in a financial institution, can you make your project open source? You need to be careful. By having the SBOM, you can have some governance on those type of things. Or maybe you're not allowed to use some libraries or projects from some certain countries. This type of information can be in the SBOM, and you can use it later. I will show you how. SBOM, good, how can we generate it? There are many ways to generate SBOM. You can use source code, you can use container image, artifact, the produced artifact, the JAR file, or in the runtime environment with the help of cloud providers, they can do this magic for you.
One tool that I really like, jack of all trades, is OWASP dep-scan. It's free, open source by OWASP. It can do scanning of the source code. It can do the scanning of the artifact. It can also do the scanning of the container images. It's really good. The result that it provides, it's really good. You can put it in your pipeline, and you can also put some rules on it that if we have too many issues, fail the pipeline, return error, whatever. Or generate this report, or generate that report. That's a really nice one. If you're using Java and Spring Boot, if it's Spring Boot, and hopefully you're using the version +3.3, it's already a 2-year-old version, you can automatically get SBOM out of the box. If you're not using Spring Boot, you can just add a dependency to your Maven, and it's pretty simple. One go, and then you have the SBOM ready.
For Docker, it's also easy. There are tons of tools that can scan a container image. I personally like Aqua Trivy. Why? Because I did a benchmark on all the available tools, from six months ago, and Trivy provided the best result. It could identify many vulnerabilities. Docker CLI itself also have some built-in tools. I think it's also using Trivy, maybe. You can read more about this one in the links that I shared in the read more section. There are a lot of tools, paid and free tools. The difference between these paid and free tools are some of the companies, they have a premium database. They do not only use the public data available regarding vulnerabilities. They have some sort of premium database of the potential vulnerabilities that can help you to detect some of the vulnerabilities sooner.
Advanced one. These are the ones that you definitely need help from your platform engineering team. They will have cost, and from timeline perspective, I would expect that these things take time, because the impact is really high on a lot of squads, probably. The first one is using dependency firewall. It's a premium solution. It's expensive. Only two companies, as far as I know, have such a solution. One is the Nexus Firewall and the JFrog Xray. What is it? It's like a proxy server that you can put some policies that I do not want to allow anyone to download packages that have, for example, high severity vulnerabilities. On the time of the request, they will block the request. Sometimes they don't even show you those packages when you're searching for a package.
For example, you say, I want the latest version. They will not show you the real latest one, because the latest one has a security issue. Or they have also some mechanisms that they will wait for 48 hours, 24 hours, to let the latest version be settled, make sure it's not a fake one or it is ok. Then they will show it on your search or when you want to retrieve it, it will let you download it. Second one is signing artifacts. Have you ever thought of signing your artifacts, for example, your container images? It's a very nice practice that if you can, when you're building your project, sign it.
Because then you can verify it wherever you want and make sure that this is exactly the one that you produced and not a hacker like me, or this guy with the hat. The container image one is a really fancy one, because it's easier than the other ones. Because when you build the container image, you can sign it and you can put some enforcement on your Kubernetes cluster, that I will not let any pod be deployed with the images that are not verified. It's a really important one. Last but not least, and it's the most important one is, monitor the SBOMs. What do I mean, monitor the SBOMs?
At the time of deployment, you generate the SBOM. That's good. Until your next deploy, maybe there will be some new vulnerabilities that are discovered about the dependencies that you have used and you are using them in production. At the time of deploy, it was good, safe, like Log4j. When they deployed, it was good. Later, they discovered the vulnerability. If you keep monitoring the SBOMs, you can detect or you can be quick in detecting the issues.
Act III - Resilience
Resilience. We know the tools, we know the things that we can do, but how can those tools help us? Let's have a flashback to Log4j. Who remembers on the day that we heard about Log4Shell? It was a busy weekend for some of the companies, and for some other companies, it was a busy year. I have worked with clients that were busy with Log4Shell for a year. Why a year? Log4Shell, a Minecraft server got hacked in December. People were ready to go for Christmas holiday, and they were thinking about what we're going to do now. On the news, everybody started calling each other that a Minecraft server got hacked, and it's also impacting us. What the hell? Minecraft server and the financial institution, e-commerce, what's the problem? Then the story was, it's something serious. The calls started that, ok, which application is affected? Who to contact? How to fix it? The last one is how to detect if we have it or not? When I was working, I was lucky, because we fixed it in an hour.
Again, I know companies that were busy with this one for a year, because they didn't have the right tool. I think you know what is the right tool now. We just need to have SBOMs in one place. If we make sure, for all the applications, all the servers, all the things that we have in production, or even the test environment, that we have SBOM, and we put them in MongoDB, because it's like a JSON. At the time of Log4Shell, we can just query the database, which applications have that problem? We can find them. We can also store a contact person in that JSON that this application belongs to this team. These are the people who can fix it. We can find the people. We can also store some metadata to see which applications need to be fixed sooner, based on the business criticality. The last time is, are we safe now? We have the real-time data, because we put it in place that when you have a new deployment, you have a new SBOM, so you can monitor it, that we started with 1,000 applications that had this issue, and now it's 500. It's something that really helps us, the SBOM in practice.
Just imagine that there are two vulnerabilities in one of our projects, and the left one is the one with the CVSS score, which is the severity one, 5.9 out of 10, and the right one is another one with 10 out of 10, critical. Which one do we need to fix first?
Questions and Answers
Participant 1: The one that affects you.
Soroosh Khodami: I heard someone, probably an architect, said it depends. Depends on what? We need to see. The left one is the application that has a public internet access, contains PII data, customer PII data, and the high EPSS score. EPSS score is like a score that says probably the hackers will manage to create an exploit for this vulnerability in a day or two, or 30 days, something like that. It says how probable it is that someone can exploit this vulnerability. The other one is in the isolated network. EPSS score is also low. The EPSS score is a live data, so every day there is a new number for this one. Just imagine, the right one was the critical, very dangerous, but the left one was not really critical, so medium. Which one do you pick to fix?
The left one, because priority and severity are not the same thing. We need to have context. That's what I said in the previous slide about prioritization and fix. You need to think of your applications, your organization, your setup, your deployment setup, to see if you can store some standard metadata in the SBOM, so in case of the next crisis, you can quickly prioritize that these applications are critical. They are the ones that if they get hacked, a lot can happen. It's really important to consider because it's not only about tools. There are platforms, one of them is Dependency-Track. It's free, open source, really awesome. You can use it in your organization to store the SBOM. What is nice about this platform is, technically, it's providing a really good result on the continuous scanning, searchability. It's amazing. I benchmarked it with also other premium ones that are really expensive, but this Dependency-Track is a really decent solution. It's free, open source.
If you have a lot of money in your organization, you can purchase one of the commercial ones and you can search. Some of these companies, they're doing a really great job, especially on the cloud. You pay the money. Next day, they scan all your cloud resources and they give you a lot of nice insight. They also have some magic to show you which one has a public internet access. Some of these tools are nice. You can search later which one is the best one.
Do you think we are prepared now? We've applied all those suggestions and recommendations. Are we ready for the next cybersecurity? Bruce is the father of cryptography stuff, I think. He says, "If you think that technology can solve security problems, you don't understand the technology and you don't understand the problems". He's right. I work with a lot of engineers. The client that I'm working with, we have 8,000 developers. I understand that the technology is not a problem at all. There is a topic that you might say, what is this, DevSecOps? Another buzzword, maybe? No, it's not a buzzword. It talks about these types of things, we need to look at these problems from different angles.
There are three pillars: there are some development works, there are some operation works, and there are some security works that currently are separated silos, but they need to work closer to each other. Instead of having the security guy at the end of the deployment that I want to go to production tomorrow, can you check my application, do the pentest and give me a sign-off? No, we need to involve them. We need to involve them in the early stages. We need to work more closely.
Then we have DevSecOps that have tools as one pillar, and the two others are like governance and the people and processes. Governance, there are a lot of tools. Some of these tools are mentioned currently, like the SBOM monitoring, and also some other tools that you probably have in your organization. A lot of people talked about the compliance and the platform teams that can build those type of things. The people and processes are very important. It's really important. In my opinion, it's the most important part, because when the things come to the people, it's really costly, it takes time, it needs a lot of thoughts and energy. The cost of change, which, for me, is the most important factor in the architectural decision, the cost of change in the people and processes is really high compared to the other things. DevSecOps is a culture, but there's also some roles in the organizations nowadays that have some people that have a little bit knowledge of all these things, and they can work closely with the DevOps teams, or developers that are delivering the products. It will be really helpful, why? Because in the modern approach, we say, let's shift the security checks to the left: early, better.
From the moment that you're designing the application, you can do something really nice. It's my favorite topic. It's not a really desirable topic, but it's very important from a security perspective, threat modeling. You can think that this application that I'm building, I'm going to deploy it, develop it, it will be an awesome project. If this application gets hacked, what can go wrong? You can just have some thoughts about it. Ok, this application probably has access to this platform, that platform, and these types of things can happen, and if these types of things can happen, we need to do something. Better think about those things before the incident happens, so we can have a little bit of thinking, and maybe we can also change our implementation a little bit because of this.
The other ones, like in develop, the 4-Eyes Principle. I think it's a standard, but still a lot of companies, a lot of squads are trying to bypass this one. Secret scanning, a lot of companies are publishing their secrets to GitHub, and the hackers, they have automation to catch those ones. I had the chance to work with some threat intelligence teams that pay a lot of money to some companies that are looking into dark web, and they have really fun stories to tell, how many passwords, tokens, secrets are on the dark web with really nice prices. A high administrator of a company, you can find his secrets, for example, the tokens in the dark web for €10,000, for example. Do the secret scanning, it's simple. If you have GitHub Advanced Security, I think you will get it by default on your project, but some of the companies, they have their private Git repository.
The rest of the things that you can see later, there is a link in the previous slide, DevSecOps guideline, you can take a look at it. There are a lot of details about all the steps that I demonstrated. You can see it on the next slide, all these steps and the technical implementation or the consideration that you need to have. Still, production is the most important part because the hackers start with the SQL injection thing that you have in your project. There are a lot of strategies to cover production. What we discussed was continuous dependency monitoring with SBOM monitoring. You can also take a look at the other ones, and most probably you're familiar with the other ones as well because they are not new.
You either die a hero or you live long enough to see yourself become the villain. If you think this is about Batman and Harvey Dent, you're wrong. I'm talking about all the dependencies and all the frameworks that we had or we still have and we don't know how to get rid of them. I found it easy to take care of those things in time when we are working, trying to upgrade the dependencies instead of having a big migration project or having a lot of hassle to fix them.
Participant 2: I've got one question about updating the dependencies. You had an argument against using the latest version. On the other hand, you advocated for using Dependabot. I was wondering, what is the difference?
Soroosh Khodami: Update your dependencies, but I'm not encouraging you to always use the latest. You can always try to use N-1 version. Still great, maybe not the latest and greatest, but it is something that has been tested. It's ok. Maybe there are some specific cases that you have to use the latest-latest, but try to stick to the N-1. In Renovate and Dependabot, you can configure that I always want to use the N-1. Dependabot can create automated pull requests for you, like, every week check if there's a new version, create a pull request, all the dependencies that can be updated. If you have good pipelines, you can also have an automated check that this dependency unit test, integration test, if they pass, we can have continuous delivery as well. Go to production.
Participant 3: Normally with Renovate, you can pin, you can give a range. If you have five sub libraries and each of them would pin urllib, probably the dependency resolver would just say like, yes, I can't find a solution, I will install the oldest version. At what level would you pin the dependencies? Like in particular in shared libraries in a company, you normally don't want to pin, for example, urllib, because the one shared library will pin it to 1.2.3, the next one to 1.2.4. Renovate will be like, I don't bump, I can't find a solution.
Soroosh Khodami: The answers depend. You need to give it a try. If Renovate cannot address your specific case in your organization, because maybe some of your teams that have a special way of releasing the version or a special way of versioning, we can still do it manually. How many internal libraries are you using every day? I don't think that the number is more than 10 or 20 maybe. You can still try to configure it, or you can also write specific scripts. We had a case that our developers said Renovate is not addressing our requirement. They write a Python script that could check the version and create a pull request. It's not that much difficult, especially with AI, you can just generate it. It's like one hour or two hours of work.
Participant 4: What about pinning to hash instead of pinning versions?
Soroosh Khodami: That's a good one. That's exactly what the lock-file actually do for you. When you installed it, you tested it, you have the lock-file, commit that one, and try to configure your pipeline to just use the lock-file instead of the package.gson, or the pip requirement file, something like that. Some of these package managers are not really smart. For example, pip, they do not offer the lock-file out of the box, you need to configure it. npm have it out of the box. Yarn have it out of the box. Maven does not have it. You can implement it with some plugins. Gradle also supports it out of the box. That's a really good suggestion, definitely, yes.
Participant 5: Do you know of any efforts for the package managers to become more robust to these things themselves rather than everybody having to work around them?
Soroosh Khodami: They are working on it. I'm not sure why they are not prioritizing these things. I was checking and I realized that npm, they added some new measurements on accounts. They currently invalidated all the tokens under npm, and you need to request new ones and they have more security measurements. They are really slow. It's a really slow process. I don't know why. Also, security experts are really talking about it every day like, why is Microsoft behind in npm? They are not picking it up faster. Also, the Maven one, I also don't know why some of the features are not available in Maven. We need to maybe request, or maybe it's about demand. We need to demand more.
Participant 6: In the company I work for, with Log4j, our main problem was that the CD/CI pipeline got overloaded and we don't have enough resources to rebuild all the software. Is there any solution for that that we can use depending on this SBOM management to prioritize the packages that need to be rebuilt first?
Soroosh Khodami: I think you might need to implement something in-house for your case, maybe. Some of these tools, they also cover special cases. You need to actually experiment with some of these tools and see which one fits more in your organization. For example, there are also some tools like Qualys. They also offer something not really different from the other companies, but they also cover some special cases for the companies that, for example, have on-prem environments. You need to really take a look at them.
See more presentations with transcripts
