Transcript
Davide de Paolis: I'm Davide, engineering manager of the platform team. We are in charge of the cloud infrastructure and the cloud spend. Recently, we noticed concerning trends and increasing costs. To reduce these costs and improve the security posture, we came up with some guidelines and best practices that you will have to follow and you will have to apply to all your projects by the end of the month. Yes, if you don't apply these changes, your deployments will break. No, we didn't have time to talk with your project managers because it's urgent. We know that you have deadlines, we have deadlines too. Compliance is a company goal and we all have to deal with it. Feel free to escalate. Let's stop it here and take a step back. This is not a good start and it's going to end poorly. I'm Davide de Paolis. I am an engineering manager of a platform team at Sevdesk. I am an AWS Community Builder. When I'm not working as a platform engineering lead, I love to spend my time climbing and listening to black metal.
The Sevdesk Platform Engineering Team (Foundational Work)
Sevdesk is a company that was founded in 2013 in Offenburg. We build accounting software for small businesses and freelancers. The company grew as many startups, and right now we are about 15 teams, considering data, security, platform, and all the core products, with about 80, 90 engineers, mostly remote. As the company started to grow 6 years ago, the need of a platform team taking care of the infrastructure was evident, and the most talented and experienced developers that had some background in DevOps and cloud engineering were taken from the product teams, to form a platform engineering team. This team was very ambitious and they immediately started rolling out roadmaps for a service catalog, dreaming of an internal developer platform, and they wanted to become a cloud center of excellence. When I joined the company, I found the Confluence page, huge documentation, and I was very impressed by that.
There was a problem. The good thing was that Sevdesk has always been a cloud-native company. There was never a time when we were on-premise or we had on-premise. We were on AWS running workloads on EC2. This team added EKS, Kubernetes cluster, for production and development because they wanted to migrate all the workloads and build a platform. In all their dreams and in all their ambition, they were also very afraid, scared of the vendor lock-in. Despite being on AWS, they did not leverage the AWS offering. On these clusters, they started adding a lot of third-party tools, services, libraries, and homegrown solutions. Keycloak for access management. Rancher to provide the developers with a nice, friendly UI for Kubernetes. Longhorn for elastic block storage. Harbor as an image repository. All these required so much work for maintenance, patches, and updates that the team had very little time then to work on the platform, on the reusable components for the teams. So much so that the team basically burned out and was dissolved.
After a few months, another team was formed. I joined that team with the goal of rebuilding and regrowing the team and prepare a plan for a platform roadmap. We wanted to reduce the burden on the team. The first thing that we have done was upgrading the setup because the company was growing, and a single account setup was not enough. First thing, we moved to AWS organization and we created a multi-account setup with different accounts by environment, by team, by domain, and department: security, data, backups, and so on. The reason for this was that you have an isolation of data, infrastructure, and access. You can limit the access to sensitive data. In case one account is compromised, all the rest are still safe. Another reason is security controls. You can tailor the policies by a specific application, and you can also adjust them depending on the environment.
You can be a bit looser in the permission for the sandbox account, for example, and let the developers do whatever they like, experiment with all the services. Only after they pass the prototype phase, you can restrict access or check the granular permission on all the other accounts. Quota allocation is also one reason. Because AWS has several quotas and limits on the services and on the APIs that are shared by all the workloads on one account, if you have multiple accounts, you reduce the risk of hitting these limits. Then, cost allocation, which allows separate billable items and assign specific costs to the teams, to the owners. This was the first foundational work that we introduced.
Cloud Engineering is Hard
Then, we wanted to reduce the burden on the team and leverage AWS. We replaced Keycloak with IAM and Single Sign-On. Longhorn with EBS. Harbor with ECR, and Inspector so that we had all the images scanned automatically. Then together with security, we set up Security Hub, AWS Config, and GuardDuty. It was awesome. We were very happy. Security was, of course, very happy because everything was natively integrated. We introduced another problem. Engineers were not so happy because cloud engineering is hard and is also quite different from software development. Change is hard. We made all these changes on our side, but those needed to be reflected on the projects, in the configuration of the projects and in the setup of the developers. The developers were used to having one account and just push to production like this. Now they have to switch from one account to another. They have to change the pipelines from pushing to Harbor to ECR.
Not only that, changing the pipelines to promote images from one account to another. That was a lot of work that we basically pushed on them. They also needed to learn these new skills because they were not used to that and they didn't know these services, and they had to somehow learn all this. Knowing what happened with the previous team, we really put a lot of effort into avoiding what I showed you in the initial meeting. We wrote documentation. We reached out to the teams with kickoff meetings. We set up a Slack channel exactly to answer the questions for the developers. We were getting over and again the same requests and sometimes some nasty comments. "That's a shitty developer experience. It was so easy before, like pushing to production or doing whatever I wanted. Why do I need to do that? Now it sucks." On our side, frustration was building up because we had to answer these questions.
We had to answer all these comments. We were asking, why devs don't just get it. Why don't they just read the documentation? We have documentation, plenty of it. Why do they keep on asking questions? You cannot see what's that because this is, of course, in an internal document, but you can see how long it is. This is an example from our Confluence page about the tagging policy. Very long, overly detailed, in my opinion, also quite boring. What's worse, in many places, outdated. Even if developers read the documentation, they would likely have made it wrong. Let's use this example, the tagging policy, to talk about internal compliance initiatives and how we can roll them out with or without friction. Tagging resources is really no big deal. It's just a bunch of tags, key and value pairs, where you define an owner or the criticality of the service, the name of the service, and a set of values. For example, with this documentation, if we document the values, nothing prevents people from making a typo or choosing a value that is not there because it's just a documentation. Let's take this as an example, a very simple example of how to roll out internal compliance without friction.
Encouraging and Expediting Change (AWS's Support)
The question is, how do we best encourage and expedite change, and how can AWS support us? There are two things to consider and both have to do with us being human. Nobody likes to be patronized and to be told what to do. This is something that security and platform teams often do. Platform teams are away from the trenches. They are not under the pressure of product. Nevertheless, they sometimes come into the team and say, tomorrow you cannot do that anymore. You have to do this. You have to follow our guidelines. We know how things should be done. Of course, people don't take it very well. Another thing is that these changes are very hard. It's like a lifestyle changing, like a diet or working out. If you are forced to do that, it's very likely to fail. Of course, we are speaking of enterprise environments, so sometimes we have to do what we have to do, but still onboard people, have them share the same purpose is very important for success.
How can we expedite change and how can AWS support us? AWS has a lot of tools that are natively integrated that allow us to check the resources, inform people, and enforce when we want to apply and be strict on these compliance measures. The nice thing is that these mechanisms are separated. You can have one round of informing, and then only afterwards becoming strict.
AWS organization, which I mentioned, was one of the first changes that we introduced in the setup of our company as tech policies. Tech policies are very simple. It's just a configuration of tags and keys. Here you see Sevdesk owner. This was what we really care more about, understanding the ownership of the services, because we had gone through a lot of transformation, a lot of reorganization, and some workloads. We didn't even know, as a platform, who were maintaining them. Sevdesk owner was one example, and the values are cloudcrew, that is my team, and then team-a, team-b, orphaned, because of this reason. Some workloads we didn't even know. They have no owner at all, and we wanted to monitor in terms of cost and in terms of incidents, in case we need to raise that to the attention of the management. It's very simple. You define the key and you define the values that are accepted.
This does not prevent you from continuing deploying without a tag. You can deploy. I cannot deploy with a different team name, but I can deploy without a tag. In order to prevent that, we have service control policies. Service control policies are guardrails that prevent undesired actions. In our case, we are denying to create a cluster, create a Lambda function, create a database, if the condition is not met, if there is no Sevdesk owner. As you can see, the configuration is fairly simple, but it's also very tedious, because you have to specify very granularly all the resources that are affected and the specific actions. You need to tell, this policy affects Lambda, and the create function, and the delete function, or whatever.
What to Enforce?
We started building this service control policy, and we, in small iteration, started to deploy that until we found that there is a limit of length in the policy, 5,000 characters. Yes, no big deal. We will split the policy into multiple policies, so that might even be a good idea, because we have one policy only for the foundational resources that we maintain, that don't change that much, and then we have another policy for all the applications that the teams are deploying. Then maybe we can create another one only for serverless, which is something that we use, but not that much. We split that. Easy. No, because there is also a limit on the number of service control policies that the account and the organizational unit can have. That was a problem. On top of that, there are minor differences between services. Some services don't have them.
Some services have request tag. Some other have resource tag. It was really time-consuming. Then we stopped, and we said, let's change approach and focus on the heavy hitters. We want to control and to introduce the compliance only for the top contributors in the cost, so the services for which we are spending a lot, and for the services that we are using the most. We don't care about Elastic Beanstalk or any very old application or service that we don't use. To do that, we used this tool called Steampipe, which is a very nice little tool that allows to run SQL query against the AWS API or any other provider that you want. Probably now, I would have explored another approach, probably with Amazon Q and some MCP servers that are available, but back then, this did the job very well. We knew what to focus on to enforce the compliance criteria. Then we needed to find out how to monitor all the other resources.
Inform and Enforce
We had our policy, a smaller policy, and we were ready to deploy that. Let's wait, because the previous team failed on that. The previous initiatives were still struggling in the communication. When we announced that we wanted to revamp, revive the tagging strategy, we said, this time will be different. We will not simply roll out a change that breaks everything. We want to work together with you. We want to allow you to plan. We will inform you properly. We weren't sure that we could live up this promise, because we knew that with this deployment, deploying the service control policy, the deployments would break, but we didn't know how much that could have been a problem. We didn't know how many resources were not compliant. It was our responsibility to understand that, inform the teams, and come up with a plan. Luckily, that was about the time when AWS made publicly available, the resource tagging standard, which is another type of control that is added on top of Security Hub.
We were already using Security Hub, and security already activated the foundation benchmark and the foundational security. We just added the tagging standard, defining the tags that we wanted to monitor. Owner, service, user data, criticality, confidentiality, these were the tags that for us were crucial, not the 25, 30 tags that were listed in the original Confluence page. They were irrelevant. This was really what we wanted to have. We activated the tagging standard. We let Security Hub run for a day. The next day, how many resources were not compliant? A lot. Way too many. It was a bad surprise. On top of that, the bad surprise was that many of those resources that were failing the control were ours. My team was owning those resources because, of course, as a platform team, we have a lot of clusters, instances, roles, permissions, security groups, and so on, and all those were not tagged. Definitely, it was good that we didn't roll that out because we would have problems, and most importantly, it would not be a good example. Platform teams should lead by example and should tell people, this is the way, and if you're not complying to that, it's a little bit uncomfortable to say.
Now we knew how many resources were not compliant. How can we inform the teams? We wanted to analyze the data and then inform the teams so that they could plan adding the tags on their resources. We came up with two ideas. First, broadcast weekly reports about legacy non-compliant resources. We know all the resources that are not tagged, and we analyze them, we assign them to a team. Then we tell each team with a Slack notification, "You have X amount of resources that are not tagged. Please do that." Then, next week, "Thank you, you are improving, or please put some more effort or plan some work on that." The second is dispatch warning on Slack, as soon as the resources were not created, because we didn't deploy the tagging policy. Teams could indeed continue deploying resources without any tag, but we want to catch that. Unfortunately, the amount of resources that were caught by security app was so high that first it was very hard to find information there.
There was a lot of noise because AWS for each deployment deploys under the hood some other resources, and so they were really way too many. They were so many that even if the teams would improve one or two resources by day, in this percentage, it was basically invisible. We said, we just need a script to parse all this information, suppress the noise, clean it up, and then we send out the message.
Should be easy? Of course not, because one of the main problems was, most of the resources were not tagged, did not have the owner. If they don't have the owner, how can we know who to notify? You might think of the created by tag, which is automatically added by AWS to every resource. This is true. It works. First, this is the tag that is yours, not the team or not the department. Second, it only works if you deploy from your machine or from your console with your account, which is something that we should not do. If you are using infrastructure as code and if you are using pipeline, you will be disappointed to see that the created tag is basically the role assumed by the GitHub runner, for example. Not much useful. Considering that tags are inconsistent, the owner was not available, and we needed to find ways to retrieve that from other sources like CloudTrail or the CI/CD, we decided to rely on all these resources, and the script became more of an epic.
At the end, we came up with this solution. Relying on AWS config, which is something that we already added together with Security Hub, we detect the resources when they are being created or edited, or the configuration is changing. Having this filter, we check if the owner tag is present or not. If not, we have this event, EventBridge triggers a Lambda. The Lambda processes this event to make sure that the information is available. If it's not available, we check the pipelines because the deployment followed a pipeline workflow. We look into the pipeline, we understand who is the owner of that pipeline repository, and then we can figure out the team that is managing that resource. We push everything to a queue. The queue triggers another Lambda aggregating the events. Then we can push them to AWS chatbot that was renamed to Amazon Q, even if it has nothing to do with an AI code assistant.
Then we push it to Slack. Why the queue in the middle? Because for every deployment, imagine we are deploying an application, Lambda, API Gateway, Dynamo, and S3. You have four resources. We don't want to send four resources plus the IAM roles to allow the permission in between. We don't want to send six messages. We don't want to spam people. That's why we have a queue in the middle. That was to create the Slack notification, and immediately go after people if they deployed something without tags, because we already informed that we were working on the tagging policy, and we want them to already make changes. If they don't, we catch them immediately. The second script is to create a report. It's very similar, still relying on EventBridge. EventBridge this time is just a schedule, weekly, trigger a Lambda. The Lambda retrieves the data from Security Hub.
Then it does basically the same, checks if the values are there, otherwise retrieves them from GitHub, and then it pushes them to a report generator. The report generator saves the stats, basically, on DynamoDB, compares them with the previous week, and then send a message to the team saying, thank you very much. You are really contributing to our success, to the success of the tagging initiative. Or, on the other way, "Shame on you. You are not doing your part. Please do so. Otherwise, in three months, six months, we will really start enforcing, and then it will be really your problem because we did our part. We did our best."
Rinse and Repeat
Having this, we are able to have a constant flow of information, of data that allow us to understand where we are and what is missing, and who has to plan work for that. The good thing with this initiative was the tagging policy, as I said, is very simple, but this process of informing, soft enforcing, and then enforcement can be applied for every internal compliance initiative. Rinse and repeat, change maybe the scripts, the concept still proves true and works within a company of any size.
How to Get to Clarity
How did all this help us, and how can this help you? The biggest reason for failure in the previous team a few years ago was that the company was growing very fast, and they felt the pressure of fixing all the problems and achieving internal compliance all of a sudden. Too much was attempted in too little time with too few resources. They complicated their lives with the vendor lock-in story, but it's a different thing, my opinion. Most importantly, they lost the focus on the goal and they lost the focus on the people. There was a disconnect. That was what caused frustration and friction. How do you avoid that? Product teams, security teams, platform teams can do anything, but they cannot do everything. It's very important to start defining a minimum viable governance, understanding what is really important for the company at the moment, and focus on that.
You cannot work on all the problems. If you look into Security Hub and you have 12,000 security vulnerabilities, you need to start from something and you need to create a prioritization. Relying on the metrics of urgency and importance and impact and effort is crucial. Focus on the stuff that is important and urgent. Then, in my opinion, it's also important to focus on the important stuff that is not urgent, so that it doesn't become urgent. Otherwise, you are always reacting to the problems and not being proactive and addressing issues. Impact, effort, I really hate the idea, the wording of low-hanging fruit, but it's true. There are a few things that, with little effort, bring a lot of impact. Focus on those. Share goals and purpose. Changing is hard. Understanding why we are doing things or why are we imposing things to others, to make it easier, it's important to share the goals and the purpose.
If people understand that this is really a company goal, not because it's written on the website or in the internal documentation, but because it's important for the success, for the customers to know that we have processes in place and we are following them, they are more likely available to support these initiatives or to put them into their own roadmap. Then, iteration. Small iteration based on feedback and changing priorities. Defining a roadmap is very important, but usually the roadmaps of platform teams have deadlines that are arbitrary. We decide that we want to have all the resources tagged by December. Of course, we don't want to fail our own goals at the end of the quarter. Still, we don't want to blow up the entire company or block the deployment and have every team screaming that they cannot proceed with the next release. Understanding this, following data-driven decisions based on the data that you have, you see, ok, we wanted to be ready by December, but we still have 50% of resources not tagged.
It doesn't make sense to do that. You can escalate. You can involve management to have some more push on the teams, but still, you need to change the priorities. It really makes no sense to just blow up everything. Then, shifting the communication and the ways of working. We already changed the way of communicating, informing proactively the teams with these messages. We opened up with the communication, with Confluence, with DevOps champions, and regular meetings where we were answering questions in person. We introduced the Tour of Duty or Embedded Expert. Basically, you have one team member going into the different teams and supporting them, explaining what we are doing, explaining how we are doing things, explaining what the changes are. Sometimes even doing the groundwork for them so that there is progress, and then they see that we are there to help them. It was mentioned in one talk, the DevOps mindset is you build it, you run it, but we are here for you. You are not alone. I love that. This is basically what we should all do. Put the responsibility on them because they are responsible for their projects, but as a platform team, we can support them. We should support them. Changing the communication and the way of collaborating is crucial.
Compliance Is a Journey
Compliance is a journey, and it takes a long time. It doesn't happen overnight. It takes longer than you expect. In order to achieve all these goals, you need to identify the quick wins, to build momentum. Otherwise, there is something that we have seen, you start very enthusiastic with a new strategy, with a new initiative. You do a round of meetings with all the teams. You onboard everyone, but then something comes in the way. There is another priority. The teams are reorganized, and nobody talks about the tagging strategy anymore. Then you are forced to change the priority and to change the deadline, but not because there is a real problem, simply because people got a little bit detached from the issue. Keep the momentum high, iterate, and have patience because it takes a long time. It takes a lot of communication. People might be frustrated, and you need to deal with that. If you remember, patience and empathy on both sides, then this is something that can really help you in achieving the goals. Changes are, at first, messy in the middle and gorgeous at the end.
Questions and Answers
Participant 1: In terms of the interaction with developers, so basically the UX for them, for the platform, have you considered using any off-the-shelf components like Backstage or some type of IDP to give them those insights, or do you choose to roll your own for different reasons?
Davide de Paolis: No, we haven't chosen to roll our own. We evaluated Backstage. We haven't decided yet. I believe, after some discussion with the team, with the chief architect and with the CTO, it was a bit premature. It is premature at the moment to come up with a solution like Backstage. Right now, we want to start slowly with a tagging strategy, with fixing some security vulnerabilities. This has nothing to do with the compliance, but more with us being the platform, migrating the workloads from the old accounts to the new setup with EKS, and supporting the teams in the carving out of a monolith. We do this by supporting them with Terraform, with creating modules. It would have slowed us down a lot starting the onboarding of a tool like Backstage. This is somehow in the plans. As team size grows, we will do that for sure.
Participant 2: My question is regarding, if you yourself have a deadline that needs to be implemented because of some regulations, like EU regulation or something, and that is affecting the compliance, how do you balance that with the product deadlines and how do you communicate that with the teams?
Davide de Paolis: If I have a deadline?
Participant 2: There are EU regulations coming regarding compliance, security, risk, and everything, which you have very tight deadlines, because then you are violating the regulations, and these have like very tight deadline.
Davide de Paolis: In this example, I'm referring to internal compliance and internal governance. If you are talking about audits and regulatory compliance, that's a different story. Then you need to be very clear that there is no way around. It happened to us, because working with an accounting software, we have, every now and then, these kinds of audits, and these get the full attention of the company. In that case, there is more push also from other sides. Normally, the problem is platform is here and has to go to all the teams and say, let's fix this. For regulatory compliance, normally, there are other departments: there is legal, there is finance, there is security. There are multiple forces pushing all in this direction. I'm not saying that it's easier, but you have more support and more mandate.
See more presentations with transcripts
