Transcript
Bergman: The next decade of software is about climate. The first time I was told about climate change, I was in primary school. That means that for my entire lifetime, we have been aware of the rapid rate of climate change, and we have deemed it important enough to teach kids in primary school. Since then, not much has changed. If anything, the situation has gotten more dire. We have used more of our finite carbon budget at increasingly higher rates. What has changed is the conversation. Climate action has risen to the top of many tech companies' agendas in the last few years. This is evidenced by many headlines. Companies are making pledges to be carbon neutral, to achieve carbon net zero, to fight climate change in our own sector in our own arena. Organizations are committing to act on climate.
Background
My name is Sara Bergman. I am a chairperson of the Writers Project in the Green Software Foundation. The Green Software Foundation, we are working to create a trusted ecosystem of people, standards, tooling, and best practices for building green software. I'm also a senior software engineer at Microsoft, where I work with M365 products as a backend engineer. I've been interested in the field of green software and IT since I was at the university where I was lucky enough to get to take a course which was truly eye opening for young Sara. I'm not alone. There are many people with a passion and engagement in this area who have been passionate and engaged for a very long time.
If we think about the climate solution, from a helicopter perspective, there are really three key phases. The first is understanding the nature of the problem. This has been known for my entire lifetime, likely been known for all of your lifetime, as well. The second is understanding the nature of the problem. Once again, I would argue that's been known my entire lifetime, and likely all of your lifetime as well. Number three is getting investment into the things that we need to do to solve this at the scale we need to solve it. Here is where things get stuck. We're still very far away from actually meaningfully solving this problem. We need massive action at scale. To quote another Swedish young woman, "No more blah, blah, blah, we need action." This is true globally, but it's also true for companies in our sector. I'm not the only one who's been working on this. As a community, there are a lot of learnings we've had over the years on how we get this number three, how do we get the investments? How do we get it happening?
What Is Green Software?
If you're looking for a light talk with some few tips and tricks, and actionable advice that you can then walk away from feeling a bit relieved, this is not the talk. We are past that. Now it's all about number three, how do we get those investments at the scale we need to do for the things that we know that we need to do? That begs the question, what is green software? There are two distinct branches here. We have green by software, and we have green software. Machine learning, has a role to play in both of these. Green by software is where we use software as part of the climate solution. We do things like use software to improve the fuel efficiency of your car or to monitor the coral reefs. This has been an area where I think we've been very good at highlighting what AI and machine learning can do for us. It's very clear the benefits that we can have here. The flip side of green by software is green software. This is where we recognize that software is a part of the climate problem. The software we build and the hardware we need to run this software has an impact on the climate.
I think this is less talked about, especially when it comes to machine learning. It begs the question, how do I do that? How do I build green software? For the answer to that question to be actionable, it needs to be very narrowly defined to a particular type of application. If you're a web developer hearing about actionable advice for neural network pruning techniques to make a machine learning training model greener, it's not going to be very applicable. Likewise, for this audience, maybe it's not so applicable if I talk about how we make an Android app more battery efficient. Green software means different things for different people. In your role, what you need to do is to be able to apply some knowledge into your very context specific domain. Because green software engineering is incredibly context specific, there's no magic bullet, there is no one thing we can do, and then magically, everything will be better. To be honest, I think that's what makes it even more interesting. We need to really apply our natural curiosity as programmers, software engineers, data scientists to really figure out what's meaningful in our area, because the only way this really works is that we teach you the fundamentals. Then you take your domain knowledge, and you apply it to your own context. The Green Software Foundation is working on a pattern's database, so you will get some insights there. Obviously, these databases won't be complete, not everyone will agree.
Green Software Engineering is about Climate Change
The focus on green software engineering is climate change. It's about carbon. Carbon isn't only the C in the periodic system, we typically mean carbon dioxide, which is a heat trapping gas that is emitted by human activity and other activities, and it creates a blanket around our planet. This is a greenhouse gas. It is not the only greenhouse gas, there are several others like methane. It becomes quite tricky to reason over all of them, so there are some very smart people who have created this conversion table, to compare all different greenhouse gases to one specific measure, which is called carbon dioxide equivalent. Typically, when we say carbon, we mean carbon dioxide equivalent. What we're focusing on in the Green Software Foundation, and what we're focusing on in this talk, is software as part of the climate problem. That naturally means we want to reduce the carbon emissions that is the result of software. That brings us to the term carbon efficiency. In our industry, everything emits carbon. That's not true for every industry. Where I come from in Sweden, we have a lot of paper and cardboard industry. They plant trees, and trees actually absorb carbon dioxide. In our industry, practically everything emits carbon. Our goal as green software engineers is to extract the maximum amount of value of every gram of carbon dioxide that our applications are responsible for emitting into the atmosphere.
Reduction vs. Neutralization
Then you may ask, why don't I just offset? It seems easier. It seems to be less work. It's a good question. Ponder this scenario, these are your emissions that you have, and your goal is to cut them in half. You can either do that by actually cutting them in half, actually reducing your emissions by 50%. Or you can do nothing, business as usual, and then you neutralize your emission by 50%. Often, these are conflated into meaning the same thing, but they are not. The goals we have set as a community for 2030, and for 2050, they are about reductions, not about neutralizations, because neutralizations are not permanent, while emissions are.
Carbon Cost of ML Training
What makes machine learning special here? Let's look at this paper. This paper by Strubell, Ganesh, and McCallum is from 2019. It's heavily cited, and it's a very well written paper. I would recommend reading it. What they did here is to characterize the dollar cost and the carbon emissions of deep learning in natural language processing. They did that by estimating the kilowatt hours required to train a variety of popular off-the-shelf NLP models. Then they converted these kilowatt hours at a fixed rate to carbon emissions and to electricity cost. They analyzed four models: Transformer, ELMo, BERT, and GPT-2. The interesting part here is this column, carbon dioxide equivalent. These numbers are quite high, they are in pounds. One of the things that a lot of people took from this paper is the neural architecture search, which emits 626,000 pounds of carbon emissions per training. That is five times the lifetime of an American car, which is mind boggling when you hear it. It's not a conversation we're really used to having. If we also look at the compute cost, we can see that this is not really the type of query we randomly do on a casual Tuesday afternoon, we are typically in the higher end of this graph. Even there, the cost is quite high. For the BERT one, it's comparable to a Trans American flight. What we can see here is the carbon cost of just training models, which is just one part of the AI or ML lifecycle, the carbon cost is quite significant.
AI and Compute
Another thing that is important to talk about when we specifically talk about machine learning and green software is this. This is an OpenAI blog post, where they look at the amount of compute that's needed for state of the art, very compute intensive models for their time. They looked from the '60s until 2018. You have petaflop days on one scale, and you have years on the other. There are two distinct areas. In the first phase, the amount of compute that AI workloads needed to run, followed Moore's Law. It had a two year doubling time, and it was supported by Moore's Law and improvement of hardware. Then there's a sharp break in the curve around 2010, or 2012, where we're starting to see a 3.4 month doubling time in the amount of compute that is needed for these large AI training rounds. That means that between 2012 and 2018, this metric grew by 600,000x, which is a very big increase. This is obviously also supported by better hardware, increased user prioritization, better use of their GPU. Why this is interesting to talk about is that the scale that AI and ML is increasing, is not just the use case, we're using ML and machine learning for more things. We're also growing the size of what we're using. In the context of green software, sure, AI is exactly the same as all the others, we still need to think about some of the fundamentals. What's different about this field is that the growth rate is really fast compared to some of the others.
Sustainable AI
What is then sustainable AI? I quite like this definition by van Wynsberghe, they say that sustainable AI is a movement to foster change in the entire lifecycle of AI products, so idea generation, training, retuning, implementation, governance, towards greater ecological integrity, and social justice. I really like that they focus on the entire lifecycle, because in the two examples we just saw, the conversation is about training. It's quite easy to do research on. If you're a production company, training is not likely where you're burning the most carbon, it's likely in your production queries of how you're utilizing these in production, because any number times a million or a billion is going to be a large number.
Power Usage Effectiveness (PUE) for Software
Now we have an understanding of what this field is and why it is important. Now, how do I do the thing? How do I reduce emissions? What we need here is a PUE for software. PUE stands for Power Usage Effectiveness. It's something we use when we talk about data centers. How much of the power that comes in actually goes to useful computation, how much goes to cooling and the likes? How do we measure the amount of carbon that our software uses, and how can we fundamentally reduce that?
Hardware Efficiency
There are three fundamental actions to reduce carbon emissions. The first one is hardware efficiency, because all the hardware, all the servers, the compute devices, the network, the client end devices, whatever hardware you're using for your software system, they emitted carbon when they were created, and they will emit carbon when they are destroyed or hopefully recycled. If our goal is to be carbon efficient, as green software engineers, that's the ultimate goal, then it follows that we must be efficient with hardware. This manifests itself in different ways, depending on your context. If you're in the cloud, and you have a typical I/O bound application, then that usually translates to reducing the number of servers that you have reserved for your usage. Especially if we look at the global average on-prem server utilization, which is 10%. Quite low. If you're an app developer, on the other hand, if you're building production code, then yes, they might mean something else. Utilizing the hardware on the device better, thinking about battery lifetime, thinking about how you can code in a way so your device doesn't become obsolete. Because how many times have you thrown away a working device just because it was out of security support. Don't let your applications or your software be the nail in the coffin for working hardware. That is not very green.
Speaking about hardware, we have the client end devices, of course, but we also have the devices that are on our side in our data centers or on-prem. Here we can think about specialized versus general purpose hardware, and specifically in the context of machine learning. If you can use specialized hardware like ASICs, or TPU, or FPGAs, to some extent, to speed up the round of training, then there can be significant efficiency gains. It's a simple math equation, if you have too slow hardware devices, and you can decrease the time, to cut the time you run the training in half, then you can just have one device. That is one way to be more hardware efficient. Simply get more out of the hardware that you already have, so that you don't have to buy more, because if you buy more, we need to create more. Utilize what you already have. Specialized hardware can be a great way to get those efficiency gains. On the other hand, if you already have general purpose hardware like CPUs, it is likely much better to use and to optimize what you already have over buying new specialized hardware. Because if you already have a CPU, the carbon that was emitted when it was created, it's already out there. That's gone. We can't affect that anymore, we have paid the price for that. Make sure you're getting the most out of it while you have it.
Another thing that can be important to think about when it comes to hardware efficiency, is server density. Assess your hosting efficiency. Think about that on-prem utilization of 10%, which is the average, and consider your own strategy. Is it good enough? Is it even good? You can have high server densities in several different ways but you can achieve it through orchestrators or cluster schedules. You can have stuff run like Kubernetes, or you can have managed. There are a lot of things and a lot of engineering investments you can do that, but the easiest first win is to just turn off those old machines. Right-size or downsize others. Don't take the biggest just because you might need it, think about what you actually need for the task, and then set up autoscaling. Turn off defunct zombie workloads that isn't doing anything. Finish your experiment, decide when they're done and shut them down. This is important to consider no matter if you're on-prem or if you're hosting in the cloud. Server density is always a good thing to think about.
Energy Efficiency
The second fundamental action to reduce carbon emissions is energy efficiency, because energy is a proxy for carbon. In the world, energy generation stands for a significant portion of the carbon emissions in the atmosphere. It follows that using less energy, we will emit less carbon. This relationship isn't exactly one to one: if you cut your energy in half, you'll always decrease your carbon in half. Sometimes it's more and sometimes it's less. There is always a relationship that if you use less energy, you emit less carbon. How do you get a handle of how much energy you're using? If you're using Azure Machine Learning, this can be a good option for you. Azure Machine Learning has released rather recently, a new set of resource metrics to help customers understand the computational and energy cost of their AI workloads across the machine learning lifecycle, which is my favorite part of it. It focuses on the entire lifetime, because where you are spending the most carbon or energy isn't necessarily the same place that I will be. Using these energy efficient machine learning tactics can really help you utilize your cloud computational resources more efficiently, which leaves you to both save costs, and also make it greener. These capabilities include surfacing some new research metrics, like the GPU energy memory utilization and computational costs for machine learning compute. This is a Microsoft initiative with the Azure Machine Learning. They are not alone. They're doing this in partnership with the Allen Institute for AI and with the Green Software Foundation. If you're already using Azure Machine Learning, I would recommend checking this out.
Another thing to think about when thinking about energy efficient software is using smaller models. Can you train models faster and more efficiently by making them smaller? There are several research initiatives which are exploring how to train models faster and more efficiently, that rely on pruning, compression, distillation, and quantization among other techniques. The goal is to use fewer computational cycles. Because once again, if we use fewer computational cycles, we use less energy. Because a server that's on consumes a baseline of energy, so that's why you should always have a high density. Whenever you run something, you increase the energy utilization. If you use fewer cycles, you can pack more things onto the same server. Another reason for shrinking the models is that edge computing and Internet of Things means that we're seeing more devices with limited capabilities in terms of memory and compute, and smaller models will be the way to go here. If this seems intriguing, I would recommend checking out these articles from my, both Microsoft and Green Software Foundation colleague, Abhishek Gupta. I think the article is really good, https://thegradient.pub/sustainable-ai/.
Carbon Awareness
This third fundamental action to reduce carbon emissions for solving our software is carbon awareness. This is where we don't use any proxies, we don't think about anything but pure carbon. We really go to the source, the core of things. This can be achieved in several different ways. One example is consuming lower carbon sources of energy. For example, I'm based in Norway, where we have hydro. Hydro can be used throughout the day. We have it at 2 a.m. and 2 p.m. The energy when I flick a light on is always going to come from hydro. If you're based in a different part of the world, your energy mix is going to be different. It likely is going to be different from day to day, if the wind is blowing, or if it's not, if the sun is shining, if it's not. That's an important consideration to have when you're being carbon aware.
Another option to be carbon aware is to consume electricity in such a way that it helps accelerate energy transition towards this low carbon future. It's all about variable power. Not all energy sources are created equally, some emit more carbon, so emit less carbon. What you need is a hosting provider with a commitment to sustainable hosting. A committed date to when they will have 24/7 low carbon power, and how are they progressing to it. Of course, this host could be you. You don't have to go to the cloud. It can very well be you, and on an on-prem solution where you have full control. Generally, it is easier to move to the public cloud. The trick here is to tell your provider that you care. As someone who is working for a cloud provider, cloud providers care about what our customers care about, so tell them what you care about. It's the way to go here. The second thing is to host in regions where there are good options for sustainable electricity. Ask your provider, what's your best region if I want to be green? What's competing for the power there?
Architecture. Think about what tasks you have that are on-demand that don't need to be on-demand. It's about this variable power. We don't want to burn a lot of electricity when there's only the carbon intense stuff available. What can we have sort of ice in our stomach and wait for the greener stuff. Machine learning training is a good option. Do you really need it now-now or can you wait 2 hours, 12, 48 hours, and use spot preemptive or flexible burstable instances to run this. Another thing to consider here is demand shaping. What is urgent versus what is important? We often think they are the same, but they are not really the same. Delay what is important but not urgent for green power.
Do less. The most efficient code is none at all. I think it needs to be said. We're also software engineers or data scientists or people in some way who work with software, so this is the most boring option. I want to write code. That's what I do for a living. Why would I stop? Think about if you're over-delivered. A big win can be changing requirements or SLAs. We've talked a bit about how being faster and using fewer computational cycles can be a way to be more carbon efficient or more energy efficient. If you're striving to really hard latency SLA for the sake of being fast, and you're doing it by throwing more hardware at the problem, like specialized caches, or whatever it is you're doing, then you're not being green. You're solving it by burning carbon. Really think about, is that needed for your business case, truly? Another thing to look at is log files. Can you reduce the retention time of the log files? Storage isn't really the biggest energy driver here. If you have so many log files, you need to buy additional storage, then we're not being hardware efficient, because we need to buy additional hardware to support our log files which maybe no one ever looks at. Relax overspecified goals and think about, are you over-delivering, or are you right-size delivering? We've gone through the three fundamental actions to reduce carbon emissions. They are being hardware efficient, being energy efficient, and being carbon aware.
How You Choose To Measure, Matters A Lot
As green software engineers, our fundamental focus is to reduce the carbon emissions. The problem is now, how do you get people to invest resources that are required into each of those three buckets? Here, how you choose to measure matters a lot. Because in order to understand what you need to do, you need to understand what the current status is. Once you start doing investments, how are they working? Are they doing anything or nothing? Measuring is something that maybe isn't fun. If we look at organizations, this is something we spend a lot of time resisting, because people are likely already measuring carbon emissions in one shape or form. To get them to measure something differently, that's hard. If we don't have a method for measuring software, which actually rewards people for doing one of the three fundamental actions that reduces carbon emissions, then people won't do those actions. You need something that guides actions, not just reports a number for the sake of reporting it.
This is where the SCI comes in, which is a standard being worked on by the Green Software Foundation, because we need a standard or a specification that actually rewards people for doing one of those fundamental actions. On the contrary, the protocol that a lot of companies have been using is the Greenhouse Gas Protocol. It's a great protocol. It's groundbreaking in its own way. It's a way to make sense and account for carbon. It's not ideal for software. Why is that? First of all, the Greenhouse Gas Protocol wants to divide emissions into these nice organizational boundaries, where Amazon's carbon should not include any of Microsoft's carbon and vice versa. For software, that doesn't make a lot of sense. For example, me I build Windows. Windows runs on many more companies than my company, but it doesn't get included into the Microsoft emissions, even though I can affect the efficiency and the carbon that is emitted because of it. For open software, they really struggle because they might not even have an organization, but they do affect a lot of other emissions. How do they get an overview over how they're doing? It's hard for them.
Consequential not Attributional
The Greenhouse Gas Protocol is attributional not consequential. This is where we tried to flip the script for the standard that's being built by Green Software Foundation where we try to be consequential instead of attribution. The Greenhouse Gas Protocol, it is a carbon accounting methodology. It does not care what carbon reductions are possible through your action. It just wants to do this, take the chunk of emissions that we have, divide it into these nice boxes, and say you're responsible for this. That's great for accounting. In reality, the picture might look like this. This is where the Green Software SCI are trying to come in. The SCI loves double book keeping. I might be responsible for some of your emissions that you might also be responsible for them. That's ok.
Growth?
Another drawback of the Greenhouse Gas Protocol is that it does not factor in growth. It's really poor feedback. What strategies are working and what strategies are not working? They typically just report a total like this, where you can see, ok, so the first quarter my emissions were at 34 tons, and the next it was 43 tons. That looks pretty bad. I think we're doing something wrong. It's very useful for reporting. It gives a clear, concise overview. It doesn't tell you what you actually achieved, and how far you have left to go. It doesn't tell you what you should be doing. That's a problem, because we need something that rewards people for doing action. For anyone who's ever reported or calculated their total, you know that we need to make some assumptions, we use averages. When we do that, we lose this high resolution that is necessary for people like us, for people on the ground, people close to the product, close to the code, to really figure this out. An overall total from Microsoft doesn't tell me what I need to do as a software engineer for a very limited part of our services. Now, if we flip the script and do this instead. This, I can work with. This is useful to me as an engineer. If we say that the carbon emissions were 3.3 grams per user, and then next quarter, they were 2.9 per user, then we took some action and they had an effect. Our total may of course still go up. This can be true at the same time, because maybe our product had a great boost in the monthly users. This is what the carbon intensity specification tries to do.
Software Carbon Intensity (SCI) Specification
The Software Carbon Intensity Specification, is a standard for carbon scoring a software application. It is being developed by the Green Software Foundation, and it's built from the ground up, biased towards actions. Actions that leads us to energy efficiency, hardware efficiency, and carbon awareness. This is much less of a reporting tool, and it's more of a tool to help software teams and people with feet on the ground to really decide where to focus your efforts in reducing it. That's what we need to do if we want to get investment into the right areas, we need to guide someone, what are those areas? This is one tool of getting that.
The SCI specification looks like this. I know it's math. It can be scary, but you should all be familiar with machine learning math. This is easy. This is just one line. The SCI, the Software Carbon Intensity is a rate. It's carbon emission per one unit of R. R is a functional unit, which you can choose. It describes how your application scales. Because green software engineering is incredibly context specific, so this needs to be something that makes sense for your use case. It can be users, or it can be API calls, machine learning jobs. It can be flops. It can be hours. It can be a benchmark application, anything that makes sense to you. That's R. The C is the carbon cost, or the amount of carbon that your software is emitting. That also consists of two parts. The first is the operational cost, where we have the energy consumed by a software system, times the location based marginal carbon emissions. Then, apart from the operational cost, you also have the embodied emissions of your software system. This is typically the hardware you need to build your software system, like a server is a good example here, or a network device, for example. There is more to be said and more to learn here, and I advise you to go to, https://grnsft.org/sci to read the full alpha version of the specification. From an AI and machine learning point of view, what's important that we start with is to instrument our current ML system to actually capture and report back the energy consumption data. Because that will provide us with this raw insight that help us make smarter decision about what actions we should take to make those changes towards a sustainable machine learning system.
Action Items
What should you do now? The first thing you should do is start the conversation. If no one is having this conversation where you work, now is the time to start. The second is get to measuring. Start instrumenting your system for gathering the data that you need, and start looking at what actions you can take to be more energy efficient, hardware efficient, and carbon aware. The third thing to do is to make green software a business priority, because we need to reach 45% reduction by 2030, and we need to be net zero by 2050. If we're still trying to pass off these engineering investments as something we do on the side when the leaders aren't looking, it's not going to work. It's not going to cut it. We need massive action at massive scale to reach these goals. We must make green software a business priority.
See more presentations with transcripts
