An unexpected change in the policy used by AWS Support raised concerns about access to customers' S3 data. The cloud provider reverted the change, stating that the permissions were not and could not be used and published a security bulletin. Security experts suggest steps to detect and prevent similar issues in the future.
On December 21st, AWS incorrectly committed and deployed a new version (v20) of the AWSSupportServiceRolePolicy, a managed IAM policy used by AWS Support automated systems. The new version inadvertently included the S3:GetObject permission that could give AWS Support teams access to customer data. Scott Piper, cloud security consultant, noticed the issue and tweeted:
AWSSupportServiceRolePolicy just got s3:GetObject. That role is supposed to only have metadata visibility. AWS Security you need to roll that back.
While Dan Urson, SIRT/Security Ops engineer, immediately acknowledged that AWS Security was working on the issue, the community was less forgiving. The change was reverted approximately eight hours later and AWS made the older (v19) version the default one. Corey Quinn, cloud economist at The Duckbill Group, initially wrote:
Just to be clear here, this means that for a time AWS support was able to read all of your S3 data. There is no mitigation; this role is mandatory. If you had CloudTrail data events enabled, you can audit. If you didn’t, it may be time to declare a security incident.
In the security bulletin, published two days after the events, AWS explains:
While these permissions were temporarily present, they were not and could not be used - only a tightly controlled set of AWS support systems may assume the AWSSupportService role, and these systems do not provide the capability to access S3 objects even if permission is granted to the role. Regardless, we are implementing additional safeguards to prevent the Support policy from inadvertently granting data access permissions.
Source: console.aws.amazon.com
Matthew Wilson, VP and distinguished engineer at AWS, summarizes:
It was never possible for AWS Support to invoke the GetObject API. I'm sorry for any confusion and concern unnecessarily caused by the policy update.
Victor Grenu, independent cloud architect, wrote an article explaining how customers can audit events and review when the support role was last used, and summarized the main takeaways in a tweet:
1. IAM is HARD, even AWS is failing
2. Changes made to IAM should always be peer-reviewed, manually, and using linting
3. Encrypt using your own customer-managed keys
While service-linked roles for AWS Support increase the transparency and auditability of support activities, not every developer still trusts this approach. The security bulletin did not clarify the reason behind the change but Wilson explains:
The practice thus far has been to update the AWSSupportServiceRolePolicy about once a month to incorporate support for new services and actions. There were many eyes on this change, laser focused on protecting customer data. But the analysis was done with additional context about the actual API access that was requested (ultimately, HeadObject which just happens to require read access to the S3 object at present I believe), the controls around all AWS Support tooling that makes use of the role, etc. The reviewers knew customer data was protected.
Piper maintains a repository of Cloud Service Provider security mistakes, a project tracking security issues, including CVEs and bug bounties, of AWS, GCP, and Azure.