Amazon Web Services (AWS) recently released the AWS CloudTrail Processing Library (CPL), a "Java client library that makes it easy to build an application that reads and processes CloudTrail log files in a fault tolerant and highly scalable manner".
AWS CloudTrail records all API calls made in an AWS account for logging and auditing use cases including security analysis, change tracking, compliance aid and operational troubleshooting, as explained in more detail in our previous coverage. It has been introduced at re:Invent 2013 and expanded over the course of 2014 to support all public AWS regions and most services.
As usual, AWS provides an API for integrating CloudTrail with custom monitoring solutions. However, implementing the logic for processing CloudTrail events required interaction with at least three involved services Amazon S3, Amazon SNS and CloudTrail itself, while considering resiliency and fault tolerance – a cumbersome task.
This has now been addressed by a "new extension to the AWS SDK for Java":
The AWS CloudTrail Processing Library, or CPL, eliminates the need to write code that polls Amazon SQS queues, reads and parses queue messages, downloads CloudTrail log files, and parses and serializes events […]. Developers can read and process CloudTrail log files in as few as 10 lines of code. CPL handles transient and enduring failures […] in a resilient and fault tolerant manner. CPL is built to scale easily and can process an unlimited number of log files in parallel.
A minimum integration just requires the implementation of an EventProcessor that receives the CloudTrail log data:
public class SampleEventsProcessor implements EventsProcessor {
public void process(List<CloudTrailEvent> events) {
int i = 0;
for (CloudTrailEvent event : events) {
System.out.println(String.format("Process event %d : %s", i++,
event.getEventData()));
}
}
}
As illustrated by Jason Fulghum in his introductory post on the Java SDK blog, events can be filtered directly within this loop. More advanced use cases can be implemented by means of a few dedicated interfaces instead:
- EventFilter – provides a callback to determine whether or not to process a log record
- EventProcessor – provides a callback to deliver log records for processing (see above)
- ExceptionHandler – provides a callback that handles exceptions while processing log files
- ProgressReporter – provides an interface for custom handling of progress
- SourceFilter – provides a callback to determine whether or not to process a log source
An EventFilter allows to selectively process events based on the granular information within the CloudTrailEventData, for example the AWS region, the originating service or even IP address, the user identity, and of course an event name such as "DeleteSecurityGroup". The CPL provides sample implementations for all interfaces, including an EventFilter that only surfaces EC2 deletion API calls:
public class SampleEventFilter implements EventFilter{
private static final String EC2_EVENTS = "ec2.amazonaws.com";
@Override
public boolean filterEvent(CloudTrailEvent event) throws CallbackException {
CloudTrailEventData eventData = event.getEventData();
String eventSource = eventData.getEventSource();
String eventName = eventData.getEventName();
return eventSource.equals(EC2_EVENTS) && eventName.startsWith("Delete");
}
}
The AWS CloudTrail Processing Library source code is available on GitHub. The CloudTrail documentation features a section on how to use CPL as well as the CPL API reference. Support is offered via issue tracker and the AWS CloudTrail forum.
