Amazon recently announced CloudWatch Logs Live Tail, an option to analyze logs in near real-time. Currently only available in the AWS console, the interactive log analytics feature helps developers detect and debug application anomalies.
Live Tail is an analytics option of the Management Console that provides customers with a real-time view of the incoming CloudWatch logs data, allowing developers to quickly validate the correct start of an application or the impact of configuration changes. According to the cloud provider, it can aid operational troubleshooting and root cause analysis. Shree Chinnasamy, senior specialist solutions architect at AWS, explains:
You can specify a filter pattern to display only log events that contain certain words or other strings (i.e., ERROR, exception, failure) and cut out the noise. The filters field is case-sensitive (...) Additionally, highlighting is a powerful feature that you can use to specify up to 5 terms to mark each event with a color-coded indicator. (..) Let’s say that you want to quickly identify the filtered events containing 404 error type. In this case, you simply add 404 as a highlighting term and each event will be marked with the assigned color indicator.
Source: https://aws.amazon.com/blogs/mt/announcing-live-tail-feature-for-amazon-cloudwatch-logs/
Matthieu Napoli, creator of the serverless framework for PHP Bref, tweets:
Given the delays are close, I imagine that the new "StartLiveTail" API is close to what we get by polling with the current API. Instead of polling, the WebSocket returns logs as they are available. I don't expect "Live Tail" to be faster, but it might be simpler to use.
CloudWatch announced as well support for account-level data protection, allowing customers to detect and mask sensitive data consistently across all log groups. The feature leverages pattern matching and ML capabilities to detect sensitive data in transit.
A maximum of 10 log groups can be scanned in one Live Tail session. To start a session, customers should assume the AWS Identity and IAM Admin Role or have an IAM policy allowing logs:* or add logs:StartLiveTail and logs:StopLiveTail actions.
Corey Quinn comments on the latest feature and the pricing in his newsletter:
Announcing "tail -f as a service", wherein it costs you a penny per minute to use after a fairly generous 1800 minutes a month perpetual free tier. I'm unreasonably happy about this; it's how I use CloudWatch Logs myself at least 95% of the time.
Live Tail is available in all AWS regions.