Oliver Berger | Tue, 17 September 2019
In previous posts we’ve discussed the differences between CloudTrail and AWS Config and that has served to raise the equally pertinent question of how you compare CloudTrail to CloudWatch.
AWS CloudWatch is a suite of monitoring tools built into one AWS service. In this post, we’ll explore each major component of CloudWatch and explain why one would consume the Metrics, Alarms, Logs and Events available within this useful service. But before we explore the many faces of CloudWatch, let’s find out a bit more about CloudTrail.
AWS CloudTrail is a log of every single API call that has taken place inside your Amazon environment. Each call is considered an event and is written in batches to an S3 bucket. These events show us details of the request, the response, the identity of the user making the request and whether the API calls came from the AWS Console, CLI, some third-party application or other AWS Service.
The process is fairly simple to break down:
CloudTrail typically logs a whole bunch of metadata about the API call that takes place and that metadata tends to vary quite significantly from service to service. At a foundational level, CloudTrail will capture the following:
type – what type of
userIdentity triggered the event (ie was it an IAM User), Root, another AWS Service etc
userIdentity – includes the ARN of the user or role that generated the request
eventSource – the service that the request was made to
eventTime – the time when the event took place
awsRegion – which region the event took place in
eventName – what the event actually was
sourceIPAddress – which IP Address the request was made from
userAgent – how the call was made (ie SDK, CLI, Console)
errorCode – if any errors occurred
errorMessage – more details on any errors
requestParameters – any parameters from the request; these vary per service and per API endpoint
responseElements – the response for any action that makes a change; these also vary on a service by service basis
additionalEventData – any further data about the event
requestID – a value that identifies the specific request; this is generated by the service itself
eventID – different from the
requestID; this is generated by CloudTrail as a unique ID for each single event
eventType – tells you if it was an
AwsApiCall that generated the event, an
AwsServiceEvent generated by a
service or if it was generated by an IAM user
apiVersion – if there was an
AwsApiCall that generated the event, this will tell you the API apiVersion
managementEvent – was this a management event?
readOnly – was the operation read only?
resources – any resources that were connected to the event; the details can include ARNs, Account IDs connected
to the resource owner and other identifiers
recipientAccountId – the account that received the event
serviceEventDetails – can contain data about the event trigger and its result
sharedEventID – if a CloudTrail event is delivered to multiple accounts this will show as a unique way of
associating the data between those accounts
vpcEndpointId – the VPC endpoint if requests were made from a VPC to a different AWS service
As a general rule, CloudTrail will deliver any event within about 15 minutes of the API call. CloudTrail will typically write logs to the allocated S3 bucket in batches every five minutes.
Actually yes, you can get notified of specific CloudTrail events immediately by consuming them via the CloudWatch Event Bus.
While CloudTrail only writes to your S3 bucket once every five minutes, it emits events on the CloudWatch Event bus as these API calls are observed. The result is a near real-time stream of information about changes in your AWS Accounts.
GorillaStack actually uses the CloudWatch Event bus to monitor CloudTrail in our popular Real Time Events tool and can alert you to any CloudTrail event in real time. You can consume these events as alerts via Chat or Email or use them to trigger Lambda functions.
This brings us to CloudWatch Events.
CloudWatch Metrics are time series performance data about your AWS services and resources. These metrics mostly relate to application performance and resource utilisation.
Because each CloudWatch Metric is time-stamped you can continually monitor and review data points in relation to a particular application or overall set of applications and resources. This in turn allows you to build up a picture about your overall operational health and, if you’d like, to set alarms when the performance falls out of an expected range or band.
A CloudWatch Alarm can be set to trigger based on the change in state or threshold of one or more CloudWatch Metrics. You can configure the CloudWatch Alarm to evaluate a combination of metrics by applying math or percentile-based expressions. You can use up to 10 CloudWatch Metrics to build math expressions as long as those CloudWatch Metrics are covering the same period.
You can set up to 5000 CloudWatch Alarms per region per account so depending on your needs that should be plenty to get some consequential outcomes.
CloudWatch alarms update in near enough real time which is what can often make them a preferable alternative to CloudTrail if your use cases demand it.
AWS CloudWatch Logs is a place to store, access and monitor logs that come from AWS Services, customer application code and other sources. CloudWatch logs allows customers to centralize their logs, retain them and then analyse/access them off one scalable platform.
The best way to explain CloudWatch Logs is through example. Here are some resources and how the type of data that they write to CloudWatch:
|Service||How CloudWatch Logs is Utilized|
|AWS EC2||Customers use the unified logging agent to automatically write logs from their EC2 instances to CloudWatch Logs.|
|AWS Glue||The output from the execution of AWS Glue crawlers is written to AWS CloudWatch Logs by default. This is a great example of how CloudWatch Logs can give an AWS Customer visibility into the operation of an AWS managed service.|
|AWS Lambda||All standard output and error from the execution of Lambda functions is written to CloudWatch Log Streams for a CloudWatch Log Group named after the given function. By default, Lambda will write log entries at the start and end of the Lambda execution with information about the request id, time, maximum memory used vs memory size, duration of the execution and the billing duration.|
|AWS Route 53||Customers can log DNS queries received by Route 53. The domain or subdomain that was requested, date and time of the request, the DNS record type (such as A or AAAA), Route 53 edge location that responded to the DNS query and the DNS response code (NoError, ServFail).|
One way of leveraging all this information is to query your CloudTrail logs with Athena.
|Detail||CloudTrail||CloudWatch Events||CloudWatch Metrics||CloudWatch Alarms||CloudWatch Logs|
|Description||Audit log of AWS API calls||Near real-time feed of operational events within the AWS Environment (including CloudTrail Events)||Time-series performance data about AWS resources and user applications||Alerts fired off state changes on percentile or math based expressions of Metric data||A centralized location to store, access and monitor logs from AWS resources, managed services and application code|
|Currency of data||Updates around every 15 minutes||Updates in near real-time||Generated periodically, frequency depends on service and some configuration options||Updates off changes in Metrics||Updates in near real-time|
|Free Tier||View, filter and download 90 days of management events. Can create one trail and get management events for free, but pay for data events. S3 costs apply.||All non-custom events||Basic (5-minute frequency) monitoring metrics, 10 Detailed (1-minute frequency) monitoring metrics, 1 million API requests (not including GetMetricData and GetMetricWidgetImage)||10 Metric Alarms (not including high resolution alarms)||5GB Data (ingestion, archive storage, and data scanned by Logs Insights queries)|
|Costs (us-east-1 as of 19 September 2019)||Management Events: one copy free per region. For additional copies, $2.00 per 100,000. Data Events: $0.10 per 100,000||Custom Events: $1.00/million events, Cross-Account Events: $1.00/million events||Depending on number of Metrics: $0.30-$0.02/metric/month. Detailed monitoring $2.10-$0.14/instance/month.||$0.10 per alarm metric for Standard Resolution, $0.30 per alarm metric for High Resolution||Data Ingestion: $0.50/GB, Data Storage: $0.30/GB, Log Insights: $0.005/GB Scanned|
|Uses||Audit AWS Changes||Track operational changes to AWS in real-time||Monitoring performance of resources and applications||Alerting off changes to metrics||Centralized logging|
If you’d like to track your most important CloudTrail Events and invoke real-time alerts and remediations for them, check out GorillaStack’s Real Time Events product.