TABLE OF CONTENT
|1. Introduction to AWS Centralized Logging in AWS|
|2. Centralized Logging: Architecture|
|3. Pricing of AWS Centralized Logging|
|5. About CloudThat|
Introduction to AWS Centralized Logging in AWS
Organizations look to analyze their system logs to obtain valuable insights that can help with troubleshooting, security and compliance, and business insights. This solution will provide a secure infrastructure to automate log aggregation from multiple AWS accounts and regions so that developers and administrators can spend more time on actual log analysis.
If we look at the typical AWS account, there are a lot of things that are happening. Services produce operational metrics, security data, and audit logs, and additionally, applications produce web-level, app-level logs. And with more customers moving towards a multi-account strategy there’s a strong desire to centralize logs and have a single view of what’s happening across your AWS account properties that’s where centralized logging comes in. It captures all activity from across AWS workloads and centralizes it for analysis. Centralizing logs comes increasingly useful for customers who want to manage multiple applications and even multiple accounts. Additionally, customers with audit and compliance requirements find it useful to be able to analyze everything all in one place.
Centralized Logging: Architecture
The architecture is fundamentally a hub and spoke. The spokes represent AWS accounts customer may have many of those, each of those have running applications, that have events, that drive logs, and customers want to view those in one place. That’s why centralized logging aggregates all those logs into the hub.
There are three key components let us look at them one by one.
Log Ingestion – the solution creates a CloudWatch log destination in all the regions by default but if one wishes to deploy that log destination in very specific regions, one can do so by changing the parameter inside the CloudFormation stack. This will need to create CloudWatch logs subscription filter in each of the spoke accounts. Once you create that subscription all the log events will come through directly to the primary account.
Log Indexing – In this component, two log indexing is focused. The Kinesis data streams are used as a target for the CloudWatch log destination. All the log events make it to the Kinesis streams that will invoke a Lambda function with that log event. The functionality of this Lambda function is to transform that into an Amazon OpenSearch service document. Once that is transformed, restore that into Kinesis Data Firehose, that Data Firehose then indexes these events or these documents into the OpenSearch service.
Visualization – Data visualization and exploration support are provided by Kibana and Amazon OpenSearch services. Kibana dashboard helps visualize all the index documents in the OpenSearch service. This dashboard allows users to dig deeper into the log events by providing customized time duration for analyzing HTTP 404 code count, HTTP 200 code count, as well as the number of packets that are being accepted, and those which are rejected. Analyzing all the log events at a specific event level is also possible by providing filters in the search section of the dashboard.
Here, there are a few sample data visualization examples of the Kibana dashboard.
Pricing of AWS Centralized Logging
The cost estimation for running Centralized logging solution depends upon factors, which are as follows:
The cluster size of AWS OpenSearch service is available in sizes like Small, Medium, and Large, and the average data transfer rate for Amazon Kinesis Data Firehose, and Amazon Kinesis Data Streams. The following table represents the cost estimation for the Centralized logging solution where a small size cluster of Amazon OpenSearch services is used:
Example Use case
Cost per month
|Amazon OpenSearch Service||Small cluster size||$806.40|
|Amazon Kinesis Data Firehose||Data ingestion (GB/second): (1 record/second x 5 KB/record) or approximately 12.359 GB/month
US East (N. Virginia) Region costs
Data ingestion: $0.029/GB for first 500 TB/month
Data processing to Amazon VPC: $0.01/GB
Amazon VPC delivery: $0.01 per hour, per Availability Zone for VPC delivery
|Amazon Kinesis Data Streams||Sample logs put approximately four to five records/minute with a data throughput of approximately 1 KB/second
1 shard ingests up to 1 MiB/second
PUT payload unit (25 KB): one record is 1 PUT payload unit
Aggregation: approximately five records/minute which equates to approximately 216,000 PUT payload units/month
Source 1: https://docs.aws.amazon.com/solutions/latest/centralized-logging/cost.html
Centralized logging can quickly deploy to start monitoring multi-account environments and position themselves to meet their audit and compliance requirements. And aside from Elasticsearch, this is a serverless solution using native AWS components like CloudWatch, CloudTrail, and VPC Flow Logs, all visualized through the Kibana dashboard. For more research, I encourage the readers to check out centralized logging and other AWS solutions on the official AWS Solutions page.
CloudThat is the official AWS (Amazon Web Services) Advanced Consulting Partner and Training partner and Microsoft gold partner, helping people develop knowledge on the cloud and help their businesses aim for higher goals using best in industry cloud computing practices and expertise. We are on a mission to build a robust cloud computing ecosystem by disseminating knowledge on technological intricacies within the cloud space. Our blogs, webinars, case studies, and white papers enable all the stakeholders in the cloud computing sphere.
Drop a query if you have any questions regarding AWS Centralized logging, AWS services, or consulting opportunities, and I will get back to you quickly. To get started, go through our Consulting page.
- Which are the AWS services providing centralized logging as a feature?
Following are the AWS services which provide centralized logging, and make logging easy and efficient:
CloudTrail logs, AWS Config logs, Elastic Load balancer logs, VPC Flow logs, S3 access logs, etc.
- What log formats are this solution compatible with?
AWS CloudTrail, AWS Lambda, Common Log Format, Space Delimited, JSON, Apache web server logs, and additional (user-specified) formats are supported.