API logs, as provided by Amazon API Gateway, serve as a critical tool for tracking and monitoring the execution of your APIs at the stage level. These logs provide a wealth of information including details about the individual making the API call, any API keys used, request parameters, as well as intricate details about both the request and the response.
The process of setting up API logging involves several key steps:
Creation of an S3 Bucket for Access Logs: This first step involves the creation of a designated S3 Bucket, which will serve as the repository for all access logs. This bucket will store all the relevant data and logs for future reference and analysis.
Creation of a Lambda Function for Log Enrichment: The lambda function is used to ensures each log record is enriched with additional information. And it will be invoked by Amazon Kinesis Data Firehose.
Creation of a Kinesis Firehose Delivery Stream: This is where the real-time streaming of API access logs from Amazon API Gateway occurs. It's a crucial component of the logging process as it ensures that all logs are delivered and stored accurately and efficiently.
Enabling of access logging on the API Stage using the Delivery Stream: The final step involves the configuration of the API Gateway to stream access logs to the Kinesis Firehose delivery stream that was created in the previous step.
S3 Bucket for Access Logs
Navigate to the S3 Management Console.
Create a new S3 bucket or use an existing one to store the API Gateway access logs.
Attach the following bucket policy to
YOUR-BUCKET. The policy grants Apiable permission for thes3:GetObjectands3:ListBucketactions.For instructions, see Adding a bucket policy by using the Amazon S3 console.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Example permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::APIABLE_ACCOUNT:root"
},
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::YOUR-BUCKET"
]
}
]
}Please replace:
YOUR-BUCKET in the policy with the correct bucket name.
APIABLE_ACCOUNT in the policy with the Apiable AWS Account number. Contact us.
Lambda Function Configuration
This lambda function is used to add the usagePlan info to the access logs.
During the function initialization, it fetches details regarding the API Gateway usage plan and loads them into memory. Upon each invocation, the function enriches every log with the usage plan name and customer name, and return it to the Kinesis Firehose stream.
Download the source code: Lambda Function Code (Ensure file name remains the same)
Create a Python 3.8 lambda function in the same region where your APIs are located
Upload the source code
In the Configuration Tab, select the general configuration and update the timeout to 5 minutes.
In the Configuration Tab, select the permissions and navigate to the execution role page. In the role detail page add the following Permission to the role
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"apigateway:GET"
],
"Resource": [
"arn:aws:apigateway:*::/usageplans",
"arn:aws:apigateway:*::/usageplans/*/*"
],
"Effect": "Allow"
}
]
}Deploy the Lambda function.
Kinesis Firehose Delivery Stream
The delivery stream is used to real-time stream API access logs from Amazon API Gateway to the s3 bucket.
Navigate to the Kinesis Data Firehose console in the region where your API Gateway is located, then choose Create delivery stream.
Under Delivery stream name, enter a name in the format amazon-apigateway-{your-delivery-stream-name}.
Chose the Direct PUT as Source and Amazon S3 as Destination.
Under the Transform and convert records section**, t**urn on data transformation, and chose the previous lambda function.
Under the Destination settings section, select the bucket created earlier.
Click the Create deliver stream button.
Enabling of access logging on the API Stage using the Delivery Stream:
Follow these instructions to enable access logging on the API stages using the Firehose delivery stream ARN.
For Log format, enter the following format:
{“api_id”: “$context.apiId”,“api_key”: “$context.identity.apiKey”,“key_id”: “$context.identity.apiKeyId”,“ip”: “$context.identity.sourceIp”,“method”: “$context.httpMethod”,“uri”: “$context.path”,“response_size”:“$context.responseLength”,“response_status”: “$context.status”,“resource_id”: “$context.resourceId”,“request_id”: “$context.requestId”,“request_latency”: “$context.responseLatency”,“request_time”:“$context.requestTime”,“stage”: “$context.stage”}