Use CloudWatch LogGroups for EC2 logging

Joris Conijn

April 10, 2024 originally published at xebia.com - 6 minutes read - 1110 words.

You can protect yourself from losing logs on Amazon EC2 by using CloudWatch Logs. Configure the CloudWatch Agent to stream your logs to a LogGroup. This protects you from losing logs. For example, when the instance is replaced by autoscaling. You are also protected against tampering of the logs. An attacker who has gained access to your system can remove the logs. But the logs in the LogGroup will contain the original log lines.

Preparations

We will need the following resources:

 InstanceSecurityGroup:
   Type: AWS::EC2::SecurityGroup
   Properties:
     GroupDescription: Security group for the test instance
     VpcId: "{{resolve:ssm:/landingzone/vpc/vpc-id}}"
     SecurityGroupEgress:
       - Description: Allow outbound connectivity to port 443.
         IpProtocol: tcp
         FromPort: 443
         ToPort: 443
         CidrIp: 0.0.0.0/0

 Role:
   Type: AWS::IAM::Role
   Properties:
     PermissionsBoundary: !Sub arn:aws:iam::${AWS::AccountId}:policy/landingzone-workload-permissions-boundary
     AssumeRolePolicyDocument:
       Version: 2012-10-17
       Statement:
         - Effect: Allow
           Action: sts:AssumeRole
           Principal:
             Service: ec2.amazonaws.com
     ManagedPolicyArns:
       - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
       - arn:aws:iam::aws:policy/AmazonInspector2ManagedCispolicy
       - arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy

 InstanceProfile:
   Type: AWS::IAM::InstanceProfile
   Properties:
     Roles:
       - !Ref Role

 LogGroup:
   Type: AWS::Logs::LogGroup
   Properties:
     LogGroupName: !Sub ${AWS::StackName}-Instance
     RetentionInDays: !!int 365

InstanceSecurityGroup, allows an outbound connection on 443. The CloudWatch Agent needs to stream the logs to the CloudWatch endpoint.
Role, holds the policies that allow the instance to send the logs to the LogGroup.
InstanceProfile, the profile used to attach the role to the EC2 instances.
LogGroup, the LogGroup used to store the logs in.

Setting up the EC2 Instance

In my previous blog I wrote about how you can create an EC2 instance using Infrastructure as Code. We will continue on that example. Instead of a single instance we will create an autoscaling group.

First we will need a LaunchTemplate. This LaunchTemplate will contain some metadata called AWS::CloudFormation::Init. The metadata has so called configSets and blocks of config. Here is an example that we will use:

 LaunchTemplate:
   Type: AWS::EC2::LaunchTemplate
   Metadata:
     AWS::CloudFormation::Init:
       configSets:
         default:
           - CloudFormationInit
           - CloudWatchLogs
       CloudFormationInit:
         files:
           /etc/cfn/cfn-hup.conf:
             owner: root
             group: root
             mode: 000400
             content: !Sub |-
               [main]
               stack=${AWS::StackId}
               region=${AWS::Region}
           /etc/cfn/hooks.d/cfn-auto-reloader.conf:
             owner: root
             group: root
             mode: 000400
             content: !Sub |-
               [cfn-auto-reloader-hook]
               triggers=post.update
               path=Resources.LaunchTemplate.Metadata.AWS::CloudFormation::Init
               action=/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --region ${AWS::Region} --resource LaunchTemplate
               runas=root
         services:
           sysvinit:
             cfn-hup:
               enabled: true
               ensureRunning: true
               files:
                 - /etc/cfn/cfn-hup.conf
                 - /etc/cfn/hooks.d/cfn-auto-reloader.conf
       CloudWatchLogs:
         packages:
           yum:
             amazon-cloudwatch-agent: []
         files:
           /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_amazon-cloudwatch-agent.json:
             owner: root
             group: root
             mode: 000400
             content: !Sub |-
               {
                 "agent": {
                   "region": "${AWS::Region}",
                   "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log",
                   "debug": false
                 },
                 "logs": {
                   "logs_collected": {
                     "files": {
                       "collect_list": [
                         {
                           "file_path": "/var/log/user-data.log",
                           "log_group_name": "${LogGroup}",
                           "log_stream_name": "{instance_id}/user-data.log"
                         },
                         {
                           "file_path": "/var/log/cfn-hup.log",
                           "log_group_name": "${LogGroup}",
                           "log_stream_name": "{instance_id}/cfn-hup.log"
                         },
                         {
                           "file_path": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log",
                           "log_group_name": "${LogGroup}",
                           "log_stream_name": "{instance_id}/amazon-cloudwatch-agent.log"
                         }
                       ]
                     }
                   },
                   "log_stream_name": "default_log_stream"
                 },
                 "metrics": {
                   "append_dimensions": {
                     "AutoScalingGroupName": "${!aws:AutoScalingGroupName}"
                   },
                   "metrics_collected": {
                     "disk": {
                       "measurement": [
                         "used_percent"
                       ],
                       "metrics_collection_interval": 60,
                       "resources": [
                         "/"
                       ]
                     }
                   }
                 }
               }
         services:
           sysvinit:
             amazon-cloudwatch-agent:
               enabled: true
               ensureRunning: true
               files:
                 - /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_amazon-cloudwatch-agent.json
   Properties:
     LaunchTemplateData:
       BlockDeviceMappings:
         - DeviceName: /dev/xvda
           Ebs:
             DeleteOnTermination: !!bool true
             Encrypted: !!bool true
             KmsKeyId: !GetAtt EncryptionKey.Arn
             VolumeSize: !!int 32
             VolumeType: gp2
       DisableApiTermination: !!bool true
       IamInstanceProfile:
         Arn: !GetAtt InstanceProfile.Arn
       ImageId: "{{resolve:ssm:/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2}}"
       InstanceType: t3.micro
       SecurityGroupIds:
         - !Ref InstanceSecurityGroup
       MetadataOptions:
         HttpTokens: required
         InstanceMetadataTags: enabled
       UserData:
         Fn::Base64: !Sub |-
           #!/bin/bash -x
           /opt/aws/bin/cfn-init --stack ${AWS::StackName} --region ${AWS::Region} --resource LaunchTemplate
           /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --region ${AWS::Region} --resource AutoScalingGroup

CloudFormationInit

The CloudFormationInit holds a configuration. This configuration is used by the cfn-init agent. It comes pre-installed with the AmazonLinux2 and AmazonLinux2023 AMIs. Under the files section you can see that we are defining 2 files:

/etc/cfn/cfn-hup.conf
/etc/cfn/hooks.d/cfn-auto-reloader.conf

The syntax is quite explanatory. You set an owner, group, mode and the content of the file. The content of the tells the EC2 instance where it can find the metadata. During the initial boot this is given in the user-data. But you might change the metadata overtime. This makes sure that the running instances checks if there is an update available.

To recap this will make sure that the running EC2 instances stay in sync with the given metadata.

CloudWatchLogs

The Amazon Linux AMIs do not come with the CloudWatch Agent you need to install the agent first. You can do this by providing the amazon-cloudwatch-agent package, under packages.

Now it’s time to configure the agent to stream the log files to CloudWatch LogGroups. We have defined 3 files that will be watched by the agent:

/var/log/user-data.log
/var/log/cfn-hup.log
/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

You can extend this list to your own needs. In this example each EC2 instance will have 3 streams. 1 per file and the instance id is used as a prefix, example: i-0000000000/user-data.log.

You can also do measurements on the instance itself and expose this as a CloudWatch Metric. In this example we are capturing a percentage of the used space of the root volume.

We have now configured the agent. But we also need to make sure it’s running. When changes are made to the configuration file, we also need to restart it. We do this by pointing to the config file. Now cfn-init knows when it changes the configuration that it also needs to restart the agent.

User Data

The user data will make sure that the configuration is applied. Since we configure the cfn-init to check the configuration it will stay in sync from here on. The second thing it will do is it will send a signal to the AutoScalingGroup resource. If the cfn-init is executed successfully it will signal a success and if not it will signal a failure.

This means that if the configuration fails the autoscaling group will receive a failure. As a result it will rollback the stack.

AutoScalingGroup

The autoscaling group looks as followed:

  AutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    CreationPolicy:
      ResourceSignal:
        Timeout: PT5M
    UpdatePolicy:
      AutoScalingRollingUpdate:
        MinInstancesInService: 1
        MaxBatchSize: 1
        PauseTime: PT5M
        WaitOnResourceSignals: !!bool true
    Properties:
      LaunchTemplate:
        LaunchTemplateId: !GetAtt LaunchTemplate.LaunchTemplateId
        Version: !GetAtt LaunchTemplate.LatestVersionNumber
      MinSize: !!int 1
      MaxSize: !!int 2
      DesiredCapacity: !!int 1
      VPCZoneIdentifier:
        - "{{resolve:ssm:/landingzone/vpc/private-subnet-1-id}}"
        - "{{resolve:ssm:/landingzone/vpc/private-subnet-2-id}}"

We will have 2 policies defined, one for creation and one for updates. If the signal configured in the user-data fails the resource will fail and trigger a rollback. If the instance is configured successfully it will continue. In the auto scaling group you can define the amount of instances required. In this example the maximum is 2 and the minimum 1. This makes sure that at least one EC2 instance is running. When a replacement is needed a new instance will be added first and if it is deployed successfully the old one is removed. This is why the max is set to 2 so that we can replace the old one gracefully.

Conclusion

Configuring a LogGroup to stream your log files is not hard. Using CloudFormation Init you can install and configure applications. In our example we configured the CloudWatch agent that will stream the logs.

When you have to deal with autoscaling events and/or malicious attackers. You can prevent the loss of logs by centralising your application log files.

Photo by Pixabay