How to monitor ephemeral storage usage in Fargate

Since there is no native CloudWatch metrics to monitor ephemeral storage on ECS, we think the only way is to use sidecar containers [1].

There is one propose for this feature, however it seems that AWS has not processed it yet. Refer to this link for more information:

https://github.com/aws/containers-roadmap/issues/1613

You can use the official docker image [amazon/cloudwatch-agent][2] to deploy a sidecar container:

Step 1

Create CloudWatch agent configuration file. You can refer to the sample settings as below:

{

  "metrics": {

    "metrics_collected": {

      "disk": {

        "measurement": [

          {

            "name": "free",

            "unit": "Gigabytes"

          },

          {

            "name": "total",

            "unit": "Gigabytes"

          },

          {

            "name": "used",

            "unit": "Gigabytes"

          },

          "used_percent",

          "inodes_free",

          "inodes_used",

          "inodes_total"

        ],

        "resources": [

          "*"

        ],

        "append_dimensions": {

          "ECScw": "ECSdisk",

          "cluster": "Default",

          "service": "Fargate"

        }

      }

    }

  }

}

You can change the metrics they want to collect by CloudWatch agent in the settings file. For details on metrics collected by CloudWatch agent, please refer to document [3]

Step 2

Upload the CloudWatch agent configuration setup file to AWS System Manager. You can refer to the “put-parameter” command for more details in the document [4]. Sample AWS CLI command to upload:

aws ssm put-parameter --name "cwagentconfigfile" --type "String" --value file://amazon-cloudwatch-agent.json --region <region_name>

Step 3

Create task definition for application container and sidecar container Set the “cwagentconfigfile” parameter for the container to transfer the CloudWatch agent configuration file created above to the sidecar container.

Sample task definition file:

{

    "containerDefinitions": [

        {

            "name": "demo",

            "image": "<your_image_uri",

            "cpu": 0,

            "portMappings": [],

            "essential": true,

            "environment": [],

            "mountPoints": [],

            "volumesFrom": [],

            "logConfiguration": {

                "logDriver": "awslogs",

                "options": {

                    "awslogs-group": "/ecs/yen-ephermeral-task",

                    "awslogs-region": "ap-northeast-1",

                    "awslogs-stream-prefix": "ecs"

                }

            }

        }

    ],

    "family": "yen-ephermeral-task",

    "taskRoleArn": "arn:aws:iam::<account_id:role/ecsTaskExecutionRole",

    "executionRoleArn": "arn:aws:iam::<accound_id>:role/ecsTaskExecutionRole",

    "networkMode": "awsvpc",

    "volumes": [],

    "placementConstraints": [],

    "requiresCompatibilities": [

        "FARGATE"

    ],

    "cpu": "256",

    "memory": "512",

    "ephemeralStorage": {"sizeInGiB": 100 }

}

As a result, CloudWatch metrics collected information about ECS Fargate’s disk usage.

Note:

• Ensure that the ECS task role has the permissions for whatever your application needs as well as the CloudWatchAgentServerPolicy and the ECS task execution role has the policies AmazonSSMReadOnlyAccess, AmazonECSTaskExecutionRolePolicy, and CloudWatchAgentServerPolicy.
• If you are not assigning the Fargate tasks to a public subnet, Ensure your service will have access to the CloudWatch and SSM services (you can set up NAT gateway or use VPC endpoints)
• Confirm from the sidecar container logs that everything is working as expected and the CloudWatch agent is correctly processing the configurations as defined in the above steps. Note: If task runs and stops, ensure check to ensure CloudWatch permissions are correct or manually create a log group entry in CloudWatch where data will write to.
• I have been suggested to use Fluentbit and cAdvisor besides aws cloudwatch agent as sidecar container. I have not tested this because lack of time. You should also tried them and I am very happy if you provide me some feedback.

References:

[1] https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-metrics.html#available_cloudwatch_metrics

[2][amazon/cloudwatch-agent]

https://hub.docker.com/r/amazon/cloudwatch-agent

[3][CloudWatch ]

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html

[4][put-parameters]

https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ssm/put-parameter.html