CFN Init and EC2 User Data

  CloudFormation

Course Main Menu

Section 4 Main Menu

EC2 User Data Overview

https://www.udemy.com/aws-cloudformation-master-class/learn/v4/t/lecture/8139272?start=0

What is CFN-Init and EC2-user data

  • Many of the CF templates will be about provisioning computing resources in your AWS Cloud
  • These resources can be either:
    • EC2 Instances
    • AutoScaling Groups
  • Usually, you want the instances to be self configured so that they can perform the job they are supposed to perform.
  • You can fully automate your EC2 fleet state with CloudFormation Init.

EC2 User data

  • This is the area of the AWS Console under EC2 Provisioning where you upload/enter your boot scripts

Example:

#!/bin/bash
yum update -y
yum install -y httpd24 php56 mysql55-server php56-mysqlnd
service httpd start
chkconfig httpd on
groupadd www
usermod -a -G www ec2-user
chown -R root:www /var/www
chmod 2775 /var/www
find /var/www -type d -exec chmod 2775 {} +
find /var/www -type f -exec chmod 0664 {} +
echo "<?php phpinfo(); ?>" > /var/www/html/phpinfo.php

EC2 User Data & Fn::Base64 in CloudFormation

https://www.udemy.com/aws-cloudformation-master-class/learn/v4/t/lecture/8162182?start=0

Same Script as above, using CloudFormation and Fn::Base64

General Outline

Resources:
   ResourceID:
      Properties:
         UserData:
            Fn::Base64: |
               #! /bin/bash
               ...

Example:

Resources:
  WebServer:
    Type: AWS::EC2::Instance
    Properties:
      UserData:
        Fn::Base64: |
           #!/bin/bash
           yum update -y
           yum install -y httpd24 php56 mysql55-server php56-mysqlnd
           service httpd start
           chkconfig httpd on
           groupadd www
           usermod -a -G www ec2-user
           chown -R root:www /var/www
           chmod 2775 /var/www
           find /var/www -type d -exec chmod 2775 {} +
           find /var/www -type f -exec chmod 0664 {} +
           echo "" > /var/www/html/phpinfo.php
Notice the red | symbol in the YAML code.  This symbol in YAML means “post everything as is, including all line breaks.”

CloudFormation Init Overview

https://www.udemy.com/aws-cloudformation-master-class/learn/v4/t/lecture/8139294?start=0

Syntax

cfn-init --stack|-s stack.name.or.id \
         --resource|-r logical.resource.id \
         --region region
         --access-key access.key \
         --secret-key secret.key \
         --role rolename\
         --credential-file|-f credential.file \
         --configsets|-c config.sets \
         --url|-u service.url \
         --http-proxy HTTP.proxy \
         --https-proxy HTTPS.proxy \
         --verbose|-v

 

The Problems with EC2 User Data

  • What if you have a very large instance configuration
    • UserData only allows a certain number of characters
  • What if you want to evolve the state of the EC2 instance without terminating it and creating a new one?
  • How do we make our EC2 user data more readable?
  • How do we know or signal that our CE2 user-data script actually completed successfully?

CloudFormation Helper Scripts

  • There are 4 Python scrips that come directly on Amazon Linux AMI, or can be installed using yum on non-Amazon Linux
    • cfn-init
      • Used to retrieve and interpret the resource metadata, installing packages, creating files and starting services
    • cfn-signal
      • A simple wrapper to signal an AWS CF CreationPolicy or WaitCondition, enabling you to synchronize other resources in the stack with the application being ready.
    • chn-get-metadata
      • A wrapper scrip making it easy to retrieve either all metadata defined for a resource or path to a specific key or subtree of the resource metadata.
    • cfn-hup
      • A daemon to check for updates to metadata and execute custom hooks when the changes are detected.
      • #This seems very similar to a cron job.
  • Usual flow:
    • cfn-init
    • cfn-signal
    • Optionally cfn-hup

AWS::CloudFormation::Init pt.1

  • A config contains the following and is executed in that order:
    • Packages: Install a list of packages on the Linux OS (mysql, wordpress, etc.)
    • Groups: defines user groups
    • Users: define users and which group they belong to
    • Sources: download an archive file and place it on the ec2 instance (tar, zip, bz2)
    • Files: create files on the ec2 instance using inline or can be pulled from a URL
    • Commands: run a series of commands
    • Services: launch a list of sysvinit

General Outline

Resources:
   ResourceID:
      Metadata:
         AWS::CloudFormation::Init:
            config:               #! /bin/bash
               packages:
                  ...

Example:

Resources:
  MyInstance:
    Type: "AWS::EC2::Instance"
    Metadata:
      AWS::CloudFormation::Init:
        config:
          packages:
            :
          groups:
            :
          users:
            :
          sources:
            :
          files:
            :
          commands:
            :
          services:
            :
    Properties:
      :

AWS::CloudFormation::Init pt.2

  • You can have multiple configs in your CloudFormation::Init
  • You create configsets with multiple configs
  • You invoke config sets from your ec2-user data

Packages

https://www.udemy.com/aws-cloudformation-master-class/learn/v4/t/lecture/8139306?start=0

  • You can install packages from the following repositories:
    • apt
    • msi
    • python
    • rpm
    • rubygems
    • yum
  • Packages are processed in the following order:
    • rpm
    • yum/apt
    • rubygems
    • python
  • You can specify a version, or no version (empty array) to get the latest version
rpm:
  epel: "http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm"
yum:
  httpd: []
  php: []
  wordpress: []
rubygems:
  chef:
    - "0.10.2"

Example from script:

  WebServerHost:
    Type: AWS::EC2::Instance
    Metadata:
      Comment: Install a simple PHP application
      AWS::CloudFormation::Init:
        config:
          packages:
            yum:
              httpd: []
              php: []
          groups:
            ...

Groups and Users

https://www.udemy.com/aws-cloudformation-master-class/learn/v4/t/lecture/8139308?start=0

  • If you want to have multiple users and groups (with an optional gid) in your ec2 instance, you and do the following.
  • Let’s add groups and users to our CloudFormation Init Metadata
groups:
  groupOne: {}
  groupTwo:
    gid: "45"
users:
  myUser:
    groups:
      - "groupOne"
      - "groupTwo"
    uid: "50"
    homeDir: "/tmp"

Example from script:

  WebServerHost:
    Type: AWS::EC2::Instance
    Metadata:
      Comment: Install a simple PHP application
      AWS::CloudFormation::Init:
        config:
          packages:
            ...
          groups:
            apache: {}
          users:
            "apache":
              groups:
                - "apache"
          sources:
            ...

Sources

https://www.udemy.com/aws-cloudformation-master-class/learn/v4/t/lecture/8139310?start=0

In short, these are used to download and extract archived (.zip, .tar, .tgz, etc.)

  • We can download whole compressed archives from the web and unpack them on the instance directly
  • It is very handy if you:
    • Have a set of standardized scripts for your instances that you store in S3 for example
    • Want to download a whole github project directly on your instance.
  • Each source is set in a key: value comfiguration
    • The key is the location of the unpacked source (Where you download to)
    • The value is the url of the file
sources:
  /etc/puppet: "https://github.com/user1/cfn-demo/tarball/master"
  /etc/myapp: "https://s3.amazonaws.com/mybucket/myapp.tar.gz"

Example from script:

  WebServerHost:
    Type: AWS::EC2::Instance
    Metadata:
      Comment: Install a simple PHP application
      AWS::CloudFormation::Init:
        config:
          ...
          sources:
            "/home/ec2-user/aws-cli": "https://github.com/aws/aws-cli/tarball/master"
          ...

Files

https://www.udemy.com/aws-cloudformation-master-class/learn/v4/t/lecture/8139312?start=0

This is pretty cool… pay close attention!

  • Files are very powerful as you have full control over any content you want.
  • They can come from a specific URL, or be written inline
files:
  /tmp/setup.mysql:
    content: !Sub |
      CREATE DATABASE ${DBName};
      CREATE USER '${dbuSERNAME}'@'localhost' IDENTIFIED BY '${DBPassword}';
      GRANT ALL ON ${DBName}.* TO '${DBUswername}'.@'localhost';
      FLUSH PRIVILEGES;
    mode" "000644"
    owner: "root"
    group: "root"

Example from script:

  WebServerHost:
    Type: AWS::EC2::Instance
    Metadata:
      Comment: Install a simple PHP application
      AWS::CloudFormation::Init:
        config:
          ...
          files:
            "/tmp/cwlogs/apacheaccess.conf":
              content: !Sub |
                [general]
                state_file= /var/awslogs/agent-state
                [/var/log/httpd/access_log]
                file = /var/log/httpd/access_log
                log_group_name = ${AWS::StackName}
                log_stream_name = {instance_id}/apache.log
                datetime_format = %d/%b/%Y:%H:%M:%S
              mode: '000400'
              owner: apache
              group: apache
            "/var/www/html/index.php":
              content: !Sub |
                
              mode: '000644'
              owner: apache
              group: apache
            "/etc/cfn/cfn-hup.conf":
              content: !Sub |
                [main]
                stack=${AWS::StackId}
                region=${AWS::Region}
              mode: "000400"
              owner: "root"
              group: "root"
            "/etc/cfn/hooks.d/cfn-auto-reloader.conf":
              content: !Sub |
                [cfn-auto-reloader-hook]
                triggers=post.update
                path=Resources.WebServerHost.Metadata.AWS::CloudFormation::Init
                action=/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource WebServerHost --region ${AWS::Region}
              mode: "000400"
              owner: "root"
              group: "root"
          commands:
            ...

Fn::Sub (substitute function)

https://www.udemy.com/aws-cloudformation-master-class/learn/v4/t/lecture/8139314?start=0

  • Fn::Sub or !Sub as a shorthand, is used to substitute variables from a text allowing you to fully customize your templates.
  • For example, you can combine Fn::Sub with References or AWS Pseudo varables
  • String must contain ${VariableName} and will substitute them.

Syntax Example 1 (less common):

!Sub
  - String
  - { Var1Name: Var1Value, Var2Name: Var2Value }

Syntax Example 2 (more common):

!Sub String

Commands

https://www.udemy.com/aws-cloudformation-master-class/learn/v4/t/lecture/8139318?start=0

  • You can run commands one at a time in alphabetical order
  • You can set a directory from which that command is run as well as environmental variables
  • You can provide a test to control whether the command is executied or not
    • Example: test:if a file does not exist, command: the download command)

Example: Call the echo command only if the file does not exist

commands:
  test:
    command: "echo \"$MAGIC\" > test.txt
    env:
      MAGIC: "I come from the environment!"
    cwd: "~"
    test: "test ! -e ~/test.txt"
    ignoreErrors: "false"
  • It is recommended to:
    • use command as a last resort since most of this can be performed in the above sections
    • Use the Files section to write your scripts, then use command to execute those scripts.

Services

https://www.udemy.com/aws-cloudformation-master-class/learn/v4/t/lecture/8139320?start=0

  • Launch several services at the instance launch
  • Ensure services are restarted when files change, or packages are updated by cfn-init.

Example from script:

  WebServerHost:
    Type: AWS::EC2::Instance
    Metadata:
      Comment: Install a simple PHP application
      AWS::CloudFormation::Init:
        config:
          ...
          services:
            sysvinit:
              httpd:
                enabled: 'true'
                ensureRunning: 'true'
              sendmail:
                enabled: 'false'
                ensureRunning: 'false'

CFN Init and Signal

https://www.udemy.com/aws-cloudformation-master-class/learn/v4/t/lecture/8162188?start=0

Calling cfn-signal

  • Use cfn-init first to launch the configuration
  • Next use cfn-signal when the configuration has completed to let CF know that the resoiurce creation has been successful
      • This has to be used in conjunction with a CreationPolicy.
    CreationPolicy:
  ResourceSignal:
    Timeout: PT5M
    • This example means we are waiting a maximum of 5 minutes for the instance to come online and be self configured. If we don’t hear from cfn-signal by then, the CF will fail and roll back.

Properties:
…
UserData:
“Fn::Base64”:
!Sub |
#!/bin/bash -xe
# Get the latest CloudFormation package
yum update -y aws-cfn-bootstrap
# Start cfn-init
/opt/aws/bin/cfn-init -s ${AWS::StackId} -r WebServerHost –region ${AWS::Region} || error_exit ‘Failed to run cfn-init’
# Start up the cfn-hup daemon to listen for changes to the EC2 instance metadata
/opt/aws/bin/cfn-hup || error_exit ‘Failed to start cfn-hup’
# All done so signal success
/opt/aws/bin/cfn-signal -e $? –stack ${AWS::StackId} –resource WebServerHost –region ${AWS::Region}

* -s == --stack (StackId)
* -r == --resource

Cfn-hup (cfn Helper Update?)

 

    • Cfn-hup can be used to tell your EC2 instnaces to look for Metadata changes every 15 minutes and apply the metadata configuration again.

 

    • Very powerful, but strongly recommended to tro try it to fully understand how it works.

 

    • Relies on a cfn-hup configuration, reference:
        • /etc/cfn/cfn-hup.conf

       

        • /etc/cfn/hooks.d/cfn-auto-reloader.conf

       

       

 

 

Example from script

 

          files:
            ...
            "/etc/cfn/cfn-hup.conf":
              content: !Sub |
                [main]
                stack=${AWS::StackId}
                region=${AWS::Region}
              mode: "000400"
              owner: "root"
              group: "root"
            "/etc/cfn/hooks.d/cfn-auto-reloader.conf":
              content: !Sub |
                [cfn-auto-reloader-hook]
                triggers=post.update
                path=Resources.WebServerHost.Metadata.AWS::CloudFormation::Init
                action=/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource WebServerHost --region ${AWS::Region}
              mode: "000400"
              owner: "root"
              group: "root"

CFN Hands On

https://www.udemy.com/aws-cloudformation-master-class/learn/v4/t/lecture/8139328?start=0

Upload the script and run it.

  • View the updates on the CF Events tab
  • View the outputs on the CD Outputs tab

Summary

https://www.udemy.com/aws-cloudformation-master-class/learn/v4/t/lecture/8139330?start=0

  • Use the template as a basis for future CloudFormation scripts.
  • For practice, create a CF script with Init for an AutoScaling group.
  • Log locations
    • EC2-user data: /var/log/cloud-init-output.log
    • cfn-init: /var/log/cfn-init.log
  • Use the logs to see which commands did not complete as expected and why.

 

LEAVE A COMMENT