{"id":27307,"date":"2015-10-01T16:14:16","date_gmt":"2015-10-01T10:44:16","guid":{"rendered":"http:\/\/www.tothenew.com\/blog\/?p=27307"},"modified":"2017-05-01T15:14:36","modified_gmt":"2017-05-01T09:44:36","slug":"chef-bootstrapping-an-autoscaled-node-using-cloud-init-cloud-config-and-userdata","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/chef-bootstrapping-an-autoscaled-node-using-cloud-init-cloud-config-and-userdata\/","title":{"rendered":"Chef: Bootstrapping an autoscaled Node using Cloud-Init, Cloud-Config and UserData"},"content":{"rendered":"

This blog refers to the bootstrapping an autoscaled Node using Cloud-Init, Cloud-Config and UserData and get it available for serving requests. Here, in this blog I am bootstrapping a web server. We will be using a yaml file and a recipe to implement the same.<\/p>\n

What is Cloud-Init?<\/h3>\n

Cloud-Init is the package which is already installed on the cloud environment, it is used to initialize the early stages of the cloud servers. It locates user-data and executes the instructions pre written on it. It understands only two userData formats – shell script and cloud-config directives.<\/p>\n

What is Cloud-Config?<\/h3>\n

It is Cloud-Init script which defines the parameters which are required when the server boots up.<\/p>\n

What is UserData?<\/h3>\n

UserData is a type of metadata that can be passed to an instance to perform different actions including executing scripts and Chef-recipes on boot. It is limited to 16KB.<\/p>\n

Chef-Server<\/h3>\n

It is the configuration management tool written in Ruby and Erlang. It is used to manage and configure organisation’s servers.<\/p>\n

Bootstrapping<\/h3>\n

While bootstrapping a node from Chef, user-data on the node will be called by the Cloud-Init program and chef-client will be installed which will register itself on Chef Server<\/a> and will pull out all the configured recipes for that \u2018type\u2019 of node.<\/p>\n

Use Case:<\/h3>\n

We had a requirement where we wanted an AWS autoscaling<\/a> node to automatically bootstrap and get itself ready for serving web requests via Nginx. Autoscaling node supposed to be a raw machine and will get its packages and services from Chef server via User data and Cloud-Init.<\/p>\n

<\/h3>\n

User Data Dissection<\/h3>\n

[js]<\/p>\n

#cloud-config<\/p>\n

# This example assumes the instance is 12.04 (precise)
\n# The default is to install from packages.<\/p>\n

# Key from
\napt_upgrade: true
\napt_sources:
\n– source: "deb $RELEASE-0.10 main"
\nkey: |
\n—–BEGIN PGP PUBLIC KEY BLOCK—–
\nVersion: GnuPG v1.4.9 (GNU\/Linux)<\/p>\n

mQGiBEppC7QRBADfsOkZU6KZK+YmKw4wev5mjKJEkVGlus+NxW8wItX5sGa6kdUu
\nzIhi7tKFo6WiwjKZq0dzctsJJjtIfr4K4vbiD9Ojg1iISQQYEQIACQUCSmkLtAIb
\n……..
\nDAAKCRApQKupg++CauISAJ9CxYPOKhOxalBnVTLeNUkAHGg2gACeIsbobtaD4ZHG
\n0GLl8EkfA8uhluM=
\n=zKAm
\n—–END PGP PUBLIC KEY BLOCK—–<\/p>\n

#The above will verify that the we are downloading is from trusted source<\/p>\n

[\/js]<\/p>\n

 <\/p>\n

The above will upgrade the package sources list and it will install chef client on the machine.
\napt_upgrade<\/strong> :- It will upgrade the package sources.
\napt_sources<\/strong> :- Here we specify the package which we are required to be installed.
\nkey<\/strong> :- Key present here will verify the key that the package is being installed from trusted source.<\/p>\n

[js]<\/p>\n

chef:
\nrun_list:
\n– "recipe[nginx]"<\/p>\n

# Valid values are ‘gems’ and ‘packages’ and ‘omnibus’
\ninstall_type: "packages"<\/p>\n

# Boolean: run ‘install_type’ code even if chef-client
\n# appears already installed.
\nforce_install: false<\/p>\n

# Chef settings
\nserver_url: "https:\/\/ip-X-X-X-X.ec2.internal\/organizations\/organisation-name"<\/p>\n

# Node Name
\n# Defaults to the instance-id if not present
\nnode_name: "#{Socket.gethostname}-test"
\n# Default validation name is organisation-validator
\nvalidation_name: "organisation-validator"
\nvalidation_key: |
\n—–BEGIN RSA PRIVATE KEY—–
\nMIIEowIBAAKCAQEAw75CltGJ1oYf980m1B9soHbDVK\/6l1dhgzd86aW8GjFpqfz8
\nQ9kXZ3wrLOxJ64+vIF1qouMzgtoGCayZ00t62MnCZodgPCzccHY0CZKOKx2V1dDW
\nPAt1veQkXODDLilFiGfTlwBud+O1JLoDa5Nxt6y9INL21IGx3sXi54Z7Df98ZS2z
\n5K5IzLjRsjmh2NhwG\/fB7xRYXaImHrjua7c309rH411Pgo1xcQb9xETU4WaOyl\/C
\n……
\nQEo7WQKBgGKenUL\/CmBmehY6EBfsYyq1Pbwn\/f3n1D0w61ef+zBD0KVnO78dZozq
\nWd2pd4EHxp4+V1NEGCF47YVfiO+D9Lk3V4ic+1kVfYCsefygdbr5i1C4zUVOGcso
\nQcbwgH\/doZL7MfiofwHP8MupY410UQnnV2it72Ml+bJf9Q7ezjsX
\n—–END RSA PRIVATE KEY—–<\/p>\n

#The above key is organisation-validator.pem which is present on the chef server<\/p>\n

[\/js]<\/p>\n

 <\/p>\n

The above code will be executed after chef-client has installed
\nrun_list<\/strong> :- Here we specify the recipes which needs to be executed after Chef has booted.
\ninstall_type<\/strong> :- It specifies which type of installation you want which can be \u2018gems\u2019, \u2018packages\u2019 and \u2018omnibus\u2019.
\nserver_url<\/strong> :- Here we specify the Chef server URL.
\nnode_name<\/strong> :- Here we specify the name of the node which you want to be registered as.
\nvalidation_name<\/strong> :- Here we give the name organisation validator.
\nvalidation_key<\/strong> :- Here we specify the organisation validator key to authenticate the node with Chef server.<\/p>\n

[js]<\/p>\n

# if install_type is ‘omnibus’, change the url to download
\nomnibus_url: "https:\/\/www.opscode.com\/chef\/install.sh"<\/p>\n

# Capture all subprocess output into a logfile
\n# Useful for troubleshooting cloud-init issues
\noutput: {all: ‘| tee -a \/var\/log\/cloud-init-output.log’}<\/p>\n

[\/js]<\/p>\n

 <\/p>\n

omnibus_url<\/strong> :- Here we specify the URL from which we can download packages if install type omnibus is selected.
\noutput<\/strong> :- It specifies where you want to store the logs of the above events.
\nruncmd<\/strong> :- Here we can write shell commands or scripts which will be executed on the machine.<\/p>\n

The above yaml script can be downloaded from here<\/a><\/p>\n

Troubleshooting<\/h3>\n

Logs can be checked at cloud-init-output.log and client.log.<\/p>\n

cloud-init-output.log<\/h4>\n

This file would contain all the logs of the events occurred when the userdata.yaml is executed. This has specified in the userdata.yaml itself.
\noutput: {all: ‘| tee -a \/var\/log\/cloud-init-output.log’}<\/p>\n

client.log<\/h4>\n

\/var\/log\/chef\/client.log
\nThis file contain the logs after Chef has installed and it tries to connect to Chef server. We can check here whether our node has connected to our Chef server.<\/p>\n

What this UserData will do?<\/h3>\n
    \n
  • It will first update the package sources.<\/li>\n
  • Then it will download and install the chef-client on that node from the defined repository.<\/li>\n
  • For initial handshake and registration, it will use organization validator key and will generate a RSA Private key (client.pem) for further communication.<\/li>\n
  • After making connection with the Chef server, it will get its packages as configured on Chef server and execute the recipe which is passed as run_list.<\/li>\n
  • Once chef-client run is completed, Server is ready !!<\/li>\n<\/ul>\n

    Conclusion<\/h3>\n

    We can bootstrap our servers automatically with the pre defined cookbooks without logging into that server instance or into chef-workstation to manually bootstrap it.<\/p>\n","protected":false},"excerpt":{"rendered":"

    This blog refers to the bootstrapping an autoscaled Node using Cloud-Init, Cloud-Config and UserData and get it available for serving requests. Here, in this blog I am bootstrapping a web server. We will be using a yaml file and a recipe to implement the same. What is Cloud-Init? Cloud-Init is the package which is already […]<\/p>\n","protected":false},"author":261,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":6},"categories":[2348],"tags":[],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/27307"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/261"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=27307"}],"version-history":[{"count":0,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/27307\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=27307"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=27307"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=27307"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}