{"id":62296,"date":"2024-06-29T08:07:11","date_gmt":"2024-06-29T02:37:11","guid":{"rendered":"https:\/\/www.tothenew.com\/blog\/?p=62296"},"modified":"2024-07-29T09:47:44","modified_gmt":"2024-07-29T04:17:44","slug":"understanding-buildkite-pipeline-snyk-integration-part-1","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/understanding-buildkite-pipeline-snyk-integration-part-1\/","title":{"rendered":"Accelerate Your Development with Buildkite Pipelines"},"content":{"rendered":"
\n

Introduction<\/h2>\n<\/div>\n
\n

In today’s fast-paced digital landscape, efficient and reliable software delivery is paramount for businesses to stay competitive. Buildkite, a powerful cloud-native CI\/CD platform, offers a robust solution to streamline the development process.<\/p>\n

Buildkite is a modern cloud-native CI\/CD platform that empowers development teams to build, test, and deploy software efficiently. At its core, a Buildkite pipeline is a series of interconnected jobs that execute in a specific order to automate the software development lifecycle. This blog post will delve into the fundamentals of Buildkite pipelines, exploring their components, configuration, and best practices.<\/p>\n

It uses a hybrid architecture to separate:<\/p>\n<\/div>\n

\n
    \n
  • \n
    The control panel<\/div>\n<\/li>\n
  • \n
    The build environment<\/div>\n<\/li>\n<\/ul>\n<\/div>\n
    \n

    Buildkite runs the control panel as a SaaS product, and we run the build environment on our infrastructure.
    \nIn other words, Buildkite handles the orchestration, and we can have our own compute unit.
    \nAs a result, we have a secure build environment to suit our particular use case and workflow.<\/p>\n

    \n
    \"Buildkite

    Buildkite Architecture<\/p><\/div>\n<\/div>\n<\/div>\n

    The hybrid architecture reduces the maintenance burden compared to self-hosted solutions and gives us the security and control missing from fully SaaS build tools.<\/p>\n

    Objective<\/h2>\n

    This article will provide a step-by-step guide for setting up a buildkite agent on EC2 and help you understand the buildkite pipeline basics.<\/p>\n

    Prerequisites<\/h2>\n
      \n
    • Amazon Linux 2023 EC2 instance<\/li>\n
    • Buildkite Account<\/li>\n
    • Github \/ Gitlab Account.<\/li>\n<\/ul>\n

      Agent Installation\u00a0<\/span><\/h3>\n
      \n
      \n

      Agents in buildkite is a program that is responsible for the execution of the pipeline. It’s a small, reliable, and cross-platform build runner that links our infrastructure to Buildkite. It checks in Buildkite for work, executes jobs, and also reports results. Agents can be installed on the local machine, any cloud servers, or any other remote machines. At least one agent will be required to run builds.<\/p>\n<\/div>\n

      In this example, we will install an agent on the “Amazon Linux 2023<\/strong>” EC2 Instance.<\/div>\n<\/div>\n
      <\/div>\n
      \n
      \"os-release\"

      os-release<\/p><\/div>\n<\/div>\n

      Step 1:<\/strong> First, Add the yum repository for our architecture (we executed uname -m to find our system’s arch).<\/p>\n

      \"uname

      system architecture<\/p><\/div>\n

      \n

      The default version of the agent is stable. Still, you can get the beta version by using unstable instead of stable in the following command, or the agent built from the repository’s main branch by using experimental instead of stable.<\/p>\n

      \n
      sudo<\/span> sh<\/span> -c 'echo -e \"[buildkite-agent]\\nname = Buildkite Pty Ltd\\nbaseurl = https:\/\/yum.buildkite.com\/buildkite-agent\/stable\/x86_64\/\\nenabled=1\\ngpgcheck=0\\npriority=1\" > \/etc\/yum.repos.d\/buildkite-agent.repo'<\/span>\r\n<\/code><\/pre>\n<\/div>\n
      \n
      Step 2:<\/strong> Install agent<\/p>\n
      sudo yum -y install buildkite-agent<\/pre>\n<\/div>\n<\/div>\n
      \"Agent
      Agent installed<\/p><\/div>\n
      \n
      Configure Agent token :<\/strong><\/p>\n<\/div>\n
      \n
      sudo sed -i \"s\/xxx\/YourActualAgentToken\/g\" \/etc\/buildkite-agent\/buildkite-agent.cfg<\/pre>\n<\/div>\n
      \n
      or, open the config file with a text editor and input your actual token here, token=\"xxx\"<\/code><\/p>\n<\/div>\n
      \n
      sudo vim \/etc\/buildkite-agent\/buildkite-agent.cfg<\/pre>\n<\/div>\n
      \"Agent
      Agent Token<\/p><\/div>\n
      \n
      Step 3:<\/strong> Start the buildkite agent and check its service status:<\/p>\n<\/div>\n
      \n
      sudo systemctl enable buildkite-agent && sudo systemctl start buildkite-agent && sudo systemctl status buildkite-agent<\/pre>\n<\/div>\n
      \"Agent
      Agent Status<\/p><\/div>\n
      \n
      The buildkite service log can be viewed using the below command:<\/p>\n<\/div>\n
      \n
      sudo journalctl -u buildkite-agent<\/pre>\n<\/div>\n
      \"Service
      Service Log<\/p><\/div>\n
      \n
      Step 4:<\/strong> SSH Setup in Agent Instance<\/p>\n<\/div>\n
      \n
      sh-4.2$ sudo su buildkite-agent\r\nbash-4.2$ mkdir -p ~\/.ssh && cd ~\/.ssh\r\nbash-4.2$ ssh-keygen -t rsa -b 4096 -C \"ec2-agent@buildkite\"<\/pre>\n
      \"SSH
      SSH Setup<\/p><\/div>\n
      Note<\/strong>: In the GitLab User Settings - SSH Keys<\/code> , Add your public key, key title, and expiration date.<\/p>\n<\/div>\n
      \"Add
      Add public key<\/p><\/div>\n
      \n
      Execute the below command to check if the SSH Setup was successful.<\/p>\n<\/div>\n
      \n
      \n
      ssh<\/span> -T git@gitlab.com<\/code><\/pre>\n<\/div>\n
      \n
      \"SSH
      Login Status check<\/p><\/div>\n<\/div>\n<\/div>\n
      \n
      Note<\/strong>: Exit from the\u00a0buildkite-agent<\/code>\u00a0user after setting up SSH.<\/p>\n<\/div>\n
      \n
      Step 5:<\/strong> Docker support for the Buildkite agent<\/p>\n<\/div>\n
      \n
      sudo yum install docker\r\nsudo systemctl enable docker.service\r\nsudo systemctl start docker.service\r\nsudo systemctl status docker.service<\/pre>\n
      \"docker
      docker status<\/p><\/div>\n
      \n
      Step 6:<\/strong> Docker-compose support for Buildkite agent<\/p>\n<\/div>\n
      \n
      # docker-compose (latest version)\r\nsudo curl -L https:\/\/github.com\/docker\/compose\/releases\/latest\/download\/docker-compose-$(uname -s)-$(uname -m) -o \/usr\/local\/bin\/docker-compose\r\n# Fix permissions after download\r\nsudo chmod +x \/usr\/local\/bin\/docker-compose\r\n# Verify success\r\ndocker-compose version<\/pre>\n
      \"docker
      docker-compose version<\/p><\/div>\n
      \n
      Step 7:<\/strong> Allow buildkite-agent<\/code> user to run the docker<\/p>\n<\/div>\n
      \n
      sudo usermod -a -G docker buildkite-agent<\/pre>\n
      \"buildkite
      Verify services<\/p><\/div>\n
      A few file locations are listed for quick access :<\/strong><\/h3>\n
        \n
      • Configuration: \/etc\/buildkite-agent\/buildkite-agent.cfg<\/li>\n
      • Agent Hooks: \/etc\/buildkite-agent\/hooks\/<\/li>\n
      • Builds: \/var\/buildkite-agent\/builds\/<\/li>\n
      • Agent user home: \/var\/lib\/buildkite-agent\/<\/li>\n
      • SSH keys: \/var\/lib\/buildkite-agent\/.ssh\/<\/li>\n
      • Logs, depending on your system:\n
          \n
        • journalctl -f -u buildkite-agent (systems)<\/li>\n
        • \/var\/log\/buildkite-agent.log (older systems)<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
          \n
          Running multiple agents<\/strong>\u00a0:
          \nYou can run as many parallel agent workers on one machine as you wish with the spawn<\/code> variable in a configuration file, or by passing the\u00a0--spawn<\/code> flag in the buidkite-agent start command.<\/p>\n
          # Start 3 workers. Each one independently fetches and executes jobs.\r\nspawn=3<\/pre>\n
          Creating a new queue for the buidkite agent :<\/strong>
          \nOpen the config file with the below command,<\/p>\n<\/div>\n
          \n
          sudo<\/span> vim<\/span> \/etc\/buildkite-agent\/buildkite-agent.cfg<\/code><\/pre>\n
          \n
          Add the below variable in the config file:<\/p>\n
          tags=\"queue=aws-ec2-agent\"<\/code><\/pre>\n<\/div>\n
          <\/div>\n<\/div>\n
          \n
          \"Agent
          Agent Queue<\/p><\/div>\n
          \n
          Note<\/strong>: To apply any changes done in\u00a0buildkite-agent.cfg<\/code> , you must execute the below command:<\/p>\n<\/div>\n
          \n
          sudo<\/span> systemctl reload-or-restart buildkite-agent<\/code><\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
          If all steps are done correctly, you will see your agent in the buildkite dashboard.<\/p>\n
          \"Agent
          Agent Queues<\/p><\/div>\n
          Pipeline Demonstration<\/h3>\n
          Pipelines are how Buildkite represents a CI\/CD workflow. You define each pipeline with a series of steps to run. When you trigger a pipeline, you create a build, and steps are dispatched as jobs to run on agents. Jobs are independent of each other and can run on different agents.<\/p>\n
          Step 1<\/strong>: Let’s create a new pipeline in Buildkite.<\/p>\n
          \"New
          New Buildkite Pipeline<\/p><\/div>\n
          You will need to provide your Git repository URL, Pipeline Name, Description and we will leave the Steps section as default.<\/p>\n
          Step 2: <\/strong>GitLab Webhook Integration<\/p>\n
          \"Gitlab
          Gitlab Webhook Integration<\/p><\/div>\n
          \"Gitlab
          Gitlab Webhook Integration<\/p><\/div>\n
          We can verify the webhook integration test, check the below sample for reference.<\/p>\n
          \"Webhook
          Webhook Integration Test<\/p><\/div>\n
          This integration triggers new builds whenever new code is pushed. Check the sample below,<\/p>\n
          \"Pipeline
          Pipeline Trigger<\/p><\/div>\n
          Conclusion<\/h2>\n
          There it is, we have successfully installed the buildkite agent on the EC2 Machine and also demonstrated setting up the sample pipeline. In Part 2 of this blog series, we will dig deeper into buidlkite pipeline & we will setup nodejs application build with the Snyk Vulnerability scan.<\/p>\n","protected":false},"excerpt":{"rendered":"
          Introduction In today’s fast-paced digital landscape, efficient and reliable software delivery is paramount for businesses to stay competitive. Buildkite, a powerful cloud-native CI\/CD platform, offers a robust solution to streamline the development process. Buildkite is a modern cloud-native CI\/CD platform that empowers development teams to build, test, and deploy software efficiently. At its core, a […]<\/p>\n","protected":false},"author":1741,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":39},"categories":[2348],"tags":[5992,4252,1892],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/62296"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/1741"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=62296"}],"version-history":[{"count":17,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/62296\/revisions"}],"predecessor-version":[{"id":63510,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/62296\/revisions\/63510"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=62296"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=62296"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=62296"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}