{"id":65202,"date":"2024-09-15T10:56:00","date_gmt":"2024-09-15T05:26:00","guid":{"rendered":"https:\/\/www.tothenew.com\/blog\/?p=65202"},"modified":"2024-09-16T14:51:29","modified_gmt":"2024-09-16T09:21:29","slug":"unlocking-seamless-data-integration-in-the-cloud-with-azure-data-factory","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/unlocking-seamless-data-integration-in-the-cloud-with-azure-data-factory\/","title":{"rendered":"Unlocking Seamless Data Integration in the Cloud with Azure Data Factory"},"content":{"rendered":"

Introduction<\/h3>\n

In today’s data-driven world, managing and transforming data from various sources is a very cumbersome task for organizations. Azure Data Factory (ADF) stands out as an extensive and robust ETL and cloud-based data integration service that helps enable businesses to streamline their complex data-driven workflows timely and with ease. \u00a0Azure Data Factory provides a scalable and flexible platform that is designed to meet your data integration needs, whether you\u2019re orchestrating your ETL processes, automating data movement between on-premise and cloud, or transforming raw data into meaningful insight.<\/p>\n

Let\u2019s understand core ADF concepts:<\/p>\n

\"Azure

Azure Data Factory Components<\/p><\/div>\n

 <\/p>\n

    \n
  1. Pipelines<\/span><\/strong>: It is a logical grouping of activities.<\/li>\n
  2. Activities:<\/span><\/strong> Each activity performs a task. An activity could be a data movement e.g. taking an ingestion, pulling data into the cloud, and performing a transformation on it. There are different types of activities:\n
      \n
    1. Data Movement:<\/em><\/strong> For example a copy activity. It moves data between sources and sinks so that we can perform operations on the data.<\/li>\n
    2. Data Transformation<\/strong>:<\/em> These are external activities where computation is done outside ADF on a data store.
      \nFor example, a stored procedure that is executed on a\u00a0 Database, Azure Functions, Spark, etc.<\/li>\n
    3. Control:<\/em> <\/strong>Native to ADF. For eg., Web-calls, validate pipelines, etc.<\/li>\n<\/ol>\n<\/li>\n
    4. Data Sets<\/span><\/strong> represent data from any data store. Data sets get consumed by an activity in a pipeline as shown in the above figure.<\/li>\n
    5. Linked Services:<\/strong><\/span> These are\u00a0 the connection strings needed to connect to the data sets<\/li>\n
    6. Integration Runtime<\/strong><\/span>: It provides the compute that allows ADF to run the pipelines.<\/li>\n<\/ol>\n

      Use Case Scenario<\/h3>\n

      Now, let\u2019s try to understand the workings of ADF with the help of a use case. Suppose we have a storage account in which users upload all sorts of data, such as CSVs, images, HTML or text files, etc. We want only the images from the first storage account, for this, we\u2019ll copy the images from the source storage account to the destination storage account. We have to use wildcards in the pipeline to select only images. This is a simple task when you just have to use a single wildcard. You can straightaway just use the Copy Activity and achieve it. But here we have to move all images be it JPEG or PNG, which means we\u2019re dealing with multiple wildcards.<\/p>\n

      Prerequisites<\/h3>\n
        \n
      1. Blank Azure Data Factory<\/li>\n
      2. Two storage accounts (source, destination). In our example, sourceadfaccount<\/em> is the source storage account and destinationadfaccount <\/em>is the destination storage account.\n
        \"Source

        Source Storage Account<\/p><\/div>\n

         <\/p>\n

        \"Destination

        Destination Storage Account contents before running the pipeline<\/p><\/div>\n

         <\/li>\n

      3. Contributor access over the subscription.<\/li>\n<\/ol>\n

        Solution<\/h3>\n
          \n
        1. Go to your ADF Studio by clicking on the Launch Studio button.\n
          \"Azure

          Azure Data Factory Launch Studio<\/p><\/div>\n

           <\/li>\n

        2. Now we need to connect to our source and destination storage accounts, which means we need to create two linked services. For this, click on Manage in the left-hand side menu and select Linked Services. Click on + and then select \u201cAzure Blob Storage\u201d as the Data store.\n
          \"Add

          Add a new Linked Service<\/p><\/div>\n

          Fill in all the information for the source storage account and test the connection. Once successful, repeat the steps for the destination storage account.<\/p>\n

          \"Choose

          Choose Azure Blob Storage Data Store as the Linked Service<\/p><\/div>\n

           <\/li>\n

        3. Next, we have to create the two datasets, sourceimg<\/em> and destimg<\/em>. For this, select the Author menu item on the left side. Click on the + button and select Dataset. Now, again select the data storage as Blob Storage and because our image files are of type binary, we\u2019ll be selecting binary format.\n
          \"Add

          Add a new Data Set<\/p><\/div>\n

          Select source linked service, browse the file path, and click on OK.<\/p>\n

          \"Add

          Add Source Data Set<\/p><\/div>\n

          Repeat the same steps for the destination dataset.<\/li>\n

        4. Now, we\u2019ll start with creating our pipeline. Click on the + button again in the Author Tab and this time select Pipeline.\n

           <\/p>\n

          \"Create

          Create a new pipeline<\/p><\/div>\n

           <\/li>\n

        5. The first task for us is to traverse our source storage account and get all the files that need to be scanned. For this we\u2019ll be using the \u201cGet Metadata\u201d activity.\n

           <\/p>\n

          \"GetMetadata\"

          Get Metadata Activity<\/p><\/div>\n

           <\/p>\n

          Go to settings and add \u201cChild Items\u201d as an argument. This returns a list of sub-folders and files in the given folder. Returned value is a list of the name and type of each child item.<\/p>\n

          \"Modify

          Modify Activity Settings<\/p><\/div>\n

           <\/li>\n

        6. Now we have all the files from the source and next we need to filter out the image files (jpeg + png). For this task, we\u2019ll be using the ADF \u201cFilter\u201d activity. \n
          \"Filter

          Filter Activity<\/p><\/div>\n

           <\/p>\n

          Use the following expressions for items and conditions respectively:
          \nItems:\u00a0<\/span><\/strong> This gets the output files from the previous activity<\/p>\n

          @activity('getimagesfromsource').output.childItems<\/pre>\n
          Conditions:<\/span><\/strong> This is used to filter out the jpeg and png files from the output we received from the last activity.<\/p>\n
          @or(contains(item().name,'.jpeg'),contains(item().name,'.png'))\r\n\r\n<\/pre>\n<\/li>\n
        7. The filter activity will give us the desired filtered image files. Next, we need to iterate the output to copy these to the destination storage account. For this task we\u2019ll be using the \u201cFor Each\u201d activity.Under the settings, for the items click on \u201cAdd dynamic content\u201d and select the filter activity’s output value.\n
           <\/p>\n
          \"For
          For Each Activity<\/p><\/div>\n
           <\/p>\n
          Now go to the \u201cActivities\u201d tab for the \u201cFor Each\u201d\u00a0 activity and add the \u201cCopy\u201d\u00a0 Activity.<\/li>\n
        8. In the \u201cCopy\u201d Activity choose the source as the sourceimg<\/em> dataset. In the File path type, select “Wildcard file path” and for the filename add \u201c@item().name\u201d. \n
          \"Copy
          Copy image source<\/p><\/div>\n
           <\/p>\n
          For the sink, add the destimg<\/em> dataset.<\/p>\n
          \"Copy
          Copy image destination<\/p><\/div>\n
           <\/li>\n
        9. Go back to the main pipeline. Make sure you have connected all the activities. It will look something like this: \n
          \"Final
          Final Data Pipeline<\/p><\/div>\n
           <\/li>\n
        10. Next, we have to\u00a0 \u201cTrigger\u201d the pipeline. For that, you need to validate and publish your pipeline. You can also choose to trigger it now or anytime manually. \n
          \"Trigger
          Trigger Now<\/p><\/div>\n
          Click on \u201cNew\/Edit\u201d. Here you can choose to run the pipeline based on a schedule or various events:<\/p>\n
          \"Create
          Create a New Trigger<\/p><\/div>\n
           <\/p>\n
          We\u2019ll be choosing to trigger the pipeline now, as we already have the data. You can see the logs and output from the activity runs window at the bottom of the screen.<\/p>\n
          \"Activity
          Activity Runs<\/p><\/div>\n
           <\/p>\n
          Destination Storage account contents after running the pipeline:<\/p>\n
          \"Destination
          Destination Storage Account<\/p><\/div>\n
           <\/li>\n<\/ol>\n
          So as you can see above, the ADF pipeline has successfully copied the image files from the source to the destination storage account. You can also add multiple activities in between if you want to perform any data transformations before actually storing the contents in the destination. For example, if you want the files to be zipped you can select the compression type while setting up the dataset etc.<\/p>\n
          Summary<\/h3>\n
          In this blog, we understood what an Azure Data Factory (ADF) is and how it solves our use case. One can achieve a lot using ADF, whether you\u2019re modernizing legacy systems, building new data workflows, or scaling your analytics capabilities. ADF can adapt to your evolving needs. It makes managing and organizing data a lot easier so that you focus on more of the decision-making.\u00a0 If you\u2019re looking to streamline your data processes, Azure Data Factory could be just the tool you need.<\/p>\n
          \n","protected":false},"excerpt":{"rendered":"
          Introduction In today’s data-driven world, managing and transforming data from various sources is a very cumbersome task for organizations. Azure Data Factory (ADF) stands out as an extensive and robust ETL and cloud-based data integration service that helps enable businesses to streamline their complex data-driven workflows timely and with ease. \u00a0Azure Data Factory provides a […]<\/p>\n","protected":false},"author":1465,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":55},"categories":[2348],"tags":[3457,1197,6445,1892],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/65202"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/1465"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=65202"}],"version-history":[{"count":22,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/65202\/revisions"}],"predecessor-version":[{"id":66129,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/65202\/revisions\/66129"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=65202"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=65202"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=65202"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}