{"id":51281,"date":"2017-09-14T08:55:37","date_gmt":"2017-09-14T03:25:37","guid":{"rendered":"http:\/\/www.tothenew.com\/blog\/?p=51281"},"modified":"2017-09-14T10:03:07","modified_gmt":"2017-09-14T04:33:07","slug":"twitter-api-integration-with-aem-using-talend","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/twitter-api-integration-with-aem-using-talend\/","title":{"rendered":"Twitter API Integration with AEM Using Talend"},"content":{"rendered":"

Recently, I came across an interesting use case to fetch tweets from Twitter and post it to AEM instance<\/a>. Since tweets content can be huge and this use case is similar to that of migration, involving extraction, transformation, and loading of the content, I decided to use an ETL tool<\/a>\u00a0called Talend.<\/p>\n

\"talend<\/p>\n

Talend<\/a> is a leading open source integration software provider. It has set of <\/span>Twitter components pack<\/span><\/a> \u00a0based on <\/span>twitter4j<\/span><\/a> library which can help you connect to both <\/span>twitter API<\/b> (in case you want to gather past tweets to build a big dataset, though you can fetch tweets from last one week only at max, as mentioned in <\/span>Twitter Developer Documentation<\/span><\/a>) and<\/span> twitter stream API<\/b> (streams the live tweets).<\/span><\/strong><\/p>\n

We just need to define the queries in “tTwitterStreamInput” or “tTwitterInput” component based on our requirement of tweet source then we can post these result to AEM using HTTP post request.<\/span><\/p>\n

Below are few simple steps that can be followed to fetch the tweets and post it to the AEM instance:<\/p>\n

\"steps<\/p>\n

Step 1: <\/b>Create your own <\/span>twitter application<\/span><\/a> as you can not use the APIs anonymously. Follow the instructions from this <\/span>link<\/span><\/a> to do the same.<\/span><\/p>\n

Step 2: <\/b>Install <\/span>Twitter components pack<\/span><\/a> in your Talend instance. These components are not available in Talend by default. You’ll have to <\/span>download<\/span><\/a> and install them manually. Follow the instructions from this <\/span>link<\/span><\/a> to install the components in Talend.<\/span><\/p>\n

Step3:\u00a0<\/b>Restart your instance and create a new job. Drag \u00a0tTwitterOAuth, tTwitterStreamInput , tJavaRow and \u00a0tTwitterOAuthClose from the components palette. Connect them as per the below diagram:<\/span><\/strong><\/p>\n

\"Job<\/p>\n

Select the <\/span>tTwitterOAuth<\/span><\/a> component, which provides the connector facilities to authenticate against a Twitter App using Twitter OAuth authentication system and fills the fields with the strings you got from your Twitter App API keys page. In this context, API and Consumer are synonyms. You can choose the connection type here based on your requirement.<\/span><\/strong><\/p>\n

\"twitter<\/p>\n

I have used\u00a0tTwitterStreamInput<\/a> which gives only structured data as an output. You can also use\u00a0tTwitterInput<\/a> component, which provides\u00a0whole JSON response from twitter API.<\/strong><\/em><\/p>\n

Write your query in the tTwitterStreamInput component, create a schema and do the column mapping of Output.<\/p>\n

\"tTwitterStreamInput<\/p>\n

\"schema<\/p>\n

You can limit the number of tweets you want in one job in this component.<\/p>\n

\"tTwitterStreamInput<\/p>\n

Connect the output to the tJavaRow in which a custom code can be written to post the data to the AEM instance. <\/span><\/p>\n

I had created a nt:unstructured node corresponding to each tweet in JCR through HTTP post request. <\/span><\/p>\n

\"creating<\/p>\n

Alternatively, you can use tHttpRequest component to post the tweets to your AEM instance.<\/p>\n

Select the connection to be closed in <\/span>tTwitterOAuthClose<\/span><\/a> component. It will close the connection on completion of sub-job.<\/span><\/strong><\/p>\n

\"<\/p>\n

Hope you find the blog helpful !!!<\/span><\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"

Recently, I came across an interesting use case to fetch tweets from Twitter and post it to AEM instance. Since tweets content can be huge and this use case is similar to that of migration, involving extraction, transformation, and loading of the content, I decided to use an ETL tool\u00a0called Talend. Talend is a leading […]<\/p>\n","protected":false},"author":1019,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":5},"categories":[1994,1,4488],"tags":[4847,4394,1832,1415,4229,4458,1157,3361,2008,1018,4732],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/51281"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/1019"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=51281"}],"version-history":[{"count":23,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/51281\/revisions"}],"predecessor-version":[{"id":51713,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/51281\/revisions\/51713"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=51281"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=51281"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=51281"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}