AWS, Big Data

Introduction to Amazon Athena

What is Amazon Athena? Amazon Athena is an analytics and interactive query service that use interactive standard SQL to analyze data stored on Simple Storage Service (S3). It is a serverless service i.e. there is no need to setup the instances for the datastore and manage the infrastructure. Also, there is no need to load data to...

by Shivam Agrawal
Tag: s3
24-Jan-2017

AWS, DevOps

Python Borrowing Resources using Fabric

In my previous blog post, I wrote about Cost Optimizations for the various AWS resources. The script consisted of six methods and the overall time required for the script to execute was roughly about 15-20 mins. While the script is running, it used to slow down my system. Also, there was a time consumption due to many reasons like...

by Aman Jain
Tag: s3
18-Jul-2016

AWS, DevOps

6 Tips for AWS Cost Optimization

As we know AWS provides a good edge in terms of cost over the on-premises data center or in a co-location environment with its On Demand and Reserved pricing. As it is rightly said "Reducing the overall cost is a high priority" and it is true for any organization whether big or small. By using AWS we can lower the IT costs, compute...

by Aman Jain
Tag: s3
30-Jun-2016

AWS, DevOps

Tweaking Logstash’s S3 plugin to create folders in YYYY/MM/DD format on AWS S3

Logstash is a service that accepts logs from a variety of systems, processes it and allows us to index it in Elasticsearch etc which can be visualised using Kibana. Our DevOps engineers have been using Logstash S3 plugin which simply puts all data in a S3 bucket location. Since we have configured files to be created in every hour on...

by Navjot Singh
Tag: s3
25-May-2016

AWS, Technology

Amazon S3 Transfer Acceleration – New Blessing for S3

Amazon has introduced a new feature for AWS S3 (Simple Storage Service) - AWS S3 Transfer Acceleration. This feature lets you transfer files from and to S3 at a much higher accelerated speed. Thus, you can save the file transfer time if you are ready to pay for it. The feature was introduced a few days back and I decided to investigate...

25-Apr-2016

AWS, Technology

AWS Lambda Invocation using Amazon S3

To start, we create a Lambda function to consume events published by Amazon S3. For any object uploaded to a bucket, S3 will invoke our Lambda function by passing event information in the form of function parameters. AWS Lambda executes the function. As the function executes, it reads the S3 event data, logs some of the event information...

by Ankit Giri
Tag: s3
10-Apr-2015

AWS

ELB Logs on S3 Bucket

Today we'll be implementing an S3 bucket policy for storing multiple Elastic Load Balancer access logs on a single S3 bucket. Instead of having multiple S3 bucket for each ELB access logs, we'll create only one S3 bucket for storing all ELB's access logs. This section walks you through the step by step guide for configuring S3...

by Vikash Jha
Tag: s3
27-Oct-2014

AWS

Adaptive Bitrate Streaming Using AWS

Adaptive Bitrate streaming is a technique employed in video streaming which detects end user's bandwidth and calibrates the video bitrate accordingly to guarantee the best viewing experience. This works by encoding source into streams of different bitrates and then each stream is fragmented into smaller multi-second chunks. A manifest...

by Hitesh Bhatia
Tag: s3
01-Apr-2014

Grails

S3 Bucket Permission

Amazon's Simple Storage Service (S3) allows its customer to maintain full control over who has access to their data with the help of its Identity Access Management (IAM) service and S3 bucket policies. For example, using Bucket Permission, one can give only reading access to one user, whereas using same permission/policy options, he...

by abhishek.tomar
Tag: s3
24-Feb-2014

AWS

IAM Role based access to S3 bucket.

In one of our projects, we came across a requirement where we were required to fetch a file containing important data from S3 in order to use it. Due to Security concerns we were not keen on storing access keys on the EC2 instance. Which basically meant that we could not configure S3 on that very system as configuring S3 would create a...

by Hitesh Bhatia
Tag: s3
29-Dec-2013

AWS

How to delete all contents/objects of Amazon S3 bucket

If you have an S3 bucket with a huge number of files, and you want to delete the bucket, you need to empty the bucket first. If the bucket has a handful of files, then it is easy enough from the AWS interface, but if the bucket has a large number of files, then AWS interface is not an option; and you must use s3cmd or may be some other...

by Deepak Mittal
Tag: s3
12-Nov-2011