Logstash is a service that accepts logs from a variety of systems, processes it and allows us to index it in Elasticsearch etc which can be visualised using Kibana. Our DevOps engineers have been using Logstash S3 plugin which simply puts all data in a S3 bucket location. Since we have configured files to be created in every […]
Amazon has introduced a new feature for AWS S3 (Simple Storage Service) – AWS S3 Transfer Acceleration. This feature lets you transfer files from and to S3 at a much higher accelerated speed. Thus, you can save the file transfer time if you are ready to pay for it. The feature was introduced a few […]
To start, we create a Lambda function to consume events published by Amazon S3. For any object uploaded to a bucket, S3 will invoke our Lambda function by passing event information in the form of function parameters. AWS Lambda executes the function. As the function executes, it reads the S3 event data, logs some of […]
Today we’ll be implementing an S3 bucket policy for storing multiple Elastic Load Balancer access logs on a single S3 bucket. Instead of having multiple S3 bucket for each ELB access logs, we’ll create only one S3 bucket for storing all ELB’s access logs. This section walks you through the step by step guide for […]
Adaptive Bitrate streaming is a technique employed in video streaming which detects end user’s bandwidth and calibrates the video bitrate accordingly to guarantee the best viewing experience. This works by encoding source into streams of different bitrates and then each stream is fragmented into smaller multi-second chunks. A manifest file is used at client’s end […]
Amazon’s Simple Storage Service (S3) allows its customer to maintain full control over who has access to their data with the help of its Identity Access Management (IAM) service and S3 bucket policies. For example, using Bucket Permission, one can give only reading access to one user, whereas using same permission/policy options, he can allow […]
In one of our projects, we came across a requirement where we were required to fetch a file containing important data from S3 in order to use it. Due to Security concerns we were not keen on storing access keys on the EC2 instance. Which basically meant that we could not configure S3 on that […]
If you have an S3 bucket with a huge number of files, and you want to delete the bucket, you need to empty the bucket first. If the bucket has a handful of files, then it is easy enough from the AWS interface, but if the bucket has a large number of files, then AWS […]
