{"id":37672,"date":"2016-07-18T11:21:34","date_gmt":"2016-07-18T05:51:34","guid":{"rendered":"http:\/\/www.tothenew.com\/blog\/?p=37672"},"modified":"2016-07-20T13:44:18","modified_gmt":"2016-07-20T08:14:18","slug":"python-borrowing-resources-using-fabric","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/python-borrowing-resources-using-fabric\/","title":{"rendered":"Python Borrowing Resources using Fabric"},"content":{"rendered":"<p>In <a href=\"http:\/\/www.tothenew.com\/blog\/6-tips-for-aws-cost-optimization\/\">my previous blog post<\/a>, I wrote about Cost Optimizations for the <a title=\"AWS DevOps\" href=\"http:\/\/www.tothenew.com\/devops-aws\">various AWS resources<\/a>. The script consisted of six methods and the overall time required for the script to execute was roughly about 15-20 mins. While the script is running, it used to slow down my system.<\/p>\n<p>Also, there was a time consumption due to many reasons like network discrepancies, computation, and the\u00a0creation of CSV, &amp; XLS files. When I run this script along with other such several scripts in consolidation on my local machine, they do a lot of computation and as a result, my system becomes slow due to high usage of compute and memory power for the execution. I have no option but to wait for it to complete. To overcome this problem there were 3 ways:<\/p>\n<ul>\n<li>Increase the compute or memory capacity of the local machine but it is not always feasible and not very handy.<\/li>\n<li>Using a high resource machine altogether. Copy and run the whole script(s) on a different machine with high resources and then get the output on our local machine, but in this case, the all the logs generated would be on the remote machine and it will keep you engage with another machine.<\/li>\n<li>We execute the script on our own system but run parts of the scripts in multiple (idle) machines utilizing their resources and get the output in our own system.<\/li>\n<\/ul>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-37714\" src=\"\/blog\/wp-ttn-blog\/uploads\/2016\/07\/Blog2-71.png\" alt=\"Blog2-7\" width=\"641\" height=\"258\" \/><\/p>\n<p>So, I chose the 3rd option. Yes, I ran my script on my own machine which made use of multiple idle machines utilizing their resources and generated the final output on my own machine. The only thing is to divide and run the parts of the script (defined as functions) as per available idle machines dynamically.<\/p>\n<p>To address this, I made use of <a href=\"http:\/\/www.fabfile.org\/\">fabric<\/a> with the help of multiple remote machines. A Fabric is a library present in python which is basically used for ssh automation over multiple servers. Using it we can execute commands remotely like uploading and downloading files from the server or other computational jobs.<\/p>\n<p>The main reason was to divide the methods and run them separately on multiple machines at the same time so that we can utilize the computation power of other idle machines keeping my own machine free from any load and I can work parallel on other jobs. To use fabric you need to create a fabfile.py in which tasks are defined as shown below:<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"size-full wp-image-37674 aligncenter\" src=\"\/blog\/wp-ttn-blog\/uploads\/2016\/07\/blog2-1.png\" alt=\"blog2-1\" width=\"252\" height=\"91\" \/><\/p>\n<p>To run the task use the following command, for example:<br \/>\n<img decoding=\"async\" loading=\"lazy\" class=\"size-full wp-image-37675 aligncenter\" src=\"\/blog\/wp-ttn-blog\/uploads\/2016\/07\/blog2-2.png\" alt=\"blog2-2\" width=\"187\" height=\"21\" \/><\/p>\n<p>\u2018<strong>remote<\/strong>\u2019 is the address of remote machine(s)<\/p>\n<p>This command will execute the task on the remote machine and create a directory named \u2018new_dir\u2019 in the remote machine or any task you want.<\/p>\n<p>For my script, I decided to make use of 2 other servers apart from my own machine so the workload for my machine limited to only one third. It can be designed in such a way that it asks the user to enter any number of IP&#8217;s of the servers where you want that script to run depending upon the number of parts\/functions of the script you want to divide. For my use-case, I made the script work with a total of 3 IP\u2019s entered including my localhost:<br \/>\n<img decoding=\"async\" loading=\"lazy\" class=\"size-full wp-image-37676 aligncenter\" src=\"\/blog\/wp-ttn-blog\/uploads\/2016\/07\/blog2-3.png\" alt=\"blog2-3\" width=\"351\" height=\"91\" \/><\/p>\n<p>In my script, I had following 6 methods. The output of these functions (CSVs) is aggregated to have a single spreadsheet containing the results.<\/p>\n<ul>\n<li>idle_elb()<\/li>\n<li>idle_rds_instances()<\/li>\n<li>underutilized_ebs_volume()<\/li>\n<li>legacy_instance_type()<\/li>\n<li>low_utilization_ec2()<\/li>\n<li>idle_eip()<\/li>\n<\/ul>\n<p>The main script, when executed with 3 IPs will create 3 files by the name file1.py, file2.py, file3.py containing 2 methods each.<\/p>\n<p><b>main script<\/b>:<br \/>\n<img decoding=\"async\" loading=\"lazy\" class=\"size-full wp-image-37679 aligncenter\" src=\"\/blog\/wp-ttn-blog\/uploads\/2016\/07\/blog2-6.png\" alt=\"blog2-6\" width=\"354\" height=\"188\" \/><\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"size-full wp-image-37677 aligncenter\" src=\"\/blog\/wp-ttn-blog\/uploads\/2016\/07\/blog2-4.png\" alt=\"blog2-4\" width=\"382\" height=\"255\" \/><\/p>\n<p>Since I have created 3 tasks in the fabfile.py each of the tasks are called using the main script for execution. The IP\u2019s are stored in an array and the tasks in the fabfile are executed using subprocess so that we can run the shell command in python.<\/p>\n<p><b>fabfile.py<\/b> :<br \/>\n<img decoding=\"async\" loading=\"lazy\" class=\"size-full wp-image-37678 aligncenter\" src=\"\/blog\/wp-ttn-blog\/uploads\/2016\/07\/blog2-5.png\" alt=\"blog2-5\" width=\"787\" height=\"659\" \/><\/p>\n<p>The output generated (CSVs) on remote servers at the specified location are copied over to my local machine once all computational work is completed for final aggregation to a spreadsheet and then are deleted from the remote servers. \u00a0The important thing to note here is that all this happens in a single SSH session with each machine thus saving from network latencies.<\/p>\n<p>The main script uses multithreading for parallel execution of methods on the remote machines. At last, final aggregation happens locally and the final xls file is generated in my local system itself.<\/p>\n<p>Prerequisites for running the script:<\/p>\n<ul>\n<li><a href=\"https:\/\/pypi.python.org\/pypi\/boto\">Boto<\/a> (Version=\u20192.40.0\u2032) should be installed in the system and<a href=\"http:\/\/docs.pythonboto.org\/en\/latest\/boto_config_tut.html\"> configured<\/a><\/li>\n<li>AWS Account (Access Key Id, Secret Key) with READ ONLY access.<\/li>\n<li>To review the created CSV file, a<a href=\"https:\/\/pypi.python.org\/pypi\/xlwt\"> xlwt<\/a> package is required to be installed<\/li>\n<li>The ssh authentication method to the machines should be key based<\/li>\n<\/ul>\n<p>Happy Coding \ud83d\ude42<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In my previous blog post, I wrote about Cost Optimizations for the various AWS resources. The script consisted of six methods and the overall time required for the script to execute was roughly about 15-20 mins. While the script is running, it used to slow down my system. Also, there was a time consumption due [&hellip;]<\/p>\n","protected":false},"author":930,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":4},"categories":[1174,2348,1],"tags":[248,1342,1262,2366,1694,1611,1916,3661,4843,1892,573,1586,1897,1749,3760,375,3761,1358,670,3662,1692,3589],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/37672"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/930"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=37672"}],"version-history":[{"count":0,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/37672\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=37672"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=37672"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=37672"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}