{"id":16699,"date":"2015-01-20T14:24:47","date_gmt":"2015-01-20T08:54:47","guid":{"rendered":"http:\/\/www.tothenew.com\/blog\/?p=16699"},"modified":"2015-08-04T16:02:51","modified_gmt":"2015-08-04T10:32:51","slug":"spark-1o1-revamping-hadoop","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/spark-1o1-revamping-hadoop\/","title":{"rendered":"Spark 1O1 – Revamping Hadoop"},"content":{"rendered":"

Big Data has witnessed a tremendous movement and growth over the last couple of years.\u00a0As per the top\u00a0research agencies, Big Data has recently emerged as the most successful \u201claunch pad\u201d, \u00a0 giving a way to the maximum number of start-up ever. \u00a0As the space evolves further, more and more organizations of varied sizes and discrete\u00a0domains have realized the\u00a0real potential it can offer for their\u00a0business. Majority of these organizations\u00a0have already factored\u00a0Big Data\u00a0\u00a0in their strategic roadmap, as\u00a0the main stream technology.<\/p>\n

Hence, It is no more a matter of ‘If’ but ‘what’, when it comes to Big Data adoption.<\/em><\/p>\n

With the potential opportunities that\u00a0comes along with Big Data, it also poses a continuous push on the Big Data vendors and technology\u00a0providers –\u00a0to regularly innovate and upgrade their technology stacks. This is critical to survive\u00a0the every growing\u00a0competition, on the face of the rapidly changing technology landscape of Big Data<\/a>.<\/p>\n

Recently, the\u00a0top vendors in the space\u00a0announced an upgrades of their respective core products\/platforms. It was primarily the\u00a0result of Spark graduating to the Apache top Level Project. Spark has been the most awaited technology ever since it’s existence and is\u00a0supposed to be a game-changer<\/strong>, pushing the boundaries\u00a0of Hadoop to next\u00a0level.<\/p>\n

Considering the above facts, we thought\u00a0to publish a series of blogs around Apache Spark that\u00a0will serve as quick start guide for anyone who wants to take a plunge into the Spark development<\/a> technology and intends to grow\u00a0as a Spark programmer.\u00a0\u00a0We will be regularly adding blogs under this series, so that it evolves logically \u00a0as we move down the series.<\/p>\n

BACKGROUND<\/strong><\/p>\n

In 2009, the University of California at Berkeley’s AMPLab\u00a0<\/strong>formally kicked off the Spark development, with the intent to address the primary shortcomings around Hadoop Development<\/a> (which I will take\u00a0up in the next section).<\/p>\n

In June 2013,\u00a0Spark made its way to Apache Incubator<\/strong>.\u00a0 With the obvious potential that Spark offered – it was able to instantly attract a large\u00a0group of contributors from the community and also received contributions from the most prevalent organizations in the Big Data space.<\/p>\n

In less than a year, on 27 February 2014<\/strong>\u2013 Spark graduated from the Incubator to Apache Top-Level Project (TLP). Since then, Spark has gathered\u00a0a lot of attention around the community and has\u00a0become one of the most active projects in the Apache Hadoop ecosystem. It has gained a massive adopting\u00a0across industry, with leading Big Data players\/organizations\u00a0including Spark as a core technology component of their respective products\/platforms\/tools.<\/p>\n

HADOOP Vs\u00a0SPARK<\/strong><\/p>\n

Needless to say, Hadoop has been a first ‘concrete’ revolutionary technology that\u00a0laid the foundation of Big Data as it exists now. It introduced\u00a0the world to a new programming paradigm which has literally changed the mindset\u00a0and approach of\u00a0dealing with huge data sets.<\/p>\n

Architecture wise, it was\u00a0composed\u00a0of two main parts\u00a0– HDFS<\/strong> (the storage or the distributed file system) and MapReduce<\/strong> (the compute or data processing engine). This\u00a0turned to\u00a0be a remarkable combo for crunching large data sets – in an efficient (parallel processing) and cost\u00a0effective (using commodity h\/w) manner.<\/p>\n

So where does\u00a0Spark fit in? \u00a0Overtime, people realized that Hadoop (rather “compute\u201d in particular) had some evident shortcomings\u00a0that constrained its adoption to a certain type of use cases only. Spark can be seen as new generation Hadoop which further takes\u00a0its\u00a0capability to a different dimension. Supporting\u00a0pseudo-distributed local mode, usually used only for testing or development purposes, It\u00a0not only provides speed to the existing\u00a0use-cases but opens up a whole new world of use-case types that have been out of\u00a0Hadoop’s reach.<\/p>\n

The table below shows a\u00a0cross-comparison between Hadoop & Spark on some of the key parameters. Clearly, Spark out performs Hadoop in\u00a0every department.<\/p>\n\n\n\n\n\n\n\n\n\n
PARAMETER<\/em><\/strong><\/td>\nHADOOP<\/em><\/strong><\/td>\nSPARK<\/em><\/strong><\/td>\n<\/tr>\n
Performance<\/em><\/strong><\/td>\nMajor issue arise as workflows become a little more complex ( MR sequencing is inefficient)<\/td>\nKnown for its performance. Benchmarks show even 100x performance compared to Hadoop.<\/td>\n<\/tr>\n
Machine Learning<\/em><\/strong><\/td>\nIncapable of handling \u2018Iterative<\/strong>\u2019 algorithms, which are predominantly used\u00a0for\u00a0ML.<\/td>\nEfficient use of all types of complex\u00a0algorithms.<\/td>\n<\/tr>\n
Streaming<\/em><\/strong><\/td>\nNot designed for event streams<\/td>\nBuilt for Real-time event streaming<\/td>\n<\/tr>\n
Implementation<\/em><\/strong><\/td>\nMR being a Low level Api makes the implementation challenging\u00a0and error prone.<\/td>\nDeveloper friendly\u00a0Api\u2019s – ease of implementation.<\/td>\n<\/tr>\n
Primary Memory Usage<\/em><\/strong><\/td>\nUses Disk I\/O\u2019s<\/td>\nIn-memory\u00a0processing engine.<\/td>\n<\/tr>\n
Analytics<\/em><\/strong><\/td>\nOffline – Batch Based Analytics<\/td>\nInteractive Analytic<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n


\nSPARK ECOSYSTEM<\/strong><\/p>\n

One of the primary focus, right from Spark’s\u00a0inception was to build a unified ecosystem around the core,\u00a0where in each ‘component’\u00a0caters to one specific area of focus. \u00a0The image below depicts the\u00a0‘Spark Ecosystem’.\u00a0Let’s quickly run through\u00a0each component one by one, for the sake of introduction. Later in the series, we will\u00a0have a dedicated post on each component, along with a hands on exercise\u00a0to\u00a0get a better hang on\u00a0the component and its capabilities.<\/p>\n

    \n
  1. Spark Streaming:<\/strong> It\u2019s the Real-time stream processing systems with the capability of auto- recovery from all kinds of failures. It is a batch based streaming solution.<\/li>\n
  2. Spark SQL:<\/strong> It\u2019s a SQL wrapper over Spark\u00a0for\u00a0querying the structured data sets. \u00a0The two important capabilities it comes with are: the directly support for direct access to Hive Queries and second, integration with various languages like\u00a0Python, Scala and Java. Apart from SQL queries it can efficiently\u00a0be used for\u00a0complex and interactive analytic algorithms. \u00a0Please note: Apache Shark is no more a supported product.<\/li>\n
  3. MLlib<\/strong>: It is the Machine Learning \u00a0\u00a0library for\u00a0Spark with\u00a0high-quality and performant algorithms. It goes a way beyond the two step Map – Reduce process. It is\u00a0interesting to note\u00a0that Apache Mahout which is currently the most stable and adopted ML library across globe –\u00a0has been\u00a0working closely with Spark to raise\u00a0the reach and performance of their algorithms.<\/li>\n
  4. GraphX<\/strong>: \u00a0It\u00a0unifies the ETL flow, exploratory analysis and iterative graph computation workflows by\u00a0leveraging the underlying Spark Api using RDD efficiently. It’s the core concept around which the Spark has been built. It requires\u00a0more than a paragraph to explain\u00a0how RDD’s function.\u00a0We will\u00a0talk about it more\u00a0in the coming Blogs.<\/li>\n<\/ol>\n

    CONCLUSION<\/strong><\/p>\n

    “Considering the overall value that Spark brings and keeping in mind the\u00a0pace, nature & direction\u00a0of its adoption, it should certainly be considered as the substitute\/replacement of Hadoop<\/em>” – UNLESS<\/strong>\u00a0one has made a conscious decision after thoroughly evaluating the tradeoffs between “cost”\u00a0vs\u00a0“expected\u00a0value” (and moreover, has no long term plans to extend, enhance or build on top of the\u00a0existing stack).<\/p>\n

    Probably there are many forums and blogs talking about various\u00a0use-cases where Hadoop might be a better choice over\u00a0Spark, but\u00a0didn’t\u00a0see a real gravity in the arguments – at least in the ones I’ve read, so far.<\/p>\n

    With this note, I end with the first part of the\u00a0Spark Series and keep a along\u00a0for the next part (s) to come shortly….. Till then, ‘Happy Sparking’<\/p>\n

     <\/p>\n","protected":false},"excerpt":{"rendered":"

    Big Data has witnessed a tremendous movement and growth over the last couple of years.\u00a0As per the top\u00a0research agencies, Big Data has recently emerged as the most successful \u201claunch pad\u201d, \u00a0 giving a way to the maximum number of start-up ever. \u00a0As the space evolves further, more and more organizations of varied sizes and discrete\u00a0domains […]<\/p>\n","protected":false},"author":153,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":4},"categories":[1395],"tags":[1515,1197,1398],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/16699"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/153"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=16699"}],"version-history":[{"count":0,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/16699\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=16699"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=16699"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=16699"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}