{"id":66515,"date":"2024-10-07T14:04:49","date_gmt":"2024-10-07T08:34:49","guid":{"rendered":"https:\/\/www.tothenew.com\/blog\/?p=66515"},"modified":"2024-10-08T16:35:05","modified_gmt":"2024-10-08T11:05:05","slug":"rss-feed-parsing-using-pyspark","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/rss-feed-parsing-using-pyspark\/","title":{"rendered":"RSS FEED PARSING using PySpark"},"content":{"rendered":"

Introduction<\/span><\/h2>\n

An RSS (Really Simple Syndication) feed is an online file that contains details about each piece of content a site has published. RSS feeds are a common way to distribute updates from websites and blogs. These feeds are often provided in XML format, and Python offers several tools to parse and extract information from them. This blog post will explore how to parse XML RSS feeds using Pyspark.<\/span><\/p>\n

\"RSS

RSS Feed Test Sample<\/span><\/p><\/div>\n

Prerequisites<\/strong><\/span><\/p>\n

Before we begin, ensure you have Python installed on your system (Link to install Python- https:\/\/www.python.org\/downloads\/<\/a> ).<\/span><\/p>\n

Understanding RSS Feeds and XML Parsing<\/strong><\/span><\/p>\n

RSS feeds contain articles, news items, or other updates in a structured XML format.\u00a0To work with these feeds, we can use the xml.etree.ElementTree or feedparser library in Python, provides an efficient way to parse XML data. Keep in mind that RSS feeds may contain additional elements beyond title, link, and description. If we use xml.etree.ElementTree we will have to adapt the parsing code to extract other elements of interest. We will be understanding the usage and implementation of the feedparser module in this blog.<\/span><\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Feature<\/strong><\/span><\/td>\nxml.etree.ElementTree<\/strong><\/span><\/td>\nFeedparser<\/strong><\/span><\/td>\n<\/tr>\n
Primary Use<\/span><\/td>\nGeneral XML parsing and manipulation<\/span><\/td>\nParsing RSS and Atom feeds<\/span><\/td>\n<\/tr>\n
Library Type<\/span><\/td>\nBuilt-in Python standard library<\/span><\/td>\nThird-party library (requires installation)<\/span><\/td>\n<\/tr>\n
Installation<\/span><\/td>\nNo installation required<\/span><\/td>\nRequires installation (pip install feedparser)<\/span><\/td>\n<\/tr>\n
Focus<\/span><\/td>\nGeneral XML data structures<\/span><\/td>\nSyndication formats (RSS, Atom)<\/span><\/td>\n<\/tr>\n
Parsing Capability<\/span><\/td>\nParses XML documents into ElementTree objects<\/span><\/td>\nParses RSS and Atom feeds into structured data<\/span><\/td>\n<\/tr>\n
XPath Support<\/span><\/td>\nBasic XPath support for querying XML<\/span><\/td>\nNone; focused on feed data extraction<\/span><\/td>\n<\/tr>\n
Element Handling<\/span><\/td>\nHandles elements, attributes, and text with tree structure<\/span><\/td>\nFocuses on extracting feed metadata and entries<\/span><\/td>\n<\/tr>\n
Feed Format Handling<\/span><\/td>\nNot specialized for RSS\/Atom feeds<\/span><\/td>\nSpecialized for handling various feed formats<\/span><\/td>\n<\/tr>\n
Data Access<\/span><\/td>\nManual traversal and querying of XML elements<\/span><\/td>\nStructured API for accessing feed information<\/span><\/td>\n<\/tr>\n
Modification Capability<\/span><\/td>\nAllows creation and modification of XML structures<\/span><\/td>\nRead-only; does not modify feeds<\/span><\/td>\n<\/tr>\n
Error Handling<\/span><\/td>\nBasic error handling for XML parsing<\/span><\/td>\nIncludes error handling for feed parsing issues<\/span><\/td>\n<\/tr>\n
Output Structure<\/span><\/td>\nProvides ElementTree objects with tag and text<\/span><\/td>\nProvides a structured dictionary-like object for feeds<\/span><\/td>\n<\/tr>\n
Common Use Cases<\/span><\/td>\nGeneral XML tasks, such as configuration files or data interchange<\/span><\/td>\nAggregating and processing feed data from news sources<\/span><\/td>\n<\/tr>\n
Performance<\/span><\/td>\nEfficient for standard XML tasks<\/span><\/td>\nOptimized for feed parsing but may be less flexible for non-feed XML<\/span><\/td>\n<\/tr>\n
what they can parse<\/span><\/td>\n<library><\/span>
\n<book><\/span>
\n<title>The Great Gatsby<\/title><\/span>
\n<author>F. Scott Fitzgerald<\/author><\/span>
\n<year>1925<\/year><\/span>
\n<\/book><\/span>
\n<book><\/span>
\n<title>To Kill a Mockingbird<\/title><\/span>
\n<author>Harper Lee<\/author><\/span>
\n<year>1960<\/year><\/span>
\n<\/book><\/span>
\n<\/library><\/span><\/td>\n		<?xml version=”1.0″ encoding=”UTF-8″ ?><\/span>
\n<rss version=”2.0″><\/span>
\n<channel><\/span>
\n<title>Sample RSS Feed<\/title><\/span>
\n<link>http:\/\/www.example.com<\/link><\/span>
\n<description>This is a sample RSS feed<\/description><\/span>
\n<item><\/span>
\n<title>First Post<\/title><\/span>
\n<link>http:\/\/www.example.com\/first-post<\/link><\/span>
\n<description>This is the description <\/description><\/span>
\n<pubDate>Mon, 01 Oct 2023 12:00:00 GMT<\/pubDate><\/span>
\n<\/item><\/span>
\n<\/channel><\/span>
\n<\/rss><\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Steps to Parse XML RSS Feeds<\/span><\/h3>\n

Let’s dive into the steps to parse an XML RSS feed using Python<\/span><\/h4>\n

Import the Required Libraries: Start by importing all the necessary libraries.<\/span>
\n
\nNote: Universal Feed Parser is a Python module that downloads and parses syndicated feeds. It can handle RSS 0.90, Netscape RSS 0.91, Userland RSS 0.91, etc, Atom 1.0 and more, CDF, and JSON feeds. It can also parse popular extension modules, like Dublin Core & Apple\u2019s iTunes extensions. To use Universal Feed Parser, you can use Python 3.8 or later versions. Universal Feed Parser is not meant to run standalone; it is a module for you to use as part of a larger Python program. Universal Feed Parser is very easy to use; it has one primary public function, “parse”. The parse function can take several arguments, but only one of them is required, and it can be a URL, a local filename, or a raw string containing feed data in any format.<\/span><\/p>\n

Let’s take\u00a0 a deep dive into parsing RSS Feed using Pyspark<\/h4>\n

1. Initialize Spark Session, Fetch, and Load RSS Feed: Create a Spark Session and upload files from remote or URL.<\/span><\/p>\n

from pyspark.sql import SparkSession\r\nimport feedparser \r\nimport glob \r\nimport os \r\n\r\nimport sys os.environ['PYSPARK_PYTHON'] = sys.executable \r\n\r\n# Initialize SparkSession \r\nspark = SparkSession.builder \\ \r\n.appName(\"RSS Feed Processor\") \\ \r\n.getOrCreate() \r\n\r\nfile_names = glob.glob('c:\/Users\/Ashita Kumar\/Downloads\/*.xml')<\/pre>\n
2. Declare Schema for the Data frame.<\/span><\/p>\n
# Define schema for DataFrame\r\nschema = StructType([\r\n StructField(\"file_name\", StringType(), True),\r\n StructField(\"feed_title\", StringType(), True),\r\n StructField(\"feed_link\", StringType(), True),\r\n StructField(\"feed_description\", StringType(), True),\r\n StructField(\"ID\", StringType(), True),\r\n StructField(\"title\", StringType(), True),\r\n StructField(\"description\", StringType(), True),\r\n StructField(\"link\", StringType(), True),\r\n StructField(\"image_link\", StringType(), True),\r\n StructField(\"condition\", StringType(), True),\r\n StructField(\"availability\", StringType(), True),\r\n StructField(\"price\", StringType(), True),\r\n StructField(\"name\", StringType(), True),\r\n StructField(\"points_value\", StringType(), True),\r\n StructField(\"ratio\", StringType(), True),\r\n StructField(\"item_group_id\", StringType(), True),\r\n StructField(\"brand\", StringType(), True),\r\n StructField(\"product_type\", StringType(), True),\r\n StructField(\"color\", StringType(), True),\r\n StructField(\"size_of_product\", StringType(), True),\r\n StructField(\"gender\", StringType(), True),\r\n StructField(\"sale_price\", StringType(), True),\r\n StructField(\"custom_label_0\", StringType(), True),\r\n StructField(\"custom_label_1\", StringType(), True),\r\n StructField(\"fb_product_category\", StringType(), True),\r\n StructField(\"age_group\", StringType(), True),\r\n])<\/pre>\n
<\/code><\/span><\/p>\n
3. Iterate through all files and parse the file using Feedparser. Use the requests library to fetch the RSS feed from a URL and load it into a feed object. We can load the RSS feed from a URL or even a storage location.<\/span><\/p>\n
# Create an empty list to store rows\r\nrows = []\r\n# Iterate through all files\r\nfor file in file_names:\r\n   print(\"Processing file:\", file)\r\n# Parse the XML file using feedparser\r\n   feed = feedparser.parse(file)\r\n# Extract feed details\r\n   feed_title = feed['feed'].get('title', '')\r\n   feed_link = feed['feed'].get('link', '')\r\n   feed_description = feed['feed'].get('description', '')\r\n# Title of the file\r\n   print(\"title of the feed \", feed['feed']['title'])\r\n# Link of the Feed\r\n   print(\"link of the feed \", feed['feed']['link'])\r\n# Description of the feed\r\n   print(\"description of the feed\", feed['feed']['description'])<\/pre>\n
\"Output
Output of print statements for feed details<\/span><\/p><\/div>\n
 <\/p>\n
 <\/p>\n
 <\/p>\n
 <\/p>\n
4. Parse RSS Items: Iterate through the RSS items to extract relevant information.<\/span><\/p>\n
# Iterate through feed entries\r\n for entry in feed.entries:\r\n   row = {\"file_name\": \"big_bazar_file\",\u00a0 # Use the actual file name\r\n         \"feed_title\": feed_title,\r\n         \"feed_link\": feed_link,\r\n         \"feed_description\": feed_description,\r\n         \"ID\": getattr(entry, 'g_id', ''),\r\n         \"title\": getattr(entry, 'g_title', ''),\r\n         \"description\": getattr(entry, 'g_description', ''),\r\n         \"link\": getattr(entry, 'g_link', ''),\r\n         \"image_link\": getattr(entry, 'g_image_link', ''),\r\n         \"condition\": getattr(entry, 'g_condition', ''),\r\n         \"availability\": getattr(entry, 'g_availability', ''),\r\n         \"price\": getattr(entry, 'g_price', ''),\r\n         \"name\": getattr(entry, 'g_name', ''),\r\n         \"points_value\": getattr(entry, 'g_points_value', ''),\r\n         \"ratio\": getattr(entry, 'g_ratio', ''),\r\n         \"item_group_id\": getattr(entry, 'g_item_group_id', ''),\r\n         \"brand\": getattr(entry, 'g_brand', ''),\r\n         \"product_type\": getattr(entry, 'g_product_type', ''),\r\n         \"color\": getattr(entry, 'g_color', ''),\r\n         \"size_of_product\": getattr(entry, 'g_size', ''),\r\n         \"gender\": getattr(entry, 'g_gender', ''),\r\n         \"sale_price\": getattr(entry, 'g_sale_price', ''),\r\n         \"custom_label_0\": getattr(entry, 'g_custom_label_0', ''),\r\n         \"custom_label_1\": getattr(entry, 'g_custom_label_1', ''),\r\n         \"fb_product_category\": getattr(entry, 'g_fb_product_category', ''),\r\n         \"age_group\": getattr(entry, 'g_age_group', '')}\r\n  rows.append(row)<\/span><\/pre>\n
5. Parse and store RSS feed item values in data frames using spark. We can create a dataframe and use for analytics or store it in parquet as required.<\/span><\/p>\n
# Create Spark DataFrame\r\ndf = spark.createDataFrame(rows, schema=schema)\r\n# Show the DataFrame with specific columns\r\ndf.show(truncate=False)\r\n# Stop the Spark session\r\nspark.stop()<\/pre>\n
\"df.show()
df.show() Output<\/span><\/p><\/div>\n
 <\/p>\n
 <\/p>\n
 <\/p>\n
 <\/p>\n
 <\/p>\n
Steps to Parse XML using Python<\/span><\/h3>\n
We can also parse RSS feeds using just Python. We’ll be able to achieve this using FeedParser and Pandas Libraries.<\/span><\/p>\n
Pros and Cons of using RSS Feed<\/strong><\/span><\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
PROS<\/strong><\/em><\/span><\/td>\nCONS<\/strong><\/em><\/span><\/td>\n<\/tr>\n
 1. Aggregated Content<\/strong><\/span><\/td>\n1. Decreased Popularity<\/strong><\/span><\/td>\n<\/tr>\n
Aggregates content from multiple sources into one feed.<\/span><\/td>\nRSS feeds are less popular now compared to social media and other news aggregators.<\/span><\/td>\n<\/tr>\n
 2. Customizable<\/strong><\/span><\/td>\n2. Requires an RSS Reader<\/strong><\/span><\/td>\n<\/tr>\n
Users can choose and customize their subscriptions based on interests.<\/span><\/td>\nAccessing RSS feeds requires an RSS reader or aggregator app.<\/span><\/td>\n<\/tr>\n
 3. Real-Time Updates<\/strong><\/span><\/td>\n3. Inconsistent Quality<\/strong><\/span><\/td>\n<\/tr>\n
Provides immediate updates when new content is published.<\/span><\/td>\nSome feeds may be outdated or have inconsistent formatting.<\/span><\/td>\n<\/tr>\n
 4. Ad-Free Experience<\/strong><\/span><\/td>\n4. Limited Multimedia Support<\/strong><\/span><\/td>\n<\/tr>\n
Delivers content without advertisements for a cleaner experience.<\/span><\/td>\nMay not handle multimedia content like videos or interactive elements well.<\/span><\/td>\n<\/tr>\n
 5. Privacy<\/strong><\/span><\/td>\n5. Fragmented Content<\/strong><\/span><\/td>\n<\/tr>\n
Subscribing to feeds does not require personal information, enhancing privacy.<\/span><\/td>\nFeeds are spread across various platforms, which can be disjointed.<\/span><\/td>\n<\/tr>\n
 6. Offline Access<\/strong><\/span><\/td>\n6. Mobile Experience May Vary<\/strong><\/span><\/td>\n<\/tr>\n
Content can be accessed offline through many RSS readers.<\/span><\/td>\nSome RSS readers might not provide a good mobile or tablet experience.<\/span><\/td>\n<\/tr>\n
 7. Easy Sharing<\/strong><\/span><\/td>\n7. Potential Information Overload<\/strong><\/span><\/td>\n<\/tr>\n
Feeds can be easily shared with others.<\/span><\/td>\nUsers may experience information overload if they subscribe to too many feeds.<\/span><\/td>\n<\/tr>\n
 8. No Algorithmic Filtering<\/strong><\/span><\/td>\n8. Static Content Delivery<\/strong><\/span><\/td>\n<\/tr>\n
Content is delivered in chronological order without algorithmic manipulation.<\/span><\/td>\nRSS feeds typically deliver static content without interactive features.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
Conclusion<\/span><\/h3>\n
Parsing XML RSS feeds using Python is a valuable skill for working with dynamic content from websites and staying up-to-date with the latest information. You can efficiently parse and extract data from RSS feeds by utilizing the feedparser library and the steps outlined in this blog post. Remember to adjust the parsing code based on the structure of the RSS feed you are working with.<\/span><\/p>\n
<\/code><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"
Introduction An RSS (Really Simple Syndication) feed is an online file that contains details about each piece of content a site has published. RSS feeds are a common way to distribute updates from websites and blogs. These feeds are often provided in XML format, and Python offers several tools to parse and extract information from […]<\/p>\n","protected":false},"author":1957,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":122},"categories":[6194],"tags":[6561,6559,6562,6560],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/66515"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/1957"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=66515"}],"version-history":[{"count":30,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/66515\/revisions"}],"predecessor-version":[{"id":68242,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/66515\/revisions\/68242"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=66515"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=66515"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=66515"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}