{"id":58125,"date":"2023-08-30T15:36:42","date_gmt":"2023-08-30T10:06:42","guid":{"rendered":"https:\/\/www.tothenew.com\/blog\/?p=58125"},"modified":"2023-09-04T17:00:10","modified_gmt":"2023-09-04T11:30:10","slug":"from-pandas-to-pyspark","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/from-pandas-to-pyspark\/","title":{"rendered":"From Pandas to Pyspark"},"content":{"rendered":"

Recently converted a Python script that relied on Pandas DataFrames to utilize PySpark DataFrames instead. The main goal is to transition data manipulation from the localized context of Pandas to the distributed processing capabilities offered by PySpark. This shift to PySpark DataFrames enables us to enhance scalability and efficiency by harnessing the power of distributed computing.<\/span><\/p>\n

Pandas stands as the indispensable library for data scientists, serving as a crucial tool for individuals seeking to manipulate and analyze data effectively.<\/span><\/p>\n

Nevertheless, while Pandas proves its usefulness and encompasses a wide range of functionalities, its limitations become evident when working with sizable datasets. To overcome this obstacle, a transition to PySpark becomes imperative as it enables distributed computing across multiple machines\u2014an advantage Pandas lacks. This article aims to simplify the process for newcomers to PySpark by offering code snippets that provide equivalents to various Pandas methods. With these readily available examples, navigating PySpark will be a smoother experience for aspiring practitioners.\u00a0<\/span><\/p>\n

<\/h2>\n

Getting started<\/b><\/span><\/h2>\n

To establish a solid foundation for our future discussions, it is imperative to start by importing the necessary libraries. This initial step lays the groundwork for further exploration and implementation.\u00a0<\/span><\/p>\n

\n
import pandas as pd<\/span>\r\nimport pyspark.sql.functions as F<\/span><\/pre>\n<\/blockquote>\n
The entry point into PySpark functionalities is the SparkSession class. Through the SparkSession instance, you can create data frames, apply all kinds of transformations, read and write files, etc.\u2026 To define a SparkSession, you can use the following :<\/span><\/p>\n
\n
 from pyspark.sql import SparkSession<\/span>\r\nspark = SparkSession\\<\/span>\r\n.builder\\<\/span>\r\n.appName('SparkByExamples.com')\\<\/span>\r\n.getOrCreate()<\/span><\/pre>\n<\/blockquote>\n
With all the preparations in place, we can now delve into the exciting comparison between Pandas and PySpark. Let’s explore the key differences and corresponding counterparts between these two powerful tools.<\/span><\/p>\n
<\/h2>\n
DataFrame creation<\/b><\/span><\/h2>\n
First, let\u2019s define a data sample we\u2019ll be using:<\/span><\/p>\n
columns = [\"employee\",\"department\",\"state\",\"salary\",\"age\"]<\/span>\r\ndata = [(\"Alain\",\"Sales\",\"Paris\",60000,34),<\/span>\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0(\"Ahmed\",\"Sales\",\"Lyon\",80000,45),<\/span>\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0(\"Ines\",\"Sales\",\"Nice\",55000,30),<\/span>\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0(\"Fatima\",\"Finance\",\"Paris\",90000,28),<\/span>\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0(\"Marie\",\"Finance\",\"Nantes\",100000,40)]<\/span><\/pre>\n
To create a <\/span>Pandas<\/b> DataFrame<\/span> , we can use the following:<\/span><\/p>\n
\n
df = pd.DataFrame(data=data, columns=columns)<\/span>\r\n# Show a few lines<\/span>\r\ndf.head(2)<\/span><\/pre>\n<\/blockquote>\n
PySpark<\/b><\/h3>\n
\n
df = spark.createDataFrame(data).toDF(*columns)<\/span>\r\n# Show a few lines<\/span>\r\ndf.limit(2).show()<\/span><\/pre>\n<\/blockquote>\n
<\/h2>\n
Specifying columns types<\/b><\/span><\/h2>\n
Pandas<\/b><\/h3>\n
\n
types_dict = {<\/span>\r\n \u00a0\u00a0\"employee\": pd.Series([r[0] for r in data], dtype='str'),<\/span>\r\n \u00a0\u00a0\"department\": pd.Series([r[1] for r in data], dtype='str'),<\/span>\r\n \u00a0\u00a0\"state\": pd.Series([r[2] for r in data], dtype='str'),<\/span>\r\n \u00a0\u00a0\"salary\": pd.Series([r[3] for r in data], dtype='int'),<\/span>\r\n \u00a0\u00a0\"age\": pd.Series([r[4] for r in data], dtype='int')<\/span>\r\n}<\/span>\r\ndf = pd.DataFrame(types_dict)<\/span><\/pre>\n<\/blockquote>\n
You can check your types by executing this line:<\/span><\/p>\n
\n
df.dtypes<\/span><\/pre>\n<\/blockquote>\n
PySpark<\/b><\/h3>\n
\n
\u00a0 \u00a0from pyspark.sql.types import StructType,StructField, StringType, IntegerType<\/span>\r\nschema = StructType([ \\<\/span>\r\n \u00a0\u00a0StructField(\"employee\",StringType(),True), \\<\/span>\r\n \u00a0\u00a0StructField(\"department\",StringType(),True), \\<\/span>\r\n \u00a0\u00a0StructField(\"state\",StringType(),True), \\<\/span>\r\n \u00a0\u00a0StructField(\"salary\", IntegerType(), True), \\<\/span>\r\n \u00a0\u00a0StructField(\"age\", IntegerType(), True) \\<\/span>\r\n ])<\/span>\r\ndf = spark.createDataFrame(data=data,schema=schema)<\/span><\/pre>\n<\/blockquote>\n
You can check your DataFrame\u2019s schema by executing :<\/span><\/p>\n
\n
\u00a0df.dtypes<\/span>\r\n# OR<\/span>\r\ndf.printSchema()<\/span><\/pre>\n<\/blockquote>\n
 <\/p>\n
Reading and writing files<\/b><\/span><\/h2>\n
Reading and writing are so similar in Pandas and PySpark. The syntax is the following for each:
\n<\/span><\/p>\n
Pandas<\/b><\/h3>\n
\n
df = pd.read_csv(path, sep=';', header=True)<\/span>\r\ndf.to_csv(path, ';', index=False)<\/span><\/pre>\n<\/blockquote>\n
PySpark<\/b><\/h3>\n
\n
df = spark.read.csv(path, sep=';')<\/span>\r\ndf.coalesce(n).write.mode('overwrite').csv(path, sep=';')<\/span><\/pre>\n<\/blockquote>\n
Add a column<\/b><\/span><\/h2>\n
Pandas<\/b><\/h3>\n
\n
score = [1, 2, 5, 7, 4]<\/span>\r\ndf[score] = score<\/span><\/pre>\n<\/blockquote>\n
Pyspark<\/b><\/h3>\n
\n
score = [1, 2, 5, 7, 4]<\/span>\r\ndf.withColumn('score', F.lit(score))<\/span><\/pre>\n<\/blockquote>\n
Filtering<\/b><\/span><\/h2>\n
Selecting certain columns in Pandas is done like below:<\/span> <\/b><\/p>\n
Pandas<\/b><\/h3>\n
\n
columns_subset = ['employee', 'salary']<\/span>\r\ndf[columns_subset].head()<\/span>\r\ndf.loc[:, columns_subset].head()<\/span><\/pre>\n<\/blockquote>\n
Whereas in PySpark, we need to use the select method with a list of columns: <\/span><\/p>\n
PySpark<\/b><\/h3>\n
df.select(['employee', 'salary']).show(5)<\/span><\/pre>\n
Concatenate dataframes<\/b><\/span><\/h2>\n
Pandas<\/b><\/h3>\n
\n
df = pd.concat([df1, df2], ignore_index = True)<\/span><\/pre>\n<\/blockquote>\n
Pyspark<\/b><\/h3>\n
\n
df = df1.union(df2)<\/span><\/pre>\n<\/blockquote>\n
Aggregations<\/b><\/span><\/h2>\n
To perform some aggregations, the syntax is almost similar between Pandas and PySpark: <\/span><\/p>\n
Pandas<\/b><\/h3>\n
\n
df.groupby('department').agg({'employee': 'count', 'salary':'max', 'age':'mean'})<\/span><\/pre>\n<\/blockquote>\n
PySpark<\/b><\/h3>\n
\n
df.groupBy('department').agg({'employee': 'count', 'salary':'max', 'age':'mean'})<\/span><\/pre>\n<\/blockquote>\n
However, the results need some tweaking to be similar in pandas and PySpark.<\/span><\/p>\n
In pandas, the column to group by becomes the index. <\/span>To get it back as a column, we need to apply the<\/span> reset_index<\/span> method:\u00a0<\/span><\/p>\n
Pandas<\/b><\/h3>\n
\n
df.groupby('department').agg({'employee': 'count', 'salary':'max', 'age':'mean'}).reset_index()<\/span><\/pre>\n<\/blockquote>\n
In<\/span> PySpark, <\/b>the names of the columns get modified to reflect the performed aggregation in the resulting data frame. If you wish to avoid this, you\u2019ll need to use the alias method.<\/span><\/p>\n

Apply a transformation over a column<\/b><\/span><\/h2>\n
Pandas<\/b><\/h3>\n
\n
df['new_score'] = df['score'].apply(lambda x: x*5)<\/span><\/pre>\n<\/blockquote>\n
Pyspark<\/b><\/h3>\n
\n
from pyspark.sql.types import FloatType<\/span>\r\n\r\ndf.withColumn('new_score', F.udf(lambda x: x*5, FloatType())('score'))<\/span><\/pre>\n<\/blockquote>\n
Conclusion<\/b><\/h2>\n
In conclusion, the striking similarities in syntax between Pandas and PySpark will greatly facilitate the transition from one framework to the other. This similarity enables a smooth and seamless migration between the two, easing the learning curve and minimizing the challenges associated with adapting to a new environment.<\/span><\/p>\n
Utilizing PySpark offers a significant advantage when dealing with large datasets due to its capability for parallel computing. However, if the dataset being handled is small, it becomes more efficient to switch back to using the versatile and widely used Pandas library. In such cases, leveraging Pandas allows quicker and more streamlined data processing, making it the preferred choice. Please refer to our blogs for more insightful content and comment if you have any questions regarding the topic.<\/span><\/p>\n
<\/div>","protected":false},"excerpt":{"rendered":"
Recently converted a Python script that relied on Pandas DataFrames to utilize PySpark DataFrames instead. The main goal is to transition data manipulation from the localized context of Pandas to the distributed processing capabilities offered by PySpark. This shift to PySpark DataFrames enables us to enhance scalability and efficiency by harnessing the power of distributed […]<\/p>\n","protected":false},"author":1624,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":25},"categories":[1395,4308,4682,4831],"tags":[],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/58125"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/1624"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=58125"}],"version-history":[{"count":3,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/58125\/revisions"}],"predecessor-version":[{"id":58296,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/58125\/revisions\/58296"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=58125"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=58125"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=58125"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}