{"id":58430,"date":"2023-09-11T11:35:18","date_gmt":"2023-09-11T06:05:18","guid":{"rendered":"https:\/\/www.tothenew.com\/blog\/?p=58430"},"modified":"2023-09-14T11:42:33","modified_gmt":"2023-09-14T06:12:33","slug":"data-quality-with-pydeequ-a-comprehensive-guide","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/data-quality-with-pydeequ-a-comprehensive-guide\/","title":{"rendered":"Data Quality with PyDeequ: A Comprehensive Guide"},"content":{"rendered":"

Inadequate data quality can adversely affect both machine learning models and the decision-making process within a business. Unaddressed data errors can result in lasting repercussions, manifesting as blemishes and jolts. It is imperative in today’s landscape to implement automated tools for monitoring data quality, enabling the timely identification and resolution of issues. This proactive approach fosters greater confidence in the integrity of data and bolsters efficiency in data handling. Consequently, adopting automated data quality monitoring should be regarded as a strategic imperative for enhancing and sustaining an organization’s data systems.<\/span><\/p>\n

In this step-by-step guide, we\u2019ll take a look at the different types of data quality checks available in AWS pyDeequ, so you can get a better understanding of how it works.<\/span><\/p>\n

What is PyDeequ?<\/strong><\/h2>\n

PyDeequ is a Python library that provides a set of tools for data quality assessment and validation in large datasets. It allows users to define data quality checks, measure data quality metrics, and identify issues or anomalies within their data. PyDeequ is often used in data preprocessing and quality assurance tasks in data analytics and machine learning workflows.<\/span><\/p>\n

Setting up PyDeequ on PySpark<\/strong><\/h2>\n

\"\"<\/p>\n

Once you have installed these dependencies, you’ll be ready to use PyDeequ for data quality checks and assessments on your datasets within a PySpark environment.<\/span><\/p>\n

Implementing PyDeequ<\/strong><\/h2>\n

Main components<\/span><\/h3>\n

\"\"<\/p>\n

Profiling<\/b>: Profiling is the process of gathering basic statistics and information about the data. In PyDeequ, the profiler provides summary statistics, data type information, and basic data distribution insights for each column in your dataset. It helps you understand the characteristics of your data quickly.<\/span><\/p>\n

Analyzer:<\/i><\/b> The analyzer component goes beyond profiling and allows you to compute more advanced statistics and metrics for your data. This includes metrics like uniqueness, completeness, and other custom-defined metrics. Analyzers help you gain a deeper understanding of data quality issues in your dataset.<\/span><\/p>\n

Constraint Suggestion: <\/i><\/b>Constraint suggestion is a powerful feature of PyDeequ that automatically generates data quality constraints based on the profiling and analysis results. It suggests constraints such as uniqueness, completeness, and data type constraints that you can apply to your data to improve its quality.<\/span><\/p>\n

Verification:<\/i><\/b> Verification is the process of running data quality checks on your dataset using the defined constraints. PyDeequ’s verification component enables you to create and run data quality checks to validate whether your data conforms to the defined constraints. It provides detailed reports on the results of these checks, helping you identify data quality issues.<\/span><\/p>\n

Data Quality Metrics:<\/b> PyDeequ includes a set of predefined data quality metrics that you can use to measure and monitor the quality of your data. These metrics include measures like data completeness, distinctness, and uniformity, among others.<\/span><\/p>\n

Anomaly Detection:<\/i><\/b> PyDeequ also offers anomaly detection capabilities to identify unusual or unexpected patterns in your data. This can be particularly useful for spotting outliers or data points that deviate significantly from the norm.<\/span><\/p>\n

Check Builder:<\/b> PyDeequ provides a convenient CheckBuilder API that allows you to construct data quality checks programmatically. You can define custom checks based on your specific data quality requirements and apply them to your data.<\/span><\/p>\n

These main components of PyDeequ work together to help you assess, analyze, and improve the quality of your data within a PySpark environment. By leveraging these components, you can gain valuable insights into your data and ensure it meets the necessary quality standards for your analytics and machine learning projects.<\/span><\/p>\n

Available Data Quality Checks<\/strong><\/h3>\n

\u25cf <\/i>Completeness Checks:<\/i><\/b> These checks make sure that the specified fields in your data structure are filled with non-empty values. AWS pyDeequ offers functions to check the completeness of columns and find the missing values.<\/span><\/p>\n

\u25cf <\/i>Uniqueness Checks: <\/b>Uniqueness checks make sure there are no duplicates in a particular column or set of columns. You can also determine the uniqueness rate of columns using AWS PyData.<\/span><\/p>\n

\u25cf<\/i> Consistency Checks: <\/i><\/b>The purpose of consistency checks is to ensure consistency across data values. With the help of AWS pyDeequ, you can identify the values that are not consistent in categorical columns. This way, you can detect data entry errors or discrepancies.<\/span><\/p>\n

\u25cf <\/i>Functional Dependency Checks: <\/i><\/b>These checks determine whether one type of column defines another type of column.<\/span><\/p>\n

\u25cf <\/i>Pattern Checks:<\/i><\/b> In a pattern check, data is validated against pre-defined patterns (such as email addresses or phone numbers). With AWS PyDeeq, you can check whether your data matches those patterns.<\/span><\/p>\n

\u25cf <\/i>Value Distribution Checks: <\/i><\/b>Value distribution checks give you an idea of how the values are distributed within a single column. With the help of AWS pyDeequ, you can see how the unique values are distributed, which helps you to see where the data is skewed and where the data is balanced.<\/span><\/p>\n

\u25cf<\/i> Custom Checks:<\/b> With AWS PyDeeq, you can create your controls based on your business rules. With this flexibility, you can address domain-specific data quality issues.<\/span><\/p>\n

Definitions of supported checks<\/b><\/h3>\n

pyDeequ provides ~40 constraints that we can verify on our dataset based on the above scenario for checking the quality of your data.<\/span><\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Constraint<\/span><\/td>\nDefinition<\/span><\/td>\n<\/tr>\n
hasSize<\/span><\/td>\nAsserts on Data Frame Size.<\/span><\/td>\n<\/tr>\n
isComplete<\/span><\/td>\nAsserts on a column completion.<\/span><\/td>\n<\/tr>\n
hasCompleteness<\/span><\/td>\nAsserts non-null values\/total_values.<\/span><\/td>\n<\/tr>\n
areComplete<\/span><\/td>\nchecks all listed columns have non-null values.<\/span><\/td>\n<\/tr>\n
areAnyComplete<\/span><\/td>\nAsserts any completion in the combined set of columns.<\/span><\/td>\n<\/tr>\n
isUnique<\/span><\/td>\nAsserts on a column’s uniqueness.<\/span><\/td>\n<\/tr>\n
hasUniqueness<\/span><\/td>\nAsserts any uniqueness in a single or combined set of key columns.<\/span><\/td>\n<\/tr>\n
hasDistinctness<\/span><\/td>\nAsserts distinctness in a single or combined set of key columns.<\/span><\/td>\n<\/tr>\n
hasUniqueValueRatio<\/span><\/td>\nAsserts a unique value ratio in a single or combined set of key columns.<\/span><\/td>\n<\/tr>\n
hasNumberofDistinctValues<\/span><\/td>\nAsserts the number of distinct values in the column.<\/span><\/td>\n<\/tr>\n
hasHistogramValues<\/span><\/td>\nAsserts on column\u2019s value distribution.<\/span><\/td>\n<\/tr>\n
hasEntropy<\/span><\/td>\nAsserts on a column entropy.<\/span><\/td>\n<\/tr>\n
hasMutualInformation<\/span><\/td>\nAsserts on a piece of mutual information between two columns.<\/span><\/td>\n<\/tr>\n
hasApproxQuantile<\/span><\/td>\nAsserts on an approximated quantile.<\/span><\/td>\n<\/tr>\n
hasMinLength<\/span><\/td>\nAsserts on the minimum length of the column.<\/span><\/td>\n<\/tr>\n
hasMaxLength<\/span><\/td>\nAsserts on the maximum length of the column.<\/span><\/td>\n<\/tr>\n
hasMin<\/span><\/td>\nAsserts on the minimum value in the column.<\/span><\/td>\n<\/tr>\n
hasMax<\/span><\/td>\nAsserts on the maximum value in the column.<\/span><\/td>\n<\/tr>\n
hasMean<\/span><\/td>\nAsserts on the mean of column values.<\/span><\/td>\n<\/tr>\n
hasSum<\/span><\/td>\nAsserts on the sum of column values.<\/span><\/td>\n<\/tr>\n
hasStandardDeviation<\/span><\/td>\nAsserts on the standard deviation of the column.<\/span><\/td>\n<\/tr>\n
hasApproxCountDistinct<\/span><\/td>\nAsserts on the approximate count distinct of the given column.<\/span><\/td>\n<\/tr>\n
hasCorrelation<\/span><\/td>\nAsserts on the Pearson correlation between two columns.<\/span><\/td>\n<\/tr>\n
hasCompleteness<\/span><\/td>\nAsserts on completed rows in a combined set of columns.<\/span><\/td>\n<\/tr>\n
haveAnyCompleteness<\/span><\/td>\nAsserts on any completion in the combined set of columns.<\/span><\/td>\n<\/tr>\n
satisfies<\/span><\/td>\nAsserts on the given condition on the data frame( Where Clause).<\/span><\/td>\n<\/tr>\n
hasPattern<\/span><\/td>\nMatches the regex Pattern.<\/span><\/td>\n<\/tr>\n
containsCreditCardNumber<\/span><\/td>\nVerifies against a Credit Card pattern.<\/span><\/td>\n<\/tr>\n
containsEmail<\/span><\/td>\nVerifies against an Email pattern.<\/span><\/td>\n<\/tr>\n
containsURL<\/span><\/td>\nVerifies against a URL pattern.<\/span><\/td>\n<\/tr>\n
containsSocialSecurityNumber<\/span><\/td>\nVerifies against the Social Security number pattern for the US.<\/span><\/td>\n<\/tr>\n
hasDataType<\/span><\/td>\nVerifies against the fraction of rows that conform to the given data type<\/span><\/td>\n<\/tr>\n
isNonNegative<\/span><\/td>\nAsserts that a column contains no negative values.<\/span><\/td>\n<\/tr>\n
isPositive<\/span><\/td>\nFind ratio positive_values\/total_values.<\/span><\/td>\n<\/tr>\n
isLessThan<\/span><\/td>\nAsserts that in each row, the value of columnA < the value of columnB.<\/span><\/td>\n<\/tr>\n
isLessThanOrEqualto<\/span><\/td>\nAsserts that in each row, the value of columnA \u2264 the value of columnB.<\/span><\/td>\n<\/tr>\n
isGreaterThan<\/span><\/td>\nAsserts that in each row, the value of columnA > the value of columnB.<\/span><\/td>\n<\/tr>\n
isGreaterThanOrEqualTo<\/span><\/td>\nAsserts that in each row, the value of columnA \u2265 to the value of columnB.<\/span><\/td>\n<\/tr>\n
isContainedIn<\/span><\/td>\nAsserts that every non-null value in a column is contained in a set of predefined values.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Example:<\/b><\/h3>\n

Pydeequ supports simple sequential addition of constraints on any Pyspark data frame and returns the quality check report in both JSON\/CSV formats.<\/span><\/p>\n

Below is a sample code for adding constraints:<\/strong><\/p>\n

\"\"<\/p>\n

Generated Report:<\/strong><\/p>\n

\"\"<\/p>\n

\u00a0Conclusion<\/strong><\/h2>\n

Data quality is one of the most important aspects of your data-driven business. AWS PyDeequ provides a complete suite of data quality controls that you can easily integrate into your workflow. With the help of its functions, you can improve the accuracy, uniformity, and dependability of your data sets, resulting in more informed decisions and insights.<\/span><\/p>\n

In this step-by-step guide, we\u2019ve covered everything you need to know about AWS PyDeequ’s various data quality controls, including completeness, uniqueness, custom, and more. With this knowledge in hand, you\u2019re better equipped to make sure your data is as good as it can be, helping your organization reach its full potential.<\/span><\/p>\n

<\/div>","protected":false},"excerpt":{"rendered":"

Inadequate data quality can adversely affect both machine learning models and the decision-making process within a business. Unaddressed data errors can result in lasting repercussions, manifesting as blemishes and jolts. It is imperative in today’s landscape to implement automated tools for monitoring data quality, enabling the timely identification and resolution of issues. This proactive approach […]<\/p>\n","protected":false},"author":1583,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":773},"categories":[1395],"tags":[5412],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/58430"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/1583"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=58430"}],"version-history":[{"count":2,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/58430\/revisions"}],"predecessor-version":[{"id":58530,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/58430\/revisions\/58530"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=58430"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=58430"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=58430"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}