How Does Snowflake Document AI Extract Data?<\/h2>\n![\"extraction](\"https:\/\/www.tothenew.com\/blog\/wp-ttn-blog\/uploads\/2025\/02\/extraction-1024x498.webp\")
extraction info from the document<\/p><\/div>\n
The process of extracting data from unstructured documents involves several key steps:<\/p>\n
\n- Document upload to stage<\/strong>\n
\n- Users upload unstructured documents (PDFs, scanned images, or text-based files) into Snowflake’s internal or external stage.<\/li>\n
- Snowflake preprocesses it to check for readability and stores metadata.<\/li>\n<\/ul>\n<\/li>\n
- Information Extraction<\/strong>\n
\n- Optical Character Recognition (OCR): <\/strong>It converts PDF files and images to natural English readable text.<\/li>\n
- Named Entity Recognition (NER): <\/strong>It identifies context-based important tags like dates, names, addresses, invoice numbers, etc.<\/li>\n<\/ul>\n<\/li>\n
- Data Structure & Storage<\/strong>\n
\n- The extracted information is converted into structured JSON formats.<\/li>\n
- The structured data is loaded into a snowflake table for querying, analytics, and reporting.<\/li>\n<\/ul>\n<\/li>\n
- Querying & Analytics\n
\n- Once the data is loaded to tables, users can perform SQL queries, ML modeling, and advanced analytics.<\/li>\n
- It enables organizations to derive hidden insights from unstructured data.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n
What\u2019s the Technology Behind Snowflake Document AI?<\/h3>\n
Snowflake Document AI leverages a combination of advanced AI\/ML models<\/strong> and proprietary algorithms<\/strong> to extract and structure data efficiently. Here are some of the key technologies behind the scenes:<\/p>\n\n- Optical Character Recognition (OCR)<\/strong>\n
\n- OCR technology helps convert images, scanned text, and PDFs into digital text.<\/li>\n
- Snowflake uses deep-learning-based OCR models to improve accuracy in handwritten and printed text recognition.<\/li>\n<\/ul>\n<\/li>\n
- Natural Language Processing (NLP)<\/strong>\n
\n- NLP models identify and categorize key entities, sections, and tables within a document.<\/li>\n
- Advanced language models (similar to Transformer-based AI models like BERT or GPT) enable context-aware text extraction.<\/li>\n<\/ul>\n<\/li>\n
- Named Entity Recognition (NER)<\/strong>\n
\n- Categories and identifies important context-based data tags such as Names, dates, addresses, invoice numbers, legal terms, etc.<\/li>\n
- It can also highly classify personal and sensitive data and tag it for business-critical applications.<\/li>\n<\/ul>\n<\/li>\n
- Machine Learning-Based Pattern Recognition<\/strong>\n
\n- Uses ML algorithms to detect tables, forms, and structured data patterns within documents.<\/li>\n
- It ensures that extracted data has its contextual meaning and relationships.<\/li>\n<\/ul>\n<\/li>\n
- Snowflake\u2019s Native AI Integration<\/strong>\n
\n- Snowflake Document AI is automatically integrated into Snowflake\u2019s ecosystem, It means users can perform SQL-based queries on extracted data without needing any additional ETL processes.<\/li>\n
- The processed data is directly available for BI tools, dashboards, and machine learning applications.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n
Oh… A lot of theory. Now it’s time to make your hands dirty. Head towards now Snowflake Snowsight interface. We will go step by step to extract the data from PDF files specifically internet bills that are uploaded to an internal stage in Snowflake. We’ll then insert the extracted information into a Snowflake table. Let’s get started!<\/p>\n
Step 1:\u00a0Create the database and the schema.<\/h4>\n-- Create the database \r\nCREATE DATABASE TESTDB; \r\n-- Create the schema inside the database \r\nCREATE SCHEMA TESTDB.TESTSCHEMA;\r\n\r\nUSE database testdb;\r\nUSE schema testschema;<\/pre>\nStep 2: Create an internal stage.<\/h4>\nCREATE OR REPLACE STAGE airtel_stage\r\nDIRECTORY = (ENABLE = TRUE)\r\nENCRYPTION = (TYPE = 'SNOWFLAKE_SSE');<\/pre>\nStep 3: Create a stream on stage.<\/h4>\nCREATE STREAM airtel_stream ON STAGE airtel_stage;\r\nSHOW STREAMS;\r\nALTER STAGE airtel_stage REFRESH;<\/pre>\nStep 4: Create a table.<\/h4>\n-- The metadata about the pdf file will be loaded to the first four columns and the information \r\n-- extracted will be loaded to the json_content column in JSON format which we need to parse JSON later.\r\n\r\nCREATE OR REPLACE TABLE airtel_bill (\r\nfile_name VARCHAR,\r\nfile_url VARCHAR,\r\nfile_size VARIANT,\r\nlast_modified VARCHAR,\r\njson_content VARCHAR\r\n);<\/pre>\nStep 5: Create a Document AI model<\/h4>\n![\"Document](\"https:\/\/www.tothenew.com\/blog\/wp-ttn-blog\/uploads\/2025\/02\/document-AI-Model-1024x439.png\")
Document AI Model<\/p><\/div>\n
\n- Go to the AI & ML menu from the left panel and select Document AI.<\/li>\n
- Click on the “+ Build”<\/strong> button top right corner to create a new model.<\/li>\n
- Provide the name of the model, database name, schema name, and description.<\/li>\n
- Click on the newly created model.<\/li>\n
- Go to the Document<\/strong> section and add a few test PDF files to train the model.<\/li>\n
- Go to the Values<\/strong> section and click on the Manage Values<\/strong> button.<\/li>\n
- Click on the “+ value”<\/strong> button to add a new Value. Provide a key name and question to extract the information.\n
\n- Example 1: The key name is “invoice_no”<\/strong> and the question would be “Extract the invoice number from this document”.<\/strong><\/li>\n
- Example 2: The key name is “bill_period”<\/strong> and the question would be “What is the billing period”.<\/strong><\/li>\n<\/ul>\n<\/li>\n
- You can add as many values as you need. It will try to extract the information from the document and If the extracted value is correct click on the “OK”<\/strong> button.<\/li>\n<\/ul>\n
![\"Manage](\"https:\/\/www.tothenew.com\/blog\/wp-ttn-blog\/uploads\/2025\/02\/manage-values.png\")
Manage Values<\/p><\/div>\n
\n- Now run this model on all of your test files and check whether it is extracting correct values or not. If not you have to change the format of your question so that it will extract correct information.<\/li>\n<\/ul>\n
![\"Extracting](\"https:\/\/www.tothenew.com\/blog\/wp-ttn-blog\/uploads\/2025\/02\/Extracting-info-1.png\")
Extracting Info<\/p><\/div>\n
\n- For testing try to extract the information from the files that have different formats because it will make your model efficient for any document.<\/li>\n
- Publish the document. It will show the “latest version”<\/strong> of your model and “extract a query section”<\/strong> to get the values from the document.<\/li>\n<\/ul>\n
Step 6: Create a task.<\/h4>\n-- Create a new task which will run interval of 1 minute and process the file uploaded to internal stage and loads the data to table airtel_bill.\r\n-- <model_name>!PREDICT() is the syntax to use to get the extracted data from a uploaded file.\r\n\r\nCREATE OR REPLACE TASK load_new_file_data\r\nWAREHOUSE = XSMALL\r\nSCHEDULE = '1 minute'\r\nCOMMENT = 'Process new files in the stage and insert data into the airtel_bill table.'\r\nWHEN SYSTEM$STREAM_HAS_DATA('airtel_stream')\r\nAS\r\nINSERT INTO airtel_bill (\r\nSELECT\r\nRELATIVE_PATH AS file_name,\r\nfile_url AS file_url,\r\nsize AS file_size,\r\nlast_modified,\r\nAIRTEL_BILL!PREDICT(GET_PRESIGNED_URL('@airtel_stage', RELATIVE_PATH), 3) AS json_content\r\nFROM airtel_stream\r\nWHERE METADATA$ACTION = 'INSERT'\r\n);\r\n\r\nALTER TASK load_new_file_data RESUME;<\/pre>\nStep 7: Show the extracted data by parsing the JSON-type column<\/h4>\n-- Use of LATERAL FLATTEN function to parse the JSON Object\r\nSELECT \r\nfile_name, \r\nfile_url, \r\nfile_size, \r\nlast_modified, \r\nb.value:value::STRING AS bill_date,\r\np.value:value::STRING AS bill_period,\r\ni.value:value::STRING AS invoice_no,\r\nn.value:value::STRING AS name,\r\nt.value:value::STRING AS total_bill,\r\no.value as ocr_score\r\nFROM \r\nairtel_bill,\r\nLATERAL FLATTEN(INPUT => PARSE_JSON(json_content):bill_date) b,\r\nLATERAL FLATTEN(INPUT => PARSE_JSON(json_content):bill_period) p,\r\nLATERAL FLATTEN(INPUT => PARSE_JSON(json_content):invoice_no) i,\r\nLATERAL FLATTEN(INPUT => PARSE_JSON(json_content):name) n,\r\nLATERAL FLATTEN(INPUT => PARSE_JSON(json_content):total_bill) t,\r\nLATERAL FLATTEN(INPUT => PARSE_JSON(json_content):__documentMetadata) o;<\/pre>\n
extraction info from the document<\/p><\/div>\n
The process of extracting data from unstructured documents involves several key steps:<\/p>\n
- \n
- Document upload to stage<\/strong>\n
- \n
- Users upload unstructured documents (PDFs, scanned images, or text-based files) into Snowflake’s internal or external stage.<\/li>\n
- Snowflake preprocesses it to check for readability and stores metadata.<\/li>\n<\/ul>\n<\/li>\n
- Information Extraction<\/strong>\n
- \n
- Optical Character Recognition (OCR): <\/strong>It converts PDF files and images to natural English readable text.<\/li>\n
- Named Entity Recognition (NER): <\/strong>It identifies context-based important tags like dates, names, addresses, invoice numbers, etc.<\/li>\n<\/ul>\n<\/li>\n
- Data Structure & Storage<\/strong>\n
- \n
- The extracted information is converted into structured JSON formats.<\/li>\n
- The structured data is loaded into a snowflake table for querying, analytics, and reporting.<\/li>\n<\/ul>\n<\/li>\n
- Querying & Analytics\n
- \n
- Once the data is loaded to tables, users can perform SQL queries, ML modeling, and advanced analytics.<\/li>\n
- It enables organizations to derive hidden insights from unstructured data.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n
What\u2019s the Technology Behind Snowflake Document AI?<\/h3>\n
Snowflake Document AI leverages a combination of advanced AI\/ML models<\/strong> and proprietary algorithms<\/strong> to extract and structure data efficiently. Here are some of the key technologies behind the scenes:<\/p>\n
- \n
- Optical Character Recognition (OCR)<\/strong>\n
- \n
- OCR technology helps convert images, scanned text, and PDFs into digital text.<\/li>\n
- Snowflake uses deep-learning-based OCR models to improve accuracy in handwritten and printed text recognition.<\/li>\n<\/ul>\n<\/li>\n
- Natural Language Processing (NLP)<\/strong>\n
- \n
- NLP models identify and categorize key entities, sections, and tables within a document.<\/li>\n
- Advanced language models (similar to Transformer-based AI models like BERT or GPT) enable context-aware text extraction.<\/li>\n<\/ul>\n<\/li>\n
- Named Entity Recognition (NER)<\/strong>\n
- \n
- Categories and identifies important context-based data tags such as Names, dates, addresses, invoice numbers, legal terms, etc.<\/li>\n
- It can also highly classify personal and sensitive data and tag it for business-critical applications.<\/li>\n<\/ul>\n<\/li>\n
- Machine Learning-Based Pattern Recognition<\/strong>\n
- \n
- Uses ML algorithms to detect tables, forms, and structured data patterns within documents.<\/li>\n
- It ensures that extracted data has its contextual meaning and relationships.<\/li>\n<\/ul>\n<\/li>\n
- Snowflake\u2019s Native AI Integration<\/strong>\n
- \n
- Snowflake Document AI is automatically integrated into Snowflake\u2019s ecosystem, It means users can perform SQL-based queries on extracted data without needing any additional ETL processes.<\/li>\n
- The processed data is directly available for BI tools, dashboards, and machine learning applications.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n
Oh… A lot of theory. Now it’s time to make your hands dirty. Head towards now Snowflake Snowsight interface. We will go step by step to extract the data from PDF files specifically internet bills that are uploaded to an internal stage in Snowflake. We’ll then insert the extracted information into a Snowflake table. Let’s get started!<\/p>\n
Step 1:\u00a0Create the database and the schema.<\/h4>\n
-- Create the database \r\nCREATE DATABASE TESTDB; \r\n-- Create the schema inside the database \r\nCREATE SCHEMA TESTDB.TESTSCHEMA;\r\n\r\nUSE database testdb;\r\nUSE schema testschema;<\/pre>\n
Step 2: Create an internal stage.<\/h4>\n
CREATE OR REPLACE STAGE airtel_stage\r\nDIRECTORY = (ENABLE = TRUE)\r\nENCRYPTION = (TYPE = 'SNOWFLAKE_SSE');<\/pre>\n
Step 3: Create a stream on stage.<\/h4>\n
CREATE STREAM airtel_stream ON STAGE airtel_stage;\r\nSHOW STREAMS;\r\nALTER STAGE airtel_stage REFRESH;<\/pre>\n
Step 4: Create a table.<\/h4>\n
-- The metadata about the pdf file will be loaded to the first four columns and the information \r\n-- extracted will be loaded to the json_content column in JSON format which we need to parse JSON later.\r\n\r\nCREATE OR REPLACE TABLE airtel_bill (\r\nfile_name VARCHAR,\r\nfile_url VARCHAR,\r\nfile_size VARIANT,\r\nlast_modified VARCHAR,\r\njson_content VARCHAR\r\n);<\/pre>\n
Step 5: Create a Document AI model<\/h4>\n
Document AI Model<\/p><\/div>\n
- \n
- Go to the AI & ML menu from the left panel and select Document AI.<\/li>\n
- Click on the “+ Build”<\/strong> button top right corner to create a new model.<\/li>\n
- Provide the name of the model, database name, schema name, and description.<\/li>\n
- Click on the newly created model.<\/li>\n
- Go to the Document<\/strong> section and add a few test PDF files to train the model.<\/li>\n
- Go to the Values<\/strong> section and click on the Manage Values<\/strong> button.<\/li>\n
- Click on the “+ value”<\/strong> button to add a new Value. Provide a key name and question to extract the information.\n
- \n
- Example 1: The key name is “invoice_no”<\/strong> and the question would be “Extract the invoice number from this document”.<\/strong><\/li>\n
- Example 2: The key name is “bill_period”<\/strong> and the question would be “What is the billing period”.<\/strong><\/li>\n<\/ul>\n<\/li>\n
- You can add as many values as you need. It will try to extract the information from the document and If the extracted value is correct click on the “OK”<\/strong> button.<\/li>\n<\/ul>\n
Manage Values<\/p><\/div>\n
- \n
- Now run this model on all of your test files and check whether it is extracting correct values or not. If not you have to change the format of your question so that it will extract correct information.<\/li>\n<\/ul>\n
Extracting Info<\/p><\/div>\n
- \n
- For testing try to extract the information from the files that have different formats because it will make your model efficient for any document.<\/li>\n
- Publish the document. It will show the “latest version”<\/strong> of your model and “extract a query section”<\/strong> to get the values from the document.<\/li>\n<\/ul>\n
Step 6: Create a task.<\/h4>\n
-- Create a new task which will run interval of 1 minute and process the file uploaded to internal stage and loads the data to table airtel_bill.\r\n-- <model_name>!PREDICT() is the syntax to use to get the extracted data from a uploaded file.\r\n\r\nCREATE OR REPLACE TASK load_new_file_data\r\nWAREHOUSE = XSMALL\r\nSCHEDULE = '1 minute'\r\nCOMMENT = 'Process new files in the stage and insert data into the airtel_bill table.'\r\nWHEN SYSTEM$STREAM_HAS_DATA('airtel_stream')\r\nAS\r\nINSERT INTO airtel_bill (\r\nSELECT\r\nRELATIVE_PATH AS file_name,\r\nfile_url AS file_url,\r\nsize AS file_size,\r\nlast_modified,\r\nAIRTEL_BILL!PREDICT(GET_PRESIGNED_URL('@airtel_stage', RELATIVE_PATH), 3) AS json_content\r\nFROM airtel_stream\r\nWHERE METADATA$ACTION = 'INSERT'\r\n);\r\n\r\nALTER TASK load_new_file_data RESUME;<\/pre>\nStep 7: Show the extracted data by parsing the JSON-type column<\/h4>\n
-- Use of LATERAL FLATTEN function to parse the JSON Object\r\nSELECT \r\nfile_name, \r\nfile_url, \r\nfile_size, \r\nlast_modified, \r\nb.value:value::STRING AS bill_date,\r\np.value:value::STRING AS bill_period,\r\ni.value:value::STRING AS invoice_no,\r\nn.value:value::STRING AS name,\r\nt.value:value::STRING AS total_bill,\r\no.value as ocr_score\r\nFROM \r\nairtel_bill,\r\nLATERAL FLATTEN(INPUT => PARSE_JSON(json_content):bill_date) b,\r\nLATERAL FLATTEN(INPUT => PARSE_JSON(json_content):bill_period) p,\r\nLATERAL FLATTEN(INPUT => PARSE_JSON(json_content):invoice_no) i,\r\nLATERAL FLATTEN(INPUT => PARSE_JSON(json_content):name) n,\r\nLATERAL FLATTEN(INPUT => PARSE_JSON(json_content):total_bill) t,\r\nLATERAL FLATTEN(INPUT => PARSE_JSON(json_content):__documentMetadata) o;<\/pre>\n
- Example 2: The key name is “bill_period”<\/strong> and the question would be “What is the billing period”.<\/strong><\/li>\n<\/ul>\n<\/li>\n
- Optical Character Recognition (OCR)<\/strong>\n
- Named Entity Recognition (NER): <\/strong>It identifies context-based important tags like dates, names, addresses, invoice numbers, etc.<\/li>\n<\/ul>\n<\/li>\n
- Optical Character Recognition (OCR): <\/strong>It converts PDF files and images to natural English readable text.<\/li>\n
![\"Info](\"https:\/\/www.tothenew.com\/blog\/wp-ttn-blog\/uploads\/2025\/02\/info-loaded.png\")