Session 4 AI Associate Series: UiPath Document Understanding Overview
- 2. 2 UiPath MVP RPA Teacher and Consultant UiPath MVP Solutions Architect Ingram Micro UiPath MVP Senior Tech Lead Proservartner Marcelo Cruz Sean Jerome Llanto Srinivas K Team Slide
- 3. 3 Get your documents processed intelligently Teach your robots to understand documents using AI-enhanced skills for data extraction and interpretation. Drag and drop these capabilities directly into your automation workflows to embed AI
- 4. 4 What is document understanding? Document Understanding Artificial Intelligence (AI) Document Processing Robotic Process Automation (RPA) Document understanding is the ability to extract and interpret information and meaning from a wide range of documents. It emerges at the intersection of document processing, AI, and RPA. Not OCR Not Computer Vision
- 5. 5 © Copyright UiPath 2022. All Rights Reserved. Document Understanding Features Document Understanding offers end-to-end capabilities to use a combination of rule-based and model-based approaches to process documents. Composable framework for flexibility and best technological choices End-to-end solution for extracting and interpreting information Support for various types of documents Support for processing different file formats Recognition of various document objects Usage of templates/rules and Machine Learning (ML) models to understand data AI-powered capabilities Model retraining capabilities Availability in cloud and on- premise
- 6. 6 Teach robots how to process your documents using intelligent drag-and-drop skills for data extraction and interpretation INTEL L I G E N T F L E X IBLE A C C U RATE EFFIC I E N T AI understands documents, takes actions, and learns from the data Getting rid of the “noise” caused by unrelated, rotated, or skewed documents Saving time and costs with seamless end-to-end automation Processing a wide range of documents and layouts, handwriting, checkboxes Machine learning (ML) skills improve over time based on the custom data Mix of template and template-less approaches for most accurate results
- 7. 7 • Like forms, passports, licenses, time sheets • Fixed in format and can contain handwriting, signatures, checkboxes • Like invoices, receipts, purchase orders, medical bills, utility bills • Containing fixed and variable parts like tables • Like contracts, agreements, emails, scripts, drug prescriptions, news • No fixed format, free-form sentences/paragraphs Which documents can be handled by Document Understanding? Structured documents Semi-structured documents Unstructured documents
- 9. 9 © Copyright UiPath 2022. All Rights Reserved. Document Understanding and the UiPath Business Automation Platform Studio Robots Orchestrator Action Center Pre-trained models available out of the box Bring your own model - custom or third-party Retrain the models Core RPA tools Human validation Integration Service AI Center
- 10. 10 1 2 3 Understand Act Receive Document Types • Structured • Semi-structured • Unstructured Document Variety • Multiple languages • Various formats • Varying templates • Handwriting • Signatures • Skewed docs • Checkboxes • Low quality scans Streamline end-to-end processes, improve business outcomes and reduce manual effort Built for: RPA and Citizen Developers – no data science skills required Built-in ML: Pre-trained ML models, data labeling, retraining Built as: SaaS, Self-hosted or Air-gapped integrated with UiPath Business Automation Platform DIGITIZE CLASSIFY EXTRACT VALIDATE RETRAIN ANALYZE End-to-end Intelligent Document Processing solution
- 11. 11 Load taxonomy for your documents How Document Understanding works Digitize images using multiple OCRs Classify documents Extract named entities in a taxonomy Validate and train supervised models Export extracted data 2 3 4 5 6 Digitize Classify Train & Validate Extract 2 3 5 4 Structured Semi-structured Unstructured Export RPA BPM API Other systems 6 1 Define taxonomy – document types and fields 1
- 12. 12 Document Understanding Typical Workflow Load taxonomy defines document types and fields for processing. Digitize documents using Optical Character Recognition (OCR) to make them machine- readable. Classify and split the files into document types. Extract information from the documents. Export the extracted data for further usage. Train classifiers based on the validated data. Train extractors based on the validated data. Validate classification results (human review). Validate extractors results (human review).
- 13. UiPath Document Understanding Overview - A Closer Look
- 14. 14 Taxonomy manager is used once at the start to define the collection of documents that you would want to process as well as business rules. Additionally, you can describe what data you would like to extract. Load taxonomy
- 15. 15 Move-For-You Co. We move so you don’t need to move PO: NP74006735 1 February 2020 PAYABLE WITHIN 15 DAYS OF RECEIPT 20800 ALMADEN AVE, SUITE 404 SAN JOSE, CA 95120-0520 T: +1 425 555 9876 F: +1 425 555 3456 E: billing@moveforyou-co.com www.moveforyou-co.com Bill To: Tony Tzeng 12345 Mango Lane Seattle, WA 98108 INVOICE DETAILS Packing services Storage fees (1 month) House move (white-glove service) Vehicle storage and transport Sales tax 10% Total Fee including Tax FEE $1,282.00 $1,884.00 $5,320.00 $5,186.00 $1,367.20 $15,039.20 Methods of payment Personal Check: Move-For-You LLC Wire Transfer: BigBank Co., Account 123456789-0987ABC Invoice No: 456200-TZE1 Digitize text in the documents using OCR
- 16. 16 Classify and split the documents Documents scanned into one file isn’t a problem – owing to classifiers, the robot can identify the document types and split the file to process the documents accordingly. Document Understanding offers different classification capabilities ranging from keyword-based to ML-based classification.
- 17. 17 Validate classification of the documents Classification Station is used to check, correct, and confirm the results of document classification and splitting.
- 18. 18 You can easily configure data extraction to choose most suitable extractor for each field. Use a combination of rule- based and model-based approaches to ensure smooth and accurate processing of different documents. Extract data from the documents
- 19. 19 Validate Extraction of Documents ▪ Validate the extracted information and handle exceptions using Validation Station. ▪ Now, retrain ML models using the data confirmed or corrected in Validation Station.
- 20. 20 © Copyright UiPath 2022. All Rights Reserved. Train Classifiers and Extractors Let the classifiers and extractors learn from the data corrected and validated in Classification Station and Validation Station, respectively.
- 21. 21 Export the Extracted Data End-to-end intelligent document processing Start & continue the document processing workflows with other automation components. Export the data for further usage/automation, for example, to an Excel spreadsheet, to SAP system, send as an email, and so on. Start Document Understanding Decisions Action Action Action End
- 23. 23 Document Types ▪ Required information found in the same place ▪ Fixed in format ▪ Examples: Forms, passports, licenses, and time sheets containing handwritten text, signatures, checkboxes ▪ Repetitive information each time ▪ Found in fixed and variable document parts such as tables ▪ Examples: Invoices, receipts, purchase orders, medical bills, bank statements, utility bills ▪ No fixed format ▪ Examples: Contracts, agreements, emails, disease descriptions, drug prescriptions, news, voice scripts Structured Semi-structured Unstructured
- 24. 24 Document Processing Methodologies Based on the document type, there are two common types of data extraction methodologies namely, rule-based and model- based. ▪ Rule-based approaches require users to create rules/templates that can best extract information from their documents. ▪ Model-based approaches rely on ML and statistical techniques. Both approaches are extremely potent tools but sometimes limited in their abilities to process optimally the range of documents companies can manage. The Document Understanding framework overcomes these limitations of an individual approach by implementing the hybrid approach. Hybrid Rule-based Model-based
- 25. 25 Document Processing Methodologies (Cont’d) Rule-based Structured fields, mostly used for structured documents Mostly structured documents, tables, checkboxes, handwriting, signatures Most structured documents (forms) Mostly semi–structured documents RegEx Based Extractor Form Extractor Forms AI Machine Learning Extractor Model-based Hybrid A combination of both – rule-based and model-based extractors Mostly documents combining both structured and less structured formats
- 26. 26 Document Processing Methodologies (Cont’d) Enables users to create and use a customized Regular Expression (RegEx) to extract information from a document. Rule-based Structured fields, mostly used for structured documents Mostly structured documents, tables, checkboxes, handwriting, signatures Most structured documents (forms) Mostly semi–structured documents RegEx Based Extractor Form Extractor Forms AI Machine Learning Extractor Model-based Hybrid A combination of both – rule-based and model-based extractors Mostly documents combining both structured and less structured formats
- 27. 27 Document Processing Methodologies (Cont’d) Enables users to create templates to extract, match, and report information by taking into consideration the words' position inside the document. Rule-based Structured fields, mostly used for structured documents Mostly structured documents, tables, checkboxes, handwriting, signatures Most structured documents (forms) Mostly semi–structured documents RegEx Based Extractor Form Extractor Forms AI Machine Learning Extractor Model-based Hybrid A combination of both – rule-based and model-based extractors Mostly documents combining both structured and less structured formats
- 28. 28 Document Processing Methodologies (Cont’d) Processes forms and documents that have similar formats and fixed formats with low diversity in layouts and provides point- and-click usage experience. Rule-based Structured fields, mostly used for structured documents Mostly structured documents, tables, checkboxes, handwriting, signatures Most structured documents (forms) Mostly semi–structured documents RegEx Based Extractor Form Extractor Forms AI Machine Learning Extractor Model-based Hybrid A combination of both – rule-based and model-based extractors Mostly documents combining both structured and less structured formats
- 29. 29 Document Processing Methodologies (Cont’d) Enables users to extract template-less similar data points from semi-structured or unstructured documents using ML models. Rule-based Structured fields, mostly used for structured documents Mostly structured documents, tables, checkboxes, handwriting, signatures Most structured documents (forms) Mostly semi–structured documents RegEx Based Extractor Form Extractor Forms AI Machine Learning Extractor Model-based Hybrid A combination of both – rule-based and model-based extractors Mostly documents combining both structured and less structured formats
- 30. 30 Rule-based or template-based approach Relies on rules (like regular expressions) and templates (including anchors) Processes fixed in format structured data Ensures high accuracy for already known documents
- 31. 31 Pre-trained models Machine learning (ML) models as a template-less approach Custom models • No-code light-weight models in Forms AI • Custom ML models in AI Center • Third-party models Model retraining Learn about sharing data for model retraining here • Invoices • Receipts • Purchase Orders • Utility Bills • Passports • ID Cards* • Legal Contracts • W-2 Forms • W-9 Forms • Delivery Notes • Remittance Advices • ACORD 125 • I9 Forms • 990 Forms • 4506T Forms • FM1003 Forms • Pay slips & personal earnings statements • Certificates of origin • EU declarations of conformity • Children’s product certificates • Certificates of incorporation • Shipping invoices • CMS1500 • Retraining via AI Center • Continuous learning loop based on human validated data
- 32. 32 Make use of pre-trained ML models to process invoices, receipts, utility bills, ID cards, and many more document types. Retrain the models to optimize them for your custom documents and improve the model accuracy over time! Bring your own model or third party models and incorporate them in your automations. Pre-trained ML models
- 33. 33 ML model training via AI Center You can use Document Manager to train your custom ML models or retrain the existing models in AI Center. This would help robots understand the specificities of your documents better. The more you work with the model, the more effective it becomes. Thus, the accuracy of the extracted data improves over time. Learn about sharing data for model retraining here.
- 34. 34 Example scenario: Mortgage packet audit post-closing Extract key loan information from documents Split the packet into underlying files for faster processing Robot monitors folder for new files, initiates document process Executed closing packet received and scanned • Document scanning • Digitization with OCR • Unattended robot • Pre-processing • Document classification (keyword, anchors, model) • RPA parallelization • Extraction • ML model-based and/or rule-based (hybrid) 1 2 3 4 Write results into line of business application Send exceptions for human review Compare / validate information across documents Check for signature present in executed fields • Signature detection • Unattended robot • Confidence / business rule- based exceptions • Validation Station • Attended robot • Action Center (Unattended RPA) • Unattended robot 5 6 7 8
- 36. Annex
- 37. 37 IDP combines Document Understanding and Communications Mining capabilities to help customers automate document processing from end to end. It delivers state-of-the-art Specialized AI and Generative AI (Gen AI) for all scenarios - documents and communications, structured, semi-structured and unstructured. Intelligent Document Processing (IDP) Introduction Extracts relevant data from documents. 70+ pre-built models to analyze and process different types of documents across industries and domains. Requests or emails with attached documents: • Multiple languages • Various formats • Handwriting • Signatures • Skewed & low-quality scans • Checkboxes • Tables Human in the loop Asking employees to validate the results if required or in case of inaccuracies and exceptions. UiPath Automation Route the extracted actions and data to downstream systems for further processing. Extracts key intent, sentiment and context data from messages. The latest advances in AI and machine learning (ML).
- 38. 38 Latest GenAI enhancement in our IDP offering Generative Extraction General availability Now Active Learning Public Preview Now, General availability April Generative Annotation General availability Now Generative Classification General availability Now Zero-Shot Discovery Public preview April Generative Extraction General availability Now Generative Validation Public preview Now, General availability April AutopilotTM for Communications Mining Public preview now Generative Annotation General availability Now Active Learning General availability Now
- 39. 39 Document Understanding: Generative Annotation (Pre-labeling) What is it? Fast & easy document annotation for ML model training with Generative AI You can annotate any document samples with Gen AI, accelerating annotation from a week to a day or two for complex scenarios, or down to minutes for simpler forms.
- 40. 40 Document Understanding: Generative Classification What is it? Document classification made easy with Generative AI Classifying documents is fast and easy with Gen AI – just define the document types, no need to write rules or train new ML models.
- 41. 41 Document Understanding: Generative Extraction What is it? Question-answering model powered by Generative AI Generative AI can answer questions and summarize content which works perfectly for free-form unstructured documents – with no need to train custom ML models.
- 42. 42 Document Understanding: Generative Validation What is it? Get a ‘second opinion’ on the extracted data from Generative AI to reduce the human validation effort With Generative AI used to confirm the output of Specialized AI, the overall automation rate increases by up to 200% and the average handle time decreases – reducing the time spent on human validation. When will it be available? • Public Preview now, GA in 2024.4 Source: Test by UiPath AI R&D on a diverse set of enterprise documents 200% increase Automation Rate
- 43. 43 Next-generation Document Understanding with active learning What? Active learning is a next-gen AI- powered experience within UiPath Document Understanding Why? • 80% faster model training—from a week, down to just a day • Anyone can train AI models—no coding or ML skills required • Guidance on model optimization— humans & AI collaborating together • Instant model evaluation—built-in model performance analytics Where and when? Public Preview now, GA in 2024.4
- 44. 44 Statement 1 Statement 2 Statement 3 Three Statement Slide
- 45. 45 “Quote text goes here. It can be short, but it shouldn’t be too long.” Author Name goes here Author Title goes here
- 46. 46 Computer Vision First Robotic Automation Early Growth Growth Global Expansion Category Leader First automation libraries for developers worldwide Desktop Automation product for Enterprise RPA Enterprise RPA Partnerships with global BPO & Consulting Firms Global offices 100 people 100+ enterprise customers Entered Japan Launched Academy Series A 700 Customers 550 People 100,000 Community 2,500 Customers 250,000 Community $200 Million Rev 31 Offices 18 Countries Raised $151 Million (or more) Cash-Flow Neutral 2005 2013 2015 2016 2017 2018 Timeline Slide
- 47. 47 Robot Attended Robot Unattended Orchestrator Studio Calendar Course foundation Enterprise- competency alt Growth Certification Course foundation 2 Equal Health Chart Course foundation 3 Error Heart Clock Course- orchestrator Exception handling Home Cloud app CRM Fast ROI Hourglass Code Desktop program First-link Automated data entry AI Keyboard Complete audit Document First-mention Automation AI Alt Label Contact email Done First-onebox Background automation AI Enabled Label alt Cost low Editor Goals Business partner-up alt App-3rd-party 2 Link special Cloud download Crown First Emoji Assessment Abstract Information Cloud upload Decrease First-like Autobiographer Advanced OCR Institutionalize Cloud secure Cultural acceptance First Flag Atom Add Input Contact email alt Ease of use First-quote Big scale Alert Alarm Leader Cost high Edit Flexibility Business partner-up App-3rd-party Link alt Corporation Ecosystem First-reply- by-email Bookmark Anniversary Link Course- advanced Enterprise competency Group of users Business partner alt 2 App-3rd-party 3 Light bulb Emoji Link special alt Icons (Gray) 1
- 48. 48 RPA champion Question and answer Question and answer 2 Ramp up Ramp up alt Read guidelines Reply Resource Redesign Redesign alt Refresh Refresh 2 Remove Remove alt RPA business analyst Reader Solutions 2 RPA developer RPA infrastructure engineer RPA service support RPA solution architect Satellite Share alt Slideshow Seamless integration Seamless integration alt Search Secure team collab Security Share Solutions RPA sponsor Thank you Solutions alt Speed Stopwatch Stories Success Technology alt 2 Technology alt 3 Support Survey Tap Touch Target Technology Technology alt Tent Student Time saver User User-OTM Validate Vendor Welcome Zoom in Visibility off Visibility on Warning Web expert Web scraping Web testing Zoom out Vector anchor Money square Location pin Lock Lock open Macro recording Managing Mobile device Money Managing alt Map Map alt Map alt 2 Media Mismatch Money circle Manage documents QA Monitoring Monitoring alt Note OCR Phone call Proof of concept Proof of concept 2 Pivot Plugin Process identification Process identification 2 Project Promoter bullhorn Public sector OCR alt Icons (Gray) 2
- 49. 49 Robot Attended Robot Unattended Orchestrator Studio Calendar Course foundation Enterprise- competency alt Growth Certification Course foundation 2 Equal Health Chart Course foundation 3 Error Heart Clock Course- orchestrator Exception handling Home Cloud app CRM Fast ROI Hourglass Code Desktop program First-link Automated data entry AI Keyboard Complete audit Document First-mention Automation AI Alt Label Contact email Done First-onebox Background automation AI Enabled Label alt Cost low Editor Goals Business partner-up alt App-3rd-party 2 Link special Cloud download Crown First Emoji Assessment Abstract Information Cloud upload Decrease First-like Autobiographer Advanced OCR Institutionalize Cloud secure Cultural acceptance First Flag Atom Add Input Contact email alt Ease of use First-quote Big scale Alert Alarm Leader Cost high Edit Flexibility Business partner-up App-3rd-party Link alt Corporation Ecosystem First-reply- by-email Bookmark Anniversary Link Course- advanced Enterprise competency Group of users Business partner alt 2 App-3rd-party 3 Light bulb Emoji Link special alt Icons (Mono) 1 49
- 50. 50 RPA champion Question and answer Question and answer 2 Ramp up Ramp up alt Read guidelines Reply Resource Redesign Redesign alt Refresh Refresh 2 Remove Remove alt RPA business analyst Reader Solutions 2 RPA developer RPA infrastructure engineer RPA service support RPA solution architect Satellite Share alt Slideshow Seamless integration Seamless integration alt Search Secure team collab Security Share Solutions RPA sponsor Thank you Solutions alt Speed Stopwatch Stories Success Technology alt 2 Technology alt 3 Support Survey Tap Touch Target Technology Technology alt Tent Student Time saver User User-OTM Validate Vendor Welcome Zoom in Visibility off Visibility on Warning Web expert Web scraping Web testing Zoom out Vector anchor Money square Location pin Lock Lock open Macro recording Managing Mobile device Money Managing alt Map Map alt Map alt 2 Media Mismatch Money circle Manage documents QA Monitoring Monitoring alt Note OCR Phone call Proof of concept Proof of concept 2 Pivot Plugin Process identification Process identification 2 Project Promoter bullhorn Public sector OCR alt Icons (Mono) 2 50
- 51. 51 Robot Attended Robot Unattended Orchestrator Studio Calendar Course foundation Enterprise- competency alt Growth Certification Course foundation 2 Equal Health Chart Course foundation 3 Error Heart Clock Course- orchestrator Exception handling Home Cloud app CRM Fast ROI Hourglass Code Desktop program First-link Automated data entry AI Keyboard Complete audit Document First-mention Automation AI Alt Label Contact email Done First-onebox Background automation AI Enabled Label alt Cost low Editor Goals Business partner-up alt App-3rd-party 2 Link special Cloud download Crown First Emoji Assessment Abstract Information Cloud upload Decrease First-like Autobiographer Advanced OCR Institutionalize Cloud secure Cultural acceptance First Flag Atom Add Input Contact email alt Ease of use First-quote Big scale Alert Alarm Leader Cost high Edit Flexibility Business partner-up App-3rd-party Link alt Corporation Ecosystem First-reply- by-email Bookmark Anniversary Link Course- advanced Enterprise competency Group of users Business partner alt 2 App-3rd-party 3 Light bulb Emoji Link special alt Icons (White) 1 51
- 52. 52 RPA champion Question and answer Question and answer 2 Ramp up Ramp up alt Read guidelines Reply Resource Redesign Redesign alt Refresh Refresh 2 Remove Remove alt RPA business analyst Reader Solutions 2 RPA developer RPA infrastructure engineer RPA service support RPA solution architect Satellite Share alt Slideshow Seamless integration Seamless integration alt Search Secure team collab Security Share Solutions RPA sponsor Thank you Solutions alt Speed Stopwatch Stories Success Technology alt 2 Technology alt 3 Support Survey Tap Touch Target Technology Technology alt Tent Student Time saver User User-OTM Validate Vendor Welcome Zoom in Visibility off Visibility on Warning Web expert Web scraping Web testing Zoom out Vector anchor Money square Location pin Lock Lock open Macro recording Managing Mobile device Money Managing alt Map Map alt Map alt 2 Media Mismatch Money circle Manage documents QA Monitoring Monitoring alt Note OCR Phone call Proof of concept Proof of concept 2 Pivot Plugin Process identification Process identification 2 Project Promoter bullhorn Public sector OCR alt Icons (White) 2 52