What Is Intelligent Document Processing? A Primer

Robotic Process Automation (RPA) continues to gain momentum, powered by fast returns on investment that free workers to focus on higher-level, decision-making tasks. But the prevalence of unstructured and semi-structured data can inhibit automation flow and require a human to solve the bottleneck.

Businesses are creating data at a very high rate today, and that will only increase as we head into the future. However, 80% of that data is “dark data,” most researchers agree — meaning it’s locked in emails, text, PDFs, and scanned documents.

Intelligent Document Processing (IDP), sometimes referred to as intelligent capture, is a set of technologies that can be used to understand and turn unstructured and semi-structured data into a structured format.

Unlike optical character recognition (OCR), IDP uses artificial intelligence (AI) technologies such as machine learning (ML) and natural language processing (NLP) to capture, classify, and extract the most difficult-to-automate data. RPA technology can then be applied to the extracted data for enhanced validation and to automatically enter it into existing applications.

Today’s IDP solutions are expected to: 

• Be easy to set up and deploy
• Recognize and extract information despite different formats
• Continuously learn and improve over time
• Be noninvasive and integrate well with existing applications

The IDP market is expected to reach USD 1.1 billion this year, according to Everest Group, as businesses adopt the technology to maintain compliance and simplify data processing — especially large volumes of data.

How does intelligent document processing work?

IDP is a necessary element in businesses today. RPA needs intelligence, and AI needs automation to scale. While RPA can automate all the rule-based tasks, IDP fills in where RPA falls short. Together, IDP and RPA provide a simple yet effective tool to automate enterprise business processes.

IDP software solutions blend the power of AI technologies, such as computer vision, OCR, NLP, and machine/deep learning (DL). By doing so, they efficiently process all types of documents, extract relevant information, and feed the output into downstream applications. 

An enterprise-grade IDP solution goes through four phases (see Figure 1):

1. Pre-processing: Documents typically can exist in varying levels of quality, which can impact the results of data extraction. The pre-processing step improves the quality of the documents by applying techniques such as noise reduction, binarization, de-skewing, and more.
   
   
2. Classification: Documents can contain multiple pages with different formats. Intelligent document classification uses AI-based technologies to automatically classify and separate multipage documents to pull out the relevant pages of information before extraction.
   
   For example, a mortgage application typically includes multiple pages with documents such as a loan application, pay stubs, bank statements, ID documents, and others that need to be accurately identified and classified.
   
   
3. Extraction: This step involves OCR to digitize documents and ML technologies to extract specific data. Typically, IDP solutions include a library of pre-trained extraction models, which are pre-populated with the right fields for extraction. The relevant information is extracted from the document(s) before it’s validated for accuracy.
   
   
4. Post-processing: Once data is extracted, it goes through a series of validation rules and AI-driven techniques to improve the extraction results. These could be predefined rules validated in an automated fashion and enhanced by using RPA techniques. A human in the loop can further validate the data, which allows the process to continuously learn and improve over time.

How is IDP different from document capture?

Even though OCR is an integral part of the larger IDP process, it is only a small piece of a very complex puzzle. IDP solutions are more resilient to changing document formats and provide greater extraction accuracies. The capability of AI to learn from human actions over time enables these solutions to improve and provide higher extraction accuracy and straight-through processing (STP) rates.

Unlike traditional OCR technologies, which are hard to set up, IDP solutions are easy to deploy. They integrate best-in-class technologies to simplify use for business users, eliminating the need to involve IT when there’s a new document or a change in format.

IDP goes above and beyond traditional capture solutions to include classifying and extracting the data — and even feeding it into workflows to take humans out of the loop, reduce errors, and accelerate end-to-end business processes.

Intelligent document processing use cases

IDP can be applied to a wide variety of business use cases across all industries where knowledge workers continue to manually process documents. Examples include horizontal use cases, such as invoice processing and purchase order processing, as well as industry-specific use cases, such as insurance claims, loan applications, know your customer (KYC), and others.

According to the Everest Group IDP Peak Matrix Survey 2019, banking, financial services and insurance (BFSI) and healthcare industries made up more than 50% of the entire IDP customer base.

Adoption of IDP solutions helps businesses streamline their operations, achieve cost savings, and improve workforce productivity. Combined with RPA, IDP is emerging as a critical capability for large organizations because it allows them to scale their automation programs.

Though many IDP use cases are possible, the ones most widely adopted are those that are easy to kick-start and have a fast time to value. Automation Anywhere’s IDP solution, Document Automation, provides prebuilt extraction packages to allow businesses to choose from a menu of use cases.

The result is that Document Automation is easy to set up, often 4x or more faster than leading competitors, and the built-in AI allows for increased return on investment over time as the bots continuously learn and improve.

Let’s take an example of a customer who deployed intelligent document processing combined with RPA to deliver end-to-end business automation. Hitachi Vantara, a wholly-owned subsidiary of Hitachi Ltd., implemented Document Automation to automate its accounts payable invoice processes.

The company, which handles more than 80,000 invoices per year from more than 1,000 vendors and in 20 different languages, was able to improve its procure-to-pay process quality and reliability as a result. The IDP technology in Document Automation helped the company reduce errors by 100% in only 10 months — and realize 75% straight-through processing of invoices.