Cognitive Document Automation - How Do We Measure Success?

Cognitive Document Automation: How Do We Measure Success?
November 20, 2019 | By Ryan Bazler

Welcome to part four of our six-part series that takes you on a journey through the latest concepts in multichannel document capture and intelligent OCR.  We’re focused on how AI has transformed what’s possible in making your documents and data work for you—and not against you.

In part one, we looked at how RPA marked a revolution in empowering businesses to solve problems associated with manual, data-centric tasks, yet was historically ineffective in automating document processing. In part two, we examined the emergence of cognitive document automation (CDA), which does the “head work” of understanding what the document or email is about, what information it contains and what to do with it. And in part three of our series, we did a deeper dive into what you should look for in a CDA solution (Hint: You’ll need much more than just OCR functionality). In part four, we tackle this question: How do we measure the success of CDA?

First, let’s acknowledge that the objective of any CDA project is to see the benefits of greater visibility, lower costs, faster processes, fewer errors and improved customer engagement.  When it comes to gauging the success of CDA, the single largest indicator is user productivity—the degree to which people become more productive because CDA helps them get work done better and faster. User productivity is made up of two components: OCR accuracy and user efficiency.

Let’s take a closer look at each of these:

OCR accuracy

Perhaps the most common question after watching a CDA technology demonstration is: What level of OCR accuracy can I expect to achieve? The short answer is: it depends and varies widely across use cases. OCR accuracy, and more generally classification and extraction accuracy, is determined by several factors, such as:
  • Scanner hardware
  • Scan resolution
  • Image quality
  • Document type (form, invoice, letter)
  • Machine-printed/hand-printed/cursive
  • Document language
  • Font type and character spacing
  • Field boxes/shadings
  • Ability to database match or check for checksums and other rules

Some of these factors will be discussed in greater depth later in our blog series. But for now, just remember this: the higher the accuracy, the more classification and extraction automation; the lower the accuracy, the less automation and more manual labor. Because accuracy varies so much, it’s recommended to perform benchmark testing for classification and extraction accuracy rates on the business’s actual real-world samples. Use these results to optimize project settings for each document type and field, and thereby increase accuracy and automation.

A related term with which you might be familiar, “straight through processing” (STP), is also used as a metric to describe CDA results. This is the measure of the percentage of documents run through the CDA “acquire, understand, integrate” process untouched by a human (see part 2 of the blog series for more information). The STP rate will never be higher than the lowest OCR accuracy field on the document. To maximize the STP rate, focus on the lowest-accuracy fields being extracted on the document, and adjust settings for those fields.

User efficiency

OCR accuracy is just one side of the coin of user productivity. The other side is user efficiency for exceptions. Documents and fields that don’t pass through untouched (known as “low confidence”) must be reviewed by a human to ensure they are classified and extracted correctly.

To put it simply, user efficiency is all about how quickly a user can review a low-confidence document or field, make a decision on what needs to be corrected/confirmed and then execute that decision. So the human validation interface must be designed for the most efficient use of eyes and hands during the document classification review and data validation process.

Here are some examples of user efficiency features in leading CDA solutions:

  • Jumping to the field that needs to be validated, skipping over confident fields
  • Highlighting that field on the actual image for context
  • Displaying an image snippet of the field in question next to the data entry area
  • Custom positioning of panels to each user’s liking
  • Correcting a single character in the field rather than re-entering the entire field
  • Hitting a hotkey to call a database lookup for a field
  • Auto-complete of the field based on the document type list or full page OCR
  • Completing a table’s worth of data by simply highlighting it
The effort spent on user efficiency and user experience will produce ten times the user productivity compared to the same effort spent on improving field OCR accuracy. That’s why it’s best to maximize a human’s work speed processing these data exceptions via effective user engagement and minimal keystrokes and mouse movements.

User productivity

This brings us to user productivity, which is the combination of OCR accuracy and user efficiency—and represents the single most important metric for a CDA project’s success. Basically, user productivity is the number of documents per hour/day/week/month each staff member can process with an acceptable level of data quality. For example, consider a mortgage application form. Some form fields will be more important than others, so an “acceptable level of data quality” will vary depending on the field. Benchmarking the CDA project to understand per-field OCR accuracy is necessary to optimize extraction rates for high-priority fields (such as social security number and annual income).

When configured effectively based on the success metric of maximizing user productivity, CDA solutions will deliver an attractive ROI and payback—frequently between 6 and 18 months from system launch. For a more detailed look into how to measure CDA success, including which metrics work and which don’t; how to measure cost and labor savings before and after automation; and how to create a compelling CDA business case, see the white paper “Cognitive Document Automation Success Metrics: The Truth About OCR Accuracy.

Stay Informed

Susbcribe to our RSS feed to get notified about the latest blog posts

Subscribe