Analyzing expense receipts with Azure Cognitive Services and Microsoft Flow
Recently, Business Applications MVP Steve Endow and I delivered a session at the User Group Summits in Phoenix, and in particular, to the GPUG Summit titled, "Microsoft Dynamics GP and Azure Services". In this course we detailed a number of Azure Services (among the hundreds) that could potentially be used with Microsoft Dynamics GP.
Being that I have also been working my way through completing a Microsoft sanctioned PowerApps, Flow, and CDS self-paced, online training class offered by edX (click here for more info on the course) and presented by Business Applications MVP Shane Young, I asked myself, what could I do with Microsoft Flow and Azure Services that could benefit Microsoft Dynamics GP users?
Playing with some Azure Services, I came across Azure Cognitive Services which offers the capability of reading images and analyzing them for text via its Computer Vision service. As it turns out, this service offers an optical character recognition (OCR) feature, which is capable of returning the full text or a JSON formatted document with the text. The idea here would be to use Microsoft Flow to read a newly created receipt in a OneDrive folder and transfer the file to Cognitive Services' Computer Vision for analysis, then get back the parsed text from the OCR service.
Let's see how it's done!
Provision the Computer Vision service
The first thing is to ensure Computer Vision has been enabled as a service on your Azure tenant. For this visit the Azure Portal, click on Create a Resource, then select Computer Vision from the AI + Machine Learning category within the Azure Marketplace.
Fill in some basic information like the resource name, location, pricing tier (there's a F0 free service!), and a resource group. Click the Create button when done. This will provision your Computer Vision resource.
Copy the service endpoint address and access keys
Once the service is provisioned, click on All Resources, select the Computer Vision resource you just created, then click on Overview.
Grab the service endpoint address and the access keys. You can obtain the access keys by clicking on Show access keys.. (two access keys are provided).
This is, by far, one of the easiest services to provision and requires no extra configuration, beyond establishing some access control to limit who can use the service, but that's not a topic for this discussion.
Setup a new Microsoft Flow flow
Over in Microsoft Flow, I started by setting up a blank flow and selected OneDrive's "When a file is created trigger" as this would setup the simple trigger point for when an expense receipt file is added to a Receipts folder I had previously created. You will then be prompted to setup the connection information for OneDrive.
NOTE: I selected my personal OneDrive for this, but this can also be done with a folder on your OneDrive for Business environment. In this case, you will want to authenticate with your Office 365 credentials.
Submit file to Computer Vision service
As it also turns out, Microsoft Flow has a connector to the Azure Computer Vision API, which exposes two actions: OCR to JSON and OCR to Text. Add a New Step and type Computer Vision in the search bar. Select Computer Vision API, then choose OCR to Text action.
Once again, you will be prompted for the connection information to your Computer Vision service on Azure. Enter the user account, the access key and service endpoint as gathered in step two, above.
Once credentials are entered, you can decide what to submit to Computer Vision. In this case, we what to send the File Content, which we can select from Dynamic content fields.
Configure Email step with Results
Upon completion, we want to send the resulting OCR from the analyzed image via email, so we will add another step to the flow. This time, we will a connector to Office 365 Outlook and will choose the Send an Email action for our next step.
We can then setup the properties for the Send an Email step. I have chosen to send the email to myself, and compose a subject line using the File name from the OneDrive directory. As body, I am including the Detected Text, which can be selected from the OCR to Text category under Dynamic content. I've included both the original file and content as part of the attachments.
Finally, I have given this flow a name and saved it.
Testing the Flow
I have dropped two receipt files within my OneDrive Receipts folder. These two receipts present various degrees of quality and text that can be recognized by the service. I was particularly interested in the second receipt (right) as this one was very folded and cracked so I was curious to see how it would be analyzed.
For the second receipt, the OCR service returned the JSON payload and a status 200, indicating it was successful in processing and delivering a response.
The actual email I received look like this and contained the following text:
Now, remember that my goal isn't to judge the accuracy of the OCR result delivered by Computer Vision, but rather to show how easy it is to build these kinds of solutions with Microsoft Flow and existing Azure services. Something like this would take an inordinate amount of time to build using traditional development tools and services.
Conceivably, I could create a simple PowerApps application that uses the Camera control to take the picture of the receipt and save it to my OneDrive folder. At this point, the receipt would be picked up by the Flow and analyzed by Computer Vision as we have established here. Why would this be important? Perhaps if you want to parse the JSON payload and rather submit to Microsoft Dynamics GP or Dynamics 365 as an account payables voucher, this would be useful.
Until next post,
MG.-
Mariano Gomez, MVP
Being that I have also been working my way through completing a Microsoft sanctioned PowerApps, Flow, and CDS self-paced, online training class offered by edX (click here for more info on the course) and presented by Business Applications MVP Shane Young, I asked myself, what could I do with Microsoft Flow and Azure Services that could benefit Microsoft Dynamics GP users?
Playing with some Azure Services, I came across Azure Cognitive Services which offers the capability of reading images and analyzing them for text via its Computer Vision service. As it turns out, this service offers an optical character recognition (OCR) feature, which is capable of returning the full text or a JSON formatted document with the text. The idea here would be to use Microsoft Flow to read a newly created receipt in a OneDrive folder and transfer the file to Cognitive Services' Computer Vision for analysis, then get back the parsed text from the OCR service.
Let's see how it's done!
Provision the Computer Vision service
The first thing is to ensure Computer Vision has been enabled as a service on your Azure tenant. For this visit the Azure Portal, click on Create a Resource, then select Computer Vision from the AI + Machine Learning category within the Azure Marketplace.
 |
| Computer Vision |
Fill in some basic information like the resource name, location, pricing tier (there's a F0 free service!), and a resource group. Click the Create button when done. This will provision your Computer Vision resource.
Copy the service endpoint address and access keys
Once the service is provisioned, click on All Resources, select the Computer Vision resource you just created, then click on Overview.
Grab the service endpoint address and the access keys. You can obtain the access keys by clicking on Show access keys.. (two access keys are provided).
 |
| Computer Vision service endpoint info |
This is, by far, one of the easiest services to provision and requires no extra configuration, beyond establishing some access control to limit who can use the service, but that's not a topic for this discussion.
Setup a new Microsoft Flow flow
Over in Microsoft Flow, I started by setting up a blank flow and selected OneDrive's "When a file is created trigger" as this would setup the simple trigger point for when an expense receipt file is added to a Receipts folder I had previously created. You will then be prompted to setup the connection information for OneDrive.
 |
| Blank flow with "When a file is created" trigger |
NOTE: I selected my personal OneDrive for this, but this can also be done with a folder on your OneDrive for Business environment. In this case, you will want to authenticate with your Office 365 credentials.
 |
| Receipts folder |
Submit file to Computer Vision service
As it also turns out, Microsoft Flow has a connector to the Azure Computer Vision API, which exposes two actions: OCR to JSON and OCR to Text. Add a New Step and type Computer Vision in the search bar. Select Computer Vision API, then choose OCR to Text action.
 |
| Computer Vision API connector - OCR to Text action |
Once again, you will be prompted for the connection information to your Computer Vision service on Azure. Enter the user account, the access key and service endpoint as gathered in step two, above.
 |
| Computer Vision API - credentials entry |
 |
| File Content from Dynamics content fields |
Configure Email step with Results
Upon completion, we want to send the resulting OCR from the analyzed image via email, so we will add another step to the flow. This time, we will a connector to Office 365 Outlook and will choose the Send an Email action for our next step.
 |
| Office 365 Outlook connector - Send Email action |
We can then setup the properties for the Send an Email step. I have chosen to send the email to myself, and compose a subject line using the File name from the OneDrive directory. As body, I am including the Detected Text, which can be selected from the OCR to Text category under Dynamic content. I've included both the original file and content as part of the attachments.
Finally, I have given this flow a name and saved it.
Testing the Flow
I have dropped two receipt files within my OneDrive Receipts folder. These two receipts present various degrees of quality and text that can be recognized by the service. I was particularly interested in the second receipt (right) as this one was very folded and cracked so I was curious to see how it would be analyzed.
 |
| Receipts |
 |
| JSON payload for second receipt |
The actual email I received look like this and contained the following text:
 |
| Receipt analysis |
Now, remember that my goal isn't to judge the accuracy of the OCR result delivered by Computer Vision, but rather to show how easy it is to build these kinds of solutions with Microsoft Flow and existing Azure services. Something like this would take an inordinate amount of time to build using traditional development tools and services.
Conceivably, I could create a simple PowerApps application that uses the Camera control to take the picture of the receipt and save it to my OneDrive folder. At this point, the receipt would be picked up by the Flow and analyzed by Computer Vision as we have established here. Why would this be important? Perhaps if you want to parse the JSON payload and rather submit to Microsoft Dynamics GP or Dynamics 365 as an account payables voucher, this would be useful.
Until next post,
MG.-
Mariano Gomez, MVP
Comments
I've seen others use custom built APIs like SmartDocumentor for OCR analysis and those seem a bit more sophisticated, to the point where they can breakdown all receipt lines with line item quantities and prices and also extract the full receipt total. I've found that to be lacking with Cognitive Services' OCR reader