Amplify Predictions identify not returning form(keyValue) and table data #641

vdvpie22 · 2020-07-14T06:43:53Z

Describe the bug
We are using amplify for our app development and one of the prediction API call is not working properly. It is working okay for Android but not for iOS.
https://docs.amplify.aws/lib/predictions/identify-text/q/platform/ios#identify-text-in-a-document
The API is not returning forms and table data in result?
We tried detectText(.form), detectText(.table), detectText(.all). Results are same for all of them.

To Reproduce
Steps to reproduce the behavior:

func detectText(_ image: URL, result:@escaping FlutterResult) {
        _ = Amplify.Predictions.identify(type: .detectText(.all), image: image, options: PredictionsIdentifyRequest.Options(), listener: { (event) in
               switch event {
               case .success(let resultIdentity):
                   let data = resultIdentity as! IdentifyDocumentTextResult
                   print(data.fullText)
                   result(data.fullText)
               case .failure(let error):
                   print(error)
                   result(error)
               }
           })
    }

data variable contains

Expected behavior
tables and keyValues should contain data as it work in Android.

Environment(please complete the following information):

Amplify Framework Version: 1.0.2
Dependency Manager: Cocoapods
Swift Version : 5.0

Device Information (please complete the following information):

Device: iPhone8 plus, Simulator
iOS Version: iOS 13
Specific to simulators:

Additional context
No any other context about the problem.

The text was updated successfully, but these errors were encountered:

wooj2 · 2020-07-14T17:09:35Z

Hi @vdvpie22,

Thanks for reaching out. It sounds like you are reporting a difference in the iOS version vs what Android offers. Can you please explain what you are seeing on Android, and how that is different on iOS? Specifically, how does the use case work for you on Android? How does it not work for you on iOS?

Thank you

vdvpie22 · 2020-07-15T04:37:31Z

Response with same document:
in Android (as expected):

{
    "fullText": "INVOICE......",
    "keyValues": [
        {
            "box": {
                "bottom": 0.33146378,
                "left": 0.7483453,
                "right": 0.8464649,
                "top": 0.32305497
            },
            "polygon": {
                "points": [
                    {
                        "x": 0.7483453,
                        "y": 0.32305497
                    },
                    {
                        "x": 0.8464648,
                        "y": 0.32305497
                    },
                    {
                        "x": 0.8464648,
                        "y": 0.33146378
                    },
                    {
                        "x": 0.7483453,
                        "y": 0.33146378
                    }
                ]
            },
            "confidence": 85.76523,
            "id": "64d01edb-0dce-4722-8d09-b2ed749dd0c1",
            "value": {
                "first": "DESPATCH NOTE",
                "second": "S538796"
            }
        },
       ......
    ],
    "lines": [
        {
            "page": 0,
            "box": {
                "bottom": 0.087794684,
                "left": 0.86288506,
                "right": 0.9639127,
                "top": 0.072275944
            },
            "polygon": {
                "points": [
                    {
                        "x": 0.86288506,
                        "y": 0.072275944
                    },
                    {
                        "x": 0.9639127,
                        "y": 0.072275944
                    },
                    {
                        "x": 0.9639127,
                        "y": 0.087794684
                    },
                    {
                        "x": 0.86288506,
                        "y": 0.087794684
                    }
                ]
            },
            "confidence": 99.8893,
            "id": "2a7531db-9649-408e-88cb-41b7f61d64eb",
            "value": "INVOICE"
        },
       ......
    ],
    "rawLineText": [
        "INVOICE",
      .....
    ],
    "selections": [],
    "tables": [
        {
            "columnSize": 7,
            "rowSize": 11,
            "box": {
                "bottom": 0.712523,
                "left": 0.0673983,
                "right": 0.9643701,
                "top": 0.36833143
            },
            "polygon": {
                "points": [
                    {
                        "x": 0.0673983,
                        "y": 0.36833143
                    },
                    {
                        "x": 0.9643701,
                        "y": 0.36833143
                    },
                    {
                        "x": 0.9643701,
                        "y": 0.7125229
                    },
                    {
                        "x": 0.0673983,
                        "y": 0.7125229
                    }
                ]
            },
            "confidence": 99.95759,
            "id": "2f80525c-3328-48f5-892e-a9cc2c956dac",
            "value": [
                {
                    "column": 0,
                    "row": 0,
                    "selected": false,
                    "box": {
                        "bottom": 0.38292885,
                        "left": 0.0673983,
                        "right": 0.14748508,
                        "top": 0.36833143
                    },
                    "polygon": {
                        "points": [
                            {
                                "x": 0.0673983,
                                "y": 0.36833143
                            },
                            {
                                "x": 0.14748506,
                                "y": 0.36833143
                            },
                            {
                                "x": 0.14748506,
                                "y": 0.38292885
                            },
                            {
                                "x": 0.0673983,
                                "y": 0.38292885
                            }
                        ]
                    },
                    "confidence": 99.95759,
                    "id": "4ff87dd2-0187-4232-a929-3fd86bc00969",
                    "value": "PRODUCT CODE"
                },
                ....
            ]
        },
        {
            "columnSize": 7,
            "rowSize": 2,
            "box": {
                "bottom": 0.36267745,
                "left": 0.045876656,
                "right": 0.94214964,
                "top": 0.31708455
            },
            "polygon": {
                "points": [
                    {
                        "x": 0.045876656,
                        "y": 0.31708455
                    },
                    {
                        "x": 0.94214964,
                        "y": 0.31708455
                    },
                    {
                        "x": 0.94214964,
                        "y": 0.36267748
                    },
                    {
                        "x": 0.045876656,
                        "y": 0.36267748
                    }
                ]
            },
            "confidence": 99.65584,
            "id": "3de1a322-46c5-4b88-9fdf-005b3d73e4eb",
            "value": [
                {
                    "column": 0,
                    "row": 0,
                    "selected": false,
                    "box": {
                        "bottom": 0.33703145,
                        "left": 0.045876656,
                        "right": 0.1599114,
                        "top": 0.31708455
                    },
                    "polygon": {
                        "points": [
                            {
                                "x": 0.045876656,
                                "y": 0.31708455
                            },
                            {
                                "x": 0.1599114,
                                "y": 0.31708455
                            },
                            {
                                "x": 0.1599114,
                                "y": 0.33703145
                            },
                            {
                                "x": 0.045876656,
                                "y": 0.33703145
                            }
                        ]
                    },
                    "confidence": 99.65584,
                    "id": "7566fd5c-0d2c-4cb3-b82e-d18ae102082c",
                    "value": "INVOICE No"
                },
               ....
            ]
        }
    ],
    "words": [
        {
            "page": 0,
            "box": {
                "bottom": 0.087794684,
                "left": 0.86288506,
                "right": 0.9639127,
                "top": 0.072275944
            },
            "polygon": {
                "points": [
                    {
                        "x": 0.86288506,
                        "y": 0.072275944
                    },
                    {
                        "x": 0.9639127,
                        "y": 0.072275944
                    },
                    {
                        "x": 0.9639127,
                        "y": 0.087794684
                    },
                    {
                        "x": 0.86288506,
                        "y": 0.087794684
                    }
                ]
            },
            "confidence": 99.8893,
            "id": "fcdda7c6-7f69-4d37-9cce-0ad348aa3075",
            "value": "INVOICE"
        },
       .....
    ]
}

In IOS:

{
    "fullText": "INVOICE......",
    "keyValues": [],
    "lines": [
        {
            "page": 0,
            "box": {
                "bottom": 0.087794684,
                "left": 0.86288506,
                "right": 0.9639127,
                "top": 0.072275944
            },
            "polygon": {
                "points": [
                    {
                        "x": 0.86288506,
                        "y": 0.072275944
                    },
                    {
                        "x": 0.9639127,
                        "y": 0.072275944
                    },
                    {
                        "x": 0.9639127,
                        "y": 0.087794684
                    },
                    {
                        "x": 0.86288506,
                        "y": 0.087794684
                    }
                ]
            },
            "confidence": 99.8893,
            "id": "2a7531db-9649-408e-88cb-41b7f61d64eb",
            "value": "INVOICE"
        },
       ......
    ],
    "rawLineText": [
        "INVOICE",
      .....
    ],
    "selections": [],
    "tables": [],
    "words": [
        {
            "page": 0,
            "box": {
                "bottom": 0.087794684,
                "left": 0.86288506,
                "right": 0.9639127,
                "top": 0.072275944
            },
            "polygon": {
                "points": [
                    {
                        "x": 0.86288506,
                        "y": 0.072275944
                    },
                    {
                        "x": 0.9639127,
                        "y": 0.072275944
                    },
                    {
                        "x": 0.9639127,
                        "y": 0.087794684
                    },
                    {
                        "x": 0.86288506,
                        "y": 0.087794684
                    }
                ]
            },
            "confidence": 99.8893,
            "id": "fcdda7c6-7f69-4d37-9cce-0ad348aa3075",
            "value": "INVOICE"
        },
       .....
    ]
}

That is, the IOS version does not return keyValue and tables as indicated in the first post

wooj2 · 2020-07-16T23:39:17Z

@vdvpie22

Thank you so much for the detailed report! Having real data made it really clear for me to dive into this.

This looks to be a bug in our code.

Specifically, our code which is parsing the response from Textract:
https://github.com/aws-amplify/amplify-ios/blob/eb5b7ff98ae9ac090dc8c71e374b8f3bcb0ee472/AmplifyPlugins/Predictions/AWSPredictionsPlugin/Support/Utils/IdentifyTextResultTransformers.swift#L69

If an element is of type .table or .keyValueSet, it may not have a value for block.text, and hence if we are parsing a .table or .keyValueSet, then we'll continue on line 70, thereby skipping that block! :(

Just to be clear about repro steps:

Setup amplify predictions w/ identify enabled
Use the following code ( or something similar)

    override func viewDidLoad() {
        super.viewDidLoad()
        let url = URL(string: "https://raw.githubusercontent.com/aws-samples/amazon-textract-code-samples/master/src-csharp/test-files/employmentapp.png")
        if let imageURL = url {
            detectText(imageURL)
        }
        
    }

    func detectText(_ image: URL) {
        _ = Amplify.Predictions.identify(type: .detectText(.all), image: image) { event in
            switch event {
            case let .success(result):
                let data = result as! IdentifyDocumentTextResult
                print(data.keyValues) // Should have values
                print(data.tables) //should also have values
            case let .failure(error):
                print(error)
            }
        }
    }

Expected behavior
The above code should print out values for data.keyValues and data.tables
For reference, here is how the android code is processing the blocks:
https://github.com/aws-amplify/amplify-android/blob/71aa76d145812170d2f9d8e3110ef907427e08ef/aws-predictions/src/main/java/com/amplifyframework/predictions/aws/service/AWSTextractService.java#L132

pavelrojkovtrepp · 2020-07-27T14:00:19Z

Hello guys.

I think I am hitting the same issue. It this fix planned for next release? It still say 1.0.5 as a milestone but it is not there.

Thank you.

ruiguoamz · 2020-07-27T15:35:23Z

Hi, @pavelrojkovtrepp. Sorry that this bug takes longer than I expected. I need more time solving this issue. And since I am already working on it. It shouldn't take long to release the fix.

vdvpie22 · 2020-08-03T05:58:00Z

Hi, any updates?

ruiguoamz · 2020-08-03T16:10:44Z

Reference to the PR which is in progress. #660

wooj2 · 2020-08-12T22:26:10Z

@vdvpie22 sorry for the delay, this fix has been released as part of Amplify 1.1.0

wooj2 added predictions Issues related to the Predictions category question General question pending-triage Issue is pending triage and removed question General question labels Jul 14, 2020

wooj2 self-assigned this Jul 16, 2020

wooj2 added bug Something isn't working and removed pending-triage Issue is pending triage labels Jul 16, 2020

drochetti added this to the 1.0.5 milestone Jul 17, 2020

wooj2 assigned ruiguoamz and unassigned wooj2 and ruiguoamz Jul 17, 2020

ruiguoamz mentioned this issue Jul 22, 2020

fix(predictions): TABLE, CELL & KEY_VALUE_SET blocks are not properly processed #660

Merged

ruiguoamz removed this from the 1.0.5 milestone Jul 27, 2020

drochetti mentioned this issue Jul 30, 2020

AWS amplify ios (awsTextract Unable to get data in table form) #679

Closed

palpatim closed this as completed Aug 12, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Amplify Predictions identify not returning form(keyValue) and table data #641

Amplify Predictions identify not returning form(keyValue) and table data #641

vdvpie22 commented Jul 14, 2020 •

edited

Loading

wooj2 commented Jul 14, 2020

vdvpie22 commented Jul 15, 2020 •

edited by raphkim

Loading

wooj2 commented Jul 16, 2020

pavelrojkovtrepp commented Jul 27, 2020

ruiguoamz commented Jul 27, 2020

vdvpie22 commented Aug 3, 2020

ruiguoamz commented Aug 3, 2020

wooj2 commented Aug 12, 2020

Amplify Predictions identify not returning form(keyValue) and table data #641

Amplify Predictions identify not returning form(keyValue) and table data #641

Comments

vdvpie22 commented Jul 14, 2020 • edited Loading

wooj2 commented Jul 14, 2020

vdvpie22 commented Jul 15, 2020 • edited by raphkim Loading

wooj2 commented Jul 16, 2020

pavelrojkovtrepp commented Jul 27, 2020

ruiguoamz commented Jul 27, 2020

vdvpie22 commented Aug 3, 2020

ruiguoamz commented Aug 3, 2020

wooj2 commented Aug 12, 2020

vdvpie22 commented Jul 14, 2020 •

edited

Loading

vdvpie22 commented Jul 15, 2020 •

edited by raphkim

Loading