SmartCrop: You have provided a FaceIndex value that exceeds the length of the zero-based detectedFaces array.

[joebernard]

I'm attempting to use the smartCrop option:

edits: {
  smartCrop : true
}

This error is logged in CloudWatch:

{ status: 400,
code: 'SmartCrop::FaceIndexOutOfRange',
message: 'You have provided a FaceIndex value that exceeds the length of the zero-based detectedFaces array. Please specify a value that is in-range.' }

It is failing because of this line. It doesn't first check if any faces were detected before it starts defining the bounding box. In my opinion, smartCrop should fail gracefully (not crop) if no faces were detected in the image. Would you accept a PR for this, or provide some advisement on how best to implement it?

[soupy1976]

@joebernard I came across this. I tried debugging it by breaking down the code and doing some console.logs(). To my surprise, when I did that I found that it did actually manage to log detectedFaces array with one entry in it, and also it managed to log the bounding box. So I think this may be more complicated. It would be interesting if you could try the same experiment and see if you get the same thing.
This is how I was logging:

console.log('faceIdx = ' + faceIdx);  
	const req = (await request);
	console.log(req);           
	console.log(req.FaceDetails[faceIdx]);
	var response = req.FaceDetails[faceIdx].BoundingBox;
	console.log(response);
	return Promise.resolve(await response);  

and this was the output of the console.log calls:
console.log(req);

{ FaceDetails: 
[ { BoundingBox: [Object],
Landmarks: [Array],
Pose: [Object],
Quality: [Object],
Confidence: 99.9921646118164 } ] }

console.log(req.FaceDetails[faceIdx]);

2019-07-19T14:59:09.751Z	1ee4d629-bdf0-48db-8c2e-6e1855135cfc	{ BoundingBox: 
{ Width: 0.08595786243677139,
Height: 0.08245693892240524,
Left: 0.3658732771873474,
Top: 0.252895712852478 },
Landmarks: 
[ { Type: 'eyeLeft',
X: 0.3878157138824463,
Y: 0.29333987832069397 },
{ Type: 'eyeRight',
X: 0.4070296287536621,
Y: 0.2882431745529175 },
{ Type: 'mouthLeft',
X: 0.4075484573841095,
Y: 0.31989386677742004 },
{ Type: 'mouthRight',
X: 0.42255866527557373,
Y: 0.31640052795410156 },
{ Type: 'nose', X: 0.3886948525905609, Y: 0.3095017373561859 } ],
Pose: 
{ Roll: -26.110822677612305,
Yaw: -56.12600326538086,
Pitch: -0.6558781266212463 },
Quality: { Brightness: 84.3766860961914, Sharpness: 46.02980041503906 },
Confidence: 99.9921646118164 }

and then the error:
2019-07-19T13:58:35.184Z 18b4b12c-703c-4d7f-b8fd-7067acaa10de TypeError: Cannot read property 'BoundingBox' of undefined

I'm not really a node developer, so it might be I was not understanding how promises and await works. This behaviour seemed very strange to me though.

Unfortunately at this point I had to forget about using SmartCrop and move on, as the lambda environment upgrade had broken my live site, so I just had to get things working as fast as possible.

[ecaron]

Another important thing to know for the change between v3 & v4 is that v3 (using thumbor rekognition - https://github.com/yu-liang-kono/thumbor_rekognition/blob/master/thumbor_rekognition/__init__.py#L32) detected multiple faces and then adjusted the focus to include them all. v4, on the other hand, just focuses on a single face (defaulting to the first one, unless specified)

[crpahl]

Any updates, or possible solutions to this? We recently updated to v4 as well and relying on smart cropping was very important to us since we used it to automatically crop user's profile pictures. We're getting the exact same error on photos that have have zero faces and it's required us to disable smart cropping for now. This feature is extremely important to us and we'd like to get it resolved soon if possible. Thank you!

[hayesry]

@crpahl Apologies for the inconvenience, I'm looking into the issue right now. For clarification, are you looking for the original image to be returned when zero faces are detected with SmartCrop?

[crpahl]

@hayesry No problem and thank you for the quick response!

Yeah exactly that. We're just looking for the original image to be returned. In our case, our users will occasionally upload profile pictures that have no faces in them (such as company logos).

[hayesry]

@crpahl Gotcha! I've added this to our feature request board and will try to have it published in the next update. Really appreciate the feedback!

[ecaron]

FWIW, I'm working on a PR that brings in https://github.com/jwagner/smartcrop.js (MIT licensed, no dependencies) and uses that plus Rekognition for some really nice results. I hope to get that submitted this weekend. I'm using it in production now, and I'm very happy with the quality of the results.

[crpahl]

@hayesry Sorry to ping you on this again. Do you have a rough idea of when this will be published? We're looking into addressing this ourselves as it has a pretty high customer impact for us, but if you think it might be addressed in the near future, we can wait slightly longer.

[SilverFoxA]

FWIW, I'm working on a PR that brings in https://github.com/jwagner/smartcrop.js (MIT licensed, no dependencies) and uses that plus Rekognition for some really nice results. I hope to get that submitted this weekend. I'm using it in production now, and I'm very happy with the quality of the results.

@ecaron Hello Eric, It would be really helpful if you could kindly push out the pull requests. Thank you

[joebernard]

Any update on the smartcrop PR?

[leofidus]

The current state on the fork from @ecaron seems to work well at least in our testing

[ecaron]

Sorry, I've been delaying submitting the change for a PR because I haven't reviewed the license tree of the dependency & I haven't written tests yet. Since there's so much interest and this ticket is getting some age on it, I'll put time to this tonight.

[leofidus]

That sounds great. One detail we have changed is to force the image to be PNG (.png() in applyEdits()). Without this webp images don't work because either smartcrop or recognition don't accept webp.

[leofidus]

Another issue that's worth pointing out with ecaron's excellent code is that AWS Recognition has a 5MB upload limit which a large image might hit. It's fairly straightforward to solve for nearly all cases by rescaling what you send to Recognition (it returns relative coordinates anyways).

[joebernard]

smartCrop is still breaking as of version 5 of serverless image handler. It breaks whether smartCrop is set to true or an object as specified in the docs. It breaks if the image actually has a face to detect or not. I can't get it working under any conditions.

From CloudWatch:

ImageRequest {
  requestType: 'Default',
  bucket: 'my-bucket',
  key: 'my-image.jpg',
  edits: {
    resize: { width: 414, height: 414, fit: 'cover', strategy: 'attention' },
    smartCrop: true
  },
  ContentType: 'image/jpeg',
  LastModified: 'Sun, 18 Oct 2020 17:14:07 GMT',
  CacheControl: 'max-age=31536000,public',
  originalImage: <Buffer ff d8 ff e0 00 10 4a 46 49 46 00 01 01 00 00 48 00 48 00 00 ff e1 00 58 45 78 69 66 00 00 4d 4d 00 2a 00 00 00 08 00 02 01 12 00 03 00 00 00 01 00 01 ... 1043242 more bytes>
}

TypeError: Cannot read property 'BoundingBox' of undefined
    at ImageHandler.getBoundingBox (/var/task/image-handler.js:220:50)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async ImageHandler.applyEdits (/var/task/image-handler.js:110:37)
    at async ImageHandler.process (/var/task/image-handler.js:27:35)
    at async Runtime.exports.handler (/var/task/index.js:24:34)

Edit:

There are conditions where it does work:
The image must have at least one face to detect. If the image does not have a face or Rekognition does not detect a face, it will break with an error.
The parameters passed to the handler must not include resize with height or width specified.

These two cases should be handled by the lambda. If no face is detected, allow the handler to continue as if smartCrop is disabled. If resize is passed, use the height and width properties to size the image while cropping a face.

[beomseoklee]

@joebernard Sorry for the late response and inactive engagement for this issue. This issue is in our backlog, but I'll take a look at this one again.

[vonec]

can we expect this to be fixed in the next release, waiting for this since July. Also I hope the images will be cached after processing as calls to rekognition are chargeable.

[joebernard]

Hi @beomseoklee, has there been any progress on this?

[G-Lenz]

We have fixed these issues with smart cropping in v5.2.0.

[joebernard]

I upgraded to v5.2.0. It returns a 400 error with these settings:

smartCrop: {
  faceIndex: 0,
  padding: 1
}

{"status":400,"code":"SmartCrop::PaddingOutOfBounds","message":"The padding value you provided exceeds the boundaries of the original image. Please try choosing a smaller value or applying padding via Sharp for greater specificity."}

No matter what padding value I try it returns the same error. If no padding is passed it does return a face very closely cropped and not really usable.

[joebernard]

Has anyone been able to get padding working with smartCrop? It seems to only work on pictures that contain a face. If the pic does not contain a face, it throws the error above.

@G-Lenz Should I open a separate issue for this?

[G-Lenz]

Hi @joebernard you can feel free to open a new issue, I have also added the issue to our backlog. Thank you for bringing this to our attention.

[joebernard]

Closing this as the faceIndex issue is resolved in 5.2.0.

I opened #263 regarding the padding issue.