SQS ReceiptHandles are not properly encoded when calling DeleteMessage

[tommy]

Calling the SQS DeleteMessage operation with the receipt handle string given in a ReceiveMessage response fails with a ReceiptHandleIsInvalid error from AWS. Looking at the error message, it appears that the library is replacing the + signs in the handle with spaces.

user=> (aws/invoke sqs {:op :ReceiveMessage :request {:QueueUrl url}})
{:Messages
 [{:MessageId "c61ea1fa-4371-435b-9af6-d867b83747a5",
   :ReceiptHandle
   "AQEBPwr8VzRpb0QmLfk6fTKH+op2XVD6TmuliAz82dYxsKncTfq2s+pPcPPlE7/ytEE1pfOe40Qw9qTZWd2sdOOlXaV8v2TcLEr35nOGlZlS6g7O1FfEKvMGMI6asvMSRfwZOvf156lpgWiYiUHerUVlK3bpnZ9ouS+lEkzLLDwTjrUsmzTxoul0cvA61B8Z1wzwCNvlRtkAMOelAGGmPBAH+HS+Rlv8FWWIap0cvcdqoy+42pVb+5fQjQXtw/Qg6n581UZw8iQlacahgkcKF/n76Qp9TdFbzCaS07sdLlRwKpGpmimKuYCxELXysUn1C4OVJFWuffQLCInuEXbTzqQxNd+CIIW1iQKr/+vu9jWHqbmj6CrzpQ/FhBHfopYKE7zbn25r6f2+9l3tRXvWbZgF04L3Ryn3Zf5APZhZDgsLUl4=",
   :MD5OfBody "136d4c574bd4e7b45bc0c19d91cbb57d",
   :Body "......"}]}

user=> (aws/invoke sqs {:op :DeleteMessage :request {:QueueUrl url :ReceiptHandle (:ReceiptHandle (first (:Messages *1)))}})

{:ErrorResponse
 {:Error
  {:Type "Sender",
   :Code "ReceiptHandleIsInvalid",
   :Message
   "The input receipt handle \"AQEBPwr8VzRpb0QmLfk6fTKH op2XVD6TmuliAz82dYxsKncTfq2s pPcPPlE7/ytEE1pfOe40Qw9qTZWd2sdOOlXaV8v2TcLEr35nOGlZlS6g7O1FfEKvMGMI6asvMSRfwZOvf156lpgWiYiUHerUVlK3bpnZ9ouS lEkzLLDwTjrUsmzTxoul0cvA61B8Z1wzwCNvlRtkAMOelAGGmPBAH HS Rlv8FWWIap0cvcdqoy 42pVb 5fQjQXtw/Qg6n581UZw8iQlacahgkcKF/n76Qp9TdFbzCaS07sdLlRwKpGpmimKuYCxELXysUn1C4OVJFWuffQLCInuEXbTzqQxNd CIIW1iQKr/ vu9jWHqbmj6CrzpQ/FhBHfopYKE7zbn25r6f2 9l3tRXvWbZgF04L3Ryn3Zf5APZhZDgsLUl4=\" is not a valid receipt handle.",
   :Detail nil},
  :RequestId "8e8e70d1-a822-5101-bb8f-245babb6f7c8"},
 :ErrorResponseAttrs
 {:xmlns "http://queue.amazonaws.com/doc/2012-11-05/"},
 :cognitect.anomalies/category :cognitect.anomalies/not-found}

(By the way, thanks for this library! I have high hopes for it!)

[dchelimsky]

I'm not sure what's going on here, but I think that's a red herring and might actually be coming from AWS:

user> (aws/invoke sqs {:op :DeleteMessage :request {:QueueUrl "a+b" :ReceiptHandle "c+d"}})
;; => {:ErrorResponse {:Error {:Code "InvalidAddress",
                         :Detail nil,
                         :Message "The address a b is not valid for this endpoint.",
                         :Type "Sender"},
                 :RequestId "be6bc4eb-23c3-57ea-a597-90616958cc5c"},
 :ErrorResponseAttrs {:xmlns "http://queue.amazonaws.com/doc/2012-11-05/"},
 :cognitect.anomalies/category :cognitect.anomalies/not-found}
user> (def m (meta *1))
;; => #'user/m
user> (-> m :http-request :body io/reader slurp)
;; => Action=DeleteMessage&Version=2012-11-05&QueueUrl=a+b&ReceiptHandle=c+d
user> (-> m :http-response :body io/reader slurp)
;; => <?xml version="1.0"?><ErrorResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><Error><Type>Sender</Type><Code>InvalidAddress</Code><Message>The address a b is not valid for this endpoint.</Message><Detail/></Error><RequestId>be6bc4eb-23c3-57ea-a597-90616958cc5c</RequestId></ErrorResponse>

You can see here that the request body has "a+b" in it, but the response body as "a b" in it.

Will keep digging.

[tommy]

Oh I see. It looks like the request body is using the content-type application/x-www-form-urlencoded, Amazon is unescaping the request parameters on their side according to url-encoding rules, where + represents a space.

With some poor-man's url-encoding (only replacing the + with %2B), I can now see the DeleteMessage request succeed:

user=> (def receipt (-> *1 :Messages first :ReceiptHandle))
"AQEBJFdXbiQ7LSt01y4R0Hat0gl6uHhniLzVXIXRoqmF+8pzCibXz356dVRBzfggzpsCF4kNNPcyOt4P8T7fy3vF+UQXlQeOABNGv3+CxGfIqaalEZK6c5OinDn8BHNKOcltDiY9gES7Tl52uDpxFGs+Bc8zle3lN+5i+otRImVHlFgss3fZC9a1tGuhGvx7/GFZFiwdsF4esGdiOKopxHPGj8gHb3eojmJPI53fFOq//E9VsUxd+LsBQ4Z/GdvwfLsYDmg+4exS2ti59V6OSpe529xE1An04WAbJNzF19p+PsU8uxxwPV/Qcpt3gFAtF8/JmkcN9IjPd1yjySSLjnSUALCJR33QQ/ZEHXZ5zeGwL6VS/w2bB2wMT+3n+nKU+yzXtSU9GlyLfzZ3yAD/wJLdd6hu4GhZdLu0v/OVXFwDygg="

user=> (aws/invoke sqs {:op :DeleteMessage :request {:QueueUrl url :ReceiptHandle receipt}})
{:ErrorResponse
 {:Error
  {:Type "Sender",
   :Code "ReceiptHandleIsInvalid",
   :Message
   "The input receipt handle \"AQEBJFdXbiQ7LSt01y4R0Hat0gl6uHhniLzVXIXRoqmF 8pzCibXz356dVRBzfggzpsCF4kNNPcyOt4P8T7fy3vF UQXlQeOABNGv3 CxGfIqaalEZK6c5OinDn8BHNKOcltDiY9gES7Tl52uDpxFGs Bc8zle3lN 5i otRImVHlFgss3fZC9a1tGuhGvx7/GFZFiwdsF4esGdiOKopxHPGj8gHb3eojmJPI53fFOq//E9VsUxd LsBQ4Z/GdvwfLsYDmg 4exS2ti59V6OSpe529xE1An04WAbJNzF19p PsU8uxxwPV/Qcpt3gFAtF8/JmkcN9IjPd1yjySSLjnSUALCJR33QQ/ZEHXZ5zeGwL6VS/w2bB2wMT 3n nKU yzXtSU9GlyLfzZ3yAD/wJLdd6hu4GhZdLu0v/OVXFwDygg=\" is not a valid receipt handle.",
   :Detail nil},
  :RequestId "5e9185bd-b92c-5248-8ede-d137cb76a3b5"},
 :ErrorResponseAttrs
 {:xmlns "http://queue.amazonaws.com/doc/2012-11-05/"},
 :cognitect.anomalies/category :cognitect.anomalies/not-found}

user=> (def escaped-receipt (clojure.string/replace receipt "+" "%2B"))
#'user/escaped-receipt

user=> (aws/invoke sqs {:op :DeleteMessage :request {:QueueUrl url :ReceiptHandle escaped-receipt}})
{:DeleteMessageResponse {:ResponseMetadata {:RequestId "2d277894-c347-56fc-b2da-2c1eb05c935f"}}, :DeleteMessageResponseAttrs {:xmlns "http://queue.amazonaws.com/doc/2012-11-05/"}}

[dchelimsky]

Good sleuthing @tommy. The fix is in, release coming shortly.

[dchelimsky]

Released in 0.8.149.

[tommy]

Verified working in 0.8.149, thanks for the quick turnaround!