Can't upload image with slow but stable uploading speed

[Oleksandr-Ivanchenk0]

Description:
When uploading image data (e.g., 2-5 MB) to S3 using AWSS3TransferUtility.default() at low upload speeds (e.g., less than 1 Mbps), the requests continuously restart if they are not completed within a specific time frame (~65 seconds). This behavior seams to be caused by NSURLSession background configuration, which manage retries autonomously and do not respect the timeoutIntervalForRequest setting.

Question:
Is there a way to prevent these repeated restarts when using the AWSS3? I don’t specifically need the advantages of NSURLSession background configuration; the default session configuration, which respects the timeoutIntervalForRequest setting (set to 600 seconds in my case), would be sufficient. However, it seems that AWSS3TransferUtility only supports upload functionality with background configuration.

For test case I use Network Link Conditioner with such preferences:

Result from Proxyman:

Logs:
2024-10-26 22:08:33:250 AR dev[33062:2601431] Retrieving credentials from keychain
2024-10-26 22:09:15:955 AR dev[33062:2601431] Couldn't locate the awsconfiguration.json file. Skipping load of awsconfiguration.json.
2024-10-26 22:09:15:955 AR dev[33062:2601431] Temporary dir Path is /Library/Caches
2024-10-26 22:09:15:970 AR dev[33062:2601431] Transfer Utility Database Path: [/Library/Caches/com/amazonaws/AWSS3TransferUtility/transfer_utility_database]
2024-10-26 22:09:15:972 AR dev[33062:2601431] In Recovery for TU Session [com.amazonaws.AWSS3TransferUtility.Default.Identifier]
2024-10-26 21:53:25:000 AR dev[30727:2580314] Value of timeoutIntervalForResource is 3000
2024-10-26 21:53:25:000 AR dev[30727:2580314] Using virtual-hosted-style access because bucket is compliant
2024-10-26 21:53:25:000 AR dev[30727:2580314] Using virtual-hosted-style access because bucket is compliant
2024-10-26 21:53:25:001 AR dev[30727:2580314] AWSS4 PresignedURL Canonical request: [PUT
/image_78E51AC9-C599-4E77-A47D-4A46FF91D99F.jpg
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA3FID7JKJ2JG3FJHM%2F20241026%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20241026T185325Z&X-Amz-Expires=2999&X-Amz-Security-Token=***
content-type:image/jpeg
host:.s3.us-west-2.amazonaws.com
x-amz-acl:public-read
content-type;host;x-amz-acl
UNSIGNED-PAYLOAD]
2024-10-26 21:53:25:001 AR dev[30727:2580314] AWS4 PresignedURL String to Sign: [AWS4-HMAC-SHA256
20241026T185325Z
20241026/us-west-2/s3/aws4_request
]
2024-10-26 21:53:25:001 AR dev[30727:2580314] AWS4 PresignedURL: [https://.s3.us-west-2.amazonaws.com/image_78E51AC9-C599-4E77-A47D-4A46FF91D99F.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=&X-Amz-Date=20241026T185325Z&X-Amz-Expires=2999&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl&X-Amz-Security-Token=***]
2024-10-26 21:53:25:001 AR dev[30727:2580314] Request headers:
{
"Content-Type" = "image/jpeg";
"User-Agent" = "aws-sdk-iOS/2.37.2 iPadOS/17.2 en_US@rg=uazzzz transfer-utility";
"x-amz-acl" = "public-read";
}
2024-10-26 21:53:25:003 AR dev[30727:2580314] Setting taskIdentifier to 2
2024-10-26 21:53:25:031 AR dev[30727:2584660] didSendBodyData called for task 2
2024-10-26 21:54:31:373 AR dev[30727:2585458] didSendBodyData called for task 2
2024-10-26 21:55:39:063 AR dev[30727:2586294] didSendBodyData called for task 2
2024-10-26 21:56:48:947 AR dev[30727:2586977] didSendBodyData called for task 2
2024-10-26 21:58:03:186 AR dev[30727:2585677] didSendBodyData called for task 2
2024-10-26 21:59:25:613 AR dev[30727:2585677] didSendBodyData called for task 2
2024-10-26 22:01:04:029 AR dev[30727:2586172] didSendBodyData called for task 2
2024-10-26 22:02:23:943 AR dev[30727:2591152] didSendBodyData called for task 2

AWS Services: AWSS3

AWSDDLog.sharedInstance.logLevel = .verbose
AWSDDLog.add(AWSDDTTYLogger.sharedInstance!)

let credentialsProvider = AWSCognitoCredentialsProvider(regionType: .USWest2, identityPoolId: EnvironmentVariables.AmazonS3.poolId)
let configuration = AWSServiceConfiguration(region: .USWest2, credentialsProvider: credentialsProvider)
        configuration?.timeoutIntervalForRequest = 600
AWSServiceManager.default().defaultServiceConfiguration = configuration
        
func testUpload() {
        let imageSize = CGSize(width: 10000, height: 10000)
        UIGraphicsBeginImageContext(imageSize)
        UIColor.red.setFill()
        UIRectFill(CGRect(origin: .zero, size: imageSize))
        let image = UIGraphicsGetImageFromCurrentImageContext()!
        UIGraphicsEndImageContext()
        
        let data = image.jpegData(compressionQuality: 1)!
        let key = "image_\(UUID().uuidString).jpg"
        let bucket = EnvironmentVariables.AmazonS3.bucket
        
        let expression = AWSS3TransferUtilityUploadExpression()
        expression.setValue("public-read", forRequestHeader: "x-amz-acl")
        AWSS3TransferUtility.default().uploadData(data,
                                                  bucket: bucket,
                                                  key: key,
                                                  contentType: "image/jpeg",
                                                  expression: expression,
                                                  completionHandler: nil)
}

Environment:

• SDK Version: 2.37.2
• Dependency Manager: Cocoapods
• Swift Version : 5

Device Information:

• Device: iPad Pro, Simulator
• iOS Version: iOS 17.2

[edisooon]

Hi @Oleksandr-Ivanchenk0 Thanks for submitting the issue! One of our team members will investigate and provide updates here.

[ruisebas]

There is no way to configure the session configuration used by the TransferUtility, which is indeed a background session due to the use cases we need to support (e.g. uploading/downloading in the background).

[Oleksandr-Ivanchenk0]

Got it, thank you for the clarification. Do you have any recommendations on how to mitigate the issue with repeated retries in this scenario?

[ruisebas]

These results may or may not be caused by how the Network Link Conditioner simulates the conditions. According to the official documentation, timeoutIntervalForRequest sets

how long (in seconds) a task should wait for additional data to arrive before giving up

i.e. it's not that the request will timeout if it's not completed in that time, but that it will timeout if no data is received at all for that amount of seconds. Accepting 600 seconds of no data sounds problematic.

I'd expect iOS to still honour the value of timeoutIntervalForRequest in real-life conditions, even when using a background session. The documentation does state that these requests will be automatically retried on failure until timeoutIntervalForResource is reached, but it doesn't clarify whether timeoutIntervalForRequest is ignored or not.

Having said that, the AWSS3TransferUtility relies in background sessions and we're bound by how iOS handles them.

---

If you really want to avoid using background sessions, then you will need to handle the upload yourself.
You could create your own class that wraps the AWSS3TransferUtility and define your own upload method that calls the transfer utility, grabs its AWSS3TransferUtilityUploadTask.request and retries it using a normal session.

This is a simple implementation that might do the trick for you and keeps the same signature as the Transfer Utility:

class TransferUtilityWrapper: NSObject, URLSessionTaskDelegate {
    private var currentUploadTask: AWSS3TransferUtilityUploadTask? = nil // For progress reporting
    private var currentUploadExpression: AWSS3TransferUtilityUploadExpression? = nil // For completion reporting

    func uploadData(
        _ data: Data,
        bucket: String,
        key: String,
        contentType: String,
        expression: AWSS3TransferUtilityUploadExpression? = nil,
        completionHandler: AWSS3TransferUtilityUploadCompletionHandlerBlock? = nil
    ) {
        let uploadExpression = AWSS3TransferUtilityUploadExpression()
        uploadExpression.internalRequestHeaders = expression?.internalRequestHeaders
        uploadExpression.internalRequestParameters = expression?.internalRequestParameters
        AWSS3TransferUtility.default().uploadData(
            data,
            bucket: bucket,
            key: key,
            contentType: contentType,
            expression: uploadExpression,
            completionHandler: nil
        ).continueWith { [weak self] task in
            guard let self,
                  let uploadTask = task.result,
                  let request = uploadTask.request else {
                return task
            }
            // Cancel the original session task
            uploadTask.sessionTask.cancel()

            // Save the expression/completion handler and the upload task
            currentUploadExpression = expression ?? .init()
            currentUploadExpression?.completionHandler = completionHandler ?? currentUploadExpression?.completionHandler
            currentUploadTask = uploadTask

            // Create a URL session with the desired configuration
            let configuration = URLSessionConfiguration.default
            configuration.timeoutIntervalForRequest = 600
            let session = URLSession(
                configuration: configuration,
                delegate: self,
                delegateQueue: nil
            )

            // Upload the data reusing the request
            let sessionUploadTask = session.uploadTask(with: request, from: data)
            sessionUploadTask.resume()

            return task
        }
    }

    func urlSession(_ session: URLSession, task: URLSessionTask, didSendBodyData bytesSent: Int64, totalBytesSent: Int64, totalBytesExpectedToSend: Int64) {
        if let currentUploadTask, let progressBlock = currentUploadExpression?.progressBlock {
            currentUploadTask.progress.totalUnitCount = totalBytesExpectedToSend
            currentUploadTask.progress.completedUnitCount = totalBytesSent
            progressBlock(currentUploadTask, currentUploadTask.progress)
        }
    }

    func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: (any Error)?) {
        if let currentUploadTask, let completionHandler = currentUploadExpression?.completionHandler {
            completionHandler(currentUploadTask, error)
        }
    }
}

Please note that this is just a simple example on how you might achieve what you want and still reuse some of the convenience that the Transfer Utility provides (e.g. generating the presigned URL), but it does not replicate everything the Transfer Utility does (e.g. it can only handle one upload at a time), nor is something officially supported.

[Oleksandr-Ivanchenk0]

@ruisebas Thank you for the detailed response and example implementation.

This issue actually originated from real app users who have slow upload speeds. In my research, I found references to similar issues in this thread and this thread, in which DTS Engineer mentioned that this behavior might be tied to how iOS handles background sessions and treats parameters like timeoutIntervalForRequest.

To verify, I conducted a test with same setup as I described in first message + modifying AWSS3TransferUtility.m, changing the line:
NSURLSessionConfiguration *configuration = [NSURLSessionConfiguration backgroundSessionConfigurationWithIdentifier:_sessionIdentifier];
to:
NSURLSessionConfiguration *configuration = [NSURLSessionConfiguration defaultSessionConfiguration];

With this change, I achieved the expected behavior of a single, long-running request without retries.

I also tried workaround you provided and observed similar results.

For now, I’ll proceed with your suggested approach. Thank you once again for the guidance.

[ruisebas]

Glad to hear. Let us know if you have any other questions.

Thanks!

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.