How to implement background uploads and offline support correctly

[hipwelljo]

State your question
I'm looking through the documentation on how to utilize the AWSS3TransferUtility to upload files and support background uploads for an iOS app, and resume them should the app be terminated and relaunched. It appears the documentation is incorrect when it comes to Background Transfers information. It states the code is suspending a transfer when the app is about to be terminated, but that's not what it does, the code suspends it in the contiueWith block - right after it started the upload. The next section, Manage a Transfer when a Suspended App Returns to the Foreground, states "When an app that has initiated a transfer becomes suspended and then returns to the foreground, the transfer may still be in progress or may have completed. In both cases, use the following code to reestablish the progress and completion handler blocks of the app" but it doesn't explain where this code should go - in the AppDelegate willEnterForeground, didBecomeActive, didFinishLaunching, elsewhere?

Trying to research how to do this the correct way I'm finding conflicting information. From what I've gathered so far reading in GitHub issues (for example, here), you don't actually need to reassign completion blocks unless the app is terminated, and you need to do it a different way not using enumerateToAssignBlocks, but again it's not explained where to put that code. We also don't need to call interceptApplication anymore? So it appears almost all of the documentation is out-of-date.

I would also like to understand what offline mode support there is. It sounds like it may be in place now, but maybe with only the multi-part uploads.

Unfortunately the sample project doesn't implement the background transfer support or anything except a basic upload, so I haven't found an example project that implements this functionality.

In short, I'm finding no information out there on the right way to piece together these APIs with the latest versions of the SDK.

Can you please update the documentation, and ideally update the sample projects? And if you can provide correct and up-to-date information here in the mean-time that would be great! Surely many others will find this useful. Thanks!

Which AWS Services are you utilizing?
AWSS3

Provide code snippets (if applicable)

Environment(please complete the following information):

• SDK Version: 2.6.29
• Dependency Manager: Cocoapods
• Swift Version : 4.2

Device Information (please complete the following information):

• Device: iPhone X
• iOS Version: iOS 12

[scb01]

@hipwelljo

Sorry to hear that you are running into difficulties using the SDK. Let me take a crack at answering your questions here

• The TransferUtility uses a NSURL Background session for uploading and downloading files. All the transfers, therefore, happen in the background.

• The app that initiated the transfers can be in the foreground or in the background at various points during the transfer. To ensure that the app is woken up in the background when a transfer is completed, you need to implement this method in the AppDelegate. This will ensure that the transfers continue to run when your app is in the background.

   func application(_ application: UIApplication, 
            handleEventsForBackgroundURLSession identifier: String, 
            completionHandler: @escaping () -> Void) {
  
         //provide the completionHandler to the TransferUtility to 
         //support background transfers.
        AWSS3TransferUtility.interceptApplication(application, 
           handleEventsForBackgroundURLSession: identifier, 
          completionHandler: completionHandler)
    }
  

• There is no need to pause the transfers when the app is transitioning to the background (and to resume when the app transitions back to the foreground). The documentation is incorrect and I will be fixing it soon.

• There is no need to use the enumerateToAssignBlocks method when the app moves from background to foreground. The ones that you setup before the app went to the background will be in place and will work fine.

• I am not sure what you mean by "offline" support. Can you clarify - the transfers need a network connection and will not proceed if the device is offline

• I have updated the sample app and tweaked the README. Please take a look at the latest and see if that works for you. Also, I have a task that I am working on to update the documentation. I will post back on this thread once that is complete.

The sample does not cover what needs to be done if the app were to be restarted. I will address that in the documentation as well, but here is a short summary of what needs to be done

• When the app is restarted, you can instantiate the TransferUtility using the AWSS3TransferUtility.s3TransferUtility(forKey: "") method. The TransferUtility uses the key to identify the NSURLSession and it is important that the key be the same across app restarts. The TransferUtility will automatically attempt to reconnect to the transfers that were in progress the last time the app was running. You can do this anywhere in the app - my preference is to do it in the appDidFinishLaunching method

• You can get the transfers and connect them to progressBlocks and completionHandlers using the following code (I am only showing this for uploads - they work in a similar fashion for downloads and multipart uploads)
  let uploadTasks = transferUtility.getUploadTasks().result

  for task in uploadTasks! {
       task.setCompletionHandler(completionHandler!)
       task.setProgressBlock(progressBlock!)
   }
  

Hopefully this gets you unblocked. I will post back in this thread once the documentation updates have been made.

[hipwelljo]

Thank you so much @cbommas! I will dive deeper into this on Monday, but wanted to answer your question.

In regards to offline mode support, I was wondering if the transfer utility will automatically stop uploading once the internet connection is lost, and automatically start uploading again once it is restored. If you try to start an upload while offline, does it successfully upload eventually if you later return to the app with an internet connection, or would it just fail immediately? I saw there's a retryLimit you can configure and I've set that to 5, but wanted to make sure it's not just going to continually try and fail 5 times in rapid succession if it's attempted to upload while offline. It sounded like it does have support for this functionality from what I've seen outside the documentation.

Then in regards to calling interceptApplication, you noted a bit ago that this was no longer necessary. Has that changed?

[scb01]

Yes, I was mistaken about the interceptApplication not being needed. What I've noticed that when the app is in the background or suspended, it will be woken up by iOS only if the handleEventsForBackgroundURLSession ( where the interceptApplication is called) is implemented in your app delegate.

I will work on your question regarding offline and will post an update here in the next few days.

[hipwelljo]

Thanks! I believe I have now implemented everything needed for uploading photos! I tested background transfers and that appears to be working nicely. I'm now testing force quitting the app before transfers are completed to test the process of restoring the progress and completion blocks. It appears the uploads are not starting after doing so. The progress block is never called after that, nor the completion block, for either of the two uploads that were ongoing before the force quit. transferUtility.getUploadTasks().result still contains 2 AWSS3TransferUtilityUploadTask objects even minutes later. But if I stop running the app and rerun it, then there's 0 elements in that array. I believe I am correctly reassigning those blocks and it just doesn't seem to be resuming those uploads. I am using version 2.6.31. Perhaps I am missing something.

Inside didFinishLaunchingWithOptions I have

AWSS3TransferUtility.register(with: awsConfiguration, transferUtilityConfiguration: transferUtilityConfiguration, forKey: NetworkingKey.TransferUtilityName) { error in
    DispatchQueue.main.async {
        let transferUtility = AWSS3TransferUtility.s3TransferUtility(forKey: NetworkingKey.TransferUtilityName)
        for task in transferUtility.getUploadTasks().result as? [AWSS3TransferUtilityUploadTask] ?? [] {
            task.setProgressBlock(photoUploadProgressBlock)
            task.setCompletionHandler(photoUploadCompletionBlock)
        }
    }
}

[scb01]

@hipwelljo

Good to hear that you have been able to get things working. Here is what is going on with the force close - When you force close the app, the tasks in the NSURLSession are canceled by iOS ( see https://developer.apple.com/documentation/foundation/nsurlsessionconfiguration/1407496-backgroundsessionconfigurationwi for more details).

I believe that the transfers are going into an "Unknown" state as the TransferUtility is unable to find them in the underlying session. You can check the status using task.status and this would likely be AWSS3TransferUtilityTransferStatusUnknown. Can you check and let me know?

[hipwelljo]

Aw okay. Yes the status is indeed unknown. So what would be the right way to go ahead and restart uploads in this scenario? Is there a way to 'naturally' test this process where we need to reassign the blocks, since force quitting actually cancels them?

[scb01]

A couple of options come to mind - a) you could put the transfer request back on the UI (all transfers that have a status of unknown) upon the UI and have the app user try them again b) you could automatically create a new transfer request for each task in an unknown status and track them on the UI.

The right answer depends on your use cases/scenarios. Let me know if you have further questions.

[hipwelljo]

👍 I think my only questions remaining are in regards to behavior when offline, returning online, etc. I'll look forward to hearing about that and let you know if I have any further questions in the meantime. I appreciate your help!

[scb01]

@hipwelljo
I ran a bunch of experiments with my test app. Here is the method I used

• Ran all the tests on a physical device ( iPhone 5c).
• Initiated transfers ( uploads, downloads, multipart uploads) and at random times flipped the phone into airplane mode, waited a while and then flipped it back out of airplane mode.
• for each interrupted transfers, I then checked if it failed, errored out, got stuck etc.
• I set thetransferUtilityConfiguration.retryLimit to 10. This setting will retry the transfer up to 10 times if there was an error
• I set the transferUtilityConfiguration.multiPartConcurrencyLimit to 1 (down from my usual value, 5). This is to make sure that only one part is uploaded at a time, so that I could simulate interruptions more easily.

My findings were as follows

• If the device was online when you initiated the transfer, the transfer would be stuck when the device went offline. The transfer would continue when the device was put back online and would always complete successfully.
• If the device was offline when you initiated the transfer, uploads and downloads would still start and finish successfully when the device went back online. Multipart uploads, however, always failed. This is working as expected - the multipart upload functionality requires connectivity to setup a multipart request and that first step will fail in this case.

[hipwelljo]

That's good to hear thanks!

In the case where an internet connection is poor and it fails to upload (not using multipart), does it call the completionHandler with an error object, and then make the request again, and repeat until the retryLimit is exceeded? Thus we could expect to receive up to 10 errors. Or does it not call the completionHandler with an error until it has exceeded the retryLimit?

Once a task has errored, its status will be AWSS3TransferUtilityTransferStatusError. Does this task persist between app launches? So on app launch is it appropriate to check if there's any tasks with an error status and recreate the uploads (or inform the user in the ui), similar to what we're doing with the unknown status? When do the tasks get cleared from the device? I am wanting to make sure we don't enter into a state of limbo where transfers are lost.

[scb01]

@hipwelljo
For your first question, the completion handler will only be called with an error if the retryLimit has been exceeded. You will only get one error.

For the second question, tasks that are completed successfully or errored out will not be persisted between launches. If the transfer was in progress and did not get to an end state before the app was terminated, it could be in a number of states

• Unknown ( indicating that it wasn't present in the underlying NSURLSession and we don't know happened to it)
• Completed ( indicating that the transfer continued and finished successfully even though the app was terminated and it was marked as completed in the NSURLSession)
• Error ( indicating that the transfer had an error )
• In Progress ( can happen if the app is opened immediately after it is terminated. Also, will happen for large MultiPart uploads)

To summarize, the TransferUtility will only keep track of the transfer state for a transfer until the transfer reaches a terminal state. After that, it will still keep the state in memory (as a convenience mechanism) for the current app session, but will not persist it between app launches.

[hipwelljo]

Thanks! A follow-up question: when it fails to upload but has not yet met the retryLimit, does it use the same transferID to try again, or does it gain a new transferID?

[scb01]

Once a transferID is assigned to a transfer, it is never changed. It will keep the transferID until the retries are exhausted.

[scb01]

@hipwelljo
Just wanted to let you know that the documentation updates have been deployed.
I am going to go ahead and close out this ticket as it looks that you are good to go.

Thank you for your patience and for working with me to resolve these issues.

[SanCHEESE]

@cbommas
Sorry for writing in the closed thread. What are the possible reasons for error status of AWSS3TransferUtilityDownloadTask fetched using utility's getDownloadTasks.result after restarting an app? I've also suspended this task on app termination. Any way to look at error description?

Thank you!

[scb01]

@SanCHEESE

I can think of a couple of reasons why it went into error status.

1. The max number of retries were reached.
2. The local file path you specified for the download is no longer valid. This can happen if you are using the temp or Library/Cache directory and it was cleared by the OS.

In any case, you can use this code snippet get more details on the error.

     let errorInfo = err.userInfo["Error"] as? [String: Any]
     if errorInfo != nil {
          print("Found error in response. Details are:")
          for element in errorInfo! {
              print(">> \(element.key): \(element.value)")
          }
     }

Also, if this doesn't help resolve the issue, it will be great if you could create a new issue and we can track it there.

[SanCHEESE]

@cbommas

Thank you for quick reply! I've found that printing AWSS3TransferUtilityTask's sessionTask.error also helps.

I've enabled verbose logging and what I've found here that AWSS3TransferUtility handles NSURLSession's NSURLSessionTask when app starts after force termination and threats this as error:

Error Domain=NSURLErrorDomain Code=-999 "(null)" UserInfo={NSURLErrorBackgroundTaskCancelledReasonKey=0, NSErrorFailingURLStringKey=https://s3.eu-central-1.amazonaws.com/dubs.dev.audio.distrib/433a5278-a693-41f8-a9c3-5d486d2019fb/en/d9eb722e-153c-4ee3-8b45-8915d5916768-fg_en_2_Bjr.aac?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJZKWAW7WYWGHPWPA%2F20181008%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20181008T081824Z&X-Amz-Expires=2999&X-Amz-SignedHeaders=host&X-Amz-Signature=8cb6b7c1a8dfdb35da25dd9ab6282992282f1d83daba1df0132c3bbbea7fef25, NSErrorFailingURLKey=https://s3.eu-central-1.amazonaws.com/dubs.dev.audio.distrib/433a5278-a693-41f8-a9c3-5d486d2019fb/en/d9eb722e-153c-4ee3-8b45-8915d5916768-fg_en_2_Bjr.aac?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJZKWAW7WYWGHPWPA%2F20181008%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20181008T081824Z&X-Amz-Expires=2999&X-Amz-SignedHeaders=host&X-Amz-Signature=8cb6b7c1a8dfdb35da25dd9ab6282992282f1d83daba1df0132c3bbbea7fef25, NSURLSessionDownloadTaskResumeData=<3c3f786d ...>}

Code -999 means that task was cancelled, NSURLErrorBackgroundTaskCancelledReasonKey=0 means that user terminated an app manually. There is NSURLSessionDownloadTaskResumeData value which has resume data to continue the download.

What I expect is that AWSS3TransferIUtility's getDownloadTasks.result return AWSS3TransferUtilityDownloadTask task with paused status and after resume, it continues from where it was suspended.

BTW, using resume data above you can resume download process by invoking

(NSURLSessionDownloadTask *)downloadTaskWithResumeData:(NSData*)resumeData

Thanks! I can create an issue if this is possible to resolve.

[SanCHEESE]

@cbommas

Just checking that you've read my comment above.

[scb01]

@SanCHEESE

Sorry, I missed seeing this. I am not sure that resumeData will work, but am happy to give it a shot.
It would be great if you could create a new issue to track this.

[AlexGingell]

@scb01 Forgive me for resurrecting this years later, but I'm seeing issues with handling when a download spans app termination and relaunch. I'm stopping execution in Xcode as a proxy for iOS background termination - I'm assuming this is not the same as the user force quitting, as sometimes I am able to continue a download doing this whereas force quitting should always cancel download sessions.

I'm having problems consistently obtaining pre-existing download tasks in order to reconnect handler blocks. I would like to do this manually because I also take this opportunity to reconnect handler blocks on the download task progress object:

NSArray<AWSS3TransferUtilityDownloadTask *> *downloadTasks = [[[self transferUtilityOSSEast1] getDownloadTasks] result];
for (AWSS3TransferUtilityDownloadTask *downloadTask in downloadTasks)
{
    NSURLSessionTask *sessionTask = [downloadTask sessionTask];
    RLogInfo(@"[RemoteFileStore] Found download task with session task %@", analyticsStringFromSessionTask(sessionTask));
    [self configureAndTrackDownloadTask:downloadTask];
}

configureAndTrackDownloadTask: sets the task.progress pausingHandler, resumingHandler and cancellationHandler, and then sets the progressHandler and completionHandler of the download task. It also begins tracking the progress object internally.

I find that it is very rare for any pre-existing download to be logged on relaunching the app, either by simply stopping execution and restarting in Xcode while the download is running (in foreground or background), or by turning on airplane mode first to ensure the download does not complete before relaunch.

If I add the following code before attempting this:

AWSS3TransferUtilityBlocks *blocks = [[AWSS3TransferUtilityBlocks alloc] initWithUploadProgress:nil
                                                                        multiPartUploadProgress:nil
                                                                               downloadProgress:nil
                                                                                uploadCompleted:nil
                                                                       multiPartUploadCompleted:nil
                                                                              downloadCompleted:nil];
[[self transferUtilityOSSEast1] enumerateToAssignBlocks:blocks];

then I find that I have better luck. Bear in mind that I need to set the task.progress handlers before I set the handlers on the task itself for internal logic related to the fact that the task completionHandler does not execute for cancelled downloads. That is why I'm passing nil - examination of AWSS3TransferUtility code leads me to believe that setting a non-nil completionHandler will fire it immediately if the task already completed which is undesirable in my case because I need the task.progress handlers to be set before the completionHandler can be run.

To recap this slightly rambling post, my question is this: on app launch, will it suffice to register the transfer utility and then enumerate the result of getDownloadTasks and set all block handlers manually, or is there some other magic connected to using enumerateToAssignBlocks: which I have failed to recognise looking through the source code? It just seems like the former method often fails to find download tasks that should be present during testing whereas I have better luck when executing enumerateToAssignBlocks:blocks, however the latter doesn't provide a way to assign task.progress-centric handler blocks prior to setting handler objects on the task object itself (unless you pass nil for those blocks and run enumerate first, which is somewhat messy).