Crash in AWSS3StorageService.upload

[smediomike]

Describe the bug

Thread 10 Crashed:
0 libsystem_kernel.dylib 0x1ec60d1d4 __pthread_kill + 8
1 libsystem_pthread.dylib 0x22477aef8 pthread_kill + 268
2 libsystem_c.dylib 0x1a3fffad8 abort + 128
3 libc++abi.dylib 0x2245895b8 abort_message + 132
4 libc++abi.dylib 0x224577bac demangling_terminate_handler() + 348
5 libobjc.A.dylib 0x19955ee14 _objc_terminate() + 156
6 libc++abi.dylib 0x22458887c std::__terminate(void ()()) + 16
7 libc++abi.dylib 0x22458bdfc __cxxabiv1::failed_throw(__cxxabiv1::__cxa_exception) + 88
8 libc++abi.dylib 0x22458bda4 __cxa_throw + 92
9 libobjc.A.dylib 0x19954344c objc_exception_throw + 448
10 CFNetwork 0x19d7d5254 -[__NSURLBackgroundSession performBlockOnQueueAndRethrowExceptions:] + 224
11 CFNetwork 0x19d7d5430 -[__NSURLBackgroundSession _uploadTaskWithTaskForClass:] + 192
12 CloudBackup 0x100b75edc AWSS3StorageService.startUpload(preSignedURL:fileURL:contentType:transferTask:startTransfer:) + 1488
13 CloudBackup 0x100b74ff8 closure #2 in AWSS3StorageService.upload(serviceKey:uploadSource:contentType:metadata:accelerate:onEvent:) + 180
14 CloudBackup 0x100b757e1 partial apply for closure #2 in AWSS3StorageService.upload(serviceKey:uploadSource:contentType:metadata:accelerate:onEvent:) + 1
15 libswift_Concurrency.dylib 0x1a7953e39 completeTaskWithClosure(swift::AsyncContext*, swift::SwiftError*) + 1

Steps To Reproduce

Steps to reproduce the behavior:
1. in our application, transfer all photos to s3 storage (about 2000~6000)
2, it will crash in this process sometimes (2/5)

Expected behavior

1, if need extra permission or error happen, handle it and callback error id, but not crash

Amplify Framework Version

2.26.1

Amplify Categories

Storage

Dependency manager

Swift PM

Swift version

5.0

CLI version

12.x

Xcode version

16

Relevant log output

Thread 10 crashed with ARM Thread State (64-bit):
    x0: 0x0000000000000000   x1: 0x0000000000000000   x2: 0x0000000000000000   x3: 0x0000000000000000
    x4: 0x000000022458df3b   x5: 0x000000016fbf5850   x6: 0x000000000000006e   x7: 0x0000000000000000
    x8: 0x6264785967e03b72   x9: 0x62647858085f4b72  x10: 0x0000000000000051  x11: 0x000000000000000b
   x12: 0x000000000000000b  x13: 0x000000019c69abbc  x14: 0x00000000001ff800  x15: 0x00000000000007fb
   x16: 0x0000000000000148  x17: 0x000000016fbf7000  x18: 0x0000000000000000  x19: 0x0000000000000006
   x20: 0x000000000002085f  x21: 0x000000016fbf70e0  x22: 0x00000001fdf7c000  x23: 0x0000000000000000
   x24: 0x000000000003661a  x25: 0x000000016fbf70e0  x26: 0x0000000000000000  x27: 0x0000000000000000
   x28: 0x000000016fbf70e0   fp: 0x000000016fbf57c0   lr: 0x000000022477aef8
    sp: 0x000000016fbf57a0   pc: 0x00000001ec60d1d4 cpsr: 0x40001000
   far: 0x0000000000000000  esr: 0x56000080  Address size fault

Is this a regression?

Yes

Regression additional context

2.17.1 SDK did not happen this issue.

Platforms

iOS

OS Version

iOS18

Device

iPhone12/14 etc

Specific to simulators

No response

Additional context

No response

[sebaland]

Hi @smediomike , thanks for reaching out.

Amplify for Swift 2.26.1 is almost 9 months old, so please update to the latest version (2.43.0 as of Oct 30) and check if the issue persists.

If you still experience crashes, then please provide a full symbolicated crashlog, as what you've shared is missing crucial information.

Thanks!

[smediomike]

CloudBackup-2024-11-01-120933.txt

Hi @ruisebas,

I upgraded SDK to 2.43.0 and ask the Japan customer to reproduce it, current version's reproduce rate already reduced (1/3), but still happen. and please change attachment extension name from "txt" to ips. which can show callstack.

Thanks!

[sebaland]

@smediomike Thanks for providing the crashlog. Unfortunately, it's missing the line numbers so we cannot know for sure exactly where the crash is being triggered:

AWSS3StorageService.startUpload(preSignedURL:fileURL:contentType:transferTask:startTransfer:) + 716

Based on the following rows in the backtrace, it seems to be related with the upload task. So I have some follow up questions:

• If you're able to reproduce this issue, could you please share the verbose logs leading to the crash? You can enable verbose logging by doing the following before calling Amplify.configure():

   Amplify.Logging.logLevel = .verbose

• To help us reproduce, could you either share a sample app that can reproduce the crash, or share code snippets on how you are interacting with the Storage's APIs?
• Are customers experiencing the crash geographically distant from the AWS region they're uploading to? Or is that irrelevant?

A symbolicated crashlog that contains the line numbers would still be helpful as well.

Thanks.

[smediomike]

@ruisebas

thanks for you information.

1, this issue currently only can be reproduced by some customers. It may diffcult to get more detailed information, but I still try.

2, Here is my upload file source code API, could u please help to review:

    @objc func uploadFile(key: String, local_path: String,_ size_progress: @escaping (_ completed: Int64) -> Void) async->Int
    {

        let fileUrl = URL(fileURLWithPath: local_path)
        
        let languages: [String] = UserDefaults.standard.object(forKey: "AppleLanguages") as! [String]
        var currentLanguage = languages.first
        if(currentLanguage == nil){
            currentLanguage = "en"
        }
   
        let soption : StorageUploadFileRequest.Options =  StorageUploadFileRequest.Options(accessLevel: StorageAccessLevel.private, metadata:["sort_lang": currentLanguage!])
        
        //print("Swift UploadFile local langauge : \(currentLanguage!)")
        
        let uploadTask = Amplify.Storage.uploadFile(
            key: key,
            local: fileUrl,
            options: soption
        )
        
        Task {
            for await progress in await uploadTask.progress {
                print("Progress: \(progress)")
                size_progress(progress.completedUnitCount)
            }
        }
        self.addFileUploadOperationWithKey(skey: key,isdata:true, operation: uploadTask)
        
        do {
            let _ = try await uploadTask.value
            self.removeFileUploadOperationWithKey(skey: key,isdata:true)
            return AWSSmedioError.Action_Success.rawValue
        } catch let error as StorageError {
            self.removeFileUploadOperationWithKey(skey: key,isdata:true)
            print("Swift UploadFile Failed: \(error.errorDescription). \(error.recoverySuggestion)")
            return self.convertStorageError(error: error)
        } catch {
            self.removeFileUploadOperationWithKey(skey: key,isdata:true)
            return AWSSmedioError.SYNC_ERROR_SERVER_NETWROK_ISSUE.rawValue
        }
        
    }

3, customer is in Japan, server also is in Japan. it should irrelevant and since 2.17 version never reported such issue from customer also.

[smediomike]

Hello @ruisebas

I get new log , it seems difference with before, like the below, Here this file is get from PHAssert, when SDK try to get file size property will cause this issue, but before I upload this file with SDK API, I also add file check ,like below, it show no problem, another thing is if user re-start application after crash and transfer again, it should be okay, can transfer.

code:

NSFileManager *fileManager = [NSFileManager defaultManager];

   NSFileManager *fileManager = [NSFileManager defaultManager];

    BOOL canUpload = NO;
    int file_error = 0;
    if ([fileManager fileExistsAtPath:file_path]) {
        if ([fileManager isReadableFileAtPath:file_path]) {
            
            NSError* error = nil;
            NSDictionary *attr = [fileManager attributesOfItemAtPath:file_path error:&error];
            if(error == nil && attr!= nil && [[attr objectForKey:NSFileSize] intValue]>0){
                canUpload = YES;
            }else {
                file_error = SYNC_ERROR_GENERAL;
            }
        } else {
            file_error = SYNC_ERROR_NO_READ_WRITE_PERMISSION;
        }
    } else {
        file_error = SYNC_ERROR_CONTENT_NOT_EXIST;
    }
    
    if(!canUpload)
    {
        result_callback(self.mBundleItem,file_error);
        return;
    }

New log:

Thread 13 Crashed:
0   libswiftCore.dylib            	       0x1908305e0 _assertionFailure(_:_:file:line:flags:) + 264
1   CloudBackup                   	       0x10155f960 static Fatal.die(reason:extra:file:line:) + 592
2   CloudBackup                   	       0x10155fc0c static Fatal.require(_:file:line:) + 140
3   CloudBackup                   	       0x10132f684 FileSystem.getFileSize(fileURL:) + 852
4   CloudBackup                   	       0x101377078 UploadSource.getFile(fileSystem:) + 972
5   CloudBackup                   	       0x101310f08 implicit closure #1 in closure #2 in AWSS3StorageService.upload(serviceKey:uploadSource:contentType:metadata:accelerate:onEvent:) + 48
6   CloudBackup                   	       0x101311f18 partial apply for implicit closure #1 in closure #2 in AWSS3StorageService.upload(serviceKey:uploadSource:contentType:metadata:accelerate:onEvent:) + 24
7   CloudBackup                   	       0x10131f4cc attempt<A>(_:fail:) + 92
8   CloudBackup                   	       0x10130ff08 closure #2 in AWSS3StorageService.upload(serviceKey:uploadSource:contentType:metadata:accelerate:onEvent:) + 552
9   CloudBackup                   	       0x101310e71 partial apply for closure #2 in AWSS3StorageService.upload(serviceKey:uploadSource:contentType:metadata:accelerate:onEvent:) + 1
10  libswift_Concurrency.dylib    	       0x19d933e39 completeTaskWithClosure(swift::AsyncContext*, swift::SwiftError*) + 1

[smediomike]

@ruisebas

Append more information, about PHAsset file upload, in our code, we copy content into one temp file and then ask AWS SDK API to upload like the above. but not use system content path directly.

[sebaland]

Thanks for the additional information. We will attempt to reproduce, and also take a look at changes that might trigger this between versions 2.17.1 and 2.26.1.

In the meantime, if you manage the get the verbose logs or any additional information, please share it here.

Thanks!

[smediomike]

CloudBackup-2024-11-05-173626.txt

CloudBackup-2024-11-05-164519.txt

@ruisebas

such two crash log which happen on uploading process. please change .txt to .ips and then Xcode open it, which can see full stack trace with AWS SDK.

and the two crash points caused two places, like such FileSystem.getFileSize and then Fatal.require; i do not know whether AWSSDK do not use fatalError to let app to crash but callback error id to app layer to handle?

another information is I provided two builds (one is delay 10 seconds to delete uploaded tmp files and another is never delete tmp files), but still happen this issue on the two builds.

[sebaland]

These seem to be unrelated crashes, at least based on the crashlogs.
Regarding the file size issue, it might be happening because of the temporary files you created being removed by the time the library attempts to upload them, as you mentioned you're attempting to upload around about 2000~6000 files.

We will continue our investigation

[smediomike]

@ruisebas

Yes, it did not related real crash, but if SDK include FatalError, it will cause App crash. There are two places in SDK which cause this according to the log. Today our other customers also report this issue (iPhone 15 ProMax/ iPhone13 ProMax ). currently this product has more than 100000 member, so could u please check this asap or add protected code and callback to Application layer to handle(we can try upload again according to error type)?

BTW, In our application, there are 6 uploading thread as Max at the same time.

[sebaland]

@smediomike sorry, could you please clarify if the FatalError is something you've experiencing in production or not?
Your latest comment is misleading.

Regarding the FatalError, may I ask why you are creating temporary files instead of just uploading the ones from disk? It seems that would unnecessarily duplicate the space used in disk.

[smediomike]

@ruisebas

Sorry, the above several log files were based on DEBUG configuration build, it may impact FatalError. I already ask our customer to help to reproduce again on release enviroment and get log.

In the above log file(release build), there is another "abort" crash happen in AWS SDK, like the below,could u please help to check this also? thanks!

Thread 13 Crashed:
0 libsystem_kernel.dylib 0x1ec60d1d4 __pthread_kill + 8
1 libsystem_pthread.dylib 0x22477aef8 pthread_kill + 268
2 libsystem_c.dylib 0x1a3fffad8 abort + 128
3 libc++abi.dylib 0x2245895b8 abort_message + 132
4 libc++abi.dylib 0x224577bac demangling_terminate_handler() + 348
5 libobjc.A.dylib 0x19955ee14 _objc_terminate() + 156
6 libc++abi.dylib 0x22458887c std::__terminate(void ()()) + 16
7 libc++abi.dylib 0x22458bdfc __cxxabiv1::failed_throw(__cxxabiv1::__cxa_exception) + 88
8 libc++abi.dylib 0x22458bda4 __cxa_throw + 92
9 libobjc.A.dylib 0x19954344c objc_exception_throw + 448
10 CFNetwork 0x19d7d5254 -[__NSURLBackgroundSession performBlockOnQueueAndRethrowExceptions:] + 224
11 CFNetwork 0x19d7d5430 -[__NSURLBackgroundSession _uploadTaskWithTaskForClass:] + 192
12 CloudBackup 0x1044c24c4 AWSS3StorageService.startUpload(preSignedURL:fileURL:contentType:transferTask:startTransfer:) + 716
13 CloudBackup 0x1044c1369 closure #2 in AWSS3StorageService.upload(serviceKey:uploadSource:contentType:metadata:accelerate:onEvent:) + 1
14 CloudBackup 0x1044c1c01 partial apply for closure #2 in AWSS3StorageService.upload(serviceKey:uploadSource:contentType:metadata:accelerate:onEvent:) + 1
15 libswift_Concurrency.dylib 0x1a7953e39 completeTaskWithClosure(swift::AsyncContext*, swift::SwiftError*) + 1

[sebaland]

Unfortunately that stack trace doesn't tell us much, it's basically the same as the original one you posted.

The last call to out library is AWSS3StorageService.startUpload(preSignedURL:fileURL:contentType:transferTask:startTransfer:), and then it crashes inside CFNetwork for some reason. Ideally the full crashlogs might give us more information.

[smediomike]

@ruisebas

finally I got AWSSDk full log from JP, and according to log, crash reason is file is not exist, due to multi-thread upload and same name file happen which cause deleting time issue.

I added same name handling and protected upload API for thread safe, this issue is gone.

Thanks for your help.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.