fix: Bug where subscription connections happen at the same time

[wooj2]

Prior to this fix, the crash looked something like this:

Seems like a race condition, where both subscription connections are coming in very quick succession, and somehow we are getting into a read/write race condition with reconciliationQueueConnectionStatus inside of AWSIncomingEventReconciliationQueue.

To reproduce this issue, you can remove the connectionStatusOperationQueue.async and run the unit test in AWSIncomingEventReconciliationQueueTests to see a very similar segfault:

2020-04-09 17:42:10.082784-0700 SampleDataStore[92033:19843535] [IncomingAsyncSubscriptionEventToAnyModelMapper] connectionState now connected
2020-04-09 17:42:10.082807-0700 SampleDataStore[92033:19843493] [IncomingAsyncSubscriptionEventToAnyModelMapper] connectionState now connected
2020-04-09 17:42:10.083090-0700 SampleDataStore[92033:19843535] -[__NSCFNumber count]: unrecognized selector sent to instance 0x8000000000000000
2020-04-09 17:42:10.103774-0700 SampleDataStore[92033:19843535] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[__NSCFNumber count]: unrecognized selector sent to instance 0x8000000000000000'
*** First throw call stack:
(
	0   CoreFoundation                      0x00007fff23c7127e __exceptionPreprocess + 350
	1   libobjc.A.dylib                     0x00007fff513fbb20 objc_exception_throw + 48
	2   CoreFoundation                      0x00007fff23c91fd4 -[NSObject(NSObject) doesNotRecognizeSelector:] + 132
	3   CoreFoundation                      0x00007fff23c75c4c ___forwarding___ + 1436
	4   CoreFoundation                      0x00007fff23c77f78 _CF_forwarding_prep_0 + 120
	5   libswiftCore.dylib                  0x00007fff5191b813 $sSD8_VariantV8setValue_6forKeyyq_n_xtF + 147
	6   libswiftCore.dylib                  0x00007fff51b10c6d $sSDyq_SgxcisTf4ngn_n + 301
	7   libswiftCore.dylib                  0x00007fff518eb7a2 $sSDyq_Sgxcis + 18
	8   AmplifyPlugins                      0x000000010e0732b5 $s14AmplifyPlugins35AWSIncomingEventReconciliationQueueC14onReceiveValue33_6BCE524E5F5E68D42C91D7334411AD0BLL07receiveI0yAA05ModelefD0O_tF + 757
	9   Combine                             0x00007fff23337433 $s7Combine11SubscribersO4SinkC7receiveyAC6DemandVxF + 35
	10  Combine                             0x00007fff23337670 $s7Combine11SubscribersO4SinkCy_xq_GAA10SubscriberA2aGP7receiveyAC6DemandV5InputQzFTW + 16
	11  Combine                             0x00007fff23337bb3 $s7Combine16AnySubscriberBoxC7receiveyAA11SubscribersO6DemandV5InputQzF + 35
	12  Combine                             0x00007fff23342f10 $s7Combine18PassthroughSubjectC7Conduit33_A517F1CF3C35FD924691D71B0A4E0FAFLLC5offeryyxF + 160
	13  Combine                             0x00007fff23342e08 $s7Combine18PassthroughSubjectC4sendyyxF + 168
	14  AmplifyPlugins                      0x000000010e079083 $s14AmplifyPlugins27AWSModelReconciliationQueueC7receive33_3FE212790B4B53CA8F25F1F4642C80CFLLyyAA034IncomingSubscriptionEventPublisherR0OF + 659
	15  AmplifyPlugins                      0x000000010e078c92 $s14AmplifyPlugins27AWSModelReconciliationQueueC9modelType14storageAdapter3api26incomingSubscriptionEventsAC0A05Model_pXp_AA013StorageEngineI0_pSgAH26APICategoryGraphQLBehavior_pAA08IncomingL14EventPublisher_pSgtcfcyAA0tluvU0OcfU0_ + 242
	16  Combine                             0x00007fff23337433 $s7Combine11SubscribersO4SinkC7receiveyAC6DemandVxF + 35
	17  Combine                             0x00007fff23337670 $s7Combine11SubscribersO4SinkCy_xq_GAA10SubscriberA2aGP7receiveyAC6DemandV5InputQzFTW + 16
	18  Combine                             0x00007fff23337bb3 $s7Combine16AnySubscriberBoxC7receiveyAA11SubscribersO6DemandV5InputQzF + 35
	19  Combine                             0x00007fff23342f10 $s7Combine18PassthroughSubjectC7Conduit33_A517F1CF3C35FD924691D71B0A4E0FAFLLC5offeryyxF + 160
	20  Combine                             0x00007fff23342e08 $s7Combine18PassthroughSubjectC4sendyyxF + 168
	21  AmplifyPlugins                      0x000000010e076093 $s14AmplifyPlugins37AWSIncomingSubscriptionEventPublisherC9onReceive33_050B108B990E83D57A3F0C1939BA23AALL12receiveValueyAA013IncomingAsyncdE0O_tF + 371
	22  Combine                             0x00007fff23337433 $s7Combine11SubscribersO4SinkC7receiveyAC6DemandVxF + 35
	23  Combine                             0x00007fff23337670 $s7Combine11SubscribersO4SinkCy_xq_GAA10SubscriberA2aGP7receiveyAC6DemandV5InputQzFTW + 16
	24  Combine                             0x00007fff23337bb3 $s7Combine16AnySubscriberBoxC7receiveyAA11SubscribersO6DemandV5InputQzF + 35
	25  Combine                             0x00007fff23342f10 $s7Combine18PassthroughSubjectC7Conduit33_A517F1CF3C35FD924691D71B0A4E0FAFLLC5offeryyxF + 160
	26  Combine                             0x00007fff23342e08 $s7Combine18PassthroughSubjectC4sendyyxF + 168
	27  AmplifyPlugins                      0x000000010e0a320e $s14AmplifyPlugins46IncomingAsyncSubscriptionEventToAnyModelMapperC7dispose33_0E4FA60B4937AFAF7EB35F5A08F28826LL2ofy0A00eF0Oys6ResultOy14AWSPluginsCore12MutationSyncVyAG0hI0VGAG20GraphQLResponseErrorOyAQGGG_tF + 798
	28  AmplifyPlugins                      0x000000010e0a2b8f $s14AmplifyPlugins46IncomingAsyncSubscriptionEventToAnyModelMapperC7receivey7Combine11SubscribersO6DemandV0A00dF0OyAJ0eF0Oys6ResultOy14AWSPluginsCore12MutationSyncVyAJ0hI0VGAJ20GraphQLResponseErrorOyAVGGGytAJ8APIErrorOGF + 1327
	29  AmplifyPlugins                      0x000000010e0a4340 $s14AmplifyPlugins46IncomingAsyncSubscriptionEventToAnyModelMapperC7Combine10SubscriberAadEP7receiveyAD11SubscribersO6DemandV5InputQzFTW + 16
	30  Combine                             0x00007fff23337bb3 $s7Combine16AnySubscriberBoxC7receiveyAA11SubscribersO6DemandV5InputQzF + 35
	31  Combine                             0x00007fff23342f10 $s7Combine18PassthroughSubjectC7Conduit33_A517F1CF3C35FD924691D71B0A4E0FAFLLC5offeryyxF + 160
	32  Combine                             0x00007fff23342e08 $s7Combine18PassthroughSubjectC4sendyyxF + 168
	33  AmplifyPlugins                      0x000000010e0a0259 $s14AmplifyPlugins39IncomingAsyncSubscriptionEventPublisherC014sendConnectionF11IfConnected5eventy0A00dF0OyAF0eF0Oys6ResultOy14AWSPluginsCore12MutationSyncVyAF8AnyModelVGAF20GraphQLResponseErrorOyARGGGytAF8APIErrorOG_tF + 121
	34  AmplifyPlugins                      0x000000010e0a05e2 $s14AmplifyPlugins39IncomingAsyncSubscriptionEventPublisherC23onUpdateListenerHandler5eventy0A00dF0OyAF0eF0Oys6ResultOy14AWSPluginsCore12MutationSyncVyAF8AnyModelVGAF20GraphQLResponseErrorOyARGGGytAF8APIErrorOG_tFyycfU_ + 130
	35  AmplifyPlugins                      0x000000010e08bfba $s14AmplifyPlugins25CancelAwareBlockOperationC4mainyyF + 154
	36  AmplifyPlugins                      0x000000010e08bffb $s14AmplifyPlugins25CancelAwareBlockOperationC4mainyyFTo + 43
	37  Foundation                          0x00007fff25755bd6 __NSOPERATION_IS_INVOKING_MAIN__ + 17
	38  Foundation                          0x00007fff25751e49 -[NSOperation start] + 731
	39  Foundation                          0x00007fff25756530 __NSOPERATIONQUEUE_IS_STARTING_AN_OPERATION__ + 17
	40  Foundation                          0x00007fff2575603a __NSOQSchedule_f + 182
	41  libdispatch.dylib                   0x000000010e5f43c9 _dispatch_block_async_invoke2 + 83
	42  libdispatch.dylib                   0x000000010e5e5d48 _dispatch_client_callout + 8
	43  libdispatch.dylib                   0x000000010e5e86ba _dispatch_continuation_pop + 552
	44  libdispatch.dylib                   0x000000010e5e7ac5 _dispatch_async_redirect_invoke + 849
	45  libdispatch.dylib                   0x000000010e5f728c _dispatch_root_queue_drain + 351
	46  libdispatch.dylib                   0x000000010e5f7b96 _dispatch_worker_thread2 + 132
	47  libsystem_pthread.dylib             0x00007fff5245f6b3 _pthread_wqthread + 583
	48  libsystem_pthread.dylib             0x00007fff5245f3fd start_wqthread + 13
)
libc++abi.dylib: terminating with uncaught exception of type NSException
(lldb)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[palpatim]

1. This is a dispatch queue, not an operation queue
2. Add a target of a serial queue for the whole SyncEngine system, to help prevent thread explosions and increase performance. (I don't know whether we already have a top-level SyncEngine subsystem queue, so if you want to follow up on this with a separate PR, feel free.)

[wooj2]

1. updated naming
2. Added to do, to be followed up soon

[palpatim]

Should protect the assignment in onReceiveCompletion, as well as cancel and any other assignments of instance variables that I missed in my scan.

[palpatim]

Consider adding a link to a backlog item for your TODO, otherwise LGTM