Merge tag 'v13.2.0' into next-major

"Bugfix release"
realm · Jan 11, 2023 · e63643e · e63643e
2 parents 46f5e8a + 1ad79b2
commit e63643e
Show file tree

Hide file tree

Showing 158 changed files with 5,537 additions and 3,329 deletions.
diff --git a/.gitmodules b/.gitmodules
@@ -1,6 +1,3 @@
 [submodule "external/catch"]
 	path = external/catch
 	url = https://github.com/catchorg/Catch2.git
-[submodule "tools/vcpkg/ports"]
-	path = tools/vcpkg/ports
-	url = https://github.com/microsoft/vcpkg.git
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -14,13 +14,93 @@
 ### Compatibility
 * Fileformat: Generates files with format v24. Reads and automatically upgrade from fileformat v10. If you want to upgrade from an earlier file format version you will have to use RealmCore v13.x.y or earlier.
 
+----------------------------------------------
+
+# 13.2.0 Release notes
+
+### Enhancements
+* Fulltext search index can now be defined on ObjectStore Property (PR [#6100](https://github.com/realm/realm-core/pull/6100))
+* Improve performance of acquiring read locks when a read lock for that version is already held. This speeds up many operations related to change notifications, and particularly refreshing a Realm which has change notifiers registered.
+* Upgrade OpenSSL from 1.1.1n to 3.0.7. ([#6097](https://github.com/realm/realm-core/pull/6097))
+* Converting flexible sync realms to bundled and local realms is now supported ([#6076](https://github.com/realm/realm-core/pull/6076))
+* Compensating write errors are now surfaced to the SDK/user after the compensating write has been applied in a download message ([#6095](https://github.com/realm/realm-core/pull/6095)).
+* Normalize sync connection parameters for device information ([#6029](https://github.com/realm/realm-core/issues/6029))
+
+### Fixed
+* Fix `BadVersion` exceptions which could occur when performing multiple writes on one thread while observing change notifications on another thread ([#6069](https://github.com/realm/realm-core/issues/6069), since v13.0.0).
+* Compare actual users (`SyncUser::operator!=`), not pointers (`shared_ptr<SyncUser>::operator!=`). ([#realm/realm-dart#1055](https://github.com/realm/realm-dart/issues/1055), since v10.2.0)
+* Core should not alter the order of the properties for additive schemas. ([#6134](https://github.com/realm/realm-core/issues/6134))
+* Core should not use the version passed in `Realm` constructor to load the schema at some particular version, but only for setting `m_frozen_version`.
+* Fixed possible segfault in sync client where async callback was using object after being deallocated ([#6053](https://github.com/realm/realm-core/issues/6053), since v11.7.0)
+* Fixed crash when using client reset with recovery and flexible sync with a single subscription ([#6070](https://github.com/realm/realm-core/issues/6070), since v12.3.0)
+* Fixed crash with wrong transaction state, during realm migration if realm is frozen due to schema mismatch ([#6144](https://github.com/realm/realm-core/issues/6144), since v13.0.0)
+
+### Breaking changes
+* Core no longer provides any vcpkg infrastructure (the ports submodule and overlay triplets), because it handles dependant libraries internally now.
+* Allow Realm instances to have a complete view of their schema, if mode is additive. ([PR #5784](https://github.com/realm/realm-core/pull/5784)).
+* `realm_sync_immediately_run_file_actions` (c-api) now takes a third argument `bool* did_run` that will be set to the result of `SyncManager::immediately_run_file_actions`. ((#6117)[https://github.com/realm/realm-core/pull/6117]) 
+* Device information in sync connection parameters was moved into a new `device_info` structure in App::Config ([PR #6066](https://github.com/realm/realm-core/pull/6066))
+* `sdk` is now a required field in the `device_device` structure in App::Config ([PR #6066](https://github.com/realm/realm-core/pull/6066))
+
+### Compatibility
+* Fileformat: Generates files with format v23. Reads and automatically upgrade from fileformat v5.
+
+-----------
+
+### Internals
+* Updates for upcoming Platform Networking feature, including new SyncSocketProvider class. ([PR #6096](https://github.com/realm/realm-core/pull/6096))
+* Updated namespaces for files moved to realm/sync/network ([PR #6109](https://github.com/realm/realm-core/pull/6109))
+* Replace util::network::Trigger with a Sync client custom trigger. ([PR #6121](https://github.com/realm/realm-core/pull/6121))
+* Create DefaultSyncSocket class ([PR #6116](https://github.com/realm/realm-core/pull/6116))
+* Improve detection of Windows target architecture when downloading prebuild dependencies. ([#6135](https://github.com/realm/realm-core/issues/6135))
+* App services request ID now logged at info level when sync client connects ([#6143](https://github.com/realm/realm-core/pull/6143]))
+* Add c_api error category for resolve errors instead of reporting unknown category. ([PR #6157](https://github.com/realm/realm-core/pull/6157))
+* Add permanent redirect (308) as a supported redirect response from the server. ([#6162](https://github.com/realm/realm-core/issues/6162))
+* Integrate DefaultSocketProvider as SyncSocketProvider in sync client. ([PR #6171](https://github.com/realm/realm-core/pull/6171))
+
+----------------------------------------------
+
+# 13.1.2 Release notes
+### Enhancements
+* Per-object compensating write error info is now available via the C API. (PR [#6103](https://github.com/realm/realm-core/pull/6103))
+### Fixed
+* Fixed `realm_add_realm_refresh_callback` and notify immediately that there is not transaction snapshot to advance to. ([#6075](https://github.com/realm/realm-core/issues/6075), since v12.6.0)
+* Fix no notification for write transaction that contains only change to backlink property. ([#4994](https://github.com/realm/realm-core/issues/4994), since v11.4.1)
+
+### Breaking changes
+* FLX Subscription API reworked to better match SDK consumption patterns ([#6065](https://github.com/realm/realm-core/pull/6065)). Not all changes are breaking, but listing them all here together.
+  * `Subscription` is now a plain struct with public fields rather than getter functions    * `has_name()` and `name()` were merged into a single `optional<string> name` field
+  * `has_name()` and `name()` were merged into a single `optional<string> name` field
+  * `SubscriptionSet` now uses the same types for `iterator` and `const_iterator` since neither was intended to support direct mutability
+  * `SubscriptionSet::get_state_change_notification()` now offers a callback-taking overload
+  * `SubscriptionSet::find()` overloads now return `const Subscription*` with `nullptr` to signal not-found, rather than an at-end iterator
+  * `MutableSubscriptionSet::erase()` now has overloads taking a `StringData name` or a `Query`. Both return a bool to indicate if anything was found and removed.
+  * `MutableSubscriptionSet::commit()` is no longer `&&`-qualified. This means you no longer need to `std::move()` the set when calling it.
+
+### Compatibility
+* Fileformat: Generates files with format v23. Reads and automatically upgrade from fileformat v5.
+
 -----------
 
 ### Internals
 * None.
 
 ----------------------------------------------
 
+# 13.1.1 Release notes
+
+### Fixed
+* Not possible to open an encrypted file on a device with a page size bigger than the one on which the file was produced. ([#8030](https://github.com/realm/realm-swift/issues/8030), since v12.11.0)
+* Fixed `realm_refresh` so it uses an argument value for the refresh result and returns any error conditions as return value. ([#6068](https://github.com/realm/realm-core/pull/6068), since  v10.4.0)
+* Fixed `realm_compact` to actually do the compaction even if the caller did not provide a `did_compact` argument. ([#6068](https://github.com/realm/realm-core/pull/6068), since v12.7.0) 
+### Breaking changes
+* ObjectId constructor made explicit, so no more implicit conversions from const char* or array of 12 bytes. It now accepts a StringData. ([#6059](https://github.com/realm/realm-core/pull/6059))
+
+### Compatibility
+* Fileformat: Generates files with format v23. Reads and automatically upgrade from fileformat v5.
+
+----------------------------------------------
+
 # 13.1.0 Release notes
 
 ### Enhancements

diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -19,6 +19,7 @@ include(SpecialtyBuilds)
 include(GetVersion)
 include(CheckCXXCompilerFlag)
 include(CheckCXXSourceRuns)
+include(AcquireRealmDependency)
 
 include(CodeCoverage)
 
@@ -130,8 +131,6 @@ if(ANDROID)
     add_compile_options(-fdata-sections -ffunction-sections -fomit-frame-pointer -fsigned-char -fstrict-aliasing -funwind-tables -no-canonical-prefixes $<$<CONFIG:Release>:-Oz>)
 endif()
 
-set(OPENSSL_VERSION ${DEP_OPENSSL_VERSION})
-
 set(CMAKE_DEBUG_POSTFIX "-dbg")
 set(CMAKE_MINSIZEDEBUG_POSTFIX "-dbg")
 
@@ -228,52 +227,15 @@ elseif(CMAKE_SYSTEM_NAME MATCHES "Linux|Android")
 endif()
 
 if(REALM_NEEDS_OPENSSL OR REALM_FORCE_OPENSSL)
-    set(OPENSSL_USE_STATIC_LIBS ON)
-    if(VCPKG_TOOLCHAIN)
-        # If we're building with vcpkg, prefer to find OpenSSL there first
-        find_package(OpenSSL)
-    endif()
-    # We aren't building with vcpkg, or it didn't have OpenSSL
-    if(NOT OpenSSL_FOUND)
-        if(ANDROID OR (CMAKE_SYSTEM_NAME STREQUAL "Linux" AND CMAKE_SYSTEM_PROCESSOR STREQUAL "x86_64"))
-            # We have prebuilt OpenSSL tarballs for Android and Linux x86_64
-            set(_realm_have_prebuilt_openssl ON)
-        endif()
-        if(NOT REALM_USE_SYSTEM_OPENSSL AND _realm_have_prebuilt_openssl)
-            # Use our own prebuilt OpenSSL
-            if(NOT OpenSSL_DIR)
-                if(NOT EXISTS ${CMAKE_CURRENT_BINARY_DIR}/openssl/lib/cmake/OpenSSL/OpenSSLConfig.cmake)
-                    if(ANDROID)
-                        set(OPENSSL_URL "http://static.realm.io/downloads/openssl/${OPENSSL_VERSION}/Android/${CMAKE_ANDROID_ARCH_ABI}/openssl.tar.gz")
-                    else()
-                        set(OPENSSL_URL "http://static.realm.io/downloads/openssl/${OPENSSL_VERSION}/Linux/x86_64/openssl.tar.gz")
-                    endif()
-
-                    message(STATUS "Getting ${OPENSSL_URL}...")
-                    file(DOWNLOAD "${OPENSSL_URL}" "${CMAKE_CURRENT_BINARY_DIR}/openssl/openssl.tar.gz" STATUS download_status)
-
-                    list(GET download_status 0 status_code)
-                    if (NOT "${status_code}" STREQUAL "0")
-                        message(FATAL_ERROR "Downloading ${url}... Failed. Status: ${download_status}")
-                    endif()
-
-                    message(STATUS "Uncompressing OpenSSL...")
-                    execute_process(
-                        COMMAND ${CMAKE_COMMAND} -E tar xfz "openssl.tar.gz"
-                        WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/openssl"
-                    )
-                endif()
-
-                set(OpenSSL_DIR "${CMAKE_CURRENT_BINARY_DIR}/openssl/lib/cmake/OpenSSL")
-            endif()
-            find_package(OpenSSL REQUIRED CONFIG)
-            target_link_libraries(OpenSSL::SSL INTERFACE OpenSSL::Crypto)
-        else()
-            # Use whatever OpenSSL CMake finds on the system
-            find_package(OpenSSL REQUIRED)
-        endif()
+    if(NOT REALM_USE_SYSTEM_OPENSSL AND (ANDROID OR WIN32 OR CMAKE_SYSTEM_NAME STREQUAL "Linux"))
+        # Use our own prebuilt OpenSSL
+        realm_acquire_dependency(openssl ${DEP_OPENSSL_VERSION} OPENSSL_CMAKE_INCLUDE_FILE)
+
+        include(${OPENSSL_CMAKE_INCLUDE_FILE})
     endif()
 
+    set(OPENSSL_USE_STATIC_LIBS ON)
+    find_package(OpenSSL REQUIRED)
     set(REALM_HAVE_OPENSSL ON)
     string(REGEX MATCH "^([0-9]+)\\.([0-9]+)" OPENSSL_VERSION_MAJOR_MINOR ${OPENSSL_VERSION})
 elseif(APPLE)
@@ -285,7 +247,10 @@ endif()
 # so for an iOS build it'll use the path from the Device plaform, which is an error on Simulator.
 # Just use -lz and let Xcode figure it out
 if(NOT APPLE AND NOT TARGET ZLIB::ZLIB)
-    if(ANDROID)
+    if(WIN32)
+        realm_acquire_dependency(zlib ${DEP_WIN32_ZLIB_VERSION} ZLIB_CMAKE_INCLUDE_FILE)
+        include(${ZLIB_CMAKE_INCLUDE_FILE})
+    elseif(ANDROID)
         # On Android FindZLIB chooses the static libz over the dynamic one, but this leads to issues
         # (see https://github.com/android/ndk/issues/1179)
         # We want to link against the stub library instead of statically linking anyway,
@@ -364,7 +329,8 @@ if(REALM_ENABLE_SYNC)
     list(APPEND REALM_EXPORTED_TARGETS Sync)
 endif()
 export(TARGETS ${REALM_EXPORTED_TARGETS} NAMESPACE Realm:: FILE RealmTargets.cmake)
-configure_file(${CMAKE_CURRENT_LIST_DIR}/tools/cmake/RealmConfig.cmake.in ${CMAKE_CURRENT_BINARY_DIR}/RealmConfig.cmake @ONLY)
+configure_file(tools/cmake/RealmConfig.cmake.in ${CMAKE_CURRENT_BINARY_DIR}/RealmConfig.cmake @ONLY)
+configure_file(tools/cmake/AcquireRealmDependency.cmake ${CMAKE_CURRENT_BINARY_DIR}/AcquireRealmDependency.cmake @ONLY)
 
 # Make the project importable from the install directory
 install(EXPORT realm
@@ -379,6 +345,11 @@ install(FILES ${CMAKE_CURRENT_BINARY_DIR}/RealmConfig.cmake
         COMPONENT devel
 )
 
+install(FILES tools/cmake/AcquireRealmDependency.cmake
+        DESTINATION lib/cmake/Realm
+        COMPONENT devel
+)
+
 # CPack
 set(CPACK_GENERATOR "TGZ")
 set(CPACK_PACKAGE_VERSION ${REALM_VERSION})

diff --git a/Jenkinsfile b/Jenkinsfile
@@ -547,12 +547,6 @@ def doBuildWindows(String buildType, boolean isUWP, String platform, boolean run
       CMAKE_BUILD_TYPE: buildType,
       REALM_ENABLE_SYNC: "ON",
       CPACK_SYSTEM_NAME: cpackSystemName,
-      CMAKE_TOOLCHAIN_FILE: '%WORKSPACE%/tools/vcpkg/ports/scripts/buildsystems/vcpkg.cmake',
-      VCPKG_MANIFEST_DIR: '%WORKSPACE%/tools/vcpkg',
-      VCPKG_OVERLAY_TRIPLETS: '%WORKSPACE%/tools/vcpkg/triplets',
-      // set a custom buildtrees path because the default one is too long and msbuild tasks fail
-      VCPKG_INSTALL_OPTIONS: '--x-buildtrees-root=%WORKSPACE%/vcpkg-buildtrees',
-      VCPKG_TARGET_TRIPLET: triplet,
       REALM_VERSION: gitDescribeVersion,
     ]
 

diff --git a/Package.swift b/Package.swift
@@ -3,7 +3,7 @@
 import PackageDescription
 import Foundation
 
-let versionStr = "13.1.0"
+let versionStr = "13.2.0"
 let versionPieces = versionStr.split(separator: "-")
 let versionCompontents = versionPieces[0].split(separator: ".")
 let versionExtra = versionPieces.count > 1 ? versionPieces[1] : ""
@@ -102,6 +102,11 @@ let notSyncServerSources: [String] = [
     "realm/sync/instruction_applier.cpp",
     "realm/sync/instruction_replication.cpp",
     "realm/sync/instructions.cpp",
+    "realm/sync/network/default_socket.cpp",
+    "realm/sync/network/http.cpp",
+    "realm/sync/network/network.cpp",
+    "realm/sync/network/network_ssl.cpp",
+    "realm/sync/network/websocket.cpp",
     "realm/sync/noinst/changeset_index.cpp",
     "realm/sync/noinst/client_history_impl.cpp",
     "realm/sync/noinst/client_impl_base.cpp",

diff --git a/dependencies.list b/dependencies.list
@@ -1,4 +1,5 @@
 PACKAGE_NAME=realm-core
-VERSION=13.1.0
-OPENSSL_VERSION=1.1.1n
+VERSION=13.2.0
+OPENSSL_VERSION=3.0.7
+WIN32_ZLIB_VERSION=1.2.13
 MDBREALM_TEST_SERVER_TAG=2022-10-21
diff --git a/doc/CHANGELOG_template.md b/doc/CHANGELOG_template.md
@@ -7,7 +7,7 @@
 ### Fixed
 * <How do the end-user experience this issue? what was the impact?> ([#????](https://github.com/realm/realm-core/issues/????), since v?.?.?)
 * None.
- 
+
 ### Breaking changes
 * None.
 

diff --git a/evergreen/build_zlib.sh b/evergreen/build_zlib.sh
diff --git a/evergreen/config.yml b/evergreen/config.yml
@@ -59,15 +59,6 @@ functions:
               set_cmake_var compiler_vars CMAKE_CXX_COMPILER PATH $(./evergreen/abspath.sh ${cxx_compiler})
           fi
 
-          if [ -n "${build_zlib|}" ]; then
-              CC="${c_compiler|}" GENERATOR="${cmake_generator|}" \
-                  ./evergreen/build_zlib.sh \
-                      -p zlib_prefix \
-                      -b v1.2.11 \
-                      -e "${extra_flags}" \
-                      -j ${max_jobs|$(grep -c proc /proc/cpuinfo)}
-          fi
-
           if [ -n "${run_tests_against_baas|}" ]; then
               set_cmake_var baas_vars REALM_ENABLE_AUTH_TESTS BOOL On
               set_cmake_var baas_vars REALM_MONGODB_ENDPOINT STRING "http://localhost:9090"
@@ -570,7 +561,7 @@ tasks:
 
 - name: object-store-tests
   tags: [ "test_suite", "for_pull_requests" ]
-  exec_timeout_secs: 2100
+  exec_timeout_secs: 2400
   commands:
   - func: "compile"
     vars:
@@ -601,7 +592,7 @@ tasks:
                 export DEVELOPER_DIR="${xcode_developer_dir}"
             fi
 
-            ./evergreen/install_baas.sh -w ./baas-work-dir -b master 2>&1 | tee install_baas_output.log
+            ./evergreen/install_baas.sh -w ./baas-work-dir -b 9398abf62ad6876c38cc675bce47b4f96786a1d6 2>&1 | tee install_baas_output.log
         fi
 
   - command: shell.exec
@@ -1017,7 +1008,6 @@ buildvariants:
     extra_flags: "-A x64"
     max_jobs: $(($(grep -c proc /proc/cpuinfo) / 2))
     fetch_missing_dependencies: On
-    build_zlib: On
     python3: "/cygdrive/c/python/python37/python.exe"
   tasks:
   - name: compile_test_and_package
@@ -1033,7 +1023,6 @@ buildvariants:
     extra_flags: "-A x64"
     max_jobs: $(($(grep -c proc /proc/cpuinfo) / 2))
     fetch_missing_dependencies: On
-    build_zlib: On
     python3: "/cygdrive/c/python/python37/python.exe"
     run_with_encryption: On
   tasks:
@@ -1050,7 +1039,6 @@ buildvariants:
     cmake_build_type: "Release"
     max_jobs: $(($(grep -c proc /proc/cpuinfo) / 2))
     fetch_missing_dependencies: On
-    build_zlib: On
     python3: "/cygdrive/c/python/python37/python.exe"
   tasks:
   - name: compile_test