Omit BEGIN/COMMIT statements for empty transactions #32647

eugeneius · 2018-04-19T20:37:52Z

If a transaction is opened and closed without any queries being run, we can safely omit the BEGIN and COMMIT statements, as they only exist to modify the connection's behaviour inside the transaction. This removes the overhead of those statements when saving a record with no changes, which makes workarounds like save if changed? unnecessary.

This implementation wraps the raw database connection in a proxy which buffers begin_transaction calls, and materializes them the next time the connection is used, or discards them if end_transaction is called first. This approach makes new connection usage inside the adapters behave correctly by default. Third party adapters can use the begin_transaction/end_transaction APIs to opt in to the new behaviour, but will still work even if they don't.

This implementation buffers transactions inside the transaction manager and materializes them the next time the connection is used. For this to work, the adapter needs to guard all connection use with a call to materialize_transactions. Because of this, adapters must opt in to get this new behaviour by implementing supports_lazy_transactions?.

If raw_connection is used to get a reference to the underlying database connection, the behaviour is disabled and transactions are opened eagerly, as we can't know how the connection will be used. However when the connection is checked back into the pool, we can assume that the application won't use the reference again and reenable lazy transactions. This prevents a single raw_connection call from disabling lazy transactions for the lifetime of the connection.

rails-bot · 2018-04-19T20:37:54Z

r? @pixeltrix

(@rails-bot has picked a reviewer for you, use r? to override)

rafaelfranca · 2018-04-19T21:13:40Z

activerecord/lib/active_record/connection_adapters/mysql2_adapter.rb

@@ -112,7 +112,7 @@ def discard! # :nodoc:
      private

        def connect
-          @connection = Mysql2::Client.new(@config)
+          @connection = LazyTransactionProxy.new(Mysql2::Client.new(@config))


I think we can push this lazy proxy creation up to the abstract class and call that method to avoiding having to remember to wrap it every time a driver connection is create.

👍 I've added a private connection= method to the abstract adapter that wraps the connection in a proxy, and changed the concrete adapters to use it when they create a connection.

yahonda · 2018-04-20T01:38:56Z

@eugeneius Let me take a look at if it "breaks" Oracle enhanced adapter. If it does I am happy to update Oracle enhanced adapter.

eugeneius · 2018-04-20T01:55:41Z

Thanks @yahonda! 🙌 I've tried to make the implementation backwards compatible, so please let me know if it does break - it might mean that I made an incorrect assumption about the adapter interface.

eugeneius · 2018-04-20T03:08:09Z

This test failure was legitimate: https://travis-ci.org/rails/rails/jobs/368924286#L1171-L1175

Executing a prepared statement doesn't touch the connection object (at least on the SQLite adapter), but does require the connection's transaction state to be up to date.

I've fixed the problem by materializing transactions in exec_query before prepared statements are executed, but that weakens the "it just works" property that I hoped to get from the proxy approach.

😕

kbrock · 2018-04-22T01:17:13Z

Thanks @eugeneius This turned out very nicely

/cc @Fryguy

eugeneius · 2018-04-22T05:24:16Z

I've moved the materialize_transactions call from exec_query into a proxy wrapping the statement object (more proxies!), which feels a bit more robust.

sgrif

This feels like a lot of complexity for very little gain, but the implementation seems fine.

sgrif · 2018-04-22T23:08:11Z

activerecord/lib/active_record/connection_adapters/abstract/lazy_transaction_proxy.rb

+module ActiveRecord
+  module ConnectionAdapters
+    class LazyTransactionProxy < SimpleDelegator # :nodoc:
+      class StatementProxy < SimpleDelegator


Does it make sense to use DelegateClass instead for both of these?

I don't think so, because the type of object they wrap depends on the adapter in use; there's no single class we can provide with the full set of methods they need to support.

This comment did make me realise that I forgot to :nodoc: the inner class, though.

yahonda · 2018-04-23T11:43:14Z

Oracle enhanced adapter needs some update to support this pull request. Here are CI results for this pull request https://travis-ci.org/yahonda/oracle-enhanced/builds/370051142 .

I am still having some jet lags due after a long flight, I'll take a look at this result in detail tomorrow.

sgrif · 2018-04-23T16:56:03Z

activerecord/lib/active_record/connection_adapters/abstract/lazy_transaction_proxy.rb

+          @connection = connection
+        end
+
+        def method_missing(*)


This needs to implement respond_to_missing? as well

Delegator already provides an implementation of respond_to_missing?:
https://github.com/ruby/ruby/blob/v2_4_4/lib/delegate.rb#L95-L104

Since we call super in method_missing, we should do the same for respond_to_missing?:

def respond_to_missing?(*) super end

I've added this to both classes, if only to document that using Delegator's behaviour unmodified is intentional.

I'm a bit confused as to why we need to call materialize_transactions in method_missing, but not respond_to_missing?. These two methods should always be in sync.

The contract between the two methods is that respond_to_missing? will return true iff called with a method that method_missing would dispatch. Usually this means making equivalent changes to both methods to keep them in sync, as you've noted. In this case however, calling materialize_transactions in method_missing doesn't affect which methods are dispatched; it's purely a side effect, and so doesn't require an equivalent change in respond_to_missing?.

Another way of looking at it: we need to call materialize_transactions in method_missing because we don't know whether the call to super will touch the socket. We don't need to call it in respond_to_missing? because we know that calling super won't touch the socket.

sgrif · 2018-04-23T16:56:16Z

activerecord/lib/active_record/connection_adapters/abstract/lazy_transaction_proxy.rb

+        __getobj__
+      end
+
+      def method_missing(*)


This needs to implement respond_to_missing? as well

eugeneius · 2018-04-23T20:42:28Z

@yahonda can you try triggering another build? I renamed LazyTransactionProxy#raw_connection to LazyTransactionProxy#proxied_connection so it wouldn't clash with this method.

matthewd · 2018-04-23T23:04:40Z

A few thoughts from an initial read-through:

Yay! ❤️ As I mentioned in Pittsburgh, it's awesome to have an implementation to look at -- this one's been sitting on the "nice to have" list for a while.

My gut feeling is that this "should" be closer to the transaction manager rather than the connection -- particularly the deferral stack, though the connection of course needs to be responsible for triggering the materialize.

Failing that, I also wonder about putting the functionality on the abstract connection itself rather than a proxy. I don't mind the code complexity needed for more thorough bookkeeping, but the proxy feels architecturally heavyweight.

Side-thought that would probably just distract us to pursue right now, but I'll mention anyway: this has some interesting parallels (deferring work on the connection until we need to run a real statement, and then potentially giving that some special handling) with #28200 and friends.

Finally, the big one, because it's about behaviour rather than implementation: I believe it's important that savepoints don't materialize the transaction -- that they too get deferred. To me, the ideal version of this change allows us to create transactions more freely, which means we can default requires_new to true, eliminating some very surprising behaviours where changes escape a transaction block. Reducing a no-op save from two queries to zero is great, of course.. I just see some other shinies in the distance.

yahonda · 2018-04-24T00:09:32Z

CI is green now with Oracle enhanced adapter. As a nature of Oracle database transaction management begin and end statements are not executed to start and finish transaction. Then some of newly added tests get "failed" but it is okay for me.

jeremy · 2018-08-23T14:00:13Z

🚀🌈🚀🌈🚀😊

…

On Thu, Aug 23, 2018 at 03:10 Matthew Draper ***@***.***> wrote: Sorry I left this sitting for ages, and thanks again! ❤️ 💚 💙 💛 💜 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#32647 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAAx-Yub-sVym022pHU6tES3UPfhBMaks5uTn-hgaJpZM4Tcdd-> .

kbrock · 2018-08-23T14:14:36Z

Thanks so much Matthew and Eugene and all others who took a stab at this and helped it go forward

Nondv · 2018-08-23T15:19:19Z

Wow. I was just wondering about omiting empty transactions and found out that it was merged several hours ago.

Thank you, rails community. You are awesome <3

`test_update_does_not_run_sql_if_record_has_not_changed` would pass without #18501 since `assert_queries` ignores BEGIN/COMMIT unless `ignore_none: true` is given. Since #32647, empty BEGIN/COMMIT is ommited. So we no longer need to use `assert_queries(0)` to ignore BEGIN/COMMIT in the queries.

It broke due new to [Lazy transactions](rails/rails#32647) feature. Workaround is taken from the Isolator gem: palkan/isolator#20

f3ndot · 2019-09-11T15:33:53Z

I would love if this would be backported to 5.2. Is that something not out of the realm of possibility?

eugeneius · 2019-09-11T20:22:44Z

I don't think so; 5.2 is only supported for security fixes at this point.

pixeltrix · 2019-09-15T09:50:48Z

@eugeneius I'm pretty sure that 5.2 is still receiving some bug fixes but they'll be limited in scope unless warranted by a new Ruby version or some other similar event (e.g. we backported a whole bunch of stuff to Rails 4.2 to make it work with Ruby 2.4's changes to to_time).

However, even given the above I agree that it's unlikely to be backported unless something else makes it necessary.

Omit BEGIN/COMMIT statements for empty transactions rails/rails#32647

for implementing supports for lazy transactions rails/rails#32647 (comment)

* Support lazy transactions * aaded materialize_transactions guard to connection uses. for implementing supports for lazy transactions rails/rails#32647 (comment) * Moved materialize_transactions calls * Fix typo * Materialize transaction in do_execute Co-authored-by: Yehuda Goldberg <[email protected]>

Rails 6 adds support for lazy transactions: rails/rails#32647 Specifically, it has all adapters default to claiming to be able to support lazy transactions, and adds logic in them to actually support it. Unfortunately, that means that the SeamlessDatabasePool adapter is now also claiming to be able to support lazy transactions, despite not having any logic added to it to do so. We could probably go in and teach it how to correctly support lazy transactions, but since we don't currently require that functionality and because we plan to migrate off of SeamlessDatabasePool, we instead simply update the adapter to explicitly NOT support lazy transactions.

It broke due new to [Lazy transactions](rails/rails#32647) feature. Workaround is taken from the Isolator gem: palkan/isolator#20

rails-bot assigned pixeltrix Apr 19, 2018

eugeneius force-pushed the lazy_transactions branch from 4ee21ad to 710c810 Compare April 19, 2018 20:48

rafaelfranca reviewed Apr 19, 2018

View reviewed changes

eugeneius force-pushed the lazy_transactions branch 3 times, most recently from e674c1e to 2d8da6d Compare April 20, 2018 01:07

eugeneius force-pushed the lazy_transactions branch 3 times, most recently from b08216a to 2839b04 Compare April 20, 2018 14:40

eugeneius force-pushed the lazy_transactions branch from 2839b04 to 47fbef6 Compare April 22, 2018 05:09

jeremy requested review from matthewd, kamipo and sgrif April 22, 2018 21:40

sgrif approved these changes Apr 22, 2018

View reviewed changes

eugeneius force-pushed the lazy_transactions branch from 47fbef6 to 4d08d52 Compare April 23, 2018 00:55

yahonda added a commit to yahonda/oracle-enhanced that referenced this pull request Apr 23, 2018

Run Oracle enhanced adapter CI for rails/rails#32647

eac9713

sgrif reviewed Apr 23, 2018

View reviewed changes

eugeneius force-pushed the lazy_transactions branch 2 times, most recently from fb88426 to 16925ad Compare April 23, 2018 20:39

sikachu mentioned this pull request Aug 23, 2018

Rails 5: empty begin commit for no reason rails/activerecord-session_store#134

Closed

georgeclaghorn mentioned this pull request Aug 28, 2018

Request: support Rails 6 lazy transactions instacart/makara#218

Open

y-yagi mentioned this pull request Sep 13, 2018

ActiveRecord issue when retrying transaction due to serialization error #33855

Closed

y-yagi mentioned this pull request Sep 24, 2018

Since save is wrapped in a transaction, it seems that unnecessary transactions occur when simple validations fail #33971

Closed

palkan mentioned this pull request Apr 11, 2019

Rails 6 compatibility palkan/isolator#20

Merged

yahonda mentioned this pull request Jul 1, 2019

[WIP] Support ActiveRecord::ReadOnlyError rsim/oracle-enhanced#1898

Closed

Envek mentioned this pull request Sep 10, 2019

Rails 6 compatibility Envek/after_commit_everywhere#6

Merged

kamipo mentioned this pull request Sep 20, 2019

Rails 6 regression in after_rollback callback not called for new records #36965

Open

JesseChavez added a commit to JesseChavez/activerecord-jdbc-adapter that referenced this pull request Dec 19, 2019

fix transaction and savepoint AR tests, issues caused by a pull request

31c39a2

Omit BEGIN/COMMIT statements for empty transactions rails/rails#32647

aidanharan mentioned this pull request Apr 23, 2020

Rails 6: Support lazy transactions rails-sqlserver/activerecord-sqlserver-adapter#780

Merged

YehudaGold mentioned this pull request Apr 23, 2020

skip tests if adapter not supports lazy transactions #39028

Closed

YehudaGold added a commit to YehudaGold/activerecord-sqlserver-adapter that referenced this pull request Apr 24, 2020

aaded materialize_transactions guard to connection uses.

6b31617

for implementing supports for lazy transactions rails/rails#32647 (comment)

aidanharan mentioned this pull request Apr 24, 2020

aaded materialize_transactions guard to connection uses. aidanharan/activerecord-sqlserver-adapter#1

Merged

aidanharan pushed a commit to aidanharan/activerecord-sqlserver-adapter that referenced this pull request Apr 27, 2020

aaded materialize_transactions guard to connection uses.

03b5480

for implementing supports for lazy transactions rails/rails#32647 (comment)

Yoshiji mentioned this pull request Dec 16, 2020

Empty previous_refresh_token if present? doorkeeper-gem/doorkeeper#1452

Merged

Hamms mentioned this pull request Jan 13, 2021

Specify that TestCase transaction should not be lazy code-dot-org/code-dot-org#38531

Merged

8 tasks

Hamms mentioned this pull request Dec 9, 2021

Disable lazy transactions in SeamlessDatabasePool code-dot-org/code-dot-org#44001

Closed

8 tasks

matthewd mentioned this pull request Mar 23, 2022

[Tests only] Flunk if test is not using SavepointTransaction #44686

Merged

shallow1729 mentioned this pull request Apr 29, 2022

Query order processed by arproxy is different from the order actually executed if lazy_transactions is enabled cookpad/arproxy#23

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Omit BEGIN/COMMIT statements for empty transactions #32647

Omit BEGIN/COMMIT statements for empty transactions #32647

eugeneius commented Apr 19, 2018 •

edited

Loading

rails-bot commented Apr 19, 2018

rafaelfranca Apr 19, 2018

eugeneius Apr 19, 2018 •

edited

Loading

yahonda commented Apr 20, 2018

eugeneius commented Apr 20, 2018

eugeneius commented Apr 20, 2018

kbrock commented Apr 22, 2018

eugeneius commented Apr 22, 2018

sgrif left a comment

sgrif Apr 22, 2018

eugeneius Apr 23, 2018

eugeneius Apr 23, 2018

yahonda commented Apr 23, 2018

sgrif Apr 23, 2018

eugeneius Apr 23, 2018

sgrif Apr 24, 2018

eugeneius Apr 24, 2018

sgrif Apr 23, 2018

eugeneius commented Apr 23, 2018

matthewd commented Apr 23, 2018

yahonda commented Apr 24, 2018

jeremy commented Aug 23, 2018 via email

kbrock commented Aug 23, 2018 •

edited

Loading

Nondv commented Aug 23, 2018

f3ndot commented Sep 11, 2019

eugeneius commented Sep 11, 2019

pixeltrix commented Sep 15, 2019

Omit BEGIN/COMMIT statements for empty transactions #32647

Omit BEGIN/COMMIT statements for empty transactions #32647

Conversation

eugeneius commented Apr 19, 2018 • edited Loading

rails-bot commented Apr 19, 2018

Choose a reason for hiding this comment

eugeneius Apr 19, 2018 • edited Loading

Choose a reason for hiding this comment

yahonda commented Apr 20, 2018

eugeneius commented Apr 20, 2018

eugeneius commented Apr 20, 2018

kbrock commented Apr 22, 2018

eugeneius commented Apr 22, 2018

sgrif left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yahonda commented Apr 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eugeneius commented Apr 23, 2018

matthewd commented Apr 23, 2018

yahonda commented Apr 24, 2018

jeremy commented Aug 23, 2018 via email

kbrock commented Aug 23, 2018 • edited Loading

Nondv commented Aug 23, 2018

f3ndot commented Sep 11, 2019

eugeneius commented Sep 11, 2019

pixeltrix commented Sep 15, 2019

eugeneius commented Apr 19, 2018 •

edited

Loading

eugeneius Apr 19, 2018 •

edited

Loading

kbrock commented Aug 23, 2018 •

edited

Loading