Replacing the Schema by a proxy #407

mikeSimonson · 2015-12-30T15:47:23Z

This PR try to explore the possibility of using Proxy object instead of the costly initialization of the schema object.

mikeSimonson · 2015-12-30T15:48:33Z

@Ocramius ?

Ocramius · 2015-12-30T15:49:28Z

@mikeSimonson the first run was just pseudo-code

mikeSimonson · 2015-12-30T15:57:18Z

it needs a first run to warm up some sort of cache ?

Ocramius · 2015-12-30T15:59:26Z

@mikeSimonson you didn't import any of the proxy lib stuff :-)

stof · 2015-12-30T16:08:32Z

lib/Doctrine/DBAL/Migrations/LazySchemaManipulator.php

+use ProxyManager\Factory\LazyLoadingGhostFactory;
+use ProxyManager\Proxy\LazyLoadingInterface;
+
+class LazySchemaManipulator


not implementing a common interface looks wrong to me

Agreed. The name also needs fixing: we just mashed it together real quick in a video call yesterday :)

Any idea on a better naming ? for the class and the interface ?

stof · 2015-12-30T16:15:16Z

@mikeSimonson I guess this is an alternative to #323.
I see 1 drawback: if the user makes any call to the Schema by mistake, they would go to the slower codebase even though they might not want it. My proposal makes it an explicit choice, making people aware that they are going the slower way. What do you think ?

stof · 2015-12-30T16:16:45Z

lib/Doctrine/DBAL/Migrations/SchemaManipulator.php

+    /** @var  AbstractSchemaManager */
+    private $schemaManager;
+
+    public function _construct(AbstractSchemaManager $schemaManager, $platform)


missing _, so this is not a constructor. This is why you are calling on null

@mikeSimonson please fix this so that tests can run for real

stof · 2015-12-30T16:32:38Z

@mikeSimonson the PR should describe what it really does IMO, not the issue you are facing currently

mikeSimonson · 2016-01-02T15:35:16Z

@stof The problem with your solution is that right now most of the people wouldn't benefit from it.
But it's definitely the way to go and the way it will be for v2 (1 interface that allow the schema and one that don't).

Can you look again at the last change I made.
I think that I took care of the issue with the double instantiation instead of the cloning.
All your others remarks should be fixed too except for the interface name.

I was thinking to something along the line of SchemaDiffProvider or SchemaProvider.
Do you have an opinion ?

Thanks for your review

mikeSimonson · 2016-01-05T00:34:39Z

@stof Do you have some time to look at this again ?

stof · 2016-01-05T16:39:32Z

lib/Doctrine/DBAL/Migrations/LazySchemaDiffProvider.php

+                Schema::class,
+                function (& $wrappedObject, $proxy,  $method, array $parameters, & $initializer) use ($originalSchemaManipulator, $fromSchema) {
+                    $initializer   = null;
+                    $wrappedObject = $originalSchemaManipulator->createToSchema($fromSchema);


shouldn't you force initializing the fromSchema here before passing it to the originalSchemaManipulator ?

The createToSchema method clone the fromSchema and the clone trigger the initialization.

then it is good

stof · 2016-01-05T18:45:43Z

@mikeSimonson it is great to see the build being green here.

stof · 2016-01-06T09:37:48Z

lib/Doctrine/DBAL/Migrations/Provider/SchemaDiffProviderInterface.php

+ * <http://www.doctrine-project.org>.
+ */
+
+namespace Doctrine\DBAL\Migrations\Provider;


why is it in a subnamespace while implementations are not ?

no valid reason

mikeSimonson · 2016-01-06T14:57:59Z

@stof @Ocramius Ok I think that this can be merged now ?

…if not needed)

in the pre method

Replacing the Schema by a proxy

mikeSimonson · 2016-01-07T09:45:29Z

@stof Do you think that you could do a profile of v1.2.1 against master to see the benefits ?

stof · 2016-01-07T22:18:07Z

Here you go. This is the result of running 109 migrations on PostgreSQL 9.3 and PHP 5.6.16 on my laptop (with XDebug enabled as I forgot to disable it before profiling):

Doctrine 1.2.1 https://blackfire.io/profiles/f96bd1bc-4539-4158-a4d6-e1823c5eea46/graph
Doctrine 1.3.0-dev (9aa590e) https://blackfire.io/profiles/4a377dc2-811d-40f7-845b-06db1198fc4c/graph
And here is the comparison: https://blackfire.io/profiles/compare/2be3159d-2821-4018-9a20-a80215695dd4/graph

We have a 97% performance improvement here (from 2min20s to 4.68s). This is similar to the 94% I reported in #323 (the difference is probably related to the fact that I have more migrations than at that time, making the effect even bigger)

Ocramius · 2016-01-07T22:21:33Z

That is quite... impressive :O

@mikeSimonson still, are you sure that we want proxymanager installed as hard dependency?

stof · 2016-01-07T22:22:56Z

@Ocramius I think we do (I'm looking forward the release of a zend-code release without the zend-stdlib dependency though, to have less transitive deps)

Ocramius · 2016-01-07T22:30:00Z

As far as I know, it doesn't require stdlib, but the eventmanager: https://github.com/zendframework/zend-code/blob/4972c33d975fa0c2b86ff381e83fde177079b249/composer.json#L17

That can't be fixed until we drop annotations support ( zendframework/zend-code#32 ), which will only happen in a major revision.

Making it an optional dependency without dropping that feature will just lead to a lot of people having headaches when they want to actually use the annotations support :-\

stof · 2016-01-07T22:40:16Z

There is a drawback in this change though: look at the metrics about the GC runs. We went from 1 to 81.
I think this is related to a circular object graph in unitialized proxies. It may be worth investigating it to see whether we can break the object graph when we don't need the proxies anymore (at the end of each migration). But the time spent in GC is not comparable to the time win, so this does not block a 1.3.0 release.

stof · 2016-01-07T22:44:38Z

And regarding the dependencies, I think a 97% perf increase justifies the mandatory dependency. If we make it optional, people would have to opt-in the fast way, which would not be much better than #323 (well, except the opt-in would be in a single place), and we would have to expose this layer outside the library (currently, it is hidden from the outside, allowing us to treat it as fully internal)

Ocramius · 2016-01-07T22:48:08Z

It may be worth investigating it to see whether we can break the object graph when we don't need the proxies anymore (at the end of each migration).

The initializer closures will obviously cause cyclic graphs. There is a way to "fix that" (simply setInitializer(null) on proxies), but:

it breaks the abstraction
obviously isn't that relevant

Agree on all the conclusions.

stof · 2016-01-07T22:49:14Z

@Ocramius the idea is to reset the initializer at the end of the migration, when we don't need the Schema object anymore (the refcount will then destroy it)

Ocramius · 2016-01-07T22:50:55Z

@stof I wouldn't do that anyway, as it may lead to unexpected bugs (in case the objects are needed after migration run, or are used in any other context)

Ocramius · 2016-01-07T22:53:04Z

@mikeSimonson this patch still lacks some tests that are required to prevent any regression that may by accident initialize a migration. I know it is a pain, but that means extremely detailed mocking :-\

Tobion · 2016-01-08T01:53:58Z

lib/Doctrine/DBAL/Migrations/Provider/SchemaDiffProviderInterface.php

+     *
+     * @return string[]
+     */
+    public function getSqlDiffToMigrate(Schema $fromSchema,Schema $toSchema);


missing space

mikeSimonson · 2016-01-08T11:36:53Z

@Ocramius What sort of test do you think are missing ?

mikeSimonson · 2016-01-08T13:34:37Z

@stof detail but when I look at the profile of the future 1.3 and that I go in the Metrics tab I see one call to GC et 81 for the profile of the 1.2. But in the comparison page in the metrics tab the numbers seems inverted. Same for the number of PDO and SQL queries. Bug in blackfire ?

Btw it make more sense to me that there would be less call to the GC with the Proxy because we are not instantiating all those Tables objects in the Schema.

Ocramius · 2016-01-08T15:16:52Z

@mikeSimonson the tests that I'm referring to would expect the schema object methods to never be called: basically a set of $schema->expects(self::never())->method('theMethod') for all methods.

mikeSimonson · 2016-01-08T15:59:14Z

@Ocramius I don't understand the goal this test ? Do you mean testing that none of the method of the schema are called in the pre and post method or something else ?

Ocramius · 2016-01-08T16:04:29Z

@mikeSimonson correct. That is just to prevent regressions though.

stof · 2016-01-08T16:05:56Z

Bug in blackfire ?

Looks like it is. then everything is good (and the issue would then be about the Schema system in DBAL relying on cyclic object graphs, which is off-topic)

stof · 2016-01-08T16:08:04Z

I reported the bug to the Blackfire team

mikeSimonson · 2016-01-08T16:09:10Z

@Ocramius There are already tests for that. It test that if those methods are used it has no impact on the sql produced fe1447a.
@stof Thanks

Ocramius · 2016-01-08T16:10:40Z

@mikeSimonson that test is still calling API methods on the schema objects - a test where no method calls on the schema ever happen is required

mikeSimonson · 2016-01-09T10:21:06Z

@Ocramius A test like that one ? #414

stof reviewed Dec 30, 2015
View reviewed changes

mikeSimonson changed the title ~~the test are failing with a different error on first and second run~~ Replacing the Schema by a proxy Jan 1, 2016

stof reviewed Jan 5, 2016
View reviewed changes

stof reviewed Jan 6, 2016
View reviewed changes

Ocramius and others added 5 commits January 6, 2016 21:52

Simple prototype of lazy schema manipulation (schema not initialized …

c7a63fe

…if not needed)

WIP lazyload the schema

6e0c1df

WIP

ff2bcef

Adding type hint and correcting dockblocks

98a016b

Not taking into account the from proxy as he might not have been used

f74cdc7

in the pre method

mikeSimonson added a commit that referenced this pull request Jan 7, 2016

Merge pull request #407 from doctrine/prototype/lazy-schema-manipulator

4461bf9

Replacing the Schema by a proxy

mikeSimonson merged commit 4461bf9 into master Jan 7, 2016

stof deleted the prototype/lazy-schema-manipulator branch January 7, 2016 21:56

stof mentioned this pull request Jan 7, 2016

Add a way to avoid reading the existing schema before each migration #323

Closed

4 tasks

Tobion reviewed Jan 8, 2016
View reviewed changes

chrisguitarguy mentioned this pull request May 5, 2017

Strange behavior on migrate: get an error "Unknown database type" #525

Closed

stof mentioned this pull request Jun 26, 2019

Less verbose output for the migrate command #834

Closed

Replacing the Schema by a proxy #407

Replacing the Schema by a proxy #407

Conversation

mikeSimonson commented Dec 30, 2015

mikeSimonson commented Dec 30, 2015

Ocramius commented Dec 30, 2015

mikeSimonson commented Dec 30, 2015

Ocramius commented Dec 30, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stof commented Dec 30, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stof commented Dec 30, 2015

mikeSimonson commented Jan 2, 2016

mikeSimonson commented Jan 5, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stof commented Jan 5, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikeSimonson commented Jan 6, 2016

mikeSimonson commented Jan 7, 2016

stof commented Jan 7, 2016

Ocramius commented Jan 7, 2016

stof commented Jan 7, 2016

Ocramius commented Jan 7, 2016

stof commented Jan 7, 2016

stof commented Jan 7, 2016

Ocramius commented Jan 7, 2016

stof commented Jan 7, 2016

Ocramius commented Jan 7, 2016

Ocramius commented Jan 7, 2016

Choose a reason for hiding this comment

mikeSimonson commented Jan 8, 2016

mikeSimonson commented Jan 8, 2016

Ocramius commented Jan 8, 2016

mikeSimonson commented Jan 8, 2016

Ocramius commented Jan 8, 2016

stof commented Jan 8, 2016

stof commented Jan 8, 2016

mikeSimonson commented Jan 8, 2016

Ocramius commented Jan 8, 2016

mikeSimonson commented Jan 9, 2016