[ENG-3651] Django3 Upgrade (#10078)

PRs:
* remove post-migrate signals from migration stream (#9964)
* move createcachetable to post migrate signal (#9944)
* [ENG-3836] Investigate waffle flags (#9950)
* [ENG-3865] Create Licenses using post-migrate signal (#9961)
* [ENG-3863] Move schema ensuring and schema blocks update to post-migrate signals (#9974)
* [ENG-3868] Move blocked email domains to post-migrate signal (#9958)
* [ENG-3866] Move citation style population out of migration stream (#9966)
* [ENG-3867] Create Storage Regions on post migration signal (#9965)
* [ENG-3836] Follow-up: Add PyYAML to Requirements (#9993)
* [ENG-3862] Move post-migrate signal out of migration stream for default providers (#9971)
* Remove unused password change view/route (#9985)
* Update import paths and fix deprecated modules for Django 3 (#9983)
* Update postgres backend and remove failover router for Django 3 (#10011)
* Move file metadata population to unit test fixtures (#9994)
* [Django Upgrade] Fix one sub-query slicing issue (#10012)
* [Django Upgrade] Upgrade django-elasticsearch-metrics (#10021)
* [Django Upgrade] Replace M2M direct assignment  (#10010)
* [Django Upgrade] Upgrade markdown and mdx_del_ins (#9984)
* [Django Upgrade] Move schema activeness/visibility update to pytest fixtures (#10029)

Commits:
* Remove osf migrations
* Remove addon migrations
* Init migrations - osf
* Init migrations - addons
* Fix migration for NotableEmailDomain
* Fix circular import with built-in admin migrations
* Add a RunSQL migration to update indexes
* Upgrade django to 3.2.15 (and related dependencies)
* Make fields defined on typedmodels subclasses nullable
* Add required on_delete=CASCADE (default) to FKs missing it
* Fix django CORS whitelist settings due to origin definition update
* Add/Enable missing templates and middlewares in api settings
* Comment out admin permissions that clash with built-in "view" ones
* Add migrations for django upgrade base fixes
* Fix system check warnings for django admin
* Fix django cache table creation
* Enable django.contrib.sessions.middleware.SessionMiddleware
* Use default truthy and falsy values from upgraded DRF 3.13.1
* Use JsonField from django.db.models and django.forms
* Fix session cookie encoding/decoding by using ensure_str()
* Use prefetch instead of Django include
* Fix CORS for ORIGINS_WHITELIST
* Rename auto generated migration for django3 upgrade fixes
* Remove QuickFile check from get_serializer_class for FileDetail view
* Add registration_schema to DraftRegistrationDetailSerializer
* Add id to DraftRegistrationDetailSerializer
* Fix middleware order and replace deprecated staticfiles with static
* Fix and rework from_db_value & to_python for EncryptedJSONField
* Fix admin login failure for django upgrade
* Wrap generated keen_read_key with ensure_str
* Clear cached storage region property when creating a new version
* Do not notify in set_password() when creating a test user
* Fix RegistrationFactory for django3 upgrade
* Remove unused update_version_metadata in file node and update tests
* Remove tests for django-include
* Remove EGAP backfill command and tests
* Set DEFAULT_AUTO_FIELD to django.db.models.AutoField (default value)
* Replace deprecated NullBooleanField + add migrations
* Replace deprecated url() with re_path() for api
* Replace deprecated url() with re_path() for admin
* Fix missing or insufficient permission test for admin preprints view
* Fix admin user view tests by using HTTP response headers object
* Fix duplicate view_node permissions in node and instn view tests
* Rework refresh_from_db() to reload GFKs
* Fix visible contributor query
* Fix handle_archive_fail and its tests
* Use a different schema to fix tests failure due to django3 upgrade
* Update URLs for URL Validation tests with Django 3
* Improve to_internal_value() in NodeRelationshipField
* Add alt fixtures to prevent both siblings & parent/child conflicts
* Fix linked_by_nodes in node related counts test
* Fix bugged NullBooleanField -> BooleanField migration
* Re-make migrations for django3 upgrade and merge them into one

Co-authored-by: Longze Chen <[email protected]>
Co-authored-by: John Tordoff <[email protected]>
Co-authored-by: Jon Walz <[email protected]>

addons/bitbucket/migrations/0001_initial.py

@@ -1,56 +1,55 @@
 # -*- coding: utf-8 -*-
-# Generated by Django 1.11.2 on 2017-07-07 15:19
+# Generated by Django 1.11.29 on 2022-08-17 19:15
 from __future__ import unicode_literals
 
-from django.conf import settings
+import addons.base.models
 from django.db import migrations, models
-import django.db.models.deletion
+import django_extensions.db.fields
 import osf.models.base
 import osf.utils.datetime_aware_jsonfield
+import osf.utils.fields
 
 
 class Migration(migrations.Migration):
 
     initial = True
 
     dependencies = [
-        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
-        ('osf', '0042_auto_20170707_1019'),
     ]
 
     operations = [
         migrations.CreateModel(
             name='NodeSettings',
             fields=[
                 ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('created', django_extensions.db.fields.CreationDateTimeField(auto_now_add=True, verbose_name='created')),
+                ('modified', django_extensions.db.fields.ModificationDateTimeField(auto_now=True, verbose_name='modified')),
                 ('_id', models.CharField(db_index=True, default=osf.models.base.generate_object_id, max_length=24, unique=True)),
-                ('deleted', models.BooleanField(default=False)),
+                ('is_deleted', models.BooleanField(default=False)),
+                ('deleted', osf.utils.fields.NonNaiveDateTimeField(blank=True, null=True)),
                 ('user', models.TextField(blank=True, null=True)),
                 ('repo', models.TextField(blank=True, null=True)),
                 ('hook_id', models.TextField(blank=True, null=True)),
-                ('external_account', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='addons_bitbucket_node_settings', to='osf.ExternalAccount')),
-                ('owner', models.OneToOneField(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='addons_bitbucket_node_settings', to='osf.AbstractNode')),
             ],
             options={
                 'abstract': False,
             },
+            bases=(models.Model, osf.models.base.QuerySetExplainMixin, addons.base.models.BaseStorageAddon),
         ),
         migrations.CreateModel(
             name='UserSettings',
             fields=[
                 ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('created', django_extensions.db.fields.CreationDateTimeField(auto_now_add=True, verbose_name='created')),
+                ('modified', django_extensions.db.fields.ModificationDateTimeField(auto_now=True, verbose_name='modified')),
                 ('_id', models.CharField(db_index=True, default=osf.models.base.generate_object_id, max_length=24, unique=True)),
-                ('deleted', models.BooleanField(default=False)),
+                ('is_deleted', models.BooleanField(default=False)),
+                ('deleted', osf.utils.fields.NonNaiveDateTimeField(blank=True, null=True)),
                 ('oauth_grants', osf.utils.datetime_aware_jsonfield.DateTimeAwareJSONField(blank=True, default=dict, encoder=osf.utils.datetime_aware_jsonfield.DateTimeAwareJSONEncoder)),
-                ('owner', models.OneToOneField(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='addons_bitbucket_user_settings', to=settings.AUTH_USER_MODEL)),
             ],
             options={
                 'abstract': False,
             },
-        ),
-        migrations.AddField(
-            model_name='nodesettings',
-            name='user_settings',
-            field=models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, to='addons_bitbucket.UserSettings'),
+            bases=(models.Model, osf.models.base.QuerySetExplainMixin),
         ),
     ]

addons/bitbucket/migrations/0002_auto_20220817_1915.py

@@ -0,0 +1,41 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.29 on 2022-08-17 19:15
+from __future__ import unicode_literals
+
+from django.conf import settings
+from django.db import migrations, models
+import django.db.models.deletion
+
+
+class Migration(migrations.Migration):
+
+    initial = True
+
+    dependencies = [
+        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
+        ('addons_bitbucket', '0001_initial'),
+        ('osf', '0001_initial'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='usersettings',
+            name='owner',
+            field=models.OneToOneField(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='addons_bitbucket_user_settings', to=settings.AUTH_USER_MODEL),
+        ),
+        migrations.AddField(
+            model_name='nodesettings',
+            name='external_account',
+            field=models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='addons_bitbucket_node_settings', to='osf.ExternalAccount'),
+        ),
+        migrations.AddField(
+            model_name='nodesettings',
+            name='owner',
+            field=models.OneToOneField(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='addons_bitbucket_node_settings', to='osf.AbstractNode'),
+        ),
+        migrations.AddField(
+            model_name='nodesettings',
+            name='user_settings',
+            field=models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, to='addons_bitbucket.UserSettings'),
+        ),
+    ]

addons/box/migrations/0001_initial.py

@@ -1,10 +1,13 @@
 # -*- coding: utf-8 -*-
-# Generated by Django 1.9 on 2017-03-23 20:34
+# Generated by Django 1.11.29 on 2022-08-17 19:15
 from __future__ import unicode_literals
 
+import addons.base.models
 from django.db import migrations, models
+import django_extensions.db.fields
 import osf.models.base
 import osf.utils.datetime_aware_jsonfield
+import osf.utils.fields
 
 
 class Migration(migrations.Migration):
@@ -19,26 +22,34 @@ class Migration(migrations.Migration):
             name='NodeSettings',
             fields=[
                 ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('created', django_extensions.db.fields.CreationDateTimeField(auto_now_add=True, verbose_name='created')),
+                ('modified', django_extensions.db.fields.ModificationDateTimeField(auto_now=True, verbose_name='modified')),
                 ('_id', models.CharField(db_index=True, default=osf.models.base.generate_object_id, max_length=24, unique=True)),
-                ('deleted', models.BooleanField(default=False)),
+                ('is_deleted', models.BooleanField(default=False)),
+                ('deleted', osf.utils.fields.NonNaiveDateTimeField(blank=True, null=True)),
                 ('folder_id', models.TextField(blank=True, null=True)),
                 ('folder_name', models.TextField(blank=True, null=True)),
                 ('folder_path', models.TextField(blank=True, null=True)),
             ],
             options={
                 'abstract': False,
             },
+            bases=(models.Model, osf.models.base.QuerySetExplainMixin, addons.base.models.BaseStorageAddon),
         ),
         migrations.CreateModel(
             name='UserSettings',
             fields=[
                 ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('created', django_extensions.db.fields.CreationDateTimeField(auto_now_add=True, verbose_name='created')),
+                ('modified', django_extensions.db.fields.ModificationDateTimeField(auto_now=True, verbose_name='modified')),
                 ('_id', models.CharField(db_index=True, default=osf.models.base.generate_object_id, max_length=24, unique=True)),
-                ('deleted', models.BooleanField(default=False)),
-                ('oauth_grants', osf.utils.datetime_aware_jsonfield.DateTimeAwareJSONField(blank=True, default=dict)),
+                ('is_deleted', models.BooleanField(default=False)),
+                ('deleted', osf.utils.fields.NonNaiveDateTimeField(blank=True, null=True)),
+                ('oauth_grants', osf.utils.datetime_aware_jsonfield.DateTimeAwareJSONField(blank=True, default=dict, encoder=osf.utils.datetime_aware_jsonfield.DateTimeAwareJSONEncoder)),
             ],
             options={
                 'abstract': False,
             },
+            bases=(models.Model, osf.models.base.QuerySetExplainMixin),
         ),
     ]

addons/box/migrations/0002_auto_20170323_1534.py → addons/box/migrations/0002_auto_20220817_1915.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Generated by Django 1.9 on 2017-03-23 20:34
+# Generated by Django 1.11.29 on 2022-08-17 19:15
 from __future__ import unicode_literals
 
 from django.conf import settings
@@ -12,9 +12,9 @@ class Migration(migrations.Migration):
     initial = True
 
     dependencies = [
+        ('addons_box', '0001_initial'),
         migrations.swappable_dependency(settings.AUTH_USER_MODEL),
         ('osf', '0001_initial'),
-        ('addons_box', '0001_initial'),
     ]
 
     operations = [

addons/dataverse/migrations/0001_initial.py

@@ -1,10 +1,13 @@
 # -*- coding: utf-8 -*-
-# Generated by Django 1.9 on 2017-03-23 20:34
+# Generated by Django 1.11.29 on 2022-08-17 19:15
 from __future__ import unicode_literals
 
+import addons.base.models
 from django.db import migrations, models
+import django_extensions.db.fields
 import osf.models.base
 import osf.utils.datetime_aware_jsonfield
+import osf.utils.fields
 
 
 class Migration(migrations.Migration):
@@ -19,8 +22,11 @@ class Migration(migrations.Migration):
             name='NodeSettings',
             fields=[
                 ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('created', django_extensions.db.fields.CreationDateTimeField(auto_now_add=True, verbose_name='created')),
+                ('modified', django_extensions.db.fields.ModificationDateTimeField(auto_now=True, verbose_name='modified')),
                 ('_id', models.CharField(db_index=True, default=osf.models.base.generate_object_id, max_length=24, unique=True)),
-                ('deleted', models.BooleanField(default=False)),
+                ('is_deleted', models.BooleanField(default=False)),
+                ('deleted', osf.utils.fields.NonNaiveDateTimeField(blank=True, null=True)),
                 ('dataverse_alias', models.TextField(blank=True, null=True)),
                 ('dataverse', models.TextField(blank=True, null=True)),
                 ('dataset_doi', models.TextField(blank=True, null=True)),
@@ -30,17 +36,22 @@ class Migration(migrations.Migration):
             options={
                 'abstract': False,
             },
+            bases=(models.Model, osf.models.base.QuerySetExplainMixin, addons.base.models.BaseStorageAddon),
         ),
         migrations.CreateModel(
             name='UserSettings',
             fields=[
                 ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('created', django_extensions.db.fields.CreationDateTimeField(auto_now_add=True, verbose_name='created')),
+                ('modified', django_extensions.db.fields.ModificationDateTimeField(auto_now=True, verbose_name='modified')),
                 ('_id', models.CharField(db_index=True, default=osf.models.base.generate_object_id, max_length=24, unique=True)),
-                ('deleted', models.BooleanField(default=False)),
-                ('oauth_grants', osf.utils.datetime_aware_jsonfield.DateTimeAwareJSONField(blank=True, default=dict)),
+                ('is_deleted', models.BooleanField(default=False)),
+                ('deleted', osf.utils.fields.NonNaiveDateTimeField(blank=True, null=True)),
+                ('oauth_grants', osf.utils.datetime_aware_jsonfield.DateTimeAwareJSONField(blank=True, default=dict, encoder=osf.utils.datetime_aware_jsonfield.DateTimeAwareJSONEncoder)),
             ],
             options={
                 'abstract': False,
             },
+            bases=(models.Model, osf.models.base.QuerySetExplainMixin),
         ),
     ]

addons/dataverse/migrations/0002_auto_20170323_1534.py → addons/dataverse/migrations/0002_auto_20220817_1915.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Generated by Django 1.9 on 2017-03-23 20:34
+# Generated by Django 1.11.29 on 2022-08-17 19:15
 from __future__ import unicode_literals
 
 from django.conf import settings