[chore] add UP rule to ruff and upgrade to Python 3.9 syntax

[danielgafni]

Summary & Motivation

Hello, I really like running ruff on Dagster.

This PR adds the pyupgrade rule which automatically updates Python syntax & usage to the minimal supported Python version (3.9 in our case).

The only exception is dagster-buildkite since it's still running on 3.8 (@gibsondan how hard would it be to bump Python in BuildKite?).

See pyproject.toml diff:

diff --git a/pyproject.toml b/pyproject.toml
index 7fcaa156fe..0ff8cb69a4 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -98,7 +98,7 @@ filterwarnings = [
 
 [tool.ruff]
 
-target-version = "py38"
+target-version = "py39"
 
 # *.py, *.ipy are included by default
 extend-include = ["*.ipynb"]
@@ -116,7 +116,7 @@ extend-exclude = [
 line-length = 100
 
 # Fail if Ruff is not running this version.
-required-version = "0.5.5"
+required-version = "0.7.2"
 
 [tool.ruff.lint]
 
@@ -181,6 +181,13 @@ ignore = [
   # (assorted perf rules) We have a lot of violations, enable when autofix is available
   "PERF401", # (manual-list-comprehension)
   "PERF402", # (manual-list-copy)
+
+  # (assorted pyupgrade rules) We have a lot of violations, enable when autofix is available
+  "UP035", # (deprecated-import)
+
+  # (assorted import rules) We have a lot of violations, enable when autofix is available
+  "F401", # (unused-import)
+
 ]
 
 # By default, ruff only uses all "E" (pycodestyle) and "F" (pyflakes) rules.
@@ -248,6 +255,9 @@ select = [
 
   # (invalid escape sequence) flag errant backslashes
   "W605",
+
+  # (pyupgrade) automatically upgrade Python syntax to newer versions
+  "UP",
 ]
 
 [tool.ruff.lint.flake8-builtins]

All other changes are automated.

Initial UP violations:

❯ ruff check --statistics
3561	UP006	[*] non-pep585-annotation
2163	UP035	[ ] deprecated-import
 457	UP008	[*] super-call-with-parameters
 162	UP015	[*] redundant-open-modes
  92	UP031	[*] printf-string-formatting
  39	UP027	[*] unpacked-list-comprehension
  39	UP034	[*] extraneous-parentheses
  22	UP037	[*] quoted-annotation
  15	UP028	[*] yield-in-for-loop
   7	UP024	[*] os-error-alias
   7	UP033	[*] lru-cache-with-maxsize-none
   7	UP036	[*] outdated-version-block
   4	UP009	[*] utf8-encoding-declaration
   4	UP013	[*] convert-typed-dict-functional-to-class
   2	UP010	[*] unnecessary-future-import
   1	UP012	[*] unnecessary-encode-utf8
   1	UP018	[*] native-literals
[*] fixable with `ruff check --fix`

Violations after ruff check --fix (before --unsafe-fixes):

❯ ruff check --statistics
3562	UP006 	[*] non-pep585-annotation
1352	UP035 	[ ] deprecated-import
 457	UP008 	[*] super-call-with-parameters
  92	UP031 	[*] printf-string-formatting
  15	UP028 	[*] yield-in-for-loop
   7	UP036 	[*] outdated-version-block
   4	TCH003	[*] typing-only-standard-library-import
[*] fixable with `ruff check --fix`

And before ignoring the unfixable rules:

❯ ruff check --statistics
237	UP035	deprecated-import
 36	F401 	unused-import

[danielgafni]

• [chore] add UP rule to ruff and upgrade to Python 3.9 syntax #25703 👈 (View in Graphite)
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Deploy preview for dagster-docs ready!

Preview available at https://dagster-docs-5tw7rwtu0-elementl.vercel.app
https://11-02--chore-add-up-rule-to-ruff.dagster.dagster-docs.io

Direct link to changed pages:

[github-actions]

Deploy preview for dagster-docs-beta ready!

Preview available at https://dagster-docs-beta-owcusk788-elementl.vercel.app

Direct link to changed pages:

[cmpadden]

I think this is great, and am in favor of inclusion of pyupgrade rules.

Will wait for approval from at least one other though to ensure consensus.

[danielgafni]

Anyone wants to give another review?

[smackesey]

Left a comment on the pyproject.toml.

Also this needs an internal companion PR to keep these OSS/internal in sync.

Otherwise nice!

[danielgafni]

After switching to py39:

ruff check --statistics
3531	UP006 	[*] non-pep585-annotation
1344	UP035 	[ ] deprecated-import
   4	TCH003	[*] typing-only-standard-library-import

After running ruff with --unsafe-fixes:

❯ ruff check --statistics
237	UP035	deprecated-import
 36	F401 	unused-import

Deprecated import errors are similar to:

python_modules/libraries/dagstermill/dagstermill/manager.py:5:1: UP035 `typing.AbstractSet` is deprecated, use `collections.abc.Set` instead
  |
3 | import uuid
4 | from collections.abc import Mapping
5 | from typing import TYPE_CHECKING, AbstractSet, Any, Optional, cast
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ UP035
6 |
7 | from dagster import (
  |

Unused import errors look like:

python_modules/libraries/dagster-pandera/dagster_pandera/__init__.py:4:45: F401 `typing.Type` imported but unused
|
2 | import re
3 | from collections.abc import Sequence
4 | from typing import TYPE_CHECKING, Callable, Type, Union
|                                             ^^^^ F401
5 |
6 | import dagster._check as check
|
= help: Remove unused import: `typing.Type`

I've added both rules to exclude.

[schrockn]

• Can you document (or point to a document) the describes all the autofixes variants being applied here? This diff to is too large to review?

With the above, we could also then announce this to end-headsup so people know what the have to "untrain" themselves from writing.

[danielgafni]

Hey @schrockn, I added violation stats to the PR description.

For sure it's impossible to review, but the number of --unsafe-fixes is pretty low (only 2 rules - check my comment above) and since CI was happy it's probably safe to apply them.

[danielgafni]

Actually, there are some failures in BuildKite because some parts of it are still running on Python 3.8 3.7.
I hope there is a single image/entrypoint which we can bump? @gibsondan @smackesey

[danielgafni]

@schrockn @cmpadden @smackesey

Hey guys, major tests seem to be passing.

I would like to get an approval here because it's very hard to keep this PR up to date (new conflicts appear all the time).

I'll rebase it again to make sure dagster-blueprints is fine now.

[schrockn]

What I want to see before we land this is the text of an announcement of this to eng-team that explains all changes here.

e.g.

* Use native types for list, dict, set in typing declarations. E.g. list[Foo] over List[Foo]. This reduces the number of imports for common use cases.

Then show a single example. Do this for all the rules.

[schrockn]

Requesting changes so that you can produce a team-facing document detailing all the changes.

[danielgafni]

got it!