Add 'ignore_basepython_conflict' option (#477)

tox provides a number of default factors - py27, py34, py35 etc. - that
are tied to particular interpreter versions. It is possible to override
these through individual sections or the global [testenv] section. For
example, consider the following 'tox.ini' file:

  [tox]
  skipsdist = True
  minversion = 2.0
  distribute = False
  envlist = py35,py27,pep8,py34-test

  [testenv]
  basepython = python3
  install_command = pip install {opts} {packages}
  commands =
    python --version

  [testenv:py27]
  basepython = python2.7

Running any target except for 'py27' will result in the same interpreter
being used. On Fedora 28 with the 'python3-tox' package:

  $ tox -qq -e py27
  Python 2.7.15
  $ tox -qq -e py35
  Python 3.6.5
  $ tox -qq -e py34-test
  Python 3.6.5

This is broken by design. Overriding these makes no sense and is a
source of common misconfigurations, as noted in #477. The only sane
thing to do here is ignore the request and use the correct interpreter
or raise a warning. There is merit to both approaches, so this
functionality is exposed by way of a new global configuration option,
'ignore_basepython_conflict'.

Signed-off-by: Stephen Finucane <[email protected]>
Closes: #477

changelog/477.bugfix.rst

@@ -0,0 +1,5 @@
+Add ``ignore_basepython_conflict``, which determines whether conflicting
+``basepython`` settings for environments containing default factors, such as
+``py27`` or ``django18-py35``, should be ignored or result in warnings. This
+was a common source of misconfiguration and is rarely, if ever, desirable from
+a user perspective - by @stephenfin

doc/config.rst

@@ -58,6 +58,18 @@ and will first lookup global tox settings in this section:
     * environment variable ``TOXENV``
     * ``tox.ini`` file's ``envlist``
 
+.. confval:: ignore_basepython_conflict=True|False(default)
+
+    .. versionadded:: 3.1.0
+
+    If ``True``, :confval:`basepython` settings that conflict with the Python
+    variant for a environments using default factors, such as ``py27`` or
+    ``py35``, will be ignored. This allows you to configure
+    :confval:`basepython` in the global testenv without affecting these
+    factors. If ``False``, the default, a warning will be emitted if a conflict
+    is identified. In a future version of tox, this warning will become an
+    error.
+
 
 Virtualenv test environment settings
 ------------------------------------
@@ -81,8 +93,15 @@ Complete list of settings that you can put into ``testenv*`` sections:
 
 .. confval:: basepython=NAME-OR-PATH
 
-    name or path to a Python interpreter which will be used for creating
-    the virtual environment. **default**: interpreter used for tox invocation.
+    Name or path to a Python interpreter which will be used for creating
+    the virtual environment; if the environment name contains a :ref:`default
+    factor <factors>`, this value will be ignored. **default**: interpreter
+    used for tox invocation.
+
+    .. versionchanged:: 3.1
+
+       Environments that use a :ref:`default factor <factors>` now ignore this
+       value, defaulting to the interpreter defined for that factor.
 
 .. confval:: commands=ARGVLIST
 
@@ -546,9 +565,6 @@ However, a better approach looks like this:
     envlist = {py27,py36}-django{15,16}
 
     [testenv]
-    basepython =
-        py27: python2.7
-        py36: python3.6
     deps =
         pytest
         django15: Django>=1.5,<1.6
@@ -612,23 +628,12 @@ Factors and factor-conditional settings
 ++++++++++++++++++++++++++++++++++++++++
 
 Parts of an environment name delimited by hyphens are called factors and can
-be used to set values conditionally:
-
-.. code-block:: ini
-
-    basepython =
-        py27: python2.7
-        py36: python3.6
-
-This conditional setting will lead to either ``python3.6`` or
-``python2.7`` used as base python, e.g. ``python3.6`` is selected if current
-environment contains ``py36`` factor.
-
-In list settings such as ``deps`` or ``commands`` you can freely intermix
-optional lines with unconditional ones:
+be used to set values conditionally. In list settings such as ``deps`` or
+``commands`` you can freely intermix optional lines with unconditional ones:
 
 .. code-block:: ini
 
+    [testenv]
     deps =
         pytest
         django15: Django>=1.5,<1.6
@@ -638,16 +643,23 @@ optional lines with unconditional ones:
 Reading it line by line:
 
 - ``pytest`` will be included unconditionally,
-- ``Django>=1.5,<1.6`` will be included for environments containing ``django15`` factor,
+- ``Django>=1.5,<1.6`` will be included for environments containing
+  ``django15`` factor,
 - ``Django>=1.6,<1.7`` similarly depends on ``django16`` factor,
 - ``unittest`` will be loaded for Python 3.6 environments.
 
-.. note::
+tox provides a number of default factors corresponding to Python interpreter
+versions. The conditional setting above will lead to either ``python3.6`` or
+``python2.7`` used as base python, e.g. ``python3.6`` is selected if current
+environment contains ``py36`` factor.
 
-    tox provides good defaults for basepython setting, so the above
-    ini-file can be further reduced by omitting the ``basepython``
-    setting.
+.. note::
 
+    Configuring :confval:`basepython` for environments using default factors
+    will result in a warning. Configure :confval:`ignore_basepython_conflict`
+    if you wish to explicitly ignore these conflicts, allowing you to define a
+    global :confval:`basepython` for all environments *except* those with
+    default factors.
 
 Complex factor conditions
 +++++++++++++++++++++++++

doc/example/general.rst

@@ -49,7 +49,6 @@ a tox test run.  Here is an example ``tox.ini`` configuration:
 .. code-block:: ini
 
     [testenv:docs]
-    basepython = python
     changedir = doc
     deps = sphinx
     commands = sphinx-build -W -b html -d {envtmpdir}/doctrees . {envtmpdir}/html
@@ -142,26 +141,23 @@ If you want to use this with Jenkins_, also checkout the :ref:`jenkins artifact
 .. _verlib: https://bitbucket.org/tarek/distutilsversion/
 
 basepython defaults, overriding
-++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++
 
-By default, for any ``pyXY`` test environment name
-the underlying "pythonX.Y" executable will be searched in
-your system ``PATH``. It must exist in order to successfully create
-virtualenv environments.  On Windows a ``pythonX.Y`` named executable
-will be searched in typical default locations using the
-``C:\PythonX.Y\python.exe`` pattern.
+For any ``pyXY`` test environment name the underlying ``pythonX.Y`` executable
+will be searched in your system ``PATH``. Similarly, for ``jython`` and
+``pypy`` the respective ``jython`` and ``pypy-c`` names will be looked for.
+The executable must exist in order to successfully create *virtualenv*
+environments. On Windows a ``pythonX.Y`` named executable will be searched in
+typical default locations using the ``C:\PythonX.Y\python.exe`` pattern.
 
-For ``jython`` and ``pypy`` the respective ``jython``
-and ``pypy-c`` names will be looked for.
-
-You can override any of the default settings by defining
-the ``basepython`` variable in a specific test environment
-section, for example:
+All other targets will use the system ``python`` instead. You can override any
+of the default settings by defining the :confval:`basepython` variable in a
+specific test environment section, for example:
 
 .. code-block:: ini
 
-    [testenv:py27]
-    basepython=/my/path/to/python2.7
+    [testenv:docs]
+    basepython = python2.7
 
 Avoiding expensive sdist
 ------------------------

src/tox/config.py

@@ -505,11 +505,32 @@ def setenv(testenv_config, value):
     )
 
     def basepython_default(testenv_config, value):
+        """Configure a sane interpreter for the environment.
+
+        If the environment contains a default factor, this will always be the
+        interpreter associated with that factor overriding anything manually
+        set.
+        """
+        for factor in testenv_config.factors:
+            if factor in tox.PYTHON.DEFAULT_FACTORS:
+                default = tox.PYTHON.DEFAULT_FACTORS[factor]
+
+                if value is None or testenv_config.config.ignore_basepython_conflict:
+                    return default
+
+                if str(value) != default:
+                    # TODO(stephenfin): Raise an exception here in tox 4.0
+                    print(
+                        "WARNING: Conflicting basepython for environment '{}'; resolve conflict "
+                        "or configure ignore_basepython_conflict".format(
+                            testenv_config.envname, str(value), default
+                        ),
+                        file=sys.stderr,
+                    )
+
         if value is None:
-            for factor in testenv_config.factors:
-                if factor in tox.PYTHON.DEFAULT_FACTORS:
-                    return tox.PYTHON.DEFAULT_FACTORS[factor]
             return sys.executable
+
         return str(value)
 
     parser.add_testenv_attribute(
@@ -893,6 +914,8 @@ def __init__(self, config, inipath):
                 "skip_missing_interpreters", False
             )
 
+        config.ignore_basepython_conflict = reader.getbool("ignore_basepython_conflict", False)
+
         # determine indexserver dictionary
         config.indexserver = {"default": IndexServerConfig("default")}
         prefix = "indexserver"

tests/test_config.py

@@ -1640,6 +1640,38 @@ def test_default_factors(self, newconfig):
         for name, config in configs.items():
             assert config.basepython == "python{}.{}".format(name[2], name[3])
 
+    def test_default_factors_conflict(self, newconfig, capsys):
+        config = newconfig(
+            """
+            [testenv]
+            basepython=python3
+            [testenv:py27]
+            commands = python --version
+        """
+        )
+        assert len(config.envconfigs) == 1
+        envconfig = config.envconfigs["py27"]
+        assert envconfig.basepython == "python3"
+        captured = capsys.readouterr()
+        assert "WARNING: Conflicting basepython" in captured.err
+
+    def test_default_factors_conflict_ignore(self, newconfig, capsys):
+        config = newconfig(
+            """
+            [tox]
+            ignore_basepython_conflict=True
+            [testenv]
+            basepython=python3
+            [testenv:py27]
+            commands = python --version
+        """
+        )
+        assert len(config.envconfigs) == 1
+        envconfig = config.envconfigs["py27"]
+        assert envconfig.basepython == "python2.7"
+        captured = capsys.readouterr()
+        assert captured.err == ""
+
     @pytest.mark.issue188
     def test_factors_in_boolean(self, newconfig):
         inisource = """

tests/test_z_cmdline.py

@@ -796,13 +796,9 @@ def test_test_piphelp(initproj, cmd):
         "example123",
         filedefs={
             "tox.ini": """
-        # content of: tox.ini
-        [testenv]
-        commands=pip -h
-        [testenv:py27]
-        basepython=python
-        [testenv:py36]
-        basepython=python
+            # content of: tox.ini
+            [testenv]
+            commands=pip -h
     """
         },
     )

tox.ini

@@ -28,7 +28,6 @@ extras = docs
 changedir = {toxinidir}
 commands = sphinx-build -d "{toxworkdir}/docs_doctree" doc "{toxworkdir}/docs_out" --color -W -bhtml
 
-
 [testenv:fix-lint]
 description = format the code base to adhere to our styles, and complain about what we cannot do automatically
 basepython = python3.6