Skip to content

Commit

Permalink
Squashed commit of the following:
Browse files Browse the repository at this point in the history
commit b0c336f
Author: Maximilian Roos <[email protected]>
Date:   Mon Oct 21 04:52:36 2019 -0400

    Whatsnew for #3419 (#3422)

    * pyupgrade --py36-plus

    * Update xarray/core/nputils.py

    Co-Authored-By: crusaderky <[email protected]>

    * Update xarray/core/parallel.py

    Co-Authored-By: crusaderky <[email protected]>

    * Update xarray/tests/test_cftime_offsets.py

    Co-Authored-By: crusaderky <[email protected]>

    * Update xarray/tests/test_cftime_offsets.py

    Co-Authored-By: crusaderky <[email protected]>

    * whatsnew

commit 2984415
Author: Rhys Doyle <[email protected]>
Date:   Mon Oct 21 01:17:47 2019 +0100

    Revert changes made in #3358 (#3411)

    * Revert #3358

    * Revision

    * Code review

    * Merge from master

    * Obsolescence note

commit 3c462b9
Author: Maximilian Roos <[email protected]>
Date:   Sun Oct 20 20:16:58 2019 -0400

    Python3.6 idioms (#3419)

    * pyupgrade --py36-plus

    * Update xarray/core/nputils.py

    Co-Authored-By: crusaderky <[email protected]>

    * Update xarray/core/parallel.py

    Co-Authored-By: crusaderky <[email protected]>

    * Update xarray/tests/test_cftime_offsets.py

    Co-Authored-By: crusaderky <[email protected]>

    * Update xarray/tests/test_cftime_offsets.py

    Co-Authored-By: crusaderky <[email protected]>

commit 9886e3c
Author: crusaderky <[email protected]>
Date:   Sun Oct 20 23:42:36 2019 +0100

    Temporarily mark pseudonetcdf-3.1 as incompatible (#3420)

commit 0f7ab0e
Author: Dan Nowacki <[email protected]>
Date:   Thu Oct 17 14:13:44 2019 -0700

    Fix and add test for groupby_bins() isnan TypeError. (#3405)

    * Fix and add test for groupby_bins() isnan TypeError.

    * Better testing

    * black

commit 6cd50cc
Author: pmallas <[email protected]>
Date:   Thu Oct 17 10:15:47 2019 -0400

    Update where docstring to make return value type more clear (#3408)

    * Update where docstring to make return value type more clear

    The where docstring was a little confusing to me.  I misunderstood "Same type as caller' to mean the values in the xarray not the xarray itself.  I think this small change will clean it up for most users.  Thanks.

    * Update xarray/core/common.py

    Co-Authored-By: Maximilian Roos <[email protected]>

commit 55b1ac0
Author: keewis <[email protected]>
Date:   Thu Oct 17 05:13:39 2019 +0200

    tests for arrays with units (#3238)

    * create the empty test file

    * add tests for data array aggregation functions

    * include pint in the ci

    * ignore missing type annotations for pint

    * really skip the tests if pint is not available

    * remove the reason from the importorskip call

    * test that the dataarray constructor does not strip the unit

    * convert every unit stripped warning to an error

    * work around pint not implementing np.allclose yet

    * remove the now unnecessary filterwarnings decorator

    * xfail all tests that depend on pint having a __array_function__

    * treat nans as equal

    * implement tests for simple arithmetic operations

    * use param's id argument to assign readable names

    * add tests for sel() and isel()

    * add more readable names for the unary arithmetics

    * xfail every test that is not yet xfailing

    These don't pass because the constructor raises a unit stripped
    warning - fixed in pint#764.

    * only xfail if pint is not the current development version

    This is test is not really reliable, but sufficient for now.

    * always use lists instead of tuples for indexing

    * add tests for loc and squeeze

    * black

    * add names and xfail marks to the parameters

    * add tests for interp and interp_like

    * implement tests for reindex

    * remove the xfail marks where it is not clear yet that pint is to blame

    * add tests for reindex_like

    * don't pass the new DataArray to a kwarg

    * xfail if not on pint dev

    * refactor the tests

    * add tests for univariate and bivariate ufuncs

    * black

    * xfail aggregation only if pint does not implement __array_function__ yet

    * remove the global filterwarnings mark

    apparently, this caused the tests to change behavior, resulting in
    different errors, or causing tests to pass that should actually fail.

    * add a test case for the repr

    * create a pytest mark that explicitly requires pint's __array_function__

    * also check the string representation in addition to the repr

    * add helpers for creating method tests

    * check that simple aggregation methods work

    * use format() instead of format strings

    * make sure the repr of method calls is different from functions

    * merge the two aggregation tests

    * explicitly check whether pint supports __array_function__

    relying on versions is somewhat fragile.

    * provide a fallback for the new base quantity

    * check that no warning is raised for both with and without coords

    * also check that the repr works both with and without coords

    * wrap all aggregation function calls

    * xfail every call that fails because of something outside xarray

    * xfail tests related to dimension coordinates and indexes

    * use the dimensions from the original array

    * allow passing arguments to the method on call

    * add tests for comparisons

    * add tests for detecting, filling and dropping missing values

    * mark the missing value tests as requiring pint to support duck arrays

    * add tests for isin, where and interpolate_na

    * reformat unit ids and add a test parameter for compatible units

    * remove an unnecessary xfail

    * add tests for the top-level replication functions (*_like)

    * check for whatever pint does with *_like functions

    * add tests for combine_first

    * xfail the bivariate ufunc tests

    * xfail the call to np.median

    * move the top-level function tests out of the DataArray namespace class

    * add cumsum and cumprod to the list of aggregation functions

    * add tests for the numpy methods

    * check for equal units directly after checking the magnitude

    * add tests for content manipulation methods

    * add tests for comparing DataArrays (equals, indentical)

    * add a test for broadcast_equals

    * refactor the comparison operation tests

    * rewrite the strip, attach and assert_equal functions and add extract

    * preserve multiindex in strip and attach

    * attach the unit from element "data" as fallback

    * fix some small typos

    * compare QuantityScalar and QuantitySequence based on their values

    * make the isel test more robust

    * add tests for reshaping and reordering

    * unify the structure of the tests

    * mark the remaining tests as requiring a recent pint version, too

    * explicitly handle quantities as parameters

    * change the repr of the function / method wrappers

    * check whether __init__ and repr / str handle units in data and coords

    * generalize array_attach_units

    * move the redefinition of DimensionalityError

    * identify quantities using isinstance

    * typo

    * skip tests with a pint version without __array_function__

    * compare DataArrays where possible

    * mark only the compatible unit as xfailing

    * preserve the name of data arrays

    * also attach units to x_mm

    * Test in more CI environments; documentation

    * What's New

    * remove a stale function

    * use Quantity directly for instance tests

    * explicitly set roll_coords to silence a deprecation warning

    * skip the whole module if pint does not implement __array_function__

    the advantage is that now forgetting to decorate a test case is not possible.

    * allow to attach units using the mapping from extract_units

    * add tests for computation methods

    resampling fails until I figure out how to use it with non-datetime coords.

    * add tests for grouped operations

    * add a test for rolling_exp

    * add a todo note for the module level skip on __array_function__

    * add a test for dot

    * use attach_units instead of manually attaching

    * modify the resample test to actually work

    * add a test for to_unstacked_dataset

    * update whats-new.rst and installing.rst

    * reformat the whats-new.rst entry

    * What's New

commit 1f81338
Author: keewis <[email protected]>
Date:   Wed Oct 16 20:54:27 2019 +0200

    Fixes to the resample docs (#3400)

    * add a missing newline to make sphinx detect the code block

    * update the link to the pandas documentation

    * explicitly state that this only works with datetime dimensions

    * also put the datetime dim requirement into the function description

    * add Series.resample and DataFrame.resample as reference

    * add the changes to whats-new.rst

    * move references to the bottom of the docstring

commit 3f9069b
Author: Joseph Hamman <[email protected]>
Date:   Mon Oct 14 14:22:08 2019 -0700

    Revert to dev version

commit 62943e2
Author: Joseph Hamman <[email protected]>
Date:   Mon Oct 14 12:51:46 2019 -0700

    Release v0.14.0

commit 30472ec
Author: Joseph Hamman <[email protected]>
Date:   Mon Oct 14 12:24:05 2019 -0700

    updates for 0.14 release [black only]

commit 4519843
Author: Joseph Hamman <[email protected]>
Date:   Mon Oct 14 12:18:54 2019 -0700

    updates for 0.14 release

commit 4f5ca73
Author: Deepak Cherian <[email protected]>
Date:   Mon Oct 14 18:06:53 2019 +0000

    Make concat more forgiving with variables that are being merged. (#3364)

    * Make concat more forgiving with variables that are being merged.

    * rename test.

    * simplify test.

    * make diff smaller.

commit ae1d8c7
Author: Crypto Jerônimo <[email protected]>
Date:   Sun Oct 13 15:38:36 2019 +0100

    Fix documentation typos (#3396)

commit 863e490
Author: Joe Hamman <[email protected]>
Date:   Sat Oct 12 17:33:33 2019 -0400

    OrderedDict --> dict, some python3.5 cleanup too (#3389)

    * OrderedDict --> dict, some python3.5 cleanup too

    * respond to part of @shoyer's review

    * fix set attr syntax on netcdf4 vars

    * fix typing errors

    * update whats new and todo comments

    * Typing annotations

    * Typing annotations

    * Fix regression

    * More substantial changes

    * More polish

    * Typing annotations

    * Rerun notebooks

commit 6851e3e
Author: crusaderky <[email protected]>
Date:   Sat Oct 12 21:05:32 2019 +0100

    Annotate LRUCache (#3395)

commit 4c05d38
Author: Stephan Hoyer <[email protected]>
Date:   Fri Oct 11 08:47:57 2019 -0700

    BUG: overrides to a dimension coordinate do not get aligned (#3393)

    Fixes GH3377

commit 3f29551
Author: Deepak Cherian <[email protected]>
Date:   Thu Oct 10 23:44:19 2019 +0000

    map_blocks (#3276)

    * map_block attempt 2

    * Address reviews: errors, args + kwargs support.

    * Works with datasets!

    * remove wrong comment.

    * Support chunks.

    * infer template.

    * cleanup

    * cleanup2

    * api.rst

    * simple shape change error check.

    * Make test more complicated.

    * Fix for when user function doesn't set DataArray.name

    * Now _to_temp_dataset works.

    * Add whats-new

    * chunks kwarg makes no sense right now.

    * review feedback:

    1. skip index graph nodes.
    2. var → name
    3. quicker dataarray creation.
    4. Add restrictions to docstring.
    5. rename chunk construction task.
    6. error when non-xarray object is returned.
    7. restore non-coord dims.

    review

    * Support nondim coords in make_meta.

    * Add Dataset.unify_chunks

    * doc updates.

    * minor.

    * update comment.

    * More complicated test dataset. Tests fail :X

    * Don't know why compute is needed.

    * work with DataArray nondim coords.

    * fastpath unify_chunks

    * comment.

    * much improved tests.

    * Change args, kwargs syntax.

    * Add dataset, dataarray methods.

    * api.rst

    * docstrings.

    * Fix unify_chunks.

    * Move assert_chunks_equal to xarray.testing.

    * minor changes.

    * Better error handling when inferring returned object

    * wip

    * wip

    * better to_array

    * Docstrings + nicer error message.

    * remove unify_chunks in map_blocks + better tests.

    * typing for unify_chunks

    * address more review comments.

    * more unify_chunks tests.

    * Just use dask.core.utils.meta_from_array

    * get tests working. assert_equal needs a lot of fixing.

    * more unify_chunks test.

    * assert_chunks_equal fixes.

    * copy over meta_from_array.

    * minor fixes.

    * raise chunks error earlier and test for map_blocks raising chunk error

    * fix.

    * Type annotations

    * py35 compat

    * make sure unify_chunks does not compute.

    * Make tests functional by call compute before assert_equal

    * Update whats-new

    * Work with attributes.

    * Support attrs and name changes.

    * more assert_equal

    * test changing coord attribute

    * fix whats new

    * rework tests to use fixtures (kind of)

    * more review changes.

    * cleanup

    * more review feedback.

    * fix unify_chunks.

    * read dask_array_compat :)

    * Dask 1.2.0 compat.

    * documentation polish

    * make_meta reflow

    * cosmetic

    * polish

    * Fix tests

    * isort

    * isort

    * Add func call to docstrings.

commit 291cb80
Author: Deepak Cherian <[email protected]>
Date:   Thu Oct 10 18:23:20 2019 +0000

    Add groupby.dims & Fix groupby reduce for DataArray (#3338)

    * Fix groupby reduce for DataArray

    * bugfix.

    * another bugfix.

    * bugfix unique_and_monotonic for object indexes (uniqueness is enough)

    * Add groupby.dims property.

    * update reduce docstring to point to xarray.ALL_DIMS

    * test for object index dims.

    * test reduce dimensions error.

    * Add whats-new

    * fix docs build

    * sq whats-new

    * one more test.

    * fix test.

    * undo monotonic change.

    * Add dimensions to repr.

    * Raise error if no bins.

    * Raise nice error if no groups were formed.

    * Some more error raising and testing.

    * Add dataset tests.

    * update whats-new.

    * fix tests.

    * make dims a cached lazy property.

    * fix whats-new.

    * whitespace

    * fix whats-new

commit 3f0049f
Author: crusaderky <[email protected]>
Date:   Wed Oct 9 19:01:29 2019 +0100

    Speed up isel and __getitem__ (#3375)

    * Variable.isel cleanup/speedup

    * Dataset.isel code cleanup

    * Speed up isel

    * What's New

    * Better error checks

    * Speedup

    * type annotations

    * Update doc/whats-new.rst

    Co-Authored-By: Maximilian Roos <[email protected]>

    * What's New

    * What's New

    * Always shallow-copy variables

commit 132733a
Author: Deepak Cherian <[email protected]>
Date:   Tue Oct 8 22:13:47 2019 +0000

    Fix concat bug when concatenating unlabeled dimensions. (#3362)

    * Fix concat bug when concatenating unlabeled dimensions.

    * Add whats-new

    * Add back older test.

    * fix test

    * Revert "fix test"

    This reverts commit c33ca34.

    * better fix

commit 6fb272c
Author: crusaderky <[email protected]>
Date:   Tue Oct 8 22:23:46 2019 +0100

    Rolling minimum dependency versions policy (#3358)

    * - Downgrade numpy to 1.14, pandas to 0.20, scipy to 0.19 (24 months old)
    - Downgrade dask to 1.1 (6 months old)
    - Don't pin patch versions

    * Apply rolling policy (see #3222)

    * Automated tool to verify the minimum versions

    * Drop Python 3.5

    * lint

    * Trivial cosmetic

    * Cosmetic

    * (temp) debug CI failure

    * Parallelize versions check script

    * Remove hacks for legacy dask

    * Documentation

    * Assorted cleanup

    * Assorted cleanup

    * Fix regression

    * Cleanup

    * type annotations upgraded to Python 3.6

    * count_not_none backport

    * pd.Index.equals on legacy pandas returned False when comparing vs. a ndarray

    * Documentation

    * pathlib cleanup

    * Slide deprecations from 0.14 to 0.15

    * More cleanups

    * More cleanups

    * Fix min_deps_check

    * Fix min_deps_check

    * Set policy of 12 months for pandas and scipy

    * Cleanup

    * Cleanup

    * Sphinx fix

    * Overhaul readthedocs environment

    * Fix test crash

    * Fix test crash

    * Prune readthedocs environment

    * Cleanup

    * Hack around versioneer bug on readthedocs CI

    * Code review

    * Prevent random timeouts in the readthedocs CI

    * What's New polish

    * Merge from Master

    * Trivial cosmetic

    * Reimplement pandas.core.common.count_not_none

commit 3e2a754
Author: Alan D. Snow <[email protected]>
Date:   Tue Oct 8 09:36:52 2019 -0500

    added geocube and rioxarray to related projects (#3383)

commit 4254b4a
Author: crusaderky <[email protected]>
Date:   Fri Oct 4 23:17:56 2019 +0100

    Lint (#3373)

    * raise exception instance, not class

    * isort

    * isort

    * Bump mypy version

commit 283b4fe
Author: Deepak Cherian <[email protected]>
Date:   Fri Oct 4 17:04:36 2019 +0000

    Docs/more fixes (#2934)

    * Move netcdf to beginning of io.rst

    * Better indexing example.

    * Start de-emphasizing pandas

    * misc.

    * compute, load, persist docstrings + text.

    * split-apply-combine.

    * np.newaxis.

    * misc.

    * some dask stuff.

    * Little more dask.

    * undo index.rst changes.

    * link to dask docs on chunks

    * Fix io.rst.

    * small changes.

    * rollingupdate.

    * joe's review

commit dfdeef7
Author: Stephan Hoyer <[email protected]>
Date:   Thu Oct 3 21:42:50 2019 -0700

    Explicitly keep track of indexes with merging (#3234)

    * Explicitly keep track of indexes in merge.py

    * Typing fixes

    * More tying fixes

    * more typing fixes

    * fixup

commit 86fb71d
Author: Deepak Cherian <[email protected]>
Date:   Thu Oct 3 15:41:50 2019 +0000

    groupby repr (#3344)

    * groupby repr

    * add test.

    * test datetime and nondim coord

    * rename test_da → repr_da

    * Add whats-new

    * Update doc/whats-new.rst

commit dd2b803
Author: Ryan May <[email protected]>
Date:   Wed Oct 2 15:43:44 2019 -0600

    Remove setting of universal wheels (#3367)

    Universal wheels indicate that one wheel supports Python 2 and 3. This
    is no longer the case for xarray. This causes builds to generate files
    with names like xarray-0.13.0-py2.py3-none-any.whl, which can cause
    pip to incorrectly install the wheel when installing from a list of
    wheel files.

commit 21705e6
Author: crusaderky <[email protected]>
Date:   Tue Oct 1 19:13:55 2019 +0100

    Revisit # noqa annotations (#3359)

commit fb575eb
Author: crusaderky <[email protected]>
Date:   Tue Oct 1 15:11:21 2019 +0100

    Fix codecov.io upload on Windows (#3360)

commit 1ab2279
Author: Deepak Cherian <[email protected]>
Date:   Mon Sep 30 21:12:22 2019 +0000

    Add how do I ... section (#3357)

    * Add how do I ... section

    * Bbugfix.

    * Update doc/howdoi.rst

    Co-Authored-By: Maximilian Roos <[email protected]>

    * Update doc/howdoi.rst

    Co-Authored-By: Maximilian Roos <[email protected]>

    * small updates.

    * Add more.

commit bd1069b
Author: Gregory Gundersen <[email protected]>
Date:   Sun Sep 29 19:39:53 2019 -0400

    Add glossary to documentation (#3352)

    * First draft at terminology glossary.

    * Made name matching rules more explicit and hopefully clearer.

    * Amended what's new.

    * Changes based on feedback.

    * More changed based on feedback.

commit b51683f
Author: Anderson Banihirwe <[email protected]>
Date:   Sun Sep 29 07:50:21 2019 -0600

    Documentation improvements (#3328)

    * Add examples for full_like, zeros_like, ones_like

    * Add examples for xr.align

    * Add examples for xr.merge

    * Update xr.where docstring

    * Update xr.dot docstring

    * Update xarray/core/common.py

    Co-Authored-By: Deepak Cherian <[email protected]>

    * Update xarray/core/common.py

    Co-Authored-By: Deepak Cherian <[email protected]>

    * Update xr.combine_by_coords docstring

    * Apply black formatting only

    * More black formatting

    * Remove unnecessary pandas bits

    * Fix indentation issues

    * Update assign and pipe

    * Update `Dataset.reindex` with examples

    * Update `Dataset.fillna` with examples

    * Address styling issues

    * Update docstring

    Co-Authored-By: Deepak Cherian <[email protected]>

commit f3c7da6
Author: Gregory Gundersen <[email protected]>
Date:   Sat Sep 28 15:57:36 2019 -0400

    Remove `complex.nc` from built docs (#3353)

    * Rolling back to prevent a different issue from leaking into this one.

    * Amended what's new.

commit 6ece6a1
Author: Tony Tung <[email protected]>
Date:   Thu Sep 26 22:45:26 2019 -0700

    Fix DataArray.to_netcdf type annotation (#3325)

    It calls DataSet.to_netcdf, which returns Union[bytes, "Delayed", None].  So this should as well.

commit 16fdac9
Author: crusaderky <[email protected]>
Date:   Thu Sep 26 10:38:46 2019 +0100

    CI test suites with pinned minimum dependencies (#3346)

    * CI test suites with pinned minimum dependencies

    * code review

    * Clarity re lxml

commit ea101f5
Author: Tom Nicholas <[email protected]>
Date:   Thu Sep 26 10:51:58 2019 +0200

    Bugfix/plot accept coord dim (#3345)

    Bug in plot.line fixed by ensuring 1D coords are cast down to their associated dims. There was previously two particular cases where this would not happen.

commit 85c9a40
Author: crusaderky <[email protected]>
Date:   Wed Sep 25 02:40:54 2019 +0100

    CI environments overhaul (#3340)

    * Rationalize and align CI environments. Add many optional dependencies to individual CI suites.

    * pynio and cdms2 are not available on Windows

    * cfgrib causes Python interpreter crash on Windows

    * dtype of np.arange defaults to int64 on Linux and int32 on Windows

    * Suppress failure to delete file on Windows

    * Mark hypotesis tests as @slow
  • Loading branch information
dcherian committed Oct 22, 2019
1 parent a2d05fe commit 8167921
Show file tree
Hide file tree
Showing 135 changed files with 6,112 additions and 2,873 deletions.
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ repos:
hooks:
- id: flake8
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v0.720 # Must match ci/requirements/*.yml
rev: v0.730 # Must match ci/requirements/*.yml
hooks:
- id: mypy
# run these occasionally, ref discussion https://github.com/pydata/xarray/pull/3194
Expand Down
4 changes: 2 additions & 2 deletions asv_bench/benchmarks/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@ def decorator(func):

def requires_dask():
try:
import dask # noqa
import dask # noqa: F401
except ImportError:
raise NotImplementedError
raise NotImplementedError()


def randn(shape, frac_nan=None, chunks=None, seed=0):
Expand Down
2 changes: 1 addition & 1 deletion asv_bench/benchmarks/dataarray_missing.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
from . import randn, requires_dask

try:
import dask # noqa
import dask # noqa: F401
except ImportError:
pass

Expand Down
2 changes: 1 addition & 1 deletion asv_bench/benchmarks/dataset_io.py
Original file line number Diff line number Diff line change
Expand Up @@ -458,7 +458,7 @@ def setup(self):
try:
import distributed
except ImportError:
raise NotImplementedError
raise NotImplementedError()
self.client = distributed.Client()
self.write = create_delayed_write()

Expand Down
40 changes: 24 additions & 16 deletions azure-pipelines.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,20 @@ jobs:
- job: Linux
strategy:
matrix:
py35-min:
conda_env: py35-min
py36-bare-minimum:
conda_env: py36-bare-minimum
py36-min-all-deps:
conda_env: py36-min-all-deps
py36-min-nep18:
conda_env: py36-min-nep18
py36:
conda_env: py36
py37:
conda_env: py37
py37-upstream-dev:
conda_env: py37
upstream_dev: true
py36-flakey:
py36-flaky:
conda_env: py36
pytest_extra_flags: --run-flaky --run-network-tests
allow_failure: true
Expand Down Expand Up @@ -78,27 +82,31 @@ jobs:
mypy .
displayName: mypy type checks
- job: Docs
- job: MinimumVersionsPolicy
pool:
vmImage: 'ubuntu-16.04'
steps:
- template: ci/azure/install.yml
parameters:
env_file: doc/environment.yml
- template: ci/azure/add-conda-to-path.yml
- bash: |
source activate xarray-tests
cd doc
sphinx-build -n -j auto -b html -d _build/doctrees . _build/html
displayName: Build HTML docs
conda install -y pyyaml
python ci/min_deps_check.py ci/requirements/py36-bare-minimum.yml
python ci/min_deps_check.py ci/requirements/py36-min-all-deps.yml
displayName: minimum versions policy
- job: LinuxHypothesis
variables:
conda_env: py36-hypothesis
- job: Docs
pool:
vmImage: 'ubuntu-16.04'
steps:
- template: ci/azure/install.yml
parameters:
env_file: ci/requirements/doc.yml
- bash: |
source activate xarray-tests
# Replicate the exact environment created by the readthedocs CI
conda install --yes --quiet -c pkgs/main mock pillow sphinx sphinx_rtd_theme
displayName: Replicate readthedocs CI environment
- bash: |
source activate xarray-tests
pytest properties
displayName: Property based tests
cd doc
sphinx-build -n -j auto -b html -d _build/doctrees . _build/html
displayName: Build HTML docs
5 changes: 3 additions & 2 deletions ci/azure/unit-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,16 @@ steps:
# https://github.com/microsoft/azure-pipelines-tasks/issues/9302
- bash: |
source activate xarray-tests
pytest xarray \
pytest \
--junitxml=junit/test-results.xml \
--cov=xarray \
--cov-report=xml \
$(pytest_extra_flags) || [ "$ALLOW_FAILURE" = "true" ]
displayName: Run tests

- bash: |
bash <(curl https://codecov.io/bash) -t 688f4d53-31bb-49b5-8370-4ce6f792cf3d
curl https://codecov.io/bash > codecov.sh
bash codecov.sh -t 688f4d53-31bb-49b5-8370-4ce6f792cf3d
displayName: Upload coverage to codecov.io

# TODO: publish coverage results to Azure, once we can merge them across
Expand Down
187 changes: 187 additions & 0 deletions ci/min_deps_check.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,187 @@
"""Fetch from conda database all available versions of the xarray dependencies and their
publication date. Compare it against requirements/py36-min-all-deps.yml to verify the
policy on obsolete dependencies is being followed. Print a pretty report :)
"""
import subprocess
import sys
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timedelta
from typing import Dict, Iterator, Tuple

import yaml

IGNORE_DEPS = {
"black",
"coveralls",
"flake8",
"hypothesis",
"mypy",
"pip",
"pytest",
"pytest-cov",
"pytest-env",
}

POLICY_MONTHS = {"python": 42, "numpy": 24, "pandas": 12, "scipy": 12}
POLICY_MONTHS_DEFAULT = 6

has_errors = False


def error(msg: str) -> None:
global has_errors
has_errors = True
print("ERROR:", msg)


def parse_requirements(fname) -> Iterator[Tuple[str, int, int]]:
"""Load requirements/py36-min-all-deps.yml
Yield (package name, major version, minor version)
"""
global has_errors

with open(fname) as fh:
contents = yaml.safe_load(fh)
for row in contents["dependencies"]:
if isinstance(row, dict) and list(row) == ["pip"]:
continue
pkg, eq, version = row.partition("=")
if pkg.rstrip("<>") in IGNORE_DEPS:
continue
if pkg.endswith("<") or pkg.endswith(">") or eq != "=":
error("package should be pinned with exact version: " + row)
continue
try:
major, minor = version.split(".")
except ValueError:
error("expected major.minor (without patch): " + row)
continue
try:
yield pkg, int(major), int(minor)
except ValueError:
error("failed to parse version: " + row)


def query_conda(pkg: str) -> Dict[Tuple[int, int], datetime]:
"""Query the conda repository for a specific package
Return map of {(major version, minor version): publication date}
"""
stdout = subprocess.check_output(
["conda", "search", pkg, "--info", "-c", "defaults", "-c", "conda-forge"]
)
out = {} # type: Dict[Tuple[int, int], datetime]
major = None
minor = None

for row in stdout.decode("utf-8").splitlines():
label, _, value = row.partition(":")
label = label.strip()
if label == "file name":
value = value.strip()[len(pkg) :]
major, minor = value.split("-")[1].split(".")[:2]
major = int(major)
minor = int(minor)
if label == "timestamp":
assert major is not None
assert minor is not None
ts = datetime.strptime(value.split()[0].strip(), "%Y-%m-%d")

if (major, minor) in out:
out[major, minor] = min(out[major, minor], ts)
else:
out[major, minor] = ts

# Hardcoded fix to work around incorrect dates in conda
if pkg == "python":
out.update(
{
(2, 7): datetime(2010, 6, 3),
(3, 5): datetime(2015, 9, 13),
(3, 6): datetime(2016, 12, 23),
(3, 7): datetime(2018, 6, 27),
(3, 8): datetime(2019, 10, 14),
}
)

return out


def process_pkg(
pkg: str, req_major: int, req_minor: int
) -> Tuple[str, int, int, str, int, int, str, str]:
"""Compare package version from requirements file to available versions in conda.
Return row to build pandas dataframe:
- package name
- major version in requirements file
- minor version in requirements file
- publication date of version in requirements file (YYYY-MM-DD)
- major version suggested by policy
- minor version suggested by policy
- publication date of version suggested by policy (YYYY-MM-DD)
- status ("<", "=", "> (!)")
"""
print("Analyzing %s..." % pkg)
versions = query_conda(pkg)

try:
req_published = versions[req_major, req_minor]
except KeyError:
error("not found in conda: " + pkg)
return pkg, req_major, req_minor, "-", 0, 0, "-", "(!)"

policy_months = POLICY_MONTHS.get(pkg, POLICY_MONTHS_DEFAULT)
policy_published = datetime.now() - timedelta(days=policy_months * 30)

policy_major = req_major
policy_minor = req_minor
policy_published_actual = req_published
for (major, minor), published in reversed(sorted(versions.items())):
if published < policy_published:
break
policy_major = major
policy_minor = minor
policy_published_actual = published

if (req_major, req_minor) < (policy_major, policy_minor):
status = "<"
elif (req_major, req_minor) > (policy_major, policy_minor):
status = "> (!)"
error("Package is too new: " + pkg)
else:
status = "="

return (
pkg,
req_major,
req_minor,
req_published.strftime("%Y-%m-%d"),
policy_major,
policy_minor,
policy_published_actual.strftime("%Y-%m-%d"),
status,
)


def main() -> None:
fname = sys.argv[1]
with ThreadPoolExecutor(8) as ex:
futures = [
ex.submit(process_pkg, pkg, major, minor)
for pkg, major, minor in parse_requirements(fname)
]
rows = [f.result() for f in futures]

print("Package Required Policy Status")
print("------------- ----------------- ----------------- ------")
fmt = "{:13} {:>1d}.{:<2d} ({:10}) {:>1d}.{:<2d} ({:10}) {}"
for row in rows:
print(fmt.format(*row))

assert not has_errors


if __name__ == "__main__":
main()
21 changes: 21 additions & 0 deletions ci/requirements/doc.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
name: xarray-docs
channels:
# Don't change to pkgs/main, as it causes random timeouts in readthedocs
- conda-forge
dependencies:
- python=3.7
- bottleneck
- cartopy
- h5netcdf
- ipython
- iris
- netcdf4
- numpy
- numpydoc
- pandas<0.25 # Hack around https://github.com/pydata/xarray/issues/3369
- rasterio
- seaborn
- sphinx
- sphinx-gallery
- sphinx_rtd_theme
- zarr
15 changes: 0 additions & 15 deletions ci/requirements/py35-min.yml

This file was deleted.

11 changes: 11 additions & 0 deletions ci/requirements/py36-bare-minimum.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
name: xarray-tests
channels:
- conda-forge
dependencies:
- python=3.6
- coveralls
- pytest
- pytest-cov
- pytest-env
- numpy=1.14
- pandas=0.24
29 changes: 0 additions & 29 deletions ci/requirements/py36-hypothesis.yml

This file was deleted.

Loading

0 comments on commit 8167921

Please sign in to comment.