Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

announce: make _chunks() helper safe against more input types #2166

Merged
merged 2 commits into from
Aug 10, 2021

Conversation

dgw
Copy link
Member

@dgw dgw commented Jul 22, 2021

Description

Fixes #2165.

Python 3 dict.keys() returns a dict_keys object, which is not subscriptable unlike the return value in py2.

Also made the generator return a consistent type (always tuple), and updated the documentation (not that it's actually used anywhere by Sphinx). Added a test, too, because let's (try to) not have this plugin bite us again.

Checklist

  • I have read CONTRIBUTING.md
  • I can and do license this contribution under the EFLv2
  • No issues are reported by make qa (runs make quality and make test)
    • I ran both the new test and make quality on both py3(.8) and py2(.7), and the patch should be clean against both. Since this can't be a straight cherry-pick back to 7.1.x due to the new test file needing different headers, I have pre-prepped an announce-py2-backport branch for myself with the appropriate code style.
  • I have tested the functionality of the things this change touches

Especially `dict_keys` objects, which is how py3 handles `dict.keys()`
(vs. being a subscriptable iterable in py2).

Added a test, too, because let's not have this plugin bite us again.
@dgw dgw added the Bugfix Generally, PRs that reference (and fix) one or more issue(s) label Jul 22, 2021
@dgw dgw added this to the 7.1.3 milestone Jul 22, 2021
@dgw dgw requested a review from a team July 22, 2021 19:05
@dgw dgw mentioned this pull request Jul 22, 2021
@Exirel
Copy link
Contributor

Exirel commented Jul 23, 2021

What about any of these 2 versions to get some chunk?

Option 1: use itertools and iter, and keep everything immutable:

import itertools

def chunk(items, size):
    iterator = iter(items)
    data = tuple(itertools.islice(iterator, size))
    while data:
        yield data
        data = tuple(itertools.islice(iterator, size))

And this version uses a mutable list, but it might be easier to read when you don't know what itertools and iter can do for you:

def chunk(items, size):
    acc = []
    for i, item in enumerate(items, start=1):
        acc.append(item)
        if i % size == 0:
            yield tuple(acc)
            acc = []
    if acc:
        yield tuple(acc)

@dgw
Copy link
Member Author

dgw commented Jul 24, 2021

I like the itertools one, but am too tired to figure out why when I forgot to wrap the call in tuple() it looped forever.

>>> while chunk := itertools.islice(iterator, 3):
...     print(chunk)
...
<itertools.islice object at 0x7f3d2b211400>
<itertools.islice object at 0x7f3d2b0e19a0>
<itertools.islice object at 0x7f3d2b211400>
[many more repetitions]
<itertools.islice object at 0x7f3d2b0e19a0>
<itertools.islice object at 0x7f3d2b211400>
<itertools.islice object at 0x7f3d2b0e19a0>
^CTraceback (most recent call last):
  File "<stdin>", line 2, in <module>
KeyboardInterrupt
>>> while chunk := tuple(itertools.islice(iterator, 3)):
...     print(chunk)
...
(1, 2, 3)
(4, 5, 6)
(7, 8)

(Note that now I'm drunk on dropping old Python versions and wish it was reasonable to support only 3.8+ already because the := assignment operator is 👌.)

@Exirel
Copy link
Contributor

Exirel commented Jul 24, 2021

but am too tired to figure out why

Because islice returns a generator, which always evaluates to True. When you convert into a tuple, you actually get an empty or a non-empty tuple, which evaluates as False or True, allowing the while loop to stop its iteration.

@dgw dgw merged commit b38a5d6 into master Aug 10, 2021
@dgw dgw deleted the announce-py3-dict.keys branch August 10, 2021 07:41
dgw added a commit that referenced this pull request Aug 10, 2021
announce: make `_chunks()` helper safe against more input types

Cherry-picked from master: b38a5d6
dgw added a commit that referenced this pull request Aug 12, 2021
Forgot that I'd prepared a separate branch for the 7.x series.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bugfix Generally, PRs that reference (and fix) one or more issue(s)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

announce error on python3
2 participants