- Sponsor
-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
StreamMapIterator (was asyncmap) #15058
Conversation
|
||
In-place version of [`amap`](:func:`amap`). | ||
""" | ||
function amap! end |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should have an actual signature so the docstring gets disambiguated.
These functions need clearer names. The pmap function gets a pass because that's a fairly standard distributed computing name. |
OK (the original names were simply an attempt to follow the @bjarthur suggested in #14843:
This seems sensible to me (shorthand for This PR now adds only one function: asyncmap(f, c...; ntasks=100) -> iterator Apply f to each element of c using at most 100 asynchronous tasks. For multiple collection arguments, apply f elementwise. Note: |
bump? |
Looks good. While it is not exported, can we have a different name for |
# | ||
#------------------------------------------------------------------------------- | ||
|
||
import Base: start, done, next |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not required?
What about just calling it (I think |
That sounds right. BTW, |
…treamMapIterator), fix JuliaLang#15058 (comment)
julia> typeof(filter(i->true, keys(Dict(1=>1))))
Filter{Function,Base.KeyIterator{Dict{Int64,Int64}}}
julia> typeof(filter(i->true, [1,2,3]))
Array{Int64,1} (to me that seems really weird because you sometimes need to do |
functions are supposed to be lowercase according to the style conventions: also, why no exports? at some point should be documented here: http://docs.julialang.org/en/release-0.4/manual/parallel-computing/ |
because #14843 (comment) |
@StefanKarpinski, your thoughts? |
I'll merge this in a day if there are no objections. It does not add any exports and will enable enhancing
|
|
||
`collect(MapIterator(f, c...))` is equivalent to `map(f, c...)` | ||
""" | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no extra newlines after docstrings
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this a hard requirement to make the docstrings machine-readable? or is it an issue of style?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like it's not required to get docstrings to be associated with the following function, but stylistically it makes the association less obvious.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, any amount of whitespace, as well as commented lines, will get ignored. But as Tony mentions, stylistically it isn't quite as obvious. Maybe worth a mention in #15136.
Also missing a fullstop as the end of the line.
@amitmurthy when/who will enhance |
Both AV tests timed out. Known issue? How do I restart AV builds? |
@amitmurthy looks like AV is happy now. |
StreamMapIterator (was asyncmap)
The |
The |
Yes, the |
As far as I can see the only difference is that If this was added to immutable Generator{I,F}
f::F
iter::I
end
Generator(f, c...) = Generator(f, zip(c...))
start(g::Generator) = start(g.iter)
done(g::Generator, s) = done(g.iter, s)
function next(g::Generator, s)
v, s2 = next(g.iter, s)
g.f(v...), s2
end |
Yes, that is the right thing do do. And if we are going with the |
I find it a bit odd to make the iterator wrapped by Generator always a Zip. It makes it harder to examine what the generator is iterating over. |
|
Yes, I think
is probably ok. |
The thing is that |
With the definition I wrote above |
Ah, yes, our messages crossed somewhere under the sea near Honolulu. |
Light is so slow. |
Should I roll a change along these lines into #15409? Or will someone with commit access just do it without a PR? |
I'll change it. |
First step in Simplifying and generalising pmap (as per #14843).
Please limit comments on this PR to technical / implementation detail relating to the code in the PR.
Please make comments relating the interface and the overall pmap refactoring in issue #14843.