Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

joinpartitions should take a promote kwarg #26

Closed
rofinn opened this issue Oct 6, 2021 · 0 comments · Fixed by #27
Closed

joinpartitions should take a promote kwarg #26

rofinn opened this issue Oct 6, 2021 · 0 comments · Fixed by #27

Comments

@rofinn
Copy link
Member

rofinn commented Oct 6, 2021

To improve usability of the joinpartitions function I think we should probably support a promote::Bool kwarg which will allow us to widen the ChainedVector eltypes as we go. This is something DataFrames.jl already supports with its append! implementation.

https://github.com/JuliaData/DataFrames.jl/blob/f690aa49e958f51e0c3c579b6def1f11be214d98/src/dataframe/dataframe.jl#L1301

Example error which is annoying to work around is.

ArgumentError: string too large (36) to convert to InlineStrings.String31
Stacktrace:
[1] stringtoolong(T::Type, n::Int64)
@ InlineStrings ~/.julia/packages/InlineStrings/2HAHk/src/InlineStrings.jl:186
[2] InlineString
@ ~/.julia/packages/InlineStrings/2HAHk/src/InlineStrings.jl:209 [inlined]
[3] convert
@ ./strings/basic.jl:232 [inlined]
[4] convert
@ ./missing.jl:69 [inlined]
[5] getpoolidx
@ ~/.julia/packages/PooledArrays/jKDrZ/src/PooledArrays.jl:548 [inlined]
[6] push!(pv::PooledArrays.PooledVector{Union{Missing, InlineStrings.String31}, UInt32, Vector{UInt32}}, v::InlineStrings.String63)
@ PooledArrays ~/.julia/packages/PooledArrays/jKDrZ/src/PooledArrays.jl:594
[7] push!
@ ~/.julia/packages/SentinelArrays/VbnBp/src/chainedvector.jl:451 [inlined]
[8] append!(A::SentinelArrays.ChainedVector{Union{Missing, InlineStrings.String31}, PooledArrays.PooledVector{Union{Missing, InlineStrings.String31}, UInt32, Vector{UInt32}}}, B::PooledArrays.PooledVector{Union{Missing, InlineStrings.String63}, UInt32, Vector{UInt32}})
@ SentinelArrays ~/.julia/packages/SentinelArrays/VbnBp/src/chainedvector.jl:608
[9] (::TableOperations.var"#23#26"{Vector{SentinelArrays.ChainedVector}, DataFrames.DataFrameColumns{DataFrames.DataFrame}})(i::Int64)
@ TableOperations ~/.julia/packages/TableOperations/ErT2o/src/TableOperations.jl:312
[10] foreach(f::TableOperations.var"#23#26"{Vector{SentinelArrays.ChainedVector}, DataFrames.DataFrameColumns{DataFrames.DataFrame}}, itr::UnitRange{Int64})
@ Base ./abstractarray.jl:2141
[11] joinpartitions(x::Tables.Partitioner{Base.Generator{Vector{String}, Tables.var"#9#10"{S3DB.var"#220#221"}}})
@rofinn rofinn changed the title joinpartitions should take a promote=false kwarg joinpartitions should take a promote kwarg Oct 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant