implement default for SubArray

[baumgold]

similar(::Type{<:SubArray}) doesn't exist. This change should handle it gracefully.

[quinnj]

Can you give a little more context around why we need this or what problem you ran into without?

[codecov-commenter]

Codecov Report

Merging #456 (d09a592) into main (074c7a7) will increase coverage by 0.12%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main     #456      +/-   ##
==========================================
+ Coverage   87.40%   87.53%   +0.12%     
==========================================
  Files          26       26              
  Lines        3271     3272       +1     
==========================================
+ Hits         2859     2864       +5     
+ Misses        412      408       -4     

Impacted Files	Coverage Δ
src/ArrowTypes/src/ArrowTypes.jl	86.85% <0.00%> (-0.50%)	⬇️

... and 4 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[baumgold]

One of the apps I maintain that relies on Arrow.jl crashed when upgrading to Arrow v2.6 with the following error/stack:

MethodError: no method matching SubArray{NamedTuple{(:QuoteSetID, :OfferSize, :BidSize), Tuple{UInt16, Union{Missing, UInt32}, Union{Missing, UInt32}}}, 1, Arrow.Struct{NamedTuple{(:QuoteSetID, :OfferSize, :BidSize), Tuple{UInt16, Union{Missing, UInt32}, Union{Missing, UInt32}}}, Tuple{Arrow.Primitive{UInt16, Vector{UInt16}}, Arrow.Primitive{Union{Missing, UInt32}, Vector{UInt32}}, Arrow.Primitive{Union{Missing, UInt32}, Vector{UInt32}}}}, Tuple{UnitRange{Int64}}, true}(::UndefInitializer, ::Tuple{Int64})

Closest candidates are:
  SubArray{T, N, P, I, L}(::Any, ::Any, ::Any, ::Any) where {T, N, P, I, L}
   @ Base subarray.jl:19

Stacktrace:
  [1] similar
    @ ./abstractarray.jl:882
  [2] similar
    @ ./abstractarray.jl:880
  [3] default
    @ ~/.julia/packages/ArrowTypes/lFz2k/src/ArrowTypes.jl:346
  [4] getindex
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/struct.jl:80 [inlined]
  [5] iterate
    @ ./abstractarray.jl:1220 [inlined]
  [6] iterate
    @ ./abstractarray.jl:1218 [inlined]
  [7] #ToList#16
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/list.jl:113
  [8] ToList
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/list.jl:100 [inlined]
  [9] #arrowvector#18
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/list.jl:214
 [10] arrowvector
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/list.jl:211 [inlined]
 [11] #arrowvector#10
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/arraytypes.jl:97
 [12] #arrowvector#3
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/arraytypes.jl:74
 [13] #49
    @ ./none:0
 [14] iterate
    @ ./generator.jl:47 [inlined]
 [15] collect_to!
    @ ./array.jl:840
 [16] collect_to!
    @ ./array.jl:848
--- the last 1 lines are repeated 1 more time ---
 [18] collect_to_with_first!
    @ ./array.jl:818
 [19] collect
    @ ./array.jl:792
 [20] _totuple
    @ ./tuple.jl:401 [inlined]
 [21] Tuple
    @ ./tuple.jl:369 [inlined]
 [22] #arrowvector#48
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/struct.jl:93
 [23] arrowvector
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/struct.jl:89 [inlined]
 [24] #arrowvector#10
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/arraytypes.jl:97
 [25] #arrowvector#3
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/arraytypes.jl:74
 [26] arrowvector
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/arraytypes.jl:58 [inlined]
 [27] #toarrowvector#2
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/arraytypes.jl:37
 [28] toarrowvector
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/arraytypes.jl:34 [inlined]
 [29] #145
    @ ~/.julia/packages/Arrow/86p6m/src/write.jl:329
 [30] eachcolumn
    @ ~/.julia/packages/Tables/AcRIE/src/utils.jl:70
 [31] toarrowtable
    @ ~/.julia/packages/Arrow/86p6m/src/write.jl:326
 [32] macro expansion
    @ ~/.julia/packages/Arrow/86p6m/src/write.jl:187 [inlined]
 [33] macro expansion
    @ ./task.jl:476 [inlined]
 [34] write
    @ ~/.julia/packages/Arrow/86p6m/src/write.jl:177
 [35] #122
    @ ~/.julia/packages/Arrow/86p6m/src/write.jl:58
 [36] #open#409
    @ ./io.jl:395
 [37] open
    @ ./io.jl:392 [inlined]
 [38] #write#121
    @ ~/.julia/packages/Arrow/86p6m/src/write.jl:57 [inlined]
 [39] write
    @ ~/.julia/packages/Arrow/86p6m/src/write.jl:56 [inlined]

[baumgold]

@quinnj - any opinions/suggestions here? Thanks.

[baumgold]

I was able to reproduce this with the following:

julia> NT = @NamedTuple{x::Int, y::Union{Missing,Int}};

julia> data = NT[(x=1,y=2), (x=2,y=missing), (x=3,y=4), (x=4,y=5)];

julia> t = [(a=1,b=view(data,1:2)), (a=2,b=view(data,3:4)), missing];

julia> Arrow.toarrowvector(t);
ERROR: MethodError: no method matching SubArray{NamedTuple{(:x, :y), Tuple{Int64, Union{Missing, Int64}}}, 1, Vector{NamedTuple{(:x, :y), Tuple{Int64, Union{Missing, Int64}}}}, Tuple{UnitRange{Int64}}, true}(::UndefInitializer, ::Tuple{Int64})

Note after I added the suggested solution from this PR, I now get the error in #458:

julia> Arrow.ArrowTypes.default(::Type{SubArray{T,N,P,I,L}}) where {T,N,P,I,L} = Arrow.ArrowTypes.default(P);

julia> Arrow.toarrowvector(t);
ERROR: MethodError: Cannot `convert` an object of type Vector{NamedTuple{(:x, :y), Tuple{Int64, Union{Missing, Int64}}}} to an object of type SubArray{NamedTuple{(:x, :y), Tuple{Int64, Union{Missing, Int64}}}, 1, Vector{NamedTuple{(:x, :y), Tuple{Int64, Union{Missing, Int64}}}}, Tuple{UnitRange{Int64}}, true}

[quinnj]

Ok, I better understand the issue this PR is trying to fix and I agree there's an actual bug, but it seems different from what is reported in #458? @baumgold, can you confirm why you think it's the same issue?

[quinnj]

Oh wait, I think I understand now: you're saying if we use your proposed change in this PR we end up with the error in #458. Ok, got it. Let me dig in a bit to think on a solution here.

[quinnj]

If we can commit my suggestion and then add @baumgold's repro as a test, I think this is good to go

[baumgold]

@quinnj - thanks for your feedback and suggestions. I've updated the code per your recommendations.

[baumgold]

ArrowTypes tests pass but Arrow tests fail because they are not using the newly fixed ArrowTypes version. @quinnj - shall we merge anyways or is there a way to get the Arrow tests to pass?

[quinnj]

I commented out the test for now; will re-enable after we tag another release.