module ArraysOfArrays

Efficient storage and handling of nested arrays.

ArraysOfArrays provides two different types of nested arrays: ArrayOfSimilarArrays and VectorOfArrays.

AbstractArrayOfSimilarArrays{T,M,N} <: AbstractArray{<:AbstractArray{T,M},N}

An array that contains arrays that have the same size/axes. The array is internally stored in flattened form as some kind of array of dimension M + N. The flattened form can be accessed via flatview(A).

Subtypes must implement (in addition to typical array operations):

flatview(A::SomeArrayOfSimilarArrays)::AbstractArray{T,M+N}

The following type aliases are defined:

• AbstractVectorOfSimilarArrays{T,M} = AbstractArrayOfSimilarArrays{T,M,1}
• AbstractArrayOfSimilarVectors{T,N} = AbstractArrayOfSimilarArrays{T,1,N}
• AbstractVectorOfSimilarVectors{T} = AbstractArrayOfSimilarArrays{T,1,1}

ArrayOfSimilarArrays{T,M,N,L,P} <: AbstractArrayOfSimilarArrays{T,M,N}

Represents a view of an array of dimension L = M + N as an array of dimension M with elements that are arrays with dimension N. All element arrays implicitly have equal size/axes.

Constructors:

ArrayOfSimilarArrays{T,M,N}(flat_data::AbstractArray)
ArrayOfSimilarArrays{T,M}(flat_data::AbstractArray)

The following type aliases are defined:

• VectorOfSimilarArrays{T,M} = AbstractArrayOfSimilarArrays{T,M,1}
• ArrayOfSimilarVectors{T,N} = AbstractArrayOfSimilarArrays{T,1,N}
• VectorOfSimilarVectors{T} = AbstractArrayOfSimilarArrays{T,1,1}

VectorOfSimilarArrays supports push!(), etc., provided the underlying array supports resizing of it's last dimension (e.g. an ElasticArray).

The nested array can also be created using the function nestedview and the wrapped flat array can be accessed using flatview afterwards:

A_flat = rand(2,3,4,5,6)
A_nested = nestedview(A_flat, 2)
A_nested isa AbstractArray{<:AbstractArray{T,2},3} where T
flatview(A_nested) === A_flat

VectorOfArrays{T,N,M} <: AbstractVector{<:AbstractArray{T,N}}

An VectorOfArrays represents a vector of N-dimensional arrays (that may differ in size). Internally, VectorOfArrays stores all elements of all arrays in a single flat vector. M must equal N - 1

The VectorOfArrays itself supports push!, unshift!, etc., but the size of each individual array in the vector is fixed. resize! can be used to shrink, but not to grow, as the size of the additional element arrays in the vector would be unknown. However, memory space for up to n arrays with a maximum size s can be reserved via sizehint!(A::VectorOfArrays, n, s::Dims{N}).

Constructors:

VectorOfArrays{T,N}()

VectorOfArrays(A::AbstractVector{<:AbstractArray})
VectorOfArrays{T}(A::AbstractVector{<:AbstractArray})
VectorOfArrays{T,N}(A::AbstractVector{<:AbstractArray})

VectorOfArrays(
    data::AbstractVector,
    elem_ptr::AbstractVector{<:Integer},
    kernel_size::AbstractVector{<:Dims}
    checks::Function = ArraysOfArrays.full_consistency_checks
)

Other suitable values for checks are ArraysOfArrays.simple_consistency_checks and ArraysOfArrays.no_consistency_checks.

VectorOfVectors is defined as an type alias:

`VectorOfVectors{T,VT,VI,VD} = VectorOfArrays{T,1,VT,VI,VD}`

VectorOfVectors{T,...} = VectorOfArrays{T,1,...}

Constructors:

VectorOfVectors(A::AbstractVector{<:AbstractVector})
VectorOfVectors{T}(A::AbstractVector{<:AbstractVector}) where {T}

VectorOfVectors(
    data::AbstractVector, elem_ptr::AbstractVector{<:Integer},
    checks::Function = full_consistency_checks
)

See also [VectorOfArrays](@ref).

abstract_nestedarray_type(T_inner::Type, ::Val{ndims_tuple})

Return the type of nested AbstractArrays. T_inner specifies the element type of the innermost layer of arrays, ndims_tuple specifies the dimensionality of each nesting layer (outer arrays first).

If ndims_tuple is empty, the returns is the (typically scalar) type T_inner itself.

consgrouped_ptrs(A::AbstractVector)

Compute an element pointer vector, suitable for creation of a VectorOfVectors that implies grouping equal consecutive entries of A.

Example:

    A = [1, 1, 2, 3, 3, 2, 2, 2]
    elem_ptr = consgrouped_ptrs(A)
    first.(VectorOfVectors(A, elem_ptr)) == [1, 2, 3, 2]

consgroupedptrs Typically, `elemptr` will be used to apply the computed grouping to other data:

    B = [1, 2, 3, 4, 5, 6, 7, 8]
    VectorOfVectors(B, elem_ptr) == [[1, 2], [3], [4, 5], [6, 7, 8]]

consgroupedview(source::AbstractVector, target)

Compute a grouping of equal consecutive elements on source via consgrouped_ptrs and apply the grouping to target, resp. each element of target. target may be an vector or a named or unnamed tuple of vectors. The result is a VectorOfVectors, resp. a tuple of such.

Example:

A = [1, 1, 2, 3, 3, 2, 2, 2]
B = [1, 2, 3, 4, 5, 6, 7, 8]
consgroupedview(A, B) == [[1, 2], [3], [4, 5], [6, 7, 8]]

consgroupedview plays well with columnar tables, too:

    using Tables, TypedTables
    data = Table(
        a = [1, 1, 2, 3, 3, 2, 2, 2],
        b = [1, 2, 3, 4, 5, 6, 7, 8],
        c = [1.1, 2.2, 3.3, 4.4, 5.5, 6.6, 7.7, 8.8]
    )

    result = Table(consgroupedview(data.a, Tables.columns(data)))

will return

     a          b          c
   ┌──────────────────────────────────────
 1 │ [1, 1]     [1, 2]     [1.1, 2.2]
 2 │ [2]        [3]        [3.3]
 3 │ [3, 3]     [4, 5]     [4.4, 5.5]
 4 │ [2, 2, 2]  [6, 7, 8]  [6.6, 7.7, 8.8]

without copying any data:

    flatview(result.a) === data.a
    flatview(result.b) === data.b
    flatview(result.c) === data.c

deepgetindex(A::AbstractArray, idxs...)
deepgetindex(A::AbstractArray{<:AbstractArray, N}, idxs...) where {N}

Recursive getindex on flat or nested arrays. If A is an array of arrays, uses the first N entries in idxs on A, then the rest on the inner array(s). If A is not a nested array, deepgetindex is equivalent to getindex.

See also deepsetindex! and deepview.

deepmap(f::Base.Callable, x::Any)
deepmap(f::Base.Callable, A::AbstractArray{<:AbstractArray{<:...}})

Applies map at the deepest possible layer of nested arrays. If A is not a nested array, deepmap behaves identical to Base.map.

deepsetindex!(A::AbstractArray, x, idxs...)
deepsetindex!(A::AbstractArray{<:AbstractArray,N}, x, idxs...) where {N}

Recursive setindex! on flat or nested arrays. If A is an array of arrays, uses the first N entries in idxs on A, then the rest on the inner array(s). If A is not a nested array, deepsetindex! is equivalent to setindex!.

See also deepgetindex and deepview.

deepview(A::AbstractArray, idxs...)
deepview(A::AbstractArray{<:AbstractArray, N}, idxs...) where {N}

Recursive view on flat or nested arrays. If A is an array of arrays, uses the first N entries in idxs on A, then the rest on the inner array(s). If A is not a nested array, deepview is equivalent to view.

See also deepgetindex and deepsetindex!.

element_ptr(A::VectorOfArrays)

Returns a copy of the internal element pointer vector of A.

flatview(A::AbstractArray)
flatview(A::AbstractArray{<:AbstractArray{<:...}})

View array A in a suitable flattened form. The shape of the flattened form will depend on the type of A. If the A is not a nested array, the return value is A itself. When no type-specific method is available, flatview will fall back to Base.Iterators.flatten.

flatview(A::ArrayOfSimilarArrays{T,M,N,L,P})::P

Returns the array of dimensionality L = M + N wrapped by A. The shape of the result may be freely changed without breaking the inner consistency of A.

flatview(A::VectorOfArrays{T})::Vector{T}

Returns the internal serialized representation of all element arrays of A as a single vector. Do not change the length of the returned vector, as it would break the inner consistency of A.

innermap(f::Base.Callable, A::AbstractArray{<:AbstractArray})

Nested map at depth 2. Equivalent to map(X -> map(f, X) A).

innersize(A:AbstractArray{<:AbstractArray}, [dim])

Returns the size of the element arrays of A. Fails if the element arrays are not of equal size.

internal_element_ptr(A::VectorOfArrays)

Returns the internal element pointer vector of A.

Do not change modify the returned vector in any way, as this would break the inner consistency of A.

Use with care, see element_ptr for a safe version of this function.

nestedview(A::AbstractArray{T,M+N}, M::Integer)
nestedview(A::AbstractArray{T,2})

AbstractArray{<:AbstractArray{T,M},N}

View array A in as an N-dimensional array of M-dimensional arrays by wrapping it into an ArrayOfSimilarArrays.

It's also possible to use a StaticVector of length S as the type of the inner arrays via

nestedview(A::AbstractArray{T}, ::Type{StaticArrays.SVector{S}})
nestedview(A::AbstractArray{T}, ::Type{StaticArrays.SVector{S,T}})