ArraysOfArrays.jl

A Julia package for efficient storage and handling of nested arrays. ArraysOfArrays provides two different types of nested arrays: ArrayOfSimilarArrays and VectorOfArrays.

This package also defines and exports the following new functions applicable to nested arrays in general:

• sliced and flatview switch between a flat and a nested (sliced) view of the same data.
• partitioned creates a view of a vector as a vector of arrays that may differ in size.
• getsplitmode, splitup and fused provide a general API to split arrays into nested (sliced or partitioned) form and to fuse them back into flat form.
• stacked and unstackmode are similar to Base.stack, but can return the original underlying data of sliced arrays without copying it.
• vecflattened concatenates the elements of nested arrays into a single vector.
• innersize returns the size of the elements of an array, provided they all have equal size.
• innermap and deepmap apply a function to the elements of the inner (resp. innermost) arrays.
• consgroupedview computes a grouping of equal consecutive elements on a vector and applies it to another vector or (named or unnamed) tuple of vectors.

Nested array types

ArraysOfArrays provides two nested-array representations. They are distinct concrete types (neither is a subtype of the other) that cover different storage needs:

Here M is the dimensionality of the inner (element) arrays and N that of the outer container.

Both keep their data in a single underlying array that is always available via flatview (and fused), so switching between the flat and the nested view is zero-copy.

Convenience aliases name the common special cases (see the Type aliases sections below). In particular, VectorOfSimilarVectors is a vector of equal-length vectors, and PartsView (also exported as VectorOfVectors) is a vector of vectors that may differ in length.

The abstract type AbstractArrayOfSimilarArrays is the supertype for custom equal-size nested-array implementations; VectorOfArrays has no package-specific abstract supertype.

ArraysOfArrays also supports Base.Slices.

Operations at a given nesting depth

Similar to axis-targeted operations in Python's AwkwardArrays, but with array-of-arrays nesting semantics:

• mapat(f, Val(d), As...) maps f over the objects at nesting depth d (d = 1 ≡ map, d = 2 ≡ innermap).
• bcastat(f, Val(d), args...) broadcasts f at depth d: nested arguments align; shallower arrays contribute one value per element of their level; scalars broadcast over everything.
• innermapreduce, innerreduce and innersum reduce over the contents of each element array.
• innersizes and innerlengths return per-element sizes/lengths (elements need not be of equal size).

For split arrays these operate on the underlying flat data, without per-element iteration, and so work on GPU arrays. Outer-level broadcasts like (x -> 2 .* x).(A) keep their usual Julia semantics (f receives whole element arrays), but return a VectorOfArrays when the results are arrays.

Which flattening function do I want?

• fused(A): the original underlying array, splitup(fused(A), getsplitmode(A)) == A.
• flatview(A): the underlying storage, requires a memory-ordered layout.
• stacked(A): elements joined along new trailing dimensions, like Base.stack.
• vecflattened(A): elements concatenated into a single vector, like reduce(vcat, A).

All four are zero-copy where possible and so may return arrays that share memory with A. In contrast, Base.stack(A) and reduce(vcat, A) always return independent arrays.

GPU support

Both array types work with GPU-resident data. An ArrayOfSimilarArrays backed by a GPU array requires no special handling. For a VectorOfArrays, the shape information (elem_ptr and kernel_size) can either stay on the host, so that element access returns device views, or live on the device as well (e.g. via Adapt.adapt), which is the layout to use inside GPU kernels. Host-side element access (V[i], iteration) requires the shape information on the host, as it would otherwise scalar-index device arrays. KernelAbstractions.get_backend returns the backend of the underlying data.

Reductions over the flat data (innersum, innerreduce, innermapreduce) run as segmented device kernels. Generic map(f, V) and outer broadcasts f.(V) with scalar results also run on the device for a device-resident VectorOfArrays with vector elements (VectorOfArrays{T,1}), provided f is device-compatible (this includes map(sum, V), which retains Base's sum semantics); the shape-insensitive reductions maximum and minimum (via map(maximum, V)/map(minimum, V)) additionally use the segmented reduction kernels for any element dimensionality, and map(argmin, V)/map(argmax, V) match Base (including NaN and signed-zero ordering) for vector elements. For VectorOfArrays{T,N} with N > 1, map keeps its generic behavior: it is correct when the shape information is host-resident and fails cleanly (rather than dropping the element shape) when it is device-resident.

ArrayOfSimilarArrays

An ArrayOfSimilarArrays offers a duality of view between representing the same data as both a flat multi-dimensional array and as an array of equally-sized arrays:

A_flat = rand(2,3,4,5,6)
A_nested = sliced(A_flat, 2)

creates a view of A_flat as an array of arrays:

A_nested isa AbstractArray{<:AbstractArray{T,2},3} where T

A_flat is always available via flatview. A_flat and A_nested are backed by the same data, no data is copied:

flatview(A_nested) === A_flat

Calling getindex on A_nested returns a view into A_flat:

fill!(A_nested[2, 4, 3], 4.2)
all(x -> x == 4.2, A_flat[:, :, 2, 4, 3])

Type aliases

The following type aliases are defined:

• VectorOfSimilarArrays{T,M} = ArrayOfSimilarArrays{T,M,1}
• ArrayOfSimilarVectors{T,N} = ArrayOfSimilarArrays{T,1,N}
• VectorOfSimilarVectors{T} = ArrayOfSimilarArrays{T,1,1}

For each of the types there is also an abstract type (AbstractArrayOfSimilarArrays, etc.).

If a VectorOfSimilarArrays is backed by an ElasticArrays.ElasticArray, additional element arrays can be pushed into it and resize! is available too:

Appending data and resizing

using ElasticArrays

A_nested = sliced(ElasticArray{Float64}(undef, 2, 3, 0), 2)

for i in 1:4
    push!(A_nested, rand(2, 3))
end
size(flatview(A_nested)) == (2, 3, 4)

resize!(A_nested, 6)
size(flatview(A_nested)) == (2, 3, 6)

There is a full duality between the nested and the flat view of the data. A_flat may be resized freely without breaking the inner consistency of A_nested: Changes in the shape of one will result in changes in the shape of the other.

VectorOfArrays

A VectorOfArrays represents a vector of arrays of equal dimensionality but different size. It is a nested interpretation of the concept of a "ragged array".

VA = VectorOfArrays{Float64, 2}()

push!(VA, rand(2, 3))
push!(VA, rand(4, 2))

size(VA[1]) == (2,3)
size(VA[2]) == (4,2)

Internally, all data is stored efficiently in a single, flat and memory-contiguous vector, accessible via flatview:

VA_flat = flatview(VA)
VA_flat isa Vector{Float64}

Calling getindex on VA returns a view into VA_flat:

VA_flat = flatview(VA)
view(VA_flat, 7:14) == vec(VA[2])

fill!(view(VA_flat, 7:14), 2.4)
all(x -> x == 2.4, VA[2])

fill!(view(VA_flat, 7:14), 4.2)
all(x -> x == 4.2, VA[2])

Type aliases

The following type aliases are defined:

• PartsView{T,VT,VI,VD,ET} = VectorOfArrays{T,1,0,VT,VI,VD,ET}
• VectorOfVectors = PartsView

Appending data and resizing

A VectorOfArrays is grown by appending data to it. resize! can be used to shrink it, but not to grow it (the size of the additional element arrays would be unknown):

length(resize!(VA, 1)) == 1

but

resize!(VA, 4)

will fail.

Note: The vector returned by flatview(VA) must not be resized directly, doing so would break the internal consistency of VA.