Advanced techniques

Structures with non-standard data layout

StructArrays support structures with custom data layout. The user is required to overload staticschema in order to define the custom layout, component to access fields of the custom layout, and createinstance(T, fields...) to create an instance of type T from its custom fields fields. In other word, given x::T, createinstance(T, (component(x, f) for f in fieldnames(staticschema(T)))...) should successfully return an instance of type T.

Here is an example of a type MyType that has as custom fields either its field data or fields of its field rest (which is a named tuple):

julia> using StructArrays
julia> struct MyType{T, NT<:NamedTuple}
           data::T
           rest::NT
       end
julia> MyType(x; kwargs...) = MyType(x, values(kwargs))Main.MyType

Let's create a small array of these objects:

julia> s = [MyType(i/5, a=6-i, b=2) for i in 1:5]5-element Vector{Main.MyType{Float64, @NamedTuple{a::Int64, b::Int64}}}:
 Main.MyType{Float64, @NamedTuple{a::Int64, b::Int64}}(0.2, (a = 5, b = 2))
 Main.MyType{Float64, @NamedTuple{a::Int64, b::Int64}}(0.4, (a = 4, b = 2))
 Main.MyType{Float64, @NamedTuple{a::Int64, b::Int64}}(0.6, (a = 3, b = 2))
 Main.MyType{Float64, @NamedTuple{a::Int64, b::Int64}}(0.8, (a = 2, b = 2))
 Main.MyType{Float64, @NamedTuple{a::Int64, b::Int64}}(1.0, (a = 1, b = 2))

The default StructArray does not unpack the NamedTuple:

julia> sa = StructArray(s);
julia> sa.rest5-element Vector{@NamedTuple{a::Int64, b::Int64}}:
 (a = 5, b = 2)
 (a = 4, b = 2)
 (a = 3, b = 2)
 (a = 2, b = 2)
 (a = 1, b = 2)
julia> sa.aERROR: type NamedTuple has no field a

Suppose we wish to give the keywords their own fields. We can define custom staticschema, component, and createinstance methods for MyType:

julia> function StructArrays.staticschema(::Type{MyType{T, NamedTuple{names, types}}}) where {T, names, types}
           # Define the desired names and eltypes of the "fields"
           return NamedTuple{(:data, names...), Base.tuple_type_cons(T, types)}
       end;
julia> function StructArrays.component(m::MyType, key::Symbol)
           # Define a component-extractor
           return key === :data ? getfield(m, 1) : getfield(getfield(m, 2), key)
       end;
julia> function StructArrays.createinstance(::Type{MyType{T, NT}}, x, args...) where {T, NT}
           # Generate an instance of MyType from components
           return MyType(x, NT(args))
       end;

and now:

julia> sa = StructArray(s);
julia> sa.a5-element Vector{Int64}:
 5
 4
 3
 2
 1
julia> sa.b5-element Vector{Int64}:
 2
 2
 2
 2
 2

The above strategy has been tested and implemented in GeometryBasics.jl.

Mutate-or-widen style accumulation

StructArrays provides a function StructArrays.append!!(dest, src) (unexported) for "mutate-or-widen" style accumulation. This function can be used via BangBang.append!! and BangBang.push!! as well.

StructArrays.append!! works like append!(dest, src) if dest can contain all element types in src iterator; i.e., it mutates dest in-place:

julia> dest = StructVector((a=[1], b=[2]))
1-element StructArray(::Array{Int64,1}, ::Array{Int64,1}) with eltype NamedTuple{(:a, :b),Tuple{Int64,Int64}}:
 (a = 1, b = 2)

julia> StructArrays.append!!(dest, [(a = 3, b = 4)])
2-element StructArray(::Array{Int64,1}, ::Array{Int64,1}) with eltype NamedTuple{(:a, :b),Tuple{Int64,Int64}}:
 (a = 1, b = 2)
 (a = 3, b = 4)

julia> ans === dest
true

Unlike append!, append!! can also widen element type of dest array:

julia> StructArrays.append!!(dest, [(a = missing, b = 6)])
3-element StructArray(::Array{Union{Missing, Int64},1}, ::Array{Int64,1}) with eltype NamedTuple{(:a, :b),Tuple{Union{Missing, Int64},Int64}}:
 NamedTuple{(:a, :b),Tuple{Union{Missing, Int64},Int64}}((1, 2))
 NamedTuple{(:a, :b),Tuple{Union{Missing, Int64},Int64}}((3, 4))
 NamedTuple{(:a, :b),Tuple{Union{Missing, Int64},Int64}}((missing, 6))

julia> ans === dest
false

Since the original array dest cannot hold the input, a new array is created (ans !== dest).

Combined with function barriers, append!! is a useful building block for implementing collect-like functions.

Using StructArrays in CUDA kernels

It is possible to combine StructArrays with CUDAnative, in order to create CUDA kernels that work on StructArrays directly on the GPU. Make sure you are familiar with the CUDAnative documentation (esp. kernels with plain CuArrays) before experimenting with kernels based on StructArrays.

using CUDAnative, CuArrays, StructArrays
d = StructArray(a = rand(100), b = rand(100))

# move to GPU
dd = replace_storage(CuArray, d)
de = similar(dd)

# a simple kernel, to copy the content of `dd` onto `de`
function kernel!(dest, src)
    i = (blockIdx().x-1)*blockDim().x + threadIdx().x
    if i <= length(dest)
        dest[i] = src[i]
    end
    return nothing
end

threads = 1024
blocks = cld(length(dd),threads)

@cuda threads=threads blocks=blocks kernel!(de, dd)