[Cryptography] Computing the nonlinearity of an 8×8 S-box

[Pierre Abbat]

One of my ciphers has an 8×8 S-box which is a permutation, which is further 
permuted by the argument and value being multiplied by bytes using two 
different multiplications. I wrote the following function, which takes a 256-
byte vector starting at index 0:

function nonlinearity(bytes::OffsetVector{UInt8})
  buf=OffsetVector(Float64[],-1)
  for b in bytes
    for i in 0:7
      push!(buf,((b>>i)&1)*-2+1)
    end
  end
  had=hadamard(buf)
  maxNonlin=√(length(had))
  (maxNonlin-maximum(abs.(had)))/maxNonlin
end

The Hadamard transform is a valid way of computing the nonlinearity of a 
Boolean function, but I'm not sure of the proper way of computing the 
nonlinearity of a function whose output is a Boolean vector. With the output 
being an 8-bit number, it computes an answer (and most of the permutations are 
at least 90% nonlinear), but if I were using 5×5 S-boxes it wouldn't work, as 
the hadamard function does nothing when the vector size is not a power of 2.

I've read some articles I found online, but couldn't quite understand how to 
handle functions with multiple-bit outputs. How can I do it?

Pierre
-- 
li fi'u vu'u fi'u fi'u du li pa