Data generator for training models

[dmrd]

Run out of 16 GB at ~30000 games of training data.

• Support IO to hf5 files
• Create an online data generator that reads sgf files and augments the data

[dmrd]

Requires using master on PyCall, but a generator is actually totally doable from Julia:

julia> @pydef type Generator
           __init__(self, tuples) = (self[:tuples] = tuples; self[:i] = 1)
           next(self) = begin
               i = self[:i]
               self[:i] = max((i + 1) % length(self[:tuples]), 1)
               return self[:tuples][i]
           end
       end
PyObject <class 'Generator'>

julia> pyeval("[next(x) for i in range(0,10)]", x=Generator([1,2,3,4,5]))
10-element Array{Any,1}:
 1
 2
 3
 4
 1
 2
 3
 4
 1
 2