I'd submit a pull request, but the logic is likely going to get heavily overwritten by a memory-efficient loader, so I'll post this here and we'll see what happens.
As written (hardmaru): load N episodes and add M*N total images to dateset.
As written (Chazzz): load N episodes and add a maximum of M images each to dateset.
--- a/vae_train.py
+++ b/vae_train.py
@@ -47,19 +47,50 @@ def create_dataset(filelist, N=10000, M=1000): # N is 10000 episodes, M is 1000 num
for i in range(N):
filename = filelist[i]
raw_data = np.load(os.path.join("record", filename))['obs']
- l = len(raw_data)
+ l = min(len(raw_data), M)
- if (idx+l) > (M*N):
- data = data[0:idx]
- print('premature break')
- break
- data[idx:idx+l] = raw_data
+ data[idx:idx+l] = raw_data[0:l]
idx += l
if ((i+1) % 100 == 0):
print("loading file", i+1)
+
+ if len(data) == M*N and idx < M*N:
+ data = data[:idx]
return data
I'd submit a pull request, but the logic is likely going to get heavily overwritten by a memory-efficient loader, so I'll post this here and we'll see what happens.
As written (hardmaru): load N episodes and add M*N total images to dateset.
As written (Chazzz): load N episodes and add a maximum of M images each to dateset.