Tensorsize do not match

[leela-uppuluri]

Hello Everyone,

When I am trying to run the transfer-learning code with these parameters [4 executors -> 23 cores -> 4*23=92 and again 92*4=368 batch_size] to submit the job either in the notebook or spark-submit it completes the model training successfully but at the time of "Validate Model" I am seeing this error.

I really appreciate if there are any suggestions or solutions for this issue to close.

[hhbyyh]

Hi @leela-uppuluri
.setValidation(EveryEpoch(), validationDF, [AUC()], 4)

According to the description, the problem occurs during validation period. Please note that the batchsize for validation is set via the fourth parameter in setValidation.
Please check:

1. what's the validation batchsize did you set?
2. is there enough validation data records in total? (more than one validation batch size)

[hhbyyh]

I tried with the following code and it runs without error

classifier = NNEstimator(lrModel, MultiLabelSoftMarginCriterion(), transformer, SeqToTensor([15])) \
     .setLearningRate(0.005).setBatchSize(4).setMaxEpoch(8).setFeaturesCol("image")\
     .setCachingSample(False)\
     .setValidation(EveryEpoch(), validationDF, [AUC()], 4)\
     .setTrainSummary(train_summary) \
     .setValidationSummary(val_summary) \
     .setCheckpoint("/home/yuhao/workspace/github/hhbyyh/BigDL-ImageProcessing-Examples/checkpoints", EveryEpoch(), False)

[leela-uppuluri]

Hi @hhbyyh
I have given the same batchsize for .setValidation() which is similar to .setBatchSize() in the classifier. Here is my code,

batch_size = 368
transformer = ChainedPreprocessing(
    [RowToImageFeature(), ImageCenterCrop(224, 224), BigDLAdapter(RandomTransformer(HFlip(), 0.5)), ImageChannelNormalize(123.0, 117.0, 104.0), ImageMatToTensor(), ImageFeatureToTensor()])

classifier = NNEstimator(lrModel, MultiLabelSoftMarginCriterion(), transformer, SeqToTensor([15])) \
    .setLearningRate(0.001).setBatchSize(batch_size).setMaxEpoch(1).setFeaturesCol("image") \
    .setCachingSample(False) \
    .setValidation(EveryEpoch(), validationDF, [AUC()], batch_size) \
    .setTrainSummary(train_summary) \
    .setValidationSummary(val_summary) \
    .setCheckpoint("/home/leelau/chestXray/checkpoint-pv", EveryEpoch(), False)

nnModel = classifier.fit(trainingDF)
print("Finished training")