BasicOptimizer.scala:89 executed in 132.93 seconds (1.774 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 76983471283000
Reset training subject: 76986181434500
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
New Minimum: -18.137234024110974 > -18.162316308594225
th(0)=-18.162316308594225;dx=-8.305330566780414E-7
New Minimum: -18.162316308594225 > -18.162317729873127
WOLFE (weak): th(2.154434690031884)=-18.162317729873127; dx=-8.035082933232554E-7 evalInputDelta=1.4212789025691563E-6
New Minimum: -18.162317729873127 > -18.16231986179127
WOLFE (weak): th(4.308869380063768)=-18.16231986179127; dx=-8.033972184128335E-7 evalInputDelta=3.553197046812784E-6
New Minimum: -18.16231986179127 > -18.162326968183287
WOLFE (weak): th(12.926608140191302)=-18.162326968183287; dx=-8.036593705456966E-7 evalInputDelta=1.065958906210085E-5
New Minimum: -18.162326968183287 > -18.16236178946396
WOLFE (weak): th(51.70643256076521)=-18.16236178946396; dx=-8.035790633116789E-7 evalInputDelta=4.548086973699128E-5
New Minimum: -18.16236178946396 > -18.162546554326134
WOLFE (weak): th(258.53216280382605)=-18.162546554326134; dx=-8.035491040909299E-7 evalInputDelta=2.302457319096618E-4
New Minimum: -18.162546554326134 > -18.163701292133062
WOLFE (weak): th(1551.1929768229563)=-18.163701292133062; dx=-8.034187879303406E-7 evalInputDelta=0.0013849835388377585
New Minimum: -18.163701292133062 > -18.172006135631136
WOLFE (weak): th(10858.350837760694)=-18.172006135631136; dx=-8.025768643073865E-7 evalInputDelta=0.00968982703691168
New Minimum: -18.172006135631136 > -18.2394523182677
WOLFE (weak): th(86866.80670208555)=-18.2394523182677; dx=-7.938061400825296E-7 evalInputDelta=0.0771360096734739
New Minimum: -18.2394523182677 > -18.81105016069641
END: th(781801.26031877)=-18.81105016069641; dx=-6.763815552475745E-7 evalInputDelta=0.6487338521021861
Fitness changed from -18.137234024110974 to -18.81105016069641
Iteration 1 complete. Error: -18.81105016069641 Total: 33.7263; Orientation: 0.0360; Line Search: 25.9143
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 3803ad4e to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-18.81105016069641;dx=-6.753017335870812E-7
New Minimum: -18.81105016069641 > -19.908944668321794
END: th(1684339.7559414052)=-19.908944668321794; dx=-4.959241889048334E-7 evalInputDelta=1.0978945076253837
Fitness changed from -18.81105016069641 to -19.908944668321794
Iteration 2 complete. Error: -19.908944668321794 Total: 8.2879; Orientation: 0.0518; Line Search: 5.2125
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 46ecfd3b to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-19.908944668321794;dx=-5.90877342925589E-7
New Minimum: -19.908944668321794 > -21.48019395816436
END: th(3628800.0)=-21.48019395816436; dx=-2.868514387255019E-7 evalInputDelta=1.5712492898425658
Fitness changed from -19.908944668321794 to -21.48019395816436
Iteration 3 complete. Error: -21.48019395816436 Total: 8.2761; Orientation: 0.0590; Line Search: 5.1272
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 72d588e8 to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-21.48019395816436;dx=-6.029539444796555E-7
New Minimum: -21.48019395816436 > -23.185941205182335
END: th(7818012.6031877)=-23.185941205182335; dx=-9.414077050637229E-8 evalInputDelta=1.7057472470179746
Fitness changed from -21.48019395816436 to -23.185941205182335
Iteration 4 complete. Error: -23.185941205182335 Total: 8.2031; Orientation: 0.0592; Line Search: 5.1926
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 227ad67 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.857e+04, gradient 7.587e-04, dot -0.526; [c2800ba5-83a9-4d77-9e0a-616af051b60c = 1.000/1.000e+00]
Orientation rejected. Popping history element from -23.185941205182335, -21.48019395816436, -19.908944668321794, -18.81105016069641
LBFGS Accumulation History: 3 points
Removed measurement 227ad67 to history. Total: 3
th(0)=-23.185941205182335;dx=-5.75435281872773E-7
New Minimum: -23.185941205182335 > -25.017939996708883
WOLF (strong): th(1.684339755941405E7)=-25.017939996708883; dx=3.0959614888829586E-8 evalInputDelta=1.8319987915265479
New Minimum: -25.017939996708883 > -25.33503063797296
WOLF (strong): th(8421698.779707026)=-25.33503063797296; dx=6.9429999293918825E-9 evalInputDelta=2.1490894327906247
END: th(2807232.9265690087)=-24.67983893463946; dx=-3.282980862787539E-7 evalInputDelta=1.4938977294571245
Fitness changed from -23.185941205182335 to -25.33503063797296
Iteration 5 complete. Error: -25.33503063797296 Total: 14.1779; Orientation: 1.0570; Line Search: 10.2242
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 3b7a54b2 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.669e+04, gradient 8.102e-04, dot -0.410; [c2800ba5-83a9-4d77-9e0a-616af051b60c = 1.000/1.000e+00]
Orientation rejected. Popping history element from -25.33503063797296, -21.48019395816436, -19.908944668321794, -18.81105016069641
LBFGS Accumulation History: 3 points
Removed measurement 3b7a54b2 to history. Total: 3
th(0)=-25.33503063797296;dx=-6.562684782297339E-7
New Minimum: -25.33503063797296 > -27.49938684080523
WOLF (strong): th(6048000.0)=-27.49938684080523; dx=4.9946686108639926E-8 evalInputDelta=2.16435620283227
END: th(3024000.0)=-27.131465590508263; dx=-3.2784876561862107E-7 evalInputDelta=1.7964349525353036
Fitness changed from -25.33503063797296 to -27.49938684080523
Iteration 6 complete. Error: -27.49938684080523 Total: 11.8725; Orientation: 1.0880; Line Search: 7.8863
Adding measurement 15db7b29 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.440e+04, gradient 7.832e-04, dot -0.434; [c2800ba5-83a9-4d77-9e0a-616af051b60c = 1.000/1.000e+00]
Orientation rejected. Popping history element from -27.49938684080523, -21.48019395816436, -19.908944668321794, -18.81105016069641
LBFGS Accumulation History: 3 points
Removed measurement 15db7b29 to history. Total: 3
th(0)=-27.49938684080523;dx=-6.132095457250917E-7
New Minimum: -27.49938684080523 > -28.53429097750924
WOLF (strong): th(6515010.502656417)=-28.53429097750924; dx=1.7372982210445983E-7 evalInputDelta=1.0349041367040108
New Minimum: -28.53429097750924 > -28.92825215525286
END: th(3257505.2513282085)=-28.92825215525286; dx=-6.730713684409735E-8 evalInputDelta=1.4288653144476307
Fitness changed from -27.49938684080523 to -28.92825215525286
Iteration 7 complete. Error: -28.92825215525286 Total: 11.2834; Orientation: 1.0333; Line Search: 7.7747
Adding measurement 14e046d5 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.410e+04, gradient 6.765e-04, dot -0.535; [c2800ba5-83a9-4d77-9e0a-616af051b60c = 1.000/1.000e+00]
Orientation rejected. Popping history element from -28.92825215525286, -21.48019395816436, -19.908944668321794, -18.81105016069641
LBFGS Accumulation History: 3 points
Removed measurement 14e046d5 to history. Total: 3
th(0)=-28.92825215525286;dx=-4.574262558380677E-7
New Minimum: -28.92825215525286 > -29.27741301669305
WOLF (strong): th(7018082.316422523)=-29.27741301669305; dx=6.512942133599804E-8 evalInputDelta=0.34916086144018976
New Minimum: -29.27741301669305 > -29.569468711726334
WOLF (strong): th(3509041.1582112615)=-29.569468711726334; dx=5.021078230202324E-8 evalInputDelta=0.6412165564734735
END: th(1169680.3860704205)=-29.418180028417737; dx=-2.631453443919235E-7 evalInputDelta=0.48992787316487707
Fitness changed from -28.92825215525286 to -29.569468711726334
Iteration 8 complete. Error: -29.569468711726334 Total: 13.7948; Orientation: 1.0111; Line Search: 10.3036
Adding measurement a45f94b to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.498e+04, gradient 7.442e-04, dot -0.474; [c2800ba5-83a9-4d77-9e0a-616af051b60c = 1.000/1.000e+00]
Orientation rejected. Popping history element from -29.569468711726334, -21.48019395816436, -19.908944668321794, -18.81105016069641
LBFGS Accumulation History: 3 points
Removed measurement a45f94b to history. Total: 3
th(0)=-29.569468711726334;dx=-5.53394354184219E-7
New Minimum: -29.569468711726334 > -30.390853702493114
WOLF (strong): th(2520000.0000000005)=-30.390853702493114; dx=5.68793493120272E-8 evalInputDelta=0.8213849907667807
END: th(1260000.0000000002)=-30.22729482928947; dx=-3.290953368588609E-7 evalInputDelta=0.6578261175631361
Fitness changed from -29.569468711726334 to -30.390853702493114
Iteration 9 complete. Error: -30.390853702493114 Total: 11.5157; Orientation: 1.0448; Line Search: 8.0836
Adding measurement 64dc6fe8 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.036e+04, gradient 6.761e-04, dot -0.527; [c2800ba5-83a9-4d77-9e0a-616af051b60c = 1.000/1.000e+00]
Orientation rejected. Popping history element from -30.390853702493114, -21.48019395816436, -19.908944668321794, -18.81105016069641
LBFGS Accumulation History: 3 points
Removed measurement 64dc6fe8 to history. Total: 3
th(0)=-30.390853702493114;dx=-4.5688208628686634E-7
New Minimum: -30.390853702493114 > -30.807478220762704
WOLF (strong): th(2714587.709440174)=-30.807478220762704; dx=9.62151150689838E-8 evalInputDelta=0.41662451826958957
New Minimum: -30.807478220762704 > -30.850283825393802
END: th(1357293.854720087)=-30.850283825393802; dx=-1.0614851137317498E-7 evalInputDelta=0.4594301229006881
Fitness changed from -30.390853702493114 to -30.850283825393802
Iteration 10 complete. Error: -30.850283825393802 Total: 11.3132; Orientation: 1.0128; Line Search: 7.8518
<a id="p-13"></a>Iteration 10
<a id="p-12"></a>
Final threshold in iteration 11: -30.850283825393802 (> -Infinity) after 132.930s (< 5400.000s)
Returns
-30.850283825393802