BasicOptimizer.scala:89 executed in 234.51 seconds (3.160 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 5730883678413
Reset training subject: 5735937940139
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
th(0)=-14.489835530519485;dx=-2.285657066054327E-6
WOLFE (weak): th(2.154434690031884)=-14.489840984344482; dx=-2.124586221776351E-6 evalInputDelta=5.453824996948242E-6
WOLFE (weak): th(4.308869380063768)=-14.4898461997509; dx=-2.124071686195058E-6 evalInputDelta=1.0669231414794922E-5
WOLFE (weak): th(12.926608140191302)=-14.48986902832985; dx=-2.124589411633906E-6 evalInputDelta=3.349781036376953E-5
WOLFE (weak): th(51.70643256076521)=-14.489968299865723; dx=-2.1239610150041696E-6 evalInputDelta=1.3276934623718262E-4
WOLFE (weak): th(258.53216280382605)=-14.49050310254097; dx=-2.123501400102132E-6 evalInputDelta=6.67572021484375E-4
WOLFE (weak): th(1551.1929768229563)=-14.493841052055359; dx=-2.1220874910532043E-6 evalInputDelta=0.004005521535873413
WOLFE (weak): th(10858.350837760694)=-14.51775375008583; dx=-2.1027658351821873E-6 evalInputDelta=0.027918219566345215
New Minimum: -14.642043143510818 > -14.703680723905563
END: th(86866.80670208555)=-14.703680723905563; dx=-1.9124229752125364E-6 evalInputDelta=0.21384519338607788
Fitness changed from -14.642043143510818 to -14.703680723905563
Iteration 1 complete. Error: -14.703680723905563 Total: 61.4488; Orientation: 0.0213; Line Search: 45.9820
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 5427c4f to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-14.703680723905563;dx=-1.8479259063322905E-6
New Minimum: -14.703680723905563 > -15.086081326007843
END: th(187148.86177126726)=-15.086081326007843; dx=-1.495265817984533E-6 evalInputDelta=0.38240060210227966
Fitness changed from -14.703680723905563 to -15.086081326007843
Iteration 2 complete. Error: -15.086081326007843 Total: 15.5777; Orientation: 0.0301; Line Search: 10.4602
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 34bd52e2 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-15.086081326007843;dx=-1.3861258072843833E-6
New Minimum: -15.086081326007843 > -15.652033537626266
END: th(403200.00000000006)=-15.652033537626266; dx=-9.43969887894436E-7 evalInputDelta=0.5659522116184235
Fitness changed from -15.086081326007843 to -15.652033537626266
Iteration 3 complete. Error: -15.652033537626266 Total: 15.3773; Orientation: 0.0303; Line Search: 10.2359
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 71d82870 to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-15.652033537626266;dx=-9.699090192610912E-7
New Minimum: -15.652033537626266 > -16.356077790260315
END: th(868668.0670208557)=-16.356077790260315; dx=-4.817751121326108E-7 evalInputDelta=0.7040442526340485
Fitness changed from -15.652033537626266 to -16.356077790260315
Iteration 4 complete. Error: -16.356077790260315 Total: 16.2654; Orientation: 0.0297; Line Search: 10.9699
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 1052cfb2 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.369e+03, gradient 8.742e-04, dot -0.730; [28c36e8e-9d09-4a9f-afa3-3230b36b0e51 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -16.356077790260315, -15.652033537626266, -15.086081326007843, -14.703680723905563
LBFGS Accumulation History: 3 points
Removed measurement 1052cfb2 to history. Total: 3
th(0)=-16.356077790260315;dx=-7.636018240928695E-7
New Minimum: -16.356077790260315 > -16.956930696964264
END: th(1871488.617712673)=-16.956930696964264; dx=-1.2769343198571014E-7 evalInputDelta=0.600852906703949
Fitness changed from -16.356077790260315 to -16.956930696964264
Iteration 5 complete. Error: -16.956930696964264 Total: 15.8227; Orientation: 0.6264; Line Search: 10.1120
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 6ea51770 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.879e+03, gradient 8.880e-04, dot -0.494; [28c36e8e-9d09-4a9f-afa3-3230b36b0e51 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -16.956930696964264, -15.652033537626266, -15.086081326007843, -14.703680723905563
LBFGS Accumulation History: 3 points
Removed measurement 6ea51770 to history. Total: 3
th(0)=-16.956930696964264;dx=-7.874947087430379E-7
New Minimum: -16.956930696964264 > -17.076613783836365
WOLF (strong): th(4032000.0000000014)=-17.076613783836365; dx=1.1532187596553222E-7 evalInputDelta=0.11968308687210083
New Minimum: -17.076613783836365 > -17.417547583580017
WOLF (strong): th(2016000.0000000007)=-17.417547583580017; dx=1.005489288226531E-7 evalInputDelta=0.4606168866157532
END: th(672000.0000000002)=-17.38731810450554; dx=-2.7846399444199254E-7 evalInputDelta=0.430387407541275
Fitness changed from -16.956930696964264 to -17.417547583580017
Iteration 6 complete. Error: -17.417547583580017 Total: 25.9470; Orientation: 0.6199; Line Search: 20.0364
Adding measurement 3c57913 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.699e+03, gradient 9.003e-04, dot -0.401; [28c36e8e-9d09-4a9f-afa3-3230b36b0e51 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -17.417547583580017, -15.652033537626266, -15.086081326007843, -14.703680723905563
LBFGS Accumulation History: 3 points
Removed measurement 3c57913 to history. Total: 3
th(0)=-17.417547583580017;dx=-8.100360837083455E-7
New Minimum: -17.417547583580017 > -17.9812992811203
WOLF (strong): th(1447780.1117014263)=-17.9812992811203; dx=5.807514385002117E-8 evalInputDelta=0.5637516975402832
END: th(723890.0558507132)=-17.905725479125977; dx=-2.869854959807186E-7 evalInputDelta=0.4881778955459595
Fitness changed from -17.417547583580017 to -17.9812992811203
Iteration 7 complete. Error: -17.9812992811203 Total: 21.0543; Orientation: 0.6190; Line Search: 15.3983
Adding measurement 7d70a691 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.872e+03, gradient 6.866e-04, dot -0.445; [28c36e8e-9d09-4a9f-afa3-3230b36b0e51 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -17.9812992811203, -15.652033537626266, -15.086081326007843, -14.703680723905563
LBFGS Accumulation History: 3 points
Removed measurement 7d70a691 to history. Total: 3
th(0)=-17.9812992811203;dx=-4.709379928925471E-7
New Minimum: -17.9812992811203 > -18.36042693257332
WOLF (strong): th(1559573.8480938943)=-18.36042693257332; dx=1.434846143167621E-8 evalInputDelta=0.3791276514530182
END: th(779786.9240469472)=-18.290207147598267; dx=-1.889573521832327E-7 evalInputDelta=0.3089078664779663
Fitness changed from -17.9812992811203 to -18.36042693257332
Iteration 8 complete. Error: -18.36042693257332 Total: 21.4008; Orientation: 0.6192; Line Search: 15.5124
Adding measurement 4a6118c5 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.194e+03, gradient 5.680e-04, dot -0.495; [28c36e8e-9d09-4a9f-afa3-3230b36b0e51 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -18.36042693257332, -15.652033537626266, -15.086081326007843, -14.703680723905563
LBFGS Accumulation History: 3 points
Removed measurement 4a6118c5 to history. Total: 3
th(0)=-18.36042693257332;dx=-3.223399728820002E-7
New Minimum: -18.36042693257332 > -18.632132709026337
WOLF (strong): th(1680000.0000000007)=-18.632132709026337; dx=2.0383478385684007E-8 evalInputDelta=0.2717057764530182
END: th(840000.0000000003)=-18.58875221014023; dx=-1.3250784235196738E-7 evalInputDelta=0.2283252775669098
Fitness changed from -18.36042693257332 to -18.632132709026337
Iteration 9 complete. Error: -18.632132709026337 Total: 20.1183; Orientation: 0.6247; Line Search: 14.6372
Adding measurement 1baa3ec8 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.382e+03, gradient 5.409e-04, dot -0.495; [28c36e8e-9d09-4a9f-afa3-3230b36b0e51 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -18.632132709026337, -15.652033537626266, -15.086081326007843, -14.703680723905563
LBFGS Accumulation History: 3 points
Removed measurement 1baa3ec8 to history. Total: 3
th(0)=-18.632132709026337;dx=-2.92322157773479E-7
New Minimum: -18.632132709026337 > -18.833891451358795
WOLF (strong): th(1809725.139626783)=-18.833891451358795; dx=6.967805630681221E-8 evalInputDelta=0.2017587423324585
New Minimum: -18.833891451358795 > -18.84280675649643
END: th(904862.5698133915)=-18.84280675649643; dx=-8.99699145227793E-8 evalInputDelta=0.21067404747009277
Fitness changed from -18.632132709026337 to -18.84280675649643
Iteration 10 complete. Error: -18.84280675649643 Total: 21.2323; Orientation: 0.6200; Line Search: 15.8077
<a id="p-13"></a>Iteration 10
<a id="p-12"></a>
Final threshold in iteration 11: -18.84280675649643 (> -Infinity) after 234.510s (< 5400.000s)
Returns
-18.84280675649643