BasicOptimizer.scala:89 executed in 2231.53 seconds (11.390 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 26138364646006
Reset training subject: 26186703856698
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
th(0)=4.379307329654694;dx=-9.55568737001139E-7
New Minimum: 4.379307329654694 > 4.3793045580387115
WOLFE (weak): th(2.154434690031884)=4.3793045580387115; dx=-9.508944622972109E-7 evalInputDelta=2.771615982055664E-6
New Minimum: 4.3793045580387115 > 4.379301935434341
WOLFE (weak): th(4.308869380063768)=4.379301935434341; dx=-8.989937373365003E-7 evalInputDelta=5.3942203521728516E-6
New Minimum: 4.379301935434341 > 4.379290670156479
WOLFE (weak): th(12.926608140191302)=4.379290670156479; dx=-1.1088823246275954E-6 evalInputDelta=1.665949821472168E-5
New Minimum: 4.379290670156479 > 4.379241526126862
WOLFE (weak): th(51.70643256076521)=4.379241526126862; dx=-1.113233760129914E-6 evalInputDelta=6.580352783203125E-5
New Minimum: 4.379241526126862 > 4.378979802131653
WOLFE (weak): th(258.53216280382605)=4.378979802131653; dx=-1.111445287800905E-6 evalInputDelta=3.275275230407715E-4
New Minimum: 4.378979802131653 > 4.377344131469727
WOLFE (weak): th(1551.1929768229563)=4.377344131469727; dx=-1.1110752114296787E-6 evalInputDelta=0.001963198184967041
New Minimum: 4.377344131469727 > 4.365583062171936
WOLFE (weak): th(10858.350837760694)=4.365583062171936; dx=-1.106017903169568E-6 evalInputDelta=0.013724267482757568
New Minimum: 4.365583062171936 > 4.270468205213547
WOLFE (weak): th(86866.80670208555)=4.270468205213547; dx=-1.0850121894992195E-6 evalInputDelta=0.10883912444114685
New Minimum: 4.270468205213547 > 3.5222449898719788
END: th(781801.26031877)=3.5222449898719788; dx=-7.929028582982948E-7 evalInputDelta=0.8570623397827148
Fitness changed from 4.379307329654694 to 3.5222449898719788
Iteration 1 complete. Error: 3.5222449898719788 Total: 638.6235; Orientation: 0.0721; Line Search: 492.1708
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 4a1b0f59 to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=3.5222449898719788;dx=-7.480004510304151E-7
New Minimum: 3.5222449898719788 > 2.3929292261600494
END: th(1684339.7559414052)=2.3929292261600494; dx=-4.4496272644843165E-7 evalInputDelta=1.1293157637119293
Fitness changed from 3.5222449898719788 to 2.3929292261600494
Iteration 2 complete. Error: 2.3929292261600494 Total: 147.7084; Orientation: 0.0960; Line Search: 97.3545
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 3b0c205b to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=2.3929292261600494;dx=-3.9421893888228525E-7
New Minimum: 2.3929292261600494 > 1.0689680576324463
END: th(3628800.0)=1.0689680576324463; dx=-2.1692348693173233E-7 evalInputDelta=1.3239611685276031
Fitness changed from 2.3929292261600494 to 1.0689680576324463
Iteration 3 complete. Error: 1.0689680576324463 Total: 148.2017; Orientation: 0.0964; Line Search: 97.8486
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 613710b8 to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=1.0689680576324463;dx=-2.550377293574052E-7
New Minimum: 1.0689680576324463 > -0.39335450530052185
END: th(7818012.6031877)=-0.39335450530052185; dx=-1.0602103332270482E-7 evalInputDelta=1.4623225629329681
Fitness changed from 1.0689680576324463 to -0.39335450530052185
Iteration 4 complete. Error: -0.39335450530052185 Total: 148.2798; Orientation: 0.0917; Line Search: 98.2789
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 559d007c to history. Total: 3
Rejected: LBFGS Orientation magnitude: 6.753e+03, gradient 4.646e-04, dot -0.693; [dd2c8a39-808c-4846-9331-f065cee0ecf0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -0.39335450530052185, 1.0689680576324463, 2.3929292261600494, 3.5222449898719788
LBFGS Accumulation History: 3 points
Removed measurement 559d007c to history. Total: 3
th(0)=-0.39335450530052185;dx=-2.0783019900946928E-7
New Minimum: -0.39335450530052185 > -1.9797546863555908
END: th(1.684339755941405E7)=-1.9797546863555908; dx=-3.9151400583771527E-8 evalInputDelta=1.586400181055069
Fitness changed from -0.39335450530052185 to -1.9797546863555908
Iteration 5 complete. Error: -1.9797546863555908 Total: 150.3002; Orientation: 2.3843; Line Search: 98.2602
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 6cdd75b1 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 6.395e+03, gradient 4.123e-04, dot -0.493; [dd2c8a39-808c-4846-9331-f065cee0ecf0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1.9797546863555908, 1.0689680576324463, 2.3929292261600494, 3.5222449898719788
LBFGS Accumulation History: 3 points
Removed measurement 6cdd75b1 to history. Total: 3
th(0)=-1.9797546863555908;dx=-1.6126860070346877E-7
New Minimum: -1.9797546863555908 > -3.2848227620124817
WOLF (strong): th(3.6288E7)=-3.2848227620124817; dx=4.429007561093094E-9 evalInputDelta=1.3050680756568909
END: th(1.8144E7)=-3.267901599407196; dx=-1.418217975486534E-8 evalInputDelta=1.2881469130516052
Fitness changed from -1.9797546863555908 to -3.2848227620124817
Iteration 6 complete. Error: -3.2848227620124817 Total: 199.7736; Orientation: 2.2214; Line Search: 146.5781
Adding measurement 2044eaa6 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.797e+03, gradient 3.988e-04, dot -0.320; [dd2c8a39-808c-4846-9331-f065cee0ecf0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -3.2848227620124817, 1.0689680576324463, 2.3929292261600494, 3.5222449898719788
LBFGS Accumulation History: 3 points
Removed measurement 2044eaa6 to history. Total: 3
th(0)=-3.2848227620124817;dx=-1.5888169678057767E-7
New Minimum: -3.2848227620124817 > -4.324150264263153
WOLF (strong): th(3.90900630159385E7)=-4.324150264263153; dx=1.5294938586035225E-8 evalInputDelta=1.0393275022506714
New Minimum: -4.324150264263153 > -4.480839312076569
END: th(1.954503150796925E7)=-4.480839312076569; dx=-4.802051196657392E-9 evalInputDelta=1.196016550064087
Fitness changed from -3.2848227620124817 to -4.480839312076569
Iteration 7 complete. Error: -4.480839312076569 Total: 198.8958; Orientation: 2.2581; Line Search: 147.6119
Adding measurement 14506e65 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.878e+03, gradient 2.136e-04, dot -0.380; [dd2c8a39-808c-4846-9331-f065cee0ecf0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -4.480839312076569, 1.0689680576324463, 2.3929292261600494, 3.5222449898719788
LBFGS Accumulation History: 3 points
Removed measurement 14506e65 to history. Total: 3
th(0)=-4.480839312076569;dx=-4.5336636960070845E-8
New Minimum: -4.480839312076569 > -5.206379383802414
WOLF (strong): th(4.210849389853513E7)=-5.206379383802414; dx=1.8192160502560856E-9 evalInputDelta=0.7255400717258453
END: th(2.1054246949267566E7)=-5.1110661923885345; dx=-1.1702719268036322E-8 evalInputDelta=0.6302268803119659
Fitness changed from -4.480839312076569 to -5.206379383802414
Iteration 8 complete. Error: -5.206379383802414 Total: 199.2391; Orientation: 2.1919; Line Search: 148.0204
Adding measurement 1154bf7d to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.237e+03, gradient 2.033e-04, dot -0.322; [dd2c8a39-808c-4846-9331-f065cee0ecf0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -5.206379383802414, 1.0689680576324463, 2.3929292261600494, 3.5222449898719788
LBFGS Accumulation History: 3 points
Removed measurement 1154bf7d to history. Total: 3
th(0)=-5.206379383802414;dx=-4.129840220140835E-8
New Minimum: -5.206379383802414 > -5.5607750415802
WOLF (strong): th(4.536000000000001E7)=-5.5607750415802; dx=5.0045557769188814E-9 evalInputDelta=0.35439565777778625
New Minimum: -5.5607750415802 > -5.605690956115723
END: th(2.2680000000000004E7)=-5.605690956115723; dx=-2.5528914261053E-9 evalInputDelta=0.3993115723133087
Fitness changed from -5.206379383802414 to -5.605690956115723
Iteration 9 complete. Error: -5.605690956115723 Total: 199.9372; Orientation: 2.1888; Line Search: 148.7431
Adding measurement 27809bf3 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.858e+03, gradient 1.262e-04, dot -0.398; [dd2c8a39-808c-4846-9331-f065cee0ecf0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -5.605690956115723, 1.0689680576324463, 2.3929292261600494, 3.5222449898719788
LBFGS Accumulation History: 3 points
Removed measurement 27809bf3 to history. Total: 3
th(0)=-5.605690956115723;dx=-1.5921172696548633E-8
New Minimum: -5.605690956115723 > -5.878508806228638
WOLF (strong): th(4.8862578769923136E7)=-5.878508806228638; dx=1.0299962885218185E-9 evalInputDelta=0.27281785011291504
END: th(2.4431289384961568E7)=-5.859376758337021; dx=-3.557776013026574E-9 evalInputDelta=0.2536858022212982
Fitness changed from -5.605690956115723 to -5.878508806228638
Iteration 10 complete. Error: -5.878508806228638 Total: 199.4035; Orientation: 2.2576; Line Search: 148.2582
<a id="p-13"></a>Iteration 10
<a id="p-12"></a>
Final threshold in iteration 11: -5.878508806228638 (> -Infinity) after 2231.524s (< 5400.000s)
Returns
-5.878508806228638