BasicOptimizer.scala:89 executed in 151.84 seconds (2.546 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 466330150441100
Reset training subject: 466333205417800
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
th(0)=-50.52177140273031;dx=-1.74993483912582E-5
WOLFE (weak): th(2.154434690031884)=-50.521807147058276; dx=-1.5752063968746672E-5 evalInputDelta=3.5744327966824585E-5
WOLFE (weak): th(4.308869380063768)=-50.52184564092165; dx=-1.577039044163641E-5 evalInputDelta=7.423819133833831E-5
WOLFE (weak): th(12.926608140191302)=-50.52198861787168; dx=-1.577266842043235E-5 evalInputDelta=2.172151413688539E-4
WOLFE (weak): th(51.70643256076521)=-50.522637508174995; dx=-1.577198789979507E-5 evalInputDelta=8.661054446861272E-4
END: th(258.53216280382605)=-50.5260990321591; dx=-1.574816459182869E-5 evalInputDelta=0.0043276294287935
Fitness changed from -50.539508480383006 to -50.539508480383006
Static Iteration Total: 26.2304; Orientation: 0.0660; Line Search: 17.5401
Iteration 1 failed. Error: -50.539508480383006
Previous Error: 0.0 -> -50.539508480383006
Retrying iteration 1
Reset training subject: 466356381015500
Corrupt weights measurement
LBFGS Accumulation History: 0 points
th(0)=-50.52177140273031;dx=-1.7496859574349077E-5
WOLFE (weak): th(556.9906600335335)=-50.53109156725178; dx=-1.577214382768192E-5 evalInputDelta=0.00932016452146911
New Minimum: -50.539508480383006 > -50.54041001304034
WOLFE (weak): th(1113.981320067067)=-50.54041001304034; dx=-1.576320896525408E-5 evalInputDelta=0.018638610310034665
New Minimum: -50.54041001304034 > -50.57767761396552
WOLFE (weak): th(3341.9439602012008)=-50.57767761396552; dx=-1.5763123049466414E-5 evalInputDelta=0.05590621123521089
New Minimum: -50.57767761396552 > -50.745253904353824
END: th(13367.775840804803)=-50.745253904353824; dx=-1.5724114938606596E-5 evalInputDelta=0.2234825016235149
Fitness changed from -50.539508480383006 to -50.745253904353824
Iteration 2 complete. Error: -50.745253904353824 Total: 20.1960; Orientation: 0.0664; Line Search: 14.3724
<a id="p-3"></a>Iteration 2
<a id="p-2"></a>
Adding measurement 472fad4f to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-50.745253904353824;dx=-1.577598286106256E-5
New Minimum: -50.745253904353824 > -51.225436654726714
WOLFE (weak): th(28800.0)=-51.225436654726714; dx=-1.5659004098254302E-5 evalInputDelta=0.48018275037289015
New Minimum: -51.225436654726714 > -51.703182976590035
WOLFE (weak): th(57600.0)=-51.703182976590035; dx=-1.5621598324925824E-5 evalInputDelta=0.9579290722362117
New Minimum: -51.703182976590035 > -53.58718541686321
WOLFE (weak): th(172800.0)=-53.58718541686321; dx=-1.5222894510666499E-5 evalInputDelta=2.841931512509383
New Minimum: -53.58718541686321 > -61.364566271743385
END: th(691200.0)=-61.364566271743385; dx=-1.2988193568030916E-5 evalInputDelta=10.619312367389561
Fitness changed from -50.745253904353824 to -61.364566271743385
Iteration 3 complete. Error: -61.364566271743385 Total: 17.8945; Orientation: 0.1097; Line Search: 14.4094
<a id="p-5"></a>Iteration 3
<a id="p-4"></a>
Adding measurement 46427a1d to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-61.364566271743385;dx=-1.3228120912111797E-5
New Minimum: -61.364566271743385 > -77.99062861911047
END: th(1489145.257750038)=-77.99062861911047; dx=-8.006460606852926E-6 evalInputDelta=16.626062347367082
Fitness changed from -61.364566271743385 to -77.99062861911047
Iteration 4 complete. Error: -77.99062861911047 Total: 9.6333; Orientation: 0.1155; Line Search: 6.0564
<a id="p-7"></a>Iteration 4
<a id="p-6"></a>
Adding measurement 1cda50b6 to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-77.99062861911047;dx=-9.265300113214235E-6
New Minimum: -77.99062861911047 > -98.07131650927342
END: th(3208266.201793153)=-98.07131650927342; dx=-3.482261834377244E-6 evalInputDelta=20.08068789016295
Fitness changed from -77.99062861911047 to -98.07131650927342
Iteration 5 complete. Error: -98.07131650927342 Total: 9.4023; Orientation: 0.1099; Line Search: 6.0072
<a id="p-9"></a>Iteration 5
<a id="p-8"></a>
Adding measurement 2eee5d2a to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.045e+04, gradient 2.768e-03, dot -0.529; [70148f09-8cff-407a-99c2-727dde652552 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -98.07131650927342, -77.99062861911047, -61.364566271743385, -50.745253904353824
LBFGS Accumulation History: 3 points
Removed measurement 2eee5d2a to history. Total: 3
th(0)=-98.07131650927342;dx=-7.64859716339814E-6
New Minimum: -98.07131650927342 > -114.21701815418874
END: th(6912000.0)=-114.21701815418874; dx=-2.284936062829534E-8 evalInputDelta=16.14570164491532
Fitness changed from -98.07131650927342 to -114.21701815418874
Iteration 6 complete. Error: -114.21701815418874 Total: 12.1882; Orientation: 2.1320; Line Search: 6.5412
Adding measurement 70117511 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.251e+04, gradient 3.141e-03, dot -0.332; [70148f09-8cff-407a-99c2-727dde652552 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -114.21701815418874, -77.99062861911047, -61.364566271743385, -50.745253904353824
LBFGS Accumulation History: 3 points
Removed measurement 70117511 to history. Total: 3
th(0)=-114.21701815418874;dx=-9.846701200187643E-6
Armijo: th(1.489145257750038E7)=-107.34959519350097; dx=2.3046828526258805E-6 evalInputDelta=-6.867422960687762
New Minimum: -114.21701815418874 > -129.3479729092373
WOLF (strong): th(7445726.28875019)=-129.3479729092373; dx=2.2125494318354682E-6 evalInputDelta=15.130954755048577
New Minimum: -129.3479729092373 > -131.58485644077973
END: th(2481908.7629167303)=-131.58485644077973; dx=-3.5109369081998457E-6 evalInputDelta=17.367838286590995
Fitness changed from -114.21701815418874 to -131.58485644077973
Iteration 7 complete. Error: -131.58485644077973 Total: 17.2795; Orientation: 2.1258; Line Search: 12.1107
Adding measurement c205dfe to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.525e+04, gradient 2.113e-03, dot -0.483; [70148f09-8cff-407a-99c2-727dde652552 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -131.58485644077973, -77.99062861911047, -61.364566271743385, -50.745253904353824
LBFGS Accumulation History: 3 points
Removed measurement c205dfe to history. Total: 3
th(0)=-131.58485644077973;dx=-4.448266527409947E-6
New Minimum: -131.58485644077973 > -145.0603337605617
END: th(5347110.336321922)=-145.0603337605617; dx=-1.0313568696115643E-6 evalInputDelta=13.475477319781959
Fitness changed from -131.58485644077973 to -145.0603337605617
Iteration 8 complete. Error: -145.0603337605617 Total: 10.8840; Orientation: 1.8837; Line Search: 6.3159
Adding measurement 3111ee87 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.276e+04, gradient 2.179e-03, dot -0.447; [70148f09-8cff-407a-99c2-727dde652552 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -145.0603337605617, -77.99062861911047, -61.364566271743385, -50.745253904353824
LBFGS Accumulation History: 3 points
Removed measurement 3111ee87 to history. Total: 3
th(0)=-145.0603337605617;dx=-4.749837427814711E-6
New Minimum: -145.0603337605617 > -148.65588346675284
WOLF (strong): th(1.1520000000000002E7)=-148.65588346675284; dx=1.1314485328775902E-6 evalInputDelta=3.59554970619115
New Minimum: -148.65588346675284 > -154.67239269235642
WOLF (strong): th(5760000.000000001)=-154.67239269235642; dx=2.2870209360081159E-7 evalInputDelta=9.612058931794735
END: th(1920000.0000000002)=-152.04572756260478; dx=-2.334875709229147E-6 evalInputDelta=6.985393802043092
Fitness changed from -145.0603337605617 to -154.67239269235642
Iteration 9 complete. Error: -154.67239269235642 Total: 16.8110; Orientation: 1.9346; Line Search: 12.0518
Adding measurement acb933b to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.518e+04, gradient 2.354e-03, dot -0.386; [70148f09-8cff-407a-99c2-727dde652552 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -154.67239269235642, -77.99062861911047, -61.364566271743385, -50.745253904353824
LBFGS Accumulation History: 3 points
Removed measurement acb933b to history. Total: 3
th(0)=-154.67239269235642;dx=-5.51348464685153E-6
New Minimum: -154.67239269235642 > -166.35052285839947
END: th(4136514.6048612176)=-166.35052285839947; dx=-1.7196206188698122E-7 evalInputDelta=11.678130166043047
Fitness changed from -154.67239269235642 to -166.35052285839947
Iteration 10 complete. Error: -166.35052285839947 Total: 10.7064; Orientation: 1.7758; Line Search: 6.0850
<a id="p-11"></a>Iteration 10
<a id="p-10"></a>
Final threshold in iteration 11: -166.35052285839947 (> -Infinity) after 151.843s (< 5400.000s)
Returns
-166.35052285839947