BasicOptimizer.scala:89 executed in 227.07 seconds (2.091 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 2774435563019
Reset training subject: 2779513686711
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
th(0)=-15.815192878246307;dx=-2.2360682918593928E-6
WOLFE (weak): th(2.154434690031884)=-15.815198302268982; dx=-2.1536821110642075E-6 evalInputDelta=5.424022674560547E-6
WOLFE (weak): th(4.308869380063768)=-15.815204560756683; dx=-2.1536428247558985E-6 evalInputDelta=1.1682510375976562E-5
WOLFE (weak): th(12.926608140191302)=-15.815225422382355; dx=-2.1532160500393794E-6 evalInputDelta=3.254413604736328E-5
WOLFE (weak): th(51.70643256076521)=-15.815326035022736; dx=-2.1538436971559905E-6 evalInputDelta=1.3315677642822266E-4
WOLFE (weak): th(258.53216280382605)=-15.815857350826263; dx=-2.1527157296244466E-6 evalInputDelta=6.644725799560547E-4
WOLFE (weak): th(1551.1929768229563)=-15.819180250167847; dx=-2.15147932845263E-6 evalInputDelta=0.003987371921539307
WOLFE (weak): th(10858.350837760694)=-15.842894494533539; dx=-2.1156077427731412E-6 evalInputDelta=0.027701616287231445
New Minimum: -15.940335512161255 > -16.021511256694794
END: th(86866.80670208555)=-16.021511256694794; dx=-1.807771110784581E-6 evalInputDelta=0.20631837844848633
Fitness changed from -15.940335512161255 to -16.021511256694794
Iteration 1 complete. Error: -16.021511256694794 Total: 60.8092; Orientation: 0.0208; Line Search: 45.7022
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 4572cdcc to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-16.021511256694794;dx=-1.7037149038550406E-6
New Minimum: -16.021511256694794 > -16.35802537202835
END: th(187148.86177126726)=-16.35802537202835; dx=-1.3044495255608537E-6 evalInputDelta=0.33651411533355713
Fitness changed from -16.021511256694794 to -16.35802537202835
Iteration 2 complete. Error: -16.35802537202835 Total: 15.1401; Orientation: 0.0293; Line Search: 9.8888
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement c82223b to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-16.35802537202835;dx=-1.1751549746127751E-6
New Minimum: -16.35802537202835 > -16.829247534275055
END: th(403200.00000000006)=-16.829247534275055; dx=-7.97941227270354E-7 evalInputDelta=0.4712221622467041
Fitness changed from -16.35802537202835 to -16.829247534275055
Iteration 3 complete. Error: -16.829247534275055 Total: 15.5174; Orientation: 0.0299; Line Search: 10.3679
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 72ba823a to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-16.829247534275055;dx=-7.559792041674302E-7
New Minimum: -16.829247534275055 > -17.411312699317932
END: th(868668.0670208557)=-17.411312699317932; dx=-4.149321975016865E-7 evalInputDelta=0.5820651650428772
Fitness changed from -16.829247534275055 to -17.411312699317932
Iteration 4 complete. Error: -17.411312699317932 Total: 15.6201; Orientation: 0.0302; Line Search: 10.1316
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 4b724d24 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.810e+03, gradient 7.300e-04, dot -0.778; [17f792a9-3d47-4093-aaed-10da40985daa = 1.000/1.000e+00]
Orientation rejected. Popping history element from -17.411312699317932, -16.829247534275055, -16.35802537202835, -16.021511256694794
LBFGS Accumulation History: 3 points
Removed measurement 4b724d24 to history. Total: 3
th(0)=-17.411312699317932;dx=-5.320020075323462E-7
New Minimum: -17.411312699317932 > -18.04156368970871
END: th(1871488.617712673)=-18.04156368970871; dx=-1.719291831085698E-7 evalInputDelta=0.6302509903907776
Fitness changed from -17.411312699317932 to -18.04156368970871
Iteration 5 complete. Error: -18.04156368970871 Total: 16.2523; Orientation: 0.6030; Line Search: 9.9838
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 63003eb0 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.893e+03, gradient 7.228e-04, dot -0.546; [17f792a9-3d47-4093-aaed-10da40985daa = 1.000/1.000e+00]
Orientation rejected. Popping history element from -18.04156368970871, -16.829247534275055, -16.35802537202835, -16.021511256694794
LBFGS Accumulation History: 3 points
Removed measurement 63003eb0 to history. Total: 3
th(0)=-18.04156368970871;dx=-5.219796561468906E-7
New Minimum: -18.04156368970871 > -18.460202157497406
WOLF (strong): th(4032000.0000000014)=-18.460202157497406; dx=4.974712614056577E-9 evalInputDelta=0.4186384677886963
END: th(2016000.0000000007)=-18.44433093070984; dx=-3.905920045336634E-8 evalInputDelta=0.40276724100112915
Fitness changed from -18.04156368970871 to -18.460202157497406
Iteration 6 complete. Error: -18.460202157497406 Total: 21.2719; Orientation: 0.5997; Line Search: 15.5600
Adding measurement 4bf5ae7e to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.353e+03, gradient 7.705e-04, dot -0.334; [17f792a9-3d47-4093-aaed-10da40985daa = 1.000/1.000e+00]
Orientation rejected. Popping history element from -18.460202157497406, -16.829247534275055, -16.35802537202835, -16.021511256694794
LBFGS Accumulation History: 3 points
Removed measurement 4bf5ae7e to history. Total: 3
th(0)=-18.460202157497406;dx=-5.932276209201045E-7
New Minimum: -18.460202157497406 > -18.595570027828217
WOLF (strong): th(4343340.335104279)=-18.595570027828217; dx=2.162625917036825E-7 evalInputDelta=0.13536787033081055
New Minimum: -18.595570027828217 > -19.00212436914444
WOLF (strong): th(2171670.1675521396)=-19.00212436914444; dx=2.5731449734520874E-8 evalInputDelta=0.5419222116470337
END: th(723890.0558507132)=-18.83219975233078; dx=-2.8262569876064056E-7 evalInputDelta=0.371997594833374
Fitness changed from -18.460202157497406 to -19.00212436914444
Iteration 7 complete. Error: -19.00212436914444 Total: 25.3011; Orientation: 0.6016; Line Search: 19.8732
Adding measurement 4b6ae42f to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.648e+03, gradient 5.596e-04, dot -0.356; [17f792a9-3d47-4093-aaed-10da40985daa = 1.000/1.000e+00]
Orientation rejected. Popping history element from -19.00212436914444, -16.829247534275055, -16.35802537202835, -16.021511256694794
LBFGS Accumulation History: 3 points
Removed measurement 4b6ae42f to history. Total: 3
th(0)=-19.00212436914444;dx=-3.1289351938390683E-7
New Minimum: -19.00212436914444 > -19.357572376728058
END: th(1559573.8480938943)=-19.357572376728058; dx=-9.316276283581223E-8 evalInputDelta=0.35544800758361816
Fitness changed from -19.00212436914444 to -19.357572376728058
Iteration 8 complete. Error: -19.357572376728058 Total: 15.8239; Orientation: 0.6076; Line Search: 10.0527
Adding measurement ec95ecd to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.246e+03, gradient 4.033e-04, dot -0.492; [17f792a9-3d47-4093-aaed-10da40985daa = 1.000/1.000e+00]
Orientation rejected. Popping history element from -19.357572376728058, -16.829247534275055, -16.35802537202835, -16.021511256694794
LBFGS Accumulation History: 3 points
Removed measurement ec95ecd to history. Total: 3
th(0)=-19.357572376728058;dx=-1.6238754494880853E-7
New Minimum: -19.357572376728058 > -19.67075628042221
END: th(3360000.0000000014)=-19.67075628042221; dx=-2.468721719374422E-8 evalInputDelta=0.31318390369415283
Fitness changed from -19.357572376728058 to -19.67075628042221
Iteration 9 complete. Error: -19.67075628042221 Total: 15.7552; Orientation: 0.5942; Line Search: 10.3866
Adding measurement 2f76cad3 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.408e+03, gradient 4.971e-04, dot -0.381; [17f792a9-3d47-4093-aaed-10da40985daa = 1.000/1.000e+00]
Orientation rejected. Popping history element from -19.67075628042221, -16.829247534275055, -16.35802537202835, -16.021511256694794
LBFGS Accumulation History: 3 points
Removed measurement 2f76cad3 to history. Total: 3
th(0)=-19.67075628042221;dx=-2.468345273882581E-7
Armijo: th(7238900.558507132)=-19.185449719429016; dx=1.218269592526206E-7 evalInputDelta=-0.4853065609931946
New Minimum: -19.67075628042221 > -19.711903989315033
WOLF (strong): th(3619450.279253566)=-19.711903989315033; dx=9.870741563648133E-8 evalInputDelta=0.041147708892822266
New Minimum: -19.711903989315033 > -19.8723424077034
END: th(1206483.4264178553)=-19.8723424077034; dx=-4.468727171285283E-8 evalInputDelta=0.20158612728118896
Fitness changed from -19.67075628042221 to -19.8723424077034
Iteration 10 complete. Error: -19.8723424077034 Total: 25.3122; Orientation: 0.5843; Line Search: 19.8375
<a id="p-13"></a>Iteration 10
<a id="p-12"></a>
Final threshold in iteration 11: -19.8723424077034 (> -Infinity) after 227.068s (< 5400.000s)
Returns
-19.8723424077034