BasicOptimizer.scala:89 executed in 298.80 seconds (3.573 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 576362736805300
Reset training subject: 576368874382000
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
New Minimum: -38.18769547202249 > -38.21844168792332
th(0)=-38.21844168792332;dx=-1.9897573345428207E-6
New Minimum: -38.21844168792332 > -38.21844366412237
END: th(2.154434690031884)=-38.21844366412237; dx=-1.734750089626402E-6 evalInputDelta=1.9761990515121397E-6
Fitness changed from -38.18769547202249 to -38.21844366412237
Iteration 1 complete. Error: -38.21844366412237 Total: 30.4450; Orientation: 0.0982; Line Search: 12.2195
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 42940e0d to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-38.21844366412237;dx=-1.7445828991165853E-6
New Minimum: -38.21844366412237 > -38.21845552131453
WOLFE (weak): th(4.641588833612779)=-38.21845552131453; dx=-1.742330024469236E-6 evalInputDelta=1.1857192163233776E-5
New Minimum: -38.21845552131453 > -38.21846540230519
WOLFE (weak): th(9.283177667225559)=-38.21846540230519; dx=-1.7419698370558057E-6 evalInputDelta=2.1738182823582974E-5
New Minimum: -38.21846540230519 > -38.21850097385041
WOLFE (weak): th(27.849533001676676)=-38.21850097385041; dx=-1.7391694191465036E-6 evalInputDelta=5.73097280422985E-5
New Minimum: -38.21850097385041 > -38.2186610453942
WOLFE (weak): th(111.3981320067067)=-38.2186610453942; dx=-1.738959185928173E-6 evalInputDelta=2.1738127183112965E-4
New Minimum: -38.2186610453942 > -38.219518701258906
WOLFE (weak): th(556.9906600335335)=-38.219518701258906; dx=-1.7388022204353836E-6 evalInputDelta=0.00107503713653756
New Minimum: -38.219518701258906 > -38.22486972303237
WOLFE (weak): th(3341.9439602012008)=-38.22486972303237; dx=-1.7390466532745627E-6 evalInputDelta=0.006426058910001586
New Minimum: -38.22486972303237 > -38.26338564820032
WOLFE (weak): th(23393.607721408407)=-38.26338564820032; dx=-1.7366425641908681E-6 evalInputDelta=0.04494198407795125
New Minimum: -38.26338564820032 > -38.57605932058477
WOLFE (weak): th(187148.86177126726)=-38.57605932058477; dx=-1.7167335137775118E-6 evalInputDelta=0.3576156564624
New Minimum: -38.57605932058477 > -41.23899422619756
END: th(1684339.7559414054)=-41.23899422619756; dx=-1.4954505703002806E-6 evalInputDelta=3.0205505620751936
Fitness changed from -38.21844366412237 to -41.23899422619756
Iteration 2 complete. Error: -41.23899422619756 Total: 68.6804; Orientation: 0.1482; Line Search: 61.2235
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 7ce5bc32 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-41.23899422619756;dx=-1.4585009088565659E-6
New Minimum: -41.23899422619756 > -46.04264317110802
END: th(3628800.0000000005)=-46.04264317110802; dx=-9.971053619482352E-7 evalInputDelta=4.8036489449104565
Fitness changed from -41.23899422619756 to -46.04264317110802
Iteration 3 complete. Error: -46.04264317110802 Total: 19.3022; Orientation: 0.1485; Line Search: 12.0052
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 628e45d3 to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-46.04264317110802;dx=-9.774377420900875E-7
New Minimum: -46.04264317110802 > -52.591032147722245
END: th(7818012.603187701)=-52.591032147722245; dx=-5.788335153943476E-7 evalInputDelta=6.548388976614227
Fitness changed from -46.04264317110802 to -52.591032147722245
Iteration 4 complete. Error: -52.591032147722245 Total: 19.7774; Orientation: 0.1313; Line Search: 12.3719
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 5985d19e to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.856e+04, gradient 8.757e-04, dot -0.730; [e5bef3c9-cce7-4b25-b742-8939a4d2183b = 1.000/1.000e+00]
Orientation rejected. Popping history element from -52.591032147722245, -46.04264317110802, -41.23899422619756, -38.21844366412237
LBFGS Accumulation History: 3 points
Removed measurement 5985d19e to history. Total: 3
th(0)=-52.591032147722245;dx=-7.665147604601413E-7
New Minimum: -52.591032147722245 > -60.25993826465745
END: th(1.6843397559414055E7)=-60.25993826465745; dx=-2.28700437506898E-7 evalInputDelta=7.668906116935204
Fitness changed from -52.591032147722245 to -60.25993826465745
Iteration 5 complete. Error: -60.25993826465745 Total: 22.4999; Orientation: 2.6029; Line Search: 12.6020
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 2936a4c2 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.920e+04, gradient 9.904e-04, dot -0.487; [e5bef3c9-cce7-4b25-b742-8939a4d2183b = 1.000/1.000e+00]
Orientation rejected. Popping history element from -60.25993826465745, -46.04264317110802, -41.23899422619756, -38.21844366412237
LBFGS Accumulation History: 3 points
Removed measurement 2936a4c2 to history. Total: 3
th(0)=-60.25993826465745;dx=-9.797511695958423E-7
New Minimum: -60.25993826465745 > -62.070148527096876
WOLF (strong): th(3.628800000000001E7)=-62.070148527096876; dx=1.238214751014341E-7 evalInputDelta=1.8102102624394263
New Minimum: -62.070148527096876 > -65.33761398335825
WOLF (strong): th(1.8144000000000004E7)=-65.33761398335825; dx=1.1536253408906237E-7 evalInputDelta=5.077675718700803
END: th(6048000.000000001)=-64.93833907436883; dx=-4.101436195041629E-7 evalInputDelta=4.678400809711377
Fitness changed from -60.25993826465745 to -65.33761398335825
Iteration 6 complete. Error: -65.33761398335825 Total: 34.0831; Orientation: 2.5190; Line Search: 24.1701
Adding measurement 7ab07815 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.334e+04, gradient 1.081e-03, dot -0.367; [e5bef3c9-cce7-4b25-b742-8939a4d2183b = 1.000/1.000e+00]
Orientation rejected. Popping history element from -65.33761398335825, -46.04264317110802, -41.23899422619756, -38.21844366412237
LBFGS Accumulation History: 3 points
Removed measurement 7ab07815 to history. Total: 3
th(0)=-65.33761398335825;dx=-1.1674432390141123E-6
New Minimum: -65.33761398335825 > -73.79599494393416
WOLF (strong): th(1.3030021005312836E7)=-73.79599494393416; dx=2.2055407915572857E-8 evalInputDelta=8.458380960575909
END: th(6515010.502656418)=-71.93311138356542; dx=-6.073504080720573E-7 evalInputDelta=6.595497400207165
Fitness changed from -65.33761398335825 to -73.79599494393416
Iteration 7 complete. Error: -73.79599494393416 Total: 27.5514; Orientation: 2.5281; Line Search: 18.9227
Adding measurement 1bfb40b5 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 5.279e+04, gradient 8.221e-04, dot -0.506; [e5bef3c9-cce7-4b25-b742-8939a4d2183b = 1.000/1.000e+00]
Orientation rejected. Popping history element from -73.79599494393416, -46.04264317110802, -41.23899422619756, -38.21844366412237
LBFGS Accumulation History: 3 points
Removed measurement 1bfb40b5 to history. Total: 3
th(0)=-73.79599494393416;dx=-6.754355969573974E-7
New Minimum: -73.79599494393416 > -78.58350068017423
END: th(1.4036164632845048E7)=-78.58350068017423; dx=-1.6590403504173304E-8 evalInputDelta=4.787505736240064
Fitness changed from -73.79599494393416 to -78.58350068017423
Iteration 8 complete. Error: -78.58350068017423 Total: 21.0824; Orientation: 2.5235; Line Search: 12.2124
Adding measurement 15b103be to history. Total: 3
Rejected: LBFGS Orientation magnitude: 6.080e+04, gradient 8.060e-04, dot -0.494; [e5bef3c9-cce7-4b25-b742-8939a4d2183b = 1.000/1.000e+00]
Orientation rejected. Popping history element from -78.58350068017423, -46.04264317110802, -41.23899422619756, -38.21844366412237
LBFGS Accumulation History: 3 points
Removed measurement 15b103be to history. Total: 3
th(0)=-78.58350068017423;dx=-6.494508788759986E-7
New Minimum: -78.58350068017423 > -79.93911148540994
WOLF (strong): th(3.024000000000001E7)=-79.93911148540994; dx=1.8951554533569708E-7 evalInputDelta=1.3556108052357132
New Minimum: -79.93911148540994 > -82.81439218748216
WOLF (strong): th(1.5120000000000006E7)=-82.81439218748216; dx=5.0203613455922634E-8 evalInputDelta=4.2308915073079305
END: th(5040000.000000002)=-81.4786979626457; dx=-3.863713796911313E-7 evalInputDelta=2.8951972824714716
Fitness changed from -78.58350068017423 to -82.81439218748216
Iteration 9 complete. Error: -82.81439218748216 Total: 33.8008; Orientation: 2.4887; Line Search: 25.1834
Adding measurement 45117123 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 6.741e+04, gradient 8.181e-04, dot -0.461; [e5bef3c9-cce7-4b25-b742-8939a4d2183b = 1.000/1.000e+00]
Orientation rejected. Popping history element from -82.81439218748216, -46.04264317110802, -41.23899422619756, -38.21844366412237
LBFGS Accumulation History: 3 points
Removed measurement 45117123 to history. Total: 3
th(0)=-82.81439218748216;dx=-6.691342529017188E-7
New Minimum: -82.81439218748216 > -87.29997153110136
END: th(1.0858350837760698E7)=-87.29997153110136; dx=-7.786349732674737E-8 evalInputDelta=4.485579343619207
Fitness changed from -82.81439218748216 to -87.29997153110136
Iteration 10 complete. Error: -87.29997153110136 Total: 20.5044; Orientation: 2.4820; Line Search: 11.9748
<a id="p-13"></a>Iteration 10
<a id="p-12"></a>
Final threshold in iteration 11: -87.29997153110136 (> -Infinity) after 298.802s (< 5400.000s)
Returns
-87.29997153110136