Code from BasicOptimizer.scala:75 executed in 1575.68 seconds (13.241 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
IterativeTrainer.wrap(trainable)
.setOrientation(orientation())
.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = trainingMonitor.log(msg)
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable, currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable, currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
.setTimeout(trainingMinutes, TimeUnit.MINUTES)
.setMaxIterations(trainingIterations)
.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
.runAndFree
.asInstanceOf[lang.Double]
Reset training subject: 949773579876400
Reset training subject: 949799920064700
Adding measurement 4c9e8c6d to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD+Trust
th(0)=4.278837449848652;dx=-1.4697309614100686E-4
New Minimum: 4.278837449848652 > 4.278631381690502
WOLFE (weak): th(2.154434690031884)=4.278631381690502; dx=-1.3553040830030397E-4 evalInputDelta=2.0606815814971924E-4
New Minimum: 4.278631381690502 > 4.278438664972782
END: th(4.308869380063768)=4.278438664972782; dx=-1.2420544048180225E-4 evalInputDelta=3.98784875869751E-4
Fitness changed from 4.278837449848652 to 4.278438664972782
Iteration 1 complete. Error: 4.278438664972782 Total: 154.3182; Orientation: 0.0035; Line Search: 76.4948
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 3df9827f to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=4.278438664972782;dx=-1.1235143415500171E-4
New Minimum: 4.278438664972782 > 4.277716066688299
END: th(9.283177667225559)=4.277716066688299; dx=-1.0072716832991344E-4 evalInputDelta=7.225982844829559E-4
Fitness changed from 4.278438664972782 to 4.277716066688299
Iteration 2 complete. Error: 4.277716066688299 Total: 76.6664; Orientation: 0.0038; Line Search: 51.0242
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 465f3675 to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=4.277716066688299;dx=-1.0012322837048423E-4
New Minimum: 4.277716066688299 > 4.276183478534222
WOLFE (weak): th(20.000000000000004)=4.276183478534222; dx=-1.008319351872617E-4 evalInputDelta=0.00153258815407753
New Minimum: 4.276183478534222 > 4.273668520152569
WOLFE (weak): th(40.00000000000001)=4.273668520152569; dx=-1.810481844382808E-4 evalInputDelta=0.004047546535730362
New Minimum: 4.273668520152569 > 4.266615800559521
WOLFE (weak): th(120.00000000000003)=4.266615800559521; dx=-1.015012456
...skipping 9962 bytes...
on 13 complete. Error: 2.9685465544462204 Total: 101.1197; Orientation: 0.1067; Line Search: 75.7790
Adding measurement 7499d0a9 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.054e+02, gradient 4.343e-03, dot -0.875; [684c3640-6902-4b03-b960-9b25d152e387 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 3.196915715932846, 3.1422639191150665, 3.021186452358961, 2.9685465544462204
LBFGS Accumulation History: 3 points
Removed measurement 5a553b5c to history. Total: 4
th(0)=2.9685465544462204;dx=-1.8860710005443973E-5
New Minimum: 2.9685465544462204 > 2.95163394510746
WOLF (strong): th(12893.302315591061)=2.95163394510746; dx=1.764362481953382E-7 evalInputDelta=0.016912609338760376
New Minimum: 2.95163394510746 > 2.945870742201805
END: th(6446.651157795531)=2.945870742201805; dx=-7.78156739568533E-6 evalInputDelta=0.022675812244415283
Fitness changed from 2.9685465544462204 to 2.945870742201805
Iteration 14 complete. Error: 2.945870742201805 Total: 102.0214; Orientation: 0.2122; Line Search: 76.4051
Adding measurement 40dc9a44 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 7.347e+01, gradient 3.055e-03, dot -0.899; [684c3640-6902-4b03-b960-9b25d152e387 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 3.1422639191150665, 3.021186452358961, 2.9685465544462204, 2.945870742201805
LBFGS Accumulation History: 3 points
Removed measurement 8248e4b to history. Total: 4
th(0)=2.945870742201805;dx=-9.333289233873365E-6
New Minimum: 2.945870742201805 > 2.871911082416773
END: th(13888.8888888889)=2.871911082416773; dx=-5.11523173204471E-6 evalInputDelta=0.07395965978503227
Fitness changed from 2.945870742201805 to 2.871911082416773
Iteration 15 complete. Error: 2.871911082416773 Total: 76.1261; Orientation: 0.1079; Line Search: 50.8534
<a id="p-15"></a>Iteration 15
<a id="p-14"></a>
Final threshold in iteration 16: 2.871911082416773 (> -Infinity) after 1575.678s (< 1800.000s)
2.871911082416773