Code from BasicOptimizer.scala:75 executed in 2597.78 seconds (23.742 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
IterativeTrainer.wrap(trainable)
.setOrientation(orientation())
.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = trainingMonitor.log(msg)
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable, currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable, currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
.setTimeout(trainingMinutes, TimeUnit.MINUTES)
.setMaxIterations(trainingIterations)
.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
.runAndFree
.asInstanceOf[lang.Double]
Reset training subject: 1424125511000200
Reset training subject: 1424157871497100
Adding measurement 3516b881 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD+Trust
th(0)=-27.439033311791718;dx=-2.225847769574606E-5
New Minimum: -27.439033311791718 > -27.43904081049065
WOLFE (weak): th(2.154434690031884)=-27.43904081049065; dx=-2.225890304977288E-5 evalInputDelta=7.498698931129866E-6
New Minimum: -27.43904081049065 > -27.439048886299133
WOLFE (weak): th(4.308869380063768)=-27.439048886299133; dx=-2.226126227625555E-5 evalInputDelta=1.5574507415294647E-5
New Minimum: -27.439048886299133 > -27.43907999309401
WOLFE (weak): th(12.926608140191302)=-27.43907999309401; dx=-2.227179836002649E-5 evalInputDelta=4.668130229035228E-5
New Minimum: -27.43907999309401 > -27.43922058865428
WOLFE (weak): th(51.70643256076521)=-27.43922058865428; dx=-2.2328528284171854E-5 evalInputDelta=1.8727686256170273E-4
New Minimum: -27.43922058865428 > -27.43997365919252
WOLFE (weak): th(258.53216280382605)=-27.43997365919252; dx=-2.2605249225352972E-5 evalInputDelta=9.403474008031765E-4
New Minimum: -27.43997365919252 > -27.4449749669681
WOLFE (weak): th(1551.1929768229563)=-27.4449749669681; dx=-2.333090876616005E-5 evalInputDelta=0.005941655176382454
New Minimum: -27.4449749669681 > -27.477402864955366
END: th(10858.350837760694)=-27.477402864955366; dx=-1.7922922874886465E-5 evalInputDelta=0.03836955316364765
Fitness changed from -27.439033311791718 to -27.477402864955366
Iteration 1 complete. Error: -27.477402864955366 Total: 345.2934; Orientation: 0.0268; Line Search: 250.2982
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 7d5ef1c3 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-27.477402864955366;dx=-3.037197258723118E-5
New Minimum: -27.477402864955366 > -27.53890236125638
END: th(23393.607721408407)=-27.53890236125638; dx=-1.5027770193304553E-5 evalInputDelta=0.061
...skipping 15305 bytes...
972 Total: 185.7757; Orientation: 0.8008; Line Search: 154.1836
Adding measurement 7535f28 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 8.406e+02, gradient 1.227e-03, dot -0.548; [e2e5fea1-cfd0-47d3-b92b-b6bc472ad2b3 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -29.866930960367124, -29.92106167723735, -30.02203716461857, -30.025577471901972
LBFGS Accumulation History: 3 points
Removed measurement 1df9f7c6 to history. Total: 4
th(0)=-30.025577471901972;dx=-1.5055566707768956E-6
New Minimum: -30.025577471901972 > -30.036013122648
WOLFE (weak): th(65455.91506173263)=-30.036013122648; dx=-1.4124913839711281E-6 evalInputDelta=0.010435650746028813
New Minimum: -30.036013122648 > -30.045879961922765
END: th(130911.83012346526)=-30.045879961922765; dx=-1.2988822370681641E-6 evalInputDelta=0.020302490020792874
Fitness changed from -30.025577471901972 to -30.045879961922765
Iteration 19 complete. Error: -30.045879961922765 Total: 123.0241; Orientation: 0.7856; Line Search: 91.4047
Adding measurement 321b7b9e to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.482e+03, gradient 1.146e-03, dot -0.911; [e2e5fea1-cfd0-47d3-b92b-b6bc472ad2b3 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -29.92106167723735, -30.02203716461857, -30.025577471901972, -30.045879961922765
LBFGS Accumulation History: 3 points
Removed measurement 3a7d914c to history. Total: 4
th(0)=-30.045879961922765;dx=-1.3142337597535555E-6
New Minimum: -30.045879961922765 > -30.08045960466067
END: th(282040.9881535545)=-30.08045960466067; dx=-1.1070564638089926E-6 evalInputDelta=0.03457964273790637
Fitness changed from -30.045879961922765 to -30.08045960466067
Iteration 20 complete. Error: -30.08045960466067 Total: 92.3099; Orientation: 0.7938; Line Search: 61.6040
<a id="p-17"></a>Iteration 20
<a id="p-16"></a>
Final threshold in iteration 21: -30.08045960466067 (> -Infinity) after 2597.779s (< 3600.000s)
-30.08045960466067