Code from BasicOptimizer.scala:75 executed in 1002.59 seconds (11.689 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
IterativeTrainer.wrap(trainable)
.setOrientation(orientation())
.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = trainingMonitor.log(msg)
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable, currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable, currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
.setTimeout(trainingMinutes, TimeUnit.MINUTES)
.setMaxIterations(trainingIterations)
.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
.runAndFree
.asInstanceOf[lang.Double]
Reset training subject: 1152791531901500
Reset training subject: 1152800322701900
Adding measurement 642a16aa to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD+Trust
th(0)=5.210621138801221;dx=-2.9610981859656042E-5
New Minimum: 5.210621138801221 > 5.210593487587124
WOLFE (weak): th(2.154434690031884)=5.210593487587124; dx=-2.9610240388784348E-5 evalInputDelta=2.7651214097090815E-5
New Minimum: 5.210593487587124 > 5.210565716838742
WOLFE (weak): th(4.308869380063768)=5.210565716838742; dx=-2.9610341238304123E-5 evalInputDelta=5.542196247887432E-5
New Minimum: 5.210565716838742 > 5.210454721370478
WOLFE (weak): th(12.926608140191302)=5.210454721370478; dx=-2.961133639015905E-5 evalInputDelta=1.6641743074252702E-4
New Minimum: 5.210454721370478 > 5.209955683410293
WOLFE (weak): th(51.70643256076521)=5.209955683410293; dx=-2.9601962012488452E-5 evalInputDelta=6.654553909282157E-4
New Minimum: 5.209955683410293 > 5.207294661899595
WOLFE (weak): th(258.53216280382605)=5.207294661899595; dx=-2.9578056275231146E-5 evalInputDelta=0.0033264769016261297
New Minimum: 5.207294661899595 > 5.190726199022352
WOLFE (weak): th(1551.1929768229563)=5.190726199022352; dx=-2.9361576637458132E-5 evalInputDelta=0.019894939778868803
New Minimum: 5.190726199022352 > 5.07505251705473
WOLFE (weak): th(10858.350837760694)=5.07505251705473; dx=-2.7740619666310342E-5 evalInputDelta=0.13556862174649087
New Minimum: 5.07505251705473 > 4.343153195847048
END: th(86866.80670208555)=4.343153195847048; dx=-1.7429875038096556E-5 evalInputDelta=0.867467942954173
Fitness changed from 5.210621138801221 to 4.343153195847048
Iteration 1 complete. Error: 4.343153195847048 Total: 101.6758; Orientation: 0.0180; Line Search: 75.8569
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 41ccb3b9 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=4.343153195847048;dx=-1.3116656564184249E-5
New Minimum: 4
...skipping 23898 bytes...
ta=0.022021754012894967
Fitness changed from -0.11238436106385935 to -0.13440611507675432
Iteration 28 complete. Error: -0.13440611507675432 Total: 24.4813; Orientation: 0.1992; Line Search: 16.1378
Adding measurement 7f27f59b to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.123e+03, gradient 2.083e-04, dot -0.870; [148240b0-9385-4c3c-8c80-9fac73071be2 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -0.08130349701675878, -0.08192081316464324, -0.11238436106385935, -0.13440611507675432
LBFGS Accumulation History: 3 points
Removed measurement 61b65d54 to history. Total: 4
th(0)=-0.13440611507675432;dx=-4.339334468267666E-8
New Minimum: -0.13440611507675432 > -0.15277982626580563
END: th(1958617.973288574)=-0.15277982626580563; dx=-3.018772139078855E-8 evalInputDelta=0.018373711189051317
Fitness changed from -0.13440611507675432 to -0.15277982626580563
Iteration 29 complete. Error: -0.15277982626580563 Total: 24.4942; Orientation: 0.2031; Line Search: 16.1991
Adding measurement 20440c6c to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.245e+03, gradient 1.966e-04, dot -0.878; [148240b0-9385-4c3c-8c80-9fac73071be2 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -0.08192081316464324, -0.11238436106385935, -0.13440611507675432, -0.15277982626580563
LBFGS Accumulation History: 3 points
Removed measurement 14b528b6 to history. Total: 4
th(0)=-0.15277982626580563;dx=-3.8650758201917245E-8
New Minimum: -0.15277982626580563 > -0.16921872281801653
END: th(4219714.506172845)=-0.16921872281801653; dx=-1.7312969078061724E-8 evalInputDelta=0.016438896552210897
Fitness changed from -0.15277982626580563 to -0.16921872281801653
Iteration 30 complete. Error: -0.16921872281801653 Total: 24.9008; Orientation: 0.2012; Line Search: 16.3280
<a id="p-21"></a>Iteration 30
<a id="p-20"></a>
Final threshold in iteration 31: -0.16921872281801653 (> -Infinity) after 1002.573s (< 3600.000s)
-0.16921872281801653