Code from BasicOptimizer.scala:75 executed in 3709.51 seconds (32.250 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
IterativeTrainer.wrap(trainable)
.setOrientation(orientation())
.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = trainingMonitor.log(msg)
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable, currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable, currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
.setTimeout(trainingMinutes, TimeUnit.MINUTES)
.setMaxIterations(trainingIterations)
.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
.runAndFree
.asInstanceOf[lang.Double]
Reset training subject: 1427105875457500
Reset training subject: 1427151812926700
Adding measurement e890591 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD+Trust
th(0)=-28.39974742817382;dx=-6.4939212516271355E-6
New Minimum: -28.39974742817382 > -28.39974843369176
WOLFE (weak): th(2.154434690031884)=-28.39974843369176; dx=-6.481124475060362E-6 evalInputDelta=1.0055179373580359E-6
New Minimum: -28.39974843369176 > -28.39975037580977
WOLFE (weak): th(4.308869380063768)=-28.39975037580977; dx=-6.469325421920331E-6 evalInputDelta=2.9476359486579895E-6
New Minimum: -28.39975037580977 > -28.399757218547165
WOLFE (weak): th(12.926608140191302)=-28.399757218547165; dx=-6.423714770206036E-6 evalInputDelta=9.790373344031877E-6
New Minimum: -28.399757218547165 > -28.39979150922348
WOLFE (weak): th(51.70643256076521)=-28.39979150922348; dx=-6.234158374768097E-6 evalInputDelta=4.408104965847315E-5
New Minimum: -28.39979150922348 > -28.39994866711398
END: th(258.53216280382605)=-28.39994866711398; dx=-5.717892452818799E-6 evalInputDelta=2.012389401571113E-4
Fitness changed from -28.39974742817382 to -28.39994866711398
Iteration 1 complete. Error: -28.39994866711398 Total: 422.4146; Orientation: 0.0481; Line Search: 281.9350
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 443ac5b8 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-28.39994866711398;dx=-5.731453176657987E-6
New Minimum: -28.39994866711398 > -28.40035988452534
WOLFE (weak): th(556.9906600335335)=-28.40035988452534; dx=-5.729117054949204E-6 evalInputDelta=4.112174113615197E-4
New Minimum: -28.40035988452534 > -28.400778513712186
WOLFE (weak): th(1113.981320067067)=-28.400778513712186; dx=-5.727053570115494E-6 evalInputDelta=8.298465982079506E-4
New Minimum: -28.400778513712186 > -28.402436342400808
WOLFE (weak): th(3341.9439602012008)=-28.402436342400808; dx=-5.716880359746438E-6 evalInputDelta=0.
...skipping 13986 bytes...
2299
New Minimum: -30.623306235919397 > -30.64207612226407
END: th(644665.1157795531)=-30.64207612226407; dx=-4.4153566373764474E-7 evalInputDelta=0.05913110139469424
Fitness changed from -30.582945020869374 to -30.64207612226407
Iteration 17 complete. Error: -30.64207612226407 Total: 238.0992; Orientation: 1.4626; Line Search: 188.7450
Adding measurement 2a79da3a to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.555e+03, gradient 8.416e-04, dot -0.793; [93d8f0dd-15ca-46cf-98f9-71f6b0d8dbd9 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -30.430563499530155, -30.518397318199277, -30.582945020869374, -30.64207612226407
LBFGS Accumulation History: 3 points
Removed measurement b386a17 to history. Total: 4
th(0)=-30.64207612226407;dx=-7.082639117282443E-7
New Minimum: -30.64207612226407 > -30.705470889806747
END: th(1388888.88888889)=-30.705470889806747; dx=-4.400629737457683E-7 evalInputDelta=0.06339476754267892
Fitness changed from -30.64207612226407 to -30.705470889806747
Iteration 18 complete. Error: -30.705470889806747 Total: 142.8708; Orientation: 1.4309; Line Search: 93.5588
Adding measurement 27d0376f to history. Total: 3
Rejected: LBFGS Orientation magnitude: 6.236e+03, gradient 8.150e-04, dot -0.830; [93d8f0dd-15ca-46cf-98f9-71f6b0d8dbd9 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -30.518397318199277, -30.582945020869374, -30.64207612226407, -30.705470889806747
LBFGS Accumulation History: 3 points
Removed measurement 41b2123e to history. Total: 4
th(0)=-30.705470889806747;dx=-6.64261151698508E-7
New Minimum: -30.705470889806747 > -30.764008305966854
END: th(2992270.4028220633)=-30.764008305966854; dx=-1.8485238155806954E-7 evalInputDelta=0.05853741616010666
Fitness changed from -30.705470889806747 to -30.764008305966854
Iteration 19 complete. Error: -30.764008305966854 Total: 142.6652; Orientation: 1.4446; Line Search: 94.2265
Final threshold in iteration 19: -30.764008305966854 (> -Infinity) after 3709.507s (< 3600.000s)
-30.764008305966854