Code from BasicOptimizer.scala:75 executed in 967.35 seconds (11.042 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
IterativeTrainer.wrap(trainable)
.setOrientation(orientation())
.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = trainingMonitor.log(msg)
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable, currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable, currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
.setTimeout(trainingMinutes, TimeUnit.MINUTES)
.setMaxIterations(trainingIterations)
.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
.runAndFree
.asInstanceOf[lang.Double]
Reset training subject: 1232934903475900
Reset training subject: 1232943268057500
Adding measurement 1d60059f to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD+Trust
th(0)=5.28758499782191;dx=-3.080391069203297E-5
New Minimum: 5.28758499782191 > 5.287556017422061
WOLFE (weak): th(2.154434690031884)=5.287556017422061; dx=-3.0803529375591793E-5 evalInputDelta=2.898039984966516E-5
New Minimum: 5.287556017422061 > 5.2875270566695
WOLFE (weak): th(4.308869380063768)=5.2875270566695; dx=-3.080344919789777E-5 evalInputDelta=5.794115241020137E-5
New Minimum: 5.2875270566695 > 5.287411115369769
WOLFE (weak): th(12.926608140191302)=5.287411115369769; dx=-3.080193662090393E-5 evalInputDelta=1.7388245214089437E-4
New Minimum: 5.287411115369769 > 5.286889225492696
WOLFE (weak): th(51.70643256076521)=5.286889225492696; dx=-3.079761916923837E-5 evalInputDelta=6.957723292142504E-4
New Minimum: 5.286889225492696 > 5.2841097567023505
WOLFE (weak): th(258.53216280382605)=5.2841097567023505; dx=-3.073482921865918E-5 evalInputDelta=0.0034752411195597688
New Minimum: 5.2841097567023505 > 5.2668285250797835
WOLFE (weak): th(1551.1929768229563)=5.2668285250797835; dx=-3.045251135533498E-5 evalInputDelta=0.020756472742126775
New Minimum: 5.2668285250797835 > 5.145845594624842
WOLFE (weak): th(10858.350837760694)=5.145845594624842; dx=-2.857118292258857E-5 evalInputDelta=0.14173940319706801
New Minimum: 5.145845594624842 > 4.403030172774304
END: th(86866.80670208555)=4.403030172774304; dx=-1.7528707385283317E-5 evalInputDelta=0.8845548250476059
Fitness changed from 5.28758499782191 to 4.403030172774304
Iteration 1 complete. Error: 4.403030172774304 Total: 97.8684; Orientation: 0.0078; Line Search: 73.4441
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 1d99ee1b to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=4.403030172774304;dx=-1.2896848657236399E-5
New Minimum: 4.403030
...skipping 23979 bytes...
8551632853013
Fitness changed from 0.020954461332736582 to -0.008200686595712534
Iteration 28 complete. Error: -0.008200686595712534 Total: 47.5585; Orientation: 0.2147; Line Search: 39.4229
Adding measurement 7d088813 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.383e+03, gradient 2.429e-04, dot -0.581; [ffef6275-fd7b-4d13-a6ee-60b02d82914a = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.040416857307030166, 0.02432413683326337, 0.020954461332736582, -0.008200686595712534
LBFGS Accumulation History: 3 points
Removed measurement a5e8260 to history. Total: 4
th(0)=-0.008200686595712534;dx=-5.9001739546008146E-8
New Minimum: -0.008200686595712534 > -0.022690015627174306
END: th(870496.877017144)=-0.022690015627174306; dx=-3.7226858017182574E-8 evalInputDelta=0.014489329031461773
Fitness changed from -0.008200686595712534 to -0.022690015627174306
Iteration 29 complete. Error: -0.022690015627174306 Total: 23.7795; Orientation: 0.2115; Line Search: 15.6968
Adding measurement 42576db9 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.314e+03, gradient 1.874e-04, dot -0.895; [ffef6275-fd7b-4d13-a6ee-60b02d82914a = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.02432413683326337, 0.020954461332736582, -0.008200686595712534, -0.022690015627174306
LBFGS Accumulation History: 3 points
Removed measurement 6c6379c to history. Total: 4
th(0)=-0.022690015627174306;dx=-3.51288915649712E-8
New Minimum: -0.022690015627174306 > -0.03730080723802774
END: th(1875428.6694101535)=-0.03730080723802774; dx=-2.5777098222151766E-8 evalInputDelta=0.014610791610853435
Fitness changed from -0.022690015627174306 to -0.03730080723802774
Iteration 30 complete. Error: -0.03730080723802774 Total: 24.0593; Orientation: 0.2143; Line Search: 16.0647
<a id="p-21"></a>Iteration 30
<a id="p-20"></a>
Final threshold in iteration 31: -0.03730080723802774 (> -Infinity) after 967.340s (< 3600.000s)
-0.03730080723802774