Code from BasicOptimizer.scala:75 executed in 3690.88 seconds (68.212 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
IterativeTrainer.wrap(trainable)
.setOrientation(orientation())
.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = trainingMonitor.log(msg)
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable, currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable, currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
.setTimeout(trainingMinutes, TimeUnit.MINUTES)
.setMaxIterations(trainingIterations)
.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
.runAndFree
.asInstanceOf[lang.Double]
Reset training subject: 1348791830649400
Reset training subject: 1348889334867000
Adding measurement 29ceefb3 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD+Trust
th(0)=22.226093649864197;dx=-1.8904718046112507E-6
New Minimum: 22.226093649864197 > 22.226091854274273
WOLFE (weak): th(2.154434690031884)=22.226091854274273; dx=-1.8873059207906216E-6 evalInputDelta=1.7955899238586426E-6
New Minimum: 22.226091854274273 > 22.226091295480728
WOLFE (weak): th(4.308869380063768)=22.226091295480728; dx=-1.887305483797562E-6 evalInputDelta=2.3543834686279297E-6
New Minimum: 22.226091295480728 > 22.22608332335949
WOLFE (weak): th(12.926608140191302)=22.22608332335949; dx=-1.8873028205338306E-6 evalInputDelta=1.0326504707336426E-5
New Minimum: 22.22608332335949 > 22.22605476528406
WOLFE (weak): th(51.70643256076521)=22.22605476528406; dx=-1.887296005748751E-6 evalInputDelta=3.888458013534546E-5
New Minimum: 22.22605476528406 > 22.225894883275032
WOLFE (weak): th(258.53216280382605)=22.225894883275032; dx=-1.8872814895142303E-6 evalInputDelta=1.987665891647339E-4
New Minimum: 22.225894883275032 > 22.22489907592535
WOLFE (weak): th(1551.1929768229563)=22.22489907592535; dx=-1.8872193013681154E-6 evalInputDelta=0.0011945739388465881
New Minimum: 22.22489907592535 > 22.217727176845074
WOLFE (weak): th(10858.350837760694)=22.217727176845074; dx=-1.886353192720892E-6 evalInputDelta=0.008366473019123077
New Minimum: 22.217727176845074 > 22.15931749343872
WOLFE (weak): th(86866.80670208555)=22.15931749343872; dx=-1.8797101651454486E-6 evalInputDelta=0.06677615642547607
New Minimum: 22.15931749343872 > 21.638649709522724
WOLFE (weak): th(781801.26031877)=21.638649709522724; dx=-1.8171226928167715E-6 evalInputDelta=0.5874439403414726
New Minimum: 21.638649709522724 > 17.869387827813625
END: th(7818012.6031877)=17.869387827813625; dx=-1.1101315312361518E-6 evalInputDelta=4.356705822050571
Fitness changed from 22.226093649864197 to 17.869387827813625
Iteration
...skipping 3402 bytes...
on: 8.9750; Line Search: 298.4288
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 175377d3 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.423e+04, gradient 8.952e-04, dot -0.615; [1ea3dde1-663a-488c-b63d-b1f5b8a5599f = 1.000/1.000e+00]
Orientation rejected. Popping history element from 14.661463484168053, 13.093659482896328, 11.807237520813942, 11.201448746025562
LBFGS Accumulation History: 3 points
Removed measurement 123a21ad to history. Total: 4
th(0)=11.201448746025562;dx=-8.014582264885084E-7
Armijo: th(3.0240000000000007E7)=17.61188332736492; dx=6.392992732608856E-7 evalInputDelta=-6.410434581339359
Armijo: th(1.5120000000000004E7)=11.261976294219494; dx=1.7150201679697338E-7 evalInputDelta=-0.06052754819393158
New Minimum: 11.201448746025562 > 10.144500702619553
END: th(5040000.000000001)=10.144500702619553; dx=-4.294350969006826E-7 evalInputDelta=1.0569480434060097
Fitness changed from 11.201448746025562 to 10.144500702619553
Iteration 6 complete. Error: 10.144500702619553 Total: 507.1742; Orientation: 9.1229; Line Search: 407.0650
Adding measurement 476a38d9 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.424e+04, gradient 6.673e-04, dot -0.894; [1ea3dde1-663a-488c-b63d-b1f5b8a5599f = 1.000/1.000e+00]
Orientation rejected. Popping history element from 13.093659482896328, 11.807237520813942, 11.201448746025562, 10.144500702619553
LBFGS Accumulation History: 3 points
Removed measurement 3582077c to history. Total: 4
th(0)=10.144500702619553;dx=-4.4531228033820053E-7
New Minimum: 10.144500702619553 > 9.52105025947094
END: th(1.0858350837760696E7)=9.52105025947094; dx=-1.5347815100702702E-7 evalInputDelta=0.623450443148613
Fitness changed from 10.144500702619553 to 9.52105025947094
Iteration 7 complete. Error: 9.52105025947094 Total: 297.8985; Orientation: 8.8901; Line Search: 195.7016
Final threshold in iteration 7: 9.52105025947094 (> -Infinity) after 3690.879s (< 3600.000s)
9.52105025947094