BasicOptimizer.scala:89 executed in 211.44 seconds (3.405 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 685683341101500
Reset training subject: 685687873350200
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
New Minimum: -817.7683810224017 > -817.973639550811
th(0)=-817.973639550811;dx=-0.007206890062622482
New Minimum: -817.973639550811 > -817.9893031085431
WOLFE (weak): th(2.154434690031884)=-817.9893031085431; dx=-0.006635234211892887 evalInputDelta=0.0156635577321822
New Minimum: -817.9893031085431 > -818.0049663663418
WOLFE (weak): th(4.308869380063768)=-818.0049663663418; dx=-0.006632924583974097 evalInputDelta=0.031326815530860586
New Minimum: -818.0049663663418 > -818.0675781987696
WOLFE (weak): th(12.926608140191302)=-818.0675781987696; dx=-0.006632282155255341 evalInputDelta=0.09393864795868012
New Minimum: -818.0675781987696 > -818.3494058163665
WOLFE (weak): th(51.70643256076521)=-818.3494058163665; dx=-0.006632207734259196 evalInputDelta=0.3757662655555123
New Minimum: -818.3494058163665 > -819.8502912117553
WOLFE (weak): th(258.53216280382605)=-819.8502912117553; dx=-0.006615511697927359 evalInputDelta=1.8766516609443897
New Minimum: -819.8502912117553 > -829.1642252895382
WOLFE (weak): th(1551.1929768229563)=-829.1642252895382; dx=-0.006537811882939185 evalInputDelta=11.190585738727236
New Minimum: -829.1642252895382 > -892.3060713678911
END: th(10858.350837760694)=-892.3060713678911; dx=-0.005817282018058579 evalInputDelta=74.33243181708019
Fitness changed from -817.7683810224017 to -892.3060713678911
Iteration 1 complete. Error: -892.3060713678911 Total: 49.5249; Orientation: 0.0606; Line Search: 36.0578
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 2087b053 to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-892.3060713678911;dx=-0.005458108119478101
New Minimum: -892.3060713678911 > -1015.2777944976439
END: th(23393.607721408407)=-1015.2777944976439; dx=-0.004108261010961492 evalInputDelta=122.97172312975272
Fitness changed from -892.3060713678911 to -1015.2777944976439
Iteration 2 complete. Error: -1015.2777944976439 Total: 14.7718; Orientation: 0.0972; Line Search: 9.1764
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 218bf1e1 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-1015.2777944976439;dx=-0.0038718750679936024
New Minimum: -1015.2777944976439 > -1189.385240365795
END: th(50400.00000000001)=-1189.385240365795; dx=-0.0024455882421770547 evalInputDelta=174.1074458681511
Fitness changed from -1015.2777944976439 to -1189.385240365795
Iteration 3 complete. Error: -1189.385240365795 Total: 14.5107; Orientation: 0.0987; Line Search: 9.3068
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 385885ba to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-1189.385240365795;dx=-0.002731161058216292
New Minimum: -1189.385240365795 > -1402.105559506844
END: th(108583.50837760697)=-1402.105559506844; dx=-0.0011907866790636275 evalInputDelta=212.720319141049
Fitness changed from -1189.385240365795 to -1402.105559506844
Iteration 4 complete. Error: -1402.105559506844 Total: 14.6792; Orientation: 0.0924; Line Search: 9.2847
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 33518c9a to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.732e+04, gradient 5.421e-02, dot -0.595; [0454506a-26a2-4315-83a4-a7a76e024b50 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1402.105559506844, -1189.385240365795, -1015.2777944976439, -892.3060713678911
LBFGS Accumulation History: 3 points
Removed measurement 33518c9a to history. Total: 3
th(0)=-1402.105559506844;dx=-0.002933141169804218
New Minimum: -1402.105559506844 > -1565.177306249998
END: th(233936.07721408413)=-1565.177306249998; dx=-8.919779997438421E-5 evalInputDelta=163.0717467431541
Fitness changed from -1402.105559506844 to -1565.177306249998
Iteration 5 complete. Error: -1565.177306249998 Total: 16.0251; Orientation: 1.7307; Line Search: 8.7860
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 7edfdc78 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.273e+04, gradient 6.329e-02, dot -0.353; [0454506a-26a2-4315-83a4-a7a76e024b50 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1565.177306249998, -1189.385240365795, -1015.2777944976439, -892.3060713678911
LBFGS Accumulation History: 3 points
Removed measurement 7edfdc78 to history. Total: 3
th(0)=-1565.177306249998;dx=-0.004003258834117387
Armijo: th(504000.0000000002)=-1506.955042461453; dx=5.843971097128336E-4 evalInputDelta=-58.22226378854498
New Minimum: -1565.177306249998 > -1712.9457084216067
WOLF (strong): th(252000.0000000001)=-1712.9457084216067; dx=7.037762547190734E-4 evalInputDelta=147.7684021716086
New Minimum: -1712.9457084216067 > -1772.520098616656
END: th(84000.00000000003)=-1772.520098616656; dx=-7.949587283364876E-4 evalInputDelta=207.34279236665793
Fitness changed from -1565.177306249998 to -1772.520098616656
Iteration 6 complete. Error: -1772.520098616656 Total: 25.1592; Orientation: 1.6466; Line Search: 18.0310
Adding measurement 343dda89 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.360e+04, gradient 5.046e-02, dot -0.432; [0454506a-26a2-4315-83a4-a7a76e024b50 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1772.520098616656, -1189.385240365795, -1015.2777944976439, -892.3060713678911
LBFGS Accumulation History: 3 points
Removed measurement 343dda89 to history. Total: 3
th(0)=-1772.520098616656;dx=-0.002537502698752456
New Minimum: -1772.520098616656 > -1932.7475261916647
WOLF (strong): th(180972.5139626783)=-1932.7475261916647; dx=9.055586552671775E-5 evalInputDelta=160.22742757500873
END: th(90486.25698133915)=-1915.8791063112515; dx=-6.612913784540503E-4 evalInputDelta=143.35900769459545
Fitness changed from -1772.520098616656 to -1932.7475261916647
Iteration 7 complete. Error: -1932.7475261916647 Total: 19.8200; Orientation: 1.5894; Line Search: 13.7770
Adding measurement 75a270fe to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.107e+04, gradient 6.497e-02, dot -0.336; [0454506a-26a2-4315-83a4-a7a76e024b50 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1932.7475261916647, -1189.385240365795, -1015.2777944976439, -892.3060713678911
LBFGS Accumulation History: 3 points
Removed measurement 75a270fe to history. Total: 3
th(0)=-1932.7475261916647;dx=-0.0042153939814401995
New Minimum: -1932.7475261916647 > -2013.5926847304547
WOLF (strong): th(194946.7310117368)=-2013.5926847304547; dx=0.0011100503387815484 evalInputDelta=80.84515853878997
New Minimum: -2013.5926847304547 > -2108.23966853866
WOLF (strong): th(97473.3655058684)=-2108.23966853866; dx=1.5086566333275084E-4 evalInputDelta=175.49214234699548
END: th(32491.121835289465)=-2050.4579976190685; dx=-0.0023556633753682114 evalInputDelta=117.71047142740372
Fitness changed from -1932.7475261916647 to -2108.23966853866
Iteration 8 complete. Error: -2108.23966853866 Total: 24.7085; Orientation: 1.5972; Line Search: 18.6220
Adding measurement 74b68428 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.276e+04, gradient 5.385e-02, dot -0.405; [0454506a-26a2-4315-83a4-a7a76e024b50 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -2108.23966853866, -1189.385240365795, -1015.2777944976439, -892.3060713678911
LBFGS Accumulation History: 3 points
Removed measurement 74b68428 to history. Total: 3
th(0)=-2108.23966853866;dx=-0.0028901798814999647
New Minimum: -2108.23966853866 > -2230.7265632524304
END: th(70000.00000000003)=-2230.7265632524304; dx=-5.631886999328819E-4 evalInputDelta=122.4868947137702
Fitness changed from -2108.23966853866 to -2230.7265632524304
Iteration 9 complete. Error: -2230.7265632524304 Total: 15.6282; Orientation: 1.7265; Line Search: 9.2830
Adding measurement 65c6eff2 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 5.871e+04, gradient 4.474e-02, dot -0.508; [0454506a-26a2-4315-83a4-a7a76e024b50 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -2230.7265632524304, -1189.385240365795, -1015.2777944976439, -892.3060713678911
LBFGS Accumulation History: 3 points
Removed measurement 65c6eff2 to history. Total: 3
th(0)=-2230.7265632524304;dx=-0.001998045161514331
New Minimum: -2230.7265632524304 > -2337.3410106358037
END: th(150810.42830223194)=-2337.3410106358037; dx=-2.0328109193972284E-5 evalInputDelta=106.61444738337332
Fitness changed from -2230.7265632524304 to -2337.3410106358037
Iteration 10 complete. Error: -2337.3410106358037 Total: 15.7491; Orientation: 1.6069; Line Search: 9.5328
<a id="p-13"></a>Iteration 10
<a id="p-12"></a>
Final threshold in iteration 11: -2337.3410106358037 (> -Infinity) after 211.443s (< 5400.000s)
Returns
-2337.3410106358037