BasicOptimizer.scala:89 executed in 312.84 seconds (3.828 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 582891681616400
Reset training subject: 582897718420000
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
New Minimum: -43.104709401934315 > -43.135811657922
th(0)=-43.135811657922;dx=-2.065558275929771E-6
New Minimum: -43.135811657922 > -43.13581515975698
WOLFE (weak): th(2.154434690031884)=-43.13581515975698; dx=-1.972000847540348E-6 evalInputDelta=3.5018349819893047E-6
New Minimum: -43.13581515975698 > -43.13582041250891
WOLFE (weak): th(4.308869380063768)=-43.13582041250891; dx=-1.9707952907501607E-6 evalInputDelta=8.754586914960782E-6
New Minimum: -43.13582041250891 > -43.13583792167743
WOLFE (weak): th(12.926608140191302)=-43.13583792167743; dx=-1.971653086473398E-6 evalInputDelta=2.6263755430022684E-5
New Minimum: -43.13583792167743 > -43.13592371650039
WOLFE (weak): th(51.70643256076521)=-43.13592371650039; dx=-1.9716150112142932E-6 evalInputDelta=1.1205857839513556E-4
New Minimum: -43.13592371650039 > -43.13636844611316
WOLFE (weak): th(258.53216280382605)=-43.13636844611316; dx=-1.971445707805399E-6 evalInputDelta=5.567881911616723E-4
New Minimum: -43.13636844611316 > -43.13914702692746
WOLFE (weak): th(1551.1929768229563)=-43.13914702692746; dx=-1.9719155653771043E-6 evalInputDelta=0.0033353690054624963
New Minimum: -43.13914702692746 > -43.1591485780968
WOLFE (weak): th(10858.350837760694)=-43.1591485780968; dx=-1.9710481542186127E-6 evalInputDelta=0.023336920174799047
New Minimum: -43.1591485780968 > -43.322209071853145
WOLFE (weak): th(86866.80670208555)=-43.322209071853145; dx=-1.9652562637329583E-6 evalInputDelta=0.18639741393114662
New Minimum: -43.322209071853145 > -44.786765585351226
WOLFE (weak): th(781801.26031877)=-44.786765585351226; dx=-1.9021525741040918E-6 evalInputDelta=1.650953927429228
New Minimum: -44.786765585351226 > -56.11760088767495
END: th(7818012.6031877)=-56.11760088767495; dx=-1.045157550240317E-6 evalInputDelta=12.981789229752955
Fitness changed from -43.104709401934315 to -56.11760088767495
Iteration 1 complete. Error: -56.11760088767495 Total: 85.3733; Orientation: 0.0915; Line Search: 67.3078
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement a2d30bd to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-56.11760088767495;dx=-1.1210819191096773E-6
New Minimum: -56.11760088767495 > -69.14398925146082
END: th(1.684339755941405E7)=-69.14398925146082; dx=-3.998430339708372E-7 evalInputDelta=13.02638836378587
Fitness changed from -56.11760088767495 to -69.14398925146082
Iteration 2 complete. Error: -69.14398925146082 Total: 19.3730; Orientation: 0.1376; Line Search: 12.2229
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 7b2e3000 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-69.14398925146082;dx=-8.182553194278019E-7
New Minimum: -69.14398925146082 > -78.2457294857308
END: th(3.6288E7)=-78.2457294857308; dx=-4.072653470939805E-9 evalInputDelta=9.101740234269982
Fitness changed from -69.14398925146082 to -78.2457294857308
Iteration 3 complete. Error: -78.2457294857308 Total: 19.6303; Orientation: 0.1349; Line Search: 12.3542
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 74e12f6c to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-78.2457294857308;dx=-1.010785003828106E-6
Armijo: th(7.8180126031877E7)=-70.46639682153547; dx=2.9454266698164853E-7 evalInputDelta=-7.779332664195337
New Minimum: -78.2457294857308 > -85.911597817081
WOLF (strong): th(3.90900630159385E7)=-85.911597817081; dx=3.1500952923786247E-7 evalInputDelta=7.665868331350197
New Minimum: -85.911597817081 > -88.4739713324423
END: th(1.3030021005312832E7)=-88.4739713324423; dx=-3.8157377876503443E-7 evalInputDelta=10.228241846711498
Fitness changed from -78.2457294857308 to -88.4739713324423
Iteration 4 complete. Error: -88.4739713324423 Total: 31.5126; Orientation: 0.1421; Line Search: 24.1711
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 6c7b8864 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 5.895e+04, gradient 6.913e-04, dot -0.656; [02defb19-d44a-4c13-afaa-6586e0b2b5e2 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -88.4739713324423, -78.2457294857308, -69.14398925146082, -56.11760088767495
LBFGS Accumulation History: 3 points
Removed measurement 6c7b8864 to history. Total: 3
th(0)=-88.4739713324423;dx=-4.774849218734537E-7
New Minimum: -88.4739713324423 > -96.36184266865692
END: th(2.807232926569009E7)=-96.36184266865692; dx=-8.021187421750154E-8 evalInputDelta=7.887871336214616
Fitness changed from -88.4739713324423 to -96.36184266865692
Iteration 5 complete. Error: -96.36184266865692 Total: 21.4409; Orientation: 2.5869; Line Search: 11.7675
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 625facfa to history. Total: 3
Rejected: LBFGS Orientation magnitude: 6.041e+04, gradient 7.261e-04, dot -0.567; [02defb19-d44a-4c13-afaa-6586e0b2b5e2 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -96.36184266865692, -78.2457294857308, -69.14398925146082, -56.11760088767495
LBFGS Accumulation History: 3 points
Removed measurement 625facfa to history. Total: 3
th(0)=-96.36184266865692;dx=-5.267253951761795E-7
Armijo: th(6.048000000000001E7)=-94.97320106047403; dx=1.846768548162865E-7 evalInputDelta=-1.388641608182894
New Minimum: -96.36184266865692 > -100.93115765083841
WOLF (strong): th(3.0240000000000004E7)=-100.93115765083841; dx=1.0042118289484938E-7 evalInputDelta=4.569314982181496
END: th(1.0080000000000002E7)=-100.57689615678308; dx=-2.3963939261388756E-7 evalInputDelta=4.215053488126159
Fitness changed from -96.36184266865692 to -100.93115765083841
Iteration 6 complete. Error: -100.93115765083841 Total: 33.6447; Orientation: 2.5248; Line Search: 23.8193
Adding measurement 345b49b2 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 7.591e+04, gradient 8.218e-04, dot -0.463; [02defb19-d44a-4c13-afaa-6586e0b2b5e2 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -100.93115765083841, -78.2457294857308, -69.14398925146082, -56.11760088767495
LBFGS Accumulation History: 3 points
Removed measurement 345b49b2 to history. Total: 3
th(0)=-100.93115765083841;dx=-6.750577608608178E-7
New Minimum: -100.93115765083841 > -108.00353352056472
WOLF (strong): th(2.1716701675521392E7)=-108.00353352056472; dx=8.286752148855879E-8 evalInputDelta=7.072375869726301
END: th(1.0858350837760696E7)=-106.92810508238448; dx=-3.091665539497416E-7 evalInputDelta=5.996947431546062
Fitness changed from -100.93115765083841 to -108.00353352056472
Iteration 7 complete. Error: -108.00353352056472 Total: 26.7821; Orientation: 2.5178; Line Search: 18.2546
Adding measurement 6b646332 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 8.112e+04, gradient 6.497e-04, dot -0.530; [02defb19-d44a-4c13-afaa-6586e0b2b5e2 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -108.00353352056472, -78.2457294857308, -69.14398925146082, -56.11760088767495
LBFGS Accumulation History: 3 points
Removed measurement 6b646332 to history. Total: 3
th(0)=-108.00353352056472;dx=-4.215829335032337E-7
New Minimum: -108.00353352056472 > -112.14123116043548
WOLF (strong): th(2.339360772140841E7)=-112.14123116043548; dx=4.730056793186589E-8 evalInputDelta=4.137697639870765
END: th(1.1696803860704206E7)=-111.69500282828352; dx=-1.6143081845570408E-7 evalInputDelta=3.6914693077188048
Fitness changed from -108.00353352056472 to -112.14123116043548
Iteration 8 complete. Error: -112.14123116043548 Total: 25.9091; Orientation: 2.4696; Line Search: 17.6732
Adding measurement 79a41d79 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 9.624e+04, gradient 6.460e-04, dot -0.514; [02defb19-d44a-4c13-afaa-6586e0b2b5e2 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -112.14123116043548, -78.2457294857308, -69.14398925146082, -56.11760088767495
LBFGS Accumulation History: 3 points
Removed measurement 79a41d79 to history. Total: 3
th(0)=-112.14123116043548;dx=-4.169233702638737E-7
New Minimum: -112.14123116043548 > -115.68108204543444
WOLF (strong): th(2.5200000000000007E7)=-115.68108204543444; dx=9.692096656904999E-8 evalInputDelta=3.5398508849989554
New Minimum: -115.68108204543444 > -115.88334352181748
END: th(1.2600000000000004E7)=-115.88334352181748; dx=-1.2516966346382984E-7 evalInputDelta=3.7421123613819987
Fitness changed from -112.14123116043548 to -115.88334352181748
Iteration 9 complete. Error: -115.88334352181748 Total: 26.8740; Orientation: 2.4958; Line Search: 18.6178
Adding measurement 47b6a8a3 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 9.819e+04, gradient 4.378e-04, dot -0.693; [02defb19-d44a-4c13-afaa-6586e0b2b5e2 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -115.88334352181748, -78.2457294857308, -69.14398925146082, -56.11760088767495
LBFGS Accumulation History: 3 points
Removed measurement 47b6a8a3 to history. Total: 3
th(0)=-115.88334352181748;dx=-1.9144240302720787E-7
New Minimum: -115.88334352181748 > -119.67825114288735
END: th(2.7145877094401743E7)=-119.67825114288735; dx=-9.361064273940622E-8 evalInputDelta=3.794907621069868
Fitness changed from -115.88334352181748 to -119.67825114288735
Iteration 10 complete. Error: -119.67825114288735 Total: 21.0360; Orientation: 2.6248; Line Search: 12.4672
<a id="p-13"></a>Iteration 10
<a id="p-12"></a>
Final threshold in iteration 11: -119.67825114288735 (> -Infinity) after 312.844s (< 5400.000s)
Returns
-119.67825114288735