BasicOptimizer.scala:89 executed in 316.88 seconds (6.779 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 589798395584500
Reset training subject: 589804341882300
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
New Minimum: -42.05877068052479 > -42.137259930995626
th(0)=-42.137259930995626;dx=-2.014586037425194E-6
New Minimum: -42.137259930995626 > -42.137261723405686
WOLFE (weak): th(2.154434690031884)=-42.137261723405686; dx=-1.929583050769181E-6 evalInputDelta=1.7924100603750048E-6
New Minimum: -42.137261723405686 > -42.13726889304516
WOLFE (weak): th(4.308869380063768)=-42.13726889304516; dx=-1.930084931949103E-6 evalInputDelta=8.96204953448887E-6
New Minimum: -42.13726889304516 > -42.137288609547426
WOLFE (weak): th(12.926608140191302)=-42.137288609547426; dx=-1.9293358856394065E-6 evalInputDelta=2.8678551799998786E-5
New Minimum: -42.137288609547426 > -42.1373692678697
WOLFE (weak): th(51.70643256076521)=-42.1373692678697; dx=-1.930126774094512E-6 evalInputDelta=1.0933687407543857E-4
New Minimum: -42.1373692678697 > -42.13780302775516
WOLFE (weak): th(258.53216280382605)=-42.13780302775516; dx=-1.930239146273258E-6 evalInputDelta=5.430967595358993E-4
New Minimum: -42.13780302775516 > -42.14051302974851
WOLFE (weak): th(1551.1929768229563)=-42.14051302974851; dx=-1.9298220746430072E-6 evalInputDelta=0.003253098752885819
New Minimum: -42.14051302974851 > -42.16002455958242
WOLFE (weak): th(10858.350837760694)=-42.16002455958242; dx=-1.9293397935222595E-6 evalInputDelta=0.022764628586791957
New Minimum: -42.16002455958242 > -42.31908762137628
WOLFE (weak): th(86866.80670208555)=-42.31908762137628; dx=-1.9229102502869917E-6 evalInputDelta=0.1818276903806577
New Minimum: -42.31908762137628 > -43.74800407054001
WOLFE (weak): th(781801.26031877)=-43.74800407054001; dx=-1.8609691871993206E-6 evalInputDelta=1.610744139544387
New Minimum: -43.74800407054001 > -54.8638348101404
END: th(7818012.6031877)=-54.8638348101404; dx=-1.0351582474698895E-6 evalInputDelta=12.726574879144778
Fitness changed from -42.05877068052479 to -54.8638348101404
Iteration 1 complete. Error: -54.8638348101404 Total: 85.8893; Orientation: 0.0927; Line Search: 68.0903
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 7143a1fe to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-54.8638348101404;dx=-1.089764129383449E-6
New Minimum: -54.8638348101404 > -67.71409479474957
END: th(1.684339755941405E7)=-67.71409479474957; dx=-4.043945278003691E-7 evalInputDelta=12.850259984609167
Fitness changed from -54.8638348101404 to -67.71409479474957
Iteration 2 complete. Error: -67.71409479474957 Total: 19.6228; Orientation: 0.1241; Line Search: 12.5069
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 67249645 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-67.71409479474957;dx=-7.857224946061628E-7
New Minimum: -67.71409479474957 > -76.85752588600134
END: th(3.6288E7)=-76.85752588600134; dx=-7.57971296045354E-9 evalInputDelta=9.14343109125177
Fitness changed from -67.71409479474957 to -76.85752588600134
Iteration 3 complete. Error: -76.85752588600134 Total: 19.8789; Orientation: 0.1483; Line Search: 12.6201
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 7beab93b to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-76.85752588600134;dx=-9.918846740628514E-7
Armijo: th(7.8180126031877E7)=-68.45857686960524; dx=2.8594264099524897E-7 evalInputDelta=-8.398949016396102
New Minimum: -76.85752588600134 > -83.59953752913398
WOLF (strong): th(3.90900630159385E7)=-83.59953752913398; dx=3.225465703613299E-7 evalInputDelta=6.742011643132642
New Minimum: -83.59953752913398 > -86.70724087824406
END: th(1.3030021005312832E7)=-86.70724087824406; dx=-3.543190684754747E-7 evalInputDelta=9.849714992242724
Fitness changed from -76.85752588600134 to -86.70724087824406
Iteration 4 complete. Error: -86.70724087824406 Total: 32.1858; Orientation: 0.1543; Line Search: 24.9099
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 7a82e4f1 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 5.745e+04, gradient 6.779e-04, dot -0.658; [5f5d676c-69fd-4b4d-8543-cf418eb3b8f2 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -86.70724087824406, -76.85752588600134, -67.71409479474957, -54.8638348101404
LBFGS Accumulation History: 3 points
Removed measurement 7a82e4f1 to history. Total: 3
th(0)=-86.70724087824406;dx=-4.586084774389238E-7
New Minimum: -86.70724087824406 > -94.11877948326227
END: th(2.807232926569009E7)=-94.11877948326227; dx=-7.325093077428443E-8 evalInputDelta=7.411538605018208
Fitness changed from -86.70724087824406 to -94.11877948326227
Iteration 5 complete. Error: -94.11877948326227 Total: 22.3275; Orientation: 2.6315; Line Search: 12.2377
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 77bd185c to history. Total: 3
Rejected: LBFGS Orientation magnitude: 5.972e+04, gradient 7.222e-04, dot -0.564; [5f5d676c-69fd-4b4d-8543-cf418eb3b8f2 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -94.11877948326227, -76.85752588600134, -67.71409479474957, -54.8638348101404
LBFGS Accumulation History: 3 points
Removed measurement 77bd185c to history. Total: 3
th(0)=-94.11877948326227;dx=-5.215328387407982E-7
Armijo: th(6.048000000000001E7)=-92.12219396909283; dx=1.9058405163470128E-7 evalInputDelta=-1.996585514169439
New Minimum: -94.11877948326227 > -98.34287866895029
WOLF (strong): th(3.0240000000000004E7)=-98.34287866895029; dx=1.1292615007959503E-7 evalInputDelta=4.224099185688019
END: th(1.0080000000000002E7)=-98.25391026413483; dx=-2.2982742800132383E-7 evalInputDelta=4.135130780872558
Fitness changed from -94.11877948326227 to -98.34287866895029
Iteration 6 complete. Error: -98.34287866895029 Total: 34.3769; Orientation: 2.5455; Line Search: 24.4104
Adding measurement 1daacb6b to history. Total: 3
Rejected: LBFGS Orientation magnitude: 7.275e+04, gradient 8.228e-04, dot -0.457; [5f5d676c-69fd-4b4d-8543-cf418eb3b8f2 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -98.34287866895029, -76.85752588600134, -67.71409479474957, -54.8638348101404
LBFGS Accumulation History: 3 points
Removed measurement 1daacb6b to history. Total: 3
th(0)=-98.34287866895029;dx=-6.76780242145383E-7
New Minimum: -98.34287866895029 > -105.33639186617258
WOLF (strong): th(2.1716701675521392E7)=-105.33639186617258; dx=9.067850549280838E-8 evalInputDelta=6.993513197222285
END: th(1.0858350837760696E7)=-104.32437218943662; dx=-3.069203574093747E-7 evalInputDelta=5.981493520486325
Fitness changed from -98.34287866895029 to -105.33639186617258
Iteration 7 complete. Error: -105.33639186617258 Total: 27.0478; Orientation: 2.4762; Line Search: 18.7585
Adding measurement 422e9744 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 8.040e+04, gradient 6.449e-04, dot -0.527; [5f5d676c-69fd-4b4d-8543-cf418eb3b8f2 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -105.33639186617258, -76.85752588600134, -67.71409479474957, -54.8638348101404
LBFGS Accumulation History: 3 points
Removed measurement 422e9744 to history. Total: 3
th(0)=-105.33639186617258;dx=-4.1508162489680566E-7
New Minimum: -105.33639186617258 > -109.36881738909496
WOLF (strong): th(2.339360772140841E7)=-109.36881738909496; dx=5.063532645859488E-8 evalInputDelta=4.032425522922381
END: th(1.1696803860704206E7)=-108.96023439679807; dx=-1.5741023349821322E-7 evalInputDelta=3.6238425306254953
Fitness changed from -105.33639186617258 to -109.36881738909496
Iteration 8 complete. Error: -109.36881738909496 Total: 26.6299; Orientation: 2.4520; Line Search: 18.4156
Adding measurement 426611d3 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 9.359e+04, gradient 6.397e-04, dot -0.511; [5f5d676c-69fd-4b4d-8543-cf418eb3b8f2 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -109.36881738909496, -76.85752588600134, -67.71409479474957, -54.8638348101404
LBFGS Accumulation History: 3 points
Removed measurement 426611d3 to history. Total: 3
th(0)=-109.36881738909496;dx=-4.0852326793014684E-7
New Minimum: -109.36881738909496 > -112.79349245974791
WOLF (strong): th(2.5200000000000007E7)=-112.79349245974791; dx=9.87124822308739E-8 evalInputDelta=3.4246750706529525
New Minimum: -112.79349245974791 > -113.01660429625453
END: th(1.2600000000000004E7)=-113.01660429625453; dx=-1.2021905978363436E-7 evalInputDelta=3.6477869071595705
Fitness changed from -109.36881738909496 to -113.01660429625453
Iteration 9 complete. Error: -113.01660429625453 Total: 26.9245; Orientation: 2.5421; Line Search: 18.3274
Adding measurement 1ea2bfec to history. Total: 3
Rejected: LBFGS Orientation magnitude: 9.673e+04, gradient 4.343e-04, dot -0.688; [5f5d676c-69fd-4b4d-8543-cf418eb3b8f2 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -113.01660429625453, -76.85752588600134, -67.71409479474957, -54.8638348101404
LBFGS Accumulation History: 3 points
Removed measurement 1ea2bfec to history. Total: 3
th(0)=-113.01660429625453;dx=-1.878192366378756E-7
New Minimum: -113.01660429625453 > -116.69122922343765
END: th(2.7145877094401743E7)=-116.69122922343765; dx=-8.832945282479588E-8 evalInputDelta=3.6746249271831175
Fitness changed from -113.01660429625453 to -116.69122922343765
Iteration 10 complete. Error: -116.69122922343765 Total: 20.7486; Orientation: 2.5270; Line Search: 12.0704
<a id="p-13"></a>Iteration 10
<a id="p-12"></a>
Final threshold in iteration 11: -116.69122922343765 (> -Infinity) after 316.883s (< 5400.000s)
Returns
-116.69122922343765