BasicOptimizer.scala:89 executed in 135.90 seconds (1.726 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 70849351238000
Reset training subject: 70852193404200
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
New Minimum: -18.90133632829671 > -18.93887311402369
th(0)=-18.93887311402369;dx=-7.72234159174789E-7
New Minimum: -18.93887311402369 > -18.938874477025436
WOLFE (weak): th(2.154434690031884)=-18.938874477025436; dx=-7.099962153379222E-7 evalInputDelta=1.363001747023418E-6
New Minimum: -18.938874477025436 > -18.93887584002708
WOLFE (weak): th(4.308869380063768)=-18.93887584002708; dx=-7.098438139521178E-7 evalInputDelta=2.726003391018139E-6
New Minimum: -18.93887584002708 > -18.938883336534378
WOLFE (weak): th(12.926608140191302)=-18.938883336534378; dx=-7.099933360677185E-7 evalInputDelta=1.022251068860669E-5
New Minimum: -18.938883336534378 > -18.938913322533903
WOLFE (weak): th(51.70643256076521)=-18.938913322533903; dx=-7.098906277055479E-7 evalInputDelta=4.0208510213801674E-5
New Minimum: -18.938913322533903 > -18.93907551870839
WOLFE (weak): th(258.53216280382605)=-18.93907551870839; dx=-7.099029119237253E-7 evalInputDelta=2.0240468469978623E-4
New Minimum: -18.93907551870839 > -18.940086146772238
WOLFE (weak): th(1551.1929768229563)=-18.940086146772238; dx=-7.098084501884314E-7 evalInputDelta=0.0012130327485486703
New Minimum: -18.940086146772238 > -18.94736066882101
WOLFE (weak): th(10858.350837760694)=-18.94736066882101; dx=-7.089782920327799E-7 evalInputDelta=0.008487554797319774
New Minimum: -18.94736066882101 > -19.00651871088473
WOLFE (weak): th(86866.80670208555)=-19.00651871088473; dx=-7.034630219731472E-7 evalInputDelta=0.06764559686104121
New Minimum: -19.00651871088473 > -19.514895592398908
END: th(781801.26031877)=-19.514895592398908; dx=-6.187884490554915E-7 evalInputDelta=0.5760224783752186
Fitness changed from -18.90133632829671 to -19.514895592398908
Iteration 1 complete. Error: -19.514895592398908 Total: 34.4917; Orientation: 0.0414; Line Search: 26.4614
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 613761e3 to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-19.514895592398908;dx=-6.147033469001456E-7
New Minimum: -19.514895592398908 > -20.523100674378377
END: th(1684339.7559414052)=-20.523100674378377; dx=-4.6505805219146257E-7 evalInputDelta=1.0082050819794688
Fitness changed from -19.514895592398908 to -20.523100674378377
Iteration 2 complete. Error: -20.523100674378377 Total: 8.5191; Orientation: 0.0508; Line Search: 5.2729
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 1e2405b2 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-20.523100674378377;dx=-5.298322354506032E-7
New Minimum: -20.523100674378377 > -22.007404601425403
END: th(3628800.0)=-22.007404601425403; dx=-2.777160366246413E-7 evalInputDelta=1.4843039270470264
Fitness changed from -20.523100674378377 to -22.007404601425403
Iteration 3 complete. Error: -22.007404601425403 Total: 8.6099; Orientation: 0.0628; Line Search: 5.4466
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 54b16abe to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-22.007404601425403;dx=-5.440938863645622E-7
New Minimum: -22.007404601425403 > -23.629579327236183
END: th(7818012.6031877)=-23.629579327236183; dx=-9.517456517008265E-8 evalInputDelta=1.6221747258107797
Fitness changed from -22.007404601425403 to -23.629579327236183
Iteration 4 complete. Error: -23.629579327236183 Total: 8.4547; Orientation: 0.0619; Line Search: 5.3734
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 15aad0cc to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.685e+04, gradient 7.130e-04, dot -0.508; [01de5fb3-4c75-4284-a1e8-c29b945b681c = 1.000/1.000e+00]
Orientation rejected. Popping history element from -23.629579327236183, -22.007404601425403, -20.523100674378377, -19.514895592398908
LBFGS Accumulation History: 3 points
Removed measurement 15aad0cc to history. Total: 3
th(0)=-23.629579327236183;dx=-5.079234171562866E-7
New Minimum: -23.629579327236183 > -25.0722830680308
WOLF (strong): th(1.684339755941405E7)=-25.0722830680308; dx=2.5436209699955384E-8 evalInputDelta=1.4427037407946166
New Minimum: -25.0722830680308 > -25.33445495615947
WOLF (strong): th(8421698.779707026)=-25.33445495615947; dx=7.491118276943472E-9 evalInputDelta=1.7048756289232863
END: th(2807232.9265690087)=-24.884099009908915; dx=-2.578402777952153E-7 evalInputDelta=1.2545196826727327
Fitness changed from -23.629579327236183 to -25.33445495615947
Iteration 5 complete. Error: -25.33445495615947 Total: 14.4144; Orientation: 1.1262; Line Search: 10.1360
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 49539ec0 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.555e+04, gradient 7.495e-04, dot -0.417; [01de5fb3-4c75-4284-a1e8-c29b945b681c = 1.000/1.000e+00]
Orientation rejected. Popping history element from -25.33445495615947, -22.007404601425403, -20.523100674378377, -19.514895592398908
LBFGS Accumulation History: 3 points
Removed measurement 49539ec0 to history. Total: 3
th(0)=-25.33445495615947;dx=-5.615077528198063E-7
New Minimum: -25.33445495615947 > -27.27394409452452
WOLF (strong): th(6048000.0)=-27.27394409452452; dx=4.175331696442853E-8 evalInputDelta=1.939489138365051
END: th(3024000.0)=-26.910904591147236; dx=-3.0267216918742386E-7 evalInputDelta=1.5764496349877675
Fitness changed from -25.33445495615947 to -27.27394409452452
Iteration 6 complete. Error: -27.27394409452452 Total: 12.2304; Orientation: 1.1106; Line Search: 7.9275
Adding measurement 608cc573 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.200e+04, gradient 7.487e-04, dot -0.421; [01de5fb3-4c75-4284-a1e8-c29b945b681c = 1.000/1.000e+00]
Orientation rejected. Popping history element from -27.27394409452452, -22.007404601425403, -20.523100674378377, -19.514895592398908
LBFGS Accumulation History: 3 points
Removed measurement 608cc573 to history. Total: 3
th(0)=-27.27394409452452;dx=-5.602924950884547E-7
New Minimum: -27.27394409452452 > -28.008454458985398
WOLF (strong): th(6515010.502656417)=-28.008454458985398; dx=1.8954033105944856E-7 evalInputDelta=0.7345103644608777
New Minimum: -28.008454458985398 > -28.500399624511726
END: th(3257505.2513282085)=-28.500399624511726; dx=-2.4034776505722124E-8 evalInputDelta=1.2264555299872058
Fitness changed from -27.27394409452452 to -28.500399624511726
Iteration 7 complete. Error: -28.500399624511726 Total: 11.3703; Orientation: 1.0525; Line Search: 7.7286
Adding measurement 2de38e18 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.057e+04, gradient 6.560e-04, dot -0.514; [01de5fb3-4c75-4284-a1e8-c29b945b681c = 1.000/1.000e+00]
Orientation rejected. Popping history element from -28.500399624511726, -22.007404601425403, -20.523100674378377, -19.514895592398908
LBFGS Accumulation History: 3 points
Removed measurement 2de38e18 to history. Total: 3
th(0)=-28.500399624511726;dx=-4.3003729933110656E-7
New Minimum: -28.500399624511726 > -28.622602537322695
WOLF (strong): th(7018082.316422523)=-28.622602537322695; dx=8.793139769908856E-8 evalInputDelta=0.12220291281096962
New Minimum: -28.622602537322695 > -29.01035903299352
WOLF (strong): th(3509041.1582112615)=-29.01035903299352; dx=7.762603137805243E-8 evalInputDelta=0.5099594084817944
END: th(1169680.3860704205)=-28.949572019399174; dx=-2.30196255654717E-7 evalInputDelta=0.449172394887448
Fitness changed from -28.500399624511726 to -29.01035903299352
Iteration 8 complete. Error: -29.01035903299352 Total: 14.2210; Orientation: 1.1783; Line Search: 10.5490
Adding measurement 7bf1bbaf to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.072e+04, gradient 7.216e-04, dot -0.453; [01de5fb3-4c75-4284-a1e8-c29b945b681c = 1.000/1.000e+00]
Orientation rejected. Popping history element from -29.01035903299352, -22.007404601425403, -20.523100674378377, -19.514895592398908
LBFGS Accumulation History: 3 points
Removed measurement 7bf1bbaf to history. Total: 3
th(0)=-29.01035903299352;dx=-5.204782685078116E-7
New Minimum: -29.01035903299352 > -29.765344698782677
WOLF (strong): th(2520000.0000000005)=-29.765344698782677; dx=6.973944686890766E-8 evalInputDelta=0.7549856657891567
END: th(1260000.0000000002)=-29.62177713249477; dx=-3.0408273386523687E-7 evalInputDelta=0.6114180995012504
Fitness changed from -29.01035903299352 to -29.765344698782677
Iteration 9 complete. Error: -29.765344698782677 Total: 11.4482; Orientation: 1.0423; Line Search: 7.7645
Adding measurement 62846a9f to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.524e+04, gradient 6.550e-04, dot -0.502; [01de5fb3-4c75-4284-a1e8-c29b945b681c = 1.000/1.000e+00]
Orientation rejected. Popping history element from -29.765344698782677, -22.007404601425403, -20.523100674378377, -19.514895592398908
LBFGS Accumulation History: 3 points
Removed measurement 62846a9f to history. Total: 3
th(0)=-29.765344698782677;dx=-4.2861251004314576E-7
New Minimum: -29.765344698782677 > -30.09823636177052
WOLF (strong): th(2714587.709440174)=-30.09823636177052; dx=1.2062670357226972E-7 evalInputDelta=0.33289166298784423
New Minimum: -30.09823636177052 > -30.17317533766188
END: th(1357293.854720087)=-30.17317533766188; dx=-7.783466783547597E-8 evalInputDelta=0.40783063887920434
Fitness changed from -29.765344698782677 to -30.17317533766188
Iteration 10 complete. Error: -30.17317533766188 Total: 11.7017; Orientation: 1.1268; Line Search: 8.0502
<a id="p-13"></a>Iteration 10
<a id="p-12"></a>
Final threshold in iteration 11: -30.17317533766188 (> -Infinity) after 135.903s (< 5400.000s)
Returns
-30.17317533766188