BasicOptimizer.scala:89 executed in 130.66 seconds (1.629 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 71063130375000
Reset training subject: 71065850415800
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
New Minimum: -24.451135927662463 > -24.493892427958272
th(0)=-24.493892427958272;dx=-6.772624507082928E-7
New Minimum: -24.493892427958272 > -24.49389453572591
WOLFE (weak): th(2.154434690031884)=-24.49389453572591; dx=-6.342204706312524E-7 evalInputDelta=2.1077676386482835E-6
New Minimum: -24.49389453572591 > -24.49389664349337
WOLFE (weak): th(4.308869380063768)=-24.49389664349337; dx=-6.343584093915444E-7 evalInputDelta=4.2155350961081695E-6
New Minimum: -24.49389664349337 > -24.49390191291122
WOLFE (weak): th(12.926608140191302)=-24.49390191291122; dx=-6.343519426336556E-7 evalInputDelta=9.484952947502734E-6
New Minimum: -24.49390191291122 > -24.493929313865767
WOLFE (weak): th(51.70643256076521)=-24.493929313865767; dx=-6.34299108468221E-7 evalInputDelta=3.6885907494621506E-5
New Minimum: -24.493929313865767 > -24.49407580306431
WOLFE (weak): th(258.53216280382605)=-24.49407580306431; dx=-6.34301820244821E-7 evalInputDelta=1.8337510603672058E-4
New Minimum: -24.49407580306431 > -24.494987388819453
WOLFE (weak): th(1551.1929768229563)=-24.494987388819453; dx=-6.3422687599156E-7 evalInputDelta=0.0010949608611809936
New Minimum: -24.494987388819453 > -24.5015476979543
WOLFE (weak): th(10858.350837760694)=-24.5015476979543; dx=-6.335525414648778E-7 evalInputDelta=0.007655269996028835
New Minimum: -24.5015476979543 > -24.55483858780674
WOLFE (weak): th(86866.80670208555)=-24.55483858780674; dx=-6.272464262542502E-7 evalInputDelta=0.06094615984846641
New Minimum: -24.55483858780674 > -25.009423033116796
END: th(781801.26031877)=-25.009423033116796; dx=-5.451412812489846E-7 evalInputDelta=0.5155306051585242
Fitness changed from -24.451135927662463 to -25.009423033116796
Iteration 1 complete. Error: -25.009423033116796 Total: 34.3719; Orientation: 0.0428; Line Search: 26.3300
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 676166de to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-25.009423033116796;dx=-5.387395266360357E-7
New Minimum: -25.009423033116796 > -25.899679876682928
END: th(1684339.7559414052)=-25.899679876682928; dx=-4.1510912246993336E-7 evalInputDelta=0.8902568435661316
Fitness changed from -25.009423033116796 to -25.899679876682928
Iteration 2 complete. Error: -25.899679876682928 Total: 8.2690; Orientation: 0.0552; Line Search: 5.1110
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 5d4452cc to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-25.899679876682928;dx=-4.595117644298059E-7
New Minimum: -25.899679876682928 > -27.280234493833113
END: th(3628800.0)=-27.280234493833113; dx=-2.7924664598931735E-7 evalInputDelta=1.3805546171501852
Fitness changed from -25.899679876682928 to -27.280234493833113
Iteration 3 complete. Error: -27.280234493833113 Total: 8.6408; Orientation: 0.0602; Line Search: 5.3141
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 352d8a30 to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-27.280234493833113;dx=-4.290750877706188E-7
New Minimum: -27.280234493833113 > -29.008533084671154
END: th(7818012.6031877)=-29.008533084671154; dx=-1.2436656155909896E-7 evalInputDelta=1.7282985908380404
Fitness changed from -27.280234493833113 to -29.008533084671154
Iteration 4 complete. Error: -29.008533084671154 Total: 8.3749; Orientation: 0.0612; Line Search: 5.2827
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement f2ab08f to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.847e+04, gradient 6.698e-04, dot -0.575; [acebaf88-720b-48ac-b733-8a495cf9e653 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -29.008533084671154, -27.280234493833113, -25.899679876682928, -25.009423033116796
LBFGS Accumulation History: 3 points
Removed measurement f2ab08f to history. Total: 3
th(0)=-29.008533084671154;dx=-4.484372825931748E-7
New Minimum: -29.008533084671154 > -30.735559064010502
END: th(1.684339755941405E7)=-30.735559064010502; dx=-1.5523841211992712E-8 evalInputDelta=1.7270259793393485
Fitness changed from -29.008533084671154 to -30.735559064010502
Iteration 5 complete. Error: -30.735559064010502 Total: 9.6044; Orientation: 1.1775; Line Search: 5.4405
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 31a0a437 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.044e+04, gradient 6.205e-04, dot -0.403; [acebaf88-720b-48ac-b733-8a495cf9e653 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -30.735559064010502, -27.280234493833113, -25.899679876682928, -25.009423033116796
LBFGS Accumulation History: 3 points
Removed measurement 31a0a437 to history. Total: 3
th(0)=-30.735559064010502;dx=-3.8473897408543775E-7
New Minimum: -30.735559064010502 > -33.18912345316091
WOLF (strong): th(3.6288E7)=-33.18912345316091; dx=3.6432020231045016E-8 evalInputDelta=2.453564389150408
New Minimum: -33.18912345316091 > -33.89309125143609
WOLF (strong): th(1.8144E7)=-33.89309125143609; dx=9.56990469340975E-9 evalInputDelta=3.1575321874255877
END: th(6048000.0)=-32.92421961430144; dx=-2.310059483404699E-7 evalInputDelta=2.1886605502909404
Fitness changed from -30.735559064010502 to -33.89309125143609
Iteration 6 complete. Error: -33.89309125143609 Total: 14.8856; Orientation: 1.1039; Line Search: 10.4783
Adding measurement 499ae015 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.900e+04, gradient 6.481e-04, dot -0.347; [acebaf88-720b-48ac-b733-8a495cf9e653 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -33.89309125143609, -27.280234493833113, -25.899679876682928, -25.009423033116796
LBFGS Accumulation History: 3 points
Removed measurement 499ae015 to history. Total: 3
th(0)=-33.89309125143609;dx=-4.1955016583036973E-7
New Minimum: -33.89309125143609 > -36.68540081702247
WOLF (strong): th(1.3030021005312834E7)=-36.68540081702247; dx=4.3272106404335604E-8 evalInputDelta=2.792309565586379
END: th(6515010.502656417)=-36.31202084088467; dx=-1.9258746718624982E-7 evalInputDelta=2.418929589448581
Fitness changed from -33.89309125143609 to -36.68540081702247
Iteration 7 complete. Error: -36.68540081702247 Total: 11.2054; Orientation: 1.0838; Line Search: 7.6388
Adding measurement 795ce183 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.540e+04, gradient 6.349e-04, dot -0.368; [acebaf88-720b-48ac-b733-8a495cf9e653 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -36.68540081702247, -27.280234493833113, -25.899679876682928, -25.009423033116796
LBFGS Accumulation History: 3 points
Removed measurement 795ce183 to history. Total: 3
th(0)=-36.68540081702247;dx=-4.030586497530391E-7
New Minimum: -36.68540081702247 > -38.01359833389893
WOLF (strong): th(1.4036164632845046E7)=-38.01359833389893; dx=1.2066257978932296E-7 evalInputDelta=1.3281975168764646
New Minimum: -38.01359833389893 > -38.625765766081216
END: th(7018082.316422523)=-38.625765766081216; dx=-3.205501027181327E-8 evalInputDelta=1.940364949058747
Fitness changed from -36.68540081702247 to -38.625765766081216
Iteration 8 complete. Error: -38.625765766081216 Total: 11.6162; Orientation: 1.0607; Line Search: 7.7855
Adding measurement 208fd3be to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.615e+04, gradient 5.826e-04, dot -0.440; [acebaf88-720b-48ac-b733-8a495cf9e653 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -38.625765766081216, -27.280234493833113, -25.899679876682928, -25.009423033116796
LBFGS Accumulation History: 3 points
Removed measurement 208fd3be to history. Total: 3
th(0)=-38.625765766081216;dx=-3.3921203802132904E-7
New Minimum: -38.625765766081216 > -38.75689547754809
WOLF (strong): th(1.5120000000000004E7)=-38.75689547754809; dx=5.683213936174553E-8 evalInputDelta=0.13112971146687613
New Minimum: -38.75689547754809 > -39.3496346496403
WOLF (strong): th(7560000.000000002)=-39.3496346496403; dx=6.863696457978889E-8 evalInputDelta=0.723868883559085
New Minimum: -39.3496346496403 > -39.38347672607816
END: th(2520000.0000000005)=-39.38347672607816; dx=-1.603667353594131E-7 evalInputDelta=0.7577109599969418
Fitness changed from -38.625765766081216 to -39.38347672607816
Iteration 9 complete. Error: -39.38347672607816 Total: 13.9287; Orientation: 1.0389; Line Search: 10.3976
Adding measurement 1e20a5f7 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.063e+04, gradient 4.722e-04, dot -0.553; [acebaf88-720b-48ac-b733-8a495cf9e653 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -39.38347672607816, -27.280234493833113, -25.899679876682928, -25.009423033116796
LBFGS Accumulation History: 3 points
Removed measurement 1e20a5f7 to history. Total: 3
th(0)=-39.38347672607816;dx=-2.228050897123706E-7
New Minimum: -39.38347672607816 > -39.98029560579192
END: th(5429175.418880348)=-39.98029560579192; dx=-3.302740295965752E-8 evalInputDelta=0.5968188797137657
Fitness changed from -39.38347672607816 to -39.98029560579192
Iteration 10 complete. Error: -39.98029560579192 Total: 9.2861; Orientation: 1.0984; Line Search: 5.4623
<a id="p-13"></a>Iteration 10
<a id="p-12"></a>
Final threshold in iteration 11: -39.98029560579192 (> -Infinity) after 130.660s (< 5400.000s)
Returns
-39.98029560579192