BasicOptimizer.scala:89 executed in 416.92 seconds (4.961 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 12305840022050
Reset training subject: 12307068222329
Adding measurement 4c1557c6 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD+Trust
th(0)=-1.3969462451218189;dx=-1.735016217797405E-4
New Minimum: -1.3969462451218189 > -1.3973614266058216
WOLFE (weak): th(2.154434690031884)=-1.3973614266058216; dx=-1.7324806709406352E-4 evalInputDelta=4.15181484002769E-4
New Minimum: -1.3973614266058216 > -1.3977761862711564
WOLFE (weak): th(4.308869380063768)=-1.3977761862711564; dx=-1.7402925741013362E-4 evalInputDelta=8.29941149337543E-4
New Minimum: -1.3977761862711564 > -1.399436338238074
WOLFE (weak): th(12.926608140191302)=-1.399436338238074; dx=-1.7468549157806542E-4 evalInputDelta=0.002490093116255032
New Minimum: -1.399436338238074 > -1.4068835208995747
WOLFE (weak): th(51.70643256076521)=-1.4068835208995747; dx=-1.7431233730435213E-4 evalInputDelta=0.009937275777755827
New Minimum: -1.4068835208995747 > -1.446379314236499
WOLFE (weak): th(258.53216280382605)=-1.446379314236499; dx=-1.7181192626177663E-4 evalInputDelta=0.04943306911468004
New Minimum: -1.446379314236499 > -1.6862857123142856
WOLFE (weak): th(1551.1929768229563)=-1.6862857123142856; dx=-1.6431561448971736E-4 evalInputDelta=0.2893394671924667
New Minimum: -1.6862857123142856 > -2.6836949402944916
END: th(10858.350837760694)=-2.6836949402944916; dx=-6.636530560992369E-5 evalInputDelta=1.2867486951726728
Fitness changed from -1.3969462451218189 to -2.6836949402944916
Iteration 1 complete. Error: -2.6836949402944916 Total: 13.7722; Orientation: 0.0182; Line Search: 10.2027
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 5ff44a30 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-2.6836949402944916;dx=-1.636391068998127E-4
New Minimum: -2.6836949402944916 > -2.81227924752149
WOLF (strong): th(23393.607721408407)=-2.81227924752149; dx=3.660758015493802E-6 evalInputDelta=0.12858430722699854
New Minimum: -2.81227924752149 > -3.3519871582750316
END: th(11696.803860704204)=-3.3519871582750316; dx=-2.768675808890074E-5 evalInputDelta=0.66829221798054
Fitness changed from -2.6836949402944916 to -3.3519871582750316
Iteration 2 complete. Error: -3.3519871582750316 Total: 4.9635; Orientation: 0.0169; Line Search: 3.6306
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 294bbe4e to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-3.3519871582750316;dx=-2.1253751808166744E-4
New Minimum: -3.3519871582750316 > -4.1831166113737
END: th(25200.000000000004)=-4.1831166113737; dx=-7.355702562303793E-6 evalInputDelta=0.8311294530986686
Fitness changed from -3.3519871582750316 to -4.1831166113737
Iteration 3 complete. Error: -4.1831166113737 Total: 3.7461; Orientation: 0.0180; Line Search: 2.4233
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 34e800d to history. Total: 3
Rejected: LBFGS Orientation magnitude: 9.727e+02, gradient 1.589e-02, dot -0.374; [032a3daa-d91f-46c0-8468-55a45c0cbcbc = 1.000/1.000e+00]
Orientation rejected. Popping history element from -4.1831166113737, -3.3519871582750316, -2.6836949402944916, -1.3969462451218189
LBFGS Accumulation History: 3 points
Removed measurement 34e800d to history. Total: 3
th(0)=-4.1831166113737;dx=-2.4577579374628257E-4
Armijo: th(54291.75418880348)=-3.416973487909679; dx=1.3788463345515878E-5 evalInputDelta=-0.7661431234640212
New Minimum: -4.1831166113737 > -4.698247160638347
WOLF (strong): th(27145.87709440174)=-4.698247160638347; dx=1.9497886686165846E-5 evalInputDelta=0.5151305492646472
New Minimum: -4.698247160638347 > -5.509251356549595
END: th(9048.625698133914)=-5.509251356549595; dx=-6.960237228068928E-5 evalInputDelta=1.326134745175895
Fitness changed from -4.1831166113737 to -5.509251356549595
Iteration 4 complete. Error: -5.509251356549595 Total: 6.5846; Orientation: 0.4686; Line Search: 4.8256
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 6789a6fc to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.251e+03, gradient 1.387e-02, dot -0.411; [032a3daa-d91f-46c0-8468-55a45c0cbcbc = 1.000/1.000e+00]
Orientation rejected. Popping history element from -5.509251356549595, -3.3519871582750316, -2.6836949402944916, -1.3969462451218189
LBFGS Accumulation History: 3 points
Removed measurement 6789a6fc to history. Total: 3
th(0)=-5.509251356549595;dx=-1.8819226216632685E-4
New Minimum: -5.509251356549595 > -6.184792899362743
END: th(19494.673101173677)=-6.184792899362743; dx=-1.9997116849408003E-5 evalInputDelta=0.6755415428131482
Fitness changed from -5.509251356549595 to -6.184792899362743
Iteration 5 complete. Error: -6.184792899362743 Total: 4.2056; Orientation: 0.4806; Line Search: 2.4310
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 459d082d to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.017e+03, gradient 1.938e-02, dot -0.317; [032a3daa-d91f-46c0-8468-55a45c0cbcbc = 1.000/1.000e+00]
Orientation rejected. Popping history element from -6.184792899362743, -3.3519871582750316, -2.6836949402944916, -1.3969462451218189
LBFGS Accumulation History: 3 points
Removed measurement 459d082d to history. Total: 3
th(0)=-6.184792899362743;dx=-3.725923835183905E-4
Armijo: th(42000.000000000015)=-3.861187466812461; dx=2.39974610007342E-5 evalInputDelta=-2.3236054325502824
Armijo: th(21000.000000000007)=-5.808340573773827; dx=6.243289443014872E-5 evalInputDelta=-0.3764523255889163
New Minimum: -6.184792899362743 > -7.562841486511379
END: th(7000.000000000003)=-7.562841486511379; dx=-3.332784427674099E-5 evalInputDelta=1.378048587148636
Fitness changed from -6.184792899362743 to -7.562841486511379
Iteration 6 complete. Error: -7.562841486511379 Total: 6.9736; Orientation: 0.4708; Line Search: 5.1972
Adding measurement 3b8eb588 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.026e+03, gradient 1.593e-02, dot -0.346; [032a3daa-d91f-46c0-8468-55a45c0cbcbc = 1.000/1.000e+00]
Orientation rejected. Popping history element from -7.562841486511379, -3.3519871582750316, -2.6836949402944916, -1.3969462451218189
LBFGS Accumulation History: 3 points
Removed measurement 3b8eb588 to history. Total: 3
th(0)=-7.562841486511379;dx=-2.5132425705594433E-4
New Minimum: -7.562841486511379 > -7.964958695222566
END: th(15081.042830223192)=-7.964958695222566; dx=-9.71399357637566E-6 evalInputDelta=0.4021172087111866
Fitness changed from -7.562841486511379 to -7.964958695222566
Iteration 7 complete. Error: -7.964958695222566 Total: 4.0753; Orientation: 0.4885; Line Search: 2.4298
Adding measurement 7a47edd8 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.040e+03, gradient 1.737e-02, dot -0.336; [032a3daa-d91f-46c0-8468-55a45c0cbcbc = 1.000/1.000e+00]
Orientation rejected. Popping history element from -7.964958695222566, -3.3519871582750316, -2.6836949402944916, -1.3969462451218189
LBFGS Accumulation History: 3 points
Removed measurement 7a47edd8 to history. Total: 3
th(0)=-7.964958695222566;dx=-2.9857130755939546E-4
Armijo: th(32491.12183528947)=-5.510022135446052; dx=5.2272729617319575E-5 evalInputDelta=-2.454936559776514
New Minimum: -7.964958695222566 > -8.191255829120799
WOLF (strong): th(16245.560917644734)=-8.191255829120799; dx=7.78755233621647E-5 evalInputDelta=0.2262971338982327
New Minimum: -8.191255829120799 > -9.051950352039126
END: th(5415.186972548245)=-9.051950352039126; dx=-1.091149573650674E-4 evalInputDelta=1.0869916568165605
Fitness changed from -7.964958695222566 to -9.051950352039126
Iteration 8 complete. Error: -9.051950352039126 Total: 6.4428; Orientation: 0.4671; Line Search: 4.8331
Adding measurement 5967050d to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.556e+03, gradient 1.408e-02, dot -0.422; [032a3daa-d91f-46c0-8468-55a45c0cbcbc = 1.000/1.000e+00]
Orientation rejected. Popping history element from -9.051950352039126,
...skipping 50242 bytes...
ment 63d730a2 to history. Total: 3
th(0)=-96.57827541041515;dx=-5.808697953017896E-4
New Minimum: -96.57827541041515 > -101.42454091589471
END: th(20658.137954906113)=-101.42454091589471; dx=-3.6130037115033296E-4 evalInputDelta=4.846265505479565
Fitness changed from -96.57827541041515 to -101.42454091589471
Iteration 59 complete. Error: -101.42454091589471 Total: 4.0734; Orientation: 0.5085; Line Search: 2.4258
Adding measurement 779f11ea to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.448e+05, gradient 2.384e-02, dot -0.119; [032a3daa-d91f-46c0-8468-55a45c0cbcbc = 1.000/1.000e+00]
Orientation rejected. Popping history element from -101.42454091589471, -3.3519871582750316, -2.6836949402944916, -1.3969462451218189
LBFGS Accumulation History: 3 points
Removed measurement 779f11ea to history. Total: 3
th(0)=-101.42454091589471;dx=-5.652376263482084E-4
New Minimum: -101.42454091589471 > -108.1003494299394
END: th(44506.60904151404)=-108.1003494299394; dx=-2.4313510445782657E-4 evalInputDelta=6.6758085140446894
Fitness changed from -101.42454091589471 to -108.1003494299394
Iteration 60 complete. Error: -108.1003494299394 Total: 4.0654; Orientation: 0.4650; Line Search: 2.4359
<a id="p-33"></a>Iteration 60
<a id="p-32"></a>
Adding measurement 1f0e56c2 to history. Total: 3
Accepted: LBFGS Orientation magnitude: 3.170e+04, gradient 3.002e-02, dot 0.146; [032a3daa-d91f-46c0-8468-55a45c0cbcbc = 1.000/1.000e+00]
th(0)=-108.1003494299394;dx=-138.5844643257215
Armijo: th(95886.58245472454)=-45.73685454074482; dx=4.670387047611198E-4 evalInputDelta=-62.36349488919458
Armijo: th(47943.29122736227)=-45.73053398202617; dx=9.328999271353833E-4 evalInputDelta=-62.36981544791323
Armijo: th(15981.097075787424)=-45.74493013233816; dx=0.002794724888491931 evalInputDelta=-62.35541929760124
Armijo: th(3995.274268946856)=-45.76254684083235; dx=0.011184110413276431 evalInputDelta=-62.33780258910705
Armijo: th(799.0548537893712)=-45.8398169538162; dx=0.05678200406675413 evalInputDelta=-62.2605324761232
Armijo: th(133.1758089648952)=-46.12393079722824; dx=0.34691084665363314 evalInputDelta=-61.97641863271116
Armijo: th(19.0251155664136)=-48.52824664724917; dx=2.3817299907637146 evalInputDelta=-59.57210278269023
Armijo: th(2.3781394458017)=-68.9971410186683; dx=12.207243961543263 evalInputDelta=-39.1032084112711
Armijo: th(0.2642377162001889)=-103.29819263453983; dx=-5.691524810096357 evalInputDelta=-4.8021567953995685
New Minimum: -108.1003494299394 > -108.816789835025
END: th(0.02642377162001889)=-108.816789835025; dx=-61.3659855112262 evalInputDelta=0.7164404050856064
Fitness changed from -108.1003494299394 to -108.816789835025
Iteration 61 complete. Error: -108.816789835025 Total: 15.7697; Orientation: 0.4575; Line Search: 13.9950
Adding measurement 5da4955a to history. Total: 4
Accepted: LBFGS Orientation magnitude: 6.756e+03, gradient 2.960e-02, dot 0.244; [032a3daa-d91f-46c0-8468-55a45c0cbcbc = 1.000/1.000e+00]
th(0)=-108.816789835025;dx=-49.05897445605776
New Minimum: -108.816789835025 > -109.053281308095
END: th(0.05692829021964868)=-109.053281308095; dx=-4.650518981080053 evalInputDelta=0.2364914730699894
Fitness changed from -108.816789835025 to -109.053281308095
Iteration 62 complete. Error: -109.053281308095 Total: 4.2334; Orientation: 0.5857; Line Search: 2.4767
Adding measurement 7a0f2834 to history. Total: 5
Rejected: LBFGS Orientation magnitude: 2.206e+07, gradient 2.903e-02, dot -0.098; [032a3daa-d91f-46c0-8468-55a45c0cbcbc = 1.000/1.000e+00]
Orientation rejected. Popping history element from -109.053281308095, -108.816789835025, -108.1003494299394, -3.3519871582750316, -2.6836949402944916, -1.3969462451218189
Rejected: LBFGS Orientation magnitude: 1.877e+07, gradient 2.903e-02, dot -0.099; [032a3daa-d91f-46c0-8468-55a45c0cbcbc = 1.000/1.000e+00]
Orientation rejected. Popping history element from -109.053281308095, -108.816789835025, -108.1003494299394, -3.3519871582750316, -2.6836949402944916
Accepted: LBFGS Orientation magnitude: 3.425e+07, gradient 2.903e-02, dot 0.099; [032a3daa-d91f-46c0-8468-55a45c0cbcbc = 1.000/1.000e+00]
Overwriting history with 4 points
th(0)=-109.053281308095;dx=-97853.53009420236
Armijo: th(0.12264828329341393)=-43.00626203494963; dx=258.59260089443524 evalInputDelta=-66.04701927314537
Armijo: th(0.061324141646706964)=-43.20624706169526; dx=550.2028157833442 evalInputDelta=-65.84703424639974
Armijo: th(0.02044138054890232)=-44.04803064730874; dx=1724.557349212696 evalInputDelta=-65.00525066078626
Armijo: th(0.00511034513722558)=-52.00574637720336; dx=5964.791428895476 evalInputDelta=-57.047534930891636
Armijo: th(0.001022069027445116)=-79.62087288166025; dx=13388.70641157751 evalInputDelta=-29.432408426434748
Armijo: th(1.7034483790751934E-4)=-103.79433822919967; dx=-4701.763082280258 evalInputDelta=-5.258943078895328
Armijo: th(2.4334976843931334E-5)=-108.94762933400617; dx=-41145.91378776433 evalInputDelta=-0.10565197408882909
New Minimum: -109.053281308095 > -109.05989430984701
END: th(3.0418721054914167E-6)=-109.05989430984701; dx=-48816.523776034635 evalInputDelta=0.00661300175201518
Fitness changed from -109.053281308095 to -109.05989430984701
Iteration 63 complete. Error: -109.05989430984701 Total: 14.2346; Orientation: 1.9750; Line Search: 11.0850
Adding measurement 487ddf6f to history. Total: 4
Accepted: LBFGS Orientation magnitude: 3.202e+09, gradient 2.908e-02, dot 0.099; [032a3daa-d91f-46c0-8468-55a45c0cbcbc = 1.000/1.000e+00]
th(0)=-109.05989430984701;dx=-9187292.69650142
Armijo: th(6.553514786711034E-6)=-87.64354854501556; dx=966614.5374189966 evalInputDelta=-21.416345764831448
Armijo: th(3.276757393355517E-6)=-97.91506703037332; dx=488146.23850764695 evalInputDelta=-11.144827279473688
Armijo: th(1.092252464451839E-6)=-106.69083181674984; dx=-1743091.3470240706 evalInputDelta=-2.369062493097175
Armijo: th(2.7306311611295974E-7)=-108.91090825762129; dx=-3834612.028615083 evalInputDelta=-0.14898605222572314
New Minimum: -109.05989430984701 > -109.0614523588651
END: th(5.461262322259195E-8)=-109.0614523588651; dx=-4581654.2258534 evalInputDelta=0.001558049018086649
Fitness changed from -109.05989430984701 to -109.0614523588651
Iteration 64 complete. Error: -109.0614523588651 Total: 9.0696; Orientation: 0.5819; Line Search: 7.3410
Adding measurement 44f512cb to history. Total: 5
Accepted: LBFGS Orientation magnitude: 3.088e+09, gradient 2.901e-02, dot 0.097; [032a3daa-d91f-46c0-8468-55a45c0cbcbc = 1.000/1.000e+00]
th(0)=-109.0614523588651;dx=-8551294.109731238
Armijo: th(1.1765932998439294E-7)=-109.01904130557881; dx=-4115936.5631284313 evalInputDelta=-0.042411053286286915
Armijo: th(5.882966499219647E-8)=-109.04982017981506; dx=-4219617.416763613 evalInputDelta=-0.011632179050039326
Armijo: th(1.9609888330732156E-8)=-109.05824221631532; dx=-4199692.855945803 evalInputDelta=-0.0032101425497756964
Armijo: th(4.902472082683039E-9)=-109.06079869733212; dx=-4471895.722198516 evalInputDelta=-6.536615329792994E-4
Armijo: th(9.804944165366078E-10)=-109.06135386217247; dx=-4462550.67474876 evalInputDelta=-9.84966926296238E-5
Armijo: th(1.6341573608943465E-10)=-109.061421019018; dx=-4459811.961851288 evalInputDelta=-3.133984709791093E-5
MIN ALPHA (2.3345105155633522E-11): th(0.0)=-109.0614523588651
Fitness changed from -109.0614523588651 to -109.0614523588651
Static Iteration Total: 12.0531; Orientation: 0.7957; Line Search: 9.7509
Iteration 65 failed. Error: -109.0614523588651
Previous Error: 0.0 -> -109.0614523588651
Retrying iteration 65
Reset training subject: 12718016999228
Adding measurement 38698bbc to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-109.0614523588651;dx=-8.43083268775657E-4
END: th(2.1544346900318838E-10)=-109.0614523588651; dx=-7.042406435755548E-4 evalInputDelta=0.0
Fitness changed from -109.0614523588651 to -109.0614523588651
Static Iteration Total: 4.7400; Orientation: 0.0183; Line Search: 2.4234
Iteration 66 failed. Error: -109.0614523588651
Previous Error: 0.0 -> -109.0614523588651
Optimization terminated 66
Final threshold in iteration 66: -109.0614523588651 (> -Infinity) after 416.917s (< 3600.000s)
Returns
-109.0614523588651