BasicOptimizer.scala:89 executed in 157.68 seconds (2.941 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 1999582502072
Reset training subject: 2000532382883
Adding measurement cd9456d to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD+Trust
th(0)=-0.29859455180397354;dx=-6.49427009844762E-4
New Minimum: -0.29859455180397354 > -0.2999975472588409
WOLFE (weak): th(2.154434690031884)=-0.2999975472588409; dx=-6.523690555376736E-4 evalInputDelta=0.001402995454867384
New Minimum: -0.2999975472588409 > -0.30140654479349055
WOLFE (weak): th(4.308869380063768)=-0.30140654479349055; dx=-6.548404750874655E-4 evalInputDelta=0.0028119929895170093
New Minimum: -0.30140654479349055 > -0.30709648264411255
WOLFE (weak): th(12.926608140191302)=-0.30709648264411255; dx=-6.63583592390049E-4 evalInputDelta=0.008501930840139005
New Minimum: -0.30709648264411255 > -0.33363796137704643
WOLFE (weak): th(51.70643256076521)=-0.33363796137704643; dx=-7.017527668770741E-4 evalInputDelta=0.03504340957307289
New Minimum: -0.33363796137704643 > -0.4934273196541844
WOLFE (weak): th(258.53216280382605)=-0.4934273196541844; dx=-8.203974149254343E-4 evalInputDelta=0.19483276785021086
New Minimum: -0.4934273196541844 > -1.7662627157197688
WOLFE (weak): th(1551.1929768229563)=-1.7662627157197688; dx=-8.516642639450816E-4 evalInputDelta=1.4676681639157954
New Minimum: -1.7662627157197688 > -5.572229170654941
END: th(10858.350837760694)=-5.572229170654941; dx=-1.8224506221767777E-4 evalInputDelta=5.273634618850967
Fitness changed from -0.29859455180397354 to -5.572229170654941
Iteration 1 complete. Error: -5.572229170654941 Total: 11.2880; Orientation: 0.0225; Line Search: 8.5718
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 46c2aed to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-5.572229170654941;dx=-0.003802761750440118
New Minimum: -5.572229170654941 > -15.557742452545181
END: th(23393.607721408407)=-15.557742452545181; dx=-5.658968172947316E-6 evalInputDelta=9.98551328189024
Fitness changed from -5.572229170654941 to -15.557742452545181
Iteration 2 complete. Error: -15.557742452545181 Total: 2.9027; Orientation: 0.0221; Line Search: 1.8777
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 129b805b to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-15.557742452545181;dx=-0.0055555187709007275
New Minimum: -15.557742452545181 > -25.34452786645349
WOLF (strong): th(50400.00000000001)=-25.34452786645349; dx=9.916201172749884E-5 evalInputDelta=9.786785413908309
New Minimum: -25.34452786645349 > -31.374971792504752
WOLF (strong): th(25200.000000000004)=-31.374971792504752; dx=9.002555272786622E-5 evalInputDelta=15.817229339959571
New Minimum: -31.374971792504752 > -33.18544014323149
END: th(8400.000000000002)=-33.18544014323149; dx=-8.123929090944161E-4 evalInputDelta=17.627697690686308
Fitness changed from -15.557742452545181 to -33.18544014323149
Iteration 3 complete. Error: -33.18544014323149 Total: 4.6870; Orientation: 0.0204; Line Search: 3.7487
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 6b4a727e to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.021e+03, gradient 6.770e-02, dot -0.112; [7bc8874d-0753-490f-aae5-cb7ebd467d69 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -33.18544014323149, -15.557742452545181, -5.572229170654941, -0.29859455180397354
LBFGS Accumulation History: 3 points
Removed measurement 6b4a727e to history. Total: 3
th(0)=-33.18544014323149;dx=-0.004547892759763736
New Minimum: -33.18544014323149 > -48.981658616867755
END: th(18097.251396267828)=-48.981658616867755; dx=-2.1799814461225065E-4 evalInputDelta=15.796218473636266
Fitness changed from -33.18544014323149 to -48.981658616867755
Iteration 4 complete. Error: -48.981658616867755 Total: 3.2170; Orientation: 0.4464; Line Search: 1.8604
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 225b151e to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.592e+03, gradient 8.650e-02, dot -0.105; [7bc8874d-0753-490f-aae5-cb7ebd467d69 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -48.981658616867755, -15.557742452545181, -5.572229170654941, -0.29859455180397354
LBFGS Accumulation History: 3 points
Removed measurement 225b151e to history. Total: 3
th(0)=-48.981658616867755;dx=-0.007448942869224724
New Minimum: -48.981658616867755 > -50.98158560021942
WOLF (strong): th(38989.346202347355)=-50.98158560021942; dx=2.4381385380900015E-4 evalInputDelta=1.9999269833516635
New Minimum: -50.98158560021942 > -64.21355666348812
WOLF (strong): th(19494.673101173677)=-64.21355666348812; dx=2.138691361819186E-4 evalInputDelta=15.231898046620366
New Minimum: -64.21355666348812 > -68.14543128257021
END: th(6498.224367057893)=-68.14543128257021; dx=-0.0012450176501072547 evalInputDelta=19.16377266570246
Fitness changed from -48.981658616867755 to -68.14543128257021
Iteration 5 complete. Error: -68.14543128257021 Total: 5.2257; Orientation: 0.4497; Line Search: 3.8756
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 12aebf80 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.010e+03, gradient 7.211e-02, dot -0.130; [7bc8874d-0753-490f-aae5-cb7ebd467d69 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -68.14543128257021, -15.557742452545181, -5.572229170654941, -0.29859455180397354
LBFGS Accumulation History: 3 points
Removed measurement 12aebf80 to history. Total: 3
th(0)=-68.14543128257021;dx=-0.005167333010513623
New Minimum: -68.14543128257021 > -82.91680341801955
END: th(14000.000000000005)=-82.91680341801955; dx=-5.34604391453777E-5 evalInputDelta=14.77137213544934
Fitness changed from -68.14543128257021 to -82.91680341801955
Iteration 6 complete. Error: -82.91680341801955 Total: 3.2001; Orientation: 0.4377; Line Search: 1.8602
Adding measurement ec2d579 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 5.798e+03, gradient 9.682e-02, dot -0.107; [7bc8874d-0753-490f-aae5-cb7ebd467d69 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -82.91680341801955, -15.557742452545181, -5.572229170654941, -0.29859455180397354
LBFGS Accumulation History: 3 points
Removed measurement ec2d579 to history. Total: 3
th(0)=-82.91680341801955;dx=-0.009327764217201436
Armijo: th(30162.085660446384)=-76.37957420324331; dx=4.839813681956698E-4 evalInputDelta=-6.5372292147762465
New Minimum: -82.91680341801955 > -92.51649979530679
WOLF (strong): th(15081.042830223192)=-92.51649979530679; dx=7.789588178780672E-4 evalInputDelta=9.59969637728723
New Minimum: -92.51649979530679 > -102.08991900525731
END: th(5027.014276741064)=-102.08991900525731; dx=-0.001688022988699813 evalInputDelta=19.17311558723776
Fitness changed from -82.91680341801955 to -102.08991900525731
Iteration 7 complete. Error: -102.08991900525731 Total: 5.1832; Orientation: 0.5356; Line Search: 3.7711
Adding measurement 2ab99ecd to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.135e+04, gradient 7.946e-02, dot -0.148; [7bc8874d-0753-490f-aae5-cb7ebd467d69 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -102.08991900525731, -15.557742452545181, -5.572229170654941, -0.29859455180397354
LBFGS Accumulation History: 3 points
Removed measurement 2ab99ecd to history. Total: 3
th(0)=-102.08991900525731;dx=-0.006300779923539765
New Minimum: -102.08991900525731 > -117.51453949272617
END: th(10830.37394509649)=-117.51453949272617; dx=-3.704261403938164E-4 evalInputDelta=15.424620487468857
Fitness changed from -102.08991900525731 to -117.51453949272617
Iteration 8 complete. Error: -117.51453949272617 Total: 3.1504; Orientation: 0.4502; Line Search: 1.8442
Adding measurement 3d1ad24d to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.811e+04, gradient 9.968e-02, dot -0.136; [7bc8874d-0753-490f-aae5-cb7ebd467d69 = 1.000/1.000e+00]
Orientation rejected. Popp
...skipping 14151 bytes...
rientation: 1.0635; Line Search: 6.6958
<a id="p-17"></a>Iteration 20
<a id="p-16"></a>
Adding measurement 70d3909c to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.653e+04, gradient 1.080e-01, dot -0.102; [7bc8874d-0753-490f-aae5-cb7ebd467d69 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -248.3331384576976, -15.557742452545181, -5.572229170654941, -0.29859455180397354
LBFGS Accumulation History: 3 points
Removed measurement 70d3909c to history. Total: 3
th(0)=-248.3331384576976;dx=-0.011639758191409638
New Minimum: -248.3331384576976 > -256.8114276565589
END: th(1205.632716049384)=-256.8114276565589; dx=-0.009082547834104268 evalInputDelta=8.478289198861319
Fitness changed from -248.3331384576976 to -256.8114276565589
Iteration 21 complete. Error: -256.8114276565589 Total: 3.2137; Orientation: 0.4352; Line Search: 1.8670
Adding measurement 3f68d913 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.249e+04, gradient 9.863e-02, dot -0.108; [7bc8874d-0753-490f-aae5-cb7ebd467d69 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -256.8114276565589, -15.557742452545181, -5.572229170654941, -0.29859455180397354
LBFGS Accumulation History: 3 points
Removed measurement 3f68d913 to history. Total: 3
th(0)=-256.8114276565589;dx=-0.00970354461723568
New Minimum: -256.8114276565589 > -270.5906877786078
END: th(2597.456946894153)=-270.5906877786078; dx=-0.006836644503617232 evalInputDelta=13.779260122048868
Fitness changed from -256.8114276565589 to -270.5906877786078
Iteration 22 complete. Error: -270.5906877786078 Total: 3.1821; Orientation: 0.4431; Line Search: 1.8696
Adding measurement 6701e419 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.713e+04, gradient 8.998e-02, dot -0.120; [7bc8874d-0753-490f-aae5-cb7ebd467d69 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -270.5906877786078, -15.557742452545181, -5.572229170654941, -0.29859455180397354
LBFGS Accumulation History: 3 points
Removed measurement 6701e419 to history. Total: 3
th(0)=-270.5906877786078;dx=-0.008074274057805413
New Minimum: -270.5906877786078 > -289.8301765689694
END: th(5596.051352253068)=-289.8301765689694; dx=-0.004627581475613054 evalInputDelta=19.239488790361634
Fitness changed from -270.5906877786078 to -289.8301765689694
Iteration 23 complete. Error: -289.8301765689694 Total: 3.1864; Orientation: 0.4406; Line Search: 1.8720
Adding measurement 690c93f9 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.742e+05, gradient 8.642e-02, dot -0.135; [7bc8874d-0753-490f-aae5-cb7ebd467d69 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -289.8301765689694, -15.557742452545181, -5.572229170654941, -0.29859455180397354
LBFGS Accumulation History: 3 points
Removed measurement 690c93f9 to history. Total: 3
th(0)=-289.8301765689694;dx=-0.007447984250824325
New Minimum: -289.8301765689694 > -312.909207079306
END: th(12056.327160493844)=-312.909207079306; dx=-0.0021872368628772744 evalInputDelta=23.079030510336565
Fitness changed from -289.8301765689694 to -312.909207079306
Iteration 24 complete. Error: -312.909207079306 Total: 3.1978; Orientation: 0.4458; Line Search: 1.8860
Adding measurement 246e01b2 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.155e+05, gradient 1.069e-01, dot -0.111; [7bc8874d-0753-490f-aae5-cb7ebd467d69 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -312.909207079306, -15.557742452545181, -5.572229170654941, -0.29859455180397354
LBFGS Accumulation History: 3 points
Removed measurement 246e01b2 to history. Total: 3
th(0)=-312.909207079306;dx=-0.011409871613990489
Armijo: th(25974.569468941536)=-291.3091678097344; dx=7.332531817384113E-4 evalInputDelta=-21.600039269571596
New Minimum: -312.909207079306 > -319.21475921391857
WOLF (strong): th(12987.284734470768)=-319.21475921391857; dx=6.813099651264507E-4 evalInputDelta=6.305552134612583
New Minimum: -319.21475921391857 > -329.805103432315
END: th(4329.094911490256)=-329.805103432315; dx=-0.002973828082670602 evalInputDelta=16.895896353008993
Fitness changed from -312.909207079306 to -329.805103432315
Iteration 25 complete. Error: -329.805103432315 Total: 6.0913; Orientation: 0.4374; Line Search: 4.7970
<a id="p-19"></a>Iteration 25
<a id="p-18"></a>
Adding measurement 5d39e14d to history. Total: 3
Accepted: LBFGS Orientation magnitude: 2.227e+04, gradient 9.034e-02, dot 0.197; [7bc8874d-0753-490f-aae5-cb7ebd467d69 = 1.000/1.000e+00]
th(0)=-329.805103432315;dx=-395.48818496964543
Armijo: th(9326.752253755116)=-87.69104423076082; dx=0.010235743374335134 evalInputDelta=-242.11405920155414
Armijo: th(4663.376126877558)=-87.71246804634589; dx=0.02039766042005372 evalInputDelta=-242.0926353859691
Armijo: th(1554.458708959186)=-87.9290263330318; dx=0.0609355194970429 evalInputDelta=-241.87607709928318
Armijo: th(388.6146772397965)=-88.69661686755589; dx=0.24182236019239237 evalInputDelta=-241.1084865647591
Armijo: th(77.7229354479593)=-89.94251365021299; dx=1.191272231577115 evalInputDelta=-239.862589782102
Armijo: th(12.953822574659883)=-104.09184083358791; dx=7.595735068544808 evalInputDelta=-225.71326259872706
Armijo: th(1.850546082094269)=-220.21542187424114; dx=31.66801936873097 evalInputDelta=-109.58968155807383
New Minimum: -329.805103432315 > -338.3995535051723
END: th(0.23131826026178362)=-338.3995535051723; dx=-51.27087587031652 evalInputDelta=8.594450072857342
Fitness changed from -329.805103432315 to -338.3995535051723
Iteration 26 complete. Error: -338.3995535051723 Total: 10.2735; Orientation: 0.4870; Line Search: 8.7975
Adding measurement 3ab32de6 to history. Total: 4
Accepted: LBFGS Orientation magnitude: 9.512e+03, gradient 1.056e-01, dot 0.105; [7bc8874d-0753-490f-aae5-cb7ebd467d69 = 1.000/1.000e+00]
th(0)=-338.3995535051723;dx=-105.30426030137743
Armijo: th(0.4983600843458104)=-311.03011811559344; dx=17.808781075878247 evalInputDelta=-27.369435389578882
Armijo: th(0.2491800421729052)=-324.4756346784763; dx=17.88771223647051 evalInputDelta=-13.923918826696024
Armijo: th(0.08306001405763506)=-333.76124746965456; dx=17.80134104852418 evalInputDelta=-4.638306035517758
Armijo: th(0.020765003514408766)=-337.25319679730245; dx=16.102388399348257 evalInputDelta=-1.1463567078698702
Armijo: th(0.004153000702881753)=-338.1710573992103; dx=15.493136787974255 evalInputDelta=-0.22849610596199454
Armijo: th(6.921667838136256E-4)=-338.36174706222334; dx=15.424414594412038 evalInputDelta=-0.03780644294897684
Armijo: th(9.888096911623223E-5)=-338.39417428126626; dx=15.303370501303853 evalInputDelta=-0.00537922390606127
Armijo: th(1.2360121139529028E-5)=-338.39888399269284; dx=15.384772809570006 evalInputDelta=-6.695124794759977E-4
Armijo: th(1.373346793281003E-6)=-338.39949578863144; dx=15.30574986795542 evalInputDelta=-5.7716540879937384E-5
Armijo: th(1.373346793281003E-7)=-338.3995535051723; dx=15.326404139300951 evalInputDelta=0.0
Armijo: th(1.2484970848009119E-8)=-338.3995535051723; dx=15.345080007239748 evalInputDelta=0.0
Armijo: th(1.0404142373340932E-9)=-338.3995535051723; dx=15.43624422131819 evalInputDelta=0.0
MIN ALPHA (8.003186441031486E-11): th(0.0)=-338.3995535051723
Fitness changed from -338.3995535051723 to -338.3995535051723
Static Iteration Total: 14.6904; Orientation: 0.6127; Line Search: 13.1498
Iteration 27 failed. Error: -338.3995535051723
Previous Error: 0.0 -> -338.3995535051723
Retrying iteration 27
Reset training subject: 2153677647720
Adding measurement 13e85ee6 to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-338.3995535051723;dx=-0.011122736393385946
END: th(2.1544346900318838E-10)=-338.3995535051723; dx=-0.009426276372216512 evalInputDelta=0.0
Fitness changed from -338.3995535051723 to -338.3995535051723
Static Iteration Total: 3.5847; Orientation: 0.0194; Line Search: 1.8340
Iteration 28 failed. Error: -338.3995535051723
Previous Error: 0.0 -> -338.3995535051723
Optimization terminated 28
Final threshold in iteration 28: -338.3995535051723 (> -Infinity) after 157.680s (< 3600.000s)
Returns
-338.3995535051723