BasicOptimizer.scala:89 executed in 510.61 seconds (10.136 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 3241856353332
Reset training subject: 3242974503163
Adding measurement 6560e33d to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD+Trust
th(0)=-7.011383287924372;dx=-8.307240762431634E-4
New Minimum: -7.011383287924372 > -7.013196997025506
WOLFE (weak): th(2.154434690031884)=-7.013196997025506; dx=-8.291503894690584E-4 evalInputDelta=0.0018137091011345419
New Minimum: -7.013196997025506 > -7.015013162687016
WOLFE (weak): th(4.308869380063768)=-7.015013162687016; dx=-8.287025245653355E-4 evalInputDelta=0.003629874762643759
New Minimum: -7.015013162687016 > -7.0222763163398065
WOLFE (weak): th(12.926608140191302)=-7.0222763163398065; dx=-8.27827357081168E-4 evalInputDelta=0.010893028415434713
New Minimum: -7.0222763163398065 > -7.055005221993279
WOLFE (weak): th(51.70643256076521)=-7.055005221993279; dx=-8.204580965378925E-4 evalInputDelta=0.043621934068907464
New Minimum: -7.055005221993279 > -7.228811787675312
WOLFE (weak): th(258.53216280382605)=-7.228811787675312; dx=-8.324521275443992E-4 evalInputDelta=0.21742849975094014
New Minimum: -7.228811787675312 > -8.357986867306611
WOLFE (weak): th(1551.1929768229563)=-8.357986867306611; dx=-7.483880025869482E-4 evalInputDelta=1.346603579382239
New Minimum: -8.357986867306611 > -12.147486291729484
END: th(10858.350837760694)=-12.147486291729484; dx=-1.1868553470179106E-4 evalInputDelta=5.136103003805112
Fitness changed from -7.011383287924372 to -12.147486291729484
Iteration 1 complete. Error: -12.147486291729484 Total: 13.7731; Orientation: 0.0205; Line Search: 10.4449
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 473f51ed to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-12.147486291729484;dx=-0.0033371169407026002
New Minimum: -12.147486291729484 > -23.609351315793262
END: th(23393.607721408407)=-23.609351315793262; dx=-8.962981312133664E-5 evalInputDelta=11.461865024063778
Fitness changed from -12.147486291729484 to -23.609351315793262
Iteration 2 complete. Error: -23.609351315793262 Total: 3.4465; Orientation: 0.0200; Line Search: 2.2996
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 6c98555a to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-23.609351315793262;dx=-0.008547100485156769
New Minimum: -23.609351315793262 > -31.388113732553304
WOLF (strong): th(50400.00000000001)=-31.388113732553304; dx=1.261979618917134E-4 evalInputDelta=7.778762416760042
New Minimum: -31.388113732553304 > -41.89445634465571
WOLF (strong): th(25200.000000000004)=-41.89445634465571; dx=7.143702349646076E-5 evalInputDelta=18.285105028862446
New Minimum: -41.89445634465571 > -47.53409291093877
END: th(8400.000000000002)=-47.53409291093877; dx=-4.917950090336898E-4 evalInputDelta=23.924741595145505
Fitness changed from -23.609351315793262 to -47.53409291093877
Iteration 3 complete. Error: -47.53409291093877 Total: 5.9001; Orientation: 0.0202; Line Search: 4.7553
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 66f8b737 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.247e+03, gradient 9.173e-02, dot -0.175; [dd12a2eb-5a1d-4a99-9e4b-36ad1770d4d0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -47.53409291093877, -23.609351315793262, -12.147486291729484, -7.011383287924372
LBFGS Accumulation History: 3 points
Removed measurement 66f8b737 to history. Total: 3
th(0)=-47.53409291093877;dx=-0.008409113060231026
New Minimum: -47.53409291093877 > -66.14106325602181
WOLF (strong): th(18097.251396267828)=-66.14106325602181; dx=1.930111549511578E-4 evalInputDelta=18.60697034508304
New Minimum: -66.14106325602181 > -71.10991030295996
END: th(9048.625698133914)=-71.10991030295996; dx=-8.823804494211313E-4 evalInputDelta=23.575817392021193
Fitness changed from -47.53409291093877 to -71.10991030295996
Iteration 4 complete. Error: -71.10991030295996 Total: 5.0543; Orientation: 0.4667; Line Search: 3.4677
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 4dd82f85 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 6.891e+03, gradient 1.241e-01, dot -0.146; [dd12a2eb-5a1d-4a99-9e4b-36ad1770d4d0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -71.10991030295996, -23.609351315793262, -12.147486291729484, -7.011383287924372
LBFGS Accumulation History: 3 points
Removed measurement 4dd82f85 to history. Total: 3
th(0)=-71.10991030295996;dx=-0.01540640359092198
New Minimum: -71.10991030295996 > -88.1114837058568
WOLF (strong): th(19494.673101173677)=-88.1114837058568; dx=6.471005240055212E-4 evalInputDelta=17.001573402896838
New Minimum: -88.1114837058568 > -104.55382634518381
END: th(9747.336550586839)=-104.55382634518381; dx=-6.911575266616228E-5 evalInputDelta=33.44391604222385
Fitness changed from -71.10991030295996 to -104.55382634518381
Iteration 5 complete. Error: -104.55382634518381 Total: 5.0279; Orientation: 0.4644; Line Search: 3.4528
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 472019d9 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 8.491e+03, gradient 1.522e-01, dot -0.118; [dd12a2eb-5a1d-4a99-9e4b-36ad1770d4d0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -104.55382634518381, -23.609351315793262, -12.147486291729484, -7.011383287924372
LBFGS Accumulation History: 3 points
Removed measurement 472019d9 to history. Total: 3
th(0)=-104.55382634518381;dx=-0.023139056131631603
Armijo: th(21000.000000000007)=-89.98067101652737; dx=6.104703042508435E-4 evalInputDelta=-14.57315532865644
New Minimum: -104.55382634518381 > -113.08714731062447
WOLF (strong): th(10500.000000000004)=-113.08714731062447; dx=0.0012041757265188646 evalInputDelta=8.533320965440666
New Minimum: -113.08714731062447 > -134.4047642503006
END: th(3500.0000000000014)=-134.4047642503006; dx=-0.003069021851925954 evalInputDelta=29.85093790511678
Fitness changed from -104.55382634518381 to -134.4047642503006
Iteration 6 complete. Error: -134.4047642503006 Total: 6.2472; Orientation: 0.4746; Line Search: 4.6512
Adding measurement 5476ea3e to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.700e+04, gradient 1.333e-01, dot -0.140; [dd12a2eb-5a1d-4a99-9e4b-36ad1770d4d0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -134.4047642503006, -23.609351315793262, -12.147486291729484, -7.011383287924372
LBFGS Accumulation History: 3 points
Removed measurement 5476ea3e to history. Total: 3
th(0)=-134.4047642503006;dx=-0.01774739510017919
New Minimum: -134.4047642503006 > -153.70424121687608
WOLF (strong): th(7540.521415111596)=-153.70424121687608; dx=0.001172611414389956 evalInputDelta=19.299476966575497
New Minimum: -153.70424121687608 > -160.04443138624265
END: th(3770.260707555798)=-160.04443138624265; dx=-0.002057453470595883 evalInputDelta=25.639667135942062
Fitness changed from -134.4047642503006 to -160.04443138624265
Iteration 7 complete. Error: -160.04443138624265 Total: 5.0295; Orientation: 0.4747; Line Search: 3.4751
Adding measurement 4d10d683 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.837e+04, gradient 1.415e-01, dot -0.140; [dd12a2eb-5a1d-4a99-9e4b-36ad1770d4d0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -160.04443138624265, -23.609351315793262, -12.147486291729484, -7.011383287924372
LBFGS Accumulation History: 3 points
Removed measurement 4d10d683 to history. Total: 3
th(0)=-160.04443138624265;dx=-0.020003754175310536
Armijo: th(8122.780458822367)=-147.34087335768632; dx=0.0029884124861829734 evalInputDelta=-12.70355802855633
New Minimum: -160.04443138624265 > -171.14811255293353
WOLF (strong): th(4061.3902294111836)=-171.14811255293353; dx=0.002065590901930217 evalInputDelta=11.103681166690876
New Minimum: -171.14811255293353 > -175.62742121193077
END: th(1353.7967431370612)=-175.62742121193077; dx=-0.008771
...skipping 68411 bytes...
51601, -324.8148265363562, -321.9429603246978, -306.3926189031546
LBFGS Accumulation History: 3 points
Removed measurement 12596df to history. Total: 3
th(0)=-1179.848647251601;dx=-0.03327317261412123
New Minimum: -1179.848647251601 > -1184.7304799533918
END: th(337.1794143905712)=-1184.7304799533918; dx=-0.021861158508528534 evalInputDelta=4.8818327017906995
Fitness changed from -1179.848647251601 to -1184.7304799533918
Iteration 80 complete. Error: -1184.7304799533918 Total: 3.9152; Orientation: 0.4732; Line Search: 2.3076
<a id="p-41"></a>Iteration 80
<a id="p-40"></a>
Adding measurement 60800ca4 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 5.018e+06, gradient 1.773e-01, dot -0.158; [dd12a2eb-5a1d-4a99-9e4b-36ad1770d4d0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1184.7304799533918, -324.8148265363562, -321.9429603246978, -306.3926189031546
LBFGS Accumulation History: 3 points
Removed measurement 60800ca4 to history. Total: 3
th(0)=-1184.7304799533918;dx=-0.03137150260761696
New Minimum: -1184.7304799533918 > -1193.9002839902255
END: th(726.4310271276823)=-1193.9002839902255; dx=-0.019178540951293475 evalInputDelta=9.169804036833739
Fitness changed from -1184.7304799533918 to -1193.9002839902255
Iteration 81 complete. Error: -1193.9002839902255 Total: 3.8991; Orientation: 0.4633; Line Search: 2.3086
Adding measurement 6e25583 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 6.378e+05, gradient 1.675e-01, dot -0.169; [dd12a2eb-5a1d-4a99-9e4b-36ad1770d4d0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1193.9002839902255, -324.8148265363562, -321.9429603246978, -306.3926189031546
LBFGS Accumulation History: 3 points
Removed measurement 6e25583 to history. Total: 3
th(0)=-1193.9002839902255;dx=-0.0280099797361195
New Minimum: -1193.9002839902255 > -1208.831565158735
END: th(1565.0482047593712)=-1208.831565158735; dx=-0.016000060025346097 evalInputDelta=14.931281168509486
Fitness changed from -1193.9002839902255 to -1208.831565158735
Iteration 82 complete. Error: -1208.831565158735 Total: 3.8542; Orientation: 0.4730; Line Search: 2.3067
Adding measurement 50675b72 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 9.278e+04, gradient 1.573e-01, dot -0.176; [dd12a2eb-5a1d-4a99-9e4b-36ad1770d4d0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1208.831565158735, -324.8148265363562, -321.9429603246978, -306.3926189031546
LBFGS Accumulation History: 3 points
Removed measurement 50675b72 to history. Total: 3
th(0)=-1208.831565158735;dx=-0.024656813749200677
New Minimum: -1208.831565158735 > -1231.669861734357
END: th(3371.794143905712)=-1231.669861734357; dx=-0.012614106428562031 evalInputDelta=22.838296575622053
Fitness changed from -1208.831565158735 to -1231.669861734357
Iteration 83 complete. Error: -1231.669861734357 Total: 3.9568; Orientation: 0.5610; Line Search: 2.2952
Adding measurement 748f7a96 to history. Total: 3
Accepted: LBFGS Orientation magnitude: 2.438e+04, gradient 1.491e-01, dot 0.240; [dd12a2eb-5a1d-4a99-9e4b-36ad1770d4d0 = 1.000/1.000e+00]
th(0)=-1231.669861734357;dx=-872.0080466607913
Armijo: th(7264.310271276824)=-115.53733359627917; dx=0.01475595402134749 evalInputDelta=-1116.1325281380778
Armijo: th(3632.155135638412)=-115.55332211600494; dx=0.029492464940445873 evalInputDelta=-1116.1165396183521
Armijo: th(1210.7183785461373)=-115.924532431278; dx=0.09006261612889055 evalInputDelta=-1115.745329303079
Armijo: th(302.6795946365343)=-117.42435430926957; dx=0.3693391261835205 evalInputDelta=-1114.2455074250875
Armijo: th(60.53591892730687)=-122.35716000356945; dx=1.9118960935774834 evalInputDelta=-1109.3127017307875
Armijo: th(10.089319821217812)=-156.56323038765484; dx=15.474012026470344 evalInputDelta=-1075.1066313467022
Armijo: th(1.441331403031116)=-627.6631973298801; dx=190.99310336185124 evalInputDelta=-604.006664404477
Armijo: th(0.1801664253788895)=-1207.7435040123264; dx=-17.664533117802886 evalInputDelta=-23.926357722030616
New Minimum: -1231.669861734357 > -1234.822031646665
END: th(0.0200184917087655)=-1234.822031646665; dx=-305.54408373599927 evalInputDelta=3.1521699123079543
Fitness changed from -1231.669861734357 to -1234.822031646665
Iteration 84 complete. Error: -1234.822031646665 Total: 13.2149; Orientation: 0.5425; Line Search: 11.5772
Adding measurement 6a6827bd to history. Total: 4
Accepted: LBFGS Orientation magnitude: 2.045e+04, gradient 1.560e-01, dot 0.185; [dd12a2eb-5a1d-4a99-9e4b-36ad1770d4d0 = 1.000/1.000e+00]
th(0)=-1234.822031646665;dx=-589.4144067086347
New Minimum: -1234.822031646665 > -1235.420345460242
END: th(0.043128532979480036)=-1235.420345460242; dx=-242.72307635324916 evalInputDelta=0.5983138135770787
Fitness changed from -1234.822031646665 to -1235.420345460242
Iteration 85 complete. Error: -1235.420345460242 Total: 3.9524; Orientation: 0.5685; Line Search: 2.2891
<a id="p-43"></a>Iteration 85
<a id="p-42"></a>
Adding measurement 6dc4500e to history. Total: 5
Rejected: LBFGS Orientation magnitude: 6.622e+04, gradient 1.672e-01, dot -0.176; [dd12a2eb-5a1d-4a99-9e4b-36ad1770d4d0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1235.420345460242, -1234.822031646665, -1231.669861734357, -324.8148265363562, -321.9429603246978, -306.3926189031546
Rejected: LBFGS Orientation magnitude: 6.290e+04, gradient 1.672e-01, dot -0.176; [dd12a2eb-5a1d-4a99-9e4b-36ad1770d4d0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1235.420345460242, -1234.822031646665, -1231.669861734357, -324.8148265363562, -321.9429603246978
Accepted: LBFGS Orientation magnitude: 2.056e+05, gradient 1.672e-01, dot 0.168; [dd12a2eb-5a1d-4a99-9e4b-36ad1770d4d0 = 1.000/1.000e+00]
Overwriting history with 4 points
th(0)=-1235.420345460242;dx=-5774.625461041352
Armijo: th(0.09291760758117595)=-868.1677349221324; dx=1885.0634636689592 evalInputDelta=-367.2526105381097
Armijo: th(0.04645880379058798)=-1073.5465558148844; dx=1707.6338451449749 evalInputDelta=-161.87378964535765
Armijo: th(0.015486267930195993)=-1202.786942057823; dx=312.032110098234 evalInputDelta=-32.63340340241916
Armijo: th(0.0038715669825489982)=-1231.249578366587; dx=-819.6668065971553 evalInputDelta=-4.1707670936550585
Armijo: th(7.743133965097996E-4)=-1234.8897079135838; dx=-1257.7315953012935 evalInputDelta=-0.5306375466582267
Armijo: th(1.2905223275163326E-4)=-1235.3438895041165; dx=-1359.3981642538656 evalInputDelta=-0.07645595612552825
Armijo: th(1.8436033250233324E-5)=-1235.4094588168887; dx=-1381.7827024922994 evalInputDelta=-0.010886643353387626
Armijo: th(2.3045041562791655E-6)=-1235.4188114390306; dx=-1377.3816349706944 evalInputDelta=-0.0015340212114551832
Armijo: th(2.5605601736435173E-7)=-1235.4202464911891; dx=-1382.4457628414098 evalInputDelta=-9.896905294226599E-5
Armijo: th(2.5605601736435173E-8)=-1235.420345460242; dx=-1372.564170775218 evalInputDelta=0.0
Armijo: th(2.327781976039561E-9)=-1235.420345460242; dx=-1389.9121916305646 evalInputDelta=0.0
Armijo: th(1.9398183133663008E-10)=-1235.4202959757165; dx=-1378.8699194207575 evalInputDelta=-4.948452556163829E-5
MIN ALPHA (1.492167933358693E-11): th(0.0)=-1235.420345460242
Fitness changed from -1235.420345460242 to -1235.420345460242
Static Iteration Total: 21.7196; Orientation: 1.7139; Line Search: 18.8879
Iteration 86 failed. Error: -1235.420345460242
Previous Error: 0.0 -> -1235.420345460242
Retrying iteration 86
Reset training subject: 3747944527528
Adding measurement 547eb256 to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-1235.420345460242;dx=-0.027906106675625604
END: th(2.1544346900318838E-10)=-1235.420345460242; dx=-0.01846553758423966 evalInputDelta=0.0
Fitness changed from -1235.420345460242 to -1235.420345460242
Static Iteration Total: 4.5215; Orientation: 0.0191; Line Search: 2.3066
Iteration 87 failed. Error: -1235.420345460242
Previous Error: 0.0 -> -1235.420345460242
Optimization terminated 87
Final threshold in iteration 87: -1235.420345460242 (> -Infinity) after 510.610s (< 3600.000s)
Returns
-1235.420345460242