BasicOptimizer.scala:89 executed in 109.45 seconds (2.462 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 3624380559200
Reset training subject: 3625024374710
Adding measurement 18e15617 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD+Trust
th(0)=-3.60542987933762;dx=-3.779867145915218E-4
New Minimum: -3.60542987933762 > -3.605468165500296
WOLFE (weak): th(2.154434690031884)=-3.605468165500296; dx=-3.7795280590320005E-4 evalInputDelta=3.828616267576024E-5
New Minimum: -3.605468165500296 > -3.605506800896154
WOLFE (weak): th(4.308869380063768)=-3.605506800896154; dx=-3.7786260201420977E-4 evalInputDelta=7.692155853389337E-5
New Minimum: -3.605506800896154 > -3.605667456767355
WOLFE (weak): th(12.926608140191302)=-3.605667456767355; dx=-3.7824133538211983E-4 evalInputDelta=2.3757742973495866E-4
New Minimum: -3.605667456767355 > -3.606390057456482
WOLFE (weak): th(51.70643256076521)=-3.606390057456482; dx=-3.7903438336414283E-4 evalInputDelta=9.60178118861954E-4
New Minimum: -3.606390057456482 > -3.610023065712245
WOLFE (weak): th(258.53216280382605)=-3.610023065712245; dx=-3.751834921492939E-4 evalInputDelta=0.0045931863746249135
New Minimum: -3.610023065712245 > -3.6295060907393015
WOLFE (weak): th(1551.1929768229563)=-3.6295060907393015; dx=-3.581904910030546E-4 evalInputDelta=0.02407621140168148
New Minimum: -3.6295060907393015 > -3.7320856956490718
END: th(10858.350837760694)=-3.7320856956490718; dx=-2.901284827351703E-4 evalInputDelta=0.12665581631145173
Fitness changed from -3.60542987933762 to -3.7320856956490718
Iteration 1 complete. Error: -3.7320856956490718 Total: 7.2112; Orientation: 0.0165; Line Search: 5.2816
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 60249e93 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-3.7320856956490718;dx=-3.69428046264245E-4
New Minimum: -3.7320856956490718 > -3.953886086128171
END: th(23393.607721408407)=-3.953886086128171; dx=-2.319357151003377E-4 evalInputDelta=0.22180039047909927
Fitness changed from -3.7320856956490718 to -3.953886086128171
Iteration 2 complete. Error: -3.953886086128171 Total: 1.9527; Orientation: 0.0167; Line Search: 1.2910
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 727bed23 to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-3.953886086128171;dx=-4.151609923757067E-4
New Minimum: -3.953886086128171 > -3.969579295562711
END: th(50400.00000000001)=-3.969579295562711; dx=-1.360830345395096E-4 evalInputDelta=0.015693209434540112
Fitness changed from -3.953886086128171 to -3.969579295562711
Iteration 3 complete. Error: -3.969579295562711 Total: 1.9316; Orientation: 0.0161; Line Search: 1.2984
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 6e5c5b6d to history. Total: 3
Rejected: LBFGS Orientation magnitude: 7.528e+03, gradient 1.911e-02, dot -0.341; [aa02667e-838d-45b7-84b6-af509a5d7e9e = 1.000/1.000e+00]
Orientation rejected. Popping history element from -3.969579295562711, -3.953886086128171, -3.7320856956490718, -3.60542987933762
LBFGS Accumulation History: 3 points
Removed measurement 6e5c5b6d to history. Total: 3
th(0)=-3.969579295562711;dx=-3.649765028928993E-4
New Minimum: -3.969579295562711 > -4.852152846100195
END: th(108583.50837760697)=-4.852152846100195; dx=-5.287561509852817E-5 evalInputDelta=0.8825735505374834
Fitness changed from -3.969579295562711 to -4.852152846100195
Iteration 4 complete. Error: -4.852152846100195 Total: 2.6314; Orientation: 0.4294; Line Search: 1.5038
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 96e52d3 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.665e+03, gradient 1.981e-02, dot -0.158; [aa02667e-838d-45b7-84b6-af509a5d7e9e = 1.000/1.000e+00]
Orientation rejected. Popping history element from -4.852152846100195, -3.953886086128171, -3.7320856956490718, -3.60542987933762
LBFGS Accumulation History: 3 points
Removed measurement 96e52d3 to history. Total: 3
th(0)=-4.852152846100195;dx=-3.921574041979447E-4
New Minimum: -4.852152846100195 > -5.065489668096764
END: th(233936.07721408413)=-5.065489668096764; dx=-1.9033222328773734E-5 evalInputDelta=0.21333682199656945
Fitness changed from -4.852152846100195 to -5.065489668096764
Iteration 5 complete. Error: -5.065489668096764 Total: 3.1319; Orientation: 0.4604; Line Search: 1.9752
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 27021d66 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.604e+03, gradient 1.609e-02, dot -0.114; [aa02667e-838d-45b7-84b6-af509a5d7e9e = 1.000/1.000e+00]
Orientation rejected. Popping history element from -5.065489668096764, -3.953886086128171, -3.7320856956490718, -3.60542987933762
LBFGS Accumulation History: 3 points
Removed measurement 27021d66 to history. Total: 3
th(0)=-5.065489668096764;dx=-2.585621324200107E-4
New Minimum: -5.065489668096764 > -5.249429920807853
END: th(504000.0000000002)=-5.249429920807853; dx=-8.313033667205766E-6 evalInputDelta=0.18394025271108916
Fitness changed from -5.065489668096764 to -5.249429920807853
Iteration 6 complete. Error: -5.249429920807853 Total: 2.4837; Orientation: 0.4429; Line Search: 1.3702
Adding measurement 2bf93ad8 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.507e+03, gradient 1.330e-02, dot -0.083; [aa02667e-838d-45b7-84b6-af509a5d7e9e = 1.000/1.000e+00]
Orientation rejected. Popping history element from -5.249429920807853, -3.953886086128171, -3.7320856956490718, -3.60542987933762
LBFGS Accumulation History: 3 points
Removed measurement 2bf93ad8 to history. Total: 3
th(0)=-5.249429920807853;dx=-1.7671343385709668E-4
New Minimum: -5.249429920807853 > -5.632686952408214
END: th(1085835.0837760698)=-5.632686952408214; dx=-3.228839478206906E-6 evalInputDelta=0.38325703160036095
Fitness changed from -5.249429920807853 to -5.632686952408214
Iteration 7 complete. Error: -5.632686952408214 Total: 2.4395; Orientation: 0.4683; Line Search: 1.3348
Adding measurement 29a1b5f to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.960e+03, gradient 1.019e-02, dot -0.056; [aa02667e-838d-45b7-84b6-af509a5d7e9e = 1.000/1.000e+00]
Orientation rejected. Popping history element from -5.632686952408214, -3.953886086128171, -3.7320856956490718, -3.60542987933762
LBFGS Accumulation History: 3 points
Removed measurement 29a1b5f to history. Total: 3
th(0)=-5.632686952408214;dx=-1.0353493002054222E-4
Armijo: th(2339360.7721408415)=-4.673126364381081; dx=-1.3016302508906056E-6 evalInputDelta=-0.9595605880271334
New Minimum: -5.632686952408214 > -5.70500305659195
END: th(1169680.3860704207)=-5.70500305659195; dx=-5.7397351899626605E-6 evalInputDelta=0.07231610418373613
Fitness changed from -5.632686952408214 to -5.70500305659195
Iteration 8 complete. Error: -5.70500305659195 Total: 3.1004; Orientation: 0.4716; Line Search: 2.0066
Adding measurement 73a9af79 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.091e+03, gradient 8.265e-03, dot -0.056; [aa02667e-838d-45b7-84b6-af509a5d7e9e = 1.000/1.000e+00]
Orientation rejected. Popping history element from -5.70500305659195, -3.953886086128171, -3.7320856956490718, -3.60542987933762
LBFGS Accumulation History: 3 points
Removed measurement 73a9af79 to history. Total: 3
th(0)=-5.70500305659195;dx=-6.81143854679925E-5
Armijo: th(2520000.000000001)=-5.058189646439562; dx=-1.3200674893166556E-6 evalInputDelta=-0.6468134101523884
New Minimum: -5.70500305659195 > -5.931133261270278
END: th(1260000.0000000005)=-5.931133261270278; dx=-6.450681561715195E-6 evalInputDelta=0.226130204678328
Fitness changed from -5.70500305659195 to -5.931133261270278
Iteration 9 complete. Error: -5.931133261270278 Total: 3.0866; Orientation: 0.4231; Line Search: 1.9877
Adding measurement 72e92162 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 7.833e+03, gradient 7.738e-03, dot -0.071; [aa02667e-838d-45b7-84b6-af509a5d7e9e = 1.000/1.000e+00]
Orientation rejected. Popping h
...skipping 14211 bytes...
ete. Error: -10.419978691554748 Total: 3.7999; Orientation: 0.4730; Line Search: 2.7432
<a id="p-19"></a>Iteration 25
<a id="p-18"></a>
Adding measurement 39b8f7c0 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.631e+05, gradient 4.792e-03, dot -0.118; [aa02667e-838d-45b7-84b6-af509a5d7e9e = 1.000/1.000e+00]
Orientation rejected. Popping history element from -10.419978691554748, -3.953886086128171, -3.7320856956490718, -3.60542987933762
LBFGS Accumulation History: 3 points
Removed measurement 39b8f7c0 to history. Total: 3
th(0)=-10.419978691554748;dx=-2.2527827293916325E-5
Armijo: th(2350341.567946288)=-9.98160927338352; dx=-4.253984306197038E-6 evalInputDelta=-0.4383694181712272
Armijo: th(1175170.783973144)=-10.293332265135644; dx=-8.579992028187887E-6 evalInputDelta=-0.12664642641910362
New Minimum: -10.419978691554748 > -10.485855579153062
END: th(391723.59465771465)=-10.485855579153062; dx=-1.472777428546343E-5 evalInputDelta=0.06587688759831423
Fitness changed from -10.419978691554748 to -10.485855579153062
Iteration 26 complete. Error: -10.485855579153062 Total: 4.4280; Orientation: 0.4152; Line Search: 3.3595
Adding measurement 1053ffc1 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.086e+06, gradient 4.762e-03, dot -0.129; [aa02667e-838d-45b7-84b6-af509a5d7e9e = 1.000/1.000e+00]
Orientation rejected. Popping history element from -10.485855579153062, -3.953886086128171, -3.7320856956490718, -3.60542987933762
LBFGS Accumulation History: 3 points
Removed measurement 1053ffc1 to history. Total: 3
th(0)=-10.485855579153062;dx=-2.252811598338232E-5
New Minimum: -10.485855579153062 > -10.56283618889629
END: th(843942.9012345688)=-10.56283618889629; dx=-1.0679882499946969E-5 evalInputDelta=0.07698060974322729
Fitness changed from -10.485855579153062 to -10.56283618889629
Iteration 27 complete. Error: -10.56283618889629 Total: 2.3598; Orientation: 0.4252; Line Search: 1.3420
Adding measurement aa4ee3e to history. Total: 3
Rejected: LBFGS Orientation magnitude: 7.146e+06, gradient 4.565e-03, dot -0.140; [aa02667e-838d-45b7-84b6-af509a5d7e9e = 1.000/1.000e+00]
Orientation rejected. Popping history element from -10.56283618889629, -3.953886086128171, -3.7320856956490718, -3.60542987933762
LBFGS Accumulation History: 3 points
Removed measurement aa4ee3e to history. Total: 3
th(0)=-10.56283618889629;dx=-2.0727803264884116E-5
New Minimum: -10.56283618889629 > -10.818680086440404
END: th(1818219.862825907)=-10.818680086440404; dx=-5.800054105996244E-6 evalInputDelta=0.2558438975441142
Fitness changed from -10.56283618889629 to -10.818680086440404
Iteration 28 complete. Error: -10.818680086440404 Total: 2.3812; Orientation: 0.4215; Line Search: 1.3595
Adding measurement 30d1b0d9 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.126e+06, gradient 4.664e-03, dot -0.125; [aa02667e-838d-45b7-84b6-af509a5d7e9e = 1.000/1.000e+00]
Orientation rejected. Popping history element from -10.818680086440404, -3.953886086128171, -3.7320856956490718, -3.60542987933762
LBFGS Accumulation History: 3 points
Removed measurement 30d1b0d9 to history. Total: 3
th(0)=-10.818680086440404;dx=-2.164890096471385E-5
Armijo: th(3917235.9465771476)=-9.937078819860995; dx=-2.0205628125948047E-6 evalInputDelta=-0.8816012665794091
Armijo: th(1958617.9732885738)=-10.641300465946365; dx=-5.3043305974492274E-6 evalInputDelta=-0.17737962049403855
New Minimum: -10.818680086440404 > -10.900094343216784
END: th(652872.657762858)=-10.900094343216784; dx=-1.1973399415676284E-5 evalInputDelta=0.08141425677638026
Fitness changed from -10.818680086440404 to -10.900094343216784
Iteration 29 complete. Error: -10.900094343216784 Total: 3.7127; Orientation: 0.4210; Line Search: 2.6726
Adding measurement 4896a60e to history. Total: 3
Accepted: LBFGS Orientation magnitude: 3.355e+04, gradient 4.601e-03, dot 0.157; [aa02667e-838d-45b7-84b6-af509a5d7e9e = 1.000/1.000e+00]
Constructing line search parameters: LBFGS+Trust
th(0)=-10.900094343216784;dx=-24.16307517747026
Armijo: th(1406571.502057615)=-6.290150766410931; dx=9.104048950712774E-7 evalInputDelta=-4.609943576805853
Armijo: th(703285.7510288075)=-6.28975052879076; dx=1.8150509784978758E-6 evalInputDelta=-4.610343814426024
Armijo: th(234428.58367626916)=-6.29213494361417; dx=5.449931131660226E-6 evalInputDelta=-4.607959399602614
Armijo: th(58607.14591906729)=-6.290751375686351; dx=2.1773065313431702E-5 evalInputDelta=-4.609342967530433
Armijo: th(11721.429183813458)=-6.275042922458678; dx=1.0613669431729409E-4 evalInputDelta=-4.625051420758106
Armijo: th(1953.5715306355762)=-6.2769306551658275; dx=6.198671425474298E-4 evalInputDelta=-4.623163688050957
Armijo: th(279.08164723365377)=-6.319974877366099; dx=0.004495684766372764 evalInputDelta=-4.580119465850685
Armijo: th(34.88520590420672)=-6.892068888872857; dx=0.032076874006727166 evalInputDelta=-4.008025454343927
Armijo: th(3.8761339893563025)=-7.975829399385514; dx=0.010749604935851566 evalInputDelta=-2.92426494383127
Armijo: th(0.38761339893563024)=-10.331119970318243; dx=-7.488469820449385 evalInputDelta=-0.5689743728985412
Armijo: th(0.035237581721420934)=-10.856779510427101; dx=-13.979135791767069 evalInputDelta=-0.043314832789683066
Armijo: th(0.0029364651434517445)=-10.895824061371012; dx=-14.80335842071388 evalInputDelta=-0.004270281845771606
Armijo: th(2.2588193411167266E-4)=-10.899758944417664; dx=-14.970742820426999 evalInputDelta=-3.353987991197016E-4
Armijo: th(1.6134423865119475E-5)=-10.90007167520934; dx=-14.98044188265483 evalInputDelta=-2.266800744443742E-5
New Minimum: -10.900094343216784 > -10.900095268440577
END: th(1.0756282576746317E-6)=-10.900095268440577; dx=-14.979231615672266 evalInputDelta=9.252237926915541E-7
Fitness changed from -10.900094343216784 to -10.900095268440577
Iteration 30 complete. Error: -10.900095268440577 Total: 11.7282; Orientation: 0.4086; Line Search: 10.7247
<a id="p-21"></a>Iteration 30
<a id="p-20"></a>
Adding measurement 4ba42354 to history. Total: 4
Rejected: LBFGS Orientation magnitude: 3.423e+09, gradient 4.604e-03, dot -0.148; [aa02667e-838d-45b7-84b6-af509a5d7e9e = 1.000/1.000e+00]
Orientation rejected. Popping history element from -10.900095268440577, -10.900094343216784, -3.953886086128171, -3.7320856956490718, -3.60542987933762
Rejected: LBFGS Orientation magnitude: 6.002e+09, gradient 4.604e-03, dot -0.148; [aa02667e-838d-45b7-84b6-af509a5d7e9e = 1.000/1.000e+00]
Orientation rejected. Popping history element from -10.900095268440577, -10.900094343216784, -3.953886086128171, -3.7320856956490718
LBFGS Accumulation History: 3 points
Removed measurement 4ba42354 to history. Total: 4
Removed measurement 4896a60e to history. Total: 3
th(0)=-10.900095268440577;dx=-2.1047079393296638E-5
Armijo: th(2.3173708319127803E-6)=-10.900094343216784; dx=-1.780965358691775E-5 evalInputDelta=-9.252237926915541E-7
Armijo: th(1.1586854159563901E-6)=-10.900094343216784; dx=-1.7784389345622594E-5 evalInputDelta=-9.252237926915541E-7
END: th(3.862284719854634E-7)=-10.900095268440577; dx=-1.7809414262041468E-5 evalInputDelta=0.0
Fitness changed from -10.900095268440577 to -10.900095268440577
Static Iteration Total: 4.1814; Orientation: 0.9126; Line Search: 2.6332
Iteration 31 failed. Error: -10.900095268440577
Previous Error: 0.0 -> -10.900095268440577
Retrying iteration 31
Reset training subject: 3730613539173
Adding measurement 4dffdbcb to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-10.900095268440577;dx=-2.1072120898088288E-5
Armijo: th(8.321040183234899E-7)=-10.900094343216784; dx=-1.78110831458544E-5 evalInputDelta=-9.252237926915541E-7
END: th(4.1605200916174497E-7)=-10.900095268440577; dx=-1.779589358309495E-5 evalInputDelta=0.0
Fitness changed from -10.900095268440577 to -10.900095268440577
Static Iteration Total: 3.2132; Orientation: 0.0155; Line Search: 1.9997
Iteration 32 failed. Error: -10.900095268440577
Previous Error: 0.0 -> -10.900095268440577
Optimization terminated 32
Final threshold in iteration 32: -10.900095268440577 (> -Infinity) after 109.446s (< 3600.000s)
Returns
-10.900095268440577