BasicOptimizer.scala:89 executed in 320.06 seconds (4.696 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 576806157245700
Reset training subject: 576812134782500
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
New Minimum: -42.25708067250935 > -42.30039937803989
th(0)=-42.30039937803989;dx=-2.052638591078849E-6
New Minimum: -42.30039937803989 > -42.30040116353718
WOLFE (weak): th(2.154434690031884)=-42.30040116353718; dx=-1.9701977130857094E-6 evalInputDelta=1.7854972895747778E-6
New Minimum: -42.30040116353718 > -42.3004065200286
WOLFE (weak): th(4.308869380063768)=-42.3004065200286; dx=-1.970155948069915E-6 evalInputDelta=7.141988710657188E-6
New Minimum: -42.3004065200286 > -42.300424374995124
WOLFE (weak): th(12.926608140191302)=-42.300424374995124; dx=-1.9703272322570187E-6 evalInputDelta=2.4996955232836626E-5
New Minimum: -42.300424374995124 > -42.30051007872952
WOLFE (weak): th(51.70643256076521)=-42.30051007872952; dx=-1.9697921222285432E-6 evalInputDelta=1.1070068963192625E-4
New Minimum: -42.30051007872952 > -42.30095287859107
WOLFE (weak): th(258.53216280382605)=-42.30095287859107; dx=-1.9694082167483116E-6 evalInputDelta=5.535005511774216E-4
New Minimum: -42.30095287859107 > -42.30373098485888
WOLFE (weak): th(1551.1929768229563)=-42.30373098485888; dx=-1.9696389227147087E-6 evalInputDelta=0.00333160681898903
New Minimum: -42.30373098485888 > -42.32372404098679
WOLFE (weak): th(10858.350837760694)=-42.32372404098679; dx=-1.9689608476063927E-6 evalInputDelta=0.023324662946897945
New Minimum: -42.32372404098679 > -42.48674302655599
WOLFE (weak): th(86866.80670208555)=-42.48674302655599; dx=-1.963653218256483E-6 evalInputDelta=0.1863436485161003
New Minimum: -42.48674302655599 > -43.951716439975876
WOLFE (weak): th(781801.26031877)=-43.951716439975876; dx=-1.9019633972132664E-6 evalInputDelta=1.651317061935984
New Minimum: -43.951716439975876 > -55.295997238671106
END: th(7818012.6031877)=-55.295997238671106; dx=-1.0463436592212155E-6 evalInputDelta=12.995597860631214
Fitness changed from -42.25708067250935 to -55.295997238671106
Iteration 1 complete. Error: -55.295997238671106 Total: 86.6020; Orientation: 0.1022; Line Search: 67.9589
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 3f68ca6a to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-55.295997238671106;dx=-1.1316350821166782E-6
New Minimum: -55.295997238671106 > -68.31823802935502
END: th(1.684339755941405E7)=-68.31823802935502; dx=-3.9871780440695067E-7 evalInputDelta=13.02224079068391
Fitness changed from -55.295997238671106 to -68.31823802935502
Iteration 2 complete. Error: -68.31823802935502 Total: 19.5601; Orientation: 0.1377; Line Search: 12.3863
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 6763ec8b to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-68.31823802935502;dx=-8.210481388088171E-7
New Minimum: -68.31823802935502 > -77.52809422042762
END: th(3.6288E7)=-77.52809422042762; dx=-6.357754580622894E-9 evalInputDelta=9.209856191072603
Fitness changed from -68.31823802935502 to -77.52809422042762
Iteration 3 complete. Error: -77.52809422042762 Total: 20.0778; Orientation: 0.1447; Line Search: 12.6623
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 529fd113 to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-77.52809422042762;dx=-1.0012754597284755E-6
Armijo: th(7.8180126031877E7)=-70.02089734976813; dx=2.9411573709343884E-7 evalInputDelta=-7.5071968706594845
New Minimum: -77.52809422042762 > -85.3072168311793
WOLF (strong): th(3.90900630159385E7)=-85.3072168311793; dx=3.1023535269825143E-7 evalInputDelta=7.779122610751685
New Minimum: -85.3072168311793 > -87.71513703508667
END: th(1.3030021005312832E7)=-87.71513703508667; dx=-3.8208137417461055E-7 evalInputDelta=10.187042814659051
Fitness changed from -77.52809422042762 to -87.71513703508667
Iteration 4 complete. Error: -87.71513703508667 Total: 33.8095; Orientation: 0.1460; Line Search: 26.0991
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 67d2dda9 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 5.926e+04, gradient 6.939e-04, dot -0.656; [195b691a-af15-437a-95c3-01b2ce95382d = 1.000/1.000e+00]
Orientation rejected. Popping history element from -87.71513703508667, -77.52809422042762, -68.31823802935502, -55.295997238671106
LBFGS Accumulation History: 3 points
Removed measurement 67d2dda9 to history. Total: 3
th(0)=-87.71513703508667;dx=-4.809303810101557E-7
New Minimum: -87.71513703508667 > -95.6035856277504
END: th(2.807232926569009E7)=-95.6035856277504; dx=-7.609683903120599E-8 evalInputDelta=7.888448592663735
Fitness changed from -87.71513703508667 to -95.6035856277504
Iteration 5 complete. Error: -95.6035856277504 Total: 22.5464; Orientation: 2.6521; Line Search: 12.5500
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 2cf78bf8 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 6.011e+04, gradient 7.277e-04, dot -0.567; [195b691a-af15-437a-95c3-01b2ce95382d = 1.000/1.000e+00]
Orientation rejected. Popping history element from -95.6035856277504, -77.52809422042762, -68.31823802935502, -55.295997238671106
LBFGS Accumulation History: 3 points
Removed measurement 2cf78bf8 to history. Total: 3
th(0)=-95.6035856277504;dx=-5.289923520661465E-7
Armijo: th(6.048000000000001E7)=-94.07337776789878; dx=1.8625068087025387E-7 evalInputDelta=-1.5302078598516289
New Minimum: -95.6035856277504 > -100.14767092399674
WOLF (strong): th(3.0240000000000004E7)=-100.14767092399674; dx=1.0445718019985098E-7 evalInputDelta=4.544085296246337
END: th(1.0080000000000002E7)=-99.85310975594622; dx=-2.392862793640002E-7 evalInputDelta=4.249524128195816
Fitness changed from -95.6035856277504 to -100.14767092399674
Iteration 6 complete. Error: -100.14767092399674 Total: 34.7842; Orientation: 2.6144; Line Search: 24.7601
Adding measurement 33a6f88c to history. Total: 3
Rejected: LBFGS Orientation magnitude: 7.554e+04, gradient 8.229e-04, dot -0.462; [195b691a-af15-437a-95c3-01b2ce95382d = 1.000/1.000e+00]
Orientation rejected. Popping history element from -100.14767092399674, -77.52809422042762, -68.31823802935502, -55.295997238671106
LBFGS Accumulation History: 3 points
Removed measurement 33a6f88c to history. Total: 3
th(0)=-100.14767092399674;dx=-6.758173154109979E-7
New Minimum: -100.14767092399674 > -107.31166434593078
WOLF (strong): th(2.1716701675521392E7)=-107.31166434593078; dx=8.121481973431595E-8 evalInputDelta=7.1639934219340375
END: th(1.0858350837760696E7)=-106.19311915811787; dx=-3.129426407177718E-7 evalInputDelta=6.045448234121125
Fitness changed from -100.14767092399674 to -107.31166434593078
Iteration 7 complete. Error: -107.31166434593078 Total: 27.2976; Orientation: 2.5633; Line Search: 18.5321
Adding measurement 2eb6ac55 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 8.055e+04, gradient 6.447e-04, dot -0.533; [195b691a-af15-437a-95c3-01b2ce95382d = 1.000/1.000e+00]
Orientation rejected. Popping history element from -107.31166434593078, -77.52809422042762, -68.31823802935502, -55.295997238671106
LBFGS Accumulation History: 3 points
Removed measurement 2eb6ac55 to history. Total: 3
th(0)=-107.31166434593078;dx=-4.148904683961673E-7
New Minimum: -107.31166434593078 > -111.45707609925542
WOLF (strong): th(2.339360772140841E7)=-111.45707609925542; dx=4.2351733208716904E-8 evalInputDelta=4.1454117533246375
END: th(1.1696803860704206E7)=-110.97050730995393; dx=-1.6107636971261075E-7 evalInputDelta=3.658842964023151
Fitness changed from -107.31166434593078 to -111.45707609925542
Iteration 8 complete. Error: -111.45707609925542 Total: 26.7728; Orientation: 2.5413; Line Search: 18.3345
Adding measurement 3f477811 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 9.641e+04, gradient 6.406e-04, dot -0.518; [195b691a-af15-437a-95c3-01b2ce95382d = 1.000/1.000e+00]
Orientation rejected. Popping history element from -111.45707609925542, -77.52809422042762, -68.31823802935502, -55.295997238671106
LBFGS Accumulation History: 3 points
Removed measurement 3f477811 to history. Total: 3
th(0)=-111.45707609925542;dx=-4.103135175352409E-7
New Minimum: -111.45707609925542 > -114.98060712962096
WOLF (strong): th(2.5200000000000007E7)=-114.98060712962096; dx=9.21544075923475E-8 evalInputDelta=3.5235310303655467
New Minimum: -114.98060712962096 > -115.15259931381921
END: th(1.2600000000000004E7)=-115.15259931381921; dx=-1.247810859267009E-7 evalInputDelta=3.695523214563792
Fitness changed from -111.45707609925542 to -115.15259931381921
Iteration 9 complete. Error: -115.15259931381921 Total: 26.6988; Orientation: 2.5178; Line Search: 18.3750
Adding measurement 2f0f08db to history. Total: 3
Rejected: LBFGS Orientation magnitude: 9.718e+04, gradient 4.368e-04, dot -0.692; [195b691a-af15-437a-95c3-01b2ce95382d = 1.000/1.000e+00]
Orientation rejected. Popping history element from -115.15259931381921, -77.52809422042762, -68.31823802935502, -55.295997238671106
LBFGS Accumulation History: 3 points
Removed measurement 2f0f08db to history. Total: 3
th(0)=-115.15259931381921;dx=-1.9062223508432872E-7
New Minimum: -115.15259931381921 > -118.94531542849364
END: th(2.7145877094401743E7)=-118.94531542849364; dx=-9.264865192872827E-8 evalInputDelta=3.792716114674434
Fitness changed from -115.15259931381921 to -118.94531542849364
Iteration 10 complete. Error: -118.94531542849364 Total: 20.7194; Orientation: 2.5766; Line Search: 12.0071
<a id="p-13"></a>Iteration 10
<a id="p-12"></a>
Final threshold in iteration 11: -118.94531542849364 (> -Infinity) after 320.059s (< 5400.000s)
Returns
-118.94531542849364