BasicOptimizer.scala:89 executed in 217.79 seconds (3.449 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 689292923500900
Reset training subject: 689297618489400
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
New Minimum: -779.5844245493877 > -779.815803571587
th(0)=-779.815803571587;dx=-0.008459286559197926
New Minimum: -779.815803571587 > -779.8345177023135
WOLFE (weak): th(2.154434690031884)=-779.8345177023135; dx=-0.007945211262355494 evalInputDelta=0.01871413072649375
New Minimum: -779.8345177023135 > -779.8532714555989
WOLFE (weak): th(4.308869380063768)=-779.8532714555989; dx=-0.00794474571221912 evalInputDelta=0.03746788401190315
New Minimum: -779.8532714555989 > -779.9282819593094
WOLFE (weak): th(12.926608140191302)=-779.9282819593094; dx=-0.00794298638882712 evalInputDelta=0.11247838772237628
New Minimum: -779.9282819593094 > -780.26557978678
WOLFE (weak): th(51.70643256076521)=-780.26557978678; dx=-0.007941948989745071 evalInputDelta=0.4497762151929692
New Minimum: -780.26557978678 > -782.060819246176
WOLFE (weak): th(258.53216280382605)=-782.060819246176; dx=-0.007912024745925835 evalInputDelta=2.2450156745890126
New Minimum: -782.060819246176 > -793.1306638883659
WOLFE (weak): th(1551.1929768229563)=-793.1306638883659; dx=-0.007727599780084133 evalInputDelta=13.31486031677889
New Minimum: -793.1306638883659 > -864.715885421333
END: th(10858.350837760694)=-864.715885421333; dx=-0.006271370919026933 evalInputDelta=84.90008184974602
Fitness changed from -779.5844245493877 to -864.715885421333
Iteration 1 complete. Error: -864.715885421333 Total: 49.9817; Orientation: 0.0609; Line Search: 36.2089
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 1f215135 to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-864.715885421333;dx=-0.00575778138538633
New Minimum: -864.715885421333 > -988.1022530588623
END: th(23393.607721408407)=-988.1022530588623; dx=-0.0038753767643651665 evalInputDelta=123.38636763752925
Fitness changed from -864.715885421333 to -988.1022530588623
Iteration 2 complete. Error: -988.1022530588623 Total: 14.9513; Orientation: 0.0987; Line Search: 9.2519
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 1489aca4 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-988.1022530588623;dx=-0.0038301851295536662
New Minimum: -988.1022530588623 > -1144.7568519122303
END: th(50400.00000000001)=-1144.7568519122303; dx=-0.002060702494345425 evalInputDelta=156.65459885336804
Fitness changed from -988.1022530588623 to -1144.7568519122303
Iteration 3 complete. Error: -1144.7568519122303 Total: 14.4710; Orientation: 0.0970; Line Search: 8.9429
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 58320011 to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-1144.7568519122303;dx=-0.003246279551542225
New Minimum: -1144.7568519122303 > -1304.8074091604476
END: th(108583.50837760697)=-1304.8074091604476; dx=-7.09623712825073E-4 evalInputDelta=160.05055724821727
Fitness changed from -1144.7568519122303 to -1304.8074091604476
Iteration 4 complete. Error: -1304.8074091604476 Total: 14.5638; Orientation: 0.0917; Line Search: 9.0604
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 5d673c13 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.379e+04, gradient 6.253e-02, dot -0.499; [2b029ee7-fc8a-4e59-961f-10f116862a5a = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1304.8074091604476, -1144.7568519122303, -988.1022530588623, -864.715885421333
LBFGS Accumulation History: 3 points
Removed measurement 5d673c13 to history. Total: 3
th(0)=-1304.8074091604476;dx=-0.003906358096167751
New Minimum: -1304.8074091604476 > -1413.4958436444022
WOLF (strong): th(233936.07721408413)=-1413.4958436444022; dx=1.846300303437694E-4 evalInputDelta=108.68843448395455
New Minimum: -1413.4958436444022 > -1437.4113450922807
END: th(116968.03860704206)=-1437.4113450922807; dx=-5.18398559217128E-5 evalInputDelta=132.60393593183312
Fitness changed from -1304.8074091604476 to -1437.4113450922807
Iteration 5 complete. Error: -1437.4113450922807 Total: 20.9148; Orientation: 1.7265; Line Search: 13.5244
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 2ea65371 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.518e+04, gradient 6.451e-02, dot -0.407; [2b029ee7-fc8a-4e59-961f-10f116862a5a = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1437.4113450922807, -1144.7568519122303, -988.1022530588623, -864.715885421333
LBFGS Accumulation History: 3 points
Removed measurement 2ea65371 to history. Total: 3
th(0)=-1437.4113450922807;dx=-0.00415831600310621
New Minimum: -1437.4113450922807 > -1495.2666986193467
WOLF (strong): th(252000.0000000001)=-1495.2666986193467; dx=5.675044914105871E-4 evalInputDelta=57.85535352706597
New Minimum: -1495.2666986193467 > -1581.3080819372296
WOLF (strong): th(126000.00000000004)=-1581.3080819372296; dx=3.354771775815544E-4 evalInputDelta=143.89673684494892
END: th(42000.000000000015)=-1562.1549218947523; dx=-0.0013947190645300657 evalInputDelta=124.74357680247158
Fitness changed from -1437.4113450922807 to -1581.3080819372296
Iteration 6 complete. Error: -1581.3080819372296 Total: 25.9714; Orientation: 1.7507; Line Search: 18.6941
Adding measurement 3ac3110b to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.786e+04, gradient 6.594e-02, dot -0.353; [2b029ee7-fc8a-4e59-961f-10f116862a5a = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1581.3080819372296, -1144.7568519122303, -988.1022530588623, -864.715885421333
LBFGS Accumulation History: 3 points
Removed measurement 3ac3110b to history. Total: 3
th(0)=-1581.3080819372296;dx=-0.004344934131521014
New Minimum: -1581.3080819372296 > -1746.16171931468
WOLF (strong): th(90486.25698133915)=-1746.16171931468; dx=6.840742432583112E-5 evalInputDelta=164.85363737745047
END: th(45243.12849066957)=-1722.0382545112057; dx=-0.001401709890339148 evalInputDelta=140.730172573976
Fitness changed from -1581.3080819372296 to -1746.16171931468
Iteration 7 complete. Error: -1746.16171931468 Total: 20.3619; Orientation: 1.7656; Line Search: 14.1202
Adding measurement 5c13ddd6 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.924e+04, gradient 5.854e-02, dot -0.409; [2b029ee7-fc8a-4e59-961f-10f116862a5a = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1746.16171931468, -1144.7568519122303, -988.1022530588623, -864.715885421333
LBFGS Accumulation History: 3 points
Removed measurement 5c13ddd6 to history. Total: 3
th(0)=-1746.16171931468;dx=-0.003422808220212894
New Minimum: -1746.16171931468 > -1865.1650061053579
WOLF (strong): th(97473.3655058684)=-1865.1650061053579; dx=2.1641720920968785E-4 evalInputDelta=119.00328679067775
END: th(48736.6827529342)=-1855.0474387465135; dx=-8.91888980447217E-4 evalInputDelta=108.88571943183342
Fitness changed from -1746.16171931468 to -1865.1650061053579
Iteration 8 complete. Error: -1865.1650061053579 Total: 20.3829; Orientation: 1.7079; Line Search: 14.2295
Adding measurement 3dc67b8c to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.592e+04, gradient 6.091e-02, dot -0.389; [2b029ee7-fc8a-4e59-961f-10f116862a5a = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1865.1650061053579, -1144.7568519122303, -988.1022530588623, -864.715885421333
LBFGS Accumulation History: 3 points
Removed measurement 3dc67b8c to history. Total: 3
th(0)=-1865.1650061053579;dx=-0.003707364640430844
New Minimum: -1865.1650061053579 > -1967.1734163514918
WOLF (strong): th(105000.00000000004)=-1967.1734163514918; dx=5.650501357574656E-4 evalInputDelta=102.00841024613396
New Minimum: -1967.1734163514918 > -1976.0686981985216
END: th(52500.00000000002)=-1976.0686981985216; dx=-5.927664653794286E-4 evalInputDelta=110.90369209316373
Fitness changed from -1865.1650061053579 to -1976.0686981985216
Iteration 9 complete. Error: -1976.0686981985216 Total: 19.6785; Orientation: 1.6562; Line Search: 13.6426
Adding measurement 2f8480f3 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.806e+04, gradient 4.640e-02, dot -0.519; [2b029ee7-fc8a-4e59-961f-10f116862a5a = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1976.0686981985216, -1144.7568519122303, -988.1022530588623, -864.715885421333
LBFGS Accumulation History: 3 points
Removed measurement 2f8480f3 to history. Total: 3
th(0)=-1976.0686981985216;dx=-0.0021487156464465593
New Minimum: -1976.0686981985216 > -2070.9305879241824
END: th(113107.82122667394)=-2070.9305879241824; dx=-1.7050449486246506E-4 evalInputDelta=94.86188972566083
Fitness changed from -1976.0686981985216 to -2070.9305879241824
Iteration 10 complete. Error: -2070.9305879241824 Total: 15.5728; Orientation: 1.7402; Line Search: 9.3318
<a id="p-13"></a>Iteration 10
<a id="p-12"></a>
Final threshold in iteration 11: -2070.9305879241824 (> -Infinity) after 217.788s (< 5400.000s)
Returns
-2070.9305879241824