Code from BasicOptimizer.scala:75 executed in 3824.46 seconds (33.874 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
IterativeTrainer.wrap(trainable)
.setOrientation(orientation())
.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = trainingMonitor.log(msg)
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable, currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable, currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
.setTimeout(trainingMinutes, TimeUnit.MINUTES)
.setMaxIterations(trainingIterations)
.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
.runAndFree
.asInstanceOf[lang.Double]
Reset training subject: 1498135198906200
Reset training subject: 1498181576798900
Adding measurement 17d19538 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD+Trust
th(0)=-28.673746918017667;dx=-5.411488103030998E-6
New Minimum: -28.673746918017667 > -28.67374757056435
WOLFE (weak): th(2.154434690031884)=-28.67374757056435; dx=-5.409815652495396E-6 evalInputDelta=6.525466815787695E-7
New Minimum: -28.67374757056435 > -28.67374910165866
WOLFE (weak): th(4.308869380063768)=-28.67374910165866; dx=-5.408369999545939E-6 evalInputDelta=2.183640994246616E-6
New Minimum: -28.67374910165866 > -28.6737540497755
WOLFE (weak): th(12.926608140191302)=-28.6737540497755; dx=-5.399704836268872E-6 evalInputDelta=7.1317578331786535E-6
New Minimum: -28.6737540497755 > -28.673781183237832
WOLFE (weak): th(51.70643256076521)=-28.673781183237832; dx=-5.3646779590020674E-6 evalInputDelta=3.4265220165252686E-5
New Minimum: -28.673781183237832 > -28.673916691293318
WOLFE (weak): th(258.53216280382605)=-28.673916691293318; dx=-5.225294642080559E-6 evalInputDelta=1.6977327565115274E-4
New Minimum: -28.673916691293318 > -28.67477032262832
WOLFE (weak): th(1551.1929768229563)=-28.67477032262832; dx=-5.155401804515103E-6 evalInputDelta=0.001023404610652534
New Minimum: -28.67477032262832 > -28.680944521911442
WOLFE (weak): th(10858.350837760694)=-28.680944521911442; dx=-5.170374264252158E-6 evalInputDelta=0.00719760389377555
New Minimum: -28.680944521911442 > -28.728221363077562
END: th(86866.80670208555)=-28.728221363077562; dx=-4.632268118040489E-6 evalInputDelta=0.054474445059895515
Fitness changed from -28.673746918017667 to -28.728221363077562
Iteration 1 complete. Error: -28.728221363077562 Total: 552.5178; Orientation: 0.0493; Line Search: 414.2785
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 69ac5752 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-28.72822136307756
...skipping 15139 bytes...
483
Adding measurement 70d7a49b to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.101e+03, gradient 9.451e-04, dot -0.751; [ab9e1187-87eb-479a-bfa6-9d073590b2f0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -30.68653289663295, -30.699809790899355, -30.764487566426396, -30.79001423592369
LBFGS Accumulation History: 3 points
Removed measurement 6978a32a to history. Total: 4
th(0)=-30.79001423592369;dx=-8.932296143409481E-7
New Minimum: -30.79001423592369 > -30.861552789186437
END: th(2792785.709300593)=-30.861552789186437; dx=-2.1829692871940336E-7 evalInputDelta=0.07153855326274794
Fitness changed from -30.79001423592369 to -30.861552789186437
Iteration 19 complete. Error: -30.861552789186437 Total: 139.0949; Orientation: 1.4056; Line Search: 91.2478
Adding measurement 38856d6e to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.082e+03, gradient 1.388e-03, dot -0.516; [ab9e1187-87eb-479a-bfa6-9d073590b2f0 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -30.699809790899355, -30.764487566426396, -30.79001423592369, -30.861552789186437
LBFGS Accumulation History: 3 points
Removed measurement 3cc817bd to history. Total: 4
th(0)=-30.861552789186437;dx=-1.9260805611104092E-6
Armijo: th(6016874.413942498)=-30.59488563550015; dx=2.9040104777289257E-7 evalInputDelta=-0.266667153686285
Armijo: th(3008437.206971249)=-30.84287577991684; dx=1.3590368270167686E-7 evalInputDelta=-0.018677009269595146
New Minimum: -30.861552789186437 > -30.929808928941686
END: th(1002812.4023237497)=-30.929808928941686; dx=-1.7668335360548144E-7 evalInputDelta=0.06825613975524902
Fitness changed from -30.861552789186437 to -30.929808928941686
Iteration 20 complete. Error: -30.929808928941686 Total: 230.2919; Orientation: 1.4241; Line Search: 183.3192
<a id="p-17"></a>Iteration 20
<a id="p-16"></a>
Final threshold in iteration 20: -30.929808928941686 (> -Infinity) after 3824.458s (< 3600.000s)
-30.929808928941686