Code from BasicOptimizer.scala:75 executed in 1419.32 seconds (13.390 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
IterativeTrainer.wrap(trainable)
.setOrientation(orientation())
.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = trainingMonitor.log(msg)
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable, currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable, currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
.setTimeout(trainingMinutes, TimeUnit.MINUTES)
.setMaxIterations(trainingIterations)
.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
.runAndFree
.asInstanceOf[lang.Double]
Reset training subject: 1580670240634300
Reset training subject: 1580688167602500
Adding measurement 3c8fe8ad to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD+Trust
th(0)=-28.921161845326424;dx=-2.1359126835111597E-5
New Minimum: -28.921161845326424 > -28.921168221781652
WOLFE (weak): th(2.154434690031884)=-28.921168221781652; dx=-2.132665962353288E-5 evalInputDelta=6.376455228718214E-6
New Minimum: -28.921168221781652 > -28.921174806853134
WOLFE (weak): th(4.308869380063768)=-28.921174806853134; dx=-2.132815081419126E-5 evalInputDelta=1.2961526710597582E-5
New Minimum: -28.921174806853134 > -28.921201165765524
WOLFE (weak): th(12.926608140191302)=-28.921201165765524; dx=-2.1335283272977478E-5 evalInputDelta=3.93204391002655E-5
New Minimum: -28.921201165765524 > -28.921323023736477
WOLFE (weak): th(51.70643256076521)=-28.921323023736477; dx=-2.1397968787858568E-5 evalInputDelta=1.6117841005325317E-4
New Minimum: -28.921323023736477 > -28.921965265025694
WOLFE (weak): th(258.53216280382605)=-28.921965265025694; dx=-2.1496585003858102E-5 evalInputDelta=8.034196992703357E-4
New Minimum: -28.921965265025694 > -28.926121798654396
WOLFE (weak): th(1551.1929768229563)=-28.926121798654396; dx=-2.289030321487261E-5 evalInputDelta=0.0049599533279724994
New Minimum: -28.926121798654396 > -28.95144474754731
END: th(10858.350837760694)=-28.95144474754731; dx=-1.7216176540417555E-5 evalInputDelta=0.030282902220886143
Fitness changed from -28.921161845326424 to -28.95144474754731
Iteration 1 complete. Error: -28.95144474754731 Total: 194.7158; Orientation: 0.0281; Line Search: 141.9876
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 2adc1e84 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-28.95144474754731;dx=-1.600986691259838E-5
New Minimum: -28.95144474754731 > -28.99793143570423
END: th(23393.607721408407)=-28.99793143570423; dx=-1.356891468000075E-5 evalIn
...skipping 14507 bytes...
.367982695500057; dx=-5.780311041756964E-7 evalInputDelta=0.05954498052597046
Fitness changed from -31.308437714974087 to -31.367982695500057
Iteration 18 complete. Error: -31.367982695500057 Total: 89.0571; Orientation: 0.8064; Line Search: 70.8012
Adding measurement 79e90571 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.659e+03, gradient 8.649e-04, dot -0.783; [bc0eb301-4d46-4771-be7c-b38b054a2f76 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -31.13844541211923, -31.22558068235715, -31.308437714974087, -31.367982695500057
LBFGS Accumulation History: 3 points
Removed measurement 4af0c50 to history. Total: 4
th(0)=-31.367982695500057;dx=-7.480603901092077E-7
New Minimum: -31.367982695500057 > -31.4273882334431
END: th(1047294.6409877221)=-31.4273882334431; dx=-4.5903794643225206E-7 evalInputDelta=0.05940553794304293
Fitness changed from -31.367982695500057 to -31.4273882334431
Iteration 19 complete. Error: -31.4273882334431 Total: 54.3150; Orientation: 0.8035; Line Search: 35.5702
Adding measurement 24b8a393 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.009e+03, gradient 8.733e-04, dot -0.797; [bc0eb301-4d46-4771-be7c-b38b054a2f76 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -31.22558068235715, -31.308437714974087, -31.367982695500057, -31.4273882334431
LBFGS Accumulation History: 3 points
Removed measurement 3f6906f4 to history. Total: 4
th(0)=-31.4273882334431;dx=-7.626660920908067E-7
New Minimum: -31.4273882334431 > -31.48601637283961
END: th(2256327.905228436)=-31.48601637283961; dx=-2.472817770322996E-7 evalInputDelta=0.0586281393965109
Fitness changed from -31.4273882334431 to -31.48601637283961
Iteration 20 complete. Error: -31.48601637283961 Total: 54.2328; Orientation: 0.7974; Line Search: 35.6112
<a id="p-17"></a>Iteration 20
<a id="p-16"></a>
Final threshold in iteration 21: -31.48601637283961 (> -Infinity) after 1419.315s (< 3600.000s)
-31.48601637283961