Code from BasicOptimizer.scala:75 executed in 995.85 seconds (11.438 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
IterativeTrainer.wrap(trainable)
.setOrientation(orientation())
.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = trainingMonitor.log(msg)
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable, currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable, currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
.setTimeout(trainingMinutes, TimeUnit.MINUTES)
.setMaxIterations(trainingIterations)
.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
.runAndFree
.asInstanceOf[lang.Double]
Reset training subject: 1157019480372800
Reset training subject: 1157027619036900
Adding measurement 8ab231 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD+Trust
th(0)=5.159011200963412;dx=-2.4326120110260436E-5
New Minimum: 5.159011200963412 > 5.1589884142142814
WOLFE (weak): th(2.154434690031884)=5.1589884142142814; dx=-2.432608499665613E-5 evalInputDelta=2.278674913025469E-5
New Minimum: 5.1589884142142814 > 5.158965473806147
WOLFE (weak): th(4.308869380063768)=5.158965473806147; dx=-2.43262061826562E-5 evalInputDelta=4.572715726514076E-5
New Minimum: 5.158965473806147 > 5.158873889344558
WOLFE (weak): th(12.926608140191302)=5.158873889344558; dx=-2.4325440637536612E-5 evalInputDelta=1.3731161885388588E-4
New Minimum: 5.158873889344558 > 5.158461105748834
WOLFE (weak): th(51.70643256076521)=5.158461105748834; dx=-2.4316873365806862E-5 evalInputDelta=5.500952145780502E-4
New Minimum: 5.158461105748834 > 5.156262153504113
WOLFE (weak): th(258.53216280382605)=5.156262153504113; dx=-2.429298665444537E-5 evalInputDelta=0.0027490474592983816
New Minimum: 5.156262153504113 > 5.142572555537079
WOLFE (weak): th(1551.1929768229563)=5.142572555537079; dx=-2.4107503887542133E-5 evalInputDelta=0.016438645426332776
New Minimum: 5.142572555537079 > 5.046914574650186
WOLFE (weak): th(10858.350837760694)=5.046914574650186; dx=-2.288422147153842E-5 evalInputDelta=0.11209662631322548
New Minimum: 5.046914574650186 > 4.4329245132069754
END: th(86866.80670208555)=4.4329245132069754; dx=-1.4744238144844636E-5 evalInputDelta=0.7260866877564363
Fitness changed from 5.159011200963412 to 4.4329245132069754
Iteration 1 complete. Error: 4.4329245132069754 Total: 99.9319; Orientation: 0.0069; Line Search: 74.4034
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 5e557671 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=4.4329245132069754;dx=-1.1123499339777064E-5
New Minimu
...skipping 24068 bytes...
InputDelta=0.022928998518070953
Fitness changed from -0.04467297580638241 to -0.06760197432445336
Iteration 28 complete. Error: -0.06760197432445336 Total: 40.6090; Orientation: 0.1907; Line Search: 32.5924
Adding measurement 4b2af0eb to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.870e+03, gradient 1.933e-04, dot -0.861; [2bcf0767-08c9-4293-b7df-dd6d5be3b3db = 1.000/1.000e+00]
Orientation rejected. Popping history element from -0.009137620484452658, -0.0322510564057753, -0.04467297580638241, -0.06760197432445336
LBFGS Accumulation History: 3 points
Removed measurement 211cf27b to history. Total: 4
th(0)=-0.06760197432445336;dx=-3.736239732951017E-8
New Minimum: -0.06760197432445336 > -0.08441151898921806
END: th(2611490.6310514323)=-0.08441151898921806; dx=-2.840126830702525E-8 evalInputDelta=0.016809544664764697
Fitness changed from -0.06760197432445336 to -0.08441151898921806
Iteration 29 complete. Error: -0.08441151898921806 Total: 24.8776; Orientation: 0.1923; Line Search: 16.3973
Adding measurement 3c5cb013 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.454e+03, gradient 1.914e-04, dot -0.881; [2bcf0767-08c9-4293-b7df-dd6d5be3b3db = 1.000/1.000e+00]
Orientation rejected. Popping history element from -0.0322510564057753, -0.04467297580638241, -0.06760197432445336, -0.08441151898921806
LBFGS Accumulation History: 3 points
Removed measurement 4afdb05e to history. Total: 4
th(0)=-0.08441151898921806;dx=-3.663080968247184E-8
New Minimum: -0.08441151898921806 > -0.1003122601392008
END: th(5626286.008230462)=-0.1003122601392008; dx=-1.826122915279497E-8 evalInputDelta=0.015900741149982744
Fitness changed from -0.08441151898921806 to -0.1003122601392008
Iteration 30 complete. Error: -0.1003122601392008 Total: 25.3697; Orientation: 0.2322; Line Search: 16.7672
<a id="p-21"></a>Iteration 30
<a id="p-20"></a>
Final threshold in iteration 31: -0.1003122601392008 (> -Infinity) after 995.843s (< 3600.000s)
-0.1003122601392008