Code from BasicOptimizer.scala:75 executed in 1580.81 seconds (11.369 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
IterativeTrainer.wrap(trainable)
.setOrientation(orientation())
.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = trainingMonitor.log(msg)
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable, currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable, currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
.setTimeout(trainingMinutes, TimeUnit.MINUTES)
.setMaxIterations(trainingIterations)
.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
.runAndFree
.asInstanceOf[lang.Double]
Reset training subject: 661829814352200
Reset training subject: 661848883803700
Adding measurement 7fba9eb3 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD+Trust
th(0)=7.039115309715271;dx=-1.354525126663835E-5
New Minimum: 7.039115309715271 > 7.039088308811188
WOLFE (weak): th(2.154434690031884)=7.039088308811188; dx=-1.3450758437251887E-5 evalInputDelta=2.7000904083251953E-5
New Minimum: 7.039088308811188 > 7.039060518145561
WOLFE (weak): th(4.308869380063768)=7.039060518145561; dx=-1.3450619004154812E-5 evalInputDelta=5.479156970977783E-5
New Minimum: 7.039060518145561 > 7.0389513075351715
WOLFE (weak): th(12.926608140191302)=7.0389513075351715; dx=-1.3452034077296971E-5 evalInputDelta=1.640021800994873E-4
New Minimum: 7.0389513075351715 > 7.038458734750748
WOLFE (weak): th(51.70643256076521)=7.038458734750748; dx=-1.3453023404838614E-5 evalInputDelta=6.565749645233154E-4
New Minimum: 7.038458734750748 > 7.035832047462463
WOLFE (weak): th(258.53216280382605)=7.035832047462463; dx=-1.3452233697579738E-5 evalInputDelta=0.003283262252807617
New Minimum: 7.035832047462463 > 7.019394427537918
WOLFE (weak): th(1551.1929768229563)=7.019394427537918; dx=-1.3470210198950732E-5 evalInputDelta=0.019720882177352905
New Minimum: 7.019394427537918 > 6.899801209568977
WOLFE (weak): th(10858.350837760694)=6.899801209568977; dx=-1.3539143920367145E-5 evalInputDelta=0.13931410014629364
New Minimum: 6.899801209568977 > 5.98417454957962
END: th(86866.80670208555)=5.98417454957962; dx=-1.083374353603455E-5 evalInputDelta=1.0549407601356506
Fitness changed from 7.039115309715271 to 5.98417454957962
Iteration 1 complete. Error: 5.98417454957962 Total: 222.4101; Orientation: 0.0548; Line Search: 166.6125
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 4259588 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=5.98417454957962;dx=-1.261996855788186E-5
New Minimum: 5.984
...skipping 15058 bytes...
87319600582 Total: 77.2013; Orientation: 1.2134; Line Search: 57.2752
Adding measurement 6c5fd24e to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.320e+03, gradient 2.026e-04, dot -0.711; [0a63d718-6939-4a12-a216-d16618212b64 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -3.843924932181835, -3.8888792395591736, -3.9272061437368393, -3.972287319600582
LBFGS Accumulation History: 3 points
Removed measurement 5c35dc96 to history. Total: 4
th(0)=-3.972287319600582;dx=-4.104117062194152E-8
New Minimum: -3.972287319600582 > -4.011519521474838
END: th(2094589.2819754444)=-4.011519521474838; dx=-5.097236809668451E-9 evalInputDelta=0.039232201874256134
Fitness changed from -3.972287319600582 to -4.011519521474838
Iteration 19 complete. Error: -4.011519521474838 Total: 58.8027; Orientation: 1.2729; Line Search: 38.3976
Adding measurement 6033f263 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.096e+03, gradient 1.818e-04, dot -0.740; [0a63d718-6939-4a12-a216-d16618212b64 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -3.8888792395591736, -3.9272061437368393, -3.972287319600582, -4.011519521474838
LBFGS Accumulation History: 3 points
Removed measurement 186ba71f to history. Total: 4
th(0)=-4.011519521474838;dx=-3.3061236014572285E-8
New Minimum: -4.011519521474838 > -4.033132299780846
WOLF (strong): th(4512655.810456873)=-4.033132299780846; dx=1.0792770879699723E-8 evalInputDelta=0.021612778306007385
New Minimum: -4.033132299780846 > -4.045511871576309
END: th(2256327.9052284365)=-4.045511871576309; dx=-8.077680914477114E-9 evalInputDelta=0.03399235010147095
Fitness changed from -4.011519521474838 to -4.045511871576309
Iteration 20 complete. Error: -4.045511871576309 Total: 76.5162; Orientation: 1.0980; Line Search: 56.9959
<a id="p-17"></a>Iteration 20
<a id="p-16"></a>
Final threshold in iteration 21: -4.045511871576309 (> -Infinity) after 1579.792s (< 3600.000s)
-4.045511871576309