Code from BasicOptimizer.scala:75 executed in 969.41 seconds (10.617 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
IterativeTrainer.wrap(trainable)
.setOrientation(orientation())
.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = trainingMonitor.log(msg)
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable, currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable, currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
.setTimeout(trainingMinutes, TimeUnit.MINUTES)
.setMaxIterations(trainingIterations)
.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
.runAndFree
.asInstanceOf[lang.Double]
Reset training subject: 1240331009918900
Reset training subject: 1240338876145500
Adding measurement 1caa1866 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD+Trust
th(0)=5.254201120326764;dx=-3.0060002489527542E-5
New Minimum: 5.254201120326764 > 5.254173200687202
WOLFE (weak): th(2.154434690031884)=5.254173200687202; dx=-3.0059378521463283E-5 evalInputDelta=2.791963956205734E-5
New Minimum: 5.254173200687202 > 5.254145097905566
WOLFE (weak): th(4.308869380063768)=5.254145097905566; dx=-3.0059048660109164E-5 evalInputDelta=5.602242119806533E-5
New Minimum: 5.254145097905566 > 5.254033044082299
WOLFE (weak): th(12.926608140191302)=5.254033044082299; dx=-3.0058024250019987E-5 evalInputDelta=1.6807624446446567E-4
New Minimum: 5.254033044082299 > 5.253528843846416
WOLFE (weak): th(51.70643256076521)=5.253528843846416; dx=-3.0048066223050407E-5 evalInputDelta=6.722764803477332E-4
New Minimum: 5.253528843846416 > 5.2508418445595435
WOLFE (weak): th(258.53216280382605)=5.2508418445595435; dx=-3.0015343303640554E-5 evalInputDelta=0.003359275767220282
New Minimum: 5.2508418445595435 > 5.234112206321278
WOLFE (weak): th(1551.1929768229563)=5.234112206321278; dx=-2.979263407142886E-5 evalInputDelta=0.020088914005485847
New Minimum: 5.234112206321278 > 5.117599786747395
WOLFE (weak): th(10858.350837760694)=5.117599786747395; dx=-2.817538566818156E-5 evalInputDelta=0.1366013335793692
New Minimum: 5.117599786747395 > 4.39759256393041
END: th(86866.80670208555)=4.39759256393041; dx=-1.6951999907518367E-5 evalInputDelta=0.8566085563963535
Fitness changed from 5.254201120326764 to 4.39759256393041
Iteration 1 complete. Error: 4.39759256393041 Total: 96.0463; Orientation: 0.0066; Line Search: 72.0417
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 1c0f32ad to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=4.39759256393041;dx=-1.2484658408260976E-5
New Minimum: 4.39
...skipping 23952 bytes...
62004139389373E-8 evalInputDelta=0.020800696951747244
Fitness changed from 0.1262613847043228 to 0.10546068775257555
Iteration 28 complete. Error: 0.10546068775257555 Total: 40.0579; Orientation: 0.2013; Line Search: 31.8100
Adding measurement 16908f89 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.838e+03, gradient 1.840e-04, dot -0.788; [450bc4d8-73d7-4d74-ad16-3eca60baa9ac = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.159440495162251, 0.14111476039326387, 0.1262613847043228, 0.10546068775257555
LBFGS Accumulation History: 3 points
Removed measurement 25f3207e to history. Total: 4
th(0)=0.10546068775257555;dx=-3.387281289113181E-8
New Minimum: 0.10546068775257555 > 0.08784562974138399
END: th(2611490.6310514323)=0.08784562974138399; dx=-2.5357321766666057E-8 evalInputDelta=0.017615058011191564
Fitness changed from 0.10546068775257555 to 0.08784562974138399
Iteration 29 complete. Error: 0.08784562974138399 Total: 23.6235; Orientation: 0.2020; Line Search: 15.5509
Adding measurement 404cc16a to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.834e+03, gradient 1.811e-04, dot -0.879; [450bc4d8-73d7-4d74-ad16-3eca60baa9ac = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.14111476039326387, 0.1262613847043228, 0.10546068775257555, 0.08784562974138399
LBFGS Accumulation History: 3 points
Removed measurement 452aea78 to history. Total: 4
th(0)=0.08784562974138399;dx=-3.2781477650890534E-8
New Minimum: 0.08784562974138399 > 0.07270561277684195
END: th(5626286.008230462)=0.07270561277684195; dx=-1.4704820965423821E-8 evalInputDelta=0.015140016964542036
Fitness changed from 0.08784562974138399 to 0.07270561277684195
Iteration 30 complete. Error: 0.07270561277684195 Total: 24.1257; Orientation: 0.2004; Line Search: 15.9705
<a id="p-21"></a>Iteration 30
<a id="p-20"></a>
Final threshold in iteration 31: 0.07270561277684195 (> -Infinity) after 969.399s (< 3600.000s)
0.07270561277684195