Code from BasicOptimizer.scala:75 executed in 3797.49 seconds (80.501 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
IterativeTrainer.wrap(trainable)
.setOrientation(orientation())
.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = trainingMonitor.log(msg)
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable, currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable, currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
.setTimeout(trainingMinutes, TimeUnit.MINUTES)
.setMaxIterations(trainingIterations)
.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
.runAndFree
.asInstanceOf[lang.Double]
Reset training subject: 1093039302796599
Reset training subject: 1093091475616500
Adding measurement 1c1f348f to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD+Trust
th(0)=1.675114558577306;dx=-1.1710392291951244E-7
New Minimum: 1.675114558577306 > 1.6751145138738224
WOLFE (weak): th(2.154434690031884)=1.6751145138738224; dx=-1.159083559632281E-7 evalInputDelta=4.470348358154297E-8
New Minimum: 1.6751145138738224 > 1.675114335059888
WOLFE (weak): th(4.308869380063768)=1.675114335059888; dx=-1.1590812387525965E-7 evalInputDelta=2.2351741790771484E-7
New Minimum: 1.675114335059888 > 1.6751131876704761
WOLFE (weak): th(12.926608140191302)=1.6751131876704761; dx=-1.159080057070914E-7 evalInputDelta=1.3709068298339844E-6
New Minimum: 1.6751131876704761 > 1.6751089730205462
WOLFE (weak): th(51.70643256076521)=1.6751089730205462; dx=-1.159073583433309E-7 evalInputDelta=5.585556759779564E-6
New Minimum: 1.6751089730205462 > 1.6750847424743776
WOLFE (weak): th(258.53216280382605)=1.6750847424743776; dx=-1.1590616313921875E-7 evalInputDelta=2.9816102928403865E-5
New Minimum: 1.6750847424743776 > 1.6749349812636956
WOLFE (weak): th(1551.1929768229563)=1.6749349812636956; dx=-1.1589226043988309E-7 evalInputDelta=1.7957731361040175E-4
New Minimum: 1.6749349812636956 > 1.6738572014499273
WOLFE (weak): th(10858.350837760694)=1.6738572014499273; dx=-1.1578829652665246E-7 evalInputDelta=0.0012573571273786222
New Minimum: 1.6738572014499273 > 1.6650921206252671
WOLFE (weak): th(86866.80670208555)=1.6650921206252671; dx=-1.1484386389790249E-7 evalInputDelta=0.010022437952038832
New Minimum: 1.6650921206252671 > 1.5883294129816492
WOLFE (weak): th(781801.26031877)=1.5883294129816492; dx=-1.0589308386193267E-7 evalInputDelta=0.08678514559565675
New Minimum: 1.5883294129816492 > 1.1146623745134723
END: th(7818012.6031877)=1.1146623745134723; dx=-4.080380469010944E-8 evalInputDelta=0.5604521840638337
Fitness changed from 1.675114558577306 to 1.1146623745134723
I
...skipping 11643 bytes...
03, gradient 7.066e-05, dot -0.623; [344ba2be-c33e-4d3e-ade5-f32325a58486 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -0.140000107516145, -0.17087883945047233, -0.19694985351381433, -0.2175210537341674
LBFGS Accumulation History: 3 points
Removed measurement 652c7ba2 to history. Total: 4
th(0)=-0.2175210537341674;dx=-4.9924218248957986E-9
New Minimum: -0.2175210537341674 > -0.2304289856564239
WOLF (strong): th(1.827625603235033E7)=-0.2304289856564239; dx=2.365499862727773E-9 evalInputDelta=0.012907931922256477
New Minimum: -0.2304289856564239 > -0.23948080326719978
END: th(9138128.016175166)=-0.23948080326719978; dx=-4.82492360637435E-10 evalInputDelta=0.02195974953303237
Fitness changed from -0.2175210537341674 to -0.23948080326719978
Iteration 14 complete. Error: -0.23948080326719978 Total: 215.1524; Orientation: 2.9162; Line Search: 160.9688
Adding measurement 463aa3da to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.347e+03, gradient 4.770e-05, dot -0.756; [344ba2be-c33e-4d3e-ade5-f32325a58486 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -0.17087883945047233, -0.19694985351381433, -0.2175210537341674, -0.23948080326719978
LBFGS Accumulation History: 3 points
Removed measurement 7a2256bb to history. Total: 4
th(0)=-0.23948080326719978;dx=-2.2755078716655728E-9
New Minimum: -0.23948080326719978 > -0.25920641060408744
WOLF (strong): th(1.9687500000000015E7)=-0.25920641060408744; dx=3.427841883015857E-11 evalInputDelta=0.019725607336887663
END: th(9843750.000000007)=-0.25468304915513507; dx=-9.770916247149704E-10 evalInputDelta=0.015202245887935284
Fitness changed from -0.23948080326719978 to -0.25920641060408744
Iteration 15 complete. Error: -0.25920641060408744 Total: 264.0820; Orientation: 2.8630; Line Search: 206.5088
<a id="p-15"></a>Iteration 15
<a id="p-14"></a>
Final threshold in iteration 15: -0.25920641060408744 (> -Infinity) after 3797.485s (< 3600.000s)
-0.25920641060408744