Code from BasicOptimizer.scala:75 executed in 1005.68 seconds (11.243 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
IterativeTrainer.wrap(trainable)
.setOrientation(orientation())
.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = trainingMonitor.log(msg)
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable, currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable, currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
.setTimeout(trainingMinutes, TimeUnit.MINUTES)
.setMaxIterations(trainingIterations)
.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
.runAndFree
.asInstanceOf[lang.Double]
Reset training subject: 1155958954464700
Reset training subject: 1155967452331900
Adding measurement 1f57666b to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD+Trust
th(0)=5.143413468555313;dx=-2.48026213069903E-5
New Minimum: 5.143413468555313 > 5.143390257434146
WOLFE (weak): th(2.154434690031884)=5.143390257434146; dx=-2.4802556277336414E-5 evalInputDelta=2.3211121166433202E-5
New Minimum: 5.143390257434146 > 5.143367082306687
WOLFE (weak): th(4.308869380063768)=5.143367082306687; dx=-2.480229271197901E-5 evalInputDelta=4.638624862529639E-5
New Minimum: 5.143367082306687 > 5.143274227243845
WOLFE (weak): th(12.926608140191302)=5.143274227243845; dx=-2.480069096693412E-5 evalInputDelta=1.3924131146758612E-4
New Minimum: 5.143274227243845 > 5.142856015801726
WOLFE (weak): th(51.70643256076521)=5.142856015801726; dx=-2.479337757595612E-5 evalInputDelta=5.574527535863183E-4
New Minimum: 5.142856015801726 > 5.140627730195788
WOLFE (weak): th(258.53216280382605)=5.140627730195788; dx=-2.4773897057876198E-5 evalInputDelta=0.002785738359524892
New Minimum: 5.140627730195788 > 5.126754853747724
WOLFE (weak): th(1551.1929768229563)=5.126754853747724; dx=-2.458081114790671E-5 evalInputDelta=0.016658614807588457
New Minimum: 5.126754853747724 > 5.030315722578781
WOLFE (weak): th(10858.350837760694)=5.030315722578781; dx=-2.3150176010206888E-5 evalInputDelta=0.11309774597653188
New Minimum: 5.030315722578781 > 4.425706033355787
END: th(86866.80670208555)=4.425706033355787; dx=-1.4652474908750666E-5 evalInputDelta=0.7177074351995252
Fitness changed from 5.143413468555313 to 4.425706033355787
Iteration 1 complete. Error: 4.425706033355787 Total: 99.3910; Orientation: 0.0075; Line Search: 74.4166
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 1e9d721 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=4.425706033355787;dx=-1.1072687569252486E-5
New Minimum: 4.425
...skipping 24028 bytes...
history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.714e+03, gradient 2.248e-04, dot -0.646; [d87ae145-db22-4c42-ba09-e77afe278591 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.038613379758474786, 0.02562828524858196, 0.004139098493531113, -0.01400809725944241
LBFGS Accumulation History: 3 points
Removed measurement 51e1e058 to history. Total: 4
th(0)=-0.01400809725944241;dx=-5.054562107434737E-8
Armijo: th(5222981.262102865)=3.501079812253671E-4; dx=4.30017219463116E-9 evalInputDelta=-0.014358205240667776
New Minimum: -0.01400809725944241 > -0.027237287997306292
END: th(2611490.6310514323)=-0.027237287997306292; dx=-1.5126287786352908E-8 evalInputDelta=0.013229190737863883
Fitness changed from -0.01400809725944241 to -0.027237287997306292
Iteration 29 complete. Error: -0.027237287997306292 Total: 33.4221; Orientation: 0.1944; Line Search: 24.9740
Adding measurement 698e1bf5 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.716e+03, gradient 2.232e-04, dot -0.708; [d87ae145-db22-4c42-ba09-e77afe278591 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.02562828524858196, 0.004139098493531113, -0.01400809725944241, -0.027237287997306292
LBFGS Accumulation History: 3 points
Removed measurement 32fd5bc to history. Total: 4
th(0)=-0.027237287997306292;dx=-4.98005858981598E-8
Armijo: th(5626286.008230462)=0.002426404652626113; dx=1.9098506376345837E-8 evalInputDelta=-0.029663692649932405
New Minimum: -0.027237287997306292 > -0.03989806164604537
END: th(2813143.004115231)=-0.03989806164604537; dx=-7.830416466720779E-9 evalInputDelta=0.012660773648739077
Fitness changed from -0.027237287997306292 to -0.03989806164604537
Iteration 30 complete. Error: -0.03989806164604537 Total: 33.4659; Orientation: 0.1973; Line Search: 25.0399
<a id="p-21"></a>Iteration 30
<a id="p-20"></a>
Final threshold in iteration 31: -0.03989806164604537 (> -Infinity) after 1005.675s (< 3600.000s)
-0.03989806164604537