Code from BasicOptimizer.scala:75 executed in 3645.56 seconds (39.322 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
IterativeTrainer.wrap(trainable)
.setOrientation(orientation())
.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = trainingMonitor.log(msg)
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable, currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable, currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
.setTimeout(trainingMinutes, TimeUnit.MINUTES)
.setMaxIterations(trainingIterations)
.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
.runAndFree
.asInstanceOf[lang.Double]
Reset training subject: 1060670399031500
Reset training subject: 1060700975233000
Adding measurement 141b3d18 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD+Trust
th(0)=-0.5660821124911308;dx=-6.869178560685675E-8
New Minimum: -0.5660821124911308 > -0.5660821869969368
WOLFE (weak): th(2.154434690031884)=-0.5660821869969368; dx=-6.844563405952946E-8 evalInputDelta=7.450580596923828E-8
New Minimum: -0.5660821869969368 > -0.5660823658108711
WOLFE (weak): th(4.308869380063768)=-0.5660823658108711; dx=-6.843935637092656E-8 evalInputDelta=2.5331974029541016E-7
New Minimum: -0.5660823658108711 > -0.566083088517189
WOLFE (weak): th(12.926608140191302)=-0.566083088517189; dx=-6.844611576137764E-8 evalInputDelta=9.760260581970215E-7
New Minimum: -0.566083088517189 > -0.5660858899354935
WOLFE (weak): th(51.70643256076521)=-0.5660858899354935; dx=-6.844570564369768E-8 evalInputDelta=3.777444362640381E-6
New Minimum: -0.5660858899354935 > -0.5661011636257172
WOLFE (weak): th(258.53216280382605)=-0.5661011636257172; dx=-6.844071073613417E-8 evalInputDelta=1.905113458633423E-5
New Minimum: -0.5661011636257172 > -0.566195972263813
WOLFE (weak): th(1551.1929768229563)=-0.566195972263813; dx=-6.843931322020202E-8 evalInputDelta=1.1385977268218994E-4
New Minimum: -0.566195972263813 > -0.5668798685073853
WOLFE (weak): th(10858.350837760694)=-0.5668798685073853; dx=-6.841843477646597E-8 evalInputDelta=7.97756016254425E-4
New Minimum: -0.5668798685073853 > -0.5724527463316917
WOLFE (weak): th(86866.80670208555)=-0.5724527463316917; dx=-6.822220054299394E-8 evalInputDelta=0.006370633840560913
New Minimum: -0.5724527463316917 > -0.6223822981119156
WOLFE (weak): th(781801.26031877)=-0.6223822981119156; dx=-6.592785380605657E-8 evalInputDelta=0.05630018562078476
New Minimum: -0.6223822981119156 > -1.008657194674015
END: th(7818012.6031877)=-1.008657194674015; dx=-4.236964374935977E-8 evalInputDelta=0.4425750821828842
Fitness changed from -0.5660821124911308 to -1
...skipping 24208 bytes...
on magnitude: 1.078e+04, gradient 3.395e-05, dot -0.617; [af1d341f-b9c5-456b-ae0b-3173fb7f688b = 1.000/1.000e+00]
Orientation rejected. Popping history element from -2.5012378469109535, -2.5051437318325043, -2.51191334053874, -2.5153335742652416
LBFGS Accumulation History: 3 points
Removed measurement 45ee6854 to history. Total: 4
th(0)=-2.5153335742652416;dx=-1.151808769958106E-9
Armijo: th(1.0909319176955444E8)=-2.307514436542988; dx=1.8178941803761978E-9 evalInputDelta=-0.2078191377222538
Armijo: th(5.454659588477722E7)=-2.4590324237942696; dx=1.3069720044043857E-9 evalInputDelta=-0.05630115047097206
New Minimum: -2.5153335742652416 > -2.5185775570571423
WOLF (strong): th(1.8182198628259074E7)=-2.5185775570571423; dx=2.7326415134623702E-11 evalInputDelta=0.0032439827919006348
New Minimum: -2.5185775570571423 > -2.5189139433205128
END: th(4545549.6570647685)=-2.5189139433205128; dx=-7.995228688290571E-10 evalInputDelta=0.0035803690552711487
Fitness changed from -2.5153335742652416 to -2.5189139433205128
Iteration 28 complete. Error: -2.5189139433205128 Total: 172.8060; Orientation: 1.6473; Line Search: 143.8709
Adding measurement 4b5a252c to history. Total: 3
Rejected: LBFGS Orientation magnitude: 8.893e+03, gradient 2.554e-05, dot -0.777; [af1d341f-b9c5-456b-ae0b-3173fb7f688b = 1.000/1.000e+00]
Orientation rejected. Popping history element from -2.5051437318325043, -2.51191334053874, -2.5153335742652416, -2.5189139433205128
LBFGS Accumulation History: 3 points
Removed measurement 7f78363f to history. Total: 4
th(0)=-2.5189139433205128;dx=-6.514893454462278E-10
New Minimum: -2.5189139433205128 > -2.5214243568480015
END: th(9793089.86644287)=-2.5214243568480015; dx=-4.941134093916063E-10 evalInputDelta=0.0025104135274887085
Fitness changed from -2.5189139433205128 to -2.5214243568480015
Iteration 29 complete. Error: -2.5214243568480015 Total: 86.4598; Orientation: 1.6455; Line Search: 57.0095
Final threshold in iteration 29: -2.5214243568480015 (> -Infinity) after 3645.561s (< 3600.000s)
-2.5214243568480015