Code from BasicOptimizer.scala:75 executed in 3592.25 seconds (16.607 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
IterativeTrainer.wrap(trainable)
.setOrientation(orientation())
.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = trainingMonitor.log(msg)
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable, currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable, currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
.setTimeout(trainingMinutes, TimeUnit.MINUTES)
.setMaxIterations(trainingIterations)
.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
.runAndFree
.asInstanceOf[lang.Double]
Reset training subject: 2735939890889
Adjust for PipelineNetwork : 298005.22765704326
Mask: [600, 750, 64]
Adjust for PipelineNetwork : 7985481.309680817
Mask: [600, 750, 64]
Adjust for PipelineNetwork : 9982600.092082508
Mask: [300, 375, 128]
Adjust for PipelineNetwork : 1.371940750725157E7
Mask: [300, 375, 128]
Adjust for PipelineNetwork : 4.495001526577726E7
Mask: [150, 188, 256]
Adjust for PipelineNetwork : 4.944768248455829E7
Mask: [150, 188, 256]
Adjust for PipelineNetwork : 2.2575599333826378E7
Mask: [150, 188, 256]
Reset training subject: 2843597494597
Adding measurement a681fee to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD+Trust
th(0)=11.664540641009808;dx=-2.6939385780940623E-7
New Minimum: 11.664540641009808 > 11.664540015161037
WOLFE (weak): th(2.154434690031884)=11.664540015161037; dx=-2.534351831476199E-7 evalInputDelta=6.258487701416016E-7
New Minimum: 11.664540015161037 > 11.664539854973555
WOLFE (weak): th(4.308869380063768)=11.664539854973555; dx=-2.534361436207555E-7 evalInputDelta=7.860362529754639E-7
New Minimum: 11.664539854973555 > 11.664537362754345
WOLFE (weak): th(12.926608140191302)=11.664537362754345; dx=-2.5343630146524926E-7 evalInputDelta=3.2782554626464844E-6
New Minimum: 11.664537362754345 > 11.664527736604214
WOLFE (weak): th(51.70643256076521)=11.664527736604214; dx=-2.5343565972184963E-7 evalInputDelta=1.290440559387207E-5
New Minimum: 11.664527736604214 > 11.664476320147514
WOLFE (weak): th(258.53216280382605)=11.664476320147514; dx=-2.5344443231878573E-7 evalInputDelta=6.432086229324341E-5
New Minimum: 11.664476320147514 > 11.66415623575449
WOLFE (weak): th(1551.1929768229563)=11.66415623575449; dx=-2.534997406251867E-7 evalInputDelta=3.84405255317688E-4
New Minimum: 11.66415623575449 > 11.661848902702332
WOLFE (weak): th(10858.350837760694)=11.661848902702332; dx=-2.5366934592902804E-7 evalInputDelta=0.0026917383074760437
New Minimum: 11.661848902702332 > 11.642923664301634
WOLFE (weak): th(86866.80670208555)=11.64292366430
...skipping 24855 bytes...
538034170866
END: th(4545549.6570647685)=3.17538034170866; dx=-7.040540629520599E-9 evalInputDelta=0.027153879404067993
Fitness changed from 3.202534221112728 to 3.17538034170866
Iteration 28 complete. Error: 3.17538034170866 Total: 139.2698; Orientation: 2.3639; Line Search: 110.6382
Adding measurement 28fe4d0f to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.291e+03, gradient 9.570e-05, dot -0.857; [c4f9907b-35e9-4051-831e-7b4f88f5dc88 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 3.25903033465147, 3.2407342195510864, 3.202534221112728, 3.17538034170866
LBFGS Accumulation History: 3 points
Removed measurement 1e35ad2 to history. Total: 4
th(0)=3.17538034170866;dx=-9.158939608620289E-9
New Minimum: 3.17538034170866 > 3.1548159196972847
END: th(9793089.86644287)=3.1548159196972847; dx=-6.464830394077032E-9 evalInputDelta=0.020564422011375427
Fitness changed from 3.17538034170866 to 3.1548159196972847
Iteration 29 complete. Error: 3.1548159196972847 Total: 84.0932; Orientation: 2.3734; Line Search: 54.5963
Adding measurement 67e363d9 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.753e+04, gradient 9.471e-05, dot -0.853; [c4f9907b-35e9-4051-831e-7b4f88f5dc88 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 3.2407342195510864, 3.202534221112728, 3.17538034170866, 3.1548159196972847
LBFGS Accumulation History: 3 points
Removed measurement 577e454 to history. Total: 4
th(0)=3.1548159196972847;dx=-8.970038552973006E-9
New Minimum: 3.1548159196972847 > 3.1155971735715866
END: th(2.1098572530864228E7)=3.1155971735715866; dx=-5.760997742674246E-9 evalInputDelta=0.03921874612569809
Fitness changed from 3.1548159196972847 to 3.1155971735715866
Iteration 30 complete. Error: 3.1155971735715866 Total: 83.1517; Orientation: 2.3477; Line Search: 54.0530
<a id="p-21"></a>Iteration 30
<a id="p-20"></a>
Final threshold in iteration 31: 3.1155971735715866 (> -Infinity) after 3592.243s (< 10800.000s)
3.1155971735715866