BasicOptimizer.scala:89 executed in 136.17 seconds (1.833 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 116803076083200
Reset training subject: 116805846612600
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
th(0)=-27.99540286529864;dx=-1.2459914667504451E-6
WOLFE (weak): th(2.154434690031884)=-27.99540588595929; dx=-1.1348874531190692E-6 evalInputDelta=3.020660649610818E-6
WOLFE (weak): th(4.308869380063768)=-27.995407396289494; dx=-1.1350330238368056E-6 evalInputDelta=4.530990853623962E-6
WOLFE (weak): th(12.926608140191302)=-27.995419478928188; dx=-1.1348486349427186E-6 evalInputDelta=1.66136295476349E-5
WOLFE (weak): th(51.70643256076521)=-27.99546780943081; dx=-1.1349352477891245E-6 evalInputDelta=6.494413216984185E-5
WOLFE (weak): th(258.53216280382605)=-27.995721543200716; dx=-1.1351949702343818E-6 evalInputDelta=3.186779020758479E-4
WOLFE (weak): th(1551.1929768229563)=-27.997310347618928; dx=-1.135083555092203E-6 evalInputDelta=0.0019074823202878122
WOLFE (weak): th(10858.350837760694)=-28.00875251352694; dx=-1.1347442929551813E-6 evalInputDelta=0.013349648228299316
New Minimum: -28.07879841291134 > -28.1017200593045
WOLFE (weak): th(86866.80670208555)=-28.1017200593045; dx=-1.123056082903479E-6 evalInputDelta=0.10631719400586093
New Minimum: -28.1017200593045 > -28.907258081569864
END: th(781801.26031877)=-28.907258081569864; dx=-1.0275448158758951E-6 evalInputDelta=0.9118552162712241
Fitness changed from -28.07879841291134 to -28.907258081569864
Iteration 1 complete. Error: -28.907258081569864 Total: 35.7206; Orientation: 0.0399; Line Search: 27.5388
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 5225f778 to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-28.907258081569864;dx=-1.0232516972816184E-6
New Minimum: -28.907258081569864 > -30.638677377370215
END: th(1684339.7559414052)=-30.638677377370215; dx=-8.76330769389913E-7 evalInputDelta=1.7314192958003503
Fitness changed from -28.907258081569864 to -30.638677377370215
Iteration 2 complete. Error: -30.638677377370215 Total: 9.2844; Orientation: 0.0684; Line Search: 5.7623
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 6bee9ffd to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-30.638677377370215;dx=-8.567717032120678E-7
New Minimum: -30.638677377370215 > -33.56631766277071
END: th(3628800.0)=-33.56631766277071; dx=-6.333445611216938E-7 evalInputDelta=2.927640285400493
Fitness changed from -30.638677377370215 to -33.56631766277071
Iteration 3 complete. Error: -33.56631766277071 Total: 8.7369; Orientation: 0.0686; Line Search: 5.5171
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 2cbb6dce to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-33.56631766277071;dx=-6.38917788598013E-7
New Minimum: -33.56631766277071 > -37.82200213916729
END: th(7818012.6031877)=-37.82200213916729; dx=-3.797898446079584E-7 evalInputDelta=4.255684476396581
Fitness changed from -33.56631766277071 to -37.82200213916729
Iteration 4 complete. Error: -37.82200213916729 Total: 9.2952; Orientation: 0.1381; Line Search: 5.8840
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement db282c to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.950e+04, gradient 7.119e-04, dot -0.714; [b1405d47-7bf0-43c6-8d20-07abd47b2677 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -37.82200213916729, -33.56631766277071, -30.638677377370215, -28.907258081569864
LBFGS Accumulation History: 3 points
Removed measurement db282c to history. Total: 3
th(0)=-37.82200213916729;dx=-5.065253064936487E-7
New Minimum: -37.82200213916729 > -42.73050463891881
END: th(1.684339755941405E7)=-42.73050463891881; dx=-1.4016427172330546E-7 evalInputDelta=4.9085024997515205
Fitness changed from -37.82200213916729 to -42.73050463891881
Iteration 5 complete. Error: -42.73050463891881 Total: 10.4531; Orientation: 1.2945; Line Search: 5.5708
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 2df20ec8 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.549e+04, gradient 7.816e-04, dot -0.453; [b1405d47-7bf0-43c6-8d20-07abd47b2677 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -42.73050463891881, -33.56631766277071, -30.638677377370215, -28.907258081569864
LBFGS Accumulation History: 3 points
Removed measurement 2df20ec8 to history. Total: 3
th(0)=-42.73050463891881;dx=-6.098046604999055E-7
New Minimum: -42.73050463891881 > -44.01127835417386
WOLF (strong): th(3.6288E7)=-44.01127835417386; dx=8.936875290240549E-8 evalInputDelta=1.2807737152550516
New Minimum: -44.01127835417386 > -45.80535447878292
WOLF (strong): th(1.8144E7)=-45.80535447878292; dx=6.276285369118769E-8 evalInputDelta=3.0748498398641075
END: th(6048000.0)=-45.52461430076235; dx=-2.3004268917970243E-7 evalInputDelta=2.7941096618435424
Fitness changed from -42.73050463891881 to -45.80535447878292
Iteration 6 complete. Error: -45.80535447878292 Total: 15.5424; Orientation: 1.3050; Line Search: 11.0718
Adding measurement 4fbdc498 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.408e+04, gradient 8.359e-04, dot -0.332; [b1405d47-7bf0-43c6-8d20-07abd47b2677 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -45.80535447878292, -33.56631766277071, -30.638677377370215, -28.907258081569864
LBFGS Accumulation History: 3 points
Removed measurement 4fbdc498 to history. Total: 3
th(0)=-45.80535447878292;dx=-6.981693639018981E-7
New Minimum: -45.80535447878292 > -50.41288168717316
WOLF (strong): th(1.3030021005312834E7)=-50.41288168717316; dx=3.5233343817839965E-8 evalInputDelta=4.607527208390245
END: th(6515010.502656417)=-49.535659260064826; dx=-3.250756029063506E-7 evalInputDelta=3.730304781281909
Fitness changed from -45.80535447878292 to -50.41288168717316
Iteration 7 complete. Error: -50.41288168717316 Total: 12.3469; Orientation: 1.2071; Line Search: 8.4524
Adding measurement 34cf2e1b to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.556e+04, gradient 7.074e-04, dot -0.410; [b1405d47-7bf0-43c6-8d20-07abd47b2677 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -50.41288168717316, -33.56631766277071, -30.638677377370215, -28.907258081569864
LBFGS Accumulation History: 3 points
Removed measurement 34cf2e1b to history. Total: 3
th(0)=-50.41288168717316;dx=-5.00172752998204E-7
New Minimum: -50.41288168717316 > -53.05794722014609
WOLF (strong): th(1.4036164632845046E7)=-53.05794722014609; dx=7.594170697732942E-8 evalInputDelta=2.6450655329729287
END: th(7018082.316422523)=-52.94441457223649; dx=-1.5757403017958867E-7 evalInputDelta=2.531532885063328
Fitness changed from -50.41288168717316 to -53.05794722014609
Iteration 8 complete. Error: -53.05794722014609 Total: 11.9316; Orientation: 1.1348; Line Search: 8.1963
Adding measurement 16bd3205 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.743e+04, gradient 7.298e-04, dot -0.373; [b1405d47-7bf0-43c6-8d20-07abd47b2677 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -53.05794722014609, -33.56631766277071, -30.638677377370215, -28.907258081569864
LBFGS Accumulation History: 3 points
Removed measurement 16bd3205 to history. Total: 3
th(0)=-53.05794722014609;dx=-5.324207121239068E-7
New Minimum: -53.05794722014609 > -55.216733025084885
WOLF (strong): th(1.5120000000000004E7)=-55.216733025084885; dx=1.5245236508635895E-7 evalInputDelta=2.1587858049387947
New Minimum: -55.216733025084885 > -55.74141735495554
END: th(7560000.000000002)=-55.74141735495554; dx=-1.0022427009559582E-7 evalInputDelta=2.683470134809447
Fitness changed from -53.05794722014609 to -55.74141735495554
Iteration 9 complete. Error: -55.74141735495554 Total: 12.6430; Orientation: 1.3338; Line Search: 8.4101
Adding measurement 272eeb40 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.747e+04, gradient 5.139e-04, dot -0.544; [b1405d47-7bf0-43c6-8d20-07abd47b2677 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -55.74141735495554, -33.56631766277071, -30.638677377370215, -28.907258081569864
LBFGS Accumulation History: 3 points
Removed measurement 272eeb40 to history. Total: 3
th(0)=-55.74141735495554;dx=-2.635617485431146E-7
New Minimum: -55.74141735495554 > -57.78636133398414
END: th(1.6287526256641045E7)=-57.78636133398414; dx=-2.7573410563179895E-8 evalInputDelta=2.044943979028602
Fitness changed from -55.74141735495554 to -57.78636133398414
Iteration 10 complete. Error: -57.78636133398414 Total: 9.7073; Orientation: 1.1837; Line Search: 5.5535
<a id="p-13"></a>Iteration 10
<a id="p-12"></a>
Final threshold in iteration 11: -57.78636133398414 (> -Infinity) after 136.174s (< 5400.000s)
Returns
-57.78636133398414