BasicOptimizer.scala:89 executed in 232.10 seconds (4.251 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 689671382979900
Reset training subject: 689676060251500
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
New Minimum: -834.953554097472 > -835.4462879204144
th(0)=-835.4462879204144;dx=-0.00777564387879767
New Minimum: -835.4462879204144 > -835.4632696893383
WOLFE (weak): th(2.154434690031884)=-835.4632696893383; dx=-0.007151452253302383 evalInputDelta=0.016981768923869822
New Minimum: -835.4632696893383 > -835.4801763058175
WOLFE (weak): th(4.308869380063768)=-835.4801763058175; dx=-0.007165382152836238 evalInputDelta=0.03388838540308825
New Minimum: -835.4801763058175 > -835.5478741520441
WOLFE (weak): th(12.926608140191302)=-835.5478741520441; dx=-0.007162700705822889 evalInputDelta=0.10158623162965341
New Minimum: -835.5478741520441 > -835.8524839946341
WOLFE (weak): th(51.70643256076521)=-835.8524839946341; dx=-0.007162384558050707 evalInputDelta=0.40619607421967885
New Minimum: -835.8524839946341 > -837.4742906501667
WOLFE (weak): th(258.53216280382605)=-837.4742906501667; dx=-0.0071380909404023815 evalInputDelta=2.028002729752302
New Minimum: -837.4742906501667 > -847.5119836910862
WOLFE (weak): th(1551.1929768229563)=-847.5119836910862; dx=-0.007023423946894763 evalInputDelta=12.065695770671823
New Minimum: -847.5119836910862 > -914.213425574138
END: th(10858.350837760694)=-914.213425574138; dx=-0.005976726693633836 evalInputDelta=78.76713765372358
Fitness changed from -834.953554097472 to -914.213425574138
Iteration 1 complete. Error: -914.213425574138 Total: 59.6689; Orientation: 0.0704; Line Search: 45.9011
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 23884344 to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-914.213425574138;dx=-0.005546674789793872
New Minimum: -914.213425574138 > -1036.9449117479674
END: th(23393.607721408407)=-1036.9449117479674; dx=-0.003978594453049 evalInputDelta=122.73148617382935
Fitness changed from -914.213425574138 to -1036.9449117479674
Iteration 2 complete. Error: -1036.9449117479674 Total: 15.6510; Orientation: 0.1003; Line Search: 9.9713
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 38fe6051 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-1036.9449117479674;dx=-0.003841147051826074
New Minimum: -1036.9449117479674 > -1203.2520517331354
END: th(50400.00000000001)=-1203.2520517331354; dx=-0.0022540568155670123 evalInputDelta=166.30713998516808
Fitness changed from -1036.9449117479674 to -1203.2520517331354
Iteration 3 complete. Error: -1203.2520517331354 Total: 16.8825; Orientation: 0.1163; Line Search: 10.4843
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 270b369c to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-1203.2520517331354;dx=-0.00287242936995679
New Minimum: -1203.2520517331354 > -1387.8500459343581
END: th(108583.50837760697)=-1387.8500459343581; dx=-9.136978821681812E-4 evalInputDelta=184.59799420122272
Fitness changed from -1203.2520517331354 to -1387.8500459343581
Iteration 4 complete. Error: -1387.8500459343581 Total: 15.4444; Orientation: 0.1013; Line Search: 9.4779
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 7412d63d to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.510e+04, gradient 6.040e-02, dot -0.526; [3efbabe4-e775-4b18-86b9-ec26f3dba1b1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1387.8500459343581, -1203.2520517331354, -1036.9449117479674, -914.213425574138
LBFGS Accumulation History: 3 points
Removed measurement 7412d63d to history. Total: 3
th(0)=-1387.8500459343581;dx=-0.0036397690768332465
New Minimum: -1387.8500459343581 > -1483.7171226349044
WOLF (strong): th(233936.07721408413)=-1483.7171226349044; dx=1.7439914364671117E-4 evalInputDelta=95.86707670054625
New Minimum: -1483.7171226349044 > -1508.647904582113
END: th(116968.03860704206)=-1508.647904582113; dx=-4.4002325928956326E-5 evalInputDelta=120.79785864775477
Fitness changed from -1387.8500459343581 to -1508.647904582113
Iteration 5 complete. Error: -1508.647904582113 Total: 21.9198; Orientation: 1.8193; Line Search: 14.3485
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 1dda1790 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.639e+04, gradient 6.412e-02, dot -0.419; [3efbabe4-e775-4b18-86b9-ec26f3dba1b1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1508.647904582113, -1203.2520517331354, -1036.9449117479674, -914.213425574138
LBFGS Accumulation History: 3 points
Removed measurement 1dda1790 to history. Total: 3
th(0)=-1508.647904582113;dx=-0.004102675958137457
New Minimum: -1508.647904582113 > -1542.7370158260935
WOLF (strong): th(252000.0000000001)=-1542.7370158260935; dx=5.82315586941123E-4 evalInputDelta=34.0891112439806
New Minimum: -1542.7370158260935 > -1644.445803302742
WOLF (strong): th(126000.00000000004)=-1644.445803302742; dx=4.8370066361868723E-4 evalInputDelta=135.79789872062906
END: th(42000.000000000015)=-1634.733617443527; dx=-0.0014179828976083049 evalInputDelta=126.0857128614141
Fitness changed from -1508.647904582113 to -1644.445803302742
Iteration 6 complete. Error: -1644.445803302742 Total: 25.6500; Orientation: 1.7201; Line Search: 18.5091
Adding measurement 15faf7 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.903e+04, gradient 6.742e-02, dot -0.356; [3efbabe4-e775-4b18-86b9-ec26f3dba1b1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1644.445803302742, -1203.2520517331354, -1036.9449117479674, -914.213425574138
LBFGS Accumulation History: 3 points
Removed measurement 15faf7 to history. Total: 3
th(0)=-1644.445803302742;dx=-0.00453963574869872
New Minimum: -1644.445803302742 > -1814.9526853336977
WOLF (strong): th(90486.25698133915)=-1814.9526853336977; dx=1.3374096700201154E-4 evalInputDelta=170.5068820309557
END: th(45243.12849066957)=-1792.8916587457259; dx=-0.0014358520785711683 evalInputDelta=148.44585544298388
Fitness changed from -1644.445803302742 to -1814.9526853336977
Iteration 7 complete. Error: -1814.9526853336977 Total: 19.8203; Orientation: 1.6267; Line Search: 13.8583
Adding measurement 6ce8c930 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.031e+04, gradient 5.975e-02, dot -0.408; [3efbabe4-e775-4b18-86b9-ec26f3dba1b1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1814.9526853336977, -1203.2520517331354, -1036.9449117479674, -914.213425574138
LBFGS Accumulation History: 3 points
Removed measurement 6ce8c930 to history. Total: 3
th(0)=-1814.9526853336977;dx=-0.0035624953560015743
New Minimum: -1814.9526853336977 > -1934.0080144611604
WOLF (strong): th(97473.3655058684)=-1934.0080144611604; dx=2.6975339794557847E-4 evalInputDelta=119.05532912746276
END: th(48736.6827529342)=-1926.3710701731377; dx=-8.831301667272403E-4 evalInputDelta=111.41838483944002
Fitness changed from -1814.9526853336977 to -1934.0080144611604
Iteration 8 complete. Error: -1934.0080144611604 Total: 20.3350; Orientation: 1.6846; Line Search: 13.9111
Adding measurement 47dabc6b to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.755e+04, gradient 6.090e-02, dot -0.394; [3efbabe4-e775-4b18-86b9-ec26f3dba1b1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1934.0080144611604, -1203.2520517331354, -1036.9449117479674, -914.213425574138
LBFGS Accumulation History: 3 points
Removed measurement 47dabc6b to history. Total: 3
th(0)=-1934.0080144611604;dx=-0.0036998539631465464
New Minimum: -1934.0080144611604 > -2036.9255631956705
WOLF (strong): th(105000.00000000004)=-2036.9255631956705; dx=5.944730658360731E-4 evalInputDelta=102.91754873451009
New Minimum: -2036.9255631956705 > -2048.811484739384
END: th(52500.00000000002)=-2048.811484739384; dx=-5.899282566830315E-4 evalInputDelta=114.80347027822359
Fitness changed from -1934.0080144611604 to -2048.811484739384
Iteration 9 complete. Error: -2048.811484739384 Total: 20.3870; Orientation: 1.7657; Line Search: 14.1038
Adding measurement 5162cd22 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 5.110e+04, gradient 4.625e-02, dot -0.530; [3efbabe4-e775-4b18-86b9-ec26f3dba1b1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -2048.811484739384, -1203.2520517331354, -1036.9449117479674, -914.213425574138
LBFGS Accumulation History: 3 points
Removed measurement 5162cd22 to history. Total: 3
th(0)=-2048.811484739384;dx=-0.0021393753986046595
New Minimum: -2048.811484739384 > -2144.2663080876873
END: th(113107.82122667394)=-2144.2663080876873; dx=-1.598672370166782E-4 evalInputDelta=95.45482334830331
Fitness changed from -2048.811484739384 to -2144.2663080876873
Iteration 10 complete. Error: -2144.2663080876873 Total: 15.2990; Orientation: 1.6259; Line Search: 9.3162
<a id="p-13"></a>Iteration 10
<a id="p-12"></a>
Final threshold in iteration 11: -2144.2663080876873 (> -Infinity) after 232.098s (< 5400.000s)
Returns
-2144.2663080876873