BasicOptimizer.scala:89 executed in 234.42 seconds (2.992 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 5397034800520
Reset training subject: 5401982066194
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
th(0)=-14.24576935172081;dx=-2.8908361733193523E-6
WOLFE (weak): th(2.154434690031884)=-14.245775550603867; dx=-2.7590821574326775E-6 evalInputDelta=6.198883056640625E-6
WOLFE (weak): th(4.308869380063768)=-14.245783269405365; dx=-2.7593009398698055E-6 evalInputDelta=1.3917684555053711E-5
WOLFE (weak): th(12.926608140191302)=-14.245812356472015; dx=-2.759097207423166E-6 evalInputDelta=4.3004751205444336E-5
WOLFE (weak): th(51.70643256076521)=-14.245942056179047; dx=-2.7591721248980826E-6 evalInputDelta=1.7270445823669434E-4
WOLFE (weak): th(258.53216280382605)=-14.246634691953659; dx=-2.758506792893882E-6 evalInputDelta=8.653402328491211E-4
WOLFE (weak): th(1551.1929768229563)=-14.250957041978836; dx=-2.753857445230423E-6 evalInputDelta=0.005187690258026123
WOLFE (weak): th(10858.350837760694)=-14.281781166791916; dx=-2.7063744806049706E-6 evalInputDelta=0.03601181507110596
New Minimum: -14.324571043252945 > -14.513157039880753
END: th(86866.80670208555)=-14.513157039880753; dx=-2.3021752436567612E-6 evalInputDelta=0.2673876881599426
Fitness changed from -14.324571043252945 to -14.513157039880753
Iteration 1 complete. Error: -14.513157039880753 Total: 60.3777; Orientation: 0.0206; Line Search: 45.5421
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 5b1aa266 to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-14.513157039880753;dx=-2.1111915421883583E-6
New Minimum: -14.513157039880753 > -14.918286383152008
END: th(187148.86177126726)=-14.918286383152008; dx=-1.4466586477584863E-6 evalInputDelta=0.4051293432712555
Fitness changed from -14.513157039880753 to -14.918286383152008
Iteration 2 complete. Error: -14.918286383152008 Total: 15.4989; Orientation: 0.0289; Line Search: 10.0620
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 6ba29e72 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-14.918286383152008;dx=-1.3834028563378996E-6
New Minimum: -14.918286383152008 > -15.455065041780472
END: th(403200.00000000006)=-15.455065041780472; dx=-8.659350385151867E-7 evalInputDelta=0.5367786586284637
Fitness changed from -14.918286383152008 to -15.455065041780472
Iteration 3 complete. Error: -15.455065041780472 Total: 15.3150; Orientation: 0.0282; Line Search: 10.0210
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 24552237 to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-15.455065041780472;dx=-1.079820529663885E-6
New Minimum: -15.455065041780472 > -16.081572502851486
END: th(868668.0670208557)=-16.081572502851486; dx=-4.130848006652917E-7 evalInputDelta=0.6265074610710144
Fitness changed from -15.455065041780472 to -16.081572502851486
Iteration 4 complete. Error: -16.081572502851486 Total: 15.3694; Orientation: 0.0284; Line Search: 10.0792
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 3a5fd9b2 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.263e+03, gradient 1.055e-03, dot -0.651; [6e6dba44-6634-47af-bf3c-41afd48b648c = 1.000/1.000e+00]
Orientation rejected. Popping history element from -16.081572502851486, -15.455065041780472, -14.918286383152008, -14.513157039880753
LBFGS Accumulation History: 3 points
Removed measurement 3a5fd9b2 to history. Total: 3
th(0)=-16.081572502851486;dx=-1.1125663309739399E-6
New Minimum: -16.081572502851486 > -16.530060172080994
END: th(1871488.617712673)=-16.530060172080994; dx=-5.4652024977783414E-9 evalInputDelta=0.44848766922950745
Fitness changed from -16.081572502851486 to -16.530060172080994
Iteration 5 complete. Error: -16.530060172080994 Total: 16.4089; Orientation: 0.6345; Line Search: 10.5099
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 52a49aa2 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.335e+03, gradient 1.178e-03, dot -0.423; [6e6dba44-6634-47af-bf3c-41afd48b648c = 1.000/1.000e+00]
Orientation rejected. Popping history element from -16.530060172080994, -15.455065041780472, -14.918286383152008, -14.513157039880753
LBFGS Accumulation History: 3 points
Removed measurement 52a49aa2 to history. Total: 3
th(0)=-16.530060172080994;dx=-1.387393782232656E-6
Armijo: th(4032000.0000000014)=-16.1190282702446; dx=2.5042565250675954E-7 evalInputDelta=-0.41103190183639526
New Minimum: -16.530060172080994 > -16.885675221681595
WOLF (strong): th(2016000.0000000007)=-16.885675221681595; dx=3.0023708366122944E-7 evalInputDelta=0.3556150496006012
New Minimum: -16.885675221681595 > -17.13318994641304
END: th(672000.0000000002)=-17.13318994641304; dx=-2.451631220491784E-7 evalInputDelta=0.6031297743320465
Fitness changed from -16.530060172080994 to -17.13318994641304
Iteration 6 complete. Error: -17.13318994641304 Total: 26.5524; Orientation: 0.5915; Line Search: 20.6114
Adding measurement 5495a576 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.798e+03, gradient 7.886e-04, dot -0.497; [6e6dba44-6634-47af-bf3c-41afd48b648c = 1.000/1.000e+00]
Orientation rejected. Popping history element from -17.13318994641304, -15.455065041780472, -14.918286383152008, -14.513157039880753
LBFGS Accumulation History: 3 points
Removed measurement 5495a576 to history. Total: 3
th(0)=-17.13318994641304;dx=-6.214364961826856E-7
New Minimum: -17.13318994641304 > -17.556320518255234
WOLF (strong): th(1447780.1117014263)=-17.556320518255234; dx=1.176892015651805E-8 evalInputDelta=0.4231305718421936
END: th(723890.0558507132)=-17.490681290626526; dx=-2.117387160553151E-7 evalInputDelta=0.3574913442134857
Fitness changed from -17.13318994641304 to -17.556320518255234
Iteration 7 complete. Error: -17.556320518255234 Total: 20.8160; Orientation: 0.6061; Line Search: 14.9043
Adding measurement 3fa9a42f to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.999e+03, gradient 8.379e-04, dot -0.437; [6e6dba44-6634-47af-bf3c-41afd48b648c = 1.000/1.000e+00]
Orientation rejected. Popping history element from -17.556320518255234, -15.455065041780472, -14.918286383152008, -14.513157039880753
LBFGS Accumulation History: 3 points
Removed measurement 3fa9a42f to history. Total: 3
th(0)=-17.556320518255234;dx=-7.013847571706401E-7
New Minimum: -17.556320518255234 > -17.83748808503151
WOLF (strong): th(1559573.8480938943)=-17.83748808503151; dx=2.2635957525827058E-7 evalInputDelta=0.28116756677627563
New Minimum: -17.83748808503151 > -17.931502401828766
END: th(779786.9240469472)=-17.931502401828766; dx=-1.1361053428978726E-7 evalInputDelta=0.3751818835735321
Fitness changed from -17.556320518255234 to -17.931502401828766
Iteration 8 complete. Error: -17.931502401828766 Total: 20.9114; Orientation: 0.6249; Line Search: 14.8727
Adding measurement 536c0bba to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.349e+03, gradient 5.663e-04, dot -0.598; [6e6dba44-6634-47af-bf3c-41afd48b648c = 1.000/1.000e+00]
Orientation rejected. Popping history element from -17.931502401828766, -15.455065041780472, -14.918286383152008, -14.513157039880753
LBFGS Accumulation History: 3 points
Removed measurement 536c0bba to history. Total: 3
th(0)=-17.931502401828766;dx=-3.2013168201263636E-7
New Minimum: -17.931502401828766 > -18.183723509311676
END: th(1680000.0000000007)=-18.183723509311676; dx=-5.843877121963155E-9 evalInputDelta=0.25222110748291016
Fitness changed from -17.931502401828766 to -18.183723509311676
Iteration 9 complete. Error: -18.183723509311676 Total: 15.9904; Orientation: 0.6099; Line Search: 10.4526
Adding measurement 5562bfc6 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.452e+03, gradient 7.046e-04, dot -0.457; [6e6dba44-6634-47af-bf3c-41afd48b648c = 1.000/1.000e+00]
Orientation rejected. Popping history element from -18.183723509311676, -15.455065041780472, -14.918286383152008, -14.513157039880753
LBFGS Accumulation History: 3 points
Removed measurement 5562bfc6 to history. Total: 3
th(0)=-18.183723509311676;dx=-4.961044962406993E-7
Armijo: th(3619450.279253566)=-17.841354608535767; dx=1.911439861827258E-7 evalInputDelta=-0.3423689007759094
New Minimum: -18.183723509311676 > -18.32092523574829
WOLF (strong): th(1809725.139626783)=-18.32092523574829; dx=1.8039153102486842E-7 evalInputDelta=0.137201726436615
New Minimum: -18.32092523574829 > -18.426321297883987
END: th(603241.7132089277)=-18.426321297883987; dx=-1.656349546019396E-7 evalInputDelta=0.2425977885723114
Fitness changed from -18.183723509311676 to -18.426321297883987
Iteration 10 complete. Error: -18.426321297883987 Total: 26.8848; Orientation: 0.6153; Line Search: 21.2490
<a id="p-13"></a>Iteration 10
<a id="p-12"></a>
Final threshold in iteration 11: -18.426321297883987 (> -Infinity) after 234.416s (< 5400.000s)
Returns
-18.426321297883987