BasicOptimizer.scala:89 executed in 228.20 seconds (2.573 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 8935089812695
Reset training subject: 8940187728928
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
th(0)=-12.851182073354721;dx=-3.800343152282076E-6
WOLFE (weak): th(2.154434690031884)=-12.851192027330399; dx=-3.799499456018697E-6 evalInputDelta=9.953975677490234E-6
WOLFE (weak): th(4.308869380063768)=-12.851202368736267; dx=-3.7990695251080997E-6 evalInputDelta=2.0295381546020508E-5
WOLFE (weak): th(12.926608140191302)=-12.851241707801819; dx=-3.7992325280888235E-6 evalInputDelta=5.963444709777832E-5
WOLFE (weak): th(51.70643256076521)=-12.851418375968933; dx=-3.798336778540777E-6 evalInputDelta=2.3630261421203613E-4
WOLFE (weak): th(258.53216280382605)=-12.852362006902695; dx=-3.798153740786301E-6 evalInputDelta=0.0011799335479736328
WOLFE (weak): th(1551.1929768229563)=-12.858252316713333; dx=-3.7916691468327267E-6 evalInputDelta=0.0070702433586120605
WOLFE (weak): th(10858.350837760694)=-12.900359511375427; dx=-3.7424602733551676E-6 evalInputDelta=0.04917743802070618
New Minimum: -13.041307717561722 > -13.219350337982178
END: th(86866.80670208555)=-13.219350337982178; dx=-3.2342170898475237E-6 evalInputDelta=0.36816826462745667
Fitness changed from -13.041307717561722 to -13.219350337982178
Iteration 1 complete. Error: -13.219350337982178 Total: 60.3507; Orientation: 0.0213; Line Search: 44.5390
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 4a46558b to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-13.219350337982178;dx=-3.048890961971669E-6
New Minimum: -13.219350337982178 > -13.81580176949501
END: th(187148.86177126726)=-13.81580176949501; dx=-2.249249678689962E-6 evalInputDelta=0.5964514315128326
Fitness changed from -13.219350337982178 to -13.81580176949501
Iteration 2 complete. Error: -13.81580176949501 Total: 15.7820; Orientation: 0.0300; Line Search: 10.4436
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 50ed9bad to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-13.81580176949501;dx=-2.0201398752449935E-6
New Minimum: -13.81580176949501 > -14.595375120639801
END: th(403200.00000000006)=-14.595375120639801; dx=-1.2448015260379337E-6 evalInputDelta=0.7795733511447906
Fitness changed from -13.81580176949501 to -14.595375120639801
Iteration 3 complete. Error: -14.595375120639801 Total: 14.8291; Orientation: 0.0301; Line Search: 9.7545
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 76f253d0 to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-14.595375120639801;dx=-1.250422831351565E-6
New Minimum: -14.595375120639801 > -15.42814987897873
END: th(868668.0670208557)=-15.42814987897873; dx=-5.597667847489913E-7 evalInputDelta=0.8327747583389282
Fitness changed from -14.595375120639801 to -15.42814987897873
Iteration 4 complete. Error: -15.42814987897873 Total: 15.8787; Orientation: 0.0303; Line Search: 10.5664
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 7f817a13 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.734e+03, gradient 9.350e-04, dot -0.730; [72165824-7f25-4ea9-b7d1-ef99d6497141 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -15.42814987897873, -14.595375120639801, -13.81580176949501, -13.219350337982178
LBFGS Accumulation History: 3 points
Removed measurement 7f817a13 to history. Total: 3
th(0)=-15.42814987897873;dx=-8.734871409128795E-7
New Minimum: -15.42814987897873 > -16.18226581811905
END: th(1871488.617712673)=-16.18226581811905; dx=-1.3543158173639235E-7 evalInputDelta=0.7541159391403198
Fitness changed from -15.42814987897873 to -16.18226581811905
Iteration 5 complete. Error: -16.18226581811905 Total: 16.1557; Orientation: 0.6441; Line Search: 10.2370
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 4c0695b5 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.579e+03, gradient 9.612e-04, dot -0.492; [72165824-7f25-4ea9-b7d1-ef99d6497141 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -16.18226581811905, -14.595375120639801, -13.81580176949501, -13.219350337982178
LBFGS Accumulation History: 3 points
Removed measurement 4c0695b5 to history. Total: 3
th(0)=-16.18226581811905;dx=-9.233433344569323E-7
Armijo: th(4032000.0000000014)=-15.767147779464722; dx=2.843469459896037E-7 evalInputDelta=-0.4151180386543274
New Minimum: -16.18226581811905 > -16.503198623657227
WOLF (strong): th(2016000.0000000007)=-16.503198623657227; dx=2.3259822227831593E-7 evalInputDelta=0.3209328055381775
New Minimum: -16.503198623657227 > -16.627759039402008
END: th(672000.0000000002)=-16.627759039402008; dx=-2.395104482801842E-7 evalInputDelta=0.445493221282959
Fitness changed from -16.18226581811905 to -16.627759039402008
Iteration 6 complete. Error: -16.627759039402008 Total: 26.0263; Orientation: 0.6187; Line Search: 19.7231
Adding measurement 34138319 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.863e+03, gradient 6.607e-04, dot -0.617; [72165824-7f25-4ea9-b7d1-ef99d6497141 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -16.627759039402008, -14.595375120639801, -13.81580176949501, -13.219350337982178
LBFGS Accumulation History: 3 points
Removed measurement 34138319 to history. Total: 3
th(0)=-16.627759039402008;dx=-4.359060491021517E-7
New Minimum: -16.627759039402008 > -17.045693039894104
END: th(1447780.1117014263)=-17.045693039894104; dx=-1.084282662033194E-7 evalInputDelta=0.41793400049209595
Fitness changed from -16.627759039402008 to -17.045693039894104
Iteration 7 complete. Error: -17.045693039894104 Total: 16.1079; Orientation: 0.9674; Line Search: 10.1155
Adding measurement 28ed9e46 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.271e+03, gradient 7.036e-04, dot -0.557; [72165824-7f25-4ea9-b7d1-ef99d6497141 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -17.045693039894104, -14.595375120639801, -13.81580176949501, -13.219350337982178
LBFGS Accumulation History: 3 points
Removed measurement 28ed9e46 to history. Total: 3
th(0)=-17.045693039894104;dx=-4.945727066496697E-7
New Minimum: -17.045693039894104 > -17.07271510362625
WOLF (strong): th(3119147.6961877886)=-17.07271510362625; dx=1.7298465605309574E-7 evalInputDelta=0.027022063732147217
New Minimum: -17.07271510362625 > -17.348508596420288
WOLF (strong): th(1559573.8480938943)=-17.348508596420288; dx=5.166072780243911E-8 evalInputDelta=0.3028155565261841
END: th(519857.94936463144)=-17.271856486797333; dx=-2.5031106248095786E-7 evalInputDelta=0.22616344690322876
Fitness changed from -17.045693039894104 to -17.348508596420288
Iteration 8 complete. Error: -17.348508596420288 Total: 25.7135; Orientation: 0.6122; Line Search: 20.0471
Adding measurement 5589505d to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.067e+03, gradient 7.315e-04, dot -0.470; [72165824-7f25-4ea9-b7d1-ef99d6497141 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -17.348508596420288, -14.595375120639801, -13.81580176949501, -13.219350337982178
LBFGS Accumulation History: 3 points
Removed measurement 5589505d to history. Total: 3
th(0)=-17.348508596420288;dx=-5.345322041435819E-7
New Minimum: -17.348508596420288 > -17.68915492296219
END: th(1120000.0000000005)=-17.68915492296219; dx=-1.1507519201233644E-8 evalInputDelta=0.34064632654190063
Fitness changed from -17.348508596420288 to -17.68915492296219
Iteration 9 complete. Error: -17.68915492296219 Total: 15.9180; Orientation: 0.5972; Line Search: 10.3105
Adding measurement 5c64830b to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.425e+03, gradient 5.381e-04, dot -0.586; [72165824-7f25-4ea9-b7d1-ef99d6497141 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -17.68915492296219, -14.595375120639801, -13.81580176949501, -13.219350337982178
LBFGS Accumulation History: 3 points
Removed measurement 5c64830b to history. Total: 3
th(0)=-17.68915492296219;dx=-2.890894081157475E-7
New Minimum: -17.68915492296219 > -17.87295937538147
WOLF (strong): th(2412966.852835711)=-17.87295937538147; dx=8.469662492170676E-8 evalInputDelta=0.183804452419281
New Minimum: -17.87295937538147 > -17.923427999019623
END: th(1206483.4264178555)=-17.923427999019623; dx=-5.33953189143221E-8 evalInputDelta=0.23427307605743408
Fitness changed from -17.68915492296219 to -17.923427999019623
Iteration 10 complete. Error: -17.923427999019623 Total: 21.1497; Orientation: 0.6375; Line Search: 15.5873
<a id="p-13"></a>Iteration 10
<a id="p-12"></a>
Final threshold in iteration 11: -17.923427999019623 (> -Infinity) after 228.202s (< 5400.000s)
Returns
-17.923427999019623