BasicOptimizer.scala:89 executed in 317.14 seconds (5.062 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 652669394971200
Reset training subject: 652675962051400
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
New Minimum: -35.556173891288445 > -35.59754452715654
th(0)=-35.59754452715654;dx=-1.8011763420306547E-6
New Minimum: -35.59754452715654 > -35.59754664885479
END: th(2.154434690031884)=-35.59754664885479; dx=-1.617615622073387E-6 evalInputDelta=2.1216982446503607E-6
Fitness changed from -35.556173891288445 to -35.59754664885479
Iteration 1 complete. Error: -35.59754664885479 Total: 32.5578; Orientation: 0.0991; Line Search: 13.0275
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 5a0a4522 to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-35.59754664885479;dx=-1.6238776650023266E-6
New Minimum: -35.59754664885479 > -35.59755725734414
WOLFE (weak): th(4.641588833612779)=-35.59755725734414; dx=-1.6234830512708193E-6 evalInputDelta=1.0608489354524409E-5
New Minimum: -35.59755725734414 > -35.59756574413335
WOLFE (weak): th(9.283177667225559)=-35.59756574413335; dx=-1.6226620355725347E-6 evalInputDelta=1.9095278560143925E-5
New Minimum: -35.59756574413335 > -35.597599691269934
WOLFE (weak): th(27.849533001676676)=-35.597599691269934; dx=-1.6225542769295857E-6 evalInputDelta=5.3042415146364874E-5
New Minimum: -35.597599691269934 > -35.59775245298397
WOLFE (weak): th(111.3981320067067)=-35.59775245298397; dx=-1.6224933825547207E-6 evalInputDelta=2.0580412918036473E-4
New Minimum: -35.59775245298397 > -35.59855868450371
WOLFE (weak): th(556.9906600335335)=-35.59855868450371; dx=-1.6223773758574478E-6 evalInputDelta=0.001012035648919607
New Minimum: -35.59855868450371 > -35.603609945893986
WOLFE (weak): th(3341.9439602012008)=-35.603609945893986; dx=-1.6219005170053056E-6 evalInputDelta=0.006063297039197835
New Minimum: -35.603609945893986 > -35.63994687434545
WOLFE (weak): th(23393.607721408407)=-35.63994687434545; dx=-1.6201138411626406E-6 evalInputDelta=0.04240022549065969
New Minimum: -35.63994687434545 > -35.935045037950445
WOLFE (weak): th(187148.86177126726)=-35.935045037950445; dx=-1.6033078335646998E-6 evalInputDelta=0.3374983890956571
New Minimum: -35.935045037950445 > -38.45945194200162
END: th(1684339.7559414054)=-38.45945194200162; dx=-1.4067120888992723E-6 evalInputDelta=2.861905293146833
Fitness changed from -35.59754664885479 to -38.45945194200162
Iteration 2 complete. Error: -38.45945194200162 Total: 70.3344; Orientation: 0.1467; Line Search: 62.6997
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 15f8f4b4 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-38.45945194200162;dx=-1.3425346136897528E-6
New Minimum: -38.45945194200162 > -43.02741974300548
END: th(3628800.0000000005)=-43.02741974300548; dx=-9.436064484698437E-7 evalInputDelta=4.56796780100386
Fitness changed from -38.45945194200162 to -43.02741974300548
Iteration 3 complete. Error: -43.02741974300548 Total: 19.7565; Orientation: 0.1415; Line Search: 12.3423
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 53d8c137 to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-43.02741974300548;dx=-9.04738927199808E-7
New Minimum: -43.02741974300548 > -49.26122708565315
END: th(7818012.603187701)=-49.26122708565315; dx=-5.512630661562204E-7 evalInputDelta=6.2338073426476655
Fitness changed from -43.02741974300548 to -49.26122708565315
Iteration 4 complete. Error: -49.26122708565315 Total: 20.9281; Orientation: 0.1534; Line Search: 13.4481
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 5c1a8c74 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.833e+04, gradient 8.356e-04, dot -0.752; [929334f3-6f2b-4a9e-86bb-23ee324c3e31 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -49.26122708565315, -43.02741974300548, -38.45945194200162, -35.59754664885479
LBFGS Accumulation History: 3 points
Removed measurement 5c1a8c74 to history. Total: 3
th(0)=-49.26122708565315;dx=-6.981249552489605E-7
New Minimum: -49.26122708565315 > -56.93566066760579
END: th(1.6843397559414055E7)=-56.93566066760579; dx=-2.424750858623264E-7 evalInputDelta=7.674433581952641
Fitness changed from -49.26122708565315 to -56.93566066760579
Iteration 5 complete. Error: -56.93566066760579 Total: 23.6613; Orientation: 3.1437; Line Search: 12.6519
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 7532ece3 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.283e+04, gradient 9.321e-04, dot -0.520; [929334f3-6f2b-4a9e-86bb-23ee324c3e31 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -56.93566066760579, -43.02741974300548, -38.45945194200162, -35.59754664885479
LBFGS Accumulation History: 3 points
Removed measurement 7532ece3 to history. Total: 3
th(0)=-56.93566066760579;dx=-8.685242840533959E-7
New Minimum: -56.93566066760579 > -59.48014232347598
WOLF (strong): th(3.628800000000001E7)=-59.48014232347598; dx=9.148044198731744E-8 evalInputDelta=2.544481655870193
New Minimum: -59.48014232347598 > -61.849832237320165
WOLF (strong): th(1.8144000000000004E7)=-61.849832237320165; dx=7.979114082034733E-8 evalInputDelta=4.914171569714377
END: th(6048000.000000001)=-61.17108170426134; dx=-3.7678587627586047E-7 evalInputDelta=4.235421036655552
Fitness changed from -56.93566066760579 to -61.849832237320165
Iteration 6 complete. Error: -61.849832237320165 Total: 35.3583; Orientation: 2.9200; Line Search: 25.2130
Adding measurement 6e67a564 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.431e+04, gradient 1.047e-03, dot -0.379; [929334f3-6f2b-4a9e-86bb-23ee324c3e31 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -61.849832237320165, -43.02741974300548, -38.45945194200162, -35.59754664885479
LBFGS Accumulation History: 3 points
Removed measurement 6e67a564 to history. Total: 3
th(0)=-61.849832237320165;dx=-1.0959876750438454E-6
New Minimum: -61.849832237320165 > -69.63241363916747
WOLF (strong): th(1.3030021005312836E7)=-69.63241363916747; dx=5.3905410238747324E-8 evalInputDelta=7.7825814018473025
END: th(6515010.502656418)=-68.05327917695509; dx=-5.598437687804538E-7 evalInputDelta=6.203446939634922
Fitness changed from -61.849832237320165 to -69.63241363916747
Iteration 7 complete. Error: -69.63241363916747 Total: 29.0497; Orientation: 2.6527; Line Search: 20.2285
Adding measurement 4402c33e to history. Total: 3
Rejected: LBFGS Orientation magnitude: 5.785e+04, gradient 8.410e-04, dot -0.492; [929334f3-6f2b-4a9e-86bb-23ee324c3e31 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -69.63241363916747, -43.02741974300548, -38.45945194200162, -35.59754664885479
LBFGS Accumulation History: 3 points
Removed measurement 4402c33e to history. Total: 3
th(0)=-69.63241363916747;dx=-7.07202106822022E-7
New Minimum: -69.63241363916747 > -74.26811336634249
WOLF (strong): th(1.4036164632845048E7)=-74.26811336634249; dx=4.1065057515333574E-8 evalInputDelta=4.635699727175023
END: th(7018082.316422524)=-73.55522005379089; dx=-2.8943703442910056E-7 evalInputDelta=3.922806414623423
Fitness changed from -69.63241363916747 to -74.26811336634249
Iteration 8 complete. Error: -74.26811336634249 Total: 27.8657; Orientation: 2.6784; Line Search: 18.8465
Adding measurement 542d6736 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 6.507e+04, gradient 8.357e-04, dot -0.469; [929334f3-6f2b-4a9e-86bb-23ee324c3e31 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -74.26811336634249, -43.02741974300548, -38.45945194200162, -35.59754664885479
LBFGS Accumulation History: 3 points
Removed measurement 542d6736 to history. Total: 3
th(0)=-74.26811336634249;dx=-6.978714209839702E-7
New Minimum: -74.26811336634249 > -78.39196442994349
WOLF (strong): th(1.5120000000000006E7)=-78.39196442994349; dx=1.1528092195516522E-7 evalInputDelta=4.123851063600995
END: th(7560000.000000003)=-78.25453213582746; dx=-2.3113743775119788E-7 evalInputDelta=3.9864187694849704
Fitness changed from -74.26811336634249 to -78.39196442994349
Iteration 9 complete. Error: -78.39196442994349 Total: 28.5939; Orientation: 2.6828; Line Search: 19.7293
Adding measurement 25c0b585 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 7.274e+04, gradient 8.504e-04, dot -0.436; [929334f3-6f2b-4a9e-86bb-23ee324c3e31 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -78.39196442994349, -43.02741974300548, -38.45945194200162, -35.59754664885479
LBFGS Accumulation History: 3 points
Removed measurement 25c0b585 to history. Total: 3
th(0)=-78.39196442994349;dx=-7.225660119214014E-7
New Minimum: -78.39196442994349 > -82.50489908500815
WOLF (strong): th(1.6287526256641047E7)=-82.50489908500815; dx=1.8302293680082358E-7 evalInputDelta=4.112934655064663
New Minimum: -82.50489908500815 > -82.78873529893168
END: th(8143763.128320524)=-82.78873529893168; dx=-2.1424282598385563E-7 evalInputDelta=4.396770868988199
Fitness changed from -78.39196442994349 to -82.78873529893168
Iteration 10 complete. Error: -82.78873529893168 Total: 27.9040; Orientation: 2.6206; Line Search: 19.1088
<a id="p-13"></a>Iteration 10
<a id="p-12"></a>
Final threshold in iteration 11: -82.78873529893168 (> -Infinity) after 317.137s (< 5400.000s)
Returns
-82.78873529893168