BasicOptimizer.scala:89 executed in 130.46 seconds (1.611 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 329236950636500
Reset training subject: 329239690582500
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
New Minimum: -99.71210718367438 > -99.94469817423932
th(0)=-99.94469817423932;dx=-2.686348514786916E-5
New Minimum: -99.94469817423932 > -99.94475125065267
END: th(2.154434690031884)=-99.94475125065267; dx=-2.3047920179477636E-5 evalInputDelta=5.3076413351504925E-5
Fitness changed from -99.71210718367438 to -99.94475125065267
Iteration 1 complete. Error: -99.94475125065267 Total: 13.0420; Orientation: 0.0370; Line Search: 5.2104
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 53037399 to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-99.94475125065267;dx=-2.3231912017011386E-5
New Minimum: -99.94475125065267 > -99.9448815290023
WOLFE (weak): th(4.641588833612779)=-99.9448815290023; dx=-2.3225408458363214E-5 evalInputDelta=1.302783496299753E-4
New Minimum: -99.9448815290023 > -99.9450021569524
WOLFE (weak): th(9.283177667225559)=-99.9450021569524; dx=-2.3220902531134668E-5 evalInputDelta=2.50906299726239E-4
New Minimum: -99.9450021569524 > -99.94549431748001
WOLFE (weak): th(27.849533001676676)=-99.94549431748001; dx=-2.320721178048087E-5 evalInputDelta=7.430668273400443E-4
New Minimum: -99.94549431748001 > -99.94772348481577
WOLFE (weak): th(111.3981320067067)=-99.94772348481577; dx=-2.3199038255625633E-5 evalInputDelta=0.002972234163095777
New Minimum: -99.94772348481577 > -99.95960671328359
WOLFE (weak): th(556.9906600335335)=-99.95960671328359; dx=-2.3187324950032648E-5 evalInputDelta=0.014855462630919192
New Minimum: -99.95960671328359 > -100.03381718911785
WOLFE (weak): th(3341.9439602012008)=-100.03381718911785; dx=-2.317700997029542E-5 evalInputDelta=0.08906593846518263
New Minimum: -100.03381718911785 > -100.56590418080228
WOLFE (weak): th(23393.607721408407)=-100.56590418080228; dx=-2.298742008888586E-5 evalInputDelta=0.6211529301496057
New Minimum: -100.56590418080228 > -104.76592000650082
WOLFE (weak): th(187148.86177126726)=-104.76592000650082; dx=-2.1623434316564788E-5 evalInputDelta=4.821168755848149
New Minimum: -104.76592000650082 > -130.0528788348163
END: th(1684339.7559414054)=-130.0528788348163; dx=-8.869159177648863E-6 evalInputDelta=30.10812758416364
Fitness changed from -99.94475125065267 to -130.0528788348163
Iteration 2 complete. Error: -130.0528788348163 Total: 28.5390; Orientation: 0.0667; Line Search: 25.4874
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 4c0e795c to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-130.0528788348163;dx=-1.2328010919150513E-5
New Minimum: -130.0528788348163 > -155.3012980214912
END: th(3628800.0000000005)=-155.3012980214912; dx=-2.016494791977306E-6 evalInputDelta=25.24841918667488
Fitness changed from -130.0528788348163 to -155.3012980214912
Iteration 3 complete. Error: -155.3012980214912 Total: 8.2597; Orientation: 0.0564; Line Search: 5.1755
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 6ade7637 to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-155.3012980214912;dx=-1.0188185360366379E-5
New Minimum: -155.3012980214912 > -160.47421200491382
WOLF (strong): th(7818012.603187701)=-160.47421200491382; dx=2.591544506427988E-6 evalInputDelta=5.172913983422632
New Minimum: -160.47421200491382 > -170.42017607211156
WOLF (strong): th(3909006.3015938504)=-170.42017607211156; dx=5.59220494364155E-7 evalInputDelta=15.118878050620367
END: th(1303002.1005312835)=-166.2195950456112; dx=-4.8910558183853E-6 evalInputDelta=10.918297024119994
Fitness changed from -155.3012980214912 to -170.42017607211156
Iteration 4 complete. Error: -170.42017607211156 Total: 13.6999; Orientation: 0.0668; Line Search: 10.4534
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 655e0d1d to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.303e+04, gradient 3.204e-03, dot -0.549; [b3ff0702-b950-4784-879d-52f015fa60a8 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -170.42017607211156, -155.3012980214912, -130.0528788348163, -99.94475125065267
LBFGS Accumulation History: 3 points
Removed measurement 655e0d1d to history. Total: 3
th(0)=-170.42017607211156;dx=-1.0244944587660828E-5
New Minimum: -170.42017607211156 > -186.02668859863155
END: th(2807232.926569009)=-186.02668859863155; dx=-2.0933864972861215E-7 evalInputDelta=15.606512526519992
Fitness changed from -170.42017607211156 to -186.02668859863155
Iteration 5 complete. Error: -186.02668859863155 Total: 9.3913; Orientation: 1.0943; Line Search: 5.2742
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 5d1ac308 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.328e+04, gradient 2.509e-03, dot -0.582; [b3ff0702-b950-4784-879d-52f015fa60a8 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -186.02668859863155, -155.3012980214912, -130.0528788348163, -99.94475125065267
LBFGS Accumulation History: 3 points
Removed measurement 5d1ac308 to history. Total: 3
th(0)=-186.02668859863155;dx=-6.280878455162301E-6
New Minimum: -186.02668859863155 > -191.16214323341512
WOLF (strong): th(6048000.000000001)=-191.16214323341512; dx=2.181866918910628E-6 evalInputDelta=5.135454634783571
New Minimum: -191.16214323341512 > -196.15513657701175
END: th(3024000.0000000005)=-196.15513657701175; dx=-3.491566365498416E-7 evalInputDelta=10.128447978380194
Fitness changed from -186.02668859863155 to -196.15513657701175
Iteration 6 complete. Error: -196.15513657701175 Total: 11.7978; Orientation: 1.0869; Line Search: 7.7847
Adding measurement 6a06cf87 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.847e+04, gradient 2.404e-03, dot -0.584; [b3ff0702-b950-4784-879d-52f015fa60a8 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -196.15513657701175, -155.3012980214912, -130.0528788348163, -99.94475125065267
LBFGS Accumulation History: 3 points
Removed measurement 6a06cf87 to history. Total: 3
th(0)=-196.15513657701175;dx=-5.766995262384689E-6
New Minimum: -196.15513657701175 > -196.98052576104777
WOLF (strong): th(6515010.502656418)=-196.98052576104777; dx=2.4405382581088296E-6 evalInputDelta=0.8253891840360268
New Minimum: -196.98052576104777 > -204.28554880411764
WOLF (strong): th(3257505.251328209)=-204.28554880411764; dx=4.768589848685331E-7 evalInputDelta=8.130412227105893
END: th(1085835.0837760696)=-201.65071239928002; dx=-3.3828603921264283E-6 evalInputDelta=5.495575822268279
Fitness changed from -196.15513657701175 to -204.28554880411764
Iteration 7 complete. Error: -204.28554880411764 Total: 13.9118; Orientation: 0.9855; Line Search: 10.5470
Adding measurement 44a4a9bd to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.299e+04, gradient 2.480e-03, dot -0.518; [b3ff0702-b950-4784-879d-52f015fa60a8 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -204.28554880411764, -155.3012980214912, -130.0528788348163, -99.94475125065267
LBFGS Accumulation History: 3 points
Removed measurement 44a4a9bd to history. Total: 3
th(0)=-204.28554880411764;dx=-6.141240445277815E-6
New Minimum: -204.28554880411764 > -212.42954963925305
END: th(2339360.772140841)=-212.42954963925305; dx=-3.877137084578524E-7 evalInputDelta=8.144000835135415
Fitness changed from -204.28554880411764 to -212.42954963925305
Iteration 8 complete. Error: -212.42954963925305 Total: 8.6849; Orientation: 1.0133; Line Search: 5.2188
Adding measurement 1929bbf9 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.738e+04, gradient 1.907e-03, dot -0.640; [b3ff0702-b950-4784-879d-52f015fa60a8 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -212.42954963925305, -155.3012980214912, -130.0528788348163, -99.94475125065267
LBFGS Accumulation History: 3 points
Removed measurement 1929bbf9 to history. Total: 3
th(0)=-212.42954963925305;dx=-3.6316998543169606E-6
New Minimum: -212.42954963925305 > -218.0101908218609
WOLF (strong): th(5040000.000000001)=-218.0101908218609; dx=7.222755377661403E-7 evalInputDelta=5.580641182607849
New Minimum: -218.0101908218609 > -218.49024896820225
END: th(2520000.0000000005)=-218.49024896820225; dx=-9.892985204018501E-7 evalInputDelta=6.060699328949198
Fitness changed from -212.42954963925305 to -218.49024896820225
Iteration 9 complete. Error: -218.49024896820225 Total: 11.2454; Orientation: 0.9746; Line Search: 7.7989
Adding measurement 26e10f4 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.956e+04, gradient 1.774e-03, dot -0.646; [b3ff0702-b950-4784-879d-52f015fa60a8 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -218.49024896820225, -155.3012980214912, -130.0528788348163, -99.94475125065267
LBFGS Accumulation History: 3 points
Removed measurement 26e10f4 to history. Total: 3
th(0)=-218.49024896820225;dx=-3.13881016230998E-6
New Minimum: -218.49024896820225 > -223.54819138684434
WOLF (strong): th(5429175.418880348)=-223.54819138684434; dx=4.258736491803124E-7 evalInputDelta=5.057942418642085
New Minimum: -223.54819138684434 > -223.81682155277574
END: th(2714587.709440174)=-223.81682155277574; dx=-7.798416628449484E-7 evalInputDelta=5.3265725845734835
Fitness changed from -218.49024896820225 to -223.81682155277574
Iteration 10 complete. Error: -223.81682155277574 Total: 11.2698; Orientation: 1.0309; Line Search: 7.8180
<a id="p-13"></a>Iteration 10
<a id="p-12"></a>
Final threshold in iteration 11: -223.81682155277574 (> -Infinity) after 130.457s (< 5400.000s)
Returns
-223.81682155277574