BasicOptimizer.scala:89 executed in 218.00 seconds (4.255 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 695410670783100
Reset training subject: 695415379861000
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
New Minimum: -918.200379819133 > -920.7727732725376
th(0)=-920.7727732725376;dx=-0.0057259023057933955
New Minimum: -920.7727732725376 > -920.7848554358396
END: th(2.154434690031884)=-920.7848554358396; dx=-0.005148495757541794 evalInputDelta=0.012082163302011395
Fitness changed from -918.200379819133 to -920.7848554358396
Iteration 1 complete. Error: -920.7848554358396 Total: 22.6096; Orientation: 0.0609; Line Search: 8.9027
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 25cf38dc to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-920.7848554358396;dx=-0.0051670082993346324
New Minimum: -920.7848554358396 > -920.8109197875533
WOLFE (weak): th(4.641588833612779)=-920.8109197875533; dx=-0.005165264708640141 evalInputDelta=0.026064351713671385
New Minimum: -920.8109197875533 > -920.8369834015139
WOLFE (weak): th(9.283177667225559)=-920.8369834015139; dx=-0.005159564072763242 evalInputDelta=0.052127965674230836
New Minimum: -920.8369834015139 > -920.9411626157232
WOLFE (weak): th(27.849533001676676)=-920.9411626157232; dx=-0.005161627719186189 evalInputDelta=0.15630717988358356
New Minimum: -920.9411626157232 > -921.409789398832
WOLFE (weak): th(111.3981320067067)=-921.409789398832; dx=-0.005159136382971447 evalInputDelta=0.6249339629923725
New Minimum: -921.409789398832 > -923.9055687677177
WOLFE (weak): th(556.9906600335335)=-923.9055687677177; dx=-0.00514470397923771 evalInputDelta=3.1207133318780507
New Minimum: -923.9055687677177 > -939.2913552247779
WOLFE (weak): th(3341.9439602012008)=-939.2913552247779; dx=-0.005021755082607798 evalInputDelta=18.5064997889383
New Minimum: -939.2913552247779 > -1037.0388734276069
END: th(23393.607721408407)=-1037.0388734276069; dx=-0.00391615082390432 evalInputDelta=116.25401799176723
Fitness changed from -920.7848554358396 to -1037.0388734276069
Iteration 2 complete. Error: -1037.0388734276069 Total: 42.7299; Orientation: 0.0967; Line Search: 37.3000
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 59d2d889 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-1037.0388734276069;dx=-0.0036152507645734857
New Minimum: -1037.0388734276069 > -1197.7908832513294
END: th(50400.00000000001)=-1197.7908832513294; dx=-0.0022842159445854012 evalInputDelta=160.75200982372257
Fitness changed from -1037.0388734276069 to -1197.7908832513294
Iteration 3 complete. Error: -1197.7908832513294 Total: 14.9545; Orientation: 0.0938; Line Search: 9.2710
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 5807109a to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-1197.7908832513294;dx=-0.0023338353257882794
New Minimum: -1197.7908832513294 > -1396.1318437024493
END: th(108583.50837760697)=-1396.1318437024493; dx=-0.0011677210883819463 evalInputDelta=198.34096045111983
Fitness changed from -1197.7908832513294 to -1396.1318437024493
Iteration 4 complete. Error: -1396.1318437024493 Total: 14.6959; Orientation: 0.0941; Line Search: 9.2310
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 28391769 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.526e+04, gradient 4.668e-02, dot -0.668; [34c947c0-2410-4201-b533-4d46d0418483 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1396.1318437024493, -1197.7908832513294, -1037.0388734276069, -920.7848554358396
LBFGS Accumulation History: 3 points
Removed measurement 28391769 to history. Total: 3
th(0)=-1396.1318437024493;dx=-0.002175623897067906
New Minimum: -1396.1318437024493 > -1575.6706508658465
END: th(233936.07721408413)=-1575.6706508658465; dx=-2.4017838475813395E-4 evalInputDelta=179.53880716339722
Fitness changed from -1396.1318437024493 to -1575.6706508658465
Iteration 5 complete. Error: -1575.6706508658465 Total: 16.8235; Orientation: 1.8121; Line Search: 9.3406
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 228656bc to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.279e+04, gradient 5.587e-02, dot -0.399; [34c947c0-2410-4201-b533-4d46d0418483 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1575.6706508658465, -1197.7908832513294, -1037.0388734276069, -920.7848554358396
LBFGS Accumulation History: 3 points
Removed measurement 228656bc to history. Total: 3
th(0)=-1575.6706508658465;dx=-0.0031197294195987905
Armijo: th(504000.0000000002)=-1554.6801921938802; dx=4.4263877561865706E-4 evalInputDelta=-20.99045867196628
New Minimum: -1575.6706508658465 > -1698.4713715573778
WOLF (strong): th(252000.0000000001)=-1698.4713715573778; dx=4.411589233481131E-4 evalInputDelta=122.8007206915313
New Minimum: -1698.4713715573778 > -1730.7345983714545
END: th(84000.00000000003)=-1730.7345983714545; dx=-5.888605932972652E-4 evalInputDelta=155.06394750560798
Fitness changed from -1575.6706508658465 to -1730.7345983714545
Iteration 6 complete. Error: -1730.7345983714545 Total: 26.4439; Orientation: 1.6986; Line Search: 19.1418
Adding measurement 295b4b6f to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.990e+04, gradient 4.457e-02, dot -0.484; [34c947c0-2410-4201-b533-4d46d0418483 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1730.7345983714545, -1197.7908832513294, -1037.0388734276069, -920.7848554358396
LBFGS Accumulation History: 3 points
Removed measurement 295b4b6f to history. Total: 3
th(0)=-1730.7345983714545;dx=-0.0019847384912949004
New Minimum: -1730.7345983714545 > -1863.8815681260437
END: th(180972.5139626783)=-1863.8815681260437; dx=-8.235002304869828E-5 evalInputDelta=133.14696975458924
Fitness changed from -1730.7345983714545 to -1863.8815681260437
Iteration 7 complete. Error: -1863.8815681260437 Total: 15.1793; Orientation: 1.6218; Line Search: 9.1098
Adding measurement 5738eb5 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.715e+04, gradient 5.422e-02, dot -0.391; [34c947c0-2410-4201-b533-4d46d0418483 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1863.8815681260437, -1197.7908832513294, -1037.0388734276069, -920.7848554358396
LBFGS Accumulation History: 3 points
Removed measurement 5738eb5 to history. Total: 3
th(0)=-1863.8815681260437;dx=-0.0029393044902960813
Armijo: th(389893.4620234736)=-1815.0772710824187; dx=5.76014625676156E-4 evalInputDelta=-48.804297043624956
New Minimum: -1863.8815681260437 > -1955.2416730419798
WOLF (strong): th(194946.7310117368)=-1955.2416730419798; dx=5.615773424909643E-4 evalInputDelta=91.36010491593606
New Minimum: -1955.2416730419798 > -1984.6879477640812
END: th(64982.24367057893)=-1984.6879477640812; dx=-6.674582268796929E-4 evalInputDelta=120.80637963803747
Fitness changed from -1863.8815681260437 to -1984.6879477640812
Iteration 8 complete. Error: -1984.6879477640812 Total: 24.3975; Orientation: 1.7297; Line Search: 18.0791
Adding measurement 17bc9024 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.630e+04, gradient 3.940e-02, dot -0.542; [34c947c0-2410-4201-b533-4d46d0418483 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1984.6879477640812, -1197.7908832513294, -1037.0388734276069, -920.7848554358396
LBFGS Accumulation History: 3 points
Removed measurement 17bc9024 to history. Total: 3
th(0)=-1984.6879477640812;dx=-0.0015519862429930705
New Minimum: -1984.6879477640812 > -2098.0041706345582
END: th(140000.00000000006)=-2098.0041706345582; dx=-3.591357887405747E-4 evalInputDelta=113.31622287047708
Fitness changed from -1984.6879477640812 to -2098.0041706345582
Iteration 9 complete. Error: -2098.0041706345582 Total: 15.5537; Orientation: 1.5866; Line Search: 9.3157
Adding measurement 38141b5e to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.422e+04, gradient 4.670e-02, dot -0.455; [34c947c0-2410-4201-b533-4d46d0418483 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -2098.0041706345582, -1197.7908832513294, -1037.0388734276069, -920.7848554358396
LBFGS Accumulation History: 3 points
Removed measurement 38141b5e to history. Total: 3
th(0)=-2098.0041706345582;dx=-0.0021784532206991047
New Minimum: -2098.0041706345582 > -2115.573326547676
WOLF (strong): th(301620.8566044639)=-2115.573326547676; dx=4.027860772604473E-4 evalInputDelta=17.56915591311781
New Minimum: -2115.573326547676 > -2179.3166360123073
WOLF (strong): th(150810.42830223194)=-2179.3166360123073; dx=1.937283850855409E-4 evalInputDelta=81.31246537774905
END: th(50270.142767410645)=-2171.137029300546; dx=-6.54618083654755E-4 evalInputDelta=73.1328586659879
Fitness changed from -2098.0041706345582 to -2179.3166360123073
Iteration 10 complete. Error: -2179.3166360123073 Total: 23.7415; Orientation: 1.6000; Line Search: 17.8572
<a id="p-13"></a>Iteration 10
<a id="p-12"></a>
Final threshold in iteration 11: -2179.3166360123073 (> -Infinity) after 217.996s (< 5400.000s)
Returns
-2179.3166360123073