BasicOptimizer.scala:89 executed in 216.57 seconds (3.828 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 690133230274800
Reset training subject: 690138520017300
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
New Minimum: -807.8817209220666 > -809.30379030374
th(0)=-809.30379030374;dx=-0.006684930588825898
New Minimum: -809.30379030374 > -809.3184632763545
WOLFE (weak): th(2.154434690031884)=-809.3184632763545; dx=-0.006198237896859571 evalInputDelta=0.014672972614448554
New Minimum: -809.3184632763545 > -809.3331745949872
WOLFE (weak): th(4.308869380063768)=-809.3331745949872; dx=-0.006197186859475122 evalInputDelta=0.029384291247197325
New Minimum: -809.3331745949872 > -809.3918627587011
WOLFE (weak): th(12.926608140191302)=-809.3918627587011; dx=-0.006191503398466104 evalInputDelta=0.08807245496109317
New Minimum: -809.3918627587011 > -809.6559840450758
WOLFE (weak): th(51.70643256076521)=-809.6559840450758; dx=-0.006193063273703262 evalInputDelta=0.35219374133578185
New Minimum: -809.6559840450758 > -811.0630370076052
WOLFE (weak): th(258.53216280382605)=-811.0630370076052; dx=-0.006181354573101076 evalInputDelta=1.7592467038651876
New Minimum: -811.0630370076052 > -819.7758077669772
WOLFE (weak): th(1551.1929768229563)=-819.7758077669772; dx=-0.00608903731587818 evalInputDelta=10.472017463237194
New Minimum: -819.7758077669772 > -877.7930422371779
END: th(10858.350837760694)=-877.7930422371779; dx=-0.0052279061700439885 evalInputDelta=68.48925193343791
Fitness changed from -807.8817209220666 to -877.7930422371779
Iteration 1 complete. Error: -877.7930422371779 Total: 50.4909; Orientation: 0.0616; Line Search: 36.1447
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 109bc848 to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-877.7930422371779;dx=-0.004882581196902687
New Minimum: -877.7930422371779 > -987.7852119767738
END: th(23393.607721408407)=-987.7852119767738; dx=-0.003651622123904117 evalInputDelta=109.99216973959585
Fitness changed from -877.7930422371779 to -987.7852119767738
Iteration 2 complete. Error: -987.7852119767738 Total: 14.7866; Orientation: 0.1003; Line Search: 9.2648
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 356a28a8 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-987.7852119767738;dx=-0.003591330631534379
New Minimum: -987.7852119767738 > -1147.9946646217481
END: th(50400.00000000001)=-1147.9946646217481; dx=-0.0022793912474431367 evalInputDelta=160.20945264497436
Fitness changed from -987.7852119767738 to -1147.9946646217481
Iteration 3 complete. Error: -1147.9946646217481 Total: 14.3910; Orientation: 0.0849; Line Search: 8.9453
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 4ae9e796 to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-1147.9946646217481;dx=-0.0027659795894335202
New Minimum: -1147.9946646217481 > -1343.5801985739445
END: th(108583.50837760697)=-1343.5801985739445; dx=-0.0010980339787045835 evalInputDelta=195.58553395219633
Fitness changed from -1147.9946646217481 to -1343.5801985739445
Iteration 4 complete. Error: -1343.5801985739445 Total: 14.8464; Orientation: 0.0936; Line Search: 9.3079
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 33e36fbf to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.818e+04, gradient 5.693e-02, dot -0.550; [2b7abb99-4b2b-40e5-a049-9c3ee8a82543 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1343.5801985739445, -1147.9946646217481, -987.7852119767738, -877.7930422371779
LBFGS Accumulation History: 3 points
Removed measurement 33e36fbf to history. Total: 3
th(0)=-1343.5801985739445;dx=-0.0032372654035007246
New Minimum: -1343.5801985739445 > -1512.466197969396
END: th(233936.07721408413)=-1512.466197969396; dx=-1.3436901658848E-4 evalInputDelta=168.88599939545156
Fitness changed from -1343.5801985739445 to -1512.466197969396
Iteration 5 complete. Error: -1512.466197969396 Total: 16.1293; Orientation: 1.7264; Line Search: 8.9228
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 5a2d0e27 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.451e+04, gradient 6.276e-02, dot -0.332; [2b7abb99-4b2b-40e5-a049-9c3ee8a82543 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1512.466197969396, -1147.9946646217481, -987.7852119767738, -877.7930422371779
LBFGS Accumulation History: 3 points
Removed measurement 5a2d0e27 to history. Total: 3
th(0)=-1512.466197969396;dx=-0.003935893500465807
Armijo: th(504000.0000000002)=-1500.8905689623077; dx=5.154463364212272E-4 evalInputDelta=-11.57562900708831
New Minimum: -1512.466197969396 > -1677.0600019081012
WOLF (strong): th(252000.0000000001)=-1677.0600019081012; dx=5.147486007131163E-4 evalInputDelta=164.59380393870515
New Minimum: -1677.0600019081012 > -1709.9018246671358
END: th(84000.00000000003)=-1709.9018246671358; dx=-7.915716618420399E-4 evalInputDelta=197.43562669773974
Fitness changed from -1512.466197969396 to -1709.9018246671358
Iteration 6 complete. Error: -1709.9018246671358 Total: 25.6436; Orientation: 1.6606; Line Search: 18.2522
Adding measurement 1a105e53 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.398e+04, gradient 4.893e-02, dot -0.416; [2b7abb99-4b2b-40e5-a049-9c3ee8a82543 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1709.9018246671358, -1147.9946646217481, -987.7852119767738, -877.7930422371779
LBFGS Accumulation History: 3 points
Removed measurement 1a105e53 to history. Total: 3
th(0)=-1709.9018246671358;dx=-0.0023882460210965777
New Minimum: -1709.9018246671358 > -1877.2923320569976
END: th(180972.5139626783)=-1877.2923320569976; dx=-4.195289773370398E-5 evalInputDelta=167.3905073898618
Fitness changed from -1709.9018246671358 to -1877.2923320569976
Iteration 7 complete. Error: -1877.2923320569976 Total: 14.8350; Orientation: 1.6707; Line Search: 8.8213
Adding measurement 3bc2e9c4 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.267e+04, gradient 6.260e-02, dot -0.333; [2b7abb99-4b2b-40e5-a049-9c3ee8a82543 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1877.2923320569976, -1147.9946646217481, -987.7852119767738, -877.7930422371779
LBFGS Accumulation History: 3 points
Removed measurement 3bc2e9c4 to history. Total: 3
th(0)=-1877.2923320569976;dx=-0.0039139301095973454
Armijo: th(389893.4620234736)=-1760.099073916011; dx=7.362969311282359E-4 evalInputDelta=-117.19325814098647
New Minimum: -1877.2923320569976 > -1956.394387642737
WOLF (strong): th(194946.7310117368)=-1956.394387642737; dx=8.514531631139495E-4 evalInputDelta=79.10205558573944
New Minimum: -1956.394387642737 > -2026.5056131183057
END: th(64982.24367057893)=-2026.5056131183057; dx=-6.270345194458821E-4 evalInputDelta=149.21328106130818
Fitness changed from -1877.2923320569976 to -2026.5056131183057
Iteration 8 complete. Error: -2026.5056131183057 Total: 24.4706; Orientation: 1.6223; Line Search: 18.2745
Adding measurement c45c5fc to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.218e+04, gradient 4.422e-02, dot -0.470; [2b7abb99-4b2b-40e5-a049-9c3ee8a82543 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -2026.5056131183057, -1147.9946646217481, -987.7852119767738, -877.7930422371779
LBFGS Accumulation History: 3 points
Removed measurement c45c5fc to history. Total: 3
th(0)=-2026.5056131183057;dx=-0.0019516115696298461
New Minimum: -2026.5056131183057 > -2150.227894898585
END: th(140000.00000000006)=-2150.227894898585; dx=-2.254500297218816E-4 evalInputDelta=123.72228178027922
Fitness changed from -2026.5056131183057 to -2150.227894898585
Iteration 9 complete. Error: -2150.227894898585 Total: 15.3352; Orientation: 1.6057; Line Search: 9.0659
Adding measurement 680e99a7 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 5.409e+04, gradient 5.511e-02, dot -0.388; [2b7abb99-4b2b-40e5-a049-9c3ee8a82543 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -2150.227894898585, -1147.9946646217481, -987.7852119767738, -877.7930422371779
LBFGS Accumulation History: 3 points
Removed measurement 680e99a7 to history. Total: 3
th(0)=-2150.227894898585;dx=-0.0030345230638129357
Armijo: th(301620.8566044639)=-2105.4947162127955; dx=6.3911441297858E-4 evalInputDelta=-44.733178685789426
New Minimum: -2150.227894898585 > -2226.091642318438
WOLF (strong): th(150810.42830223194)=-2226.091642318438; dx=5.441638239558329E-4 evalInputDelta=75.8637474198531
New Minimum: -2226.091642318438 > -2249.186519611035
END: th(50270.142767410645)=-2249.186519611035; dx=-7.4199129651094E-4 evalInputDelta=98.95862471244982
Fitness changed from -2150.227894898585 to -2249.186519611035
Iteration 10 complete. Error: -2249.186519611035 Total: 24.7389; Orientation: 1.6184; Line Search: 18.5565
<a id="p-13"></a>Iteration 10
<a id="p-12"></a>
Final threshold in iteration 11: -2249.186519611035 (> -Infinity) after 216.574s (< 5400.000s)
Returns
-2249.186519611035