BasicOptimizer.scala:89 executed in 2176.05 seconds (7.770 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 28812371540076
Reset training subject: 28861888979081
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
th(0)=2.616383522748947;dx=-5.115856733188539E-7
New Minimum: 2.616383522748947 > 2.616382598876953
END: th(2.154434690031884)=2.616382598876953; dx=-4.416451040057942E-7 evalInputDelta=9.238719940185547E-7
Fitness changed from 2.616383522748947 to 2.616382598876953
Iteration 1 complete. Error: 2.616382598876953 Total: 246.3876; Orientation: 0.0689; Line Search: 98.2294
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 12878859 to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=2.616382598876953;dx=-4.396824703117342E-7
New Minimum: 2.616382598876953 > 2.6163796186447144
WOLFE (weak): th(4.641588833612779)=2.6163796186447144; dx=-4.392054316180193E-7 evalInputDelta=2.9802322387695312E-6
New Minimum: 2.6163796186447144 > 2.6163774728775024
WOLFE (weak): th(9.283177667225559)=2.6163774728775024; dx=-4.390338054264475E-7 evalInputDelta=5.125999450683594E-6
New Minimum: 2.6163774728775024 > 2.6163678765296936
WOLFE (weak): th(27.849533001676676)=2.6163678765296936; dx=-4.387080720679874E-7 evalInputDelta=1.4722347259521484E-5
New Minimum: 2.6163678765296936 > 2.616325855255127
WOLFE (weak): th(111.3981320067067)=2.616325855255127; dx=-4.384170416035968E-7 evalInputDelta=5.6743621826171875E-5
New Minimum: 2.616325855255127 > 2.616100937128067
END: th(556.9906600335335)=2.616100937128067; dx=-3.715461245761277E-7 evalInputDelta=2.816617488861084E-4
Fitness changed from 2.616382598876953 to 2.616100937128067
Iteration 2 complete. Error: 2.616100937128067 Total: 344.4384; Orientation: 0.0938; Line Search: 293.6579
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 8df2649 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=2.616100937128067;dx=-3.7973947967556357E-7
New Minimum: 2.616100937128067 > 2.6155921816825867
WOLFE (weak): th(1200.0)=2.6155921816825867; dx=-3.667031318532359E-7 evalInputDelta=5.087554454803467E-4
New Minimum: 2.6155921816825867 > 2.6150847673416138
WOLFE (weak): th(2400.0)=2.6150847673416138; dx=-3.661968282874742E-7 evalInputDelta=0.001016169786453247
New Minimum: 2.6150847673416138 > 2.613055795431137
WOLFE (weak): th(7200.0)=2.613055795431137; dx=-3.6592368497510307E-7 evalInputDelta=0.0030451416969299316
New Minimum: 2.613055795431137 > 2.6039413809776306
WOLFE (weak): th(28800.0)=2.6039413809776306; dx=-3.647586303084992E-7 evalInputDelta=0.012159556150436401
New Minimum: 2.6039413809776306 > 2.5558714270591736
WOLFE (weak): th(144000.0)=2.5558714270591736; dx=-3.5953542668650404E-7 evalInputDelta=0.06022951006889343
New Minimum: 2.5558714270591736 > 2.276866614818573
END: th(864000.0)=2.276866614818573; dx=-3.116803406870801E-7 evalInputDelta=0.339234322309494
Fitness changed from 2.616100937128067 to 2.276866614818573
Iteration 3 complete. Error: 2.276866614818573 Total: 392.3449; Orientation: 0.0919; Line Search: 342.5427
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 4ec3a340 to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=2.276866614818573;dx=-3.463328719242476E-7
New Minimum: 2.276866614818573 > 1.660993218421936
END: th(1861431.5721875476)=1.660993218421936; dx=-2.020593687116212E-7 evalInputDelta=0.615873396396637
Fitness changed from 2.276866614818573 to 1.660993218421936
Iteration 4 complete. Error: 1.660993218421936 Total: 148.7533; Orientation: 0.0895; Line Search: 98.4467
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 9cb2745 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.055e+03, gradient 4.786e-04, dot -0.699; [b7300cba-35d6-4831-927a-931f8d1ff0f2 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 1.660993218421936, 2.276866614818573, 2.616100937128067, 2.616382598876953
LBFGS Accumulation History: 3 points
Removed measurement 9cb2745 to history. Total: 3
th(0)=1.660993218421936;dx=-2.2893998555556095E-7
New Minimum: 1.660993218421936 > 0.8708539009094238
END: th(4010332.752241441)=0.8708539009094238; dx=-1.2027796343411234E-7 evalInputDelta=0.7901393175125122
Fitness changed from 1.660993218421936 to 0.8708539009094238
Iteration 5 complete. Error: 0.8708539009094238 Total: 148.7962; Orientation: 2.3628; Line Search: 97.0951
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 5844e674 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.732e+03, gradient 3.511e-04, dot -0.685; [b7300cba-35d6-4831-927a-931f8d1ff0f2 = 1.000/1.000e+00]
Orientation rejected. Popping history element from 0.8708539009094238, 2.276866614818573, 2.616100937128067, 2.616382598876953
LBFGS Accumulation History: 3 points
Removed measurement 5844e674 to history. Total: 3
th(0)=0.8708539009094238;dx=-1.232302375733745E-7
New Minimum: 0.8708539009094238 > -0.018360942602157593
END: th(8640000.0)=-0.018360942602157593; dx=-6.171247546950882E-8 evalInputDelta=0.8892148435115814
Fitness changed from 0.8708539009094238 to -0.018360942602157593
Iteration 6 complete. Error: -0.018360942602157593 Total: 149.1815; Orientation: 2.1914; Line Search: 97.5257
Adding measurement 4c8c7fc0 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.924e+03, gradient 3.388e-04, dot -0.568; [b7300cba-35d6-4831-927a-931f8d1ff0f2 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -0.018360942602157593, 2.276866614818573, 2.616100937128067, 2.616382598876953
LBFGS Accumulation History: 3 points
Removed measurement 4c8c7fc0 to history. Total: 3
th(0)=-0.018360942602157593;dx=-1.0927700859892082E-7
New Minimum: -0.018360942602157593 > -1.004823386669159
END: th(1.8614315721875478E7)=-1.004823386669159; dx=-2.5771330020042208E-8 evalInputDelta=0.9864624440670013
Fitness changed from -0.018360942602157593 to -1.004823386669159
Iteration 7 complete. Error: -1.004823386669159 Total: 149.2026; Orientation: 2.2720; Line Search: 97.8714
Adding measurement 787f6933 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 5.842e+03, gradient 3.002e-04, dot -0.390; [b7300cba-35d6-4831-927a-931f8d1ff0f2 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1.004823386669159, 2.276866614818573, 2.616100937128067, 2.616382598876953
LBFGS Accumulation History: 3 points
Removed measurement 787f6933 to history. Total: 3
th(0)=-1.004823386669159;dx=-9.006944290171215E-8
New Minimum: -1.004823386669159 > -1.7263377010822296
WOLF (strong): th(4.0103327522414416E7)=-1.7263377010822296; dx=1.5760647372810042E-9 evalInputDelta=0.7215143144130707
New Minimum: -1.7263377010822296 > -1.740604817867279
END: th(2.0051663761207208E7)=-1.740604817867279; dx=-5.2119502166182194E-9 evalInputDelta=0.7357814311981201
Fitness changed from -1.004823386669159 to -1.740604817867279
Iteration 8 complete. Error: -1.740604817867279 Total: 197.1450; Orientation: 2.2701; Line Search: 146.5910
Adding measurement 1b117ee to history. Total: 3
Rejected: LBFGS Orientation magnitude: 5.370e+03, gradient 2.403e-04, dot -0.372; [b7300cba-35d6-4831-927a-931f8d1ff0f2 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -1.740604817867279, 2.276866614818573, 2.616100937128067, 2.616382598876953
LBFGS Accumulation History: 3 points
Removed measurement 1b117ee to history. Total: 3
th(0)=-1.740604817867279;dx=-6.094769459803804E-8
New Minimum: -1.740604817867279 > -2.2734669744968414
WOLF (strong): th(4.320000000000001E7)=-2.2734669744968414; dx=9.620259504942404E-9 evalInputDelta=0.5328621566295624
New Minimum: -2.2734669744968414 > -2.403788924217224
END: th(2.1600000000000004E7)=-2.403788924217224; dx=-5.0124847585264805E-9 evalInputDelta=0.6631841063499451
Fitness changed from -1.740604817867279 to -2.403788924217224
Iteration 9 complete. Error: -2.403788924217224 Total: 199.4729; Orientation: 2.2047; Line Search: 148.7146
Adding measurement 2bca17c8 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 5.245e+03, gradient 1.783e-04, dot -0.425; [b7300cba-35d6-4831-927a-931f8d1ff0f2 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -2.403788924217224, 2.276866614818573, 2.616100937128067, 2.616382598876953
LBFGS Accumulation History: 3 points
Removed measurement 2bca17c8 to history. Total: 3
th(0)=-2.403788924217224;dx=-3.1791763462660664E-8
New Minimum: -2.403788924217224 > -2.8228434920310974
WOLF (strong): th(4.65357893046887E7)=-2.8228434920310974; dx=4.781147006660439E-9 evalInputDelta=0.4190545678138733
New Minimum: -2.8228434920310974 > -2.8401841819286346
END: th(2.326789465234435E7)=-2.8401841819286346; dx=-5.879451345193551E-9 evalInputDelta=0.4363952577114105
Fitness changed from -2.403788924217224 to -2.8401841819286346
Iteration 10 complete. Error: -2.8401841819286346 Total: 198.5928; Orientation: 2.2677; Line Search: 147.2696
<a id="p-13"></a>Iteration 10
<a id="p-12"></a>
Final threshold in iteration 11: -2.8401841819286346 (> -Infinity) after 2176.051s (< 5400.000s)
Returns
-2.8401841819286346