BasicOptimizer.scala:89 executed in 397.53 seconds (4.127 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 2569875133724
Reset training subject: 2571255751624
Adding measurement 43b0bcf7 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD+Trust
th(0)=-2.759705148077441;dx=-5.4740737575272086E-5
New Minimum: -2.759705148077441 > -2.759840119728441
WOLFE (weak): th(2.154434690031884)=-2.759840119728441; dx=-5.4724323591654747E-5 evalInputDelta=1.3497165100018194E-4
New Minimum: -2.759840119728441 > -2.7599746293398777
WOLFE (weak): th(4.308869380063768)=-2.7599746293398777; dx=-5.418727159797237E-5 evalInputDelta=2.694812624368659E-4
New Minimum: -2.7599746293398777 > -2.7605147272112793
WOLFE (weak): th(12.926608140191302)=-2.7605147272112793; dx=-5.3993436143952154E-5 evalInputDelta=8.095791338385006E-4
New Minimum: -2.7605147272112793 > -2.7629543249280455
WOLFE (weak): th(51.70643256076521)=-2.7629543249280455; dx=-5.4757394105167654E-5 evalInputDelta=0.0032491768506046625
New Minimum: -2.7629543249280455 > -2.7759890747210134
WOLFE (weak): th(258.53216280382605)=-2.7759890747210134; dx=-5.434464311357175E-5 evalInputDelta=0.016283926643572588
New Minimum: -2.7759890747210134 > -2.858439548207004
WOLFE (weak): th(1551.1929768229563)=-2.858439548207004; dx=-5.339966603750199E-5 evalInputDelta=0.09873440012956314
New Minimum: -2.858439548207004 > -3.430423792912614
END: th(10858.350837760694)=-3.430423792912614; dx=-4.670494836693214E-5 evalInputDelta=0.6707186448351732
Fitness changed from -2.759705148077441 to -3.430423792912614
Iteration 1 complete. Error: -3.430423792912614 Total: 15.6591; Orientation: 0.0169; Line Search: 11.5959
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 3aba3bf to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-3.430423792912614;dx=-8.212155934699808E-5
New Minimum: -3.430423792912614 > -4.951700700528192
END: th(23393.607721408407)=-4.951700700528192; dx=-2.5339517114625615E-5 evalInputDelta=1.5212769076155777
Fitness changed from -3.430423792912614 to -4.951700700528192
Iteration 2 complete. Error: -4.951700700528192 Total: 4.2726; Orientation: 0.0172; Line Search: 2.7852
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 45e1ffa to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-4.951700700528192;dx=-1.6638739855882921E-4
New Minimum: -4.951700700528192 > -6.633028184793468
END: th(50400.00000000001)=-6.633028184793468; dx=-9.069847969828096E-7 evalInputDelta=1.6813274842652763
Fitness changed from -4.951700700528192 to -6.633028184793468
Iteration 3 complete. Error: -6.633028184793468 Total: 4.2935; Orientation: 0.0171; Line Search: 2.7867
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 55538442 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 5.190e+03, gradient 1.589e-02, dot -0.093; [abc7744b-6c0b-4b11-8d14-4e0b7c2dc2ed = 1.000/1.000e+00]
Orientation rejected. Popping history element from -6.633028184793468, -4.951700700528192, -3.430423792912614, -2.759705148077441
LBFGS Accumulation History: 3 points
Removed measurement 55538442 to history. Total: 3
th(0)=-6.633028184793468;dx=-2.4817264496597663E-4
New Minimum: -6.633028184793468 > -10.036402829308614
END: th(108583.50837760697)=-10.036402829308614; dx=-2.9631666647100127E-6 evalInputDelta=3.4033746445151456
Fitness changed from -6.633028184793468 to -10.036402829308614
Iteration 4 complete. Error: -10.036402829308614 Total: 4.6936; Orientation: 0.4585; Line Search: 2.7647
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 2339e10a to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.533e+03, gradient 1.783e-02, dot -0.057; [abc7744b-6c0b-4b11-8d14-4e0b7c2dc2ed = 1.000/1.000e+00]
Orientation rejected. Popping history element from -10.036402829308614, -4.951700700528192, -3.430423792912614, -2.759705148077441
LBFGS Accumulation History: 3 points
Removed measurement 2339e10a to history. Total: 3
th(0)=-10.036402829308614;dx=-3.1093555108859916E-4
New Minimum: -10.036402829308614 > -13.265874947705834
WOLF (strong): th(233936.07721408413)=-13.265874947705834; dx=3.2633362693778444E-6 evalInputDelta=3.2294721183972204
New Minimum: -13.265874947705834 > -16.806724425618228
END: th(116968.03860704206)=-16.806724425618228; dx=-3.807409458979655E-6 evalInputDelta=6.770321596309614
Fitness changed from -10.036402829308614 to -16.806724425618228
Iteration 5 complete. Error: -16.806724425618228 Total: 6.1734; Orientation: 0.4619; Line Search: 4.2293
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 540e39aa to history. Total: 3
Rejected: LBFGS Orientation magnitude: 7.509e+03, gradient 2.072e-02, dot -0.056; [abc7744b-6c0b-4b11-8d14-4e0b7c2dc2ed = 1.000/1.000e+00]
Orientation rejected. Popping history element from -16.806724425618228, -4.951700700528192, -3.430423792912614, -2.759705148077441
LBFGS Accumulation History: 3 points
Removed measurement 540e39aa to history. Total: 3
th(0)=-16.806724425618228;dx=-4.232619358250907E-4
New Minimum: -16.806724425618228 > -17.514256832268977
WOLF (strong): th(252000.0000000001)=-17.514256832268977; dx=9.474559272689864E-6 evalInputDelta=0.7075324066507491
New Minimum: -17.514256832268977 > -23.965431823836106
END: th(126000.00000000004)=-23.965431823836106; dx=-8.487423302873861E-6 evalInputDelta=7.158707398217878
Fitness changed from -16.806724425618228 to -23.965431823836106
Iteration 6 complete. Error: -23.965431823836106 Total: 6.1250; Orientation: 0.4764; Line Search: 4.1847
Adding measurement 474faaae to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.002e+04, gradient 2.349e-02, dot -0.050; [abc7744b-6c0b-4b11-8d14-4e0b7c2dc2ed = 1.000/1.000e+00]
Orientation rejected. Popping history element from -23.965431823836106, -4.951700700528192, -3.430423792912614, -2.759705148077441
LBFGS Accumulation History: 3 points
Removed measurement 474faaae to history. Total: 3
th(0)=-23.965431823836106;dx=-5.489106469583876E-4
Armijo: th(271458.77094401745)=-18.310706834838324; dx=1.0563300799151021E-5 evalInputDelta=-5.654724988997781
New Minimum: -23.965431823836106 > -26.285427513173712
WOLF (strong): th(135729.38547200873)=-26.285427513173712; dx=1.172450957612395E-5 evalInputDelta=2.319995689337606
New Minimum: -26.285427513173712 > -32.84647308759736
END: th(45243.12849066957)=-32.84647308759736; dx=-5.1042327375275804E-5 evalInputDelta=8.881041263761254
Fitness changed from -23.965431823836106 to -32.84647308759736
Iteration 7 complete. Error: -32.84647308759736 Total: 7.8572; Orientation: 0.4594; Line Search: 6.0354
Adding measurement 3478503a to history. Total: 3
Rejected: LBFGS Orientation magnitude: 7.178e+04, gradient 2.476e-02, dot -0.074; [abc7744b-6c0b-4b11-8d14-4e0b7c2dc2ed = 1.000/1.000e+00]
Orientation rejected. Popping history element from -32.84647308759736, -4.951700700528192, -3.430423792912614, -2.759705148077441
LBFGS Accumulation History: 3 points
Removed measurement 3478503a to history. Total: 3
th(0)=-32.84647308759736;dx=-6.066197083809109E-4
Armijo: th(97473.3655058684)=-32.282219751007936; dx=1.657056662156021E-5 evalInputDelta=-0.5642533365894238
New Minimum: -32.84647308759736 > -38.89042774714418
END: th(48736.6827529342)=-38.89042774714418; dx=-8.950892041402978E-6 evalInputDelta=6.04395465954682
Fitness changed from -32.84647308759736 to -38.89042774714418
Iteration 8 complete. Error: -38.89042774714418 Total: 5.9693; Orientation: 0.4683; Line Search: 4.1755
Adding measurement 20ab091a to history. Total: 3
Accepted: LBFGS Orientation magnitude: 2.632e+04, gradient 2.329e-02, dot 0.115; [abc7744b-6c0b-4b11-8d14-4e0b7c2dc2ed = 1.000/1.000e+00]
Constructing line search parameters: LBFGS+Trust
th(0)=-38.89042774714418;dx=-70.61532182512539
Armijo: th(105000.00000000004)=-8.42289221184438; dx=9.476167234737009E-5 evalInputDelta=-30.467535535299803
Armijo: th(52500.00000000002)=-8.421631768275564; dx=1.907998951
...skipping 36079 bytes...
rch: 2.7967
Adding measurement 4a0218e3 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 5.353e+04, gradient 1.489e-02, dot -0.175; [abc7744b-6c0b-4b11-8d14-4e0b7c2dc2ed = 1.000/1.000e+00]
Orientation rejected. Popping history element from -102.35491342021744, -4.951700700528192, -3.430423792912614, -2.759705148077441
LBFGS Accumulation History: 3 points
Removed measurement 4a0218e3 to history. Total: 3
th(0)=-102.35491342021744;dx=-2.1910787141274515E-4
New Minimum: -102.35491342021744 > -117.45569740872887
END: th(149228.03606008177)=-117.45569740872887; dx=-9.815237835316459E-5 evalInputDelta=15.100783988511424
Fitness changed from -102.35491342021744 to -117.45569740872887
Iteration 38 complete. Error: -117.45569740872887 Total: 4.6724; Orientation: 0.5500; Line Search: 2.7856
Adding measurement 531e432d to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.765e+05, gradient 1.611e-02, dot -0.186; [abc7744b-6c0b-4b11-8d14-4e0b7c2dc2ed = 1.000/1.000e+00]
Orientation rejected. Popping history element from -117.45569740872887, -4.951700700528192, -3.430423792912614, -2.759705148077441
LBFGS Accumulation History: 3 points
Removed measurement 531e432d to history. Total: 3
th(0)=-117.45569740872887;dx=-2.570105483090203E-4
New Minimum: -117.45569740872887 > -125.67913413652327
END: th(321502.057613169)=-125.67913413652327; dx=-2.7849080463082514E-5 evalInputDelta=8.223436727794407
Fitness changed from -117.45569740872887 to -125.67913413652327
Iteration 39 complete. Error: -125.67913413652327 Total: 4.6141; Orientation: 0.4594; Line Search: 2.8166
Adding measurement 2b5257d1 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.656e+05, gradient 2.180e-02, dot -0.124; [abc7744b-6c0b-4b11-8d14-4e0b7c2dc2ed = 1.000/1.000e+00]
Orientation rejected. Popping history element from -125.67913413652327, -4.951700700528192, -3.430423792912614, -2.759705148077441
LBFGS Accumulation History: 3 points
Removed measurement 2b5257d1 to history. Total: 3
th(0)=-125.67913413652327;dx=-4.679600132894239E-4
Armijo: th(692655.1858384407)=-74.54014281800463; dx=2.3540556068159517E-5 evalInputDelta=-51.13899131851865
Armijo: th(346327.59291922033)=-111.14870964821306; dx=3.331926602357704E-5 evalInputDelta=-14.530424488310217
New Minimum: -125.67913413652327 > -145.04877057977845
END: th(115442.53097307344)=-145.04877057977845; dx=-4.709981616113191E-5 evalInputDelta=19.369636443255175
Fitness changed from -125.67913413652327 to -145.04877057977845
Iteration 40 complete. Error: -145.04877057977845 Total: 7.9784; Orientation: 0.4809; Line Search: 6.1622
<a id="p-25"></a>Iteration 40
<a id="p-24"></a>
Adding measurement 44dfafdf to history. Total: 3
Accepted: LBFGS Orientation magnitude: 1.445e+05, gradient 1.947e-02, dot 0.173; [abc7744b-6c0b-4b11-8d14-4e0b7c2dc2ed = 1.000/1.000e+00]
th(0)=-145.04877057977845;dx=-486.84711372787365
Armijo: th(248713.39343346964)=-82.73445686488581; dx=1.8435695860406377E-4 evalInputDelta=-62.314313714892634
Armijo: th(124356.69671673482)=-82.73445686488581; dx=3.7300669984935685E-4 evalInputDelta=-62.314313714892634
Armijo: th(41452.23223891161)=-82.73373342283998; dx=0.0011122284273935791 evalInputDelta=-62.31503715693847
Armijo: th(10363.058059727902)=-82.73286735845123; dx=0.004373355788179309 evalInputDelta=-62.31590322132722
Armijo: th(2072.6116119455805)=-82.77213970832999; dx=0.02220821210540151 evalInputDelta=-62.27663087144846
Armijo: th(345.43526865759674)=-82.75381631363742; dx=0.13244667636094387 evalInputDelta=-62.29495426614103
Armijo: th(49.34789552251382)=-83.38231777862916; dx=0.9442876307899609 evalInputDelta=-61.66645280114929
Armijo: th(6.168486940314227)=-85.92425846489242; dx=7.278824523789073 evalInputDelta=-59.12451211488603
Armijo: th(0.6853874378126918)=-108.0861314177966; dx=48.09056658465615 evalInputDelta=-36.962639161981855
New Minimum: -145.04877057977845 > -152.23878151776165
END: th(0.06853874378126919)=-152.23878151776165; dx=-88.89177849682146 evalInputDelta=7.190010937983203
Fitness changed from -145.04877057977845 to -152.23878151776165
Iteration 41 complete. Error: -152.23878151776165 Total: 17.3155; Orientation: 0.4536; Line Search: 15.3702
Adding measurement 505933bf to history. Total: 4
Accepted: LBFGS Orientation magnitude: 2.626e+04, gradient 1.890e-02, dot 0.188; [abc7744b-6c0b-4b11-8d14-4e0b7c2dc2ed = 1.000/1.000e+00]
th(0)=-152.23878151776165;dx=-94.21103417722333
New Minimum: -152.23878151776165 > -152.2901900853486
WOLF (strong): th(0.14766224721357338)=-152.2901900853486; dx=3.873666327693144 evalInputDelta=0.051408567586946674
WOLF (strong): th(0.07383112360678669)=-152.2691485975812; dx=3.7021988924005154 evalInputDelta=0.03036707981954123
WOLF (strong): th(0.024610374535595562)=-152.25033615024444; dx=3.680272277296096 evalInputDelta=0.011554632482784655
WOLF (strong): th(0.0061525936338988904)=-152.2416167112139; dx=3.592474943182828 evalInputDelta=0.002835193452256135
WOLF (strong): th(0.0012305187267797782)=-152.2393677251115; dx=3.6660310884150324 evalInputDelta=5.862073498406062E-4
WOLF (strong): th(2.0508645446329636E-4)=-152.238860430421; dx=3.635207419831932 evalInputDelta=7.891265934745206E-5
WOLF (strong): th(2.9298064923328052E-5)=-152.23878715438153; dx=3.514655470189907 evalInputDelta=5.6366198748492025E-6
Armijo: th(3.6622581154160065E-6)=-152.23877024452125; dx=3.6493980434655757 evalInputDelta=-1.1273240403397722E-5
Armijo: th(4.0691756837955627E-7)=-152.23878151776165; dx=3.661516478104722 evalInputDelta=0.0
Armijo: th(4.0691756837955626E-8)=-152.23877588114155; dx=3.588031888869881 evalInputDelta=-5.636620102222878E-6
Armijo: th(3.6992506216323296E-9)=-152.23877024452125; dx=3.671388808697974 evalInputDelta=-1.1273240403397722E-5
Armijo: th(3.0827088513602745E-10)=-152.23877024452125; dx=3.5483419950488235 evalInputDelta=-1.1273240403397722E-5
MIN ALPHA (2.3713145010463648E-11): th(0.14766224721357338)=-152.2901900853486
Fitness changed from -152.23878151776165 to -152.2901900853486
Iteration 42 complete. Error: -152.2901900853486 Total: 21.5099; Orientation: 0.5746; Line Search: 19.6054
Adding measurement bc457ed to history. Total: 5
Rejected: LBFGS Orientation magnitude: 2.167e+07, gradient 1.893e-02, dot -0.131; [abc7744b-6c0b-4b11-8d14-4e0b7c2dc2ed = 1.000/1.000e+00]
Orientation rejected. Popping history element from -152.2901900853486, -152.23878151776165, -145.04877057977845, -4.951700700528192, -3.430423792912614, -2.759705148077441
Rejected: LBFGS Orientation magnitude: 6.330e+07, gradient 1.893e-02, dot -0.128; [abc7744b-6c0b-4b11-8d14-4e0b7c2dc2ed = 1.000/1.000e+00]
Orientation rejected. Popping history element from -152.2901900853486, -152.23878151776165, -145.04877057977845, -4.951700700528192, -3.430423792912614
Accepted: LBFGS Orientation magnitude: 1.257e+08, gradient 1.893e-02, dot 0.128; [abc7744b-6c0b-4b11-8d14-4e0b7c2dc2ed = 1.000/1.000e+00]
Overwriting history with 4 points
th(0)=-152.2901900853486;dx=-306490.0429143646
Armijo: th(2.1544346900318838E-10)=-152.29017881591372; dx=-79075.63727120624 evalInputDelta=-1.1269434878613538E-5
Armijo: th(1.0772173450159419E-10)=-152.29018445063124; dx=-77651.42943968697 evalInputDelta=-5.634717354041641E-6
MIN ALPHA (3.590724483386473E-11): th(0.0)=-152.2901900853486
Fitness changed from -152.2901900853486 to -152.2901900853486
Static Iteration Total: 9.2455; Orientation: 2.2163; Line Search: 5.6897
Iteration 43 failed. Error: -152.2901900853486
Previous Error: 0.0 -> -152.2901900853486
Retrying iteration 43
Reset training subject: 2961943945631
Adding measurement 452a04a2 to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-152.2901900853486;dx=-3.566371161129972E-4
END: th(2.1544346900318838E-10)=-152.2901900853486; dx=-2.6868699622663593E-4 evalInputDelta=0.0
Fitness changed from -152.2901900853486 to -152.2901900853486
Static Iteration Total: 5.4608; Orientation: 0.0168; Line Search: 2.7812
Iteration 44 failed. Error: -152.2901900853486
Previous Error: 0.0 -> -152.2901900853486
Optimization terminated 44
Final threshold in iteration 44: -152.2901900853486 (> -Infinity) after 397.530s (< 3600.000s)
Returns
-152.2901900853486