BasicOptimizer.scala:89 executed in 243.94 seconds (3.008 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 1733768482773
Reset training subject: 1734609232892
Adding measurement 12580466 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD+Trust
th(0)=-2.7605577464823456;dx=-7.324568067615606E-4
New Minimum: -2.7605577464823456 > -2.7619861019431906
WOLFE (weak): th(2.154434690031884)=-2.7619861019431906; dx=-7.318195071842617E-4 evalInputDelta=0.0014283554608449656
New Minimum: -2.7619861019431906 > -2.7634164026583923
WOLFE (weak): th(4.308869380063768)=-2.7634164026583923; dx=-7.252120572958878E-4 evalInputDelta=0.0028586561760466367
New Minimum: -2.7634164026583923 > -2.7691119676237386
WOLFE (weak): th(12.926608140191302)=-2.7691119676237386; dx=-7.257034533818613E-4 evalInputDelta=0.008554221141392926
New Minimum: -2.7691119676237386 > -2.7946234273734754
WOLFE (weak): th(51.70643256076521)=-2.7946234273734754; dx=-7.253809056473373E-4 evalInputDelta=0.03406568089112971
New Minimum: -2.7946234273734754 > -2.929671813347973
WOLFE (weak): th(258.53216280382605)=-2.929671813347973; dx=-7.081367582863139E-4 evalInputDelta=0.16911406686562724
New Minimum: -2.929671813347973 > -3.7281930695372965
END: th(1551.1929768229563)=-3.7281930695372965; dx=-6.119446964237383E-4 evalInputDelta=0.9676353230549508
Fitness changed from -2.7605577464823456 to -3.7281930695372965
Iteration 1 complete. Error: -3.7281930695372965 Total: 8.7530; Orientation: 0.0203; Line Search: 6.2038
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 5c6eb1a4 to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-3.7281930695372965;dx=-7.018178898991396E-4
New Minimum: -3.7281930695372965 > -5.3904895042252665
END: th(3341.943960201201)=-5.3904895042252665; dx=-3.8050374232150437E-4 evalInputDelta=1.66229643468797
Fitness changed from -3.7281930695372965 to -5.3904895042252665
Iteration 2 complete. Error: -5.3904895042252665 Total: 2.6694; Orientation: 0.0196; Line Search: 1.7723
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 3d371fa8 to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-5.3904895042252665;dx=-5.479236146300064E-4
New Minimum: -5.3904895042252665 > -7.546595815874633
END: th(7200.000000000001)=-7.546595815874633; dx=-1.9419224116863987E-4 evalInputDelta=2.156106311649366
Fitness changed from -5.3904895042252665 to -7.546595815874633
Iteration 3 complete. Error: -7.546595815874633 Total: 2.6739; Orientation: 0.0206; Line Search: 1.7934
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 66b23e7e to history. Total: 3
Rejected: LBFGS Orientation magnitude: 9.423e+02, gradient 2.528e-02, dot -0.345; [8eed2d40-ca1c-468b-8a20-5ab6633341e1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -7.546595815874633, -5.3904895042252665, -3.7281930695372965, -2.7605577464823456
LBFGS Accumulation History: 3 points
Removed measurement 66b23e7e to history. Total: 3
th(0)=-7.546595815874633;dx=-6.359512136992635E-4
New Minimum: -7.546595815874633 > -8.86593077545546
WOLF (strong): th(15511.929768229566)=-8.86593077545546; dx=6.726891768646035E-6 evalInputDelta=1.3193349595808277
New Minimum: -8.86593077545546 > -9.055575462103963
END: th(7755.964884114783)=-9.055575462103963; dx=-5.586644493725626E-5 evalInputDelta=1.5089796462293306
Fitness changed from -7.546595815874633 to -9.055575462103963
Iteration 4 complete. Error: -9.055575462103963 Total: 4.2155; Orientation: 0.4907; Line Search: 2.8533
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 18e30a to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.421e+03, gradient 3.057e-02, dot -0.313; [8eed2d40-ca1c-468b-8a20-5ab6633341e1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -9.055575462103963, -5.3904895042252665, -3.7281930695372965, -2.7605577464823456
LBFGS Accumulation History: 3 points
Removed measurement 18e30a to history. Total: 3
th(0)=-9.055575462103963;dx=-9.31794031318361E-4
New Minimum: -9.055575462103963 > -9.424369174749826
WOLF (strong): th(16709.71980100601)=-9.424369174749826; dx=8.632000112054086E-5 evalInputDelta=0.36879371264586247
New Minimum: -9.424369174749826 > -10.39960152058277
WOLF (strong): th(8354.859900503005)=-10.39960152058277; dx=1.0967171552703922E-4 evalInputDelta=1.344026058478807
New Minimum: -10.39960152058277 > -10.474628722135847
END: th(2784.953300167668)=-10.474628722135847; dx=-2.7264793804150785E-4 evalInputDelta=1.419053260031884
Fitness changed from -9.055575462103963 to -10.474628722135847
Iteration 5 complete. Error: -10.474628722135847 Total: 5.6807; Orientation: 0.4896; Line Search: 4.3130
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 22a09b7 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.576e+03, gradient 2.194e-02, dot -0.371; [8eed2d40-ca1c-468b-8a20-5ab6633341e1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -10.474628722135847, -5.3904895042252665, -3.7281930695372965, -2.7605577464823456
LBFGS Accumulation History: 3 points
Removed measurement 22a09b7 to history. Total: 3
th(0)=-10.474628722135847;dx=-4.7892866567025855E-4
New Minimum: -10.474628722135847 > -11.70137192960428
END: th(6000.000000000002)=-11.70137192960428; dx=-1.3631592224545923E-4 evalInputDelta=1.2267432074684326
Fitness changed from -10.474628722135847 to -11.70137192960428
Iteration 6 complete. Error: -11.70137192960428 Total: 3.1442; Orientation: 0.5007; Line Search: 1.7774
Adding measurement 5dbd33ff to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.102e+03, gradient 2.638e-02, dot -0.331; [8eed2d40-ca1c-468b-8a20-5ab6633341e1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -11.70137192960428, -5.3904895042252665, -3.7281930695372965, -2.7605577464823456
LBFGS Accumulation History: 3 points
Removed measurement 5dbd33ff to history. Total: 3
th(0)=-11.70137192960428;dx=-6.938563042806896E-4
New Minimum: -11.70137192960428 > -12.121405446373872
WOLF (strong): th(12926.608140191307)=-12.121405446373872; dx=8.026018812305779E-5 evalInputDelta=0.42003351676959255
New Minimum: -12.121405446373872 > -12.686166435355782
WOLF (strong): th(6463.3040700956535)=-12.686166435355782; dx=4.697468242567549E-5 evalInputDelta=0.984794505751502
END: th(2154.4346900318847)=-12.518521738655085; dx=-2.652801446324921E-4 evalInputDelta=0.8171498090508056
Fitness changed from -11.70137192960428 to -12.686166435355782
Iteration 7 complete. Error: -12.686166435355782 Total: 4.8777; Orientation: 0.4862; Line Search: 3.5510
Adding measurement 4bca6f56 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.610e+03, gradient 3.140e-02, dot -0.222; [8eed2d40-ca1c-468b-8a20-5ab6633341e1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -12.686166435355782, -5.3904895042252665, -3.7281930695372965, -2.7605577464823456
LBFGS Accumulation History: 3 points
Removed measurement 4bca6f56 to history. Total: 3
th(0)=-12.686166435355782;dx=-9.833922377225487E-4
New Minimum: -12.686166435355782 > -13.916144016896896
END: th(4641.588833612781)=-13.916144016896896; dx=-4.379071008329391E-6 evalInputDelta=1.2299775815411138
Fitness changed from -12.686166435355782 to -13.916144016896896
Iteration 8 complete. Error: -13.916144016896896 Total: 3.1086; Orientation: 0.4897; Line Search: 1.7702
Adding measurement 73cde9d4 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.817e+03, gradient 2.560e-02, dot -0.311; [8eed2d40-ca1c-468b-8a20-5ab6633341e1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -13.916144016896896, -5.3904895042252665, -3.7281930695372965, -2.7605577464823456
LBFGS Accumulation History: 3 points
Removed measurement 73cde9d4 to history. Total: 3
th(0)=-13.916144016896896;dx=-6.53251936584553E-4
New Minimum: -13.916144016896896 > -14.63030726101065
WOLF (strong): th(10000.000000000005)=-14.6
...skipping 35563 bytes...
tion 45
<a id="p-26"></a>
Adding measurement 766567ef to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.156e+05, gradient 2.772e-02, dot -0.249; [8eed2d40-ca1c-468b-8a20-5ab6633341e1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -87.9765705737874, -5.3904895042252665, -3.7281930695372965, -2.7605577464823456
LBFGS Accumulation History: 3 points
Removed measurement 766567ef to history. Total: 3
th(0)=-87.9765705737874;dx=-7.664473403855603E-4
Armijo: th(65241.23070219461)=-77.049663641205; dx=2.125960936260813E-4 evalInputDelta=-10.926906932582398
Armijo: th(32620.615351097305)=-87.75342149325036; dx=1.845870819563908E-4 evalInputDelta=-0.2231490805370413
New Minimum: -87.9765705737874 > -91.28910082719521
END: th(10873.538450365768)=-91.28910082719521; dx=-2.1572688109302977E-4 evalInputDelta=3.3125302534078145
Fitness changed from -87.9765705737874 to -91.28910082719521
Iteration 46 complete. Error: -91.28910082719521 Total: 4.9571; Orientation: 0.4902; Line Search: 3.6042
Adding measurement 68a84b26 to history. Total: 3
Accepted: LBFGS Orientation magnitude: 1.014e+05, gradient 2.251e-02, dot 0.333; [8eed2d40-ca1c-468b-8a20-5ab6633341e1 = 1.000/1.000e+00]
th(0)=-91.28910082719521;dx=-758.9027881412409
Armijo: th(23426.328440863545)=-21.18605785535234; dx=8.614726569404003E-4 evalInputDelta=-70.10304297184287
Armijo: th(11713.164220431772)=-21.1863905622612; dx=0.0017247017701510973 evalInputDelta=-70.10271026493402
Armijo: th(3904.3880734772574)=-21.203212415825156; dx=0.005196626768010908 evalInputDelta=-70.08588841137006
Armijo: th(976.0970183693144)=-21.239866892641622; dx=0.020853475436860718 evalInputDelta=-70.0492339345536
Armijo: th(195.21940367386287)=-21.24988342727283; dx=0.10508038020992885 evalInputDelta=-70.03921739992238
Armijo: th(32.536567278977145)=-21.550525391340397; dx=0.6370470009934948 evalInputDelta=-69.73857543585481
Armijo: th(4.648081039853878)=-23.23642392504192; dx=4.413602826230128 evalInputDelta=-68.0526769021533
Armijo: th(0.5810101299817347)=-57.62860420117249; dx=57.65678757899727 evalInputDelta=-33.66049662602272
New Minimum: -91.28910082719521 > -109.57070429870534
END: th(0.06455668110908164)=-109.57070429870534; dx=-130.7474979827294 evalInputDelta=18.281603471510124
Fitness changed from -91.28910082719521 to -109.57070429870534
Iteration 47 complete. Error: -109.57070429870534 Total: 10.4113; Orientation: 0.4741; Line Search: 9.1102
Adding measurement 4d219152 to history. Total: 4
Accepted: LBFGS Orientation magnitude: 6.055e+04, gradient 3.243e-02, dot 0.135; [8eed2d40-ca1c-468b-8a20-5ab6633341e1 = 1.000/1.000e+00]
th(0)=-109.57070429870534;dx=-265.02538884031213
Armijo: th(0.13908315325473147)=-104.50727584044458; dx=38.13154299468157 evalInputDelta=-5.063428458260759
Armijo: th(0.06954157662736574)=-108.89879480632077; dx=-1.087043331709392 evalInputDelta=-0.6719094923845716
New Minimum: -109.57070429870534 > -109.92721097817777
END: th(0.023180525542455244)=-109.92721097817777; dx=-42.446656373051866 evalInputDelta=0.35650667947243164
Fitness changed from -109.57070429870534 to -109.92721097817777
Iteration 48 complete. Error: -109.92721097817777 Total: 5.0087; Orientation: 0.5901; Line Search: 3.5835
Adding measurement bfe0e6f to history. Total: 5
Accepted: LBFGS Orientation magnitude: 3.033e+04, gradient 3.332e-02, dot 0.133; [8eed2d40-ca1c-468b-8a20-5ab6633341e1 = 1.000/1.000e+00]
th(0)=-109.92721097817777;dx=-134.29175425140494
Armijo: th(0.04994092836183573)=-109.63942236794817; dx=-10.068793548184809 evalInputDelta=-0.28778861022959745
Armijo: th(0.024970464180917865)=-109.85369598083449; dx=-14.83211954048113 evalInputDelta=-0.07351499734328115
Armijo: th(0.008323488060305955)=-109.92088106164455; dx=-18.40148666603602 evalInputDelta=-0.006329916533218238
Armijo: th(0.0020808720150764888)=-109.92653553153856; dx=-19.94757181765887 evalInputDelta=-6.754466392067116E-4
Armijo: th(4.1617440301529773E-4)=-109.92719562716388; dx=-20.176848152192186 evalInputDelta=-1.535101388583371E-5
New Minimum: -109.92721097817777 > -109.92721609518193
END: th(6.936240050254963E-5)=-109.92721609518193; dx=-20.647606124200383 evalInputDelta=5.117004164389982E-6
Fitness changed from -109.92721097817777 to -109.92721609518193
Iteration 49 complete. Error: -109.92721609518193 Total: 8.0395; Orientation: 0.7189; Line Search: 6.4861
Adding measurement 2ac52ec0 to history. Total: 6
Rejected: LBFGS Orientation magnitude: 6.729e+07, gradient 3.332e-02, dot -0.122; [8eed2d40-ca1c-468b-8a20-5ab6633341e1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -109.92721609518193, -109.92721097817777, -109.57070429870534, -91.28910082719521, -5.3904895042252665, -3.7281930695372965, -2.7605577464823456
Rejected: LBFGS Orientation magnitude: 1.653e+08, gradient 3.332e-02, dot -0.123; [8eed2d40-ca1c-468b-8a20-5ab6633341e1 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -109.92721609518193, -109.92721097817777, -109.57070429870534, -91.28910082719521, -5.3904895042252665, -3.7281930695372965
Accepted: LBFGS Orientation magnitude: 1.195e+08, gradient 3.332e-02, dot 0.121; [8eed2d40-ca1c-468b-8a20-5ab6633341e1 = 1.000/1.000e+00]
Overwriting history with 5 points
th(0)=-109.92721609518193;dx=-479810.91276948206
Armijo: th(1.494367618265779E-4)=-88.94435965066837; dx=111623.67859907064 evalInputDelta=-20.98285644451356
Armijo: th(7.471838091328895E-5)=-101.88628032757995; dx=97550.0343685417 evalInputDelta=-8.04093576760198
Armijo: th(2.4906126971096314E-5)=-108.87682961588447; dx=8970.403052298683 evalInputDelta=-1.0503864792974582
Armijo: th(6.226531742774079E-6)=-109.87668941936869; dx=-44954.36647454946 evalInputDelta=-0.05052667581324499
New Minimum: -109.92721609518193 > -109.92935498206013
END: th(1.2453063485548158E-6)=-109.92935498206013; dx=-63977.20064364542 evalInputDelta=0.0021388868781997417
Fitness changed from -109.92721609518193 to -109.92935498206013
Iteration 50 complete. Error: -109.92935498206013 Total: 8.3641; Orientation: 2.0908; Line Search: 5.4202
<a id="p-29"></a>Iteration 50
<a id="p-28"></a>
Adding measurement 73b121b7 to history. Total: 5
Accepted: LBFGS Orientation magnitude: 1.435e+08, gradient 3.337e-02, dot 0.120; [8eed2d40-ca1c-468b-8a20-5ab6633341e1 = 1.000/1.000e+00]
th(0)=-109.92935498206013;dx=-574335.020175105
Armijo: th(2.6829311970434318E-6)=-109.90539502418345; dx=-58817.87594552039 evalInputDelta=-0.023959957876684257
Armijo: th(1.3414655985217159E-6)=-109.92436076984724; dx=-69553.84494942629 evalInputDelta=-0.004994212212892535
Armijo: th(4.4715519950723863E-7)=-109.92904285044408; dx=-74527.7123352137 evalInputDelta=-3.1213161605592177E-4
Armijo: th(1.1178879987680966E-7)=-109.92932428062899; dx=-75336.73631056394 evalInputDelta=-3.0701431143143054E-5
Armijo: th(2.2357759975361933E-8)=-109.92934986515553; dx=-76092.26864981599 evalInputDelta=-5.1169046031418475E-6
Armijo: th(3.726293329226989E-9)=-109.92935498206013; dx=-76375.28235765445 evalInputDelta=0.0
Armijo: th(5.323276184609984E-10)=-109.92934986515553; dx=-76531.79806764728 evalInputDelta=-5.1169046031418475E-6
MIN ALPHA (6.65409523076248E-11): th(0.0)=-109.92935498206013
Fitness changed from -109.92935498206013 to -109.92935498206013
Static Iteration Total: 10.4214; Orientation: 0.7296; Line Search: 8.8086
Iteration 51 failed. Error: -109.92935498206013
Previous Error: 0.0 -> -109.92935498206013
Retrying iteration 51
Reset training subject: 1974184316250
Adding measurement 7bd192a8 to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-109.92935498206013;dx=-0.0011119754565778418
END: th(2.1544346900318838E-10)=-109.92935498206013; dx=-9.348667784139034E-4 evalInputDelta=0.0
Fitness changed from -109.92935498206013 to -109.92935498206013
Static Iteration Total: 3.5268; Orientation: 0.0222; Line Search: 1.7936
Iteration 52 failed. Error: -109.92935498206013
Previous Error: 0.0 -> -109.92935498206013
Optimization terminated 52
Final threshold in iteration 52: -109.92935498206013 (> -Infinity) after 243.943s (< 3600.000s)
Returns
-109.92935498206013