BasicOptimizer.scala:89 executed in 132.14 seconds (1.633 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 331940841053300
Reset training subject: 331943467726200
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
New Minimum: -109.7171464079413 > -109.82438029818282
th(0)=-109.82438029818282;dx=-2.9191249947092854E-5
New Minimum: -109.82438029818282 > -109.8244329909661
END: th(2.154434690031884)=-109.8244329909661; dx=-2.197294753844032E-5 evalInputDelta=5.269278328512428E-5
Fitness changed from -109.7171464079413 to -109.8244329909661
Iteration 1 complete. Error: -109.8244329909661 Total: 12.8479; Orientation: 0.0367; Line Search: 5.0219
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 21ac053a to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-109.8244329909661;dx=-2.19847901871266E-5
New Minimum: -109.8244329909661 > -109.82455154963607
WOLFE (weak): th(4.641588833612779)=-109.82455154963607; dx=-2.1969860878148142E-5 evalInputDelta=1.1855866996768327E-4
New Minimum: -109.82455154963607 > -109.82466571712322
WOLFE (weak): th(9.283177667225559)=-109.82466571712322; dx=-2.1963189830495293E-5 evalInputDelta=2.327261571224426E-4
New Minimum: -109.82466571712322 > -109.82513116795798
WOLFE (weak): th(27.849533001676676)=-109.82513116795798; dx=-2.1951512383462424E-5 evalInputDelta=6.981769918752434E-4
New Minimum: -109.82513116795798 > -109.82722128135089
WOLFE (weak): th(111.3981320067067)=-109.82722128135089; dx=-2.1932048755103917E-5 evalInputDelta=0.0027882903847853413
New Minimum: -109.82722128135089 > -109.83834300151887
WOLFE (weak): th(556.9906600335335)=-109.83834300151887; dx=-2.1919367188115008E-5 evalInputDelta=0.013910010552763197
New Minimum: -109.83834300151887 > -109.90780960687113
WOLFE (weak): th(3341.9439602012008)=-109.90780960687113; dx=-2.1906894707702487E-5 evalInputDelta=0.08337661590502421
New Minimum: -109.90780960687113 > -110.40583416854655
WOLFE (weak): th(23393.607721408407)=-110.40583416854655; dx=-2.172945149026347E-5 evalInputDelta=0.581401177580446
New Minimum: -110.40583416854655 > -114.30188850515228
WOLFE (weak): th(187148.86177126726)=-114.30188850515228; dx=-2.005434728672768E-5 evalInputDelta=4.477455514186175
New Minimum: -114.30188850515228 > -134.16355067568026
END: th(1684339.7559414054)=-134.16355067568026; dx=-5.119867728864578E-6 evalInputDelta=24.339117684714154
Fitness changed from -109.8244329909661 to -134.16355067568026
Iteration 2 complete. Error: -134.16355067568026 Total: 29.1354; Orientation: 0.0605; Line Search: 25.9627
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 3b8a035c to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-134.16355067568026;dx=-1.0937475277229533E-5
New Minimum: -134.16355067568026 > -151.3720802210217
END: th(3628800.0000000005)=-151.3720802210217; dx=-5.079126237608704E-7 evalInputDelta=17.208529545341435
Fitness changed from -134.16355067568026 to -151.3720802210217
Iteration 3 complete. Error: -151.3720802210217 Total: 8.0185; Orientation: 0.0600; Line Search: 5.0456
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 5cb19069 to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-151.3720802210217;dx=-1.1234386408153912E-5
Armijo: th(7818012.603187701)=-146.89512819513837; dx=3.4432029924705283E-6 evalInputDelta=-4.476952025883321
New Minimum: -151.3720802210217 > -162.48502309215678
WOLF (strong): th(3909006.3015938504)=-162.48502309215678; dx=2.2779153061623263E-6 evalInputDelta=11.11294287113509
END: th(1303002.1005312835)=-162.3936857143111; dx=-4.1950339659985645E-6 evalInputDelta=11.0216054932894
Fitness changed from -151.3720802210217 to -162.48502309215678
Iteration 4 complete. Error: -162.48502309215678 Total: 13.1891; Orientation: 0.0602; Line Search: 10.1664
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 1d1068de to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.865e+04, gradient 3.438e-03, dot -0.513; [25fbc75e-e6c9-428f-ac9c-8b7fdbdc90b6 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -162.48502309215678, -151.3720802210217, -134.16355067568026, -109.8244329909661
LBFGS Accumulation History: 3 points
Removed measurement 1d1068de to history. Total: 3
th(0)=-162.48502309215678;dx=-1.1812269512519295E-5
New Minimum: -162.48502309215678 > -178.73306920111045
WOLF (strong): th(2807232.926569009)=-178.73306920111045; dx=5.691226186457771E-7 evalInputDelta=16.248046108953673
END: th(1403616.4632845046)=-175.82273041932805; dx=-4.940374310738421E-6 evalInputDelta=13.337707327171273
Fitness changed from -162.48502309215678 to -178.73306920111045
Iteration 5 complete. Error: -178.73306920111045 Total: 11.7009; Orientation: 1.1042; Line Search: 7.6868
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 3e402434 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.631e+04, gradient 2.594e-03, dot -0.527; [25fbc75e-e6c9-428f-ac9c-8b7fdbdc90b6 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -178.73306920111045, -151.3720802210217, -134.16355067568026, -109.8244329909661
LBFGS Accumulation History: 3 points
Removed measurement 3e402434 to history. Total: 3
th(0)=-178.73306920111045;dx=-6.727191915919849E-6
New Minimum: -178.73306920111045 > -188.50063553014374
WOLF (strong): th(3024000.0000000005)=-188.50063553014374; dx=1.6463931566610388E-7 evalInputDelta=9.767566329033286
END: th(1512000.0000000002)=-186.6725865500644; dx=-2.8581456834850128E-6 evalInputDelta=7.939517348953956
Fitness changed from -178.73306920111045 to -188.50063553014374
Iteration 6 complete. Error: -188.50063553014374 Total: 11.9093; Orientation: 1.0373; Line Search: 7.7435
Adding measurement 63b35ff8 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.164e+04, gradient 2.444e-03, dot -0.540; [25fbc75e-e6c9-428f-ac9c-8b7fdbdc90b6 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -188.50063553014374, -151.3720802210217, -134.16355067568026, -109.8244329909661
LBFGS Accumulation History: 3 points
Removed measurement 63b35ff8 to history. Total: 3
th(0)=-188.50063553014374;dx=-5.970715802644246E-6
New Minimum: -188.50063553014374 > -195.89423490913066
WOLF (strong): th(3257505.251328209)=-195.89423490913066; dx=9.843074677107942E-7 evalInputDelta=7.39359937898692
END: th(1628752.6256641045)=-195.5839245424273; dx=-1.96827711352011E-6 evalInputDelta=7.083289012283558
Fitness changed from -188.50063553014374 to -195.89423490913066
Iteration 7 complete. Error: -195.89423490913066 Total: 11.1804; Orientation: 0.9973; Line Search: 7.5678
Adding measurement 7c1a6cd7 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.779e+04, gradient 2.565e-03, dot -0.476; [25fbc75e-e6c9-428f-ac9c-8b7fdbdc90b6 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -195.89423490913066, -151.3720802210217, -134.16355067568026, -109.8244329909661
LBFGS Accumulation History: 3 points
Removed measurement 7c1a6cd7 to history. Total: 3
th(0)=-195.89423490913066;dx=-6.575846807488314E-6
New Minimum: -195.89423490913066 > -201.96117609012444
WOLF (strong): th(3509041.158211262)=-201.96117609012444; dx=2.194565250428232E-6 evalInputDelta=6.06694118099378
New Minimum: -201.96117609012444 > -203.56308408389037
END: th(1754520.579105631)=-203.56308408389037; dx=-1.3767211523869574E-6 evalInputDelta=7.668849174759714
Fitness changed from -195.89423490913066 to -203.56308408389037
Iteration 8 complete. Error: -203.56308408389037 Total: 11.2531; Orientation: 1.0113; Line Search: 7.8122
Adding measurement 7dfc191b to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.442e+04, gradient 1.686e-03, dot -0.690; [25fbc75e-e6c9-428f-ac9c-8b7fdbdc90b6 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -203.56308408389037, -151.3720802210217, -134.16355067568026, -109.8244329909661
LBFGS Accumulation History: 3 points
Removed measurement 7dfc191b to history. Total: 3
th(0)=-203.56308408389037;dx=-2.838458085299564E-6
New Minimum: -203.56308408389037 > -210.58078853587014
END: th(3780000.0000000014)=-210.58078853587014; dx=-8.927376058666673E-7 evalInputDelta=7.0177044519797676
Fitness changed from -203.56308408389037 to -210.58078853587014
Iteration 9 complete. Error: -210.58078853587014 Total: 8.7141; Orientation: 0.9704; Line Search: 5.1305
Adding measurement 429f0be to history. Total: 3
Rejected: LBFGS Orientation magnitude: 4.558e+04, gradient 2.003e-03, dot -0.544; [25fbc75e-e6c9-428f-ac9c-8b7fdbdc90b6 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -210.58078853587014, -151.3720802210217, -134.16355067568026, -109.8244329909661
LBFGS Accumulation History: 3 points
Removed measurement 429f0be to history. Total: 3
th(0)=-210.58078853587014;dx=-4.000265051569049E-6
Armijo: th(8143763.128320524)=-206.87190355561313; dx=1.8225949264555345E-6 evalInputDelta=-3.708884980257011
New Minimum: -210.58078853587014 > -214.52925538268428
WOLF (strong): th(4071881.564160262)=-214.52925538268428; dx=8.676823853745507E-7 evalInputDelta=3.948466846814142
New Minimum: -214.52925538268428 > -214.66224648963095
END: th(1357293.8547200873)=-214.66224648963095; dx=-1.6217583864357505E-6 evalInputDelta=4.081457953760804
Fitness changed from -210.58078853587014 to -214.66224648963095
Iteration 10 complete. Error: -214.66224648963095 Total: 13.7195; Orientation: 1.0349; Line Search: 10.2706
<a id="p-13"></a>Iteration 10
<a id="p-12"></a>
Final threshold in iteration 11: -214.66224648963095 (> -Infinity) after 132.139s (< 5400.000s)
Returns
-214.66224648963095