BasicOptimizer.scala:89 executed in 129.41 seconds (1.627 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
val trainer = new IterativeTrainer(trainable)
trainer.setOrientation(orientation())
trainer.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = {
trainingMonitor.log(msg)
BasicOptimizer.this.log(msg)
}
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable.addRef().asInstanceOf[Trainable], currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage()
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable.addRef().asInstanceOf[Trainable], currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
trainer.setTimeout(trainingMinutes, TimeUnit.MINUTES)
trainer.setMaxIterations(trainingIterations)
trainer.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
trainer.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
val result = trainer.run.asInstanceOf[lang.Double]
trainer.freeRef()
result
Reset training subject: 111215223216200
Reset training subject: 111217978177000
Corrupt weights measurement
LBFGS Accumulation History: 0 points
Constructing line search parameters: GD+Trust
th(0)=-30.03415289901718;dx=-1.2210969856547573E-6
WOLFE (weak): th(2.154434690031884)=-30.034155714632213; dx=-1.1023449187010948E-6 evalInputDelta=2.8156150335689745E-6
WOLFE (weak): th(4.308869380063768)=-30.034158530246984; dx=-1.1025514505422642E-6 evalInputDelta=5.631229804237137E-6
WOLFE (weak): th(12.926608140191302)=-30.034168384896603; dx=-1.1023369479574836E-6 evalInputDelta=1.5485879423238202E-5
WOLFE (weak): th(51.70643256076521)=-30.034214842486957; dx=-1.10259763236754E-6 evalInputDelta=6.194346977750342E-5
WOLFE (weak): th(258.53216280382605)=-30.03446120729544; dx=-1.1025038064409722E-6 evalInputDelta=3.0830827826022755E-4
WOLFE (weak): th(1551.1929768229563)=-30.036002701216706; dx=-1.1017533901774664E-6 evalInputDelta=0.0018498021995263514
WOLFE (weak): th(10858.350837760694)=-30.047096308528648; dx=-1.101336683109721E-6 evalInputDelta=0.012943409511468218
New Minimum: -30.10559300632656 > -30.137134853751164
END: th(86866.80670208555)=-30.137134853751164; dx=-1.089236167523109E-6 evalInputDelta=0.10298195473398408
Fitness changed from -30.10559300632656 to -30.137134853751164
Iteration 1 complete. Error: -30.137134853751164 Total: 31.8036; Orientation: 0.0376; Line Search: 23.7605
<a id="p-3"></a>Iteration 1
<a id="p-2"></a>
Adding measurement 48bc566f to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=-30.137134853751164;dx=-1.0934944382547622E-6
New Minimum: -30.137134853751164 > -30.355699471489345
WOLFE (weak): th(187148.86177126726)=-30.355699471489345; dx=-1.0657009427246183E-6 evalInputDelta=0.21856461773818125
New Minimum: -30.355699471489345 > -30.569050247370566
WOLFE (weak): th(374297.7235425345)=-30.569050247370566; dx=-1.0418132104319864E-6 evalInputDelta=0.431915393619402
New Minimum: -30.569050247370566 > -31.383280632023798
END: th(1122893.1706276035)=-31.383280632023798; dx=-9.696230309279529E-7 evalInputDelta=1.2461457782726342
Fitness changed from -30.137134853751164 to -31.383280632023798
Iteration 2 complete. Error: -31.383280632023798 Total: 13.7646; Orientation: 0.0551; Line Search: 10.6588
<a id="p-5"></a>Iteration 2
<a id="p-4"></a>
Adding measurement 6ffb86fe to history. Total: 1
LBFGS Accumulation History: 2 points
th(0)=-31.383280632023798;dx=-9.570351356555465E-7
New Minimum: -31.383280632023798 > -33.65271772329015
END: th(2419200.0000000005)=-33.65271772329015; dx=-7.80245248941104E-7 evalInputDelta=2.2694370912663544
Fitness changed from -31.383280632023798 to -33.65271772329015
Iteration 3 complete. Error: -33.65271772329015 Total: 8.5894; Orientation: 0.0573; Line Search: 5.5141
<a id="p-7"></a>Iteration 3
<a id="p-6"></a>
Adding measurement 360ec974 to history. Total: 2
LBFGS Accumulation History: 3 points
th(0)=-33.65271772329015;dx=-7.640274224037261E-7
New Minimum: -33.65271772329015 > -37.242062909558186
END: th(5212008.402125134)=-37.242062909558186; dx=-5.210335773491635E-7 evalInputDelta=3.5893451862680337
Fitness changed from -33.65271772329015 to -37.242062909558186
Iteration 4 complete. Error: -37.242062909558186 Total: 8.4399; Orientation: 0.0549; Line Search: 5.1626
<a id="p-9"></a>Iteration 4
<a id="p-8"></a>
Adding measurement 24d7ddc7 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.735e+04, gradient 7.418e-04, dot -0.805; [136d6235-b186-4e81-97f1-4dc789754bc3 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -37.242062909558186, -33.65271772329015, -31.383280632023798, -30.137134853751164
LBFGS Accumulation History: 3 points
Removed measurement 24d7ddc7 to history. Total: 3
th(0)=-37.242062909558186;dx=-5.489936370995785E-7
New Minimum: -37.242062909558186 > -42.16309203439006
END: th(1.1228931706276037E7)=-42.16309203439006; dx=-2.8405351505931496E-7 evalInputDelta=4.921029124831875
Fitness changed from -37.242062909558186 to -42.16309203439006
Iteration 5 complete. Error: -42.16309203439006 Total: 9.6890; Orientation: 1.1055; Line Search: 5.3768
<a id="p-11"></a>Iteration 5
<a id="p-10"></a>
Adding measurement 6e97d78f to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.450e+04, gradient 6.792e-04, dot -0.597; [136d6235-b186-4e81-97f1-4dc789754bc3 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -42.16309203439006, -33.65271772329015, -31.383280632023798, -30.137134853751164
LBFGS Accumulation History: 3 points
Removed measurement 6e97d78f to history. Total: 3
th(0)=-42.16309203439006;dx=-4.6073648822995033E-7
New Minimum: -42.16309203439006 > -46.77810172102045
END: th(2.4192000000000004E7)=-46.77810172102045; dx=-5.198027944937986E-8 evalInputDelta=4.615009686630387
Fitness changed from -42.16309203439006 to -46.77810172102045
Iteration 6 complete. Error: -46.77810172102045 Total: 9.4701; Orientation: 1.1856; Line Search: 5.3234
Adding measurement 62e265de to history. Total: 3
Rejected: LBFGS Orientation magnitude: 1.877e+04, gradient 7.978e-04, dot -0.320; [136d6235-b186-4e81-97f1-4dc789754bc3 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -46.77810172102045, -33.65271772329015, -31.383280632023798, -30.137134853751164
LBFGS Accumulation History: 3 points
Removed measurement 62e265de to history. Total: 3
th(0)=-46.77810172102045;dx=-6.3637071143917E-7
Armijo: th(5.212008402125134E7)=-45.847628952012016; dx=1.058097502037382E-7 evalInputDelta=-0.930472769008432
New Minimum: -46.77810172102045 > -49.64562934865304
WOLF (strong): th(2.606004201062567E7)=-49.64562934865304; dx=1.373454879988619E-7 evalInputDelta=2.8675276276325903
New Minimum: -49.64562934865304 > -50.69977326871706
END: th(8686680.670208557)=-50.69977326871706; dx=-1.8528642863043153E-7 evalInputDelta=3.9216715476966115
Fitness changed from -46.77810172102045 to -50.69977326871706
Iteration 7 complete. Error: -50.69977326871706 Total: 14.3984; Orientation: 1.0512; Line Search: 10.5556
Adding measurement 37549af1 to history. Total: 3
Rejected: LBFGS Orientation magnitude: 2.915e+04, gradient 5.873e-04, dot -0.483; [136d6235-b186-4e81-97f1-4dc789754bc3 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -50.69977326871706, -33.65271772329015, -31.383280632023798, -30.137134853751164
LBFGS Accumulation History: 3 points
Removed measurement 37549af1 to history. Total: 3
th(0)=-50.69977326871706;dx=-3.4467010096679866E-7
New Minimum: -50.69977326871706 > -53.673993960456365
END: th(1.8714886177126728E7)=-53.673993960456365; dx=-2.4760155413532395E-8 evalInputDelta=2.974220691739305
Fitness changed from -50.69977326871706 to -53.673993960456365
Iteration 8 complete. Error: -53.673993960456365 Total: 9.2905; Orientation: 1.0446; Line Search: 5.3950
Adding measurement 1d67d3df to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.064e+04, gradient 6.992e-04, dot -0.361; [136d6235-b186-4e81-97f1-4dc789754bc3 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -53.673993960456365, -33.65271772329015, -31.383280632023798, -30.137134853751164
LBFGS Accumulation History: 3 points
Removed measurement 1d67d3df to history. Total: 3
th(0)=-53.673993960456365;dx=-4.88624305180059E-7
Armijo: th(4.032000000000001E7)=-52.4957335053659; dx=1.1669619947177518E-7 evalInputDelta=-1.1782604550904665
New Minimum: -53.673993960456365 > -55.526486789821334
WOLF (strong): th(2.0160000000000004E7)=-55.526486789821334; dx=1.2653442571902204E-7 evalInputDelta=1.8524928293649694
New Minimum: -55.526486789821334 > -56.14494630593263
END: th(6720000.000000001)=-56.14494630593263; dx=-1.7762897336205975E-7 evalInputDelta=2.470952345476263
Fitness changed from -53.673993960456365 to -56.14494630593263
Iteration 9 complete. Error: -56.14494630593263 Total: 14.4350; Orientation: 1.1325; Line Search: 10.6454
Adding measurement 72a399ec to history. Total: 3
Rejected: LBFGS Orientation magnitude: 3.832e+04, gradient 4.888e-04, dot -0.543; [136d6235-b186-4e81-97f1-4dc789754bc3 = 1.000/1.000e+00]
Orientation rejected. Popping history element from -56.14494630593263, -33.65271772329015, -31.383280632023798, -30.137134853751164
LBFGS Accumulation History: 3 points
Removed measurement 72a399ec to history. Total: 3
th(0)=-56.14494630593263;dx=-2.3860264904723173E-7
New Minimum: -56.14494630593263 > -58.35867311120267
END: th(1.4477801117014261E7)=-58.35867311120267; dx=-7.543601731946553E-8 evalInputDelta=2.213726805270042
Fitness changed from -56.14494630593263 to -58.35867311120267
Iteration 10 complete. Error: -58.35867311120267 Total: 8.9724; Orientation: 0.9948; Line Search: 5.2953
<a id="p-13"></a>Iteration 10
<a id="p-12"></a>
Final threshold in iteration 11: -58.35867311120267 (> -Infinity) after 129.413s (< 5400.000s)
Returns
-58.35867311120267