Iteration 1 Iteration 1

Iteration 2 Iteration 2

Iteration 3 Iteration 3

Iteration 4 Iteration 4

Iteration 5 Iteration 5

Iteration 10 Iteration 10

Iteration 15 Iteration 15

Code from BasicOptimizer.scala:75 executed in 3666.57 seconds (33.941 gc):

    val lineSearchInstance: LineSearchStrategy = lineSearchFactory
    IterativeTrainer.wrap(trainable)
      .setOrientation(orientation())
      .setMonitor(new TrainingMonitor() {
        override def clear(): Unit = trainingMonitor.clear()
  
        override def log(msg: String): Unit = trainingMonitor.log(msg)
  
        override def onStepFail(currentPoint: Step): Boolean = {
          BasicOptimizer.this.onStepFail(trainable, currentPoint)
        }
  
        override def onStepComplete(currentPoint: Step): Unit = {
          if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
            val image = currentImage
            timelineAnimation += image
            val caption = "Iteration " + currentPoint.iteration
            out.p(caption + "\n" + out.jpg(image, caption))
          }
          BasicOptimizer.this.onStepComplete(trainable, currentPoint)
          trainingMonitor.onStepComplete(currentPoint)
          super.onStepComplete(currentPoint)
        }
      })
      .setTimeout(trainingMinutes, TimeUnit.MINUTES)
      .setMaxIterations(trainingIterations)
      .setLineSearchFactory((_: CharSequence) => lineSearchInstance)
      .setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
      .runAndFree
      .asInstanceOf[lang.Double]

Logging:

    Reset training subject: 1055007821679099
    Reset training subject: 1055060978333000
    Adding measurement 3a62755a to history. Total: 0
    LBFGS Accumulation History: 1 points
    Constructing line search parameters: GD+Trust
    th(0)=-0.5683215335011482;dx=-6.898395595379401E-8
    New Minimum: -0.5683215335011482 > -0.5683216527104378
    WOLFE (weak): th(2.154434690031884)=-0.5683216527104378; dx=-6.868942222570603E-8 evalInputDelta=1.1920928955078125E-7
    New Minimum: -0.5683216527104378 > -0.5683217495679855
    WOLFE (weak): th(4.308869380063768)=-0.5683217495679855; dx=-6.868290086155453E-8 evalInputDelta=2.1606683731079102E-7
    New Minimum: -0.5683217495679855 > -0.5683225318789482
    WOLFE (weak): th(12.926608140191302)=-0.5683225318789482; dx=-6.869875965114347E-8 evalInputDelta=9.98377799987793E-7
    New Minimum: -0.5683225318789482 > -0.5683254823088646
    WOLFE (weak): th(51.70643256076521)=-0.5683254823088646; dx=-6.869517941212851E-8 evalInputDelta=3.948807716369629E-6
    New Minimum: -0.5683254823088646 > -0.5683406814932823
    WOLFE (weak): th(258.53216280382605)=-0.5683406814932823; dx=-6.869536440283367E-8 evalInputDelta=1.9147992134094238E-5
    New Minimum: -0.5683406814932823 > -0.568435937166214
    WOLFE (weak): th(1551.1929768229563)=-0.568435937166214; dx=-6.868323108580613E-8 evalInputDelta=1.1440366506576538E-4
    New Minimum: -0.568435937166214 > -0.5691213086247444
    WOLFE (weak): th(10858.350837760694)=-0.5691213086247444; dx=-6.865038024304426E-8 evalInputDelta=7.997751235961914E-4
    New Minimum: -0.5691213086247444 > -0.5747068151831627
    WOLFE (weak): th(86866.80670208555)=-0.5747068151831627; dx=-6.842490803412719E-8 evalInputDelta=0.006385281682014465
    New Minimum: -0.5747068151831627 > -0.6247304603457451
    WOLFE (weak): th(781801.26031877)=-0.6247304603457451; dx=-6.614717658159607E-8 evalInputDelta=0.05640892684459686
    New Minimum: -0.6247304603457451 > -1.0178711712360382
    END: th(7818012.6031877)=-1.0178711712360382; dx=-4.330336318489431E-8 evalInputDelta=0.44954963773489
    Fitness changed from -0.5683215335011482 

...skipping 11819 bytes...

    11E-9 evalInputDelta=0.012568458914756775
    Fitness changed from -2.3889106810092926 to -2.4014791399240494
    Iteration 14 complete. Error: -2.4014791399240494 Total: 257.7414; Orientation: 1.9950; Line Search: 204.7517
    Adding measurement 62990ea3 to history. Total: 3
    Rejected: LBFGS Orientation magnitude: 3.094e+03, gradient 3.639e-05, dot -0.949; [9ec609af-62d5-4d73-a894-8d1d84d58d26 = 1.000/1.000e+00]
    Orientation rejected. Popping history element from -2.368079423904419, -2.376838818192482, -2.3889106810092926, -2.4014791399240494
    LBFGS Accumulation History: 3 points
    Removed measurement 10f03ab4 to history. Total: 4
    th(0)=-2.4014791399240494;dx=-1.3227536329390267E-9
    New Minimum: -2.4014791399240494 > -2.4094203487038612
    END: th(1.750000000000001E7)=-2.4094203487038612; dx=-1.145413326139436E-9 evalInputDelta=0.007941208779811859
    Fitness changed from -2.4014791399240494 to -2.4094203487038612
    Iteration 15 complete. Error: -2.4094203487038612 Total: 159.5152; Orientation: 2.0340; Line Search: 105.5265
    <a id="p-15"></a>Iteration 15
    <a id="p-14"></a>![Iteration 15](etc/394bb6f9-c0cb-44c6-88ea-9a1b7ed22657.jpg)
    
    Adding measurement 32c903fd to history. Total: 3
    Rejected: LBFGS Orientation magnitude: 8.006e+03, gradient 3.468e-05, dot -0.962; [9ec609af-62d5-4d73-a894-8d1d84d58d26 = 1.000/1.000e+00]
    Orientation rejected. Popping history element from -2.376838818192482, -2.3889106810092926, -2.4014791399240494, -2.4094203487038612
    LBFGS Accumulation History: 3 points
    Removed measurement 17d4ac58 to history. Total: 4
    th(0)=-2.4094203487038612;dx=-1.2012540239555312E-9
    New Minimum: -2.4094203487038612 > -2.4219851419329643
    END: th(3.770260707555799E7)=-2.4219851419329643; dx=-9.757765335158654E-10 evalInputDelta=0.012564793229103088
    Fitness changed from -2.4094203487038612 to -2.4219851419329643
    Iteration 16 complete. Error: -2.4219851419329643 Total: 157.4009; Orientation: 2.0201; Line Search: 101.8335
    Final threshold in iteration 16: -2.4219851419329643 (> -Infinity) after 3666.565s (< 3600.000s)
    

Returns:

    -2.4219851419329643