Code from BasicOptimizer.scala:75 executed in 48.54 seconds (0.000 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
IterativeTrainer.wrap(trainable)
.setOrientation(orientation())
.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = trainingMonitor.log(msg)
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable, currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable, currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
.setTimeout(trainingMinutes, TimeUnit.MINUTES)
.setMaxIterations(trainingIterations)
.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
.runAndFree
.asInstanceOf[lang.Double]
Reset training subject: 720165616368
THREADS = 64
DEBUG_LIFECYCLE = false
DOUBLE_CACHE_MODE = WEAK
THREADS = 64
SINGLE_THREADED = false
BACKPROP_AGG_SIZE = 1
Initialized CoreSettings = {
"doubleCacheMode" : "WEAK",
"backpropAggregationSize" : 1,
"jvmThreads" : 64,
"singleThreaded" : false
}
Forcing Single-GPU Mode
Found 1 devices
Device 0 - Tesla V100-SXM2-16GB
Default Limit StackSize = 1024
Default Limit FifoSize = 1048576
Default Limit HeapSize = 8388608
Default Limit SyncDepth = 2
Default Limit PendingLaunch = 2048
Configured Limit StackSize = 1024
Configured Limit FifoSize = 8388608
Configured Limit HeapSize = 4194304
Configured Limit SyncDepth = 2
Configured Limit PendingLaunch = 2048
Reset training subject: 767980233860
Adding measurement 1c1fa3b5 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD+Trust
th(0)=1.6711207101518109;dx=9.896531924801988E-11 (ERROR: Starting derivative negative)
Fitness changed from 0.5078205522897911 to 0.5078205522897911
Static Iteration Total: 48.5261; Orientation: 0.0831; Line Search: 0.4548
Iteration 1 failed. Error: 0.5078205522897911
Previous Error: 0.0 -> 0.5078205522897911
Optimization terminated 1
Final threshold in iteration 1: 0.5078205522897911 (> -Infinity) after 48.527s (< 3600.000s)
0.5078205522897911