Code from BasicOptimizer.scala:75 executed in 44.39 seconds (0.339 gc):
val lineSearchInstance: LineSearchStrategy = lineSearchFactory
IterativeTrainer.wrap(trainable)
.setOrientation(orientation())
.setMonitor(new TrainingMonitor() {
override def clear(): Unit = trainingMonitor.clear()
override def log(msg: String): Unit = trainingMonitor.log(msg)
override def onStepFail(currentPoint: Step): Boolean = {
BasicOptimizer.this.onStepFail(trainable, currentPoint)
}
override def onStepComplete(currentPoint: Step): Unit = {
if (0 < logEvery && (0 == currentPoint.iteration % logEvery || currentPoint.iteration < logEvery)) {
val image = currentImage
timelineAnimation += image
val caption = "Iteration " + currentPoint.iteration
out.p(caption + "\n" + out.jpg(image, caption))
}
BasicOptimizer.this.onStepComplete(trainable, currentPoint)
trainingMonitor.onStepComplete(currentPoint)
super.onStepComplete(currentPoint)
}
})
.setTimeout(trainingMinutes, TimeUnit.MINUTES)
.setMaxIterations(trainingIterations)
.setLineSearchFactory((_: CharSequence) => lineSearchInstance)
.setTerminateThreshold(java.lang.Double.NEGATIVE_INFINITY)
.runAndFree
.asInstanceOf[lang.Double]
Reset training subject: 500547805731
THREADS = 64
DEBUG_LIFECYCLE = false
DOUBLE_CACHE_MODE = WEAK
THREADS = 64
SINGLE_THREADED = false
BACKPROP_AGG_SIZE = 1
Initialized CoreSettings = {
"doubleCacheMode" : "WEAK",
"backpropAggregationSize" : 1,
"jvmThreads" : 64,
"singleThreaded" : false
}
Forcing Single-GPU Mode
Found 1 devices
Device 0 - Tesla V100-SXM2-16GB
Default Limit StackSize = 1024
Default Limit FifoSize = 1048576
Default Limit HeapSize = 8388608
Default Limit SyncDepth = 2
Default Limit PendingLaunch = 2048
Configured Limit StackSize = 1024
Configured Limit FifoSize = 8388608
Configured Limit HeapSize = 4194304
Configured Limit SyncDepth = 2
Configured Limit PendingLaunch = 2048
Reset training subject: 542991506120
Adding measurement 2ed4ae54 to history. Total: 0
LBFGS Accumulation History: 1 points
Constructing line search parameters: GD+Trust
th(0)=1.6711385250091553;dx=9.896509247574664E-11 (ERROR: Starting derivative negative)
Fitness changed from 0.5078068971633911 to 0.5078068971633911
Static Iteration Total: 43.3157; Orientation: 0.0939; Line Search: 0.5486
Iteration 1 failed. Error: 0.5078068971633911
Previous Error: 0.0 -> 0.5078068971633911
Retrying iteration 1
Reset training subject: 543864563573
Adding measurement 21cc31ca to history. Total: 0
LBFGS Accumulation History: 1 points
th(0)=1.6711385250091553;dx=9.896509247574664E-11 (ERROR: Starting derivative negative)
Fitness changed from 0.5078068971633911 to 0.5078068971633911
Static Iteration Total: 1.0625; Orientation: 0.0728; Line Search: 0.8117
Iteration 2 failed. Error: 0.5078068971633911
Previous Error: 0.0 -> 0.5078068971633911
Optimization terminated 2
Final threshold in iteration 2: 0.5078068971633911 (> -Infinity) after 44.379s (< 3600.000s)
0.5078068971633911