Created
July 11, 2016 14:06
-
-
Save fac2003/7d948f7687194e03a3f066bf0965ef99 to your computer and use it in GitHub Desktop.
running out of memory with dl4j multi-GPU
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [mas2182@node007 dl4j-spark-cdh5-examples]$ java -Xmx10g -cp target/dl4j-spark-cdh5-examples-1.0-SNAPSHOT.jar org.deeplearning4j.examples.rnn.GravesLSTMCharModellingExample |tee GPU-benchmark-0.4.0-1.txt | |
| o.n.n.NativeOps - Number of threads used for linear algebra 32 | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [0]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [1]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [2]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [3]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [4]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [5]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [6]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.a.c.i.BasicContextPool - Creating new stream for thread: [1], device: [7]... | |
| o.n.j.c.CudaAffinityManager - Single device is forced, mapping to device [0] | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [27] to device [0], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [28] to device [0], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [29] to device [0], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [30] to device [0], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [31] to device [0], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [32] to device [0], out of [8] devices... | |
| o.n.j.h.i.CudaZeroHandler - Creating bucketID: 2 | |
| o.n.j.h.i.CudaZeroHandler - Creating bucketID: 4 | |
| o.n.j.h.i.CudaZeroHandler - Creating bucketID: 0 | |
| o.n.j.h.i.CudaZeroHandler - Creating bucketID: 3 | |
| o.n.j.h.i.CudaZeroHandler - Creating bucketID: 5 | |
| o.n.j.h.i.CudaZeroHandler - Creating bucketID: 1 | |
| Using existing text file at /tmp/Shakespeare.txt | |
| o.n.j.c.CudaAffinityManager - Mapping thread [150] to device [0], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [137] to device [1], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [139] to device [2], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [143] to device [3], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [149] to device [4], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [146] to device [5], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [133] to device [6], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [121] to device [7], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [144] to device [0], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [128] to device [1], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [138] to device [2], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [131] to device [3], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [135] to device [4], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [123] to device [5], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [119] to device [6], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [125] to device [7], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [129] to device [0], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [130] to device [1], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [127] to device [2], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [136] to device [3], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [122] to device [4], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [124] to device [5], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [148] to device [6], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [134] to device [7], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [132] to device [0], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [120] to device [1], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [126] to device [2], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [145] to device [3], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [140] to device [4], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [142] to device [5], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [141] to device [6], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Mapping thread [147] to device [7], out of [8] devices... | |
| o.d.s.i.p.ParameterAveragingTrainingMaster - Starting training of split 1 of 1. workerMiniBatchSize=8, averagingFreq=3, Configured for 32 workers | |
| [Stage 4:=======> (4 + 28) / 32]o.n.j.c.CudaAffinityManager - Manually mapping thread [387] to device [7], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [390] to device [4], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [385] to device [2], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [389] to device [5], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [384] to device [1], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [391] to device [3], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [388] to device [5], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [386] to device [2], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [392] to device [4], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [394] to device [1], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [395] to device [0], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [393] to device [1], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [399] to device [6], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [400] to device [2], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [401] to device [5], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [402] to device [0], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [403] to device [7], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [404] to device [1], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [406] to device [2], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [405] to device [0], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [407] to device [4], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [408] to device [4], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [409] to device [6], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [410] to device [3], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [411] to device [6], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [412] to device [0], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [413] to device [7], out of [8] devices... | |
| o.n.j.c.CudaAffinityManager - Manually mapping thread [414] to device [5], out of [8] devices... | |
| CUDA error at /skymind/libnd4j/blas/cuda/NativeOps.cu:3947 code=77(<unknown>) "result" | |
| CUDA error at /skymind/libnd4j/blas/cuda/NativeOps.cu:3947 code=77(<unknown>) "result" | |
| CUDA error at /skymind/libnd4j/blas/cuda/NativeOps.cu:3947 code=77(<unknown>) "result" | |
| Failed on [139634353886016] -> [35652985856], size: [307692], direction: [0], result: [77] | |
| Failed on [139634690209024] -> [35816996864], size: [307692], direction: [0], result: [77] | |
| Failed on [139633548760464] -> [35528638464], size: [307692], direction: [0], result: [77] | |
| CUDA error at /skymind/libnd4j/blas/cuda/NativeOps.cu:3947 code=77(<unknown>) "result" | |
| Failed on [139634287676976] -> [35513217536], size: [307692], direction: [0], result: [77] | |
| CUDA error at /skymind/libnd4j/blas/cuda/NativeOps.cu:3988 code=77(<unknown>) "result" | |
| CUDA error at /skymind/libnd4j/blas/cuda/NativeOps.cu:3947 code=77(<unknown>) "result" | |
| Failed on [139634152799744] -> [35562808320], size: [307692], direction: [0], result: [77] | |
| CUDA error at /skymind/libnd4j/blas/cuda/NativeOps.cu:3947 code=77(<unknown>) "result" | |
| Failed on [35904815104] -> [180770963456], size: [2239508], direction: [1], result: [77] | |
| CUDA error at /skymind/libnd4j/blas/cuda/NativeOps.cu:3988 code=77(<unknown>) "result" | |
| o.n.j.h.i.CudaZeroHandler - Out of [DEVICE] memory, host memory will be used instead: deviceId: [5], requested bytes: [2239508] | |
| o.n.j.h.i.CudaZeroHandler - Out of [DEVICE] memory, host memory will be used instead: deviceId: [5], requested bytes: [2239508] | |
| o.n.j.h.i.CudaZeroHandler - Out of [DEVICE] memory, host memory will be used instead: deviceId: [4], requested bytes: [2239508] | |
| o.a.s.e.Executor - Exception in task 0.0 in stage 4.0 (TID 128) | |
| java.lang.IllegalStateException: MemcpyAsync relocate H2D failed: [35904815104] -> [180770963456] | |
| at org.nd4j.jita.handler.impl.CudaZeroHandler.relocate(CudaZeroHandler.java:366) ~[dl4j-spark-cdh5-examples-1.0-SNAPSHOT.jar:na] | |
| at org.nd4j.jita.handler.impl.CudaZeroHandler.getDevicePointer(CudaZeroHandler.java:733) ~[dl4j-spark-cdh5-examples-1.0-SNAPSHOT.jar:na] | |
| at org.nd4j.jita.allocator.impl.AtomicAllocator.getPointer(AtomicAllocator.java:256) ~[dl4j-spark-cdh5-examples-1.0-SNAPSHOT.jar:na] | |
| at org.nd4j.linalg.jcublas.ops.executioner.JCudaExecutioner.invoke(JCudaExecutioner.java:1001) ~[dl4j-spark-cdh5-examples-1.0-SNAPSHOT.jar:na] | |
| at org.nd4j.linalg.jcublas.ops.executioner.JCudaExecutioner.exec(JCudaExecutioner.java:552) ~[dl4j-spark-cdh5-examples-1.0-SNAPSHOT.jar:na] | |
| o.a.s.e.Executor - Exception in task 26.0 in stage 4.0 (TID 154) | |
| java.lang.RuntimeException: java.lang.RuntimeException: java.lang.IllegalStateException: MemcpyAsync H2H failed: [139634287676976] -> [35513217536] | |
| at org.nd4j.linalg.api.ndarray.BaseNDArrayProxy.readObject(BaseNDArrayProxy.java:68) ~[dl4j-spark-cdh5-examples-1.0-SNAPSHOT.jar:na] | |
| at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source) ~[na:na] | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91] | |
| at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91] |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment