Last active
January 23, 2019 21:25
-
-
Save ryanpersaud/149262950ef9d86df5e3716027671d66 to your computer and use it in GitHub Desktop.
Error encountered when attempting to launch Storm topology with HDF 2.1
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Darrell Kienzle noted the following: | |
I downloaded HDF 2.1 and tweaked my vagrant rig to load all the new stuff from HDF instead of HDP. There was one less storm dependency | |
on HDF – no need for the “atlas_metadata” which HDP includes & requires. Anyway, when I tried to run the parser, I got the exception | |
below. | |
Long story short, it _looks_ like the submission is a two-step process now. First, it tries to do some fixups in | |
ClientJarTransformerRunner” and output a tmp jar in /tmp. This is per: | |
• STORM-1202: Migrate APIs to org.apache.storm, but try to provide some form of backwards compatability | |
Then it tries to run the fixed up jar. But since the first one throws an exception, there’s no jar for it there to run! | |
I _think_ this is the culprit | |
Storm.py from HortonWorks says: | |
transform_class = confvalue("client.jartransformer.class", [CLUSTER_CONF_DIR]) | |
if (transform_class != None and transform_class != "null"): | |
tmpjar = os.path.join(tempfile.gettempdir(), uuid.uuid1().hex+".jar") | |
exec_storm_class("org.apache.storm.daemon.ClientJarTransformerRunner", args=[transform_class, jarfile, tmpjar], fork=True, daemon=False) | |
Storm.py from Apache 1.0.2 says: | |
transform_class = confvalue("client.jartransformer.class", [CLUSTER_CONF_DIR]) | |
if (transform_class != None and transform_class != "nil"): | |
tmpjar = os.path.join(tempfile.gettempdir(), uuid.uuid1().hex+".jar") | |
exec_storm_class("org.apache.storm.daemon.ClientJarTransformerRunner", args=[transform_class, jarfile, tmpjar], fork=True, daemon=False) | |
I think the workaround is to add that key to storm.ini and set it to “null” | |
I added this to storm.yaml: | |
client.jartransformer.class: "null" | |
==> storm: /opt/storm/bin/storm: line 2: /usr/hdf/2.1.0.0-165/etc/default/hadoop: No such file or directory | |
==> storm: Running: java -server -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdf/2.1.0.0-165/storm -Dstorm.log.dir=/usr/hdf/2.1.0.0-165/storm/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/hdf/2.1.0.0-165/storm/lib/asm-5.0.3.jar:/usr/hdf/2.1.0.0-165/storm/lib/clojure-1.7.0.jar:/usr/hdf/2.1.0.0-165/storm/lib/disruptor-3.3.2.jar:/usr/hdf/2.1.0.0-165/storm/lib/kryo-3.0.3.jar:/usr/hdf/2.1.0.0-165/storm/lib/log4j-api-2.1.jar:/usr/hdf/2.1.0.0-165/storm/lib/log4j-core-2.1.jar:/usr/hdf/2.1.0.0-165/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdf/2.1.0.0-165/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdf/2.1.0.0-165/storm/lib/minlog-1.3.0.jar:/usr/hdf/2.1.0.0-165/storm/lib/objenesis-2.1.jar:/usr/hdf/2.1.0.0-165/storm/lib/reflectasm-1.10.1.jar:/usr/hdf/2.1.0.0-165/storm/lib/ring-cors-0.1.5.jar:/usr/hdf/2.1.0.0-165/storm/lib/servlet-api-2.5.jar:/usr/hdf/2.1.0.0-165/storm/lib/slf4j-api-1.7.7.jar:/usr/hdf/2.1.0.0-165/storm/lib/storm-core-1.0.2.2.1.0.0-165.jar:/usr/hdf/2.1.0.0-165/storm/lib/storm-rename-hack-1.0.2.2.1.0.0-165.jar:/usr/hdf/2.1.0.0-165/storm/lib/zookeeper.jar:/usr/hdf/2.1.0.0-165/storm/lib/jackson-core-2.3.1.jar org.apache.storm.daemon.ClientJarTransformerRunner nil /vagrant/stormparser-0.75_storm1.jar /tmp/060dc788c45e11e680c0000c29bbb94c.jar | |
==> storm: Exception in thread "main" | |
==> storm: java.lang.RuntimeException: java.lang.ClassNotFoundException: nil | |
==> storm: at org.apache.storm.utils.Utils.newInstance(Utils.java:183) | |
==> storm: at org.apache.storm.utils.Utils.jarTransformer(Utils.java:203) | |
==> storm: at org.apache.storm.daemon.ClientJarTransformerRunner.main(ClientJarTransformerRunner.java:34) | |
==> storm: Caused by: java.lang.ClassNotFoundException: nil | |
==> storm: at java.net.URLClassLoader.findClass(URLClassLoader.java:381) | |
==> storm: at java.lang.ClassLoader.loadClass(ClassLoader.java:424) | |
==> storm: at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) | |
==> storm: at java.lang.ClassLoader.loadClass(ClassLoader.java:357) | |
==> storm: at java.lang.Class.forName0(Native Method) | |
==> storm: at java.lang.Class.forName(Class.java:264) | |
==> storm: at org.apache.storm.utils.Utils.newInstance(Utils.java:181) | |
==> storm: ... 2 more | |
==> storm: Running: java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdf/2.1.0.0-165/storm -Dstorm.log.dir=/usr/hdf/2.1.0.0-165/storm/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/hdf/2.1.0.0-165/storm/lib/asm-5.0.3.jar:/usr/hdf/2.1.0.0-165/storm/lib/clojure-1.7.0.jar:/usr/hdf/2.1.0.0-165/storm/lib/disruptor-3.3.2.jar:/usr/hdf/2.1.0.0-165/storm/lib/kryo-3.0.3.jar:/usr/hdf/2.1.0.0-165/storm/lib/log4j-api-2.1.jar:/usr/hdf/2.1.0.0-165/storm/lib/log4j-core-2.1.jar:/usr/hdf/2.1.0.0-165/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdf/2.1.0.0-165/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdf/2.1.0.0-165/storm/lib/minlog-1.3.0.jar:/usr/hdf/2.1.0.0-165/storm/lib/objenesis-2.1.jar:/usr/hdf/2.1.0.0-165/storm/lib/reflectasm-1.10.1.jar:/usr/hdf/2.1.0.0-165/storm/lib/ring-cors-0.1.5.jar:/usr/hdf/2.1.0.0-165/storm/lib/servlet-api-2.5.jar:/usr/hdf/2.1.0.0-165/storm/lib/slf4j-api-1.7.7.jar:/usr/hdf/2.1.0.0-165/storm/lib/storm-core-1.0.2.2.1.0.0-165.jar:/usr/hdf/2.1.0.0-165/storm/lib/storm-rename-hack-1.0.2.2.1.0.0-165.jar:/usr/hdf/2.1.0.0-165/storm/lib/zookeeper.jar:/usr/hdf/2.1.0.0-165/storm/lib/jackson-core-2.3.1.jar:/tmp/060dc788c45e11e680c0000c29bbb94c.jar:/usr/hdf/2.1.0.0-165/storm/conf:/usr/hdf/2.1.0.0-165/storm/bin -Dstorm.jar=/tmp/060dc788c45e11e680c0000c29bbb94c.jar -Dstorm.dependency.jars= -Dstorm.dependency.artifacts={} org.apache.storm.flux.Flux --remote /vagrant/flux/parser_json1.yaml -f /vagrant/flux/local.properties | |
==> storm: Error: Could not find or load main class org.apache.storm.flux.Flux | |
==> storm: Traceback (most recent call last): | |
==> storm: File "/usr/hdf/2.1.0.0-165/storm/bin/storm.py", line 883, in <module> | |
==> storm: main() | |
==> storm: File "/usr/hdf/2.1.0.0-165/storm/bin/storm.py", line 880, in main | |
==> storm: (COMMANDS.get(COMMAND, unknown_command))(*ARGS) | |
==> storm: File "/usr/hdf/2.1.0.0-165/storm/bin/storm.py", line 313, in jar | |
==> storm: os.remove(tmpjar) | |
==> storm: OSError: [Errno 2] No such file or directory: '/tmp/060dc788c45e11e680c0000c29bbb94c.jar' | |
The SSH command responded with a non-zero exit status. Vagrant | |
assumes that this means the command failed. The output for this command | |
should be in the log above. Please read the output to determine what | |
went wrong. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Excuse me, has this problem been solved? I have encountered the same problem.