Created
November 5, 2014 23:02
-
-
Save eyala/d71116d54b69c4f075a7 to your computer and use it in GitHub Desktop.
build/test/logs/TEST-org.apache.pig.test.TestCommit.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Testsuite: org.apache.pig.test.TestCommit | |
| Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 31.061 sec | |
| ------------- Standard Output --------------- | |
| 2014-11-05 21:15:05,143 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:/// | |
| 2014-11-05 21:15:05,854 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable | |
| 2014-11-05 21:15:05,902 [main] WARN org.apache.pig.newplan.BaseOperatorPlan - Encountered Warning IMPLICIT_CAST_TO_INT 2 time(s). | |
| 2014-11-05 21:15:05,920 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: COGROUP,GROUP_BY,ORDER_BY,FILTER | |
| 2014-11-05 21:15:05,971 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer, PredicatePushdownOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter]} | |
| 2014-11-05 21:15:06,085 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false | |
| 2014-11-05 21:15:06,104 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.CombinerOptimizerUtil - Choosing to move algebraic foreach to combiner | |
| 2014-11-05 21:15:06,124 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SecondaryKeyOptimizerMR - Using Secondary Key Optimization for MapReduce node scope-49 | |
| 2014-11-05 21:15:06,127 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer - Rewrite: POPackage->POForEach to POPackage(JoinPackager) | |
| 2014-11-05 21:15:06,139 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 4 | |
| 2014-11-05 21:15:06,140 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 4 | |
| 2014-11-05 21:15:06,183 [main] INFO org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script settings are added to the job | |
| 2014-11-05 21:15:06,193 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 | |
| 2014-11-05 21:15:06,195 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Reduce phase detected, estimating # of required reducers. | |
| 2014-11-05 21:15:06,196 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Using reducer estimator: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator | |
| 2014-11-05 21:15:06,199 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=145 | |
| 2014-11-05 21:15:06,199 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting Parallelism to 1 | |
| 2014-11-05 21:15:06,223 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job | |
| 2014-11-05 21:15:06,228 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, will not generate code. | |
| 2014-11-05 21:15:06,228 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Starting process to move generated code to distributed cacche | |
| 2014-11-05 21:15:06,228 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Distributed cache not supported or needed in local mode. Setting key [pig.schematuple.local.dir] with code temp directory: /tmp/pig_junit_tmp1236119171/1415214906228-0 | |
| 2014-11-05 21:15:06,309 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission. | |
| 2014-11-05 21:15:06,321 [JobControl] WARN org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. | |
| 2014-11-05 21:15:06,328 [JobControl] WARN org.apache.hadoop.mapred.JobClient - No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). | |
| 2014-11-05 21:15:06,359 [JobControl] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 | |
| 2014-11-05 21:15:06,359 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1 | |
| 2014-11-05 21:15:06,365 [JobControl] WARN org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library not loaded | |
| 2014-11-05 21:15:06,367 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1 | |
| 2014-11-05 21:15:06,369 [JobControl] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 | |
| 2014-11-05 21:15:06,369 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1 | |
| 2014-11-05 21:15:06,370 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1 | |
| 2014-11-05 21:15:06,620 [Thread-2] INFO org.apache.hadoop.util.ProcessTree - setsid exited with exit code 0 | |
| 2014-11-05 21:15:06,631 [Thread-2] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@22762e43 | |
| 2014-11-05 21:15:06,649 [Thread-2] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader - Current split being processed file:/tmp/pig_junit_tmp1236119171/tmp722023357692972726voter:0+82 | |
| 2014-11-05 21:15:06,655 [Thread-2] INFO org.apache.hadoop.mapred.MapTask - io.sort.mb = 100 | |
| 2014-11-05 21:15:06,692 [Thread-2] INFO org.apache.hadoop.mapred.MapTask - data buffer = 79691776/99614720 | |
| 2014-11-05 21:15:06,692 [Thread-2] INFO org.apache.hadoop.mapred.MapTask - record buffer = 262144/327680 | |
| 2014-11-05 21:15:06,705 [Thread-2] INFO org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not set... will not generate code. | |
| 2014-11-05 21:15:06,719 [Thread-2] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map - Aliases being processed per job phase (AliasName[line,offset]): M: a[1,4],c[3,4],e[5,4],b[2,4],d[4,4],e[5,4] C: R: | |
| 2014-11-05 21:15:06,727 [Thread-2] INFO org.apache.hadoop.mapred.MapTask - Starting flush of map output | |
| 2014-11-05 21:15:06,734 [Thread-2] INFO org.apache.hadoop.mapred.MapTask - Finished spill 0 | |
| 2014-11-05 21:15:06,736 [Thread-2] INFO org.apache.hadoop.mapred.Task - Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting | |
| 2014-11-05 21:15:06,817 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0001 | |
| 2014-11-05 21:15:06,817 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases a,b,c,d,e | |
| 2014-11-05 21:15:06,817 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: a[1,4],c[3,4],e[5,4],b[2,4],d[4,4],e[5,4] C: R: | |
| 2014-11-05 21:15:06,820 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete | |
| 2014-11-05 21:15:06,820 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_local_0001] | |
| 2014-11-05 21:15:09,597 [Thread-2] INFO org.apache.hadoop.mapred.LocalJobRunner - | |
| 2014-11-05 21:15:09,597 [Thread-2] INFO org.apache.hadoop.mapred.Task - Task 'attempt_local_0001_m_000000_0' done. | |
| 2014-11-05 21:15:09,605 [Thread-2] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@17f58b1d | |
| 2014-11-05 21:15:09,609 [Thread-2] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader - Current split being processed file:/tmp/pig_junit_tmp1236119171/tmp102360492696102533student:0+63 | |
| 2014-11-05 21:15:09,610 [Thread-2] INFO org.apache.hadoop.mapred.MapTask - io.sort.mb = 100 | |
| 2014-11-05 21:15:09,643 [Thread-2] INFO org.apache.hadoop.mapred.MapTask - data buffer = 79691776/99614720 | |
| 2014-11-05 21:15:09,643 [Thread-2] INFO org.apache.hadoop.mapred.MapTask - record buffer = 262144/327680 | |
| 2014-11-05 21:15:09,656 [Thread-2] WARN org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been initialized | |
| 2014-11-05 21:15:09,670 [Thread-2] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map - Aliases being processed per job phase (AliasName[line,offset]): M: a[1,4],c[3,4],e[5,4],b[2,4],d[4,4],e[5,4] C: R: | |
| 2014-11-05 21:15:09,671 [Thread-2] INFO org.apache.hadoop.mapred.MapTask - Starting flush of map output | |
| 2014-11-05 21:15:09,673 [Thread-2] INFO org.apache.hadoop.mapred.MapTask - Finished spill 0 | |
| 2014-11-05 21:15:09,675 [Thread-2] INFO org.apache.hadoop.mapred.Task - Task:attempt_local_0001_m_000001_0 is done. And is in the process of commiting | |
| 2014-11-05 21:15:09,822 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 12% complete | |
| 2014-11-05 21:15:09,823 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_local_0001] | |
| 2014-11-05 21:15:12,601 [Thread-2] INFO org.apache.hadoop.mapred.LocalJobRunner - | |
| 2014-11-05 21:15:12,602 [Thread-2] INFO org.apache.hadoop.mapred.Task - Task 'attempt_local_0001_m_000001_0' done. | |
| 2014-11-05 21:15:12,622 [Thread-2] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7fef9cd1 | |
| 2014-11-05 21:15:12,622 [Thread-2] INFO org.apache.hadoop.mapred.LocalJobRunner - | |
| 2014-11-05 21:15:12,626 [Thread-2] INFO org.apache.hadoop.mapred.Merger - Merging 2 sorted segments | |
| 2014-11-05 21:15:12,631 [Thread-2] INFO org.apache.hadoop.mapred.Merger - Down to the last merge-pass, with 2 segments left of total size: 204 bytes | |
| 2014-11-05 21:15:12,631 [Thread-2] INFO org.apache.hadoop.mapred.LocalJobRunner - | |
| 2014-11-05 21:15:12,648 [Thread-2] WARN org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been initialized | |
| 2014-11-05 21:15:12,681 [Thread-2] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce - Aliases being processed per job phase (AliasName[line,offset]): M: a[1,4],c[3,4],e[5,4],b[2,4],d[4,4],e[5,4] C: R: | |
| 2014-11-05 21:15:12,686 [Thread-2] INFO org.apache.hadoop.mapred.Task - Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting | |
| 2014-11-05 21:15:12,687 [Thread-2] INFO org.apache.hadoop.mapred.LocalJobRunner - | |
| 2014-11-05 21:15:12,687 [Thread-2] INFO org.apache.hadoop.mapred.Task - Task attempt_local_0001_r_000000_0 is allowed to commit now | |
| 2014-11-05 21:15:12,689 [Thread-2] INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task 'attempt_local_0001_r_000000_0' to file:/tmp/temp-158856303/tmp1228236859 | |
| 2014-11-05 21:15:15,619 [Thread-2] INFO org.apache.hadoop.mapred.LocalJobRunner - reduce > reduce | |
| 2014-11-05 21:15:15,621 [Thread-2] INFO org.apache.hadoop.mapred.Task - Task 'attempt_local_0001_r_000000_0' done. | |
| 2014-11-05 21:15:15,829 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 25% complete | |
| 2014-11-05 21:15:15,849 [main] INFO org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script settings are added to the job | |
| 2014-11-05 21:15:15,850 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 | |
| 2014-11-05 21:15:15,850 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Reduce phase detected, estimating # of required reducers. | |
| 2014-11-05 21:15:15,850 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Using reducer estimator: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator | |
| 2014-11-05 21:15:15,851 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=164 | |
| 2014-11-05 21:15:15,851 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting Parallelism to 1 | |
| 2014-11-05 21:15:15,858 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job | |
| 2014-11-05 21:15:15,884 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission. | |
| 2014-11-05 21:15:15,886 [JobControl] WARN org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. | |
| 2014-11-05 21:15:15,887 [JobControl] WARN org.apache.hadoop.mapred.JobClient - No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). | |
| 2014-11-05 21:15:15,897 [JobControl] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 | |
| 2014-11-05 21:15:15,897 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1 | |
| 2014-11-05 21:15:15,897 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1 | |
| 2014-11-05 21:15:15,972 [Thread-9] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@10144efe | |
| 2014-11-05 21:15:15,977 [Thread-9] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader - Current split being processed file:/tmp/temp-158856303/tmp1228236859/part-r-00000:0+164 | |
| 2014-11-05 21:15:15,977 [Thread-9] INFO org.apache.hadoop.mapred.MapTask - io.sort.mb = 100 | |
| 2014-11-05 21:15:16,006 [Thread-9] INFO org.apache.hadoop.mapred.MapTask - data buffer = 79691776/99614720 | |
| 2014-11-05 21:15:16,006 [Thread-9] INFO org.apache.hadoop.mapred.MapTask - record buffer = 262144/327680 | |
| 2014-11-05 21:15:16,015 [Thread-9] WARN org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been initialized | |
| 2014-11-05 21:15:16,021 [Thread-9] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map - Aliases being processed per job phase (AliasName[line,offset]): M: h[8,4],g[7,4] C: h[8,4],g[7,4] R: h[8,4] | |
| 2014-11-05 21:15:16,022 [Thread-9] INFO org.apache.hadoop.mapred.MapTask - Starting flush of map output | |
| 2014-11-05 21:15:16,033 [Thread-9] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine - Aliases being processed per job phase (AliasName[line,offset]): M: h[8,4],g[7,4] C: h[8,4],g[7,4] R: h[8,4] | |
| 2014-11-05 21:15:16,035 [Thread-9] INFO org.apache.hadoop.mapred.MapTask - Finished spill 0 | |
| 2014-11-05 21:15:16,036 [Thread-9] INFO org.apache.hadoop.mapred.Task - Task:attempt_local_0002_m_000000_0 is done. And is in the process of commiting | |
| 2014-11-05 21:15:16,385 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0002 | |
| 2014-11-05 21:15:16,385 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases g,h | |
| 2014-11-05 21:15:16,385 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: h[8,4],g[7,4] C: h[8,4],g[7,4] R: h[8,4] | |
| 2014-11-05 21:15:18,965 [Thread-9] INFO org.apache.hadoop.mapred.LocalJobRunner - | |
| 2014-11-05 21:15:18,966 [Thread-9] INFO org.apache.hadoop.mapred.Task - Task 'attempt_local_0002_m_000000_0' done. | |
| 2014-11-05 21:15:18,977 [Thread-9] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@a637589 | |
| 2014-11-05 21:15:18,977 [Thread-9] INFO org.apache.hadoop.mapred.LocalJobRunner - | |
| 2014-11-05 21:15:18,978 [Thread-9] INFO org.apache.hadoop.mapred.Merger - Merging 1 sorted segments | |
| 2014-11-05 21:15:18,978 [Thread-9] INFO org.apache.hadoop.mapred.Merger - Down to the last merge-pass, with 1 segments left of total size: 61 bytes | |
| 2014-11-05 21:15:18,978 [Thread-9] INFO org.apache.hadoop.mapred.LocalJobRunner - | |
| 2014-11-05 21:15:18,990 [Thread-9] WARN org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been initialized | |
| 2014-11-05 21:15:19,000 [Thread-9] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce - Aliases being processed per job phase (AliasName[line,offset]): M: h[8,4],g[7,4] C: h[8,4],g[7,4] R: h[8,4] | |
| 2014-11-05 21:15:19,001 [Thread-9] INFO org.apache.hadoop.mapred.Task - Task:attempt_local_0002_r_000000_0 is done. And is in the process of commiting | |
| 2014-11-05 21:15:19,002 [Thread-9] INFO org.apache.hadoop.mapred.LocalJobRunner - | |
| 2014-11-05 21:15:19,002 [Thread-9] INFO org.apache.hadoop.mapred.Task - Task attempt_local_0002_r_000000_0 is allowed to commit now | |
| 2014-11-05 21:15:19,003 [Thread-9] INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task 'attempt_local_0002_r_000000_0' to file:/tmp/temp-158856303/tmp-1874861385 | |
| 2014-11-05 21:15:19,395 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 37% complete | |
| 2014-11-05 21:15:19,395 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_local_0002] | |
| 2014-11-05 21:15:21,973 [Thread-9] INFO org.apache.hadoop.mapred.LocalJobRunner - reduce > reduce | |
| 2014-11-05 21:15:21,974 [Thread-9] INFO org.apache.hadoop.mapred.Task - Task 'attempt_local_0002_r_000000_0' done. | |
| 2014-11-05 21:15:22,399 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete | |
| 2014-11-05 21:15:22,407 [main] INFO org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script settings are added to the job | |
| 2014-11-05 21:15:22,408 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 | |
| 2014-11-05 21:15:22,409 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Reduce phase detected, estimating # of required reducers. | |
| 2014-11-05 21:15:22,409 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Using reducer estimator: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator | |
| 2014-11-05 21:15:22,411 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=51 | |
| 2014-11-05 21:15:22,411 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting Parallelism to 1 | |
| 2014-11-05 21:15:22,422 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job | |
| 2014-11-05 21:15:22,468 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission. | |
| 2014-11-05 21:15:22,470 [JobControl] WARN org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. | |
| 2014-11-05 21:15:22,471 [JobControl] WARN org.apache.hadoop.mapred.JobClient - No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). | |
| 2014-11-05 21:15:22,484 [JobControl] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 | |
| 2014-11-05 21:15:22,484 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1 | |
| 2014-11-05 21:15:22,485 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1 | |
| 2014-11-05 21:15:22,550 [Thread-12] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4902d674 | |
| 2014-11-05 21:15:22,552 [Thread-12] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader - Current split being processed file:/tmp/temp-158856303/tmp-1874861385/part-r-00000:0+51 | |
| 2014-11-05 21:15:22,553 [Thread-12] INFO org.apache.hadoop.mapred.MapTask - io.sort.mb = 100 | |
| 2014-11-05 21:15:22,588 [Thread-12] INFO org.apache.hadoop.mapred.MapTask - data buffer = 79691776/99614720 | |
| 2014-11-05 21:15:22,588 [Thread-12] INFO org.apache.hadoop.mapred.MapTask - record buffer = 262144/327680 | |
| 2014-11-05 21:15:22,595 [Thread-12] WARN org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been initialized | |
| 2014-11-05 21:15:22,598 [Thread-12] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map - Aliases being processed per job phase (AliasName[line,offset]): M: i[9,4] C: R: | |
| 2014-11-05 21:15:22,599 [Thread-12] INFO org.apache.hadoop.mapred.MapTask - Starting flush of map output | |
| 2014-11-05 21:15:22,600 [Thread-12] INFO org.apache.hadoop.mapred.MapTask - Finished spill 0 | |
| 2014-11-05 21:15:22,602 [Thread-12] INFO org.apache.hadoop.mapred.Task - Task:attempt_local_0003_m_000000_0 is done. And is in the process of commiting | |
| 2014-11-05 21:15:22,969 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0003 | |
| 2014-11-05 21:15:22,969 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases i | |
| 2014-11-05 21:15:22,969 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: i[9,4] C: R: | |
| 2014-11-05 21:15:25,546 [Thread-12] INFO org.apache.hadoop.mapred.LocalJobRunner - | |
| 2014-11-05 21:15:25,547 [Thread-12] INFO org.apache.hadoop.mapred.Task - Task 'attempt_local_0003_m_000000_0' done. | |
| 2014-11-05 21:15:25,562 [Thread-12] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@61effe24 | |
| 2014-11-05 21:15:25,562 [Thread-12] INFO org.apache.hadoop.mapred.LocalJobRunner - | |
| 2014-11-05 21:15:25,564 [Thread-12] INFO org.apache.hadoop.mapred.Merger - Merging 1 sorted segments | |
| 2014-11-05 21:15:25,565 [Thread-12] INFO org.apache.hadoop.mapred.Merger - Down to the last merge-pass, with 1 segments left of total size: 66 bytes | |
| 2014-11-05 21:15:25,565 [Thread-12] INFO org.apache.hadoop.mapred.LocalJobRunner - | |
| 2014-11-05 21:15:25,569 [Thread-12] WARN org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been initialized | |
| 2014-11-05 21:15:25,583 [Thread-12] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce - Aliases being processed per job phase (AliasName[line,offset]): M: i[9,4] C: R: | |
| 2014-11-05 21:15:25,584 [Thread-12] INFO org.apache.hadoop.mapred.Task - Task:attempt_local_0003_r_000000_0 is done. And is in the process of commiting | |
| 2014-11-05 21:15:25,585 [Thread-12] INFO org.apache.hadoop.mapred.LocalJobRunner - | |
| 2014-11-05 21:15:25,585 [Thread-12] INFO org.apache.hadoop.mapred.Task - Task attempt_local_0003_r_000000_0 is allowed to commit now | |
| 2014-11-05 21:15:25,587 [Thread-12] INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task 'attempt_local_0003_r_000000_0' to file:/tmp/temp-158856303/tmp1846078548 | |
| 2014-11-05 21:15:25,977 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 62% complete | |
| 2014-11-05 21:15:25,978 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_local_0003] | |
| 2014-11-05 21:15:28,555 [Thread-12] INFO org.apache.hadoop.mapred.LocalJobRunner - reduce > reduce | |
| 2014-11-05 21:15:28,557 [Thread-12] INFO org.apache.hadoop.mapred.Task - Task 'attempt_local_0003_r_000000_0' done. | |
| 2014-11-05 21:15:28,981 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 75% complete | |
| 2014-11-05 21:15:28,988 [main] INFO org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script settings are added to the job | |
| 2014-11-05 21:15:28,989 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 | |
| 2014-11-05 21:15:28,990 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Reduce phase detected, estimating # of required reducers. | |
| 2014-11-05 21:15:28,990 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting Parallelism to 1 | |
| 2014-11-05 21:15:29,000 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job | |
| 2014-11-05 21:15:29,012 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission. | |
| 2014-11-05 21:15:29,015 [JobControl] WARN org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. | |
| 2014-11-05 21:15:29,016 [JobControl] WARN org.apache.hadoop.mapred.JobClient - No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). | |
| 2014-11-05 21:15:29,023 [JobControl] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 | |
| 2014-11-05 21:15:29,023 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1 | |
| 2014-11-05 21:15:29,023 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1 | |
| 2014-11-05 21:15:29,082 [Thread-15] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@767bdf2a | |
| 2014-11-05 21:15:29,086 [Thread-15] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader - Current split being processed file:/tmp/temp-158856303/tmp-1874861385/part-r-00000:0+51 | |
| 2014-11-05 21:15:29,087 [Thread-15] INFO org.apache.hadoop.mapred.MapTask - io.sort.mb = 100 | |
| 2014-11-05 21:15:29,149 [Thread-15] INFO org.apache.hadoop.mapred.MapTask - data buffer = 79691776/99614720 | |
| 2014-11-05 21:15:29,149 [Thread-15] INFO org.apache.hadoop.mapred.MapTask - record buffer = 262144/327680 | |
| 2014-11-05 21:15:29,154 [Thread-15] WARN org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been initialized | |
| 2014-11-05 21:15:29,155 [Thread-15] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map - Aliases being processed per job phase (AliasName[line,offset]): M: i[9,4] C: R: | |
| 2014-11-05 21:15:29,159 [Thread-15] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 | |
| 2014-11-05 21:15:29,159 [Thread-15] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1 | |
| 2014-11-05 21:15:29,161 [Thread-15] INFO org.apache.hadoop.mapred.MapTask - Starting flush of map output | |
| 2014-11-05 21:15:29,162 [Thread-15] INFO org.apache.hadoop.mapred.MapTask - Finished spill 0 | |
| 2014-11-05 21:15:29,164 [Thread-15] INFO org.apache.hadoop.mapred.Task - Task:attempt_local_0004_m_000000_0 is done. And is in the process of commiting | |
| 2014-11-05 21:15:29,512 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0004 | |
| 2014-11-05 21:15:29,513 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases i | |
| 2014-11-05 21:15:29,513 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: i[9,4] C: R: | |
| 2014-11-05 21:15:32,073 [Thread-15] INFO org.apache.hadoop.mapred.LocalJobRunner - | |
| 2014-11-05 21:15:32,074 [Thread-15] INFO org.apache.hadoop.mapred.Task - Task 'attempt_local_0004_m_000000_0' done. | |
| 2014-11-05 21:15:32,088 [Thread-15] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@6759a771 | |
| 2014-11-05 21:15:32,088 [Thread-15] INFO org.apache.hadoop.mapred.LocalJobRunner - | |
| 2014-11-05 21:15:32,090 [Thread-15] INFO org.apache.hadoop.mapred.Merger - Merging 1 sorted segments | |
| 2014-11-05 21:15:32,090 [Thread-15] INFO org.apache.hadoop.mapred.Merger - Down to the last merge-pass, with 1 segments left of total size: 57 bytes | |
| 2014-11-05 21:15:32,090 [Thread-15] INFO org.apache.hadoop.mapred.LocalJobRunner - | |
| 2014-11-05 21:15:32,104 [Thread-15] WARN org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been initialized | |
| 2014-11-05 21:15:32,108 [Thread-15] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce - Aliases being processed per job phase (AliasName[line,offset]): M: i[9,4] C: R: | |
| 2014-11-05 21:15:32,109 [Thread-15] INFO org.apache.hadoop.mapred.Task - Task:attempt_local_0004_r_000000_0 is done. And is in the process of commiting | |
| 2014-11-05 21:15:32,109 [Thread-15] INFO org.apache.hadoop.mapred.LocalJobRunner - | |
| 2014-11-05 21:15:32,109 [Thread-15] INFO org.apache.hadoop.mapred.Task - Task attempt_local_0004_r_000000_0 is allowed to commit now | |
| 2014-11-05 21:15:32,111 [Thread-15] INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task 'attempt_local_0004_r_000000_0' to file:/tmp/temp-158856303/tmp-665727465 | |
| 2014-11-05 21:15:32,521 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 87% complete | |
| 2014-11-05 21:15:32,521 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_local_0004] | |
| 2014-11-05 21:15:35,082 [Thread-15] INFO org.apache.hadoop.mapred.LocalJobRunner - reduce > reduce | |
| 2014-11-05 21:15:35,084 [Thread-15] INFO org.apache.hadoop.mapred.Task - Task 'attempt_local_0004_r_000000_0' done. | |
| 2014-11-05 21:15:35,536 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete | |
| 2014-11-05 21:15:35,539 [main] INFO org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics: | |
| HadoopVersion PigVersion UserId StartedAt FinishedAt Features | |
| 1.0.4 0.14.0-SNAPSHOT eyal 2014-11-05 21:15:06 2014-11-05 21:15:35 COGROUP,GROUP_BY,ORDER_BY,FILTER | |
| Success! | |
| Job Stats (time in seconds): | |
| JobId Maps Reduces MaxMapTime MinMapTime AvgMapTime MedianMapTime MaxReduceTime MinReduceTime AvgReduceTime MedianReducetime Alias Feature Outputs | |
| job_local_0001 1 1 n/a n/a n/a n/a n/a n/a n/a n/a a,b,c,d,e COGROUP | |
| job_local_0002 1 1 n/a n/a n/a n/a n/a n/a n/a n/a g,h GROUP_BY,COMBINER | |
| job_local_0003 1 1 n/a n/a n/a n/a n/a n/a n/a n/a i SAMPLER | |
| job_local_0004 1 1 n/a n/a n/a n/a n/a n/a n/a n/a i ORDER_BY file:/tmp/temp-158856303/tmp-665727465, | |
| Input(s): | |
| Successfully read 4 records from: "/tmp/pig_junit_tmp1236119171/tmp102360492696102533student" | |
| Successfully read 3 records from: "/tmp/pig_junit_tmp1236119171/tmp722023357692972726voter" | |
| Output(s): | |
| Successfully stored 2 records in: "file:/tmp/temp-158856303/tmp-665727465" | |
| Counters: | |
| Total records written : 2 | |
| Total bytes written : 0 | |
| Spillable Memory Manager spill count : 0 | |
| Total bags proactively spilled: 0 | |
| Total records proactively spilled: 0 | |
| Job DAG: | |
| job_local_0001 -> job_local_0002, | |
| job_local_0002 -> job_local_0003, | |
| job_local_0003 -> job_local_0004, | |
| job_local_0004 | |
| 2014-11-05 21:15:35,544 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success! | |
| 2014-11-05 21:15:35,545 [main] WARN org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been initialized | |
| 2014-11-05 21:15:35,546 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 | |
| 2014-11-05 21:15:35,546 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1 | |
| 2014-11-05 21:15:35,577 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:/// | |
| 2014-11-05 21:15:35,685 [main] WARN org.apache.pig.newplan.BaseOperatorPlan - Encountered Warning IMPLICIT_CAST_TO_DOUBLE 1 time(s). | |
| ------------- ---------------- --------------- | |
| Testcase: testCheckin1 took 30.9 sec | |
| Testcase: testCheckin2 took 0.145 sec | |
| Caused an ERROR | |
| Unable to store alias i | |
| org.apache.pig.PigException: ERROR 1002: Unable to store alias i | |
| at org.apache.pig.PigServer.storeEx(PigServer.java:1038) | |
| at org.apache.pig.PigServer.store(PigServer.java:997) | |
| at org.apache.pig.PigServer.store(PigServer.java:965) | |
| at org.apache.pig.test.TestCommit.testCheckin2(TestCommit.java:103) | |
| Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 6000: Output Location Validation Failed for: 'testCheckin2-output.txt More info to follow: | |
| Output directory testCheckin2-output.txt already exists | |
| at org.apache.pig.newplan.logical.visitor.InputOutputFileValidatorVisitor.visit(InputOutputFileValidatorVisitor.java:95) | |
| at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66) | |
| at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64) | |
| at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) | |
| at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) | |
| at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) | |
| at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) | |
| at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) | |
| at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53) | |
| at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52) | |
| at org.apache.pig.newplan.logical.relational.LogicalPlan.validate(LogicalPlan.java:212) | |
| at org.apache.pig.PigServer$Graph.compile(PigServer.java:1767) | |
| at org.apache.pig.PigServer$Graph.access$300(PigServer.java:1443) | |
| at org.apache.pig.PigServer.storeEx(PigServer.java:1033) | |
| Caused by: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory testCheckin2-output.txt already exists | |
| at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137) | |
| at org.apache.pig.newplan.logical.visitor.InputOutputFileValidatorVisitor.visit(InputOutputFileValidatorVisitor.java:69) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment