Created
August 12, 2022 15:27
-
-
Save lukemarsden/cb175735e1cdcab10b5b9ab1c95c7730 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
22/08/12 16:24:39 WARN Utils: Your hostname, mind resolves to a loopback address: 127.0.1.1; using 10.1.255.235 instead (on interface enp6s0f0) | |
22/08/12 16:24:39 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address | |
22/08/12 16:24:40 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable | |
22/08/12 16:24:40 INFO SparkContext: Running Spark version 3.3.0 | |
22/08/12 16:24:40 INFO ResourceUtils: ============================================================== | |
22/08/12 16:24:40 INFO ResourceUtils: No custom resources configured for spark.driver. | |
22/08/12 16:24:40 INFO ResourceUtils: ============================================================== | |
22/08/12 16:24:40 INFO SparkContext: Submitted application: spark.py | |
22/08/12 16:24:40 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0) | |
22/08/12 16:24:40 INFO ResourceProfile: Limiting resource is cpu | |
22/08/12 16:24:40 INFO ResourceProfileManager: Added ResourceProfile id: 0 | |
22/08/12 16:24:40 INFO SecurityManager: Changing view acls to: luke | |
22/08/12 16:24:40 INFO SecurityManager: Changing modify acls to: luke | |
22/08/12 16:24:40 INFO SecurityManager: Changing view acls groups to: | |
22/08/12 16:24:40 INFO SecurityManager: Changing modify acls groups to: | |
22/08/12 16:24:40 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(luke); groups with view permissions: Set(); users with modify permissions: Set(luke); groups with modify permissions: Set() | |
22/08/12 16:24:41 INFO Utils: Successfully started service 'sparkDriver' on port 45909. | |
22/08/12 16:24:41 INFO SparkEnv: Registering MapOutputTracker | |
22/08/12 16:24:41 INFO SparkEnv: Registering BlockManagerMaster | |
22/08/12 16:24:41 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information | |
22/08/12 16:24:41 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up | |
22/08/12 16:24:41 INFO SparkEnv: Registering BlockManagerMasterHeartbeat | |
22/08/12 16:24:41 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-9de0714f-1adf-4b7d-bacc-77154eb0cf5a | |
22/08/12 16:24:41 INFO MemoryStore: MemoryStore started with capacity 434.4 MiB | |
22/08/12 16:24:41 INFO SparkEnv: Registering OutputCommitCoordinator | |
22/08/12 16:24:41 INFO Utils: Successfully started service 'SparkUI' on port 4040. | |
22/08/12 16:24:41 INFO SparkContext: Added JAR file:///home/luke/pp/pachyderm/spark/hadoop-aws-3.3.3.jar at spark://10.1.255.235:45909/jars/hadoop-aws-3.3.3.jar with timestamp 1660317880795 | |
22/08/12 16:24:41 INFO SparkContext: Added JAR file:///home/luke/pp/pachyderm/spark/aws-java-sdk-bundle-1.12.264.jar at spark://10.1.255.235:45909/jars/aws-java-sdk-bundle-1.12.264.jar with timestamp 1660317880795 | |
22/08/12 16:24:41 INFO Executor: Starting executor ID driver on host 10.1.255.235 | |
22/08/12 16:24:41 INFO Executor: Starting executor with user classpath (userClassPathFirst = false): '' | |
22/08/12 16:24:41 INFO Executor: Fetching spark://10.1.255.235:45909/jars/hadoop-aws-3.3.3.jar with timestamp 1660317880795 | |
22/08/12 16:24:41 INFO TransportClientFactory: Successfully created connection to /10.1.255.235:45909 after 32 ms (0 ms spent in bootstraps) | |
22/08/12 16:24:41 INFO Utils: Fetching spark://10.1.255.235:45909/jars/hadoop-aws-3.3.3.jar to /tmp/spark-7ddbe69a-a202-4991-b257-71af8961db3e/userFiles-1e00ef57-8902-4eb1-a9e5-60aee3067fb4/fetchFileTemp6439226125535153377.tmp | |
22/08/12 16:24:41 INFO Executor: Adding file:/tmp/spark-7ddbe69a-a202-4991-b257-71af8961db3e/userFiles-1e00ef57-8902-4eb1-a9e5-60aee3067fb4/hadoop-aws-3.3.3.jar to class loader | |
22/08/12 16:24:41 INFO Executor: Fetching spark://10.1.255.235:45909/jars/aws-java-sdk-bundle-1.12.264.jar with timestamp 1660317880795 | |
22/08/12 16:24:41 INFO Utils: Fetching spark://10.1.255.235:45909/jars/aws-java-sdk-bundle-1.12.264.jar to /tmp/spark-7ddbe69a-a202-4991-b257-71af8961db3e/userFiles-1e00ef57-8902-4eb1-a9e5-60aee3067fb4/fetchFileTemp7017306744932639789.tmp | |
22/08/12 16:24:42 INFO Executor: Adding file:/tmp/spark-7ddbe69a-a202-4991-b257-71af8961db3e/userFiles-1e00ef57-8902-4eb1-a9e5-60aee3067fb4/aws-java-sdk-bundle-1.12.264.jar to class loader | |
22/08/12 16:24:42 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38343. | |
22/08/12 16:24:42 INFO NettyBlockTransferService: Server created on 10.1.255.235:38343 | |
22/08/12 16:24:42 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy | |
22/08/12 16:24:42 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.1.255.235, 38343, None) | |
22/08/12 16:24:42 INFO BlockManagerMasterEndpoint: Registering block manager 10.1.255.235:38343 with 434.4 MiB RAM, BlockManagerId(driver, 10.1.255.235, 38343, None) | |
22/08/12 16:24:42 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.1.255.235, 38343, None) | |
22/08/12 16:24:42 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 10.1.255.235, 38343, None) | |
[('spark.driver.extraJavaOptions', '-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED'), ('spark.hadoop.fs.s3a.connection.ssl.enabled', 'false'), ('spark.app.initial.jar.urls', 'spark://10.1.255.235:45909/jars/hadoop-aws-3.3.3.jar,spark://10.1.255.235:45909/jars/aws-java-sdk-bundle-1.12.264.jar'), ('spark.driver.host', '10.1.255.235'), ('spark.hadoop.fs.s3a.path.style.access', 'true'), ('spark.repl.local.jars', 'file:///home/luke/pp/pachyderm/spark/hadoop-aws-3.3.3.jar,file:///home/luke/pp/pachyderm/spark/aws-java-sdk-bundle-1.12.264.jar'), ('spark.app.id', 'local-1660317881500'), ('spark.executor.id', 'driver'), ('spark.hadoop.fs.s3a.change.detection.mode', 'none'), ('spark.jars', 'file:///home/luke/pp/pachyderm/spark/hadoop-aws-3.3.3.jar,file:///home/luke/pp/pachyderm/spark/aws-java-sdk-bundle-1.12.264.jar'), ('spark.hadoop.fs.s3a.impl', 'org.apache.hadoop.fs.s3a.S3AFileSystem'), ('spark.app.submitTime', '1660317880334'), ('spark.hadoop.fs.s3a.change.detection.version.required', 'false'), ('spark.app.name', 'spark.py'), ('spark.rdd.compress', 'True'), ('spark.hadoop.fs.s3a.endpoint', 'http://localhost:30600'), ('spark.executor.extraJavaOptions', '-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED'), ('spark.hadoop.fs.s3a.access.key', 'lemon'), ('spark.serializer.objectStreamReset', '100'), ('spark.master', 'local[*]'), ('spark.submit.pyFiles', ''), ('spark.driver.port', '45909'), ('spark.hadoop.fs.s3a.secret.key', 'lemon'), ('spark.submit.deployMode', 'client'), ('spark.app.startTime', '1660317880795')] | |
22/08/12 16:24:42 DEBUG FileSystem: Looking for FS supporting file | |
22/08/12 16:24:42 DEBUG FileSystem: looking for configuration option fs.file.impl | |
22/08/12 16:24:42 DEBUG FileSystem: Looking in service filesystems for implementation class | |
22/08/12 16:24:42 DEBUG FileSystem: FS for file is class org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem | |
22/08/12 16:24:42 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir. | |
22/08/12 16:24:42 INFO SharedState: Warehouse path is 'file:/home/luke/pp/pachyderm/spark/spark-warehouse'. | |
22/08/12 16:24:42 DEBUG FsUrlStreamHandlerFactory: Creating handler for protocol jar | |
22/08/12 16:24:42 DEBUG FileSystem: Looking for FS supporting jar | |
22/08/12 16:24:42 DEBUG FileSystem: looking for configuration option fs.jar.impl | |
22/08/12 16:24:42 DEBUG FileSystem: Looking in service filesystems for implementation class | |
22/08/12 16:24:42 DEBUG FsUrlStreamHandlerFactory: Unknown protocol jar, delegating to default implementation | |
22/08/12 16:24:43 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$pythonToJava$1 | |
22/08/12 16:24:43 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$pythonToJava$1) is now cleaned +++ | |
22/08/12 16:24:43 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$toJavaArray$1 | |
22/08/12 16:24:43 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$toJavaArray$1) is now cleaned +++ | |
22/08/12 16:24:43 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$applySchemaToPythonRDD$1 | |
22/08/12 16:24:43 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$applySchemaToPythonRDD$1) is now cleaned +++ | |
22/08/12 16:24:43 DEBUG CatalystSqlParser: Parsing command: spark_grouping_id | |
22/08/12 16:24:45 DEBUG WholeStageCodegenExec: | |
/* 001 */ public Object generate(Object[] references) { | |
/* 002 */ return new GeneratedIteratorForCodegenStage1(references); | |
/* 003 */ } | |
/* 004 */ | |
/* 005 */ // codegenStageId=1 | |
/* 006 */ final class GeneratedIteratorForCodegenStage1 extends org.apache.spark.sql.execution.BufferedRowIterator { | |
/* 007 */ private Object[] references; | |
/* 008 */ private scala.collection.Iterator[] inputs; | |
/* 009 */ private scala.collection.Iterator rdd_input_0; | |
/* 010 */ private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[] rdd_mutableStateArray_0 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[2]; | |
/* 011 */ | |
/* 012 */ public GeneratedIteratorForCodegenStage1(Object[] references) { | |
/* 013 */ this.references = references; | |
/* 014 */ } | |
/* 015 */ | |
/* 016 */ public void init(int index, scala.collection.Iterator[] inputs) { | |
/* 017 */ partitionIndex = index; | |
/* 018 */ this.inputs = inputs; | |
/* 019 */ rdd_input_0 = inputs[0]; | |
/* 020 */ rdd_mutableStateArray_0[0] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(2, 0); | |
/* 021 */ rdd_mutableStateArray_0[1] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(2, 64); | |
/* 022 */ | |
/* 023 */ } | |
/* 024 */ | |
/* 025 */ protected void processNext() throws java.io.IOException { | |
/* 026 */ while ( rdd_input_0.hasNext()) { | |
/* 027 */ InternalRow rdd_row_0 = (InternalRow) rdd_input_0.next(); | |
/* 028 */ ((org.apache.spark.sql.execution.metric.SQLMetric) references[0] /* numOutputRows */).add(1); | |
/* 029 */ // common sub-expressions | |
/* 030 */ | |
/* 031 */ boolean rdd_isNull_0 = rdd_row_0.isNullAt(0); | |
/* 032 */ long rdd_value_0 = rdd_isNull_0 ? | |
/* 033 */ -1L : (rdd_row_0.getLong(0)); | |
/* 034 */ boolean project_isNull_0 = rdd_isNull_0; | |
/* 035 */ UTF8String project_value_0 = null; | |
/* 036 */ if (!rdd_isNull_0) { | |
/* 037 */ project_value_0 = UTF8String.fromString(String.valueOf(rdd_value_0)); | |
/* 038 */ } | |
/* 039 */ boolean rdd_isNull_1 = rdd_row_0.isNullAt(1); | |
/* 040 */ double rdd_value_1 = rdd_isNull_1 ? | |
/* 041 */ -1.0 : (rdd_row_0.getDouble(1)); | |
/* 042 */ boolean project_isNull_2 = rdd_isNull_1; | |
/* 043 */ UTF8String project_value_2 = null; | |
/* 044 */ if (!rdd_isNull_1) { | |
/* 045 */ project_value_2 = UTF8String.fromString(String.valueOf(rdd_value_1)); | |
/* 046 */ } | |
/* 047 */ rdd_mutableStateArray_0[1].reset(); | |
/* 048 */ | |
/* 049 */ rdd_mutableStateArray_0[1].zeroOutNullBytes(); | |
/* 050 */ | |
/* 051 */ if (project_isNull_0) { | |
/* 052 */ rdd_mutableStateArray_0[1].setNullAt(0); | |
/* 053 */ } else { | |
/* 054 */ rdd_mutableStateArray_0[1].write(0, project_value_0); | |
/* 055 */ } | |
/* 056 */ | |
/* 057 */ if (project_isNull_2) { | |
/* 058 */ rdd_mutableStateArray_0[1].setNullAt(1); | |
/* 059 */ } else { | |
/* 060 */ rdd_mutableStateArray_0[1].write(1, project_value_2); | |
/* 061 */ } | |
/* 062 */ append((rdd_mutableStateArray_0[1].getRow())); | |
/* 063 */ if (shouldStop()) return; | |
/* 064 */ } | |
/* 065 */ } | |
/* 066 */ | |
/* 067 */ } | |
22/08/12 16:24:45 DEBUG CodeGenerator: | |
/* 001 */ public Object generate(Object[] references) { | |
/* 002 */ return new GeneratedIteratorForCodegenStage1(references); | |
/* 003 */ } | |
/* 004 */ | |
/* 005 */ // codegenStageId=1 | |
/* 006 */ final class GeneratedIteratorForCodegenStage1 extends org.apache.spark.sql.execution.BufferedRowIterator { | |
/* 007 */ private Object[] references; | |
/* 008 */ private scala.collection.Iterator[] inputs; | |
/* 009 */ private scala.collection.Iterator rdd_input_0; | |
/* 010 */ private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[] rdd_mutableStateArray_0 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[2]; | |
/* 011 */ | |
/* 012 */ public GeneratedIteratorForCodegenStage1(Object[] references) { | |
/* 013 */ this.references = references; | |
/* 014 */ } | |
/* 015 */ | |
/* 016 */ public void init(int index, scala.collection.Iterator[] inputs) { | |
/* 017 */ partitionIndex = index; | |
/* 018 */ this.inputs = inputs; | |
/* 019 */ rdd_input_0 = inputs[0]; | |
/* 020 */ rdd_mutableStateArray_0[0] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(2, 0); | |
/* 021 */ rdd_mutableStateArray_0[1] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(2, 64); | |
/* 022 */ | |
/* 023 */ } | |
/* 024 */ | |
/* 025 */ protected void processNext() throws java.io.IOException { | |
/* 026 */ while ( rdd_input_0.hasNext()) { | |
/* 027 */ InternalRow rdd_row_0 = (InternalRow) rdd_input_0.next(); | |
/* 028 */ ((org.apache.spark.sql.execution.metric.SQLMetric) references[0] /* numOutputRows */).add(1); | |
/* 029 */ // common sub-expressions | |
/* 030 */ | |
/* 031 */ boolean rdd_isNull_0 = rdd_row_0.isNullAt(0); | |
/* 032 */ long rdd_value_0 = rdd_isNull_0 ? | |
/* 033 */ -1L : (rdd_row_0.getLong(0)); | |
/* 034 */ boolean project_isNull_0 = rdd_isNull_0; | |
/* 035 */ UTF8String project_value_0 = null; | |
/* 036 */ if (!rdd_isNull_0) { | |
/* 037 */ project_value_0 = UTF8String.fromString(String.valueOf(rdd_value_0)); | |
/* 038 */ } | |
/* 039 */ boolean rdd_isNull_1 = rdd_row_0.isNullAt(1); | |
/* 040 */ double rdd_value_1 = rdd_isNull_1 ? | |
/* 041 */ -1.0 : (rdd_row_0.getDouble(1)); | |
/* 042 */ boolean project_isNull_2 = rdd_isNull_1; | |
/* 043 */ UTF8String project_value_2 = null; | |
/* 044 */ if (!rdd_isNull_1) { | |
/* 045 */ project_value_2 = UTF8String.fromString(String.valueOf(rdd_value_1)); | |
/* 046 */ } | |
/* 047 */ rdd_mutableStateArray_0[1].reset(); | |
/* 048 */ | |
/* 049 */ rdd_mutableStateArray_0[1].zeroOutNullBytes(); | |
/* 050 */ | |
/* 051 */ if (project_isNull_0) { | |
/* 052 */ rdd_mutableStateArray_0[1].setNullAt(0); | |
/* 053 */ } else { | |
/* 054 */ rdd_mutableStateArray_0[1].write(0, project_value_0); | |
/* 055 */ } | |
/* 056 */ | |
/* 057 */ if (project_isNull_2) { | |
/* 058 */ rdd_mutableStateArray_0[1].setNullAt(1); | |
/* 059 */ } else { | |
/* 060 */ rdd_mutableStateArray_0[1].write(1, project_value_2); | |
/* 061 */ } | |
/* 062 */ append((rdd_mutableStateArray_0[1].getRow())); | |
/* 063 */ if (shouldStop()) return; | |
/* 064 */ } | |
/* 065 */ } | |
/* 066 */ | |
/* 067 */ } | |
22/08/12 16:24:45 INFO CodeGenerator: Code generated in 173.205468 ms | |
22/08/12 16:24:45 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$doExecute$4$adapted | |
22/08/12 16:24:45 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$doExecute$4$adapted) is now cleaned +++ | |
22/08/12 16:24:45 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$executeTake$2 | |
22/08/12 16:24:45 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$executeTake$2) is now cleaned +++ | |
22/08/12 16:24:45 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$runJob$5 | |
22/08/12 16:24:45 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$runJob$5) is now cleaned +++ | |
22/08/12 16:24:45 INFO SparkContext: Starting job: showString at NativeMethodAccessorImpl.java:0 | |
22/08/12 16:24:45 DEBUG DAGScheduler: eagerlyComputePartitionsForRddAndAncestors for RDD 6 took 0.000872 seconds | |
22/08/12 16:24:45 DEBUG DAGScheduler: Merging stage rdd profiles: Set() | |
22/08/12 16:24:45 INFO DAGScheduler: Got job 0 (showString at NativeMethodAccessorImpl.java:0) with 1 output partitions | |
22/08/12 16:24:45 INFO DAGScheduler: Final stage: ResultStage 0 (showString at NativeMethodAccessorImpl.java:0) | |
22/08/12 16:24:45 INFO DAGScheduler: Parents of final stage: List() | |
22/08/12 16:24:45 INFO DAGScheduler: Missing parents: List() | |
22/08/12 16:24:45 DEBUG DAGScheduler: submitStage(ResultStage 0 (name=showString at NativeMethodAccessorImpl.java:0;jobs=0)) | |
22/08/12 16:24:45 DEBUG DAGScheduler: missing: List() | |
22/08/12 16:24:45 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[6] at showString at NativeMethodAccessorImpl.java:0), which has no missing parents | |
22/08/12 16:24:45 DEBUG DAGScheduler: submitMissingTasks(ResultStage 0) | |
22/08/12 16:24:45 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 12.4 KiB, free 434.4 MiB) | |
22/08/12 16:24:45 DEBUG BlockManager: Put block broadcast_0 locally took 32 ms | |
22/08/12 16:24:45 DEBUG BlockManager: Putting block broadcast_0 without replication took 33 ms | |
22/08/12 16:24:45 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 6.5 KiB, free 434.4 MiB) | |
22/08/12 16:24:45 DEBUG BlockManagerMasterEndpoint: Updating block info on master broadcast_0_piece0 for BlockManagerId(driver, 10.1.255.235, 38343, None) | |
22/08/12 16:24:45 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.1.255.235:38343 (size: 6.5 KiB, free: 434.4 MiB) | |
22/08/12 16:24:45 DEBUG BlockManagerMaster: Updated info of block broadcast_0_piece0 | |
22/08/12 16:24:45 DEBUG BlockManager: Told master about block broadcast_0_piece0 | |
22/08/12 16:24:45 DEBUG BlockManager: Put block broadcast_0_piece0 locally took 9 ms | |
22/08/12 16:24:45 DEBUG BlockManager: Putting block broadcast_0_piece0 without replication took 9 ms | |
22/08/12 16:24:45 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1513 | |
22/08/12 16:24:45 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[6] at showString at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0)) | |
22/08/12 16:24:45 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks resource profile 0 | |
22/08/12 16:24:45 DEBUG TaskSetManager: Epoch for TaskSet 0.0: 0 | |
22/08/12 16:24:45 DEBUG TaskSetManager: Adding pending tasks took 2 ms | |
22/08/12 16:24:45 DEBUG TaskSetManager: Valid locality levels for TaskSet 0.0: NO_PREF, ANY | |
22/08/12 16:24:45 DEBUG TaskSchedulerImpl: parentName: , name: TaskSet_0.0, runningTasks: 0 | |
22/08/12 16:24:45 DEBUG TaskSetManager: Valid locality levels for TaskSet 0.0: NO_PREF, ANY | |
22/08/12 16:24:45 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (10.1.255.235, executor driver, partition 0, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map() | |
22/08/12 16:24:45 DEBUG TaskSetManager: No tasks for locality level NO_PREF, so moving to locality level ANY | |
22/08/12 16:24:45 INFO Executor: Running task 0.0 in stage 0.0 (TID 0) | |
22/08/12 16:24:45 DEBUG ExecutorMetricsPoller: stageTCMP: (0, 0) -> 1 | |
22/08/12 16:24:45 DEBUG BlockManager: Getting local block broadcast_0 | |
22/08/12 16:24:45 DEBUG BlockManager: Level for block broadcast_0 is StorageLevel(disk, memory, deserialized, 1 replicas) | |
22/08/12 16:24:46 INFO PythonRunner: Times: total = 427, boot = 367, init = 60, finish = 0 | |
22/08/12 16:24:46 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1789 bytes result sent to driver | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (0, 0) -> 0 | |
22/08/12 16:24:46 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 571 ms on 10.1.255.235 (executor driver) (1/1) | |
22/08/12 16:24:46 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool | |
22/08/12 16:24:46 INFO PythonAccumulatorV2: Connected to AccumulatorServer at host: 127.0.0.1 port: 49069 | |
22/08/12 16:24:46 INFO DAGScheduler: ResultStage 0 (showString at NativeMethodAccessorImpl.java:0) finished in 0.785 s | |
22/08/12 16:24:46 DEBUG DAGScheduler: After removal of stage 0, remaining stages = 0 | |
22/08/12 16:24:46 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job | |
22/08/12 16:24:46 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished | |
22/08/12 16:24:46 INFO DAGScheduler: Job 0 finished: showString at NativeMethodAccessorImpl.java:0, took 0.837916 s | |
22/08/12 16:24:46 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$executeTake$2 | |
22/08/12 16:24:46 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$executeTake$2) is now cleaned +++ | |
22/08/12 16:24:46 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$runJob$5 | |
22/08/12 16:24:46 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$runJob$5) is now cleaned +++ | |
22/08/12 16:24:46 INFO SparkContext: Starting job: showString at NativeMethodAccessorImpl.java:0 | |
22/08/12 16:24:46 DEBUG DAGScheduler: eagerlyComputePartitionsForRddAndAncestors for RDD 6 took 0.000216 seconds | |
22/08/12 16:24:46 DEBUG DAGScheduler: Merging stage rdd profiles: Set() | |
22/08/12 16:24:46 INFO DAGScheduler: Got job 1 (showString at NativeMethodAccessorImpl.java:0) with 4 output partitions | |
22/08/12 16:24:46 INFO DAGScheduler: Final stage: ResultStage 1 (showString at NativeMethodAccessorImpl.java:0) | |
22/08/12 16:24:46 INFO DAGScheduler: Parents of final stage: List() | |
22/08/12 16:24:46 INFO DAGScheduler: Missing parents: List() | |
22/08/12 16:24:46 DEBUG DAGScheduler: submitStage(ResultStage 1 (name=showString at NativeMethodAccessorImpl.java:0;jobs=1)) | |
22/08/12 16:24:46 DEBUG DAGScheduler: missing: List() | |
22/08/12 16:24:46 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[6] at showString at NativeMethodAccessorImpl.java:0), which has no missing parents | |
22/08/12 16:24:46 DEBUG DAGScheduler: submitMissingTasks(ResultStage 1) | |
22/08/12 16:24:46 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 12.4 KiB, free 434.4 MiB) | |
22/08/12 16:24:46 DEBUG BlockManager: Put block broadcast_1 locally took 1 ms | |
22/08/12 16:24:46 DEBUG BlockManager: Putting block broadcast_1 without replication took 2 ms | |
22/08/12 16:24:46 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 6.6 KiB, free 434.4 MiB) | |
22/08/12 16:24:46 DEBUG BlockManagerMasterEndpoint: Updating block info on master broadcast_1_piece0 for BlockManagerId(driver, 10.1.255.235, 38343, None) | |
22/08/12 16:24:46 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.1.255.235:38343 (size: 6.6 KiB, free: 434.4 MiB) | |
22/08/12 16:24:46 DEBUG BlockManagerMaster: Updated info of block broadcast_1_piece0 | |
22/08/12 16:24:46 DEBUG BlockManager: Told master about block broadcast_1_piece0 | |
22/08/12 16:24:46 DEBUG BlockManager: Put block broadcast_1_piece0 locally took 2 ms | |
22/08/12 16:24:46 DEBUG BlockManager: Putting block broadcast_1_piece0 without replication took 3 ms | |
22/08/12 16:24:46 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1513 | |
22/08/12 16:24:46 INFO DAGScheduler: Submitting 4 missing tasks from ResultStage 1 (MapPartitionsRDD[6] at showString at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(1, 2, 3, 4)) | |
22/08/12 16:24:46 INFO TaskSchedulerImpl: Adding task set 1.0 with 4 tasks resource profile 0 | |
22/08/12 16:24:46 DEBUG TaskSetManager: Epoch for TaskSet 1.0: 0 | |
22/08/12 16:24:46 DEBUG TaskSetManager: Adding pending tasks took 0 ms | |
22/08/12 16:24:46 DEBUG TaskSetManager: Valid locality levels for TaskSet 1.0: NO_PREF, ANY | |
22/08/12 16:24:46 DEBUG TaskSchedulerImpl: parentName: , name: TaskSet_1.0, runningTasks: 0 | |
22/08/12 16:24:46 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1) (10.1.255.235, executor driver, partition 1, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map() | |
22/08/12 16:24:46 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 2) (10.1.255.235, executor driver, partition 2, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map() | |
22/08/12 16:24:46 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 3) (10.1.255.235, executor driver, partition 3, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map() | |
22/08/12 16:24:46 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 4) (10.1.255.235, executor driver, partition 4, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map() | |
22/08/12 16:24:46 DEBUG TaskSetManager: No tasks for locality level NO_PREF, so moving to locality level ANY | |
22/08/12 16:24:46 INFO Executor: Running task 0.0 in stage 1.0 (TID 1) | |
22/08/12 16:24:46 INFO Executor: Running task 1.0 in stage 1.0 (TID 2) | |
22/08/12 16:24:46 INFO Executor: Running task 2.0 in stage 1.0 (TID 3) | |
22/08/12 16:24:46 INFO Executor: Running task 3.0 in stage 1.0 (TID 4) | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 1 | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 2 | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 3 | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 4 | |
22/08/12 16:24:46 DEBUG BlockManager: Getting local block broadcast_1 | |
22/08/12 16:24:46 DEBUG BlockManager: Level for block broadcast_1 is StorageLevel(disk, memory, deserialized, 1 replicas) | |
22/08/12 16:24:46 INFO PythonRunner: Times: total = 62, boot = 6, init = 56, finish = 0 | |
22/08/12 16:24:46 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1789 bytes result sent to driver | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 3 | |
22/08/12 16:24:46 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 88 ms on 10.1.255.235 (executor driver) (1/4) | |
22/08/12 16:24:46 INFO PythonRunner: Times: total = 67, boot = 9, init = 58, finish = 0 | |
22/08/12 16:24:46 INFO Executor: Finished task 3.0 in stage 1.0 (TID 4). 1789 bytes result sent to driver | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 2 | |
22/08/12 16:24:46 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 4) in 91 ms on 10.1.255.235 (executor driver) (2/4) | |
22/08/12 16:24:46 INFO PythonRunner: Times: total = 74, boot = -98, init = 172, finish = 0 | |
22/08/12 16:24:46 INFO Executor: Finished task 1.0 in stage 1.0 (TID 2). 1789 bytes result sent to driver | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 1 | |
22/08/12 16:24:46 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 2) in 98 ms on 10.1.255.235 (executor driver) (3/4) | |
22/08/12 16:24:46 INFO PythonRunner: Times: total = 84, boot = 3, init = 80, finish = 1 | |
22/08/12 16:24:46 INFO Executor: Finished task 2.0 in stage 1.0 (TID 3). 1789 bytes result sent to driver | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 0 | |
22/08/12 16:24:46 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 3) in 106 ms on 10.1.255.235 (executor driver) (4/4) | |
22/08/12 16:24:46 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool | |
22/08/12 16:24:46 INFO DAGScheduler: ResultStage 1 (showString at NativeMethodAccessorImpl.java:0) finished in 0.125 s | |
22/08/12 16:24:46 DEBUG DAGScheduler: After removal of stage 1, remaining stages = 0 | |
22/08/12 16:24:46 INFO DAGScheduler: Job 1 is finished. Cancelling potential speculative or zombie tasks for this job | |
22/08/12 16:24:46 INFO TaskSchedulerImpl: Killing all running tasks in stage 1: Stage finished | |
22/08/12 16:24:46 INFO DAGScheduler: Job 1 finished: showString at NativeMethodAccessorImpl.java:0, took 0.131962 s | |
22/08/12 16:24:46 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$executeTake$2 | |
22/08/12 16:24:46 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$executeTake$2) is now cleaned +++ | |
22/08/12 16:24:46 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$runJob$5 | |
22/08/12 16:24:46 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$runJob$5) is now cleaned +++ | |
22/08/12 16:24:46 INFO SparkContext: Starting job: showString at NativeMethodAccessorImpl.java:0 | |
22/08/12 16:24:46 DEBUG DAGScheduler: eagerlyComputePartitionsForRddAndAncestors for RDD 6 took 0.000154 seconds | |
22/08/12 16:24:46 DEBUG DAGScheduler: Merging stage rdd profiles: Set() | |
22/08/12 16:24:46 INFO DAGScheduler: Got job 2 (showString at NativeMethodAccessorImpl.java:0) with 11 output partitions | |
22/08/12 16:24:46 INFO DAGScheduler: Final stage: ResultStage 2 (showString at NativeMethodAccessorImpl.java:0) | |
22/08/12 16:24:46 INFO DAGScheduler: Parents of final stage: List() | |
22/08/12 16:24:46 INFO DAGScheduler: Missing parents: List() | |
22/08/12 16:24:46 DEBUG DAGScheduler: submitStage(ResultStage 2 (name=showString at NativeMethodAccessorImpl.java:0;jobs=2)) | |
22/08/12 16:24:46 DEBUG DAGScheduler: missing: List() | |
22/08/12 16:24:46 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[6] at showString at NativeMethodAccessorImpl.java:0), which has no missing parents | |
22/08/12 16:24:46 DEBUG DAGScheduler: submitMissingTasks(ResultStage 2) | |
22/08/12 16:24:46 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 12.4 KiB, free 434.4 MiB) | |
22/08/12 16:24:46 DEBUG BlockManager: Put block broadcast_2 locally took 1 ms | |
22/08/12 16:24:46 DEBUG BlockManager: Putting block broadcast_2 without replication took 1 ms | |
22/08/12 16:24:46 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 6.6 KiB, free 434.3 MiB) | |
22/08/12 16:24:46 DEBUG BlockManagerMasterEndpoint: Updating block info on master broadcast_2_piece0 for BlockManagerId(driver, 10.1.255.235, 38343, None) | |
22/08/12 16:24:46 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.1.255.235:38343 (size: 6.6 KiB, free: 434.4 MiB) | |
22/08/12 16:24:46 DEBUG BlockManagerMaster: Updated info of block broadcast_2_piece0 | |
22/08/12 16:24:46 DEBUG BlockManager: Told master about block broadcast_2_piece0 | |
22/08/12 16:24:46 DEBUG BlockManager: Put block broadcast_2_piece0 locally took 2 ms | |
22/08/12 16:24:46 DEBUG BlockManager: Putting block broadcast_2_piece0 without replication took 2 ms | |
22/08/12 16:24:46 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1513 | |
22/08/12 16:24:46 INFO DAGScheduler: Submitting 11 missing tasks from ResultStage 2 (MapPartitionsRDD[6] at showString at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15)) | |
22/08/12 16:24:46 INFO TaskSchedulerImpl: Adding task set 2.0 with 11 tasks resource profile 0 | |
22/08/12 16:24:46 DEBUG TaskSetManager: Epoch for TaskSet 2.0: 0 | |
22/08/12 16:24:46 DEBUG TaskSetManager: Adding pending tasks took 0 ms | |
22/08/12 16:24:46 DEBUG TaskSetManager: Valid locality levels for TaskSet 2.0: NO_PREF, ANY | |
22/08/12 16:24:46 DEBUG TaskSchedulerImpl: parentName: , name: TaskSet_2.0, runningTasks: 0 | |
22/08/12 16:24:46 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 5) (10.1.255.235, executor driver, partition 5, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map() | |
22/08/12 16:24:46 INFO TaskSetManager: Starting task 1.0 in stage 2.0 (TID 6) (10.1.255.235, executor driver, partition 6, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map() | |
22/08/12 16:24:46 INFO TaskSetManager: Starting task 2.0 in stage 2.0 (TID 7) (10.1.255.235, executor driver, partition 7, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map() | |
22/08/12 16:24:46 INFO TaskSetManager: Starting task 3.0 in stage 2.0 (TID 8) (10.1.255.235, executor driver, partition 8, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map() | |
22/08/12 16:24:46 INFO TaskSetManager: Starting task 4.0 in stage 2.0 (TID 9) (10.1.255.235, executor driver, partition 9, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map() | |
22/08/12 16:24:46 INFO TaskSetManager: Starting task 5.0 in stage 2.0 (TID 10) (10.1.255.235, executor driver, partition 10, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map() | |
22/08/12 16:24:46 INFO TaskSetManager: Starting task 6.0 in stage 2.0 (TID 11) (10.1.255.235, executor driver, partition 11, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map() | |
22/08/12 16:24:46 INFO TaskSetManager: Starting task 7.0 in stage 2.0 (TID 12) (10.1.255.235, executor driver, partition 12, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map() | |
22/08/12 16:24:46 INFO TaskSetManager: Starting task 8.0 in stage 2.0 (TID 13) (10.1.255.235, executor driver, partition 13, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map() | |
22/08/12 16:24:46 INFO TaskSetManager: Starting task 9.0 in stage 2.0 (TID 14) (10.1.255.235, executor driver, partition 14, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map() | |
22/08/12 16:24:46 INFO TaskSetManager: Starting task 10.0 in stage 2.0 (TID 15) (10.1.255.235, executor driver, partition 15, PROCESS_LOCAL, 4471 bytes) taskResourceAssignments Map() | |
22/08/12 16:24:46 DEBUG TaskSetManager: No tasks for locality level NO_PREF, so moving to locality level ANY | |
22/08/12 16:24:46 INFO Executor: Running task 1.0 in stage 2.0 (TID 6) | |
22/08/12 16:24:46 INFO Executor: Running task 0.0 in stage 2.0 (TID 5) | |
22/08/12 16:24:46 INFO Executor: Running task 2.0 in stage 2.0 (TID 7) | |
22/08/12 16:24:46 INFO Executor: Running task 3.0 in stage 2.0 (TID 8) | |
22/08/12 16:24:46 INFO Executor: Running task 5.0 in stage 2.0 (TID 10) | |
22/08/12 16:24:46 INFO Executor: Running task 4.0 in stage 2.0 (TID 9) | |
22/08/12 16:24:46 INFO Executor: Running task 6.0 in stage 2.0 (TID 11) | |
22/08/12 16:24:46 INFO Executor: Running task 7.0 in stage 2.0 (TID 12) | |
22/08/12 16:24:46 INFO Executor: Running task 8.0 in stage 2.0 (TID 13) | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 3 | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 2 | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 4 | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 1 | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 5 | |
22/08/12 16:24:46 INFO Executor: Running task 9.0 in stage 2.0 (TID 14) | |
22/08/12 16:24:46 INFO Executor: Running task 10.0 in stage 2.0 (TID 15) | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 6 | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 8 | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 7 | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 9 | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 10 | |
22/08/12 16:24:46 DEBUG BlockManager: Getting local block broadcast_2 | |
22/08/12 16:24:46 DEBUG BlockManager: Level for block broadcast_2 is StorageLevel(disk, memory, deserialized, 1 replicas) | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 11 | |
22/08/12 16:24:46 INFO PythonRunner: Times: total = 66, boot = -51, init = 117, finish = 0 | |
22/08/12 16:24:46 INFO Executor: Finished task 2.0 in stage 2.0 (TID 7). 1789 bytes result sent to driver | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 10 | |
22/08/12 16:24:46 INFO TaskSetManager: Finished task 2.0 in stage 2.0 (TID 7) in 90 ms on 10.1.255.235 (executor driver) (1/11) | |
22/08/12 16:24:46 INFO PythonRunner: Times: total = 75, boot = -54, init = 129, finish = 0 | |
22/08/12 16:24:46 INFO PythonRunner: Times: total = 76, boot = 11, init = 65, finish = 0 | |
22/08/12 16:24:46 INFO PythonRunner: Times: total = 82, boot = -59, init = 141, finish = 0 | |
22/08/12 16:24:46 INFO Executor: Finished task 6.0 in stage 2.0 (TID 11). 1789 bytes result sent to driver | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 9 | |
22/08/12 16:24:46 INFO TaskSetManager: Finished task 6.0 in stage 2.0 (TID 11) in 99 ms on 10.1.255.235 (executor driver) (2/11) | |
22/08/12 16:24:46 INFO Executor: Finished task 7.0 in stage 2.0 (TID 12). 1789 bytes result sent to driver | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 8 | |
22/08/12 16:24:46 INFO TaskSetManager: Finished task 7.0 in stage 2.0 (TID 12) in 102 ms on 10.1.255.235 (executor driver) (3/11) | |
22/08/12 16:24:46 INFO Executor: Finished task 3.0 in stage 2.0 (TID 8). 1789 bytes result sent to driver | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 7 | |
22/08/12 16:24:46 INFO TaskSetManager: Finished task 3.0 in stage 2.0 (TID 8) in 110 ms on 10.1.255.235 (executor driver) (4/11) | |
22/08/12 16:24:46 INFO PythonRunner: Times: total = 96, boot = 8, init = 88, finish = 0 | |
22/08/12 16:24:46 INFO PythonRunner: Times: total = 97, boot = 3, init = 93, finish = 1 | |
22/08/12 16:24:46 INFO Executor: Finished task 9.0 in stage 2.0 (TID 14). 1789 bytes result sent to driver | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 6 | |
22/08/12 16:24:46 INFO Executor: Finished task 5.0 in stage 2.0 (TID 10). 1789 bytes result sent to driver | |
22/08/12 16:24:46 INFO TaskSetManager: Finished task 9.0 in stage 2.0 (TID 14) in 116 ms on 10.1.255.235 (executor driver) (5/11) | |
22/08/12 16:24:46 INFO PythonRunner: Times: total = 98, boot = -44, init = 142, finish = 0 | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 5 | |
22/08/12 16:24:46 INFO PythonRunner: Times: total = 100, boot = 12, init = 87, finish = 1 | |
22/08/12 16:24:46 INFO TaskSetManager: Finished task 5.0 in stage 2.0 (TID 10) in 120 ms on 10.1.255.235 (executor driver) (6/11) | |
22/08/12 16:24:46 INFO Executor: Finished task 0.0 in stage 2.0 (TID 5). 1789 bytes result sent to driver | |
22/08/12 16:24:46 INFO Executor: Finished task 1.0 in stage 2.0 (TID 6). 1789 bytes result sent to driver | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 4 | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 3 | |
22/08/12 16:24:46 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 5) in 128 ms on 10.1.255.235 (executor driver) (7/11) | |
22/08/12 16:24:46 INFO TaskSetManager: Finished task 1.0 in stage 2.0 (TID 6) in 128 ms on 10.1.255.235 (executor driver) (8/11) | |
22/08/12 16:24:46 INFO PythonRunner: Times: total = 112, boot = 24, init = 88, finish = 0 | |
22/08/12 16:24:46 INFO PythonRunner: Times: total = 117, boot = 19, init = 98, finish = 0 | |
22/08/12 16:24:46 INFO PythonRunner: Times: total = 106, boot = 16, init = 90, finish = 0 | |
22/08/12 16:24:46 INFO Executor: Finished task 8.0 in stage 2.0 (TID 13). 1789 bytes result sent to driver | |
22/08/12 16:24:46 INFO Executor: Finished task 4.0 in stage 2.0 (TID 9). 1789 bytes result sent to driver | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 2 | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 1 | |
22/08/12 16:24:46 INFO Executor: Finished task 10.0 in stage 2.0 (TID 15). 1830 bytes result sent to driver | |
22/08/12 16:24:46 INFO TaskSetManager: Finished task 4.0 in stage 2.0 (TID 9) in 145 ms on 10.1.255.235 (executor driver) (9/11) | |
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 0 | |
22/08/12 16:24:46 INFO TaskSetManager: Finished task 8.0 in stage 2.0 (TID 13) in 144 ms on 10.1.255.235 (executor driver) (10/11) | |
22/08/12 16:24:46 INFO TaskSetManager: Finished task 10.0 in stage 2.0 (TID 15) in 144 ms on 10.1.255.235 (executor driver) (11/11) | |
22/08/12 16:24:46 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool | |
22/08/12 16:24:46 INFO DAGScheduler: ResultStage 2 (showString at NativeMethodAccessorImpl.java:0) finished in 0.162 s | |
22/08/12 16:24:46 DEBUG DAGScheduler: After removal of stage 2, remaining stages = 0 | |
22/08/12 16:24:46 INFO DAGScheduler: Job 2 is finished. Cancelling potential speculative or zombie tasks for this job | |
22/08/12 16:24:46 INFO TaskSchedulerImpl: Killing all running tasks in stage 2: Stage finished | |
22/08/12 16:24:46 INFO DAGScheduler: Job 2 finished: showString at NativeMethodAccessorImpl.java:0, took 0.168241 s | |
22/08/12 16:24:46 DEBUG GenerateSafeProjection: code for createexternalrow(input[0, string, true].toString, input[1, string, true].toString, StructField(a,StringType,true), StructField(b,StringType,true)): | |
/* 001 */ public java.lang.Object generate(Object[] references) { | |
/* 002 */ return new SpecificSafeProjection(references); | |
/* 003 */ } | |
/* 004 */ | |
/* 005 */ class SpecificSafeProjection extends org.apache.spark.sql.catalyst.expressions.codegen.BaseProjection { | |
/* 006 */ | |
/* 007 */ private Object[] references; | |
/* 008 */ private InternalRow mutableRow; | |
/* 009 */ | |
/* 010 */ | |
/* 011 */ public SpecificSafeProjection(Object[] references) { | |
/* 012 */ this.references = references; | |
/* 013 */ mutableRow = (InternalRow) references[references.length - 1]; | |
/* 014 */ | |
/* 015 */ } | |
/* 016 */ | |
/* 017 */ public void initialize(int partitionIndex) { | |
/* 018 */ | |
/* 019 */ } | |
/* 020 */ | |
/* 021 */ public java.lang.Object apply(java.lang.Object _i) { | |
/* 022 */ InternalRow i = (InternalRow) _i; | |
/* 023 */ org.apache.spark.sql.Row value_5 = CreateExternalRow_0(i); | |
/* 024 */ if (false) { | |
/* 025 */ mutableRow.setNullAt(0); | |
/* 026 */ } else { | |
/* 027 */ | |
/* 028 */ mutableRow.update(0, value_5); | |
/* 029 */ } | |
/* 030 */ | |
/* 031 */ return mutableRow; | |
/* 032 */ } | |
/* 033 */ | |
/* 034 */ | |
/* 035 */ private org.apache.spark.sql.Row CreateExternalRow_0(InternalRow i) { | |
/* 036 */ Object[] values_0 = new Object[2]; | |
/* 037 */ | |
/* 038 */ boolean isNull_2 = i.isNullAt(0); | |
/* 039 */ UTF8String value_2 = isNull_2 ? | |
/* 040 */ null : (i.getUTF8String(0)); | |
/* 041 */ boolean isNull_1 = true; | |
/* 042 */ java.lang.String value_1 = null; | |
/* 043 */ if (!isNull_2) { | |
/* 044 */ isNull_1 = false; | |
/* 045 */ if (!isNull_1) { | |
/* 046 */ | |
/* 047 */ Object funcResult_0 = null; | |
/* 048 */ funcResult_0 = value_2.toString(); | |
/* 049 */ value_1 = (java.lang.String) funcResult_0; | |
/* 050 */ | |
/* 051 */ } | |
/* 052 */ } | |
/* 053 */ if (isNull_1) { | |
/* 054 */ values_0[0] = null; | |
/* 055 */ } else { | |
/* 056 */ values_0[0] = value_1; | |
/* 057 */ } | |
/* 058 */ | |
/* 059 */ boolean isNull_4 = i.isNullAt(1); | |
/* 060 */ UTF8String value_4 = isNull_4 ? | |
/* 061 */ null : (i.getUTF8String(1)); | |
/* 062 */ boolean isNull_3 = true; | |
/* 063 */ java.lang.String value_3 = null; | |
/* 064 */ if (!isNull_4) { | |
/* 065 */ isNull_3 = false; | |
/* 066 */ if (!isNull_3) { | |
/* 067 */ | |
/* 068 */ Object funcResult_1 = null; | |
/* 069 */ funcResult_1 = value_4.toString(); | |
/* 070 */ value_3 = (java.lang.String) funcResult_1; | |
/* 071 */ | |
/* 072 */ } | |
/* 073 */ } | |
/* 074 */ if (isNull_3) { | |
/* 075 */ values_0[1] = null; | |
/* 076 */ } else { | |
/* 077 */ values_0[1] = value_3; | |
/* 078 */ } | |
/* 079 */ | |
/* 080 */ final org.apache.spark.sql.Row value_0 = new org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema(values_0, ((org.apache.spark.sql.types.StructType) references[0] /* schema */)); | |
/* 081 */ | |
/* 082 */ return value_0; | |
/* 083 */ } | |
/* 084 */ | |
/* 085 */ } | |
22/08/12 16:24:46 DEBUG CodeGenerator: | |
/* 001 */ public java.lang.Object generate(Object[] references) { | |
/* 002 */ return new SpecificSafeProjection(references); | |
/* 003 */ } | |
/* 004 */ | |
/* 005 */ class SpecificSafeProjection extends org.apache.spark.sql.catalyst.expressions.codegen.BaseProjection { | |
/* 006 */ | |
/* 007 */ private Object[] references; | |
/* 008 */ private InternalRow mutableRow; | |
/* 009 */ | |
/* 010 */ | |
/* 011 */ public SpecificSafeProjection(Object[] references) { | |
/* 012 */ this.references = references; | |
/* 013 */ mutableRow = (InternalRow) references[references.length - 1]; | |
/* 014 */ | |
/* 015 */ } | |
/* 016 */ | |
/* 017 */ public void initialize(int partitionIndex) { | |
/* 018 */ | |
/* 019 */ } | |
/* 020 */ | |
/* 021 */ public java.lang.Object apply(java.lang.Object _i) { | |
/* 022 */ InternalRow i = (InternalRow) _i; | |
/* 023 */ org.apache.spark.sql.Row value_5 = CreateExternalRow_0(i); | |
/* 024 */ if (false) { | |
/* 025 */ mutableRow.setNullAt(0); | |
/* 026 */ } else { | |
/* 027 */ | |
/* 028 */ mutableRow.update(0, value_5); | |
/* 029 */ } | |
/* 030 */ | |
/* 031 */ return mutableRow; | |
/* 032 */ } | |
/* 033 */ | |
/* 034 */ | |
/* 035 */ private org.apache.spark.sql.Row CreateExternalRow_0(InternalRow i) { | |
/* 036 */ Object[] values_0 = new Object[2]; | |
/* 037 */ | |
/* 038 */ boolean isNull_2 = i.isNullAt(0); | |
/* 039 */ UTF8String value_2 = isNull_2 ? | |
/* 040 */ null : (i.getUTF8String(0)); | |
/* 041 */ boolean isNull_1 = true; | |
/* 042 */ java.lang.String value_1 = null; | |
/* 043 */ if (!isNull_2) { | |
/* 044 */ isNull_1 = false; | |
/* 045 */ if (!isNull_1) { | |
/* 046 */ | |
/* 047 */ Object funcResult_0 = null; | |
/* 048 */ funcResult_0 = value_2.toString(); | |
/* 049 */ value_1 = (java.lang.String) funcResult_0; | |
/* 050 */ | |
/* 051 */ } | |
/* 052 */ } | |
/* 053 */ if (isNull_1) { | |
/* 054 */ values_0[0] = null; | |
/* 055 */ } else { | |
/* 056 */ values_0[0] = value_1; | |
/* 057 */ } | |
/* 058 */ | |
/* 059 */ boolean isNull_4 = i.isNullAt(1); | |
/* 060 */ UTF8String value_4 = isNull_4 ? | |
/* 061 */ null : (i.getUTF8String(1)); | |
/* 062 */ boolean isNull_3 = true; | |
/* 063 */ java.lang.String value_3 = null; | |
/* 064 */ if (!isNull_4) { | |
/* 065 */ isNull_3 = false; | |
/* 066 */ if (!isNull_3) { | |
/* 067 */ | |
/* 068 */ Object funcResult_1 = null; | |
/* 069 */ funcResult_1 = value_4.toString(); | |
/* 070 */ value_3 = (java.lang.String) funcResult_1; | |
/* 071 */ | |
/* 072 */ } | |
/* 073 */ } | |
/* 074 */ if (isNull_3) { | |
/* 075 */ values_0[1] = null; | |
/* 076 */ } else { | |
/* 077 */ values_0[1] = value_3; | |
/* 078 */ } | |
/* 079 */ | |
/* 080 */ final org.apache.spark.sql.Row value_0 = new org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema(values_0, ((org.apache.spark.sql.types.StructType) references[0] /* schema */)); | |
/* 081 */ | |
/* 082 */ return value_0; | |
/* 083 */ } | |
/* 084 */ | |
/* 085 */ } | |
22/08/12 16:24:46 INFO CodeGenerator: Code generated in 15.263034 ms | |
+---+---+ | |
| a| b| | |
+---+---+ | |
| 1|2.0| | |
+---+---+ | |
22/08/12 16:24:46 DEBUG FileSystem: Starting: Acquiring creator semaphore for s3a://master.rando2/nonemptyprefix8 | |
22/08/12 16:24:46 DEBUG FileSystem: Acquiring creator semaphore for s3a://master.rando2/nonemptyprefix8: duration 0:00.000s | |
22/08/12 16:24:46 DEBUG FileSystem: Starting: Creating FS s3a://master.rando2/nonemptyprefix8 | |
22/08/12 16:24:46 DEBUG FileSystem: Looking for FS supporting s3a | |
22/08/12 16:24:46 DEBUG FileSystem: looking for configuration option fs.s3a.impl | |
22/08/12 16:24:46 DEBUG Configuration: Reloading 9 existing configurations | |
22/08/12 16:24:46 DEBUG FileSystem: Filesystem s3a defined in configuration option | |
22/08/12 16:24:46 DEBUG FileSystem: FS for s3a is class org.apache.hadoop.fs.s3a.S3AFileSystem | |
22/08/12 16:24:46 DEBUG AbstractService: Service: NoopAuditor entered state INITED | |
22/08/12 16:24:46 DEBUG AbstractService: Service NoopAuditor is started | |
22/08/12 16:24:46 DEBUG S3AFileSystem: Initializing S3AFileSystem for master.rando2 | |
22/08/12 16:24:46 DEBUG S3AUtils: Propagating entries under fs.s3a.bucket.master.rando2. | |
22/08/12 16:24:46 DEBUG S3AUtils: Data is unencrypted | |
22/08/12 16:24:46 DEBUG S3ARetryPolicy: Retrying on recoverable AWS failures 7 times with an initial interval of 500ms | |
22/08/12 16:24:46 DEBUG MetricsSystemImpl: from system property: null | |
22/08/12 16:24:46 DEBUG MetricsSystemImpl: from environment variable: null | |
22/08/12 16:24:46 DEBUG MetricsConfig: Could not locate file hadoop-metrics2-s3a-file-system.properties | |
org.apache.hadoop.shaded.org.apache.commons.configuration2.ex.ConfigurationException: Could not locate: org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileLocator@1191d4eb[fileName=hadoop-metrics2-s3a-file-system.properties,basePath=<null>,sourceURL=,encoding=<null>,fileSystem=<null>,locationStrategy=<null>] | |
at org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileLocatorUtils.locateOrThrow(FileLocatorUtils.java:346) | |
at org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileHandler.load(FileHandler.java:972) | |
at org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileHandler.load(FileHandler.java:702) | |
at org.apache.hadoop.metrics2.impl.MetricsConfig.loadFirst(MetricsConfig.java:118) | |
at org.apache.hadoop.metrics2.impl.MetricsConfig.create(MetricsConfig.java:97) | |
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configure(MetricsSystemImpl.java:482) | |
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:188) | |
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:163) | |
at org.apache.hadoop.fs.s3a.S3AInstrumentation.getMetricsSystem(S3AInstrumentation.java:251) | |
at org.apache.hadoop.fs.s3a.S3AInstrumentation.registerAsMetricsSource(S3AInstrumentation.java:264) | |
at org.apache.hadoop.fs.s3a.S3AInstrumentation.<init>(S3AInstrumentation.java:235) | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:466) | |
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469) | |
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174) | |
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574) | |
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521) | |
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540) | |
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365) | |
at org.apache.spark.sql.execution.datasources.DataSource.planForWritingFileFormat(DataSource.scala:461) | |
at org.apache.spark.sql.execution.datasources.DataSource.planForWriting(DataSource.scala:558) | |
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390) | |
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:363) | |
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239) | |
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) | |
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) | |
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
at java.base/java.lang.reflect.Method.invoke(Method.java:566) | |
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) | |
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) | |
at py4j.Gateway.invoke(Gateway.java:282) | |
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) | |
at py4j.commands.CallCommand.execute(CallCommand.java:79) | |
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) | |
at py4j.ClientServerConnection.run(ClientServerConnection.java:106) | |
at java.base/java.lang.Thread.run(Thread.java:829) | |
22/08/12 16:24:46 DEBUG MetricsConfig: Could not locate file hadoop-metrics2.properties | |
org.apache.hadoop.shaded.org.apache.commons.configuration2.ex.ConfigurationException: Could not locate: org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileLocator@79eafac4[fileName=hadoop-metrics2.properties,basePath=<null>,sourceURL=,encoding=<null>,fileSystem=<null>,locationStrategy=<null>] | |
at org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileLocatorUtils.locateOrThrow(FileLocatorUtils.java:346) | |
at org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileHandler.load(FileHandler.java:972) | |
at org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileHandler.load(FileHandler.java:702) | |
at org.apache.hadoop.metrics2.impl.MetricsConfig.loadFirst(MetricsConfig.java:118) | |
at org.apache.hadoop.metrics2.impl.MetricsConfig.create(MetricsConfig.java:97) | |
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configure(MetricsSystemImpl.java:482) | |
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:188) | |
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:163) | |
at org.apache.hadoop.fs.s3a.S3AInstrumentation.getMetricsSystem(S3AInstrumentation.java:251) | |
at org.apache.hadoop.fs.s3a.S3AInstrumentation.registerAsMetricsSource(S3AInstrumentation.java:264) | |
at org.apache.hadoop.fs.s3a.S3AInstrumentation.<init>(S3AInstrumentation.java:235) | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:466) | |
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469) | |
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174) | |
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574) | |
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521) | |
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540) | |
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365) | |
at org.apache.spark.sql.execution.datasources.DataSource.planForWritingFileFormat(DataSource.scala:461) | |
at org.apache.spark.sql.execution.datasources.DataSource.planForWriting(DataSource.scala:558) | |
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390) | |
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:363) | |
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239) | |
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) | |
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) | |
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
at java.base/java.lang.reflect.Method.invoke(Method.java:566) | |
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) | |
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) | |
at py4j.Gateway.invoke(Gateway.java:282) | |
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) | |
at py4j.commands.CallCommand.execute(CallCommand.java:79) | |
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) | |
at py4j.ClientServerConnection.run(ClientServerConnection.java:106) | |
at java.base/java.lang.Thread.run(Thread.java:829) | |
22/08/12 16:24:46 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties | |
22/08/12 16:24:46 DEBUG MetricsConfig: poking parent 'PropertiesConfiguration' for key: period | |
22/08/12 16:24:46 DEBUG MetricsConfig: poking parent 'PropertiesConfiguration' for key: periodMillis | |
22/08/12 16:24:46 DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableCounterLong org.apache.hadoop.metrics2.impl.MetricsSystemImpl.droppedPubAll with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName="Ops", valueName="Time", about="", interval=10, type=DEFAULT, value={"Dropped updates by all sinks"}) | |
22/08/12 16:24:46 DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableStat org.apache.hadoop.metrics2.impl.MetricsSystemImpl.publishStat with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName="Ops", valueName="Time", about="", interval=10, type=DEFAULT, value={"Publish", "Publishing stats"}) | |
22/08/12 16:24:46 DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableStat org.apache.hadoop.metrics2.impl.MetricsSystemImpl.snapshotStat with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName="Ops", valueName="Time", about="", interval=10, type=DEFAULT, value={"Snapshot", "Snapshot stats"}) | |
22/08/12 16:24:46 DEBUG MetricsConfig: poking parent 'PropertiesConfiguration' for key: source.source.start_mbeans | |
22/08/12 16:24:46 DEBUG MetricsConfig: poking parent 'MetricsConfig' for key: source.start_mbeans | |
22/08/12 16:24:46 DEBUG MetricsConfig: poking parent 'PropertiesConfiguration' for key: *.source.start_mbeans | |
22/08/12 16:24:46 DEBUG MetricsSourceAdapter: Updating attr cache... | |
22/08/12 16:24:46 DEBUG MetricsSourceAdapter: Done. # tags & metrics=10 | |
22/08/12 16:24:46 DEBUG MetricsSourceAdapter: Updating info cache... | |
22/08/12 16:24:46 DEBUG MetricsSystemImpl: [javax.management.MBeanAttributeInfo[description=Metrics context, name=tag.Context, type=java.lang.String, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of active metrics sources, name=NumActiveSources, type=java.lang.Integer, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of all registered metrics sources, name=NumAllSources, type=java.lang.Integer, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of active metrics sinks, name=NumActiveSinks, type=java.lang.Integer, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of all registered metrics sinks, name=NumAllSinks, type=java.lang.Integer, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Dropped updates by all sinks, name=DroppedPubAll, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of ops for publishing stats, name=PublishNumOps, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Average time for publishing stats, name=PublishAvgTime, type=java.lang.Double, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of ops for snapshot stats, name=SnapshotNumOps, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Average time for snapshot stats, name=SnapshotAvgTime, type=java.lang.Double, read-only, descriptor={}]] | |
22/08/12 16:24:46 DEBUG MetricsSourceAdapter: Done | |
22/08/12 16:24:46 DEBUG MBeans: Registered Hadoop:service=s3a-file-system,name=MetricsSystem,sub=Stats | |
22/08/12 16:24:46 DEBUG MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered. | |
22/08/12 16:24:46 INFO MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s). | |
22/08/12 16:24:46 INFO MetricsSystemImpl: s3a-file-system metrics system started | |
22/08/12 16:24:46 DEBUG MBeans: Registered Hadoop:service=s3a-file-system,name=MetricsSystem,sub=Control | |
22/08/12 16:24:46 DEBUG MetricsSystemImpl: S3AMetrics1-master.rando2, | |
22/08/12 16:24:46 DEBUG MetricsConfig: poking parent 'PropertiesConfiguration' for key: source.source.start_mbeans | |
22/08/12 16:24:46 DEBUG MetricsConfig: poking parent 'MetricsConfig' for key: source.start_mbeans | |
22/08/12 16:24:46 DEBUG MetricsConfig: poking parent 'PropertiesConfiguration' for key: *.source.start_mbeans | |
22/08/12 16:24:46 DEBUG MetricsSourceAdapter: Updating attr cache... | |
22/08/12 16:24:46 DEBUG MetricsSourceAdapter: Done. # tags & metrics=187 | |
22/08/12 16:24:46 DEBUG MetricsSourceAdapter: Updating info cache... | |
22/08/12 16:24:46 DEBUG MetricsSystemImpl: [javax.management.MBeanAttributeInfo[description=Metrics context, name=tag.Context, type=java.lang.String, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=A unique identifier for the instance, name=tag.s3aFileSystemId, type=java.lang.String, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Hostname from the FS URL, name=tag.bucket, type=java.lang.String, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Local hostname, name=tag.Hostname, type=java.lang.String, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of directories created through the object store., name=directories_created, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of directories deleted through the object store., name=directories_deleted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of files copied within the object store., name=files_copied, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of bytes copied within the object store., name=files_copied_bytes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of files created through the object store., name=files_created, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of files deleted from the object store., name=files_deleted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of files whose delete request was rejected, name=files_delete_rejected, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of fake directory entries created in the object store., name=fake_directories_created, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of fake directory deletes submitted to object store., name=fake_directories_deleted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Errors caught and ignored, name=ignored_errors, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of createNonRecursive(), name=op_create_non_recursive, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of hflush(), name=op_hflush, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of hsync(), name=op_hsync, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of listLocatedStatus(), name=op_list_located_status, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of open(), name=op_open, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object copy requests, name=object_copy_requests, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Objects deleted in delete requests, name=object_delete_objects, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of requests for object metadata, name=object_metadata_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object put/multipart upload completed count, name=object_put_request_completed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=number of bytes uploaded, name=object_put_bytes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of S3 Select requests issued, name=object_select_requests, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of times the TCP stream was aborted, name=stream_aborted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Bytes read from an input stream in read() calls, name=stream_read_bytes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of bytes discarded by aborting an input stream, name=stream_read_bytes_discarded_in_abort, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of bytes read and discarded when closing an input stream, name=stream_read_bytes_discarded_in_close, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of times the TCP stream was closed, name=stream_read_closed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total count of times an attempt to close an input stream was made, name=stream_read_close_operations, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of exceptions raised during input stream reads, name=stream_read_exceptions, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of readFully() operations in an input stream, name=stream_read_fully_operations, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total count of times an input stream to object store data was opened, name=stream_read_opened, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of read() operations in an input stream, name=stream_read_operations, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of incomplete read() operations in an input stream, name=stream_read_operations_incomplete, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of version mismatches encountered while reading an input stream, name=stream_read_version_mismatches, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of executed seek operations which went backwards in a stream, name=stream_read_seek_backward_operations, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of bytes moved backwards during seek operations in an input stream, name=stream_read_bytes_backwards_on_seek, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of bytes read and discarded during seek() in an input stream, name=stream_read_seek_bytes_discarded, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of bytes skipped during forward seek operations an input stream, name=stream_read_seek_bytes_skipped, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of executed seek operations which went forward in an input stream, name=stream_read_seek_forward_operations, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of seek operations in an input stream, name=stream_read_seek_operations, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of times the seek policy was dynamically changed in an input stream, name=stream_read_seek_policy_changed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total count of bytes read from an input stream, name=stream_read_total_bytes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of stream write failures reported, name=stream_write_exceptions, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of failures when finalizing a multipart upload, name=stream_write_exceptions_completing_upload, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of block/partition uploads completed, name=stream_write_block_uploads, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of number of block uploads committed, name=stream_write_block_uploads_committed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of number of block uploads aborted, name=stream_write_block_uploads_aborted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of total time taken for uploads to complete, name=stream_write_total_time, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of total data uploaded, name=stream_write_total_data, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of bytes written to output stream (including all not yet uploaded), name=stream_write_bytes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of files to commit created, name=committer_commits_created, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of files committed, name=committer_commits_completed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of successful jobs, name=committer_jobs_completed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of failed jobs, name=committer_jobs_failed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of successful tasks, name=committer_tasks_completed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of failed tasks, name=committer_tasks_failed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Amount of data committed, name=committer_bytes_committed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of bytes uploaded duing commit operations, name=committer_bytes_uploaded, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of commits failed, name=committer_commits.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of commits aborted, name=committer_commits_aborted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of commits reverted, name=committer_commits_reverted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of files created under 'magic' paths, name=committer_magic_files_created, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store put one metadata path request, name=s3guard_metadatastore_put_path_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store initialization times, name=s3guard_metadatastore_initialization, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store records deleted, name=s3guard_metadatastore_record_deletes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store records read, name=s3guard_metadatastore_record_reads, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store records written, name=s3guard_metadatastore_record_writes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store retry events, name=s3guard_metadatastore_retry, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store throttled events, name=s3guard_metadatastore_throttled, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store authoritative directories updated from S3, name=s3guard_metadatastore_authoritative_directories_updated, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=requests made of the remote store, name=store_io_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=retried requests made of the remote store, name=store_io_retry, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Requests throttled and retried, name=store_io_throttled, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Uploader Instantiated, name=multipart_instantiated, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Part Put Operation, name=multipart_upload_part_put, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Part Put Bytes, name=multipart_upload_part_put_bytes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Upload Aborted, name=multipart_upload_aborted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Upload Abort Unner Path Invoked, name=multipart_upload_abort_under_path_invoked, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Upload Completed, name=multipart_upload_completed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Upload Started, name=multipart_upload_started, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Audit access check was rejected, name=audit_access_check_failure, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Audit Span Created, name=audit_span_creation, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Audit failure/rejection, name=audit_failure, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=AWS request made, name=audit_request_execution, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Current number of active put requests, name=object_put_request_active, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=number of bytes queued for upload/being actively uploaded, name=object_put_bytes_pending, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of block/partition uploads active, name=stream_write_block_uploads_active, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Gauge of block/partitions uploads queued to be written, name=stream_write_block_uploads_pending, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Gauge of data queued to be written, name=stream_write_block_uploads_data_pending, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=gauge to indicate if client side encryption is enabled, name=client_side_encryption_enabled, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Executor acquired., name=action_executor_acquired, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Executor acquired., name=action_executor_acquired.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=HEAD request., name=action_http_head_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=HEAD request., name=action_http_head_request.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=GET request., name=action_http_get_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=GET request., name=action_http_get_request.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of abort(), name=op_abort, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of abort(), name=op_abort.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of access(), name=op_access, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of access(), name=op_access.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of copyFromLocalFile(), name=op_copy_from_local_file, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of copyFromLocalFile(), name=op_copy_from_local_file.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of create(), name=op_create, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of create(), name=op_create.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of delete(), name=op_delete, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of delete(), name=op_delete.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of exists(), name=op_exists, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of exists(), name=op_exists.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getContentSummary(), name=op_get_content_summary, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getContentSummary(), name=op_get_content_summary.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getDelegationToken(), name=op_get_delegation_token, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getDelegationToken(), name=op_get_delegation_token.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getFileChecksum(), name=op_get_file_checksum, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getFileChecksum(), name=op_get_file_checksum.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getFileStatus(), name=op_get_file_status, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getFileStatus(), name=op_get_file_status.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of globStatus(), name=op_glob_status, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of globStatus(), name=op_glob_status.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of isDirectory(), name=op_is_directory, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of isDirectory(), name=op_is_directory.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of isFile(), name=op_is_file, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of isFile(), name=op_is_file.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of listFiles(), name=op_list_files, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of listFiles(), name=op_list_files.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of listStatus(), name=op_list_status, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of listStatus(), name=op_list_status.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of mkdirs(), name=op_mkdirs, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of mkdirs(), name=op_mkdirs.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of rename(), name=op_rename, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of rename(), name=op_rename.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getXAttrs(Path path), name=op_xattr_get_map, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getXAttrs(Path path), name=op_xattr_get_map.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getXAttr(Path, String), name=op_xattr_get_named, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getXAttr(Path, String), name=op_xattr_get_named.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of xattr(), name=op_xattr_get_named_map, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of xattr(), name=op_xattr_get_named_map.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getXAttrs(Path path, List<String> names), name=op_xattr_list, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getXAttrs(Path path, List<String> names), name=op_xattr_list.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object delete requests, name=object_delete_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object delete requests, name=object_delete_request.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object bulk delete requests, name=object_bulk_delete_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object bulk delete requests, name=object_bulk_delete_request.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of object listings made, name=object_list_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of object listings made, name=object_list_request.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of continued object listings made, name=object_continue_list_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of continued object listings made, name=object_continue_list_request.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object multipart upload initiated, name=object_multipart_initiated, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object multipart upload initiated, name=object_multipart_initiated.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object multipart upload aborted, name=object_multipart_aborted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object multipart upload aborted, name=object_multipart_aborted.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object put/multipart upload count, name=object_put_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object put/multipart upload count, name=object_put_request.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total queue duration of all block uploads, name=stream_write_queue_duration, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total queue duration of all block uploads, name=stream_write_queue_duration.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Duration Tracking of time to commit an entire job, name=committer_commit_job, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Duration Tracking of time to commit an entire job, name=committer_commit_job.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Duration Tracking of time to materialize a file in job commit, name=committer_materialize_file, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Duration Tracking of time to materialize a file in job commit, name=committer_materialize_file.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Duration Tracking of files uploaded from a local staging path, name=committer_stage_file_upload, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Duration Tracking of files uploaded from a local staging path, name=committer_stage_file_upload.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Store Existence Probe, name=store_exists_probe, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Store Existence Probe, name=store_exists_probe.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of delegation tokens issued, name=delegation_tokens_issued, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of delegation tokens issued, name=delegation_tokens_issued.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Upload List, name=multipart_upload_list, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Upload List, name=multipart_upload_list.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of ops for s3Guard metadata store put one metadata path latency with 1s interval, name=S3guard_metadatastore_put_path_latencyNumOps, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=50 percentile latency with 1 second interval for s3Guard metadata store put one metadata path latency, name=S3guard_metadatastore_put_path_latency50thPercentileLatency, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=75 percentile latency with 1 second interval for s3Guard metadata store put one metadata path latency, name=S3guard_metadatastore_put_path_latency75thPercentileLatency, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=90 percentile latency with 1 second interval for s3Guard metadata store put one metadata path latency, name=S3guard_metadatastore_put_path_latency90thPercentileLatency, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=95 percentile latency with 1 second interval for s3Guard metadata store put one metadata path latency, name=S3guard_metadatastore_put_path_latency95thPercentileLatency, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=99 percentile latency with 1 second interval for s3Guard metadata store put one metadata path latency, name=S3guard_metadatastore_put_path_latency99thPercentileLatency, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of events for s3Guard metadata store throttle rate with 1s interval, name=S3guard_metadatastore_throttle_rateNumEvents, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=50 percentile frequency (Hz) with 1 second interval for s3Guard metadata store throttle rate, name=S3guard_metadatastore_throttle_rate50thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=75 percentile frequency (Hz) with 1 second interval for s3Guard metadata store throttle rate, name=S3guard_metadatastore_throttle_rate75thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=90 percentile frequency (Hz) with 1 second interval for s3Guard metadata store throttle rate, name=S3guard_metadatastore_throttle_rate90thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=95 percentile frequency (Hz) with 1 second interval for s3Guard metadata store throttle rate, name=S3guard_metadatastore_throttle_rate95thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=99 percentile frequency (Hz) with 1 second interval for s3Guard metadata store throttle rate, name=S3guard_metadatastore_throttle_rate99thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of events for rate of S3 request throttling with 1s interval, name=Store_io_throttle_rateNumEvents, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=50 percentile frequency (Hz) with 1 second interval for rate of S3 request throttling, name=Store_io_throttle_rate50thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=75 percentile frequency (Hz) with 1 second interval for rate of S3 request throttling, name=Store_io_throttle_rate75thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=90 percentile frequency (Hz) with 1 second interval for rate of S3 request throttling, name=Store_io_throttle_rate90thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=95 percentile frequency (Hz) with 1 second interval for rate of S3 request throttling, name=Store_io_throttle_rate95thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=99 percentile frequency (Hz) with 1 second interval for rate of S3 request throttling, name=Store_io_throttle_rate99thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}]] | |
22/08/12 16:24:46 DEBUG MetricsSourceAdapter: Done | |
22/08/12 16:24:46 DEBUG MBeans: Registered Hadoop:service=s3a-file-system,name=S3AMetrics1-master.rando2 | |
22/08/12 16:24:46 DEBUG MetricsSourceAdapter: MBean for source S3AMetrics1-master.rando2 registered. | |
22/08/12 16:24:46 DEBUG MetricsSystemImpl: Registered source S3AMetrics1-master.rando2 | |
22/08/12 16:24:46 DEBUG S3AFileSystem: Client Side Encryption enabled: false | |
22/08/12 16:24:46 DEBUG S3ARetryPolicy: Retrying on recoverable AWS failures 7 times with an initial interval of 500ms | |
22/08/12 16:24:46 DEBUG S3GuardExistsRetryPolicy: Retrying on recoverable S3Guard table/S3 inconsistencies 7 times with an initial interval of 2000ms | |
22/08/12 16:24:46 DEBUG S3AUtils: Value of fs.s3a.paging.maximum is 5000 | |
22/08/12 16:24:46 DEBUG S3AUtils: Value of fs.s3a.block.size is 33554432 | |
22/08/12 16:24:46 DEBUG S3AUtils: Value of fs.s3a.readahead.range is 65536 | |
22/08/12 16:24:46 DEBUG S3AUtils: Value of fs.s3a.max.total.tasks is 32 | |
22/08/12 16:24:46 DEBUG S3AUtils: Value of fs.s3a.threads.keepalivetime is 60 | |
22/08/12 16:24:46 DEBUG S3AUtils: Value of fs.s3a.executor.capacity is 16 | |
22/08/12 16:24:46 DEBUG SignerManager: No custom signers specified | |
22/08/12 16:24:46 DEBUG AuditIntegration: auditing is disabled | |
22/08/12 16:24:46 DEBUG AbstractService: Service: NoopAuditManagerS3A entered state INITED | |
22/08/12 16:24:46 DEBUG CompositeService: NoopAuditManagerS3A: initing services, size=0 | |
22/08/12 16:24:46 DEBUG CompositeService: Adding service NoopAuditor | |
22/08/12 16:24:46 DEBUG AbstractService: Service: NoopAuditor entered state INITED | |
22/08/12 16:24:46 DEBUG CompositeService: NoopAuditManagerS3A: starting services, size=1 | |
22/08/12 16:24:46 DEBUG AbstractService: Service NoopAuditor is started | |
22/08/12 16:24:46 DEBUG AbstractService: Service NoopAuditManagerS3A is started | |
22/08/12 16:24:46 DEBUG AuditIntegration: Started Audit Manager Service NoopAuditManagerS3A in state NoopAuditManagerS3A: STARTED | |
22/08/12 16:24:46 DEBUG S3AUtils: Value of fs.s3a.internal.upload.part.count.limit is 10000 | |
22/08/12 16:24:46 DEBUG S3ARetryPolicy: Retrying on recoverable AWS failures 7 times with an initial interval of 500ms | |
22/08/12 16:24:46 DEBUG S3AUtils: Credential provider class is org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider | |
22/08/12 16:24:46 DEBUG S3AUtils: Credential provider class is org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider | |
22/08/12 16:24:46 DEBUG S3AUtils: Credential provider class is com.amazonaws.auth.EnvironmentVariableCredentialsProvider | |
22/08/12 16:24:46 DEBUG S3AUtils: Credential provider class is org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider | |
22/08/12 16:24:47 DEBUG S3AUtils: For URI s3a://master.rando2/nonemptyprefix8, using credentials AWSCredentialProviderList[refcount= 1: [TemporaryAWSCredentialsProvider, SimpleAWSCredentialsProvider, EnvironmentVariableCredentialsProvider, org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider@717fe440] | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Using credential provider AWSCredentialProviderList[refcount= 1: [TemporaryAWSCredentialsProvider, SimpleAWSCredentialsProvider, EnvironmentVariableCredentialsProvider, org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider@717fe440] | |
22/08/12 16:24:47 DEBUG S3AUtils: Value of fs.s3a.connection.maximum is 96 | |
22/08/12 16:24:47 DEBUG S3AUtils: Value of fs.s3a.attempts.maximum is 20 | |
22/08/12 16:24:47 DEBUG S3AUtils: Value of fs.s3a.connection.establish.timeout is 5000 | |
22/08/12 16:24:47 DEBUG S3AUtils: Value of fs.s3a.connection.timeout is 200000 | |
22/08/12 16:24:47 DEBUG S3AUtils: Value of fs.s3a.socket.send.buffer is 8192 | |
22/08/12 16:24:47 DEBUG S3AUtils: Value of fs.s3a.socket.recv.buffer is 8192 | |
22/08/12 16:24:47 DEBUG S3AUtils: Using User-Agent: Hadoop 3.3.2 | |
22/08/12 16:24:47 DEBUG S3AUtils: Data is unencrypted | |
22/08/12 16:24:47 DEBUG AmazonWebServiceClient: Internal logging successfully configured to commons logger: true | |
22/08/12 16:24:47 DEBUG AwsSdkMetrics: Admin mbean registered under com.amazonaws.management:type=AwsSdkMetrics | |
22/08/12 16:24:47 DEBUG DefaultS3ClientFactory: Creating endpoint configuration for "http://localhost:30600" | |
22/08/12 16:24:47 DEBUG DefaultS3ClientFactory: Endpoint URI = http://localhost:30600 | |
22/08/12 16:24:47 DEBUG DefaultS3ClientFactory: Endpoint http://localhost:30600 is not the default; parsing | |
22/08/12 16:24:47 DEBUG DefaultS3ClientFactory: Region for endpoint http://localhost:30600, URI http://localhost:30600 is determined as null | |
22/08/12 16:24:47 DEBUG FsUrlStreamHandlerFactory: Creating handler for protocol http | |
22/08/12 16:24:47 DEBUG FsUrlStreamHandlerFactory: Unknown protocol http, delegating to default implementation | |
22/08/12 16:24:47 DEBUG CsmConfigurationProviderChain: Unable to load configuration from com.amazonaws.monitoring.EnvironmentVariableCsmConfigurationProvider@6e2dc702: Unable to load Client Side Monitoring configurations from environment variables! | |
22/08/12 16:24:47 DEBUG CsmConfigurationProviderChain: Unable to load configuration from com.amazonaws.monitoring.SystemPropertyCsmConfigurationProvider@52ebaf37: Unable to load Client Side Monitoring configurations from system properties variables! | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Closing connections idle longer than 60000 MILLISECONDS | |
22/08/12 16:24:47 DEBUG CsmConfigurationProviderChain: Unable to load configuration from com.amazonaws.monitoring.ProfileCsmConfigurationProvider@9ad30ff: The 'default' profile does not define all the required properties! | |
22/08/12 16:24:47 DEBUG S3AFileSystem: skipping check for bucket existence | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Input fadvise policy = normal | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Change detection policy = ETagChangeDetectionPolicy mode=None | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Filesystem support for magic committers is enabled | |
22/08/12 16:24:47 DEBUG S3AUtils: Value of fs.s3a.fast.upload.active.blocks is 4 | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Using S3ABlockOutputStream with buffer = disk; block=67108864; queue limit=4 | |
22/08/12 16:24:47 DEBUG S3Guard: Metastore option source [core-default.xml] | |
22/08/12 16:24:47 DEBUG S3Guard: Using NullMetadataStore metadata store for s3a filesystem | |
22/08/12 16:24:47 DEBUG S3AFileSystem: S3Guard is disabled on this bucket: master.rando2 | |
22/08/12 16:24:47 DEBUG DirectoryPolicyImpl: Directory markers will be deleted | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Directory marker retention policy is DirectoryMarkerRetention{policy='delete'} | |
22/08/12 16:24:47 DEBUG S3AUtils: Value of fs.s3a.multipart.purge.age is 86400 | |
22/08/12 16:24:47 DEBUG S3AUtils: Value of fs.s3a.bulk.delete.page.size is 250 | |
22/08/12 16:24:47 DEBUG FileSystem: Creating FS s3a://master.rando2/nonemptyprefix8: duration 0:00.715s | |
22/08/12 16:24:47 DEBUG FileCommitProtocol: Creating committer org.apache.spark.sql.execution.datasources.SQLHadoopMapReduceCommitProtocol; job 31b8f587-d086-4d3b-bf6d-4ce13ea2fecc; output=s3a://master.rando2/nonemptyprefix8; dynamic=false | |
22/08/12 16:24:47 DEBUG FileCommitProtocol: Using (String, String, Boolean) constructor | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists by 1 with final value 1 | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8 (nonemptyprefix8); needEmptyDirectory=false | |
22/08/12 16:24:47 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8 | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 1 | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 1 | |
22/08/12 16:24:47 DEBUG S3AFileSystem: HEAD nonemptyprefix8 with change tracker null | |
22/08/12 16:24:47 DEBUG Invoker: Starting: create credentials | |
22/08/12 16:24:47 DEBUG Invoker: create credentials: duration 0:00.004s | |
22/08/12 16:24:47 DEBUG AWSCredentialProviderList: No credentials from TemporaryAWSCredentialsProvider: org.apache.hadoop.fs.s3a.auth.NoAwsCredentialsException: Session credentials in Hadoop configuration: No AWS Credentials | |
22/08/12 16:24:47 DEBUG AWSCredentialProviderList: Using credentials from SimpleAWSCredentialsProvider | |
22/08/12 16:24:47 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix8 Headers: (amz-sdk-invocation-id: daff2a9c-2d42-a289-c168-fd308a999145, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD | |
/master.rando2/nonemptyprefix8 | |
amz-sdk-invocation-id:daff2a9c-2d42-a289-c168-fd308a999145 | |
amz-sdk-request:attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-type:application/octet-stream | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
x-amz-date:20220812T152447Z | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date | |
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | |
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152447Z | |
20220812/us-east-1/s3/aws4_request | |
09eb0b140b531b6a30b0a76354ce47f4c0aa77ef031f1d8595e5f14111629d1b" | |
22/08/12 16:24:47 DEBUG AWS4Signer: Generating a new signing key as the signing key not available in the cache for the date 1660262400000 | |
22/08/12 16:24:47 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:47 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 0; route allocated: 0 of 96; total allocated: 0 of 96] | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG MainClientExec: Opening connection {}->http://localhost:30600 | |
22/08/12 16:24:47 DEBUG DefaultHttpClientConnectionOperator: Connecting to localhost/127.0.0.1:30600 | |
22/08/12 16:24:47 DEBUG DefaultHttpClientConnectionOperator: Connection established 127.0.0.1:59728<->127.0.0.1:30600 | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:47 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix8 HTTP/1.1 | |
22/08/12 16:24:47 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix8 HTTP/1.1 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: daff2a9c-2d42-a289-c168-fd308a999145 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-request: attempt=1;max=21 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=7a1f1dab9b137bd736489828a6596cc4039d538e9e0c24fca5853bef77d74cd5 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152447Z | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix8 HTTP/1.1[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: daff2a9c-2d42-a289-c168-fd308a999145[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=7a1f1dab9b137bd736489828a6596cc4039d538e9e0c24fca5853bef77d74cd5[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152447Z[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 3daab1a0-6106-4800-bbde-d1a1b7688a30[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 3daab1a0-6106-4800-bbde-d1a1b7688a30[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:47 GMT[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Length: 238[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Type: application/xml | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 3daab1a0-6106-4800-bbde-d1a1b7688a30 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 3daab1a0-6106-4800-bbde-d1a1b7688a30 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:47 GMT | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Length: 238 | |
22/08/12 16:24:47 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:47 GMT | |
22/08/12 16:24:47 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 3daab1a0-6106-4800-bbde-d1a1b7688a30; S3 Extended Request ID: 3daab1a0-6106-4800-bbde-d1a1b7688a30; Proxy: null), S3 Extended Request ID: 3daab1a0-6106-4800-bbde-d1a1b7688a30 | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 1 | |
22/08/12 16:24:47 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 3daab1a0-6106-4800-bbde-d1a1b7688a30; S3 Extended Request ID: 3daab1a0-6106-4800-bbde-d1a1b7688a30; Proxy: null), S3 Extended Request ID: 3daab1a0-6106-4800-bbde-d1a1b7688a30], RequestType=[GetObjectMetadataRequest], AWSRequestID=[3daab1a0-6106-4800-bbde-d1a1b7688a30], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=0, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[136.637], HttpClientSendRequestTime=[2.668], HttpRequestTime=[69.853], ApiCallLatency=[120.482], RequestSigningTime=[32.408], CredentialsRequestTime=[5.573, 0.016], HttpClientReceiveResponseTime=[18.711], | |
22/08/12 16:24:47 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix8/ delimiter=/ keys=2 requester pays=false | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Starting: LIST | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 1 | |
22/08/12 16:24:47 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix8/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 22c79b53-2c3d-5be0-a595-a586ebb84541, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET | |
/master.rando2/ | |
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix8%2F | |
amz-sdk-invocation-id:22c79b53-2c3d-5be0-a595-a586ebb84541 | |
amz-sdk-request:ttl=20220812T152807Z;attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-type:application/octet-stream | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
x-amz-date:20220812T152447Z | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date | |
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | |
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152447Z | |
20220812/us-east-1/s3/aws4_request | |
019acf4980f1ae26885efbc8e4913bd8ac4c94c2eaa3ccf2f282556c4ef43fb8" | |
22/08/12 16:24:47 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:47 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:47 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F&fetch-owner=false HTTP/1.1 | |
22/08/12 16:24:47 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F&fetch-owner=false HTTP/1.1 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 22c79b53-2c3d-5be0-a595-a586ebb84541 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=359683b5b489e76389fdb97672339667f5e7a0471f09890bbe933ab7c1b410e6 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152447Z | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Length: 0 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F&fetch-owner=false HTTP/1.1[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 22c79b53-2c3d-5be0-a595-a586ebb84541[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=359683b5b489e76389fdb97672339667f5e7a0471f09890bbe933ab7c1b410e6[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152447Z[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: cd3e943f-8e26-426b-9953-1dea8efbcf46[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: cd3e943f-8e26-426b-9953-1dea8efbcf46[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:47 GMT[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Length: 276[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix8/</Prefix></ListBucketResult>" | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Type: application/xml | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: cd3e943f-8e26-426b-9953-1dea8efbcf46 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: cd3e943f-8e26-426b-9953-1dea8efbcf46 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:47 GMT | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Length: 276 | |
22/08/12 16:24:47 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:47 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:47 GMT | |
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler | |
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2 | |
22/08/12 16:24:47 DEBUG request: Received successful response: 200, AWS Request ID: cd3e943f-8e26-426b-9953-1dea8efbcf46 | |
22/08/12 16:24:47 DEBUG requestId: x-amzn-RequestId: not available | |
22/08/12 16:24:47 DEBUG requestId: AWS Request ID: cd3e943f-8e26-426b-9953-1dea8efbcf46 | |
22/08/12 16:24:47 DEBUG requestId: AWS Extended Request ID: cd3e943f-8e26-426b-9953-1dea8efbcf46 | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 2 | |
22/08/12 16:24:47 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[cd3e943f-8e26-426b-9953-1dea8efbcf46], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[37.947], ClientExecuteTime=[80.903], HttpClientSendRequestTime=[1.577], HttpRequestTime=[23.382], ApiCallLatency=[80.641], RequestSigningTime=[0.526], CredentialsRequestTime=[0.008, 0.012], HttpClientReceiveResponseTime=[19.702], | |
22/08/12 16:24:47 DEBUG S3AFileSystem: LIST: duration 0:00.083s | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8 | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists.failures by 1 with final value 1 | |
22/08/12 16:24:47 INFO ParquetFileFormat: Using default output committer for Parquet: org.apache.parquet.hadoop.ParquetOutputCommitter | |
22/08/12 16:24:47 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121624471812156148346658793_0000}; taskId=attempt_202208121624471812156148346658793_0000_m_000000_0, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@7b9e186f}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix8 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121624471812156148346658793_0000}; taskId=attempt_202208121624471812156148346658793_0000_m_000000_0, status=''} | |
22/08/12 16:24:47 INFO FileOutputCommitter: File Output Committer Algorithm version is 1 | |
22/08/12 16:24:47 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false | |
22/08/12 16:24:47 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter | |
22/08/12 16:24:47 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121624471812156148346658793_0000}; taskId=attempt_202208121624471812156148346658793_0000_m_000000_0, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@3ff4f746}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix8 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121624471812156148346658793_0000}; taskId=attempt_202208121624471812156148346658793_0000_m_000000_0, status=''} | |
22/08/12 16:24:47 INFO FileOutputCommitter: File Output Committer Algorithm version is 1 | |
22/08/12 16:24:47 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false | |
22/08/12 16:24:47 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter op_mkdirs by 1 with final value 1 | |
22/08/12 16:24:47 DEBUG MkdirOperation: Making directory: s3a://master.rando2/nonemptyprefix8/_temporary/0 | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8/_temporary/0 (nonemptyprefix8/_temporary/0); needEmptyDirectory=false | |
22/08/12 16:24:47 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8/_temporary/0 | |
22/08/12 16:24:47 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix8/_temporary/0/ delimiter=/ keys=2 requester pays=false | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Starting: LIST | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 2 | |
22/08/12 16:24:47 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix8/_temporary/0/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: a9501fd6-6372-0185-fe8b-85005b8d2afe, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET | |
/master.rando2/ | |
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F | |
amz-sdk-invocation-id:a9501fd6-6372-0185-fe8b-85005b8d2afe | |
amz-sdk-request:ttl=20220812T152807Z;attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-type:application/octet-stream | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
x-amz-date:20220812T152447Z | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date | |
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | |
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152447Z | |
20220812/us-east-1/s3/aws4_request | |
f4caaffa42a171e08d2295eab15dda7db3dac909fcde4b5507747310c177f6ee" | |
22/08/12 16:24:47 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:47 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:47 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1 | |
22/08/12 16:24:47 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: a9501fd6-6372-0185-fe8b-85005b8d2afe | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=caa695eb2a1be3f95e9101d66cb8389d82d23bdc653e4b1bdd50537606be4705 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152447Z | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Length: 0 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: a9501fd6-6372-0185-fe8b-85005b8d2afe[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=caa695eb2a1be3f95e9101d66cb8389d82d23bdc653e4b1bdd50537606be4705[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152447Z[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 6aafe837-cd7b-40b7-838a-f3089fe62a59[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 6aafe837-cd7b-40b7-838a-f3089fe62a59[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:47 GMT[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Length: 289[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix8/_temporary/0/</Prefix></ListBucketResult>" | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Type: application/xml | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 6aafe837-cd7b-40b7-838a-f3089fe62a59 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 6aafe837-cd7b-40b7-838a-f3089fe62a59 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:47 GMT | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Length: 289 | |
22/08/12 16:24:47 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:47 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:47 GMT | |
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler | |
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2 | |
22/08/12 16:24:47 DEBUG request: Received successful response: 200, AWS Request ID: 6aafe837-cd7b-40b7-838a-f3089fe62a59 | |
22/08/12 16:24:47 DEBUG requestId: x-amzn-RequestId: not available | |
22/08/12 16:24:47 DEBUG requestId: AWS Request ID: 6aafe837-cd7b-40b7-838a-f3089fe62a59 | |
22/08/12 16:24:47 DEBUG requestId: AWS Extended Request ID: 6aafe837-cd7b-40b7-838a-f3089fe62a59 | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 3 | |
22/08/12 16:24:47 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[6aafe837-cd7b-40b7-838a-f3089fe62a59], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[2.28], ClientExecuteTime=[26.321], HttpClientSendRequestTime=[1.173], HttpRequestTime=[22.087], ApiCallLatency=[26.062], RequestSigningTime=[0.581], CredentialsRequestTime=[0.009, 0.004], HttpClientReceiveResponseTime=[19.775], | |
22/08/12 16:24:47 DEBUG S3AFileSystem: LIST: duration 0:00.027s | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8/_temporary/0 | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8/_temporary/0 (nonemptyprefix8/_temporary/0); needEmptyDirectory=false | |
22/08/12 16:24:47 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8/_temporary/0 | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 2 | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 2 | |
22/08/12 16:24:47 DEBUG S3AFileSystem: HEAD nonemptyprefix8/_temporary/0 with change tracker null | |
22/08/12 16:24:47 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix8/_temporary/0 Headers: (amz-sdk-invocation-id: e8b4813b-fc7a-8967-7924-64c90534fc39, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD | |
/master.rando2/nonemptyprefix8/_temporary/0 | |
amz-sdk-invocation-id:e8b4813b-fc7a-8967-7924-64c90534fc39 | |
amz-sdk-request:ttl=20220812T152807Z;attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-type:application/octet-stream | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
x-amz-date:20220812T152447Z | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date | |
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | |
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152447Z | |
20220812/us-east-1/s3/aws4_request | |
eaf3b5deb5e1b4da4d9933c57201ae5d00e2b93686fc151bf229f841c3d7ef43" | |
22/08/12 16:24:47 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:47 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:47 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix8/_temporary/0 HTTP/1.1 | |
22/08/12 16:24:47 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix8/_temporary/0 HTTP/1.1 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: e8b4813b-fc7a-8967-7924-64c90534fc39 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=73372ffd119ab8114dac440234fc87dcb379c69f423c1088ebf21092968a77cd | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152447Z | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix8/_temporary/0 HTTP/1.1[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: e8b4813b-fc7a-8967-7924-64c90534fc39[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=73372ffd119ab8114dac440234fc87dcb379c69f423c1088ebf21092968a77cd[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152447Z[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: ff19ddf7-5f6c-4726-9cc9-db2290704a16[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: ff19ddf7-5f6c-4726-9cc9-db2290704a16[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:47 GMT[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Length: 251[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Type: application/xml | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: ff19ddf7-5f6c-4726-9cc9-db2290704a16 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: ff19ddf7-5f6c-4726-9cc9-db2290704a16 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:47 GMT | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Length: 251 | |
22/08/12 16:24:47 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:47 GMT | |
22/08/12 16:24:47 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: ff19ddf7-5f6c-4726-9cc9-db2290704a16; S3 Extended Request ID: ff19ddf7-5f6c-4726-9cc9-db2290704a16; Proxy: null), S3 Extended Request ID: ff19ddf7-5f6c-4726-9cc9-db2290704a16 | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 4 | |
22/08/12 16:24:47 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: ff19ddf7-5f6c-4726-9cc9-db2290704a16; S3 Extended Request ID: ff19ddf7-5f6c-4726-9cc9-db2290704a16; Proxy: null), S3 Extended Request ID: ff19ddf7-5f6c-4726-9cc9-db2290704a16], RequestType=[GetObjectMetadataRequest], AWSRequestID=[ff19ddf7-5f6c-4726-9cc9-db2290704a16], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[22.965], HttpClientSendRequestTime=[0.845], HttpRequestTime=[21.581], ApiCallLatency=[22.746], RequestSigningTime=[0.311], CredentialsRequestTime=[0.006, 0.003], HttpClientReceiveResponseTime=[19.578], | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8/_temporary/0 | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8/_temporary (nonemptyprefix8/_temporary); needEmptyDirectory=false | |
22/08/12 16:24:47 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8/_temporary | |
22/08/12 16:24:47 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix8/_temporary/ delimiter=/ keys=2 requester pays=false | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Starting: LIST | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 3 | |
22/08/12 16:24:47 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix8/_temporary/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 6f197e9e-ece8-147a-639b-49f0ce133788, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET | |
/master.rando2/ | |
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F | |
amz-sdk-invocation-id:6f197e9e-ece8-147a-639b-49f0ce133788 | |
amz-sdk-request:ttl=20220812T152807Z;attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-type:application/octet-stream | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
x-amz-date:20220812T152447Z | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date | |
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | |
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152447Z | |
20220812/us-east-1/s3/aws4_request | |
4a87fed889492bff077761a34fbd8ca98c2c69f7ef878d776f8583cd8f2e691a" | |
22/08/12 16:24:47 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:47 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:47 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F&fetch-owner=false HTTP/1.1 | |
22/08/12 16:24:47 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F&fetch-owner=false HTTP/1.1 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 6f197e9e-ece8-147a-639b-49f0ce133788 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=862667900d080a98e9cbbae8321a52d307bf7a38375c16e85475d94b6c426ff9 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152447Z | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Length: 0 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F&fetch-owner=false HTTP/1.1[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 6f197e9e-ece8-147a-639b-49f0ce133788[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=862667900d080a98e9cbbae8321a52d307bf7a38375c16e85475d94b6c426ff9[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152447Z[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 91414309-86a8-4b18-af1b-2b7f37d37b21[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 91414309-86a8-4b18-af1b-2b7f37d37b21[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:47 GMT[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Length: 287[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix8/_temporary/</Prefix></ListBucketResult>" | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Type: application/xml | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 91414309-86a8-4b18-af1b-2b7f37d37b21 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 91414309-86a8-4b18-af1b-2b7f37d37b21 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:47 GMT | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Length: 287 | |
22/08/12 16:24:47 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:47 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:47 GMT | |
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler | |
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2 | |
22/08/12 16:24:47 DEBUG request: Received successful response: 200, AWS Request ID: 91414309-86a8-4b18-af1b-2b7f37d37b21 | |
22/08/12 16:24:47 DEBUG requestId: x-amzn-RequestId: not available | |
22/08/12 16:24:47 DEBUG requestId: AWS Request ID: 91414309-86a8-4b18-af1b-2b7f37d37b21 | |
22/08/12 16:24:47 DEBUG requestId: AWS Extended Request ID: 91414309-86a8-4b18-af1b-2b7f37d37b21 | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 5 | |
22/08/12 16:24:47 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[91414309-86a8-4b18-af1b-2b7f37d37b21], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[1.872], ClientExecuteTime=[22.39], HttpClientSendRequestTime=[1.189], HttpRequestTime=[18.897], ApiCallLatency=[22.139], RequestSigningTime=[0.55], CredentialsRequestTime=[0.007, 0.005], HttpClientReceiveResponseTime=[16.639], | |
22/08/12 16:24:47 DEBUG S3AFileSystem: LIST: duration 0:00.023s | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8/_temporary | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8/_temporary (nonemptyprefix8/_temporary); needEmptyDirectory=false | |
22/08/12 16:24:47 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8/_temporary | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 3 | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 3 | |
22/08/12 16:24:47 DEBUG S3AFileSystem: HEAD nonemptyprefix8/_temporary with change tracker null | |
22/08/12 16:24:47 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix8/_temporary Headers: (amz-sdk-invocation-id: 51b43d6b-15fa-d3a7-0f7c-294df2b09065, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD | |
/master.rando2/nonemptyprefix8/_temporary | |
amz-sdk-invocation-id:51b43d6b-15fa-d3a7-0f7c-294df2b09065 | |
amz-sdk-request:ttl=20220812T152807Z;attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-type:application/octet-stream | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
x-amz-date:20220812T152447Z | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date | |
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | |
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152447Z | |
20220812/us-east-1/s3/aws4_request | |
edc7e55f1ec0754e6c860fee6c88f79b470b0b5144e6d8c332c6b1c1b4d57f03" | |
22/08/12 16:24:47 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:47 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:47 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix8/_temporary HTTP/1.1 | |
22/08/12 16:24:47 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix8/_temporary HTTP/1.1 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 51b43d6b-15fa-d3a7-0f7c-294df2b09065 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=2574e43ca3e37e49613b59ca5b2ea39f6745f8b8c29c8e940813a73669a48211 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152447Z | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix8/_temporary HTTP/1.1[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 51b43d6b-15fa-d3a7-0f7c-294df2b09065[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=2574e43ca3e37e49613b59ca5b2ea39f6745f8b8c29c8e940813a73669a48211[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152447Z[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: b803373f-f269-4997-a6dd-b1da6a68a524[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: b803373f-f269-4997-a6dd-b1da6a68a524[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:47 GMT[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Length: 249[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Type: application/xml | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: b803373f-f269-4997-a6dd-b1da6a68a524 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: b803373f-f269-4997-a6dd-b1da6a68a524 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:47 GMT | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Length: 249 | |
22/08/12 16:24:47 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:47 GMT | |
22/08/12 16:24:47 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: b803373f-f269-4997-a6dd-b1da6a68a524; S3 Extended Request ID: b803373f-f269-4997-a6dd-b1da6a68a524; Proxy: null), S3 Extended Request ID: b803373f-f269-4997-a6dd-b1da6a68a524 | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 6 | |
22/08/12 16:24:47 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: b803373f-f269-4997-a6dd-b1da6a68a524; S3 Extended Request ID: b803373f-f269-4997-a6dd-b1da6a68a524; Proxy: null), S3 Extended Request ID: b803373f-f269-4997-a6dd-b1da6a68a524], RequestType=[GetObjectMetadataRequest], AWSRequestID=[b803373f-f269-4997-a6dd-b1da6a68a524], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[16.192], HttpClientSendRequestTime=[0.69], HttpRequestTime=[15.04], ApiCallLatency=[16.003], RequestSigningTime=[0.313], CredentialsRequestTime=[0.005, 0.003], HttpClientReceiveResponseTime=[13.422], | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8/_temporary | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8 (nonemptyprefix8); needEmptyDirectory=false | |
22/08/12 16:24:47 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8 | |
22/08/12 16:24:47 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix8/ delimiter=/ keys=2 requester pays=false | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Starting: LIST | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 4 | |
22/08/12 16:24:47 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix8/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 7aa1f9d8-63be-c804-2d80-759c8c87cc8d, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET | |
/master.rando2/ | |
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix8%2F | |
amz-sdk-invocation-id:7aa1f9d8-63be-c804-2d80-759c8c87cc8d | |
amz-sdk-request:ttl=20220812T152807Z;attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-type:application/octet-stream | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
x-amz-date:20220812T152447Z | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date | |
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | |
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152447Z | |
20220812/us-east-1/s3/aws4_request | |
b37845cd551c8cb5d2915212bbf0f39774dd03e3ab428c5cf88d16fdb5838b6e" | |
22/08/12 16:24:47 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:47 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:47 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F&fetch-owner=false HTTP/1.1 | |
22/08/12 16:24:47 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F&fetch-owner=false HTTP/1.1 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 7aa1f9d8-63be-c804-2d80-759c8c87cc8d | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=79eb3148c7e5c30911c8b005724da88280c6e23d24e8e0ef46ed56b6584a8d02 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152447Z | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Length: 0 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F&fetch-owner=false HTTP/1.1[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 7aa1f9d8-63be-c804-2d80-759c8c87cc8d[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=79eb3148c7e5c30911c8b005724da88280c6e23d24e8e0ef46ed56b6584a8d02[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152447Z[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 6131f265-28b9-4d23-a8f1-cc70617cb6af[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 6131f265-28b9-4d23-a8f1-cc70617cb6af[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:47 GMT[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Length: 276[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix8/</Prefix></ListBucketResult>" | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Type: application/xml | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 6131f265-28b9-4d23-a8f1-cc70617cb6af | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 6131f265-28b9-4d23-a8f1-cc70617cb6af | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:47 GMT | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Length: 276 | |
22/08/12 16:24:47 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:47 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:47 GMT | |
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler | |
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2 | |
22/08/12 16:24:47 DEBUG request: Received successful response: 200, AWS Request ID: 6131f265-28b9-4d23-a8f1-cc70617cb6af | |
22/08/12 16:24:47 DEBUG requestId: x-amzn-RequestId: not available | |
22/08/12 16:24:47 DEBUG requestId: AWS Request ID: 6131f265-28b9-4d23-a8f1-cc70617cb6af | |
22/08/12 16:24:47 DEBUG requestId: AWS Extended Request ID: 6131f265-28b9-4d23-a8f1-cc70617cb6af | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 7 | |
22/08/12 16:24:47 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[6131f265-28b9-4d23-a8f1-cc70617cb6af], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[1.891], ClientExecuteTime=[17.214], HttpClientSendRequestTime=[0.746], HttpRequestTime=[14.048], ApiCallLatency=[17.015], RequestSigningTime=[0.384], CredentialsRequestTime=[0.006, 0.003], HttpClientReceiveResponseTime=[12.559], | |
22/08/12 16:24:47 DEBUG S3AFileSystem: LIST: duration 0:00.018s | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8 | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8 (nonemptyprefix8); needEmptyDirectory=false | |
22/08/12 16:24:47 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8 | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 4 | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 4 | |
22/08/12 16:24:47 DEBUG S3AFileSystem: HEAD nonemptyprefix8 with change tracker null | |
22/08/12 16:24:47 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix8 Headers: (amz-sdk-invocation-id: 916bbbb3-be37-d245-ea23-ebeb9132a28b, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD | |
/master.rando2/nonemptyprefix8 | |
amz-sdk-invocation-id:916bbbb3-be37-d245-ea23-ebeb9132a28b | |
amz-sdk-request:ttl=20220812T152807Z;attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-type:application/octet-stream | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
x-amz-date:20220812T152447Z | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date | |
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | |
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152447Z | |
20220812/us-east-1/s3/aws4_request | |
eaf3b75856e032fd560c7786eedad643c3aeec9f6ed34755f79636dfc263fbac" | |
22/08/12 16:24:47 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:47 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:47 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix8 HTTP/1.1 | |
22/08/12 16:24:47 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix8 HTTP/1.1 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 916bbbb3-be37-d245-ea23-ebeb9132a28b | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=4bebe31525db92dca60f96ba1a28fde085378b487ee8cdf8ea5bf9e3b98397a7 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152447Z | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix8 HTTP/1.1[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 916bbbb3-be37-d245-ea23-ebeb9132a28b[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=4bebe31525db92dca60f96ba1a28fde085378b487ee8cdf8ea5bf9e3b98397a7[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152447Z[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 38a86ef9-0270-4ffc-a709-d68818fccdaa[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 38a86ef9-0270-4ffc-a709-d68818fccdaa[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:47 GMT[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Length: 238[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Type: application/xml | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 38a86ef9-0270-4ffc-a709-d68818fccdaa | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 38a86ef9-0270-4ffc-a709-d68818fccdaa | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:47 GMT | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Length: 238 | |
22/08/12 16:24:47 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:47 GMT | |
22/08/12 16:24:47 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 38a86ef9-0270-4ffc-a709-d68818fccdaa; S3 Extended Request ID: 38a86ef9-0270-4ffc-a709-d68818fccdaa; Proxy: null), S3 Extended Request ID: 38a86ef9-0270-4ffc-a709-d68818fccdaa | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 8 | |
22/08/12 16:24:47 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 38a86ef9-0270-4ffc-a709-d68818fccdaa; S3 Extended Request ID: 38a86ef9-0270-4ffc-a709-d68818fccdaa; Proxy: null), S3 Extended Request ID: 38a86ef9-0270-4ffc-a709-d68818fccdaa], RequestType=[GetObjectMetadataRequest], AWSRequestID=[38a86ef9-0270-4ffc-a709-d68818fccdaa], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[15.768], HttpClientSendRequestTime=[0.575], HttpRequestTime=[14.201], ApiCallLatency=[15.56], RequestSigningTime=[0.284], CredentialsRequestTime=[0.006, 0.003], HttpClientReceiveResponseTime=[12.65], | |
22/08/12 16:24:47 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8 | |
22/08/12 16:24:47 DEBUG Invoker: Starting: PUT 0-byte object | |
22/08/12 16:24:47 DEBUG S3AFileSystem: PUT 0 bytes to nonemptyprefix8/_temporary/0/ | |
22/08/12 16:24:47 DEBUG S3AFileSystem: PUT start 0 bytes | |
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter object_put_request by 1 with final value 1 | |
22/08/12 16:24:47 DEBUG request: Sending Request: PUT http://localhost:30600 /master.rando2/nonemptyprefix8/_temporary/0/ Headers: (amz-sdk-invocation-id: 798a1364-fafa-c571-a88f-423c786d6d8f, Content-Length: 0, Content-Type: application/x-directory, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 Canonical Request: '"PUT | |
/master.rando2/nonemptyprefix8/_temporary/0/ | |
amz-sdk-invocation-id:798a1364-fafa-c571-a88f-423c786d6d8f | |
amz-sdk-request:attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-length:86 | |
content-type:application/x-directory | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD | |
x-amz-date:20220812T152447Z | |
x-amz-decoded-content-length:0 | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length | |
STREAMING-AWS4-HMAC-SHA256-PAYLOAD" | |
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152447Z | |
20220812/us-east-1/s3/aws4_request | |
e148dd69e9e93d8a12a1c71aa0bda527f3f418edfce3dc8ddd0da013f327ca3a" | |
22/08/12 16:24:47 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:47 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:47 DEBUG MainClientExec: Executing request PUT /master.rando2/nonemptyprefix8/_temporary/0/ HTTP/1.1 | |
22/08/12 16:24:47 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> PUT /master.rando2/nonemptyprefix8/_temporary/0/ HTTP/1.1 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 798a1364-fafa-c571-a88f-423c786d6d8f | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-request: attempt=1;max=21 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=7d6cd2284b14c1caccea796a2a3f89b0bc7b3d3ae388a85eea793152e131fbc4 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Type: application/x-directory | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152447Z | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> x-amz-decoded-content-length: 0 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Length: 86 | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Expect: 100-continue | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "PUT /master.rando2/nonemptyprefix8/_temporary/0/ HTTP/1.1[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 798a1364-fafa-c571-a88f-423c786d6d8f[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=7d6cd2284b14c1caccea796a2a3f89b0bc7b3d3ae388a85eea793152e131fbc4[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Type: application/x-directory[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152447Z[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "x-amz-decoded-content-length: 0[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Length: 86[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Expect: 100-continue[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "HTTP/1.1 100 Continue[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << HTTP/1.1 100 Continue | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "0;chunk-signature=1417326b39cf805f273fb1b4b19e48c7a14f8b1e9acc0f1572f8b6f6b3d0c1c4[\r][\n]" | |
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Etag: "0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8"[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "X-Amz-Version-Id: 2d00fbfd57b840f0863941d59e62cdad[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:48 GMT[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Content-Length: 0[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Etag: "0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8" | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << X-Amz-Version-Id: 2d00fbfd57b840f0863941d59e62cdad | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:48 GMT | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Content-Length: 0 | |
22/08/12 16:24:48 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:48 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:48 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:48 GMT | |
22/08/12 16:24:48 DEBUG request: Received successful response: 200, AWS Request ID: null | |
22/08/12 16:24:48 DEBUG requestId: x-amzn-RequestId: not available | |
22/08/12 16:24:48 DEBUG requestId: AWS Request ID: not available | |
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 9 | |
22/08/12 16:24:48 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[PutObjectRequest], AWSRequestID=[null], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[0.315], ClientExecuteTime=[186.683], HttpClientSendRequestTime=[3.475], HttpRequestTime=[182.177], ApiCallLatency=[186.265], RequestSigningTime=[1.006], CredentialsRequestTime=[0.009, 0.005], HttpClientReceiveResponseTime=[177.452], | |
22/08/12 16:24:48 DEBUG S3AFileSystem: PUT completed success=true; 0 bytes | |
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter object_put_request_completed by 1 with final value 1 | |
22/08/12 16:24:48 DEBUG S3AFileSystem: Finished write to nonemptyprefix8/_temporary/0/, len 0. etag 0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8, version 2d00fbfd57b840f0863941d59e62cdad | |
22/08/12 16:24:48 DEBUG CallableSupplier: Starting: Waiting for task completion | |
22/08/12 16:24:48 DEBUG S3AFileSystem: Starting: Deleting 2 keys | |
22/08/12 16:24:48 DEBUG S3AFileSystem: Initiating delete operation for 2 objects | |
22/08/12 16:24:48 DEBUG S3AFileSystem: nonemptyprefix8/_temporary/ | |
22/08/12 16:24:48 DEBUG S3AFileSystem: nonemptyprefix8/ | |
22/08/12 16:24:48 DEBUG S3AFileSystem: Starting: DELETE 2 keys | |
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter object_bulk_delete_request by 1 with final value 1 | |
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter object_delete_objects by 2 with final value 2 | |
22/08/12 16:24:48 DEBUG request: Sending Request: POST http://localhost:30600 /master.rando2/ Parameters: ({"delete":[null]}Headers: (amz-sdk-invocation-id: f4f9232d-9938-70d8-c32c-9c611681ca83, Content-Length: 135, Content-MD5: xFbx26Nz2FGJIP9zZ3RhAg==, Content-Type: application/xml, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:48 DEBUG AWS4Signer: AWS4 Canonical Request: '"POST | |
/master.rando2/ | |
delete= | |
amz-sdk-invocation-id:f4f9232d-9938-70d8-c32c-9c611681ca83 | |
amz-sdk-request:ttl=20220812T152808Z;attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-length:135 | |
content-md5:xFbx26Nz2FGJIP9zZ3RhAg== | |
content-type:application/xml | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:16f9f85365f43ddbe703d6dd461ae2d8d9ef1785e41655b9c0d396a819924aed | |
x-amz-date:20220812T152448Z | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date | |
16f9f85365f43ddbe703d6dd461ae2d8d9ef1785e41655b9c0d396a819924aed" | |
22/08/12 16:24:48 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152448Z | |
20220812/us-east-1/s3/aws4_request | |
e99663c839db631169021f8ed01103d6016b2b9ea36cb07379745ab0ecafea7b" | |
22/08/12 16:24:48 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:48 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:48 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:48 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:48 DEBUG MainClientExec: Executing request POST /master.rando2/?delete HTTP/1.1 | |
22/08/12 16:24:48 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> POST /master.rando2/?delete HTTP/1.1 | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: f4f9232d-9938-70d8-c32c-9c611681ca83 | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152808Z;attempt=1;max=21 | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=73cf3673181055080f1d69875af40ac920a41fe7f1366c449457754ab3c59832 | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Content-MD5: xFbx26Nz2FGJIP9zZ3RhAg== | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Content-Type: application/xml | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: 16f9f85365f43ddbe703d6dd461ae2d8d9ef1785e41655b9c0d396a819924aed | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152448Z | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Content-Length: 135 | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "POST /master.rando2/?delete HTTP/1.1[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: f4f9232d-9938-70d8-c32c-9c611681ca83[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152808Z;attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=73cf3673181055080f1d69875af40ac920a41fe7f1366c449457754ab3c59832[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Content-MD5: xFbx26Nz2FGJIP9zZ3RhAg==[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: 16f9f85365f43ddbe703d6dd461ae2d8d9ef1785e41655b9c0d396a819924aed[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152448Z[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Content-Length: 135[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "<Delete><Quiet>true</Quiet><Object><Key>nonemptyprefix8/_temporary/</Key></Object><Object><Key>nonemptyprefix8/</Key></Object></Delete>" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 7d9090db-110a-4ee5-8c26-ef863774f9d4[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 7d9090db-110a-4ee5-8c26-ef863774f9d4[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:48 GMT[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Content-Length: 116[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"></DeleteResult>" | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Content-Type: application/xml | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 7d9090db-110a-4ee5-8c26-ef863774f9d4 | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 7d9090db-110a-4ee5-8c26-ef863774f9d4 | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:48 GMT | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Content-Length: 116 | |
22/08/12 16:24:48 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:48 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:48 GMT | |
22/08/12 16:24:48 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$DeleteObjectsHandler | |
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:48 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:48 DEBUG request: Received successful response: 200, AWS Request ID: 7d9090db-110a-4ee5-8c26-ef863774f9d4 | |
22/08/12 16:24:48 DEBUG requestId: x-amzn-RequestId: not available | |
22/08/12 16:24:48 DEBUG requestId: AWS Request ID: 7d9090db-110a-4ee5-8c26-ef863774f9d4 | |
22/08/12 16:24:48 DEBUG requestId: AWS Extended Request ID: 7d9090db-110a-4ee5-8c26-ef863774f9d4 | |
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 10 | |
22/08/12 16:24:48 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[DeleteObjectsRequest], AWSRequestID=[7d9090db-110a-4ee5-8c26-ef863774f9d4], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[2.522], ClientExecuteTime=[50.971], HttpClientSendRequestTime=[0.714], HttpRequestTime=[45.489], ApiCallLatency=[50.678], RequestSigningTime=[1.65], CredentialsRequestTime=[0.009, 0.004], HttpClientReceiveResponseTime=[43.856], | |
22/08/12 16:24:48 DEBUG S3AFileSystem: DELETE 2 keys: duration 0:00.122s | |
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter fake_directories_deleted by 2 with final value 2 | |
22/08/12 16:24:48 DEBUG S3AFileSystem: Deleting 2 keys: duration 0:00.125s | |
22/08/12 16:24:48 DEBUG CallableSupplier: Waiting for task completion: duration 0:00.126s | |
22/08/12 16:24:48 DEBUG Invoker: PUT 0-byte object : duration 0:00.323s | |
22/08/12 16:24:48 DEBUG Progress: PUT nonemptyprefix8/_temporary/0: 0 bytes | |
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter directories_created by 1 with final value 1 | |
22/08/12 16:24:48 DEBUG WholeStageCodegenExec: | |
/* 001 */ public Object generate(Object[] references) { | |
/* 002 */ return new GeneratedIteratorForCodegenStage1(references); | |
/* 003 */ } | |
/* 004 */ | |
/* 005 */ // codegenStageId=1 | |
/* 006 */ final class GeneratedIteratorForCodegenStage1 extends org.apache.spark.sql.execution.BufferedRowIterator { | |
/* 007 */ private Object[] references; | |
/* 008 */ private scala.collection.Iterator[] inputs; | |
/* 009 */ private scala.collection.Iterator rdd_input_0; | |
/* 010 */ private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[] rdd_mutableStateArray_0 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[1]; | |
/* 011 */ | |
/* 012 */ public GeneratedIteratorForCodegenStage1(Object[] references) { | |
/* 013 */ this.references = references; | |
/* 014 */ } | |
/* 015 */ | |
/* 016 */ public void init(int index, scala.collection.Iterator[] inputs) { | |
/* 017 */ partitionIndex = index; | |
/* 018 */ this.inputs = inputs; | |
/* 019 */ rdd_input_0 = inputs[0]; | |
/* 020 */ rdd_mutableStateArray_0[0] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(2, 0); | |
/* 021 */ | |
/* 022 */ } | |
/* 023 */ | |
/* 024 */ protected void processNext() throws java.io.IOException { | |
/* 025 */ while ( rdd_input_0.hasNext()) { | |
/* 026 */ InternalRow rdd_row_0 = (InternalRow) rdd_input_0.next(); | |
/* 027 */ ((org.apache.spark.sql.execution.metric.SQLMetric) references[0] /* numOutputRows */).add(1); | |
/* 028 */ boolean rdd_isNull_0 = rdd_row_0.isNullAt(0); | |
/* 029 */ long rdd_value_0 = rdd_isNull_0 ? | |
/* 030 */ -1L : (rdd_row_0.getLong(0)); | |
/* 031 */ boolean rdd_isNull_1 = rdd_row_0.isNullAt(1); | |
/* 032 */ double rdd_value_1 = rdd_isNull_1 ? | |
/* 033 */ -1.0 : (rdd_row_0.getDouble(1)); | |
/* 034 */ rdd_mutableStateArray_0[0].reset(); | |
/* 035 */ | |
/* 036 */ rdd_mutableStateArray_0[0].zeroOutNullBytes(); | |
/* 037 */ | |
/* 038 */ if (rdd_isNull_0) { | |
/* 039 */ rdd_mutableStateArray_0[0].setNullAt(0); | |
/* 040 */ } else { | |
/* 041 */ rdd_mutableStateArray_0[0].write(0, rdd_value_0); | |
/* 042 */ } | |
/* 043 */ | |
/* 044 */ if (rdd_isNull_1) { | |
/* 045 */ rdd_mutableStateArray_0[0].setNullAt(1); | |
/* 046 */ } else { | |
/* 047 */ rdd_mutableStateArray_0[0].write(1, rdd_value_1); | |
/* 048 */ } | |
/* 049 */ append((rdd_mutableStateArray_0[0].getRow())); | |
/* 050 */ if (shouldStop()) return; | |
/* 051 */ } | |
/* 052 */ } | |
/* 053 */ | |
/* 054 */ } | |
22/08/12 16:24:48 DEBUG CodeGenerator: | |
/* 001 */ public Object generate(Object[] references) { | |
/* 002 */ return new GeneratedIteratorForCodegenStage1(references); | |
/* 003 */ } | |
/* 004 */ | |
/* 005 */ // codegenStageId=1 | |
/* 006 */ final class GeneratedIteratorForCodegenStage1 extends org.apache.spark.sql.execution.BufferedRowIterator { | |
/* 007 */ private Object[] references; | |
/* 008 */ private scala.collection.Iterator[] inputs; | |
/* 009 */ private scala.collection.Iterator rdd_input_0; | |
/* 010 */ private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[] rdd_mutableStateArray_0 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[1]; | |
/* 011 */ | |
/* 012 */ public GeneratedIteratorForCodegenStage1(Object[] references) { | |
/* 013 */ this.references = references; | |
/* 014 */ } | |
/* 015 */ | |
/* 016 */ public void init(int index, scala.collection.Iterator[] inputs) { | |
/* 017 */ partitionIndex = index; | |
/* 018 */ this.inputs = inputs; | |
/* 019 */ rdd_input_0 = inputs[0]; | |
/* 020 */ rdd_mutableStateArray_0[0] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(2, 0); | |
/* 021 */ | |
/* 022 */ } | |
/* 023 */ | |
/* 024 */ protected void processNext() throws java.io.IOException { | |
/* 025 */ while ( rdd_input_0.hasNext()) { | |
/* 026 */ InternalRow rdd_row_0 = (InternalRow) rdd_input_0.next(); | |
/* 027 */ ((org.apache.spark.sql.execution.metric.SQLMetric) references[0] /* numOutputRows */).add(1); | |
/* 028 */ boolean rdd_isNull_0 = rdd_row_0.isNullAt(0); | |
/* 029 */ long rdd_value_0 = rdd_isNull_0 ? | |
/* 030 */ -1L : (rdd_row_0.getLong(0)); | |
/* 031 */ boolean rdd_isNull_1 = rdd_row_0.isNullAt(1); | |
/* 032 */ double rdd_value_1 = rdd_isNull_1 ? | |
/* 033 */ -1.0 : (rdd_row_0.getDouble(1)); | |
/* 034 */ rdd_mutableStateArray_0[0].reset(); | |
/* 035 */ | |
/* 036 */ rdd_mutableStateArray_0[0].zeroOutNullBytes(); | |
/* 037 */ | |
/* 038 */ if (rdd_isNull_0) { | |
/* 039 */ rdd_mutableStateArray_0[0].setNullAt(0); | |
/* 040 */ } else { | |
/* 041 */ rdd_mutableStateArray_0[0].write(0, rdd_value_0); | |
/* 042 */ } | |
/* 043 */ | |
/* 044 */ if (rdd_isNull_1) { | |
/* 045 */ rdd_mutableStateArray_0[0].setNullAt(1); | |
/* 046 */ } else { | |
/* 047 */ rdd_mutableStateArray_0[0].write(1, rdd_value_1); | |
/* 048 */ } | |
/* 049 */ append((rdd_mutableStateArray_0[0].getRow())); | |
/* 050 */ if (shouldStop()) return; | |
/* 051 */ } | |
/* 052 */ } | |
/* 053 */ | |
/* 054 */ } | |
22/08/12 16:24:48 INFO CodeGenerator: Code generated in 23.15445 ms | |
22/08/12 16:24:48 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$doExecute$4$adapted | |
22/08/12 16:24:48 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$doExecute$4$adapted) is now cleaned +++ | |
22/08/12 16:24:48 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$write$21 | |
22/08/12 16:24:48 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$write$21) is now cleaned +++ | |
22/08/12 16:24:48 INFO SparkContext: Starting job: save at NativeMethodAccessorImpl.java:0 | |
22/08/12 16:24:48 DEBUG DAGScheduler: eagerlyComputePartitionsForRddAndAncestors for RDD 8 took 0.000745 seconds | |
22/08/12 16:24:48 DEBUG DAGScheduler: Merging stage rdd profiles: Set() | |
22/08/12 16:24:48 INFO DAGScheduler: Got job 3 (save at NativeMethodAccessorImpl.java:0) with 1 output partitions | |
22/08/12 16:24:48 INFO DAGScheduler: Final stage: ResultStage 3 (save at NativeMethodAccessorImpl.java:0) | |
22/08/12 16:24:48 INFO DAGScheduler: Parents of final stage: List() | |
22/08/12 16:24:48 INFO DAGScheduler: Missing parents: List() | |
22/08/12 16:24:48 DEBUG DAGScheduler: submitStage(ResultStage 3 (name=save at NativeMethodAccessorImpl.java:0;jobs=3)) | |
22/08/12 16:24:48 DEBUG DAGScheduler: missing: List() | |
22/08/12 16:24:48 INFO DAGScheduler: Submitting ResultStage 3 (CoalescedRDD[8] at save at NativeMethodAccessorImpl.java:0), which has no missing parents | |
22/08/12 16:24:48 DEBUG DAGScheduler: submitMissingTasks(ResultStage 3) | |
22/08/12 16:24:48 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 215.2 KiB, free 434.1 MiB) | |
22/08/12 16:24:48 DEBUG BlockManager: Put block broadcast_3 locally took 1 ms | |
22/08/12 16:24:48 DEBUG BlockManager: Putting block broadcast_3 without replication took 1 ms | |
22/08/12 16:24:48 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 77.9 KiB, free 434.1 MiB) | |
22/08/12 16:24:48 DEBUG BlockManagerMasterEndpoint: Updating block info on master broadcast_3_piece0 for BlockManagerId(driver, 10.1.255.235, 38343, None) | |
22/08/12 16:24:48 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 10.1.255.235:38343 (size: 77.9 KiB, free: 434.3 MiB) | |
22/08/12 16:24:48 DEBUG BlockManagerMaster: Updated info of block broadcast_3_piece0 | |
22/08/12 16:24:48 DEBUG BlockManager: Told master about block broadcast_3_piece0 | |
22/08/12 16:24:48 DEBUG BlockManager: Put block broadcast_3_piece0 locally took 2 ms | |
22/08/12 16:24:48 DEBUG BlockManager: Putting block broadcast_3_piece0 without replication took 2 ms | |
22/08/12 16:24:48 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1513 | |
22/08/12 16:24:48 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 3 (CoalescedRDD[8] at save at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0)) | |
22/08/12 16:24:48 INFO TaskSchedulerImpl: Adding task set 3.0 with 1 tasks resource profile 0 | |
22/08/12 16:24:48 DEBUG TaskSetManager: Epoch for TaskSet 3.0: 0 | |
22/08/12 16:24:48 DEBUG TaskSetManager: Adding pending tasks took 0 ms | |
22/08/12 16:24:48 DEBUG TaskSetManager: Valid locality levels for TaskSet 3.0: NO_PREF, ANY | |
22/08/12 16:24:48 DEBUG TaskSchedulerImpl: parentName: , name: TaskSet_3.0, runningTasks: 0 | |
22/08/12 16:24:48 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 16) (10.1.255.235, executor driver, partition 0, PROCESS_LOCAL, 5295 bytes) taskResourceAssignments Map() | |
22/08/12 16:24:48 DEBUG TaskSetManager: No tasks for locality level NO_PREF, so moving to locality level ANY | |
22/08/12 16:24:48 INFO Executor: Running task 0.0 in stage 3.0 (TID 16) | |
22/08/12 16:24:48 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 1 | |
22/08/12 16:24:48 DEBUG BlockManager: Getting local block broadcast_3 | |
22/08/12 16:24:48 DEBUG BlockManager: Level for block broadcast_3 is StorageLevel(disk, memory, deserialized, 1 replicas) | |
22/08/12 16:24:48 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121624485385779855500771815_0003}; taskId=attempt_202208121624485385779855500771815_0003_m_000000_16, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@57e9a743}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix8 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121624485385779855500771815_0003}; taskId=attempt_202208121624485385779855500771815_0003_m_000000_16, status=''} | |
22/08/12 16:24:48 INFO FileOutputCommitter: File Output Committer Algorithm version is 1 | |
22/08/12 16:24:48 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false | |
22/08/12 16:24:48 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter | |
22/08/12 16:24:48 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121624485385779855500771815_0003}; taskId=attempt_202208121624485385779855500771815_0003_m_000000_16, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@3da9886}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix8 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121624485385779855500771815_0003}; taskId=attempt_202208121624485385779855500771815_0003_m_000000_16, status=''} | |
22/08/12 16:24:48 INFO FileOutputCommitter: File Output Committer Algorithm version is 1 | |
22/08/12 16:24:48 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false | |
22/08/12 16:24:48 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter | |
22/08/12 16:24:48 INFO CodecConfig: Compression: SNAPPY | |
22/08/12 16:24:48 INFO CodecConfig: Compression: SNAPPY | |
22/08/12 16:24:48 INFO ParquetOutputFormat: Parquet block size to 134217728 | |
22/08/12 16:24:48 INFO ParquetOutputFormat: Validation is off | |
22/08/12 16:24:48 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes | |
22/08/12 16:24:48 INFO ParquetOutputFormat: Parquet properties are: | |
Parquet page size to 1048576 | |
Parquet dictionary page size to 1048576 | |
Dictionary is true | |
Writer version is: PARQUET_1_0 | |
Page size checking is: estimated | |
Min row count for page size check is: 100 | |
Max row count for page size check is: 10000 | |
Truncate length for column indexes is: 64 | |
Truncate length for statistics min/max is: 2147483647 | |
Bloom filter enabled: false | |
Max Bloom filter size for a column is 1048576 | |
Bloom filter expected number of distinct values are: null | |
Page row count limit to 20000 | |
Writing page checksums is: on | |
22/08/12 16:24:48 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema: | |
{ | |
"type" : "struct", | |
"fields" : [ { | |
"name" : "a", | |
"type" : "long", | |
"nullable" : true, | |
"metadata" : { } | |
}, { | |
"name" : "b", | |
"type" : "double", | |
"nullable" : true, | |
"metadata" : { } | |
} ] | |
} | |
and corresponding Parquet message type: | |
message spark_schema { | |
optional int64 a; | |
optional double b; | |
} | |
22/08/12 16:24:48 DEBUG EncryptionPropertiesFactory: EncryptionPropertiesFactory is not configured - name not found in hadoop config | |
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter op_create by 1 with final value 1 | |
22/08/12 16:24:48 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet (nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet); needEmptyDirectory=false | |
22/08/12 16:24:48 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet | |
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 5 | |
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 5 | |
22/08/12 16:24:48 DEBUG S3AFileSystem: HEAD nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet with change tracker null | |
22/08/12 16:24:48 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet Headers: (amz-sdk-invocation-id: 932d78b1-791a-d543-8013-eb2cf915725f, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:48 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD | |
/master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet | |
amz-sdk-invocation-id:932d78b1-791a-d543-8013-eb2cf915725f | |
amz-sdk-request:ttl=20220812T152808Z;attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-type:application/octet-stream | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
x-amz-date:20220812T152448Z | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date | |
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | |
22/08/12 16:24:48 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152448Z | |
20220812/us-east-1/s3/aws4_request | |
864a04a69c8c070f0d465e3922bd7968cdb900b53e289ac3f5b60d031fd148b8" | |
22/08/12 16:24:48 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:48 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:48 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:48 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:48 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet HTTP/1.1 | |
22/08/12 16:24:48 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet HTTP/1.1 | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 932d78b1-791a-d543-8013-eb2cf915725f | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152808Z;attempt=1;max=21 | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=8289e572a608af7c1197644d4efd48b52f8f8bcf7410daa22943c2a5d202e22d | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152448Z | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet HTTP/1.1[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 932d78b1-791a-d543-8013-eb2cf915725f[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152808Z;attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=8289e572a608af7c1197644d4efd48b52f8f8bcf7410daa22943c2a5d202e22d[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152448Z[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 4be2e478-732d-4a97-87e9-cf2ac708c76a[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 4be2e478-732d-4a97-87e9-cf2ac708c76a[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:48 GMT[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Content-Length: 389[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Content-Type: application/xml | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 4be2e478-732d-4a97-87e9-cf2ac708c76a | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 4be2e478-732d-4a97-87e9-cf2ac708c76a | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:48 GMT | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Content-Length: 389 | |
22/08/12 16:24:48 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:48 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:48 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:48 GMT | |
22/08/12 16:24:48 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 4be2e478-732d-4a97-87e9-cf2ac708c76a; S3 Extended Request ID: 4be2e478-732d-4a97-87e9-cf2ac708c76a; Proxy: null), S3 Extended Request ID: 4be2e478-732d-4a97-87e9-cf2ac708c76a | |
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 11 | |
22/08/12 16:24:48 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 4be2e478-732d-4a97-87e9-cf2ac708c76a; S3 Extended Request ID: 4be2e478-732d-4a97-87e9-cf2ac708c76a; Proxy: null), S3 Extended Request ID: 4be2e478-732d-4a97-87e9-cf2ac708c76a], RequestType=[GetObjectMetadataRequest], AWSRequestID=[4be2e478-732d-4a97-87e9-cf2ac708c76a], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[20.796], HttpClientSendRequestTime=[0.85], HttpRequestTime=[18.873], ApiCallLatency=[20.476], RequestSigningTime=[0.643], CredentialsRequestTime=[0.01, 0.009], HttpClientReceiveResponseTime=[16.92], | |
22/08/12 16:24:48 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet/ delimiter=/ keys=2 requester pays=false | |
22/08/12 16:24:48 DEBUG S3AFileSystem: Starting: LIST | |
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 5 | |
22/08/12 16:24:48 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 2abe7ab3-f0a3-f96d-99bb-99cce65be5bc, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:48 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET | |
/master.rando2/ | |
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2Fpart-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet%2F | |
amz-sdk-invocation-id:2abe7ab3-f0a3-f96d-99bb-99cce65be5bc | |
amz-sdk-request:ttl=20220812T152808Z;attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-type:application/octet-stream | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
x-amz-date:20220812T152448Z | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date | |
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | |
22/08/12 16:24:48 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152448Z | |
20220812/us-east-1/s3/aws4_request | |
3f30b94b744865d8f0e7128a4e36531e3ddc20b660b5749c7627ed39f44b5c8d" | |
22/08/12 16:24:48 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:48 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:48 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:48 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:48 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2Fpart-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1 | |
22/08/12 16:24:48 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2Fpart-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1 | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 2abe7ab3-f0a3-f96d-99bb-99cce65be5bc | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152808Z;attempt=1;max=21 | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=23a08d83bb0abb3cc211e6589dac3c2c1fef887550c2906583b1c797c612eec9 | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152448Z | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Content-Length: 0 | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2Fpart-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 2abe7ab3-f0a3-f96d-99bb-99cce65be5bc[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152808Z;attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=23a08d83bb0abb3cc211e6589dac3c2c1fef887550c2906583b1c797c612eec9[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152448Z[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 7e04903a-9d8d-4db3-8b5d-ceeffb51682d[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 7e04903a-9d8d-4db3-8b5d-ceeffb51682d[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:48 GMT[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Content-Length: 427[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]" | |
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet/</Prefix></ListBucketResult>" | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Content-Type: application/xml | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 7e04903a-9d8d-4db3-8b5d-ceeffb51682d | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 7e04903a-9d8d-4db3-8b5d-ceeffb51682d | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:48 GMT | |
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Content-Length: 427 | |
22/08/12 16:24:48 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:48 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:48 GMT | |
22/08/12 16:24:48 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler | |
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:48 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:48 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler | |
22/08/12 16:24:48 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2 | |
22/08/12 16:24:48 DEBUG request: Received successful response: 200, AWS Request ID: 7e04903a-9d8d-4db3-8b5d-ceeffb51682d | |
22/08/12 16:24:48 DEBUG requestId: x-amzn-RequestId: not available | |
22/08/12 16:24:48 DEBUG requestId: AWS Request ID: 7e04903a-9d8d-4db3-8b5d-ceeffb51682d | |
22/08/12 16:24:48 DEBUG requestId: AWS Extended Request ID: 7e04903a-9d8d-4db3-8b5d-ceeffb51682d | |
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 12 | |
22/08/12 16:24:48 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[7e04903a-9d8d-4db3-8b5d-ceeffb51682d], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[32.111], ClientExecuteTime=[54.843], HttpClientSendRequestTime=[0.686], HttpRequestTime=[21.041], ApiCallLatency=[54.551], RequestSigningTime=[0.454], CredentialsRequestTime=[0.008, 0.005], HttpClientReceiveResponseTime=[19.491], | |
22/08/12 16:24:48 DEBUG S3AFileSystem: LIST: duration 0:00.055s | |
22/08/12 16:24:48 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet | |
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter files_created by 1 with final value 1 | |
22/08/12 16:24:48 DEBUG S3ARetryPolicy: Retrying on recoverable AWS failures 7 times with an initial interval of 500ms | |
22/08/12 16:24:48 DEBUG S3ABlockOutputStream: Initialized S3ABlockOutputStream for nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet output to FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-8560763669495856341.tmp, state=Writing, dataSize=0, limit=67108864} | |
22/08/12 16:24:48 DEBUG ParquetFileWriter: 0: start | |
22/08/12 16:24:48 DEBUG MemoryManager: Allocated total memory pool is: 1020054720 | |
22/08/12 16:24:48 INFO CodecPool: Got brand-new compressor [.snappy] | |
22/08/12 16:24:48 DEBUG RunLengthBitPackingHybridEncoder: Encoding: RunLengthBitPackingHybridEncoder with bithWidth: 1 initialCapacity 64 | |
22/08/12 16:24:48 DEBUG CapacityByteArrayOutputStream: initial slab of size 64 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(36) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 36 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 36 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(21) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 21 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 21 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(33) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 33 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 33 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(1) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 1 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 1 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(41) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 41 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 41 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(85) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 85 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 85 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(59) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 59 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 59 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(58) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 58 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 58 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(15) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 15 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 15 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanBroadcast(0) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning broadcast 0 | |
22/08/12 16:24:48 DEBUG TorrentBroadcast: Unpersisting TorrentBroadcast 0 | |
22/08/12 16:24:48 DEBUG BlockManagerStorageEndpoint: removing broadcast 0 | |
22/08/12 16:24:48 DEBUG CapacityByteArrayOutputStream: initial slab of size 1024 | |
22/08/12 16:24:48 DEBUG BlockManager: Removing broadcast 0 | |
22/08/12 16:24:48 DEBUG BlockManager: Removing block broadcast_0_piece0 | |
22/08/12 16:24:48 DEBUG MemoryStore: Block broadcast_0_piece0 of size 6705 dropped from memory (free 455149761) | |
22/08/12 16:24:48 DEBUG BlockManagerMasterEndpoint: Updating block info on master broadcast_0_piece0 for BlockManagerId(driver, 10.1.255.235, 38343, None) | |
22/08/12 16:24:48 INFO BlockManagerInfo: Removed broadcast_0_piece0 on 10.1.255.235:38343 in memory (size: 6.5 KiB, free: 434.3 MiB) | |
22/08/12 16:24:48 DEBUG BlockManagerMaster: Updated info of block broadcast_0_piece0 | |
22/08/12 16:24:48 DEBUG BlockManager: Told master about block broadcast_0_piece0 | |
22/08/12 16:24:48 DEBUG BlockManager: Removing block broadcast_0 | |
22/08/12 16:24:48 DEBUG MemoryStore: Block broadcast_0 of size 12720 dropped from memory (free 455162481) | |
22/08/12 16:24:48 DEBUG BlockManagerStorageEndpoint: Done removing broadcast 0, response is 0 | |
22/08/12 16:24:48 DEBUG BlockManagerStorageEndpoint: Sent response: 0 to 10.1.255.235:45909 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned broadcast 0 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(13) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 13 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 13 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(73) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 73 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 73 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(11) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 11 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 11 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(71) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 71 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 71 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(28) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 28 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 28 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(48) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 48 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 48 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(75) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 75 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 75 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanBroadcast(2) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning broadcast 2 | |
22/08/12 16:24:48 DEBUG TorrentBroadcast: Unpersisting TorrentBroadcast 2 | |
22/08/12 16:24:48 DEBUG RunLengthBitPackingHybridEncoder: Encoding: RunLengthBitPackingHybridEncoder with bithWidth: 1 initialCapacity 64 | |
22/08/12 16:24:48 DEBUG CapacityByteArrayOutputStream: initial slab of size 64 | |
22/08/12 16:24:48 DEBUG CapacityByteArrayOutputStream: initial slab of size 1024 | |
22/08/12 16:24:48 DEBUG BlockManagerStorageEndpoint: removing broadcast 2 | |
22/08/12 16:24:48 DEBUG BlockManager: Removing broadcast 2 | |
22/08/12 16:24:48 DEBUG BlockManager: Removing block broadcast_2_piece0 | |
22/08/12 16:24:48 DEBUG MemoryStore: Block broadcast_2_piece0 of size 6713 dropped from memory (free 455169194) | |
22/08/12 16:24:48 DEBUG BlockManagerMasterEndpoint: Updating block info on master broadcast_2_piece0 for BlockManagerId(driver, 10.1.255.235, 38343, None) | |
22/08/12 16:24:48 INFO BlockManagerInfo: Removed broadcast_2_piece0 on 10.1.255.235:38343 in memory (size: 6.6 KiB, free: 434.3 MiB) | |
22/08/12 16:24:48 DEBUG BlockManagerMaster: Updated info of block broadcast_2_piece0 | |
22/08/12 16:24:48 DEBUG BlockManager: Told master about block broadcast_2_piece0 | |
22/08/12 16:24:48 DEBUG BlockManager: Removing block broadcast_2 | |
22/08/12 16:24:48 DEBUG MemoryStore: Block broadcast_2 of size 12720 dropped from memory (free 455181914) | |
22/08/12 16:24:48 DEBUG BlockManagerStorageEndpoint: Done removing broadcast 2, response is 0 | |
22/08/12 16:24:48 DEBUG BlockManagerStorageEndpoint: Sent response: 0 to 10.1.255.235:45909 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned broadcast 2 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(76) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 76 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 76 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(77) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 77 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 77 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(16) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 16 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 16 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(26) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 26 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 26 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(83) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 83 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 83 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(7) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 7 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 7 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(44) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 44 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 44 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(82) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 82 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 82 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(50) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 50 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 50 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(66) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 66 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 66 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(43) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 43 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 43 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(68) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 68 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 68 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(17) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 17 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 17 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(78) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 78 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 78 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(60) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 60 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 60 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(57) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 57 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 57 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(23) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 23 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 23 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(72) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 72 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 72 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(12) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 12 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 12 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(46) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 46 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 46 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(18) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 18 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 18 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(2) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 2 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 2 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(5) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 5 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 5 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(49) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 49 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 49 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(25) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 25 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 25 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(14) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 14 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 14 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(19) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 19 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 19 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(6) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 6 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 6 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(4) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 4 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 4 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(62) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 62 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 62 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(38) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 38 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 38 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(35) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 35 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 35 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(22) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 22 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 22 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(45) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 45 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 45 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(47) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 47 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 47 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(20) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 20 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 20 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(70) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 70 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 70 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(64) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 64 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 64 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(3) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 3 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 3 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(32) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 32 | |
22/08/12 16:24:48 DEBUG MemoryManager: Adjust block size from 134,217,728 to 134,217,728 for writer: org.apache.parquet.hadoop.InternalParquetRecordWriter@40045bfe | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 32 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(61) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 61 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 61 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(87) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 87 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 87 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(51) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 51 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 51 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(29) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 29 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 29 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(74) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 74 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 74 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(39) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 39 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 39 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(27) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 27 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 27 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(10) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 10 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 10 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(30) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 30 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 30 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(40) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 40 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 40 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(54) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 54 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 54 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(86) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 86 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 86 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(67) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 67 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 67 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(56) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 56 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 56 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(81) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 81 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 81 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(24) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 24 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 24 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(53) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 53 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 53 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(55) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 55 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 55 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(31) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 31 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 31 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(69) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 69 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 69 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(65) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 65 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 65 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(42) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 42 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 42 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(52) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 52 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 52 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(34) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 34 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 34 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(8) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 8 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 8 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanBroadcast(1) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning broadcast 1 | |
22/08/12 16:24:48 DEBUG TorrentBroadcast: Unpersisting TorrentBroadcast 1 | |
22/08/12 16:24:48 DEBUG BlockManagerStorageEndpoint: removing broadcast 1 | |
22/08/12 16:24:48 DEBUG BlockManager: Removing broadcast 1 | |
22/08/12 16:24:48 DEBUG BlockManager: Removing block broadcast_1 | |
22/08/12 16:24:48 DEBUG MemoryStore: Block broadcast_1 of size 12720 dropped from memory (free 455194634) | |
22/08/12 16:24:48 DEBUG BlockManager: Removing block broadcast_1_piece0 | |
22/08/12 16:24:48 DEBUG MemoryStore: Block broadcast_1_piece0 of size 6711 dropped from memory (free 455201345) | |
22/08/12 16:24:48 DEBUG BlockManagerMasterEndpoint: Updating block info on master broadcast_1_piece0 for BlockManagerId(driver, 10.1.255.235, 38343, None) | |
22/08/12 16:24:48 INFO BlockManagerInfo: Removed broadcast_1_piece0 on 10.1.255.235:38343 in memory (size: 6.6 KiB, free: 434.3 MiB) | |
22/08/12 16:24:48 DEBUG BlockManagerMaster: Updated info of block broadcast_1_piece0 | |
22/08/12 16:24:48 DEBUG BlockManager: Told master about block broadcast_1_piece0 | |
22/08/12 16:24:48 DEBUG BlockManagerStorageEndpoint: Done removing broadcast 1, response is 0 | |
22/08/12 16:24:48 DEBUG BlockManagerStorageEndpoint: Sent response: 0 to 10.1.255.235:45909 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned broadcast 1 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(80) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 80 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 80 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(63) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 63 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 63 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(79) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 79 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 79 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(9) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 9 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 9 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(37) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 37 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 37 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(84) | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 84 | |
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 84 | |
22/08/12 16:24:48 INFO PythonRunner: Times: total = 50, boot = -2330, init = 2379, finish = 1 | |
22/08/12 16:24:48 INFO PythonRunner: Times: total = 51, boot = -2362, init = 2413, finish = 0 | |
22/08/12 16:24:49 INFO PythonRunner: Times: total = 50, boot = -2419, init = 2469, finish = 0 | |
22/08/12 16:24:49 INFO PythonRunner: Times: total = 51, boot = -2467, init = 2518, finish = 0 | |
22/08/12 16:24:49 INFO PythonRunner: Times: total = 49, boot = -2490, init = 2539, finish = 0 | |
22/08/12 16:24:49 INFO PythonRunner: Times: total = 49, boot = -2533, init = 2582, finish = 0 | |
22/08/12 16:24:49 INFO PythonRunner: Times: total = 49, boot = -2594, init = 2643, finish = 0 | |
22/08/12 16:24:49 INFO PythonRunner: Times: total = 46, boot = -2635, init = 2681, finish = 0 | |
22/08/12 16:24:49 INFO PythonRunner: Times: total = 47, boot = -2676, init = 2723, finish = 0 | |
22/08/12 16:24:49 INFO PythonRunner: Times: total = 50, boot = -2721, init = 2771, finish = 0 | |
22/08/12 16:24:49 INFO PythonRunner: Times: total = 52, boot = -2774, init = 2826, finish = 0 | |
22/08/12 16:24:49 INFO PythonRunner: Times: total = 75, boot = -502, init = 577, finish = 0 | |
22/08/12 16:24:49 INFO PythonRunner: Times: total = 50, boot = -526, init = 576, finish = 0 | |
22/08/12 16:24:49 INFO PythonRunner: Times: total = 55, boot = -526, init = 581, finish = 0 | |
22/08/12 16:24:49 INFO PythonRunner: Times: total = 83, boot = -531, init = 614, finish = 0 | |
22/08/12 16:24:49 DEBUG RecordConsumerLoggingWrapper: <!-- start message --> | |
22/08/12 16:24:49 DEBUG MessageColumnIO: < MESSAGE START > | |
22/08/12 16:24:49 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={}}: [] r:0 | |
22/08/12 16:24:49 DEBUG RecordConsumerLoggingWrapper: <a> | |
22/08/12 16:24:49 DEBUG MessageColumnIO: startField(a, 0) | |
22/08/12 16:24:49 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={}}: [a] r:0 | |
22/08/12 16:24:49 DEBUG RecordConsumerLoggingWrapper: 1 | |
22/08/12 16:24:49 DEBUG MessageColumnIO: addLong(1) | |
22/08/12 16:24:49 DEBUG MessageColumnIO: r: 0 | |
22/08/12 16:24:49 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={}}: [a] r:0 | |
22/08/12 16:24:49 DEBUG RecordConsumerLoggingWrapper: </a> | |
22/08/12 16:24:49 DEBUG MessageColumnIO: endField(a, 0) | |
22/08/12 16:24:49 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={0}}: [] r:0 | |
22/08/12 16:24:49 DEBUG RecordConsumerLoggingWrapper: <b> | |
22/08/12 16:24:49 DEBUG MessageColumnIO: startField(b, 1) | |
22/08/12 16:24:49 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={0}}: [b] r:0 | |
22/08/12 16:24:49 DEBUG RecordConsumerLoggingWrapper: 2.0 | |
22/08/12 16:24:49 DEBUG MessageColumnIO: addDouble(2.0) | |
22/08/12 16:24:49 DEBUG MessageColumnIO: r: 0 | |
22/08/12 16:24:49 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={0}}: [b] r:0 | |
22/08/12 16:24:49 DEBUG RecordConsumerLoggingWrapper: </b> | |
22/08/12 16:24:49 DEBUG MessageColumnIO: endField(b, 1) | |
22/08/12 16:24:49 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={0, 1}}: [] r:0 | |
22/08/12 16:24:49 DEBUG MessageColumnIO: < MESSAGE END > | |
22/08/12 16:24:49 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={0, 1}}: [] r:0 | |
22/08/12 16:24:49 DEBUG RecordConsumerLoggingWrapper: <!-- end message --> | |
22/08/12 16:24:49 INFO PythonRunner: Times: total = 51, boot = -563, init = 614, finish = 0 | |
22/08/12 16:24:49 DEBUG RecordConsumerLoggingWrapper: <!-- flush --> | |
22/08/12 16:24:49 DEBUG InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 24 | |
22/08/12 16:24:49 DEBUG ParquetFileWriter: 4: start block | |
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: used 0 slabs, adding new slab of size 64 | |
22/08/12 16:24:49 DEBUG DictionaryValuesWriter: max dic id 0 | |
22/08/12 16:24:49 DEBUG RunLengthBitPackingHybridEncoder: Encoding: RunLengthBitPackingHybridEncoder with bithWidth: 0 initialCapacity 1024 | |
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: initial slab of size 1024 | |
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: used 0 slabs, adding new slab of size 1024 | |
22/08/12 16:24:49 DEBUG DictionaryValuesWriter: rle encoded bytes 1 | |
22/08/12 16:24:49 DEBUG BytesInput: BytesInput from array of 1 bytes | |
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: used 0 slabs, adding new slab of size 1024 | |
22/08/12 16:24:49 DEBUG PlainValuesWriter: writing a buffer of size 8 | |
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 0 bytes to out | |
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 6 bytes to out | |
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: { | |
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 4 bytes to out | |
22/08/12 16:24:49 DEBUG BytesUtils: write le int: 2 => 2 0 0 0 | |
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 2 bytes to out | |
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: } | |
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 8 bytes to out | |
22/08/12 16:24:49 DEBUG BytesInput: converted 16 to byteArray of 16 bytes | |
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 23 bytes to out | |
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 16 bytes to out | |
22/08/12 16:24:49 DEBUG BytesInput: converted 39 to byteArray of 39 bytes | |
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: initial slab of size 64 | |
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: initial slab of size 1024 | |
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: used 0 slabs, adding new slab of size 64 | |
22/08/12 16:24:49 DEBUG DictionaryValuesWriter: max dic id 0 | |
22/08/12 16:24:49 DEBUG RunLengthBitPackingHybridEncoder: Encoding: RunLengthBitPackingHybridEncoder with bithWidth: 0 initialCapacity 1024 | |
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: initial slab of size 1024 | |
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: used 0 slabs, adding new slab of size 1024 | |
22/08/12 16:24:49 DEBUG DictionaryValuesWriter: rle encoded bytes 1 | |
22/08/12 16:24:49 DEBUG BytesInput: BytesInput from array of 1 bytes | |
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: used 0 slabs, adding new slab of size 1024 | |
22/08/12 16:24:49 DEBUG PlainValuesWriter: writing a buffer of size 8 | |
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 0 bytes to out | |
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 6 bytes to out | |
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: { | |
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 4 bytes to out | |
22/08/12 16:24:49 DEBUG BytesUtils: write le int: 2 => 2 0 0 0 | |
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 2 bytes to out | |
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: } | |
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 8 bytes to out | |
22/08/12 16:24:49 DEBUG BytesInput: converted 16 to byteArray of 16 bytes | |
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 23 bytes to out | |
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 16 bytes to out | |
22/08/12 16:24:49 DEBUG BytesInput: converted 39 to byteArray of 39 bytes | |
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: initial slab of size 64 | |
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: initial slab of size 1024 | |
22/08/12 16:24:49 DEBUG ParquetFileWriter: 4: write data pages | |
22/08/12 16:24:49 DEBUG ParquetFileWriter: 4: write data pages content | |
22/08/12 16:24:49 DEBUG ParquetFileWriter: 43: end column | |
22/08/12 16:24:49 DEBUG ColumnChunkPageWriteStore: written 39B for [a] optional int64 a: 1 values, 14B raw, 16B comp, 1 pages, encodings: [PLAIN] | |
22/08/12 16:24:49 DEBUG ParquetFileWriter: 43: write data pages | |
22/08/12 16:24:49 DEBUG ParquetFileWriter: 43: write data pages content | |
22/08/12 16:24:49 DEBUG ParquetFileWriter: 82: end column | |
22/08/12 16:24:49 DEBUG ColumnChunkPageWriteStore: written 39B for [b] optional double b: 1 values, 14B raw, 16B comp, 1 pages, encodings: [PLAIN] | |
22/08/12 16:24:49 DEBUG ParquetFileWriter: 82: end block | |
22/08/12 16:24:49 DEBUG ParquetFileWriter: 82: column indexes | |
22/08/12 16:24:49 DEBUG ParquetFileWriter: 144: offset indexes | |
22/08/12 16:24:49 DEBUG ParquetFileWriter: 164: bloom filters | |
22/08/12 16:24:49 DEBUG ParquetFileWriter: 164: end | |
22/08/12 16:24:49 DEBUG ParquetFileWriter: 710: footer length = 546 | |
22/08/12 16:24:49 DEBUG BytesUtils: write le int: 546 => 34 2 0 0 | |
22/08/12 16:24:49 DEBUG S3ABlockOutputStream: S3ABlockOutputStream{WriteOperationHelper {bucket=master.rando2}, blockSize=67108864, activeBlock=FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-8560763669495856341.tmp, state=Writing, dataSize=718, limit=67108864} Statistics=counters=((action_executor_acquired=0) (stream_write_bytes=718) (stream_write_block_uploads=0) (stream_write_exceptions_completing_upload=0) (object_multipart_aborted.failures=0) (multipart_upload_completed=0) (op_hsync=0) (stream_write_queue_duration=0) (op_abort=0) (stream_write_total_time=0) (op_hflush=0) (object_multipart_aborted=0) (stream_write_exceptions=0) (stream_write_total_data=0) (action_executor_acquired.failures=0) (op_abort.failures=0) (multipart_upload_completed.failures=0)); | |
gauges=((stream_write_block_uploads_pending=0) (stream_write_block_uploads_data_pending=0)); | |
minimums=((op_abort.min=-1) (action_executor_acquired.failures.min=-1) (action_executor_acquired.min=-1) (object_multipart_aborted.failures.min=-1) (object_multipart_aborted.min=-1) (multipart_upload_completed.failures.min=-1) (multipart_upload_completed.min=-1) (op_abort.failures.min=-1)); | |
maximums=((op_abort.max=-1) (object_multipart_aborted.max=-1) (op_abort.failures.max=-1) (object_multipart_aborted.failures.max=-1) (multipart_upload_completed.failures.max=-1) (action_executor_acquired.failures.max=-1) (multipart_upload_completed.max=-1) (action_executor_acquired.max=-1)); | |
means=((multipart_upload_completed.mean=(samples=0, sum=0, mean=0.0000)) (multipart_upload_completed.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_aborted.failures.mean=(samples=0, sum=0, mean=0.0000)) (action_executor_acquired.mean=(samples=0, sum=0, mean=0.0000)) (op_abort.mean=(samples=0, sum=0, mean=0.0000)) (op_abort.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_aborted.mean=(samples=0, sum=0, mean=0.0000)) (action_executor_acquired.failures.mean=(samples=0, sum=0, mean=0.0000))); | |
}: Closing block #1: current block= FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-8560763669495856341.tmp, state=Writing, dataSize=718, limit=67108864} | |
22/08/12 16:24:49 DEBUG S3ABlockOutputStream: Executing regular upload for WriteOperationHelper {bucket=master.rando2} | |
22/08/12 16:24:49 DEBUG S3ADataBlocks: Start datablock[1] upload | |
22/08/12 16:24:49 DEBUG S3ADataBlocks: FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-8560763669495856341.tmp, state=Writing, dataSize=718, limit=67108864}: entering state Upload | |
22/08/12 16:24:49 DEBUG IOStatisticsStoreImpl: Incrementing counter stream_write_block_uploads by 1 with final value 1 | |
22/08/12 16:24:49 DEBUG S3ABlockOutputStream: Clearing active block | |
22/08/12 16:24:49 DEBUG Invoker: Starting: Writing Object | |
22/08/12 16:24:49 DEBUG S3AFileSystem: PUT 718 bytes to nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet | |
22/08/12 16:24:49 DEBUG S3AFileSystem: PUT start 718 bytes | |
22/08/12 16:24:49 DEBUG IOStatisticsStoreImpl: Incrementing counter object_put_request by 1 with final value 2 | |
22/08/12 16:24:49 DEBUG Mimetypes: Loading mime types from file in the classpath: mime.types | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/activemessage' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ez' to 'application/andrew-inset' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/applefile' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'atom' to 'application/atom+xml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/atomicmail' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/batch-smtp' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/beep+xml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/cals-1840' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/cnrp+xml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/commonground' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/cpl+xml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/cybercash' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/dca-rft' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/dec-dx' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/dvcs' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/edi-consent' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/edifact' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/edi-x12' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/eshop' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/font-tdpfr' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/http' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/hyperstudio' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/iges' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/index' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/index.cmd' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/index.obj' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/index.response' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/index.vnd' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/iotp' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/ipp' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/isup' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'json' to 'application/json' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'hqx' to 'application/mac-binhex40' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'cpt' to 'application/mac-compactpro' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/macwriteii' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/marc' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/mathematica' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mathml' to 'application/mathml+xml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'doc' to 'application/msword' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/news-message-id' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/news-transmission' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/ocsp-request' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/ocsp-response' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'bin' to 'application/octet-stream' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'dms' to 'application/octet-stream' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'lha' to 'application/octet-stream' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'lzh' to 'application/octet-stream' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'exe' to 'application/octet-stream' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'class' to 'application/octet-stream' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'so' to 'application/octet-stream' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'dll' to 'application/octet-stream' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'dmg' to 'application/octet-stream' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'oda' to 'application/oda' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ogg' to 'application/ogg' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/parityfec' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'pdf' to 'application/pdf' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pgp-encrypted' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pgp-keys' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pgp-signature' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pkcs10' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pkcs7-mime' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pkcs7-signature' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pkix-cert' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pkix-crl' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pkixcmp' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ai' to 'application/postscript' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'eps' to 'application/postscript' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ps' to 'application/postscript' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/prs.alvestrand.titrax-sheet' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/prs.cww' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/prs.nprend' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/prs.plucker' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/qsig' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'rdf' to 'application/rdf+xml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/reginfo+xml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/remote-printing' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/riscos' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/rtf' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/sdp' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/set-payment' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/set-payment-initiation' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/set-registration' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/set-registration-initiation' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/sgml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/sgml-open-catalog' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/sieve' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/slate' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'smi' to 'application/smil' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'smil' to 'application/smil' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'gram' to 'application/srgs' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'grxml' to 'application/srgs+xml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/timestamp-query' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/timestamp-reply' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/tve-trigger' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vemmi' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.3gpp.pic-bw-large' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.3gpp.pic-bw-small' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.3gpp.pic-bw-var' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.3gpp.sms' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.3m.post-it-notes' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.accpac.simply.aso' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.accpac.simply.imp' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.acucobol' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.acucorp' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.adobe.xfdf' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.aether.imp' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.amiga.ami' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.anser-web-certificate-issue-initiation' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.anser-web-funds-transfer-initiation' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.audiograph' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.blueice.multipass' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.bmi' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.businessobjects' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.canon-cpdl' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.canon-lips' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.cinderella' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.claymore' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.commerce-battelle' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.commonspace' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.contact.cmsg' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.cosmocaller' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.criticaltools.wbs+xml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ctc-posml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.cups-postscript' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.cups-raster' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.cups-raw' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.curl' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.cybank' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.data-vision.rdz' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.dna' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.dpgraph' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.dreamfactory' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.dxr' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ecdis-update' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ecowin.chart' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ecowin.filerequest' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ecowin.fileupdate' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ecowin.series' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ecowin.seriesrequest' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ecowin.seriesupdate' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.enliven' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.epson.esf' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.epson.msf' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.epson.quickanime' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.epson.salt' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.epson.ssf' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ericsson.quickcall' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.eudora.data' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fdf' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ffsns' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fints' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.flographit' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.framemaker' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fsc.weblaunch' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujitsu.oasys' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujitsu.oasys2' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujitsu.oasys3' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujitsu.oasysgp' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujitsu.oasysprs' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujixerox.ddd' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujixerox.docuworks' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujixerox.docuworks.binder' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fut-misnet' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.grafeq' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.groove-account' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.groove-help' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.groove-identity-message' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.groove-injector' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.groove-tool-message' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.groove-tool-template' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.groove-vcard' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hbci' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hhe.lesson-player' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hp-hpgl' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hp-hpid' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hp-hps' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hp-pcl' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hp-pclxl' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.httphone' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hzn-3d-crossword' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ibm.afplinedata' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ibm.electronic-media' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ibm.minipay' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ibm.modcap' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ibm.rights-management' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ibm.secure-container' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.informix-visionary' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.intercon.formnet' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.intertrust.digibox' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.intertrust.nncp' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.intu.qbo' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.intu.qfx' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.irepository.package+xml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.is-xpr' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-directory-service' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-jpnstore-wakeup' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-payment-wakeup' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-registration' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-registration-wakeup' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-setstore-wakeup' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-verification' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-verification-wakeup' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.jisp' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.karbon' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.kchart' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.kformula' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.kivio' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.kontour' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.kpresenter' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.kspread' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.kword' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kenameaapp' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.koan' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.liberty-request+xml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.llamagraphics.life-balance.desktop' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.llamagraphics.life-balance.exchange+xml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.lotus-1-2-3' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.lotus-approach' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.lotus-freelance' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.lotus-notes' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.lotus-organizer' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.lotus-screencam' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.lotus-wordpro' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mcd' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mediastation.cdkey' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.meridian-slingshot' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.micrografx.flo' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.micrografx.igx' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mif' to 'application/vnd.mif' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.minisoft-hp3000-save' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mitsubishi.misty-guard.trustweb' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mobius.daf' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mobius.dis' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mobius.mbk' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mobius.mqy' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mobius.msl' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mobius.plc' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mobius.txf' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mophun.application' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mophun.certificate' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.motorola.flexsuite' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.motorola.flexsuite.adsi' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.motorola.flexsuite.fis' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.motorola.flexsuite.gotap' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.motorola.flexsuite.kmr' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.motorola.flexsuite.ttc' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.motorola.flexsuite.wem' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'xul' to 'application/vnd.mozilla.xul+xml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ms-artgalry' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ms-asf' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'xls' to 'application/vnd.ms-excel' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ms-lrm' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ppt' to 'application/vnd.ms-powerpoint' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ms-project' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ms-tnef' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ms-works' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ms-wpl' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mseq' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.msign' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.music-niff' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.musician' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.netfpx' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.noblenet-directory' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.noblenet-sealer' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.noblenet-web' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.novadigm.edm' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.novadigm.edx' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.novadigm.ext' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.obn' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.osa.netdeploy' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.palm' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.pg.format' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.pg.osasli' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.powerbuilder6' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.powerbuilder6-s' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.powerbuilder7' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.powerbuilder7-s' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.powerbuilder75' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.powerbuilder75-s' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.previewsystems.box' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.publishare-delta-tree' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.pvi.ptid1' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.pwg-multiplexed' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.pwg-xhtml-print+xml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.quark.quarkxpress' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.rapid' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'rm' to 'application/vnd.rn-realmedia' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.s3sms' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.sealed.net' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.seemail' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.shana.informed.formdata' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.shana.informed.formtemplate' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.shana.informed.interchange' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.shana.informed.package' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.smaf' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.sss-cod' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.sss-dtf' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.sss-ntf' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.street-stream' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.svd' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.swiftview-ics' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.triscape.mxs' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.trueapp' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.truedoc' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ufdl' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.alert' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.alert-wbxml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.bearer-choice' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.bearer-choice-wbxml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.cacheop' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.cacheop-wbxml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.channel' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.channel-wbxml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.list' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.list-wbxml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.listcmd' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.listcmd-wbxml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.signal' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.vcx' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.vectorworks' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.vidsoft.vidconference' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.visio' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.visionary' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.vividence.scriptfile' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.vsf' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.wap.sic' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.wap.slc' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'wbxml' to 'application/vnd.wap.wbxml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'wmlc' to 'application/vnd.wap.wmlc' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'wmlsc' to 'application/vnd.wap.wmlscriptc' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.webturbo' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.wrq-hp3000-labelled' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.wt.stf' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.wv.csp+wbxml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.xara' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.xfdl' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.yamaha.hv-dic' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.yamaha.hv-script' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.yamaha.hv-voice' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.yellowriver-custom-menu' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'vxml' to 'application/voicexml+xml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/watcherinfo+xml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/whoispp-query' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/whoispp-response' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/wita' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/wordperfect5.1' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'bcpio' to 'application/x-bcpio' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'vcd' to 'application/x-cdlink' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'pgn' to 'application/x-chess-pgn' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/x-compress' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'cpio' to 'application/x-cpio' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'csh' to 'application/x-csh' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'dcr' to 'application/x-director' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'dir' to 'application/x-director' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'dxr' to 'application/x-director' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'dvi' to 'application/x-dvi' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'spl' to 'application/x-futuresplash' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'gtar' to 'application/x-gtar' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'gz' to 'application/x-gzip' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'hdf' to 'application/x-hdf' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'js' to 'application/x-javascript' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'jnlp' to 'application/x-java-jnlp-file' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'skp' to 'application/x-koan' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'skd' to 'application/x-koan' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'skt' to 'application/x-koan' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'skm' to 'application/x-koan' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'latex' to 'application/x-latex' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'nc' to 'application/x-netcdf' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'cdf' to 'application/x-netcdf' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'sh' to 'application/x-sh' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'shar' to 'application/x-shar' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'swf' to 'application/x-shockwave-flash' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'sit' to 'application/x-stuffit' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'sv4cpio' to 'application/x-sv4cpio' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'sv4crc' to 'application/x-sv4crc' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'tar' to 'application/x-tar' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'tcl' to 'application/x-tcl' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'tex' to 'application/x-tex' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'texinfo' to 'application/x-texinfo' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'texi' to 'application/x-texinfo' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 't' to 'application/x-troff' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'tr' to 'application/x-troff' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'roff' to 'application/x-troff' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'man' to 'application/x-troff-man' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'me' to 'application/x-troff-me' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ms' to 'application/x-troff-ms' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ustar' to 'application/x-ustar' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'src' to 'application/x-wais-source' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/x400-bp' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'xhtml' to 'application/xhtml+xml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'xht' to 'application/xhtml+xml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'xslt' to 'application/xslt+xml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'xml' to 'application/xml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'xsl' to 'application/xml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'dtd' to 'application/xml-dtd' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/xml-external-parsed-entity' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'zip' to 'application/zip' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/32kadpcm' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/amr' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/amr-wb' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'au' to 'audio/basic' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'snd' to 'audio/basic' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/cn' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/dat12' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/dsr-es201108' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/dvi4' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/evrc' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/evrc0' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g722' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g.722.1' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g723' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g726-16' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g726-24' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g726-32' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g726-40' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g728' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g729' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g729D' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g729E' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/gsm' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/gsm-efr' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/l8' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/l16' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/l20' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/l24' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/lpc' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mid' to 'audio/midi' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'midi' to 'audio/midi' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'kar' to 'audio/midi' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/mpa' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/mpa-robust' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'm4a' to 'audio/mp4a-latm' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'm4p' to 'audio/mp4a-latm' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mpga' to 'audio/mpeg' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mp2' to 'audio/mpeg' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mp3' to 'audio/mpeg' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/parityfec' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/pcma' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/pcmu' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/prs.sid' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/qcelp' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/red' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/smv' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/smv0' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/telephone-event' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/tone' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vdvi' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.3gpp.iufp' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.cisco.nse' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.cns.anp1' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.cns.inf1' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.digital-winds' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.everad.plj' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.lucent.voice' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.nortel.vbk' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.nuera.ecelp4800' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.nuera.ecelp7470' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.nuera.ecelp9600' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.octel.sbc' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.qcelp' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.rhetorex.32kadpcm' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.vmx.cvsd' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'aif' to 'audio/x-aiff' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'aiff' to 'audio/x-aiff' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'aifc' to 'audio/x-aiff' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/x-alaw-basic' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'm3u' to 'audio/x-mpegurl' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ram' to 'audio/x-pn-realaudio' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ra' to 'audio/x-pn-realaudio' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/x-pn-realaudio-plugin' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'wav' to 'audio/x-wav' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'pdb' to 'chemical/x-pdb' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'xyz' to 'chemical/x-xyz' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'bmp' to 'image/bmp' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'cgm' to 'image/cgm' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/g3fax' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'gif' to 'image/gif' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ief' to 'image/ief' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'jpeg' to 'image/jpeg' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'jpg' to 'image/jpeg' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'jpe' to 'image/jpeg' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'jp2' to 'image/jp2' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/naplps' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'pict' to 'image/pict' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'pic' to 'image/pict' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'pct' to 'image/pict' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'png' to 'image/png' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/prs.btif' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/prs.pti' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'svg' to 'image/svg+xml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/t38' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'tiff' to 'image/tiff' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'tif' to 'image/tiff' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/tiff-fx' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.cns.inf2' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'djvu' to 'image/vnd.djvu' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'djv' to 'image/vnd.djvu' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.dwg' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.dxf' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.fastbidsheet' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.fpx' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.fst' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.fujixerox.edmics-mmr' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.fujixerox.edmics-rlc' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.globalgraphics.pgb' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.mix' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.ms-modi' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.net-fpx' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.svf' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'wbmp' to 'image/vnd.wap.wbmp' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.xiff' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ras' to 'image/x-cmu-raster' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'pntg' to 'image/x-macpaint' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'pnt' to 'image/x-macpaint' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mac' to 'image/x-macpaint' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ico' to 'image/x-icon' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'pnm' to 'image/x-portable-anymap' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'pbm' to 'image/x-portable-bitmap' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'pgm' to 'image/x-portable-graymap' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ppm' to 'image/x-portable-pixmap' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'qtif' to 'image/x-quicktime' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'qti' to 'image/x-quicktime' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'rgb' to 'image/x-rgb' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'xbm' to 'image/x-xbitmap' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'xpm' to 'image/x-xpixmap' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'xwd' to 'image/x-xwindowdump' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/delivery-status' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/disposition-notification' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/external-body' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/http' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/news' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/partial' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/rfc822' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/s-http' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/sip' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/sipfrag' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'igs' to 'model/iges' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'iges' to 'model/iges' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'msh' to 'model/mesh' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mesh' to 'model/mesh' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'silo' to 'model/mesh' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.dwf' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.flatland.3dml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.gdl' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.gs-gdl' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.gtw' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.mts' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.parasolid.transmit.binary' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.parasolid.transmit.text' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.vtu' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'wrl' to 'model/vrml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'vrml' to 'model/vrml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/alternative' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/appledouble' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/byteranges' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/digest' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/encrypted' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/form-data' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/header-set' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/mixed' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/parallel' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/related' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/report' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/signed' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/voice-message' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ics' to 'text/calendar' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ifb' to 'text/calendar' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'css' to 'text/css' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/directory' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/enriched' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'html' to 'text/html' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'htm' to 'text/html' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/parityfec' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'asc' to 'text/plain' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'txt' to 'text/plain' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/prs.lines.tag' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/rfc822-headers' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'rtx' to 'text/richtext' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'rtf' to 'text/rtf' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'sgml' to 'text/sgml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'sgm' to 'text/sgml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/t140' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'tsv' to 'text/tab-separated-values' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/uri-list' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.abc' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.curl' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.dmclientscript' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.fly' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.fmi.flexstor' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.in3d.3dml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.in3d.spot' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.iptc.nitf' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.iptc.newsml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.latex-z' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.motorola.reflex' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.ms-mediapackage' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.net2phone.commcenter.command' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.sun.j2me.app-descriptor' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.wap.si' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.wap.sl' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'wml' to 'text/vnd.wap.wml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'wmls' to 'text/vnd.wap.wmlscript' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'etx' to 'text/x-setext' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/xml' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/xml-external-parsed-entity' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension '3gp' to 'video/3gpp' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/bmpeg' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/bt656' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/celb' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/dv' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/h261' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/h263' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/h263-1998' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/h263-2000' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/jpeg' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/mp1s' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/mp2p' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/mp2t' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mp4' to 'video/mp4' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/mp4v-es' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/mpv' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mpeg' to 'video/mpeg' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mpg' to 'video/mpeg' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mpe' to 'video/mpeg' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/nv' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ogv' to 'video/ogg' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/parityfec' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/pointer' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'qt' to 'video/quicktime' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mov' to 'video/quicktime' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/smpte292m' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/vnd.fvt' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/vnd.motorola.video' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/vnd.motorola.videop' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mxu' to 'video/vnd.mpegurl' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'm4u' to 'video/vnd.mpegurl' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/vnd.nokia.interleaved-multimedia' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/vnd.objectvideo' | |
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/vnd.vivo' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'webm' to 'video/webm' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'dv' to 'video/x-dv' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'dif' to 'video/x-dv' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'flv' to 'video/x-flv' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'm4v' to 'video/x-m4v' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'wmv' to 'video/x-ms-wmv' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'avi' to 'video/x-msvideo' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'movie' to 'video/x-sgi-movie' | |
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ice' to 'x-conference/x-cooltalk' | |
22/08/12 16:24:49 DEBUG Mimetypes: Extension 'tmp' is unrecognized in mime type listing, using default mime type: 'application/octet-stream' | |
22/08/12 16:24:49 DEBUG request: Sending Request: PUT http://localhost:30600 /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet Headers: (amz-sdk-invocation-id: 26236f27-4f41-11b2-2c3c-2d72077ebda7, Content-Length: 718, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:49 DEBUG AWS4Signer: AWS4 Canonical Request: '"PUT | |
/master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet | |
amz-sdk-invocation-id:26236f27-4f41-11b2-2c3c-2d72077ebda7 | |
amz-sdk-request:attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-length:892 | |
content-type:application/octet-stream | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD | |
x-amz-date:20220812T152449Z | |
x-amz-decoded-content-length:718 | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length | |
STREAMING-AWS4-HMAC-SHA256-PAYLOAD" | |
22/08/12 16:24:49 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152449Z | |
20220812/us-east-1/s3/aws4_request | |
77ff5913ac812a24ded91dc2ce594049c2445d92e2a6358cdaa66e84f10d6527" | |
22/08/12 16:24:49 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:49 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:49 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:49 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:49 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:49 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:49 DEBUG MainClientExec: Executing request PUT /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet HTTP/1.1 | |
22/08/12 16:24:49 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> PUT /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet HTTP/1.1 | |
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 26236f27-4f41-11b2-2c3c-2d72077ebda7 | |
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> amz-sdk-request: attempt=1;max=21 | |
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=f5989455b323a7bedf2733a772ede181b7d343cde4703e03ab29a7af7e8067f0 | |
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream | |
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD | |
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152449Z | |
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> x-amz-decoded-content-length: 718 | |
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> Content-Length: 892 | |
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> Expect: 100-continue | |
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "PUT /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet HTTP/1.1[\r][\n]" | |
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 26236f27-4f41-11b2-2c3c-2d72077ebda7[\r][\n]" | |
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=f5989455b323a7bedf2733a772ede181b7d343cde4703e03ab29a7af7e8067f0[\r][\n]" | |
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" | |
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD[\r][\n]" | |
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152449Z[\r][\n]" | |
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "x-amz-decoded-content-length: 718[\r][\n]" | |
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "Content-Length: 892[\r][\n]" | |
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "Expect: 100-continue[\r][\n]" | |
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 << "HTTP/1.1 100 Continue[\r][\n]" | |
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 << HTTP/1.1 100 Continue | |
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "2ce;chunk-signature=95afe3e597425c2053d0489a240992cf32b2c4bffce41f604e72055dcc9f824e[\r][\n]" | |
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "PAR1[0x15][0x0][0x15][0x1c][0x15] [0x15][0xfe][0xc8][0xd0][0xb6][0xb][0x1c][0x15][0x2][0x15][0x0][0x15][0x6][0x15][0x8][0x0][0x0][0xe]4[0x2][0x0][0x0][0x0][0x3][0x1][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x15][0x0][0x15][0x1c][0x15] [0x15][0x9d][0xb3][0x9d][0xee][0x3][0x1c][0x15][0x2][0x15][0x0][0x15][0x6][0x15][0x8][0x0][0x0][0xe]4[0x2][0x0][0x0][0x0][0x3][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0]@[0x19][0x11][0x2][0x19][0x18][0x8][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x19][0x18][0x8][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x15][0x2][0x19][0x16][0x0][0x0][0x19][0x11][0x2][0x19][0x18][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x0]@[0x19][0x18][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x0]@[0x15][0x2][0x19][0x16][0x0][0x0][0x19][0x1c][0x16][0x8][0x15]N[0x16][0x0][0x0][0x0][0x19][0x1c][0x16]V[0x15]N[0x16][0x0][0x0][0x0][0x15][0x2][0x19]<H[0xc]spark_schema[0x15][0x4][0x0][0x15][0x4]%[0x2][0x18][0x1]a[0x0][0x15][\n]" | |
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "%[0x2][0x18][0x1]b[0x0][0x16][0x2][0x19][0x1c][0x19],&[0x8][0x1c][0x15][0x4][0x19]5[0x8][0x6][0x0][0x19][0x18][0x1]a[0x15][0x2][0x16][0x2][0x16]J[0x16]N&[0x8]<[0x18][0x8][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x18][0x8][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x16][0x0]([0x8][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x18][0x8][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x19][0x1c][0x15][0x0][0x15][0x0][0x15][0x2][0x0][0x0][0x16][0xa0][0x2][0x15][0x14][0x16][0xa4][0x1][0x15]>[0x0]&V[0x1c][0x15][\n]" | |
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "[0x19]5[0x8][0x6][0x0][0x19][0x18][0x1]b[0x15][0x2][0x16][0x2][0x16]J[0x16]N&V<[0x18][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x0]@[0x18][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x0]@[0x16][0x0]([0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x0]@[0x18][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x0]@[0x0][0x19][0x1c][0x15][0x0][0x15][0x0][0x15][0x2][0x0][0x0][0x16][0xb4][0x2][0x15][0x14][0x16][0xe2][0x1][0x15]>[0x0][0x16][0x94][0x1][0x16][0x2]&[0x8][0x16][0x9c][0x1][0x14][0x0][0x0][0x19],[0x18][0x18]org.apache.spark.version[0x18][0x5]3.3.0[0x0][0x18])org.apache.spark.sql.parquet.row.metadata[0x18][0x90][0x1]{"type":"struct","fields":[{"name":"a","type":"long","nullable":true,"metadata":{}},{"name":"b","type":"double","nullable":true,"metadata":{}}]}[0x0][0x18]Jparquet-mr version 1.12.2 (build 77e30c8093386ec52c3cfa6c34b7ef3321322c94)[0x19],[0x1c][0x0][0x0][0x1c][0x0][0x0][0x0]"[0x2][0x0][0x0]PAR1[\r][\n]" | |
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "0;chunk-signature=5da53b38411f1cee0108e3369ee46ed8fd850bf4b032bdfecabb8b450bb5a019[\r][\n]" | |
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "Etag: "0a03a8cbced07244b6b9fdfaa86b9b37c3070344e57dde0757bc57e6a7ab045d"[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "X-Amz-Version-Id: 519c48fef92047a29bc78fca8b6c1bf3[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:50 GMT[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "Content-Length: 0[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 << Etag: "0a03a8cbced07244b6b9fdfaa86b9b37c3070344e57dde0757bc57e6a7ab045d" | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 << X-Amz-Version-Id: 519c48fef92047a29bc78fca8b6c1bf3 | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:50 GMT | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 << Content-Length: 0 | |
22/08/12 16:24:50 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:50 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:50 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:50 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:50 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:50 GMT | |
22/08/12 16:24:50 DEBUG request: Received successful response: 200, AWS Request ID: null | |
22/08/12 16:24:50 DEBUG requestId: x-amzn-RequestId: not available | |
22/08/12 16:24:50 DEBUG requestId: AWS Request ID: not available | |
22/08/12 16:24:50 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 13 | |
22/08/12 16:24:50 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[PutObjectRequest], AWSRequestID=[null], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[0.19], ClientExecuteTime=[869.845], HttpClientSendRequestTime=[3.405], HttpRequestTime=[867.332], ApiCallLatency=[869.087], RequestSigningTime=[0.677], CredentialsRequestTime=[0.013, 0.007], HttpClientReceiveResponseTime=[862.463], | |
22/08/12 16:24:50 DEBUG S3AFileSystem: PUT completed success=true; 718 bytes | |
22/08/12 16:24:50 DEBUG IOStatisticsStoreImpl: Incrementing counter object_put_bytes by 718 with final value 718 | |
22/08/12 16:24:50 DEBUG IOStatisticsStoreImpl: Incrementing counter object_put_request_completed by 1 with final value 2 | |
22/08/12 16:24:50 DEBUG S3AFileSystem: Finished write to nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet, len 718. etag 0a03a8cbced07244b6b9fdfaa86b9b37c3070344e57dde0757bc57e6a7ab045d, version 519c48fef92047a29bc78fca8b6c1bf3 | |
22/08/12 16:24:50 DEBUG CallableSupplier: Starting: Waiting for task completion | |
22/08/12 16:24:50 DEBUG S3AFileSystem: Starting: Deleting 5 keys | |
22/08/12 16:24:50 DEBUG S3AFileSystem: Initiating delete operation for 5 objects | |
22/08/12 16:24:50 DEBUG S3AFileSystem: nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/ | |
22/08/12 16:24:50 DEBUG S3AFileSystem: nonemptyprefix8/_temporary/0/_temporary/ | |
22/08/12 16:24:50 DEBUG S3AFileSystem: nonemptyprefix8/_temporary/0/ | |
22/08/12 16:24:50 DEBUG S3AFileSystem: nonemptyprefix8/_temporary/ | |
22/08/12 16:24:50 DEBUG S3AFileSystem: nonemptyprefix8/ | |
22/08/12 16:24:50 DEBUG S3AFileSystem: Starting: DELETE 5 keys | |
22/08/12 16:24:50 DEBUG IOStatisticsStoreImpl: Incrementing counter object_bulk_delete_request by 1 with final value 2 | |
22/08/12 16:24:50 DEBUG IOStatisticsStoreImpl: Incrementing counter object_delete_objects by 5 with final value 7 | |
22/08/12 16:24:50 DEBUG request: Sending Request: POST http://localhost:30600 /master.rando2/ Parameters: ({"delete":[null]}Headers: (amz-sdk-invocation-id: a62c5d64-b53b-ef06-3b3b-bb32dff9b5b8, Content-Length: 387, Content-MD5: toKYq0RGHKQSGL9sB1/cdg==, Content-Type: application/xml, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:50 DEBUG AWS4Signer: AWS4 Canonical Request: '"POST | |
/master.rando2/ | |
delete= | |
amz-sdk-invocation-id:a62c5d64-b53b-ef06-3b3b-bb32dff9b5b8 | |
amz-sdk-request:ttl=20220812T152810Z;attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-length:387 | |
content-md5:toKYq0RGHKQSGL9sB1/cdg== | |
content-type:application/xml | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:3bbb784825b9f4bd2627b155ff84e230e583c52472a55c175691a5da26426020 | |
x-amz-date:20220812T152450Z | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date | |
3bbb784825b9f4bd2627b155ff84e230e583c52472a55c175691a5da26426020" | |
22/08/12 16:24:50 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152450Z | |
20220812/us-east-1/s3/aws4_request | |
f0d5d3665185b4539cd1e3482bc457e01e6cacdaec50d590325cedcabe563b5f" | |
22/08/12 16:24:50 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:50 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:50 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:50 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:50 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:50 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:50 DEBUG MainClientExec: Executing request POST /master.rando2/?delete HTTP/1.1 | |
22/08/12 16:24:50 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> POST /master.rando2/?delete HTTP/1.1 | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: a62c5d64-b53b-ef06-3b3b-bb32dff9b5b8 | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152810Z;attempt=1;max=21 | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=31376faf52d9c0f96abf9757bb37f9f955b52cc54a00c5d87c6a2d4e0d5fb41d | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> Content-MD5: toKYq0RGHKQSGL9sB1/cdg== | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> Content-Type: application/xml | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: 3bbb784825b9f4bd2627b155ff84e230e583c52472a55c175691a5da26426020 | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152450Z | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> Content-Length: 387 | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "POST /master.rando2/?delete HTTP/1.1[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: a62c5d64-b53b-ef06-3b3b-bb32dff9b5b8[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152810Z;attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=31376faf52d9c0f96abf9757bb37f9f955b52cc54a00c5d87c6a2d4e0d5fb41d[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "Content-MD5: toKYq0RGHKQSGL9sB1/cdg==[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: 3bbb784825b9f4bd2627b155ff84e230e583c52472a55c175691a5da26426020[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152450Z[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "Content-Length: 387[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "<Delete><Quiet>true</Quiet><Object><Key>nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/</Key></Object><Object><Key>nonemptyprefix8/_temporary/0/_temporary/</Key></Object><Object><Key>nonemptyprefix8/_temporary/0/</Key></Object><Object><Key>nonemptyprefix8/_temporary/</Key></Object><Object><Key>nonemptyprefix8/</Key></Object></Delete>" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: f338b1a2-c6fc-4af3-935f-27189524aeb3[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: f338b1a2-c6fc-4af3-935f-27189524aeb3[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:50 GMT[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "Content-Length: 116[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"></DeleteResult>" | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 << Content-Type: application/xml | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: f338b1a2-c6fc-4af3-935f-27189524aeb3 | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: f338b1a2-c6fc-4af3-935f-27189524aeb3 | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:50 GMT | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 << Content-Length: 116 | |
22/08/12 16:24:50 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:50 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:50 GMT | |
22/08/12 16:24:50 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$DeleteObjectsHandler | |
22/08/12 16:24:50 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:50 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:50 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:50 DEBUG request: Received successful response: 200, AWS Request ID: f338b1a2-c6fc-4af3-935f-27189524aeb3 | |
22/08/12 16:24:50 DEBUG requestId: x-amzn-RequestId: not available | |
22/08/12 16:24:50 DEBUG requestId: AWS Request ID: f338b1a2-c6fc-4af3-935f-27189524aeb3 | |
22/08/12 16:24:50 DEBUG requestId: AWS Extended Request ID: f338b1a2-c6fc-4af3-935f-27189524aeb3 | |
22/08/12 16:24:50 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 14 | |
22/08/12 16:24:50 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[DeleteObjectsRequest], AWSRequestID=[f338b1a2-c6fc-4af3-935f-27189524aeb3], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[1.572], ClientExecuteTime=[95.418], HttpClientSendRequestTime=[1.283], HttpRequestTime=[92.192], ApiCallLatency=[95.067], RequestSigningTime=[0.543], CredentialsRequestTime=[0.012, 0.008], HttpClientReceiveResponseTime=[89.956], | |
22/08/12 16:24:50 DEBUG S3AFileSystem: DELETE 5 keys: duration 0:00.097s | |
22/08/12 16:24:50 DEBUG IOStatisticsStoreImpl: Incrementing counter fake_directories_deleted by 5 with final value 7 | |
22/08/12 16:24:50 DEBUG S3AFileSystem: Deleting 5 keys: duration 0:00.097s | |
22/08/12 16:24:50 DEBUG CallableSupplier: Waiting for task completion: duration 0:00.098s | |
22/08/12 16:24:50 DEBUG Invoker: Writing Object: duration 0:00.991s | |
22/08/12 16:24:50 DEBUG S3ADataBlocks: FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-8560763669495856341.tmp, state=Upload, dataSize=718, limit=67108864}: entering state Closed | |
22/08/12 16:24:50 DEBUG S3ADataBlocks: Closed FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-8560763669495856341.tmp, state=Closed, dataSize=718, limit=67108864} | |
22/08/12 16:24:50 DEBUG S3ADataBlocks: Closing FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-8560763669495856341.tmp, state=Closed, dataSize=718, limit=67108864} | |
22/08/12 16:24:50 DEBUG S3ADataBlocks: block[1]: closeBlock() | |
22/08/12 16:24:50 DEBUG S3ABlockOutputStream: Upload complete to nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet by WriteOperationHelper {bucket=master.rando2} | |
22/08/12 16:24:50 DEBUG S3ABlockOutputStream: Statistics: OutputStreamStatistics{counters=((stream_write_exceptions=0) (multipart_upload_completed.failures=0) (multipart_upload_completed=0) (stream_write_block_uploads=1) (object_multipart_aborted.failures=0) (op_hsync=0) (op_abort=0) (stream_write_queue_duration=0) (stream_write_bytes=718) (object_multipart_aborted=0) (op_abort.failures=0) (stream_write_exceptions_completing_upload=0) (action_executor_acquired=0) (stream_write_total_data=718) (op_hflush=0) (action_executor_acquired.failures=0) (stream_write_total_time=0)); | |
gauges=((stream_write_block_uploads_pending=1) (stream_write_block_uploads_data_pending=0)); | |
minimums=((op_abort.failures.min=-1) (action_executor_acquired.failures.min=-1) (multipart_upload_completed.min=-1) (action_executor_acquired.min=-1) (multipart_upload_completed.failures.min=-1) (object_multipart_aborted.failures.min=-1) (object_multipart_aborted.min=-1) (op_abort.min=-1)); | |
maximums=((op_abort.failures.max=-1) (action_executor_acquired.max=-1) (object_multipart_aborted.max=-1) (action_executor_acquired.failures.max=-1) (multipart_upload_completed.failures.max=-1) (op_abort.max=-1) (multipart_upload_completed.max=-1) (object_multipart_aborted.failures.max=-1)); | |
means=((object_multipart_aborted.mean=(samples=0, sum=0, mean=0.0000)) (op_abort.mean=(samples=0, sum=0, mean=0.0000)) (multipart_upload_completed.mean=(samples=0, sum=0, mean=0.0000)) (action_executor_acquired.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_aborted.failures.mean=(samples=0, sum=0, mean=0.0000)) (multipart_upload_completed.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_abort.failures.mean=(samples=0, sum=0, mean=0.0000)) (action_executor_acquired.failures.mean=(samples=0, sum=0, mean=0.0000))); | |
, blocksActive=0, blockUploadsCompleted=0, blocksAllocated=1, blocksReleased=1, blocksActivelyAllocated=0, transferDuration=0 ms, totalUploadDuration=0 ms, effectiveBandwidth=0.0 bytes/s} | |
22/08/12 16:24:50 DEBUG IOStatisticsStoreImpl: Incrementing counter stream_write_total_data by 718 with final value 718 | |
22/08/12 16:24:50 DEBUG IOStatisticsStoreImpl: Incrementing counter op_get_file_status by 1 with final value 1 | |
22/08/12 16:24:50 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet (nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet); needEmptyDirectory=false | |
22/08/12 16:24:50 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet | |
22/08/12 16:24:50 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 6 | |
22/08/12 16:24:50 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 6 | |
22/08/12 16:24:50 DEBUG S3AFileSystem: HEAD nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet with change tracker null | |
22/08/12 16:24:50 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet Headers: (amz-sdk-invocation-id: 27b050b9-f85f-e928-3680-5830e9e4c362, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:50 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD | |
/master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet | |
amz-sdk-invocation-id:27b050b9-f85f-e928-3680-5830e9e4c362 | |
amz-sdk-request:ttl=20220812T152810Z;attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-type:application/octet-stream | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
x-amz-date:20220812T152450Z | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date | |
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | |
22/08/12 16:24:50 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152450Z | |
20220812/us-east-1/s3/aws4_request | |
901b3afc5808d40b024f5b646f56c6bdc400a9e7f70fda96b91a1942a94f6e06" | |
22/08/12 16:24:50 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:50 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:50 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:50 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:50 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:50 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:50 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet HTTP/1.1 | |
22/08/12 16:24:50 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet HTTP/1.1 | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 27b050b9-f85f-e928-3680-5830e9e4c362 | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152810Z;attempt=1;max=21 | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=c436f5583498a8da5c7433ed0827e388de288d702b7f4553162ba2e2172cf68a | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152450Z | |
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet HTTP/1.1[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 27b050b9-f85f-e928-3680-5830e9e4c362[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152810Z;attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=c436f5583498a8da5c7433ed0827e388de288d702b7f4553162ba2e2172cf68a[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152450Z[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 7fb54d66-eb63-49b0-baf8-394beed0fb78[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 7fb54d66-eb63-49b0-baf8-394beed0fb78[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:51 GMT[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Length: 389[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Type: application/xml | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 7fb54d66-eb63-49b0-baf8-394beed0fb78 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 7fb54d66-eb63-49b0-baf8-394beed0fb78 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:51 GMT | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Length: 389 | |
22/08/12 16:24:51 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:51 GMT | |
22/08/12 16:24:51 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 7fb54d66-eb63-49b0-baf8-394beed0fb78; S3 Extended Request ID: 7fb54d66-eb63-49b0-baf8-394beed0fb78; Proxy: null), S3 Extended Request ID: 7fb54d66-eb63-49b0-baf8-394beed0fb78 | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 15 | |
22/08/12 16:24:51 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 7fb54d66-eb63-49b0-baf8-394beed0fb78; S3 Extended Request ID: 7fb54d66-eb63-49b0-baf8-394beed0fb78; Proxy: null), S3 Extended Request ID: 7fb54d66-eb63-49b0-baf8-394beed0fb78], RequestType=[GetObjectMetadataRequest], AWSRequestID=[7fb54d66-eb63-49b0-baf8-394beed0fb78], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[755.373], HttpClientSendRequestTime=[2.076], HttpRequestTime=[753.718], ApiCallLatency=[755.1], RequestSigningTime=[0.433], CredentialsRequestTime=[0.008, 0.007], HttpClientReceiveResponseTime=[750.435], | |
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet/ delimiter=/ keys=2 requester pays=false | |
22/08/12 16:24:51 DEBUG S3AFileSystem: Starting: LIST | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 6 | |
22/08/12 16:24:51 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: f97ebf25-72c6-b816-2176-e7fb83f7ce8b, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET | |
/master.rando2/ | |
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2Fpart-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet%2F | |
amz-sdk-invocation-id:f97ebf25-72c6-b816-2176-e7fb83f7ce8b | |
amz-sdk-request:ttl=20220812T152811Z;attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-type:application/octet-stream | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
x-amz-date:20220812T152451Z | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date | |
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | |
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152451Z | |
20220812/us-east-1/s3/aws4_request | |
b47271e8efca2189609898dc4cac47b15df6d34d4c57b2d391b80c7fc8aec663" | |
22/08/12 16:24:51 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:51 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:51 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2Fpart-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1 | |
22/08/12 16:24:51 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2Fpart-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: f97ebf25-72c6-b816-2176-e7fb83f7ce8b | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=900733557a3486e91db8a71ed5763399cd3d754b52e257494b0f82e766107e81 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152451Z | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Length: 0 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2Fpart-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: f97ebf25-72c6-b816-2176-e7fb83f7ce8b[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=900733557a3486e91db8a71ed5763399cd3d754b52e257494b0f82e766107e81[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152451Z[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 8eb704a4-3211-44e6-8fad-ee79fa440b71[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 8eb704a4-3211-44e6-8fad-ee79fa440b71[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:51 GMT[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Length: 427[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet/</Prefix></ListBucketResult>" | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Type: application/xml | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 8eb704a4-3211-44e6-8fad-ee79fa440b71 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 8eb704a4-3211-44e6-8fad-ee79fa440b71 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:51 GMT | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Length: 427 | |
22/08/12 16:24:51 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:51 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:51 GMT | |
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler | |
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2 | |
22/08/12 16:24:51 DEBUG request: Received successful response: 200, AWS Request ID: 8eb704a4-3211-44e6-8fad-ee79fa440b71 | |
22/08/12 16:24:51 DEBUG requestId: x-amzn-RequestId: not available | |
22/08/12 16:24:51 DEBUG requestId: AWS Request ID: 8eb704a4-3211-44e6-8fad-ee79fa440b71 | |
22/08/12 16:24:51 DEBUG requestId: AWS Extended Request ID: 8eb704a4-3211-44e6-8fad-ee79fa440b71 | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 16 | |
22/08/12 16:24:51 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[8eb704a4-3211-44e6-8fad-ee79fa440b71], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[2.01], ClientExecuteTime=[24.744], HttpClientSendRequestTime=[1.1], HttpRequestTime=[20.892], ApiCallLatency=[24.477], RequestSigningTime=[0.579], CredentialsRequestTime=[0.008, 0.007], HttpClientReceiveResponseTime=[17.786], | |
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST: duration 0:00.026s | |
22/08/12 16:24:51 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter op_get_file_status.failures by 1 with final value 1 | |
22/08/12 16:24:51 DEBUG BasicWriteTaskStatsTracker: File s3a://master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet is not yet visible | |
java.io.FileNotFoundException: No such file or directory: s3a://master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3866) | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3688) | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getFileStatus$24(S3AFileSystem.java:3556) | |
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499) | |
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444) | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337) | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356) | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3554) | |
at org.apache.spark.sql.execution.datasources.BasicWriteTaskStatsTracker.getFileSize(BasicWriteStatsTracker.scala:89) | |
at org.apache.spark.sql.execution.datasources.BasicWriteTaskStatsTracker.getFileSize(BasicWriteStatsTracker.scala:73) | |
at org.apache.spark.sql.execution.datasources.BasicWriteTaskStatsTracker.updateFileStats(BasicWriteStatsTracker.scala:151) | |
at org.apache.spark.sql.execution.datasources.BasicWriteTaskStatsTracker.closeFile(BasicWriteStatsTracker.scala:146) | |
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.$anonfun$releaseCurrentWriter$1(FileFormatDataWriter.scala:65) | |
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.$anonfun$releaseCurrentWriter$1$adapted(FileFormatDataWriter.scala:65) | |
at scala.collection.immutable.List.foreach(List.scala:431) | |
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseCurrentWriter(FileFormatDataWriter.scala:65) | |
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseResources(FileFormatDataWriter.scala:75) | |
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:105) | |
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:332) | |
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538) | |
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:338) | |
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$21(FileFormatWriter.scala:256) | |
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) | |
at org.apache.spark.scheduler.Task.run(Task.scala:136) | |
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) | |
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504) | |
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) | |
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) | |
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) | |
at java.base/java.lang.Thread.run(Thread.java:829) | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists by 1 with final value 2 | |
22/08/12 16:24:51 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16 (nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16); needEmptyDirectory=false | |
22/08/12 16:24:51 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16 | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 7 | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 7 | |
22/08/12 16:24:51 DEBUG S3AFileSystem: HEAD nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16 with change tracker null | |
22/08/12 16:24:51 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16 Headers: (amz-sdk-invocation-id: 6c3a185e-0f14-f0c3-2eed-ed33a6b8df5e, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD | |
/master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16 | |
amz-sdk-invocation-id:6c3a185e-0f14-f0c3-2eed-ed33a6b8df5e | |
amz-sdk-request:ttl=20220812T152811Z;attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-type:application/octet-stream | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
x-amz-date:20220812T152451Z | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date | |
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | |
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152451Z | |
20220812/us-east-1/s3/aws4_request | |
c697e3a3f9dd16fef539157287952cc455f77d3859c05e8352fc3d64774bc6ac" | |
22/08/12 16:24:51 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:51 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:51 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16 HTTP/1.1 | |
22/08/12 16:24:51 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16 HTTP/1.1 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 6c3a185e-0f14-f0c3-2eed-ed33a6b8df5e | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=2d252e1819288bb106f29b54518889e14de29614d6e4bb66d866df1c1785aaa2 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152451Z | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16 HTTP/1.1[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 6c3a185e-0f14-f0c3-2eed-ed33a6b8df5e[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=2d252e1819288bb106f29b54518889e14de29614d6e4bb66d866df1c1785aaa2[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152451Z[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: c7e4dc4b-fc95-406a-87c7-304916a96ca7[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: c7e4dc4b-fc95-406a-87c7-304916a96ca7[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:51 GMT[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Length: 321[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Type: application/xml | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: c7e4dc4b-fc95-406a-87c7-304916a96ca7 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: c7e4dc4b-fc95-406a-87c7-304916a96ca7 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:51 GMT | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Length: 321 | |
22/08/12 16:24:51 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:51 GMT | |
22/08/12 16:24:51 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: c7e4dc4b-fc95-406a-87c7-304916a96ca7; S3 Extended Request ID: c7e4dc4b-fc95-406a-87c7-304916a96ca7; Proxy: null), S3 Extended Request ID: c7e4dc4b-fc95-406a-87c7-304916a96ca7 | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 17 | |
22/08/12 16:24:51 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: c7e4dc4b-fc95-406a-87c7-304916a96ca7; S3 Extended Request ID: c7e4dc4b-fc95-406a-87c7-304916a96ca7; Proxy: null), S3 Extended Request ID: c7e4dc4b-fc95-406a-87c7-304916a96ca7], RequestType=[GetObjectMetadataRequest], AWSRequestID=[c7e4dc4b-fc95-406a-87c7-304916a96ca7], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[21.524], HttpClientSendRequestTime=[0.497], HttpRequestTime=[20.419], ApiCallLatency=[21.339], RequestSigningTime=[0.288], CredentialsRequestTime=[0.007, 0.003], HttpClientReceiveResponseTime=[19.029], | |
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/ delimiter=/ keys=2 requester pays=false | |
22/08/12 16:24:51 DEBUG S3AFileSystem: Starting: LIST | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 7 | |
22/08/12 16:24:51 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 8b9dc119-a5b5-1907-841a-40737bfed560, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET | |
/master.rando2/ | |
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2F | |
amz-sdk-invocation-id:8b9dc119-a5b5-1907-841a-40737bfed560 | |
amz-sdk-request:ttl=20220812T152811Z;attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-type:application/octet-stream | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
x-amz-date:20220812T152451Z | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date | |
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | |
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152451Z | |
20220812/us-east-1/s3/aws4_request | |
a761af39b70452fa9e5f865c72ded9254ff2583b10356a1f0332125e26183463" | |
22/08/12 16:24:51 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:51 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:51 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2F&fetch-owner=false HTTP/1.1 | |
22/08/12 16:24:51 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2F&fetch-owner=false HTTP/1.1 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 8b9dc119-a5b5-1907-841a-40737bfed560 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=8929be5086660625b3e189da58ff1d3c4546d1c660718febb3b4d5b379f955c8 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152451Z | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Length: 0 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2F&fetch-owner=false HTTP/1.1[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 8b9dc119-a5b5-1907-841a-40737bfed560[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=8929be5086660625b3e189da58ff1d3c4546d1c660718febb3b4d5b379f955c8[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152451Z[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: f21dc649-ab17-47ba-b9cf-9f4a27494ad7[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: f21dc649-ab17-47ba-b9cf-9f4a27494ad7[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:51 GMT[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Length: 359[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/</Prefix></ListBucketResult>" | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Type: application/xml | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: f21dc649-ab17-47ba-b9cf-9f4a27494ad7 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: f21dc649-ab17-47ba-b9cf-9f4a27494ad7 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:51 GMT | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Length: 359 | |
22/08/12 16:24:51 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:51 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:51 GMT | |
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler | |
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2 | |
22/08/12 16:24:51 DEBUG request: Received successful response: 200, AWS Request ID: f21dc649-ab17-47ba-b9cf-9f4a27494ad7 | |
22/08/12 16:24:51 DEBUG requestId: x-amzn-RequestId: not available | |
22/08/12 16:24:51 DEBUG requestId: AWS Request ID: f21dc649-ab17-47ba-b9cf-9f4a27494ad7 | |
22/08/12 16:24:51 DEBUG requestId: AWS Extended Request ID: f21dc649-ab17-47ba-b9cf-9f4a27494ad7 | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 18 | |
22/08/12 16:24:51 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[f21dc649-ab17-47ba-b9cf-9f4a27494ad7], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[1.969], ClientExecuteTime=[19.841], HttpClientSendRequestTime=[0.513], HttpRequestTime=[16.61], ApiCallLatency=[19.577], RequestSigningTime=[0.321], CredentialsRequestTime=[0.007, 0.003], HttpClientReceiveResponseTime=[15.378], | |
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST: duration 0:00.020s | |
22/08/12 16:24:51 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16 | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists.failures by 1 with final value 2 | |
22/08/12 16:24:51 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_202208121624485385779855500771815_0003_m_000000_16 | |
22/08/12 16:24:51 WARN BasicWriteTaskStatsTracker: Expected 1 files, but only saw 0. This could be due to the output format not writing empty files, or files being not immediately visible in the filesystem. | |
22/08/12 16:24:51 INFO Executor: Finished task 0.0 in stage 3.0 (TID 16). 2914 bytes result sent to driver | |
22/08/12 16:24:51 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 0 | |
22/08/12 16:24:51 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 16) in 3418 ms on 10.1.255.235 (executor driver) (1/1) | |
22/08/12 16:24:51 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool | |
22/08/12 16:24:51 INFO DAGScheduler: ResultStage 3 (save at NativeMethodAccessorImpl.java:0) finished in 3.452 s | |
22/08/12 16:24:51 DEBUG DAGScheduler: After removal of stage 3, remaining stages = 0 | |
22/08/12 16:24:51 INFO DAGScheduler: Job 3 is finished. Cancelling potential speculative or zombie tasks for this job | |
22/08/12 16:24:51 INFO TaskSchedulerImpl: Killing all running tasks in stage 3: Stage finished | |
22/08/12 16:24:51 DEBUG SQLHadoopMapReduceCommitProtocol: onTaskCommit(org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage@224dab57) | |
22/08/12 16:24:51 INFO DAGScheduler: Job 3 finished: save at NativeMethodAccessorImpl.java:0, took 3.459553 s | |
22/08/12 16:24:51 INFO FileFormatWriter: Start to commit write Job 1579bf8c-1bfb-448e-994b-bf13ffb6fe44. | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter op_list_status by 1 with final value 1 | |
22/08/12 16:24:51 DEBUG Invoker: Starting: listStatus | |
22/08/12 16:24:51 DEBUG S3AFileSystem: List status for path: s3a://master.rando2/nonemptyprefix8/_temporary/0 | |
22/08/12 16:24:51 DEBUG S3AFileSystem: listStatus: doing listObjects for directory nonemptyprefix8/_temporary/0/ | |
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix8/_temporary/0/ delimiter=/ keys=5000 requester pays=false | |
22/08/12 16:24:51 DEBUG S3AFileSystem: Starting: LIST | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 8 | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 1 | |
22/08/12 16:24:51 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["5000"],"prefix":["nonemptyprefix8/_temporary/0/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 3011fe8b-f41b-86d6-56f5-d169b3fffbc1, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET | |
/master.rando2/ | |
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=5000&prefix=nonemptyprefix8%2F_temporary%2F0%2F | |
amz-sdk-invocation-id:3011fe8b-f41b-86d6-56f5-d169b3fffbc1 | |
amz-sdk-request:ttl=20220812T152811Z;attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-type:application/octet-stream | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
x-amz-date:20220812T152451Z | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date | |
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | |
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152451Z | |
20220812/us-east-1/s3/aws4_request | |
e3511709d2eee863d6c53c409365922ac12c78c26b0c9d1aa5782f5ab33d05c3" | |
22/08/12 16:24:51 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:51 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:51 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=5000&prefix=nonemptyprefix8%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1 | |
22/08/12 16:24:51 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=5000&prefix=nonemptyprefix8%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 3011fe8b-f41b-86d6-56f5-d169b3fffbc1 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=6c71d26c6694c471b3bb54ffeeecd8d246a0b64fa79abe27f20f0c188608d575 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152451Z | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Length: 0 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=5000&prefix=nonemptyprefix8%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 3011fe8b-f41b-86d6-56f5-d169b3fffbc1[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=6c71d26c6694c471b3bb54ffeeecd8d246a0b64fa79abe27f20f0c188608d575[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152451Z[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: af36717a-5c80-4d34-84f8-b041b08c5376[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: af36717a-5c80-4d34-84f8-b041b08c5376[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:51 GMT[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Length: 292[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>5000</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix8/_temporary/0/</Prefix></ListBucketResult>" | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Type: application/xml | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: af36717a-5c80-4d34-84f8-b041b08c5376 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: af36717a-5c80-4d34-84f8-b041b08c5376 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:51 GMT | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Length: 292 | |
22/08/12 16:24:51 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:51 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:51 GMT | |
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler | |
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2 | |
22/08/12 16:24:51 DEBUG request: Received successful response: 200, AWS Request ID: af36717a-5c80-4d34-84f8-b041b08c5376 | |
22/08/12 16:24:51 DEBUG requestId: x-amzn-RequestId: not available | |
22/08/12 16:24:51 DEBUG requestId: AWS Request ID: af36717a-5c80-4d34-84f8-b041b08c5376 | |
22/08/12 16:24:51 DEBUG requestId: AWS Extended Request ID: af36717a-5c80-4d34-84f8-b041b08c5376 | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 19 | |
22/08/12 16:24:51 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[af36717a-5c80-4d34-84f8-b041b08c5376], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[1.787], ClientExecuteTime=[18.346], HttpClientSendRequestTime=[0.488], HttpRequestTime=[15.265], ApiCallLatency=[18.142], RequestSigningTime=[0.393], CredentialsRequestTime=[0.007, 0.004], HttpClientReceiveResponseTime=[14.104], | |
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST: duration 0:00.019s | |
22/08/12 16:24:51 DEBUG S3AFileSystem: Added 0 entries; ignored 0; hasNext=false; hasMoreObjects=false | |
22/08/12 16:24:51 DEBUG S3AFileSystem: All entries in batch were filtered...continuing | |
22/08/12 16:24:51 DEBUG S3AFileSystem: Start iterating the provided status. | |
22/08/12 16:24:51 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8/_temporary/0 (nonemptyprefix8/_temporary/0); needEmptyDirectory=false | |
22/08/12 16:24:51 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8/_temporary/0 | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 8 | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 8 | |
22/08/12 16:24:51 DEBUG S3AFileSystem: HEAD nonemptyprefix8/_temporary/0 with change tracker null | |
22/08/12 16:24:51 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix8/_temporary/0 Headers: (amz-sdk-invocation-id: e494962e-c10c-7eb4-0a66-d6491ec50ef4, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD | |
/master.rando2/nonemptyprefix8/_temporary/0 | |
amz-sdk-invocation-id:e494962e-c10c-7eb4-0a66-d6491ec50ef4 | |
amz-sdk-request:ttl=20220812T152811Z;attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-type:application/octet-stream | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
x-amz-date:20220812T152451Z | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date | |
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | |
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152451Z | |
20220812/us-east-1/s3/aws4_request | |
8b09a1ea04a37a1688c0ddb5c634fc72c81ac78ee150f8c6952bcf472c1f980e" | |
22/08/12 16:24:51 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:51 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:51 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix8/_temporary/0 HTTP/1.1 | |
22/08/12 16:24:51 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix8/_temporary/0 HTTP/1.1 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: e494962e-c10c-7eb4-0a66-d6491ec50ef4 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=da3421c4076f13bd1eefe66623084ae3b32b8d9d1ce3d3562b00ba8a0722643c | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152451Z | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix8/_temporary/0 HTTP/1.1[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: e494962e-c10c-7eb4-0a66-d6491ec50ef4[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=da3421c4076f13bd1eefe66623084ae3b32b8d9d1ce3d3562b00ba8a0722643c[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152451Z[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: d782dd59-cc6f-4369-9ffe-1014297ddb9d[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: d782dd59-cc6f-4369-9ffe-1014297ddb9d[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:51 GMT[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Length: 251[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Type: application/xml | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: d782dd59-cc6f-4369-9ffe-1014297ddb9d | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: d782dd59-cc6f-4369-9ffe-1014297ddb9d | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:51 GMT | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Length: 251 | |
22/08/12 16:24:51 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:51 GMT | |
22/08/12 16:24:51 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: d782dd59-cc6f-4369-9ffe-1014297ddb9d; S3 Extended Request ID: d782dd59-cc6f-4369-9ffe-1014297ddb9d; Proxy: null), S3 Extended Request ID: d782dd59-cc6f-4369-9ffe-1014297ddb9d | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 20 | |
22/08/12 16:24:51 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: d782dd59-cc6f-4369-9ffe-1014297ddb9d; S3 Extended Request ID: d782dd59-cc6f-4369-9ffe-1014297ddb9d; Proxy: null), S3 Extended Request ID: d782dd59-cc6f-4369-9ffe-1014297ddb9d], RequestType=[GetObjectMetadataRequest], AWSRequestID=[d782dd59-cc6f-4369-9ffe-1014297ddb9d], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[15.068], HttpClientSendRequestTime=[0.654], HttpRequestTime=[13.776], ApiCallLatency=[14.845], RequestSigningTime=[0.319], CredentialsRequestTime=[0.007, 0.003], HttpClientReceiveResponseTime=[12.373], | |
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix8/_temporary/0/ delimiter=/ keys=2 requester pays=false | |
22/08/12 16:24:51 DEBUG S3AFileSystem: Starting: LIST | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 9 | |
22/08/12 16:24:51 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix8/_temporary/0/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 102525c2-7281-4568-38a0-0d644d4e8781, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET | |
/master.rando2/ | |
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F | |
amz-sdk-invocation-id:102525c2-7281-4568-38a0-0d644d4e8781 | |
amz-sdk-request:ttl=20220812T152811Z;attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-type:application/octet-stream | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
x-amz-date:20220812T152451Z | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date | |
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | |
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152451Z | |
20220812/us-east-1/s3/aws4_request | |
b1b6315da502059e7d80bd950896f44a02b4be883c275342cb87f26ca4b9f0ef" | |
22/08/12 16:24:51 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:51 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:51 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1 | |
22/08/12 16:24:51 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 102525c2-7281-4568-38a0-0d644d4e8781 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=c21dc7ada620ea45e9b848b24fc70f8f42e2da7b4d1e2e2f1846cb69a00aaeac | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152451Z | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Length: 0 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 102525c2-7281-4568-38a0-0d644d4e8781[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=c21dc7ada620ea45e9b848b24fc70f8f42e2da7b4d1e2e2f1846cb69a00aaeac[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152451Z[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 35778686-73a8-4b45-a002-e6315654ad1f[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 35778686-73a8-4b45-a002-e6315654ad1f[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:51 GMT[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Length: 289[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix8/_temporary/0/</Prefix></ListBucketResult>" | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Type: application/xml | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 35778686-73a8-4b45-a002-e6315654ad1f | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 35778686-73a8-4b45-a002-e6315654ad1f | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:51 GMT | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Length: 289 | |
22/08/12 16:24:51 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:51 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:51 GMT | |
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler | |
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2 | |
22/08/12 16:24:51 DEBUG request: Received successful response: 200, AWS Request ID: 35778686-73a8-4b45-a002-e6315654ad1f | |
22/08/12 16:24:51 DEBUG requestId: x-amzn-RequestId: not available | |
22/08/12 16:24:51 DEBUG requestId: AWS Request ID: 35778686-73a8-4b45-a002-e6315654ad1f | |
22/08/12 16:24:51 DEBUG requestId: AWS Extended Request ID: 35778686-73a8-4b45-a002-e6315654ad1f | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 21 | |
22/08/12 16:24:51 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[35778686-73a8-4b45-a002-e6315654ad1f], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[1.608], ClientExecuteTime=[17.536], HttpClientSendRequestTime=[0.496], HttpRequestTime=[14.694], ApiCallLatency=[17.336], RequestSigningTime=[0.331], CredentialsRequestTime=[0.006, 0.003], HttpClientReceiveResponseTime=[13.589], | |
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST: duration 0:00.018s | |
22/08/12 16:24:51 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8/_temporary/0 | |
22/08/12 16:24:51 DEBUG Invoker: listStatus: duration 0:00.057s | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter op_list_status.failures by 1 with final value 1 | |
22/08/12 16:24:51 ERROR FileFormatWriter: Aborting job 1579bf8c-1bfb-448e-994b-bf13ffb6fe44. | |
java.io.FileNotFoundException: No such file or directory: s3a://master.rando2/nonemptyprefix8/_temporary/0 | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3866) | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3688) | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:3300) | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$null$20(S3AFileSystem.java:3264) | |
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:117) | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listStatus$21(S3AFileSystem.java:3263) | |
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499) | |
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444) | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337) | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356) | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:3262) | |
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1972) | |
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:2014) | |
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getAllCommittedTaskPaths(FileOutputCommitter.java:334) | |
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJobInternal(FileOutputCommitter.java:404) | |
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:377) | |
at org.apache.parquet.hadoop.ParquetOutputCommitter.commitJob(ParquetOutputCommitter.java:48) | |
at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:192) | |
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$25(FileFormatWriter.scala:267) | |
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) | |
at org.apache.spark.util.Utils$.timeTakenMs(Utils.scala:642) | |
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:267) | |
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:186) | |
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:113) | |
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:111) | |
at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:125) | |
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98) | |
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109) | |
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169) | |
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95) | |
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779) | |
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64) | |
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98) | |
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94) | |
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584) | |
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176) | |
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584) | |
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30) | |
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267) | |
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263) | |
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30) | |
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30) | |
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560) | |
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94) | |
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81) | |
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79) | |
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:116) | |
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:860) | |
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390) | |
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:363) | |
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239) | |
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) | |
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) | |
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
at java.base/java.lang.reflect.Method.invoke(Method.java:566) | |
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) | |
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) | |
at py4j.Gateway.invoke(Gateway.java:282) | |
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) | |
at py4j.commands.CallCommand.execute(CallCommand.java:79) | |
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) | |
at py4j.ClientServerConnection.run(ClientServerConnection.java:106) | |
at java.base/java.lang.Thread.run(Thread.java:829) | |
22/08/12 16:24:51 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8/_temporary (nonemptyprefix8/_temporary); needEmptyDirectory=true | |
22/08/12 16:24:51 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8/_temporary | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 9 | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 9 | |
22/08/12 16:24:51 DEBUG S3AFileSystem: HEAD nonemptyprefix8/_temporary with change tracker null | |
22/08/12 16:24:51 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix8/_temporary Headers: (amz-sdk-invocation-id: e7507793-ba60-18ef-a010-f51284433e19, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD | |
/master.rando2/nonemptyprefix8/_temporary | |
amz-sdk-invocation-id:e7507793-ba60-18ef-a010-f51284433e19 | |
amz-sdk-request:ttl=20220812T152811Z;attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-type:application/octet-stream | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
x-amz-date:20220812T152451Z | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date | |
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | |
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152451Z | |
20220812/us-east-1/s3/aws4_request | |
98bdab4f625a5e1d2a55a8523f53e80a6d21f926bb162c9aebecda5f6f725761" | |
22/08/12 16:24:51 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:51 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:51 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix8/_temporary HTTP/1.1 | |
22/08/12 16:24:51 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix8/_temporary HTTP/1.1 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: e7507793-ba60-18ef-a010-f51284433e19 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=1a966b377e5ec0ceaff0728e2ff5fe4458c42ac90422ff136eec6a6cdf8bd78d | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152451Z | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix8/_temporary HTTP/1.1[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: e7507793-ba60-18ef-a010-f51284433e19[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=1a966b377e5ec0ceaff0728e2ff5fe4458c42ac90422ff136eec6a6cdf8bd78d[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152451Z[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 3de2dd19-68e9-4f52-aff3-04b834d9d45a[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 3de2dd19-68e9-4f52-aff3-04b834d9d45a[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:51 GMT[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Length: 249[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Type: application/xml | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 3de2dd19-68e9-4f52-aff3-04b834d9d45a | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 3de2dd19-68e9-4f52-aff3-04b834d9d45a | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:51 GMT | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Length: 249 | |
22/08/12 16:24:51 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:51 GMT | |
22/08/12 16:24:51 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 3de2dd19-68e9-4f52-aff3-04b834d9d45a; S3 Extended Request ID: 3de2dd19-68e9-4f52-aff3-04b834d9d45a; Proxy: null), S3 Extended Request ID: 3de2dd19-68e9-4f52-aff3-04b834d9d45a | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 22 | |
22/08/12 16:24:51 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 3de2dd19-68e9-4f52-aff3-04b834d9d45a; S3 Extended Request ID: 3de2dd19-68e9-4f52-aff3-04b834d9d45a; Proxy: null), S3 Extended Request ID: 3de2dd19-68e9-4f52-aff3-04b834d9d45a], RequestType=[GetObjectMetadataRequest], AWSRequestID=[3de2dd19-68e9-4f52-aff3-04b834d9d45a], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[15.174], HttpClientSendRequestTime=[0.456], HttpRequestTime=[14.17], ApiCallLatency=[14.979], RequestSigningTime=[0.217], CredentialsRequestTime=[0.007, 0.003], HttpClientReceiveResponseTime=[12.958], | |
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix8/_temporary/ delimiter=/ keys=2 requester pays=false | |
22/08/12 16:24:51 DEBUG S3AFileSystem: Starting: LIST | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 10 | |
22/08/12 16:24:51 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix8/_temporary/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 42665700-8a06-44b3-9b47-1dd927dcbcea, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET | |
/master.rando2/ | |
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F | |
amz-sdk-invocation-id:42665700-8a06-44b3-9b47-1dd927dcbcea | |
amz-sdk-request:ttl=20220812T152811Z;attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-type:application/octet-stream | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
x-amz-date:20220812T152451Z | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date | |
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | |
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152451Z | |
20220812/us-east-1/s3/aws4_request | |
b0507684371063ddfc5ff4dd6ab9000619ee52f59de7230c4d3df288590c111e" | |
22/08/12 16:24:51 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:51 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:51 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F&fetch-owner=false HTTP/1.1 | |
22/08/12 16:24:51 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F&fetch-owner=false HTTP/1.1 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 42665700-8a06-44b3-9b47-1dd927dcbcea | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=ff64a0de247a9499296ab46858db31d9b09bc37dd52951ee12334699d9f86bb0 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152451Z | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Length: 0 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F&fetch-owner=false HTTP/1.1[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 42665700-8a06-44b3-9b47-1dd927dcbcea[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=ff64a0de247a9499296ab46858db31d9b09bc37dd52951ee12334699d9f86bb0[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152451Z[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 311c11a7-8c34-4e47-a231-4f55043280c7[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 311c11a7-8c34-4e47-a231-4f55043280c7[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:51 GMT[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Length: 287[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix8/_temporary/</Prefix></ListBucketResult>" | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Type: application/xml | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 311c11a7-8c34-4e47-a231-4f55043280c7 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 311c11a7-8c34-4e47-a231-4f55043280c7 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:51 GMT | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Length: 287 | |
22/08/12 16:24:51 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:51 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:51 GMT | |
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler | |
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2 | |
22/08/12 16:24:51 DEBUG request: Received successful response: 200, AWS Request ID: 311c11a7-8c34-4e47-a231-4f55043280c7 | |
22/08/12 16:24:51 DEBUG requestId: x-amzn-RequestId: not available | |
22/08/12 16:24:51 DEBUG requestId: AWS Request ID: 311c11a7-8c34-4e47-a231-4f55043280c7 | |
22/08/12 16:24:51 DEBUG requestId: AWS Extended Request ID: 311c11a7-8c34-4e47-a231-4f55043280c7 | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 23 | |
22/08/12 16:24:51 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[311c11a7-8c34-4e47-a231-4f55043280c7], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[1.769], ClientExecuteTime=[17.922], HttpClientSendRequestTime=[0.475], HttpRequestTime=[15.02], ApiCallLatency=[17.736], RequestSigningTime=[0.266], CredentialsRequestTime=[0.006, 0.003], HttpClientReceiveResponseTime=[13.818], | |
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST: duration 0:00.018s | |
22/08/12 16:24:51 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8/_temporary | |
22/08/12 16:24:51 DEBUG S3AFileSystem: Couldn't delete s3a://master.rando2/nonemptyprefix8/_temporary - does not exist: java.io.FileNotFoundException: No such file or directory: s3a://master.rando2/nonemptyprefix8/_temporary | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter ignored_errors by 1 with final value 1 | |
22/08/12 16:24:51 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc (nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc); needEmptyDirectory=true | |
22/08/12 16:24:51 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 10 | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 10 | |
22/08/12 16:24:51 DEBUG S3AFileSystem: HEAD nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc with change tracker null | |
22/08/12 16:24:51 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc Headers: (amz-sdk-invocation-id: b399a93c-20e2-c4fc-8e8f-fac69f36d235, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD | |
/master.rando2/nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc | |
amz-sdk-invocation-id:b399a93c-20e2-c4fc-8e8f-fac69f36d235 | |
amz-sdk-request:ttl=20220812T152811Z;attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-type:application/octet-stream | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
x-amz-date:20220812T152451Z | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date | |
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | |
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152451Z | |
20220812/us-east-1/s3/aws4_request | |
15860be2bf2524f1fb7674bc176ab7da26428ecc133b5794ec5a51628b2f32ed" | |
22/08/12 16:24:51 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:51 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:51 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc HTTP/1.1 | |
22/08/12 16:24:51 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc HTTP/1.1 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: b399a93c-20e2-c4fc-8e8f-fac69f36d235 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=9ab50c61b7c0bc5ae1f2416adefad6e998ea1f56834c75b2fe9d83ec8a9a7dc8 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152451Z | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc HTTP/1.1[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: b399a93c-20e2-c4fc-8e8f-fac69f36d235[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=9ab50c61b7c0bc5ae1f2416adefad6e998ea1f56834c75b2fe9d83ec8a9a7dc8[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152451Z[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 1708d265-86bd-4376-8203-97c31bb17675[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 1708d265-86bd-4376-8203-97c31bb17675[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:51 GMT[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Length: 290[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Type: application/xml | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 1708d265-86bd-4376-8203-97c31bb17675 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 1708d265-86bd-4376-8203-97c31bb17675 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:51 GMT | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Length: 290 | |
22/08/12 16:24:51 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:51 GMT | |
22/08/12 16:24:51 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 1708d265-86bd-4376-8203-97c31bb17675; S3 Extended Request ID: 1708d265-86bd-4376-8203-97c31bb17675; Proxy: null), S3 Extended Request ID: 1708d265-86bd-4376-8203-97c31bb17675 | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 24 | |
22/08/12 16:24:51 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 1708d265-86bd-4376-8203-97c31bb17675; S3 Extended Request ID: 1708d265-86bd-4376-8203-97c31bb17675; Proxy: null), S3 Extended Request ID: 1708d265-86bd-4376-8203-97c31bb17675], RequestType=[GetObjectMetadataRequest], AWSRequestID=[1708d265-86bd-4376-8203-97c31bb17675], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[14.502], HttpClientSendRequestTime=[0.494], HttpRequestTime=[13.466], ApiCallLatency=[14.307], RequestSigningTime=[0.262], CredentialsRequestTime=[0.006, 0.003], HttpClientReceiveResponseTime=[12.258], | |
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc/ delimiter=/ keys=2 requester pays=false | |
22/08/12 16:24:51 DEBUG S3AFileSystem: Starting: LIST | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 11 | |
22/08/12 16:24:51 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 4602ce4e-6b53-2da6-893c-d421f16d381c, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, ) | |
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET | |
/master.rando2/ | |
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix8%2F.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc%2F | |
amz-sdk-invocation-id:4602ce4e-6b53-2da6-893c-d421f16d381c | |
amz-sdk-request:ttl=20220812T152811Z;attempt=1;max=21 | |
amz-sdk-retry:0/0/500 | |
content-type:application/octet-stream | |
host:localhost:30600 | |
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
x-amz-date:20220812T152451Z | |
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date | |
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" | |
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256 | |
20220812T152451Z | |
20220812/us-east-1/s3/aws4_request | |
61f3f8c13b34f3760cc32e7971be556e54e9e09677dda81206ff2dec99d03626" | |
22/08/12 16:24:51 DEBUG RequestAddCookies: CookieSpec selected: default | |
22/08/12 16:24:51 DEBUG RequestAuthCache: Auth cache not set in the context | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000 | |
22/08/12 16:24:51 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc%2F&fetch-owner=false HTTP/1.1 | |
22/08/12 16:24:51 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc%2F&fetch-owner=false HTTP/1.1 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Host: localhost:30600 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 4602ce4e-6b53-2da6-893c-d421f16d381c | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=53e7ec49f932c5ef39cc90a689142c9d7de1139aa6c2e749b3259c6f256d863f | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152451Z | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Length: 0 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc%2F&fetch-owner=false HTTP/1.1[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 4602ce4e-6b53-2da6-893c-d421f16d381c[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=53e7ec49f932c5ef39cc90a689142c9d7de1139aa6c2e749b3259c6f256d863f[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152451Z[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 5076f85d-0a42-4fd9-b066-797a75c1dc78[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 5076f85d-0a42-4fd9-b066-797a75c1dc78[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:51 GMT[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Length: 328[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "[\r][\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]" | |
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc/</Prefix></ListBucketResult>" | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Type: application/xml | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 5076f85d-0a42-4fd9-b066-797a75c1dc78 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 5076f85d-0a42-4fd9-b066-797a75c1dc78 | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:51 GMT | |
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Length: 328 | |
22/08/12 16:24:51 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS | |
22/08/12 16:24:51 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:51 GMT | |
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds | |
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0 | |
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96] | |
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler | |
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2 | |
22/08/12 16:24:51 DEBUG request: Received successful response: 200, AWS Request ID: 5076f85d-0a42-4fd9-b066-797a75c1dc78 | |
22/08/12 16:24:51 DEBUG requestId: x-amzn-RequestId: not available | |
22/08/12 16:24:51 DEBUG requestId: AWS Request ID: 5076f85d-0a42-4fd9-b066-797a75c1dc78 | |
22/08/12 16:24:51 DEBUG requestId: AWS Extended Request ID: 5076f85d-0a42-4fd9-b066-797a75c1dc78 | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 25 | |
22/08/12 16:24:51 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[5076f85d-0a42-4fd9-b066-797a75c1dc78], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[1.598], ClientExecuteTime=[19.183], HttpClientSendRequestTime=[0.469], HttpRequestTime=[16.476], ApiCallLatency=[18.99], RequestSigningTime=[0.326], CredentialsRequestTime=[0.007, 0.004], HttpClientReceiveResponseTime=[15.417], | |
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST: duration 0:00.020s | |
22/08/12 16:24:51 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc | |
22/08/12 16:24:51 DEBUG S3AFileSystem: Couldn't delete s3a://master.rando2/nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc - does not exist: java.io.FileNotFoundException: No such file or directory: s3a://master.rando2/nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc | |
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter ignored_errors by 1 with final value 2 | |
Traceback (most recent call last): | |
File "/home/luke/pp/pachyderm/spark/spark.py", line 48, in <module> | |
df.coalesce(1).write.format("parquet").mode("overwrite").save("s3a://master.rando2/nonemptyprefix8") | |
File "/home/luke/pp/pachyderm/venv/lib/python3.10/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 968, in save | |
File "/home/luke/pp/pachyderm/venv/lib/python3.10/site-packages/pyspark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__ | |
File "/home/luke/pp/pachyderm/venv/lib/python3.10/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 190, in deco | |
File "/home/luke/pp/pachyderm/venv/lib/python3.10/site-packages/pyspark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value | |
py4j.protocol.Py4JJavaError: An error occurred while calling o243.save. | |
: org.apache.spark.SparkException: Job aborted. | |
at org.apache.spark.sql.errors.QueryExecutionErrors$.jobAbortedError(QueryExecutionErrors.scala:638) | |
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:278) | |
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:186) | |
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:113) | |
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:111) | |
at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:125) | |
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98) | |
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109) | |
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169) | |
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95) | |
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779) | |
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64) | |
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98) | |
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94) | |
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584) | |
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176) | |
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584) | |
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30) | |
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267) | |
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263) | |
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30) | |
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30) | |
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560) | |
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94) | |
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81) | |
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79) | |
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:116) | |
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:860) | |
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390) | |
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:363) | |
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239) | |
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) | |
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) | |
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
at java.base/java.lang.reflect.Method.invoke(Method.java:566) | |
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) | |
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) | |
at py4j.Gateway.invoke(Gateway.java:282) | |
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) | |
at py4j.commands.CallCommand.execute(CallCommand.java:79) | |
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) | |
at py4j.ClientServerConnection.run(ClientServerConnection.java:106) | |
at java.base/java.lang.Thread.run(Thread.java:829) | |
Caused by: java.io.FileNotFoundException: No such file or directory: s3a://master.rando2/nonemptyprefix8/_temporary/0 | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3866) | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3688) | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:3300) | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$null$20(S3AFileSystem.java:3264) | |
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:117) | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listStatus$21(S3AFileSystem.java:3263) | |
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499) | |
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444) | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337) | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356) | |
at org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:3262) | |
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1972) | |
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:2014) | |
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getAllCommittedTaskPaths(FileOutputCommitter.java:334) | |
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJobInternal(FileOutputCommitter.java:404) | |
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:377) | |
at org.apache.parquet.hadoop.ParquetOutputCommitter.commitJob(ParquetOutputCommitter.java:48) | |
at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:192) | |
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$25(FileFormatWriter.scala:267) | |
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) | |
at org.apache.spark.util.Utils$.timeTakenMs(Utils.scala:642) | |
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:267) | |
... 41 more | |
22/08/12 16:24:51 INFO SparkContext: Invoking stop() from shutdown hook | |
22/08/12 16:24:51 INFO SparkUI: Stopped Spark web UI at http://10.1.255.235:4040 | |
22/08/12 16:24:51 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! | |
22/08/12 16:24:52 INFO MemoryStore: MemoryStore cleared | |
22/08/12 16:24:52 INFO BlockManager: BlockManager stopped | |
22/08/12 16:24:52 INFO BlockManagerMaster: BlockManagerMaster stopped | |
22/08/12 16:24:52 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! | |
22/08/12 16:24:52 INFO SparkContext: Successfully stopped SparkContext | |
22/08/12 16:24:52 INFO ShutdownHookManager: Shutdown hook called | |
22/08/12 16:24:52 INFO ShutdownHookManager: Deleting directory /tmp/spark-675a0fcb-380d-4f5f-8991-c288769c744f | |
22/08/12 16:24:52 INFO ShutdownHookManager: Deleting directory /tmp/spark-7ddbe69a-a202-4991-b257-71af8961db3e/pyspark-3de0ca21-97fc-444f-ae94-cbae398755d5 | |
22/08/12 16:24:52 INFO ShutdownHookManager: Deleting directory /tmp/spark-7ddbe69a-a202-4991-b257-71af8961db3e | |
22/08/12 16:24:52 DEBUG FileSystem: FileSystem.close() by method: org.apache.hadoop.fs.FilterFileSystem.close(FilterFileSystem.java:529)); Key: (luke (auth:SIMPLE))@file://; URI: file:///; Object Identity Hash: 60fc0331 | |
22/08/12 16:24:52 DEBUG FileSystem: FileSystem.close() by method: org.apache.hadoop.fs.RawLocalFileSystem.close(RawLocalFileSystem.java:759)); Key: null; URI: file:///; Object Identity Hash: 3221f19 | |
22/08/12 16:24:52 DEBUG S3AFileSystem: Filesystem s3a://master.rando2 is closed | |
22/08/12 16:24:52 DEBUG S3AFileSystem: IOStatistics: counters=((stream_read_exceptions=0) (object_metadata_request=10) (op_hsync=0) (object_multipart_aborted.failures=0) (s3guard_metadatastore_record_reads=0) (op_glob_status.failures=0) (op_list_located_status=0) (stream_read_fully_operations=0) (stream_read_operations=0) (files_copied=0) (object_continue_list_request.failures=0) (op_get_file_checksum.failures=0) (multipart_upload_part_put=0) (store_exists_probe.failures=0) (action_executor_acquired=0) (store_io_retry=0) (object_put_request=2) (multipart_upload_aborted=0) (multipart_instantiated=0) (stream_read_operations_incomplete=0) (object_put_request.failures=0) (stream_read_seek_policy_changed=0) (object_put_request_completed=2) (fake_directories_created=0) (stream_read_bytes_discarded_in_close=0) (s3guard_metadatastore_record_deletes=0) (stream_write_exceptions=0) (object_continue_list_request=0) (delegation_tokens_issued=0) (directories_deleted=0) (op_access.failures=0) (committer_bytes_uploaded=0) (committer_commit_job.failures=0) (op_get_delegation_token=0) (op_exists.failures=2) (op_list_files=0) (stream_read_seek_bytes_discarded=0) (files_copied_bytes=0) (op_create.failures=0) (committer_stage_file_upload=0) (s3guard_metadatastore_initialization=0) (op_get_file_checksum=0) (op_mkdirs.failures=0) (stream_read_version_mismatches=0) (multipart_upload_completed=0) (op_access=0) (object_delete_request=0) (stream_read_opened=0) (audit_span_creation=0) (op_exists=2) (s3guard_metadatastore_retry=0) (op_copy_from_local_file=0) (committer_tasks_failed=0) (stream_write_block_uploads=1) (committer_commit_job=0) (multipart_upload_list=0) (multipart_upload_part_put_bytes=0) (stream_write_total_data=1436) (multipart_upload_started=0) (object_bulk_delete_request=2) (committer_magic_files_created=0) (files_created=1) (stream_read_seek_operations=0) (op_get_content_summary=0) (op_create_non_recursive=0) (stream_read_bytes_discarded_in_abort=0) (stream_read_total_bytes=0) (committer_jobs_failed=0) (op_copy_from_local_file.failures=0) (object_multipart_aborted=0) (stream_write_block_uploads_committed=0) (op_is_file.failures=0) (op_is_directory.failures=0) (s3guard_metadatastore_put_path_request=0) (stream_write_queue_duration=0) (op_is_file=0) (committer_commits_created=0) (committer_tasks_completed=0) (action_executor_acquired.failures=0) (object_delete_objects=7) (ignored_errors=2) (stream_write_exceptions_completing_upload=0) (committer_bytes_committed=0) (stream_read_close_operations=0) (op_list_status=1) (object_copy_requests=0) (fake_directories_deleted=7) (stream_read_bytes=0) (s3guard_metadatastore_record_writes=0) (store_exists_probe=0) (op_hflush=0) (op_xattr_get_named=0) (object_multipart_initiated.failures=0) (action_http_get_request=0) (op_is_directory=0) (action_http_get_request.failures=0) (op_xattr_get_named_map.failures=0) (committer_materialize_file=0) (op_open=0) (s3guard_metadatastore_authoritative_directories_updated=0) (stream_read_bytes_backwards_on_seek=0) (audit_failure=0) (committer_commits.failures=0) (op_delete.failures=0) (files_delete_rejected=0) (files_deleted=0) (action_http_head_request=10) (op_rename=0) (s3guard_metadatastore_throttled=0) (stream_write_total_time=0) (committer_commits_completed=0) (op_get_file_status=1) (object_put_bytes=718) (stream_read_seek_bytes_skipped=0) (op_abort=0) (object_list_request.failures=0) (stream_read_seek_forward_operations=0) (op_create=1) (store_io_throttled=0) (multipart_upload_abort_under_path_invoked=0) (store_io_request=25) (object_list_request=11) (op_xattr_get_named.failures=0) (op_mkdirs=1) (committer_commits_reverted=0) (op_rename.failures=0) (op_delete=0) (op_list_status.failures=1) (op_xattr_get_map=0) (object_delete_request.failures=0) (op_xattr_get_named_map=0) (op_xattr_list.failures=0) (op_get_delegation_token.failures=0) (committer_commits_aborted=0) (object_bulk_delete_request.failures=0) (op_get_file_status.failures=1) (committer_stage_file_upload.failures=0) (delegation_tokens_issued.failures=0) (op_get_content_summary.failures=0) (op_glob_status=0) (stream_write_queue_duration.failures=0) (stream_write_bytes=718) (op_abort.failures=0) (committer_materialize_file.failures=0) (multipart_upload_list.failures=0) (op_xattr_list=0) (audit_request_execution=0) (object_multipart_initiated=0) (op_list_files.failures=0) (stream_write_block_uploads_aborted=0) (op_xattr_get_map.failures=0) (object_select_requests=0) (directories_created=1) (stream_aborted=0) (stream_read_seek_backward_operations=0) (committer_jobs_completed=0) (action_http_head_request.failures=0) (stream_read_closed=0) (audit_access_check_failure=0)); | |
gauges=((stream_write_block_uploads_active=0) (object_put_bytes_pending=0) (object_put_request_active=0) (client_side_encryption_enabled=0) (stream_write_block_uploads_pending=1) (stream_write_block_uploads_data_pending=0)); | |
minimums=((op_get_content_summary.failures.min=-1) (object_bulk_delete_request.failures.min=-1) (op_list_status.min=-1) (op_get_file_status.min=-1) (op_delete.failures.min=-1) (op_exists.failures.min=43) (op_copy_from_local_file.min=-1) (delegation_tokens_issued.failures.min=-1) (op_xattr_get_map.failures.min=-1) (op_is_directory.failures.min=-1) (committer_commit_job.min=-1) (multipart_upload_list.min=-1) (op_xattr_list.min=-1) (op_delete.min=-1) (op_exists.min=-1) (op_glob_status.min=-1) (object_put_request.failures.min=-1) (action_http_get_request.min=-1) (object_continue_list_request.failures.min=-1) (object_delete_request.failures.min=-1) (object_multipart_aborted.failures.min=-1) (object_multipart_initiated.min=-1) (committer_materialize_file.failures.min=-1) (store_exists_probe.min=-1) (op_xattr_get_named.min=-1) (object_multipart_aborted.min=-1) (op_access.min=-1) (store_exists_probe.failures.min=-1) (op_mkdirs.min=451) (op_is_file.min=-1) (op_is_file.failures.min=-1) (multipart_upload_list.failures.min=-1) (delegation_tokens_issued.min=-1) (op_xattr_list.failures.min=-1) (object_list_request.failures.min=-1) (action_http_get_request.failures.min=-1) (op_get_file_checksum.failures.min=-1) (op_rename.failures.min=-1) (op_glob_status.failures.min=-1) (action_http_head_request.failures.min=-1) (op_create.failures.min=-1) (op_list_files.failures.min=-1) (stream_write_queue_duration.min=-1) (action_executor_acquired.failures.min=-1) (op_xattr_get_named_map.min=-1) (op_abort.min=-1) (op_get_delegation_token.min=-1) (op_list_status.failures.min=57) (op_get_file_checksum.min=-1) (op_list_files.min=-1) (committer_stage_file_upload.min=-1) (op_abort.failures.min=-1) (op_get_delegation_token.failures.min=-1) (stream_write_queue_duration.failures.min=-1) (op_xattr_get_named.failures.min=-1) (committer_stage_file_upload.failures.min=-1) (op_rename.min=-1) (op_copy_from_local_file.failures.min=-1) (object_multipart_initiated.failures.min=-1) (op_create.min=98) (op_access.failures.min=-1) (action_http_head_request.min=14) (object_delete_request.min=-1) (committer_materialize_file.min=-1) (op_xattr_get_map.min=-1) (committer_commit_job.failures.min=-1) (object_continue_list_request.min=-1) (object_list_request.min=18) (op_get_content_summary.min=-1) (op_xattr_get_named_map.failures.min=-1) (object_put_request.min=189) (op_get_file_status.failures.min=782) (op_is_directory.min=-1) (object_bulk_delete_request.min=96) (action_executor_acquired.min=-1) (op_mkdirs.failures.min=-1)); | |
maximums=((multipart_upload_list.failures.max=-1) (op_list_files.failures.max=-1) (committer_materialize_file.failures.max=-1) (op_get_file_status.failures.max=782) (object_multipart_initiated.max=-1) (object_delete_request.failures.max=-1) (op_access.failures.max=-1) (op_xattr_list.failures.max=-1) (op_get_delegation_token.max=-1) (op_glob_status.max=-1) (op_get_delegation_token.failures.max=-1) (object_multipart_initiated.failures.max=-1) (op_rename.max=-1) (op_get_file_status.max=-1) (op_exists.max=-1) (stream_write_queue_duration.failures.max=-1) (op_glob_status.failures.max=-1) (op_list_status.failures.max=57) (op_is_directory.failures.max=-1) (op_xattr_get_named.max=-1) (action_http_get_request.failures.max=-1) (op_get_file_checksum.max=-1) (action_executor_acquired.failures.max=-1) (op_get_content_summary.max=-1) (object_continue_list_request.failures.max=-1) (op_mkdirs.failures.max=-1) (op_xattr_get_named_map.max=-1) (object_multipart_aborted.failures.max=-1) (committer_commit_job.max=-1) (op_is_file.max=-1) (committer_stage_file_upload.failures.max=-1) (op_delete.failures.max=-1) (delegation_tokens_issued.max=-1) (op_copy_from_local_file.max=-1) (object_put_request.max=893) (op_delete.max=-1) (object_bulk_delete_request.failures.max=-1) (op_mkdirs.max=451) (committer_materialize_file.max=-1) (committer_commit_job.failures.max=-1) (op_xattr_get_map.failures.max=-1) (op_xattr_get_named_map.failures.max=-1) (object_put_request.failures.max=-1) (op_is_file.failures.max=-1) (object_list_request.failures.max=-1) (object_continue_list_request.max=-1) (op_create.max=98) (object_multipart_aborted.max=-1) (store_exists_probe.failures.max=-1) (action_http_head_request.failures.max=-1) (object_list_request.max=82) (op_get_content_summary.failures.max=-1) (op_xattr_list.max=-1) (delegation_tokens_issued.failures.max=-1) (object_bulk_delete_request.max=120) (op_copy_from_local_file.failures.max=-1) (store_exists_probe.max=-1) (action_executor_acquired.max=-1) (multipart_upload_list.max=-1) (op_xattr_get_map.max=-1) (op_xattr_get_named.failures.max=-1) (op_get_file_checksum.failures.max=-1) (action_http_head_request.max=756) (op_list_status.max=-1) (op_exists.failures.max=239) (op_access.max=-1) (op_is_directory.max=-1) (committer_stage_file_upload.max=-1) (op_abort.failures.max=-1) (op_create.failures.max=-1) (op_list_files.max=-1) (stream_write_queue_duration.max=-1) (op_abort.max=-1) (action_http_get_request.max=-1) (op_rename.failures.max=-1) (object_delete_request.max=-1)); | |
means=((op_exists.failures.mean=(samples=2, sum=282, mean=141.0000)) (committer_commit_job.failures.mean=(samples=0, sum=0, mean=0.0000)) (committer_stage_file_upload.mean=(samples=0, sum=0, mean=0.0000)) (committer_commit_job.mean=(samples=0, sum=0, mean=0.0000)) (delegation_tokens_issued.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_is_directory.mean=(samples=0, sum=0, mean=0.0000)) (action_http_get_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_get_file_checksum.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_xattr_get_named.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_is_file.mean=(samples=0, sum=0, mean=0.0000)) (op_xattr_get_named_map.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_initiated.mean=(samples=0, sum=0, mean=0.0000)) (object_put_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_xattr_list.mean=(samples=0, sum=0, mean=0.0000)) (op_get_file_status.mean=(samples=0, sum=0, mean=0.0000)) (op_is_directory.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_mkdirs.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_aborted.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_initiated.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_put_request.mean=(samples=2, sum=1082, mean=541.0000)) (op_access.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_abort.mean=(samples=0, sum=0, mean=0.0000)) (op_delete.mean=(samples=0, sum=0, mean=0.0000)) (store_exists_probe.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_glob_status.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_copy_from_local_file.mean=(samples=0, sum=0, mean=0.0000)) (op_rename.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_list_files.mean=(samples=0, sum=0, mean=0.0000)) (op_abort.failures.mean=(samples=0, sum=0, mean=0.0000)) (store_exists_probe.mean=(samples=0, sum=0, mean=0.0000)) (object_delete_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (stream_write_queue_duration.mean=(samples=0, sum=0, mean=0.0000)) (op_create.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_is_file.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_list_status.failures.mean=(samples=1, sum=57, mean=57.0000)) (object_continue_list_request.mean=(samples=0, sum=0, mean=0.0000)) (op_get_file_checksum.mean=(samples=0, sum=0, mean=0.0000)) (op_xattr_get_named.mean=(samples=0, sum=0, mean=0.0000)) (committer_materialize_file.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_xattr_list.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_copy_from_local_file.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_xattr_get_named_map.mean=(samples=0, sum=0, mean=0.0000)) (op_rename.mean=(samples=0, sum=0, mean=0.0000)) (multipart_upload_list.mean=(samples=0, sum=0, mean=0.0000)) (committer_materialize_file.mean=(samples=0, sum=0, mean=0.0000)) (op_xattr_get_map.failures.mean=(samples=0, sum=0, mean=0.0000)) (multipart_upload_list.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_get_file_status.failures.mean=(samples=1, sum=782, mean=782.0000)) (op_exists.mean=(samples=0, sum=0, mean=0.0000)) (op_mkdirs.mean=(samples=1, sum=451, mean=451.0000)) (object_list_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_bulk_delete_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (action_http_head_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (action_executor_acquired.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_aborted.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_get_delegation_token.failures.mean=(samples=0, sum=0, mean=0.0000)) (committer_stage_file_upload.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_get_content_summary.mean=(samples=0, sum=0, mean=0.0000)) (op_get_content_summary.failures.mean=(samples=0, sum=0, mean=0.0000)) (action_executor_acquired.mean=(samples=0, sum=0, mean=0.0000)) (op_list_status.mean=(samples=0, sum=0, mean=0.0000)) (op_delete.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_create.mean=(samples=1, sum=98, mean=98.0000)) (object_delete_request.mean=(samples=0, sum=0, mean=0.0000)) (stream_write_queue_duration.failures.mean=(samples=0, sum=0, mean=0.0000)) (delegation_tokens_issued.mean=(samples=0, sum=0, mean=0.0000)) (object_bulk_delete_request.mean=(samples=2, sum=216, mean=108.0000)) (object_continue_list_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_xattr_get_map.mean=(samples=0, sum=0, mean=0.0000)) (op_glob_status.mean=(samples=0, sum=0, mean=0.0000)) (action_http_get_request.mean=(samples=0, sum=0, mean=0.0000)) (object_list_request.mean=(samples=11, sum=324, mean=29.4545)) (op_get_delegation_token.mean=(samples=0, sum=0, mean=0.0000)) (op_access.mean=(samples=0, sum=0, mean=0.0000)) (action_http_head_request.mean=(samples=10, sum=1047, mean=104.7000)) (op_list_files.failures.mean=(samples=0, sum=0, mean=0.0000))); | |
22/08/12 16:24:52 DEBUG FileSystem: FileSystem.close() by method: org.apache.hadoop.fs.s3a.S3AFileSystem.close(S3AFileSystem.java:4069)); Key: (luke (auth:SIMPLE))@s3a://master.rando2; URI: s3a://master.rando2; Object Identity Hash: 17051619 | |
22/08/12 16:24:52 DEBUG PoolingHttpClientConnectionManager: Connection manager is shutting down | |
22/08/12 16:24:52 DEBUG IdleConnectionReaper: Reaper thread: | |
java.lang.InterruptedException: sleep interrupted | |
at java.base/java.lang.Thread.sleep(Native Method) | |
at com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:188) | |
22/08/12 16:24:52 DEBUG IdleConnectionReaper: Shutting down reaper thread. | |
22/08/12 16:24:52 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: Close connection | |
22/08/12 16:24:52 DEBUG PoolingHttpClientConnectionManager: Connection manager shut down | |
22/08/12 16:24:52 DEBUG S3AFileSystem: Gracefully shutting down executor service. Waiting max 30 SECONDS | |
22/08/12 16:24:52 DEBUG S3AFileSystem: Succesfully shutdown executor service | |
22/08/12 16:24:52 DEBUG S3AFileSystem: Gracefully shutting down executor service. Waiting max 30 SECONDS | |
22/08/12 16:24:52 DEBUG S3AFileSystem: Succesfully shutdown executor service | |
22/08/12 16:24:52 DEBUG MBeans: Unregistering Hadoop:service=s3a-file-system,name=S3AMetrics1-master.rando2 | |
22/08/12 16:24:52 DEBUG S3AInstrumentation: Shutting down metrics publisher | |
22/08/12 16:24:52 DEBUG MetricsSystemImpl: refCount=1 | |
22/08/12 16:24:52 INFO MetricsSystemImpl: Stopping s3a-file-system metrics system... | |
22/08/12 16:24:52 DEBUG MBeans: Unregistering Hadoop:service=s3a-file-system,name=MetricsSystem,sub=Stats | |
22/08/12 16:24:52 INFO MetricsSystemImpl: s3a-file-system metrics system stopped. | |
22/08/12 16:24:52 DEBUG MBeans: Unregistering Hadoop:service=s3a-file-system,name=MetricsSystem,sub=Control | |
22/08/12 16:24:52 INFO MetricsSystemImpl: s3a-file-system metrics system shutdown complete. | |
22/08/12 16:24:52 DEBUG SignerManager: Unregistering fs from 0 initializers | |
22/08/12 16:24:52 DEBUG AbstractService: Service: NoopAuditManagerS3A entered state STOPPED | |
22/08/12 16:24:52 DEBUG CompositeService: NoopAuditManagerS3A: stopping services, size=1 | |
22/08/12 16:24:52 DEBUG CompositeService: Stopping service #0: Service NoopAuditor in state NoopAuditor: STARTED | |
22/08/12 16:24:52 DEBUG AbstractService: Service: NoopAuditor entered state STOPPED | |
22/08/12 16:24:52 DEBUG S3AFileSystem: Closing AWSCredentialProviderList[refcount= 1: [TemporaryAWSCredentialsProvider, SimpleAWSCredentialsProvider, EnvironmentVariableCredentialsProvider, org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider@717fe440] last provider: SimpleAWSCredentialsProvider | |
22/08/12 16:24:52 DEBUG AWSCredentialProviderList: Closing AWSCredentialProviderList[refcount= 0: [TemporaryAWSCredentialsProvider, SimpleAWSCredentialsProvider, EnvironmentVariableCredentialsProvider, org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider@717fe440] last provider: SimpleAWSCredentialsProvider | |
22/08/12 16:24:52 DEBUG S3AFileSystem: Statistics for s3a://master.rando2: counters=((action_http_head_request=10) | |
(directories_created=1) | |
(fake_directories_deleted=7) | |
(files_created=1) | |
(ignored_errors=2) | |
(object_bulk_delete_request=2) | |
(object_delete_objects=7) | |
(object_list_request=11) | |
(object_metadata_request=10) | |
(object_put_bytes=718) | |
(object_put_request=2) | |
(object_put_request_completed=2) | |
(op_create=1) | |
(op_exists=2) | |
(op_exists.failures=2) | |
(op_get_file_status=1) | |
(op_get_file_status.failures=1) | |
(op_list_status=1) | |
(op_list_status.failures=1) | |
(op_mkdirs=1) | |
(store_io_request=25) | |
(stream_write_block_uploads=1) | |
(stream_write_bytes=718) | |
(stream_write_total_data=1436)); | |
gauges=((stream_write_block_uploads_pending=1)); | |
minimums=((action_http_head_request.min=14) | |
(object_bulk_delete_request.min=96) | |
(object_list_request.min=18) | |
(object_put_request.min=189) | |
(op_create.min=98) | |
(op_exists.failures.min=43) | |
(op_get_file_status.failures.min=782) | |
(op_list_status.failures.min=57) | |
(op_mkdirs.min=451)); | |
maximums=((action_http_head_request.max=756) | |
(object_bulk_delete_request.max=120) | |
(object_list_request.max=82) | |
(object_put_request.max=893) | |
(op_create.max=98) | |
(op_exists.failures.max=239) | |
(op_get_file_status.failures.max=782) | |
(op_list_status.failures.max=57) | |
(op_mkdirs.max=451)); | |
means=((action_http_head_request.mean=(samples=10, sum=1047, mean=104.7000)) | |
(object_bulk_delete_request.mean=(samples=2, sum=216, mean=108.0000)) | |
(object_list_request.mean=(samples=11, sum=324, mean=29.4545)) | |
(object_put_request.mean=(samples=2, sum=1082, mean=541.0000)) | |
(op_create.mean=(samples=1, sum=98, mean=98.0000)) | |
(op_exists.failures.mean=(samples=2, sum=282, mean=141.0000)) | |
(op_get_file_status.failures.mean=(samples=1, sum=782, mean=782.0000)) | |
(op_list_status.failures.mean=(samples=1, sum=57, mean=57.0000)) | |
(op_mkdirs.mean=(samples=1, sum=451, mean=451.0000))); | |
22/08/12 16:24:52 DEBUG ShutdownHookManager: Completed shutdown in 0.067 seconds; Timeouts: 0 | |
22/08/12 16:24:52 DEBUG ShutdownHookManager: ShutdownHookManager completed shutdown. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment