Skip to content

Instantly share code, notes, and snippets.

@michellemay
michellemay / hdfs_to_s3.scala
Last active May 12, 2017 15:02
HDFS to S3 using spark and hadoop
import org.apache.hadoop.fs.{FileSystem, Path, FileUtil}
def elapsed[A](f: => A): (A, Double) = {
val s = System.nanoTime
val ret = f
val elapsed = (System.nanoTime - s) / 1e9
(ret, elapsed)
}
val inputPathStr = "hdfs://namenode/user/hadoop/archive/"