Viacheslav Baranov SlavikBaranov
SlavikBaranov
/ cartesian-product.sc
Created
August 28, 2017 09:01
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| // These two lines are only for code completion in IDEA, | |
| // don't paste them into spark-shell | |
| val spark: org.apache.spark.sql.SparkSession | |
| import spark.implicits._ | |
| import scala.util.Random | |
| @transient val sc = spark.sparkContext | |
| // # of rows 1M-10M. Running time is quadratic, |
SlavikBaranov
/ r_packages.txt
Created
April 20, 2017 08:30
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| data.table lubridate ggplot2 h2o lift plotly party Hmisc caret caretEnsemble markdown knitr RColorBrewer revealjs rmarkdown boot corrplot dplyr forecast foreign ggplot2 ggthemes MASS Matrix matrixStats mclust moments plotrix plotROC ROCR plyr psych reshape2 rlm sqldf stringr tidyr Weighted.Desc.Stat xgboost elasticnet glmnet party Hmisc |
SlavikBaranov
/ white-helv.css
Created
November 23, 2016 16:08
Helvetica theme for RevealJS presentation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| /** | |
| * White theme for reveal.js. This is the opposite of the 'black' theme. | |
| * | |
| * By Hakim El Hattab, http://hakim.se | |
| */ | |
| section.has-dark-background, section.has-dark-background h1, section.has-dark-background h2, section.has-dark-background h3, section.has-dark-background h4, section.has-dark-background h5, section.has-dark-background h6 { | |
| color: #fff; } | |
| /********************************************* | |
| * GLOBAL STYLES |
SlavikBaranov
/ spark_join.sc
Created
October 18, 2016 13:44
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| // These two lines are only for code completion in IDEA, | |
| // don't paste them into spark-shell | |
| val spark: org.apache.spark.sql.SparkSession | |
| import spark.implicits._ | |
| import scala.util.Random | |
| @transient val sc = spark.sparkContext | |
| // # of rows 1M-10M. Running time is quadratic, |
SlavikBaranov
/ AsyncStream.scala
Created
June 16, 2016 20:33
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| // Requires "com.twitter" %% "util-core" % "6.34.0" | |
| import com.twitter.concurrent.AsyncStream | |
| import com.twitter.util.{Await, Future} | |
| object TestStream extends App { | |
| def foo(read: () => Future[String], write: String => Future[Unit]): Future[Unit] = { | |
| def stream: AsyncStream[String] = AsyncStream.fromFuture(read()).flatMap(_ +:: stream) |
SlavikBaranov
/ parquet_to_csv.scala
Created
April 6, 2016 08:25
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| /* | |
| 1. Download spark-csv & apache commons-csv | |
| http://mvnrepository.com/artifact/com.databricks/spark-csv_2.10/1.3.0 | |
| http://mvnrepository.com/artifact/org.apache.commons/commons-csv/1.2 | |
| 2. Run spark-shell with command: | |
| spark-shell --jars /<path to>/spark-csv_2.10-1.3.0.jar,/<path to>/commons-csv-1.2.jar | |
| */ |