It's great for beginners. Then it turns into a mess.
import copy | |
# write to a path using the Hudi format | |
def hudi_write(df, schema, table, path, mode, hudi_options): | |
hudi_options = { | |
"hoodie.datasource.write.recordkey.field": "recordkey", | |
"hoodie.datasource.write.precombine.field": "precombine_field", | |
"hoodie.datasource.write.partitionpath.field": "partitionpath_field", | |
"hoodie.datasource.write.operation": "write_operaion", | |
"hoodie.datasource.write.table.type": "table_type", |
Using our beloved docker and docker-compose, we can very quickly bring up an Apache Airflow instance on our mac.
About the only thing you need to customize in this docker-compose.yml file is the volumes section. This will tell docker to map the given directory containing your Airflow DAGs/plugins to container file system.
version: '3'
services:
#!/usr/bin/python | |
# $Id: $ | |
# Converts Oracle, SQL-Server, and other DDL to Snowflake DDL | |
def usage(): | |
print """\ | |
# Usage: sql2sf.py input-file [output-file] | |
""" |
sudo add-apt-repository -y ppa:apt-fast/stable | |
sudo add-apt-repository -y ppa:graphics-drivers/ppa | |
sudo apt-get update | |
sudo apt-get -y install apt-fast | |
# prompts | |
sudo apt-fast -y upgrade | |
sudo apt-fast install -y python3-pip ubuntu-drivers-common libvorbis-dev libflac-dev libsndfile-dev cmake build-essential libgflags-dev libgoogle-glog-dev libgtest-dev google-mock zlib1g-dev libeigen3-dev libboost-all-dev libasound2-dev libogg-dev libtool libfftw3-dev libbz2-dev liblzma-dev libgoogle-glog0v5 gcc-6 gfortran-6 g++-6 doxygen graphviz libsox-fmt-all parallel exuberant-ctags vim-nox python-powerline python3-pip ack lsyncd | |
sudo apt-fast install -y tigervnc-standalone-server firefox mesa-common-dev |
$ java -version
java version "1.7.0_171"
OpenJDK Runtime Environment (rhel-2.6.13.0.el7_4-x86_64 u171-b01)
OpenJDK 64-Bit Server VM (build 24.171-b01, mixed mode)
For scala to be set up JDK 8 or greater version is required
if JDK/OpenJDK version is less than 1.8 then follow the below steps
/!\ Be very carrefull in your setup : any misconfiguration make all the git config to fail silently ! Go trought this guide step by step and it should be fine 😉
- Generate your SSH keys as per your git provider documentation.
- Add each public SSH keys to your git providers acounts.
- In your
~/.ssh/config
, set each ssh key for each repository as in this exemple:
# | |
# Some constants | |
# | |
aws_profile = "your_profile" | |
aws_region = "your_region" | |
s3_bucket = "your_bucket" | |
# | |
# Reading environment variables from aws credential file | |
# |
http://www.oreilly.com/programming/free/files/microservices-for-java-developers.pdf
http://www.oreilly.com/programming/free/files/microservices-for-java-developers.epub
http://www.oreilly.com/programming/free/files/microservices-for-java-developers.mobi
http://www.oreilly.com/programming/free/files/modern-java-ee-design-patterns.pdf
http://www.oreilly.com/programming/free/files/modern-java-ee-design-patterns.epub
http://www.oreilly.com/programming/free/files/modern-java-ee-design-patterns.mobi
# Install R + RStudio on Ubuntu 14.04 | |
sudo apt-key adv –keyserver keyserver.ubuntu.com –recv-keys E084DAB9 | |
# Ubuntu 12.04: precise | |
# Ubuntu 14.04: trusty | |
# Ubuntu 16.04: xenial | |
# Basic format of next line deb https://<my.favorite.cran.mirror>/bin/linux/ubuntu <enter your ubuntu version>/ | |
sudo add-apt-repository 'deb https://ftp.ussg.iu.edu/CRAN/bin/linux/ubuntu trusty/' | |
sudo apt-get update |