There's multiple ways to generate a kfp component:
- from python function
- from file/text
You'll use component.load_component_from_<file|text>
when you need to interface with a command line tool.
If you would like to work in the same environment as jupyterlab, check out Theia, the cloud IDE. The following highlights how we can install theia in your jupyter notebook's container.
For more information here's a post describing how Google Cloud Platform is using Theia as their default cloud IDE: https://cloud.google.com/blog/products/application-development/introducing-cloud-shell-editor.
Presto | Spark |
---|---|
array_join(array[year, month, day], '-', 'NA') |
CONCAT_WS('-', col1, col2, col3) |
DATE_ADD('day', -7, date '2021-07-01') |
date_sub('2021-07-01', 7) |
array_join('one', 'two', 'three')[1] # one indexed |
array_join('one', 'two', 'three')[0] # zero indexed |
-- ALTER TABLE schema.table DROP IF EXISTS PARTITION (year='2021', month='01', day='11', hour='01') | |
ALTER TABLE pricing.demand_tbl ADD | |
PARTITION (year='2021', month='01', day='11', hour='01') | |
LOCATION 's3://datascience-bucket/wesley.goi/data/pricing/demand_tbl/year=2021/month=01/day=11/hour=01' |
Installing kubeflow on localmachine is not a simple task. Documentation on the official website might be outdated. At the time of writing, the solutions suggested include miniKF and microk8s. The later sets up GPU passthrough effortlessly.
Use the nbconvert / jupyter-nbconvert CLI to converting IPYNB to HTML
$ NOTEBOOK=file_name.ipynb
$ jupyter-nbconvert --to html $NOTEBOOK
You can add tags to hide cells, eg. code or outputs which you do not want in shown in the HTML
You got your machine, now you want to start working immediately on your 1st day at work. What do you do?
curl -O https://iterm2.com/downloads/stable/iTerm2-3_3_12.zip
Step 1: Install gcsFUSE (linux)
export GCSFUSE_REPO=gcsfuse-`lsb_release -c -s`
echo "deb http://packages.cloud.google.com/apt $GCSFUSE_REPO main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo apt-get install gcsfuse