Skip to content

Instantly share code, notes, and snippets.

View fifar's full-sized avatar

Huihua Zhang fifar

12 followers · 23 following
View GitHub Profile
@fifar
fifar / v2.4.4_v3.0.0_KryoSerializer.diff
Created August 5, 2020 13:23
Changes of core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala between v2.4.4 and v3.0.0
diff --git a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala b/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
index 7271004..cdaab59 100644
--- a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
+++ b/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
@@ -30,6 +30,7 @@ import scala.util.control.NonFatal
import com.esotericsoftware.kryo.{Kryo, KryoException, Serializer => KryoClassSerializer}
import com.esotericsoftware.kryo.io.{Input => KryoInput, Output => KryoOutput}
import com.esotericsoftware.kryo.io.{UnsafeInput => KryoUnsafeInput, UnsafeOutput => KryoUnsafeOutput}
+import com.esotericsoftware.kryo.pool.{KryoCallback, KryoFactory, KryoPool}
import com.esotericsoftware.kryo.serializers.{JavaSerializer => KryoJavaSerializer}
@fifar
fifar / consul.sh
Created July 17, 2020 13:18
#!/bin/sh
### BEGIN INIT INFO
# Provides: consul
# Required-Start: $local_fs $remote_fs
# Required-Stop: $local_fs $remote_fs
# Default-Start: 2 3 4 5
# Default-Stop: S 0 1 6
# Short-Description: Consul service discovery framework
# Description: Healthchecks local services and registers
# them in a central consul database.
@fifar
fifar / jump
Created June 29, 2020 08:07
#!/bin/bash
# Black 0;30 Dark Gray 1;30
# Red 0;31 Light Red 1;31
# Green 0;32 Light Green 1;32
# Brown/Orange 0;33 Yellow 1;33
# Blue 0;34 Light Blue 1;34
# Purple 0;35 Light Purple 1;35
# Cyan 0;36 Light Cyan 1;36
# Light Gray 0;37 White 1;37
@fifar
fifar / node_affinity.yaml
Created April 20, 2020 06:49
node affinity for CPU and GPU pods
singleuser:
profileList:
- display_name: "Default"
description: |
Tubi Data Runtime
default: True
kubespawner_override:
image: <private-docker-registry>/tubi-data-runtime
node_affinity_preferred:
- weight: 1
@fifar
fifar / tensorboard_notebook.diff
Created April 13, 2020 04:17
TensorBoard 2.2.0 patch
diff --git a/tensorboard/notebook.py b/tensorboard/notebook.py
index fe0e13aa..ab774377 100644
--- a/tensorboard/notebook.py
+++ b/tensorboard/notebook.py
@@ -378,8 +378,17 @@ def _display_ipython(port, height, display_handle):
<script>
(function() {
const frame = document.getElementById(%JSON_ID%);
- const url = new URL("/", window.location);
- url.port = %PORT%;
@fifar
fifar / profile_list.yaml
Last active June 5, 2020 07:31
CPU and GPU profiles in config.yaml
singleuser:
profileList:
- display_name: "Default"
description: |
Tubi Data Runtime
default: True
kubespawner_override:
image: <private-docker-registry>/tubi-data-runtime
extra_resource_limits:
nvidia.com/gpu: "0"
@fifar
fifar / pod_description.yaml
Created April 13, 2020 04:11
result of kubectl describe pod
Volumes:
home:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: efs-persist
ReadOnly: false
@fifar
fifar / storage.yaml
Last active June 5, 2020 07:30
singleuser storage in config.yaml
singleuser:
storage:
homeMountPath: '/home/tubi/notebooks/{username}'
type: "static"
static:
pvcName: "efs-persist"
subPath: 'home/{username}'
extraVolumeMounts:
- name: home
mountPath: /home/tubi/notebooks/shared
@fifar
fifar / auth.yaml
Last active June 5, 2020 07:30
auth in config.yaml
auth:
type: custom
custom:
className: oauthenticator.generic.GenericOAuthenticator
config:
login_service: "Okta"
client_id: "{{ okta_client_id }}"
client_secret: "{{ okta_client_secret }}"
token_url: https://{{ tubi_okta_domain }}/oauth2/v1/token
userdata_url: https://{{ tubi_okta_domain }}/oauth2/v1/userinfo