Henry Zhu hzhu212

Wrap shell command with subprocess

create a simple HDFS client for example:

import logging
import subprocess
from typing import List, Tuple, Union, TextIO, BinaryIO

A simple Python retrying decorator

Define:

import functools
import logging

logger = logging.getLogger(__name__)

任意一个 cell 出现异常执行 callback

# this function will be called on exceptions in any cell
def when_exception(shell, etype, evalue, tb, tb_offset=None):
    # still show the error within the notebook, don't just swallow it
    shell.showtraceback((etype, evalue, tb), tb_offset=tb_offset)

    err_msg = f'Unexpected exception occured: {repr(evalue)}'
 logger.error(err_msg)

A wrapper for Python protobuf object:

class PbMessage(object):
    """protobuf 消息的包装类，封装一些通用方法"""

    def __init__(self, pb_class):
        self._pb_class = pb_class
        self._pb_obj = None
        self._raw_msg = None

Python Stream object act like Java Stream, features include lazy calculating, slicing, chain-call etc.

stream.py:

import functools
import itertools


class Stream(object):

a threading-safe cache decorator, cache result of function returns

from functools import wraps
import threading


class CallCache(object):
    def __init__(self):
 self.lock = threading.RLock()

Windows cmd/bat 相关

后台执行命令

cmd 中后台执行命令，相当于 bash 中的&。

start /b <your_command>

	from concurrent.futures import as_completed
	from concurrent.futures.process import ProcessPoolExecutor
	import os
	import psutil
	import time


	def kill_child_processes(parent_pid):
	try:
	parent = psutil.Process(parent_pid)

	package com.example.udf;

	import com.google.common.base.Throwables;
	import org.apache.hadoop.hive.ql.exec.Description;
	import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
	import org.apache.hadoop.hive.ql.exec.UDFArgumentLengthException;
	import org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException;
	import org.apache.hadoop.hive.ql.metadata.HiveException;
	import org.apache.hadoop.hive.ql.udf.generic.GenericUDF;
	import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;