davidgin’s gists

davidgin / schema.sql

Created June 15, 2016 08:52 — forked from fernandomantoan/schema.sql

Schema for PostgreSQL to use with JdbcTokenStore (Spring Security OAuth2)

	create table oauth_client_details (
	client_id VARCHAR(256) PRIMARY KEY,
	resource_ids VARCHAR(256),
	client_secret VARCHAR(256),
	scope VARCHAR(256),
	authorized_grant_types VARCHAR(256),
	web_server_redirect_uri VARCHAR(256),
	authorities VARCHAR(256),
	access_token_validity INTEGER,
	refresh_token_validity INTEGER,

davidgin / gitignore

Created May 31, 2016 08:44

davidgin / gist:5709580d7573491a980bf475752d1819

Created May 8, 2016 08:15

hdfs scala utils for modified folders

	import org.apache.hadoop.fs._
	import org.apache.hadoop.conf.Configuration
	import scala.collection.mutable.ListBuffer


	object HdfsUtil {


	val pf = new PathFilter {
	override def accept(path: Path): Boolean = {

davidgin / MultipleOutputs.scala

Created September 26, 2015 16:42 — forked from silasdavis/MultipleOutputs.scala

Wrapping OutputFormat to produce multiple outputs with hadoop MultipleOutputs

	/**
	* This file contains the core idea of wrapping an underlying OutputFormat with an OutputFormat
	* with an augmented key that writes to partitions using MultipleOutputs (or something similar)
	*/

	package model.hadoop

	import model.hadoop.HadoopIO.MultipleOutputer
	import model.hadoop.HadoopIO.MultipleOutputer._
	import org.apache.hadoop.io.{DataInputBuffer, NullWritable}

davidgin / query.hive.template

Created January 18, 2013 19:59 — forked from tovbinm/query.hive.template

	ADD JAR s3://<s3-bucket>/jars/hive_contrib-0.5.jar;

	CREATE TEMPORARY FUNCTION now as 'com.mt.utils.udf.Now';
	CREATE TEMPORARY FUNCTION user_agent_f as 'com.mt.utils.UserAgent';

	set hive.merge.mapredfiles=true;
	set hive.merge.mapfiles=true;
	set hive.merge.size.per.task=500000000;

	CREATE EXTERNAL TABLE data

davidgin / gist:4151316

Created November 26, 2012 23:21 — forked from criccomini/gist:3775858

Sort Reducer Input Values in Hadoop

	public class SortReducerByValues {
	public static final String INPUT = "/tmp/data_in";
	public static final String OUTPUT = "/tmp/data_out";

	public static void main(String[] args) throws IOException {
	new SortReducerByValues().run();
	}

	public void run() throws IOException {
	JobConf conf = new JobConf();

davidgin / java-shell-script-wrapper.sh

Created September 12, 2012 12:16

java-shell-script-wrapper from http://saltnlight5.blogspot.ie/2012/08/a-better-java-shell-script-wrapper.html

	#!/usr/bin/env bash
	#
	# Copyright 2012 Zemian Deng
	#
	# Licensed under the Apache License, Version 2.0 (the "License");
	# you may not use this file except in compliance with the License.
	# You may obtain a copy of the License at
	#
	# http://www.apache.org/licenses/LICENSE-2.0
	#