DaisukeMiyamoto / create_compute_environment_and_job_queue.py

Last active March 20, 2022 17:04

Create compute environment and job queue for AWS Batch with Boto3 (Polling and Waiter implementations)

	####################################################################
	# Create compute environment and job queue for AWS Batch with Boto3
	# Daisuke Miyamoto
	# 20200330
	#

	import boto3
	import botocore.waiter
	from botocore.exceptions import WaiterError
	import time

goliatone / README.md

Last active September 21, 2024 00:57 — forked from colophonemes/create_triggers

Postgres TRIGGER to call NOTIFY with a JSON payload

This TRIGGER function calls PosgreSQL's NOTIFY command with a JSON payload. You can listen for these calls and then send the JSON payload to a message queue (like AMQP/RabbitMQ) or trigger other actions.

Create the trigger with notify_trigger.sql.

When declaring the trigger, supply the column names you want the JSON payload to contain as arguments to the function (see create_triggers.sql)

The payload returns a JSON object:

uhho / pandas_s3_streaming.py

Last active December 2, 2022 18:57

Streaming pandas DataFrame to/from S3 with on-the-fly processing and GZIP compression

	def s3_to_pandas(client, bucket, key, header=None):

	# get key using boto3 client
	obj = client.get_object(Bucket=bucket, Key=key)
	gz = gzip.GzipFile(fileobj=obj['Body'])

	# load stream directly to DF
	return pd.read_csv(gz, header=header, dtype=str)

	def s3_to_pandas_with_processing(client, bucket, key, header=None):

blaylockbk / multipro_template.py

Last active May 20, 2024 16:39

Template for Python multiprocessing and multithreading

	import multiprocessing
	from multiprocessing.dummy import Pool as ThreadPool

	import numpy as np


	def my_multipro(items, func, max_cpus=12):
	"""Do an embarrassingly parallel task using multiprocessing.

	Use this for CPU bound tasks.

eldondevcg / get_cloudwatch_logs.py

Last active January 13, 2023 21:11

Pull down cloudwatch logs with boto

	# IF YOU INCUR HUGE COSTS WITH THIS OR IT BREAKS DON'T BLAME ME License

	# This is a throw-away script I wrote to pull the json events for all of the streams from a cloudwatch log
	# For some reason, the naive way to do vpc network logging does logging to different streams in a cloudwatch
	# log based on interface.
	# Great for diagnosing lots of things, and generating verbose logs, but for the broad-stroke analysis I was doing,
	# all I really wanted was the basic data. This would have been easier if I had logged to s3, but I did not see a
	# way to do that in 2 clicks.

	group_name = 'CHANGEME'

Zearin / python_decorator_guide.md

Last active August 22, 2024 06:22

The best explanation of Python decorators I’ve ever seen. (An archived answer from StackOverflow.)

NOTE: This is a question I found on StackOverflow which I’ve archived here, because the answer is so effing phenomenal.

Q: How can I make a chain of function decorators in Python?

If you are not into long explanations, see [Paolo Bergantino’s answer][2].

rgreenjr / postgres_queries_and_commands.sql

Last active September 19, 2024 05:47

Useful PostgreSQL Queries and Commands

	-- show running queries (pre 9.2)
	SELECT procpid, age(clock_timestamp(), query_start), usename, current_query
	FROM pg_stat_activity
	WHERE current_query != '<IDLE>' AND current_query NOT ILIKE '%pg_stat_activity%'
	ORDER BY query_start desc;

	-- show running queries (9.2)
	SELECT pid, age(clock_timestamp(), query_start), usename, query
	FROM pg_stat_activity
	WHERE query != '<IDLE>' AND query NOT ILIKE '%pg_stat_activity%'

josef-pkt / stationary_block_bootstrap.py

Created April 2, 2012 01:14

Performs a stationary block bootstrap resampling of elements from a time series, or rows of an input matrix representing a multivariate time series.

	#Copyright (c) 2011, Josh Hemann (hemann @ colorado . edu)
	#All rights reserved.
	#
	#Redistribution and use in source and binary forms, with or without
	#modification, are permitted provided that the following conditions are met:
	# * Redistributions of source code must retain the above copyright
	# notice, this list of conditions and the following disclaimer.
	# * Redistributions in binary form must reproduce the above copyright
	# notice, this list of conditions and the following disclaimer in the
	# documentation and/or other materials provided with the distribution.

Riyad Parvez riyadparvez

Q: How can I make a chain of function decorators in Python?