-
Go to https://developer.apple.com/downloads/index.action and search for "Command line tools" and choose the one for your Mac OSX
-
Go to http://brew.sh/ and enter the one-liner into the Terminal, you now have
brew
installed (a better Mac ports) -
Install transmission-daemon with
brew install transmission
-
Copy the startup config for launchctl with
ln -sfv /usr/local/opt/transmission/*.plist ~/Library/LaunchAgents
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# suppose my data file name has the following format "datatfile_YYYY_MM_DD.csv"; this file arrives in S3 every day. | |
file_suffix = "{{ execution_date.strftime('%Y-%m-%d') }}" | |
bucket_key_template = 's3://[bucket_name]/datatfile_{}.csv'.format(file_suffix) | |
file_sensor = S3KeySensor( | |
task_id='s3_key_sensor_task', | |
poke_interval=60 * 30, # (seconds); checking file every half an hour | |
timeout=60 * 60 * 12, # timeout in 12 hours | |
bucket_key=bucket_key_template, | |
bucket_name=None, | |
wildcard_match=False, |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from airflow import DAG | |
from airflow.operators.sensors import S3KeySensor | |
from airflow.operators import BashOperator | |
from datetime import datetime, timedelta | |
yday = datetime.combine(datetime.today() - timedelta(1), | |
datetime.min.time()) | |
default_args = { | |
'owner': 'msumit', |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
with DAG(**dag_config) as dag: | |
# Declare pipeline start and end task | |
start_task = DummyOperator(task_id='pipeline_start') | |
end_task = DummyOperator(task_id='pipeline_end') | |
for account_details in pipeline_config['task_details']['accounts']: | |
#Declare Account Start and End Task | |
if account_details['runable']: | |
acct_start_task = DummyOperator(task_id=account_details['account_id'] + '_start') | |
acct_start_task.set_upstream(start_task) |
A running example of the code from:
- http://marcio.io/2015/07/handling-1-million-requests-per-minute-with-golang
- http://nesv.github.io/golang/2014/02/25/worker-queues-in-go.html
This gist creates a working example from blog post, and a alternate example using simple worker pool.
TLDR: if you want simple and controlled concurrency use a worker pool.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
/* | |
* Licensed to the Apache Software Foundation (ASF) under one or more | |
* contributor license agreements. See the NOTICE file distributed with | |
* this work for additional information regarding copyright ownership. | |
* The ASF licenses this file to You under the Apache License, Version 2.0 | |
* (the "License"); you may not use this file except in compliance with | |
* the License. You may obtain a copy of the License at | |
* | |
* http://www.apache.org/licenses/LICENSE-2.0 | |
* |
Moved to git repository: https://github.com/denji/golang-tls
# Key considerations for algorithm "RSA" ≥ 2048-bit
openssl genrsa -out server.key 2048
# Key considerations for algorithm "ECDSA" ≥ secp384r1
# List ECDSA the supported curves (openssl ecparam -list_curves)
I've been working with Kafka for over 7 years. I inevitably find myself doing the same set of activities while I'm developing or working with someone else's system. Here's a set of Kafka productivity hacks for doing a few things way faster than you're probably doing them now. 🔥
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import akka.actor.{ ActorSystem, Props } | |
import akka.io.IO | |
import com.typesafe.config.Config | |
import com.typesafe.scalalogging.LazyLogging | |
import spray.can.Http | |
sealed class HealthStatusService(actorNames: List[String], webConfig: Config)(implicit system: ActorSystem) extends LazyLogging { | |
private[status] val healthCheckPath: String = webConfig.getString("status-path") | |
private[status] val timeoutDuration: Int = webConfig.getInt("timeout") |
NewerOlder