Skip to content

Instantly share code, notes, and snippets.

View filipelenfers's full-sized avatar

Filipe Pais Lenfers filipelenfers

View GitHub Profile
@filipelenfers
filipelenfers / remove_glue_table_partition_add_again.py
Created April 21, 2020 14:59
Remove all partitions from a Glue Table and add again (sometimes solves the HIVE_PARTITION_SCHEMA_MISMATCH in Athena)
# !!!THIS SCRIPT ASSUME HIVE COMPATIBLE PARTITIONS!!!
#
# When you get the following error:
# HIVE_PARTITION_SCHEMA_MISMATCH: There is a mismatch between the table and partition schemas. The types are incompatible and cannot be coerced.
# what happened is that some column type changed in the table, and partitions have they own version of the schema,
# and the version that partitions have cached are not compatible with the table actual version on glue.
# We need to:
# 1. remove partitions
# 2. add partitions again
#
@filipelenfers
filipelenfers / avro_write_file_in_memory.py
Last active August 23, 2024 16:04
Generate AVRO bytes as a in-memory file, with the schema, Python3.
import io
import snappy
import avro.schema
from avro.datafile import DataFileReader, DataFileWriter
from avro.io import DatumReader, DatumWriter, BinaryEncoder
#Generate avro bytes as a in-memory file, with the schema.
def generate_avro_bytes(schema,avro_dicts, codec='snappy'):
bytes_writer = io.BytesIO()
writer = DataFileWriter(bytes_writer, DatumWriter(), schema, codec=codec)
@filipelenfers
filipelenfers / README.md
Created November 19, 2019 17:00 — forked from davideicardi/README.md
Write and read Avro records from bytes array

Avro serialization

There are 4 possible serialization format when using avro:

@filipelenfers
filipelenfers / dict_to_dynamodb_item.py
Last active October 10, 2019 19:10 — forked from JamieCressey/dict_to_dynamodb_item.py
Coverts a standard Python3 dictionary to a Boto3 DynamoDB item
def dict_to_item(raw:dict):
return {
key: _dict_to_item_recurse(value)
for key, value in raw.items()
}
def _dict_to_item_recurse(raw):
if isinstance(raw, dict):
return {
@filipelenfers
filipelenfers / VagrantFile.rb
Created September 26, 2019 23:22
VagrantFile Zepellin/Glue - article - step 1
# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64"
config.vm.network "forwarded_port", guest: 8080, host: 8080
config.vm.provider "virtualbox" do |vb|
@filipelenfers
filipelenfers / airflow-scheduler.service
Last active September 17, 2019 18:43
airflow-scheduler.service for Ubuntu - see #CHANGE in the file
# Apache Airflow Scheduler Service file for Ubuntu 18
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
@filipelenfers
filipelenfers / airflow-webserver.service
Last active September 17, 2019 18:43
airflow-webserver.service for Ubuntu - see #CHANGE in the file
# Apache Airflow Service file for Ubuntu 18
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
@filipelenfers
filipelenfers / Vagrantfile-scala-dev.rb
Last active September 17, 2019 14:26
Vagrantfile - scala dev
# -*- mode: ruby -*-
# vi: set ft=ruby :
# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|
@filipelenfers
filipelenfers / Vagrantfile.rb
Created August 23, 2019 12:54
Vagrantfile with docker
-*- mode: ruby -*-
# vi: set ft=ruby :
# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|
# The most common configuration options are documented and commented below.
# For a complete reference, please see the online documentation at
@filipelenfers
filipelenfers / .hyper.js
Created January 5, 2018 13:05
Hyper.js using Zsh on Windows with WSL
//I like to use zsh(with oh-my-zsh) as my default shell.
//First define your default distro for wsl
//run this on cmd.exe
//wslconfig /l
//wslconfig /s Ubuntu
//use this options on preferences of Hyper.js termina to use it as default.
{
shell: 'C:\\Windows\\System32\\bash.exe',