You have created a VM, deployed your services in Docker on it, services (for example, Kafka + Kafka Connect) started writing data.
Suddenly you find that the disk volume is running out.
df -h
Whenever you need to store your data on S3 / Data Lake / External table choose file format wisely:
## Enable debugging messages and exit on error | |
# set -ex | |
## Assign variables directly or use ENV variables | |
TS=`date +"%Y-%m-%d-%H-%M-%S-%Z"` | |
# OXR_TOKEN='' | |
# BASE_CURRENCY='RUB' | |
# SYMBOLS='AED,EUR,GBP,RUB,USD' | |
## Chain commands |
{{ | |
config( | |
materialized='ephemeral' | |
) | |
}} | |
WITH accepted AS ( | |
SELECT DISTINCT | |
XN Merge Left Join DS_DIST_NONE (cost=6768461698952.58..64203387414945.72 rows=11491303 width=4361) | |
-> XN Hash Left Join DS_BCAST_INNER (cost=5768461175265.66..63203382323457.28 rows=11491303 width=3845) | |
-> XN Merge Left Join DS_DIST_NONE (cost=4765778722374.23..62200689246924.30 rows=11491303 width=3321) | |
-> XN Hash Left Join DS_DIST_INNER (cost=4765778722374.23..62200689217006.26 rows=11491303 width=3285) | |
-> XN Merge Left Join DS_DIST_NONE (cost=3765767462785.01..36310404425364.76 rows=11491303 width=3284) | |
-> XN Hash Left Join DS_DIST_NONE (cost=3765767462785.01..36310403604540.56 rows=11491303 width=3276) | |
-> XN Hash Left Join DS_DIST_INNER (cost=3765766240852.13..36310400055617.20 rows=11491303 width=3251) | |
-> XN Hash Left Join DS_DIST_NONE (cost=1765764690131.17..34310372401253.97 rows=11491303 width=3250) | |
-> XN Hash Left Join DS_DIST_NONE (cost=1765764584059.78..34308739121313.73 rows=11491303 width=2734) | |
-> XN Hash Left Join DS_DIST_INNER (co |
select | |
_id as audit_id | |
, resource_id | |
, driver_id | |
, "user_id" | |
, "event" | |
, created_at | |
, resource_type |
Data Warehouse Analyst – Analytics Engineer @ OTUS.ru
Follow instructions in metabase_clickhouse.md
below step by step.
Data Warehouse Analyst – Analytics Engineer @ OTUS.ru
Follow instructions in data_vault_tutorial.md
step by step.
Add README.md with answers to the following questions: