Skip to content

Instantly share code, notes, and snippets.

@nire0510

nire0510/split_csv.sh

Created July 17, 2023 13:29
Show Gist options
  • Save nire0510/52d6198611a005d507caca71f2e8bd4b to your computer and use it in GitHub Desktop.
Save nire0510/52d6198611a005d507caca71f2e8bd4b to your computer and use it in GitHub Desktop.
Download ZIP
Split large CSV file into smaller chunks
Raw
split_csv.sh
FILE=$1
BASENAME=`basename $FILE .csv`
ROWS=$2
echo Splitting $FILE into multiple files of $ROWS rows each...
# split to chunks:
split -l $ROWS -d $FILE $BASENAME\_
# append file extension:
for file in $(find $BASENAME\_*);
do mv $file $file.csv;
done;
# add header row to all chunks except of the first:
for file in $(find . -type f -name "$BASENAME\_*.csv" -not -name "$BASENAME\_00.csv");
do echo "$(head -1 $BASENAME\_00.csv)\n$(cat $file)" > $file;
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment