You can avoid a bufio.ErrTooLong
error ("bufio.Scanner: token too long"
) by using something like this as your scanner.Split()
function:
const maxTokenLength = bufio.MaxScanTokenSize / 2
func scanLinesOrGiveLong(data []byte, atEOF bool) (advance int, token []byte, err error) {
advance, token, err = bufio.ScanLines(data, atEOF)
if advance > 0 || token != nil || err != nil {
// bufio.ScanLines found something, use it
return
}
// bufio.ScanLines found nothing
// if our buffer is still a reasonable size, continue scanning for regular lines
if len(data) < maxTokenLength {
return
}
// our buffer is getting massive, stop waiting for line breaks and return data now
// this avoids bufio.ErrTooLong
return maxTokenLength, data[0:maxTokenLength], nil
}
Like this:
package main
import (
"bufio"
"os"
)
func main() {
scanner := bufio.NewScanner(os.Stdin)
scanner.Split(scanLinesOrGiveLong)
for scanner.Scan() {
os.Stdout.Write(scanner.Bytes())
}
}
I had this issue while using logrus.Write()
, so I've included an extension for that below.