Created
February 10, 2021 10:49
-
-
Save jacace/1625e49cf9b2adce9e9b754f559d7301 to your computer and use it in GitHub Desktop.
Create resources in the Stack driver at the partition level
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dstream.foreachRDD { rdd => | |
rdd.foreachPartition { partitionOfRecords => | |
// ConnectionPool is a static, lazily initialized pool of connections | |
val connection = ConnectionPool.getConnection() | |
partitionOfRecords.foreach(record => connection.send(record)) | |
ConnectionPool.returnConnection(connection) // return to the pool for future reuse | |
} | |
} |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment