Created
April 26, 2018 19:05
-
-
Save calonso/5a13c7fa7a0c542eb458d56a00b6992e to your computer and use it in GitHub Desktop.
Apache Beam function to get an updating side input
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| def updatingSchemas(sc: ScioContext, refreshFreq: Duration, tableNames: List[String]) { | |
| sc.customInput("Tick", GenerateSequence.from(0).withRate(1, refreshFreq)) | |
| .withName("Retrieve schemas") | |
| .flatMap { _ => | |
| tableNames.map(t => (t, bq.getTableSchema(t))) | |
| } | |
| .withName("Set windowing") | |
| .withFixedWindows(refreshFreq) | |
| .withName("To side input") | |
| .asMapSideInput | |
| } |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment