Last active
October 13, 2023 21:20
-
-
Save ericphanson/ffe42145bbee476012354abbf23df783 to your computer and use it in GitHub Desktop.
Arrow IPC with named pipe
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
julia> include("consumer.jl"); | |
Julia done: got Arrow.Table with 4 rows, 3 columns, and schema: | |
:f0 Union{Missing, Int64} | |
:f1 Union{Missing, String} | |
:f2 Union{Missing, Bool} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
using Arrow | |
pipe = "py_to_julia" | |
ispath(pipe) && rm(pipe) | |
run(`mkfifo $pipe`) | |
@sync begin | |
@async run(`python producer.py`) | |
@async begin | |
x = Arrow.Table(read(pipe)) | |
print("Julia done: got $x") | |
end | |
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import pyarrow as pa | |
# create a pyarrow record batch (could be a table also) | |
data = [ | |
pa.array([1, 2, 3, 4]), | |
pa.array(['foo', 'bar', 'baz', None]), | |
pa.array([True, None, False, True]) | |
] | |
batch = pa.record_batch(data, names=['f0', 'f1', 'f2']) | |
# Write it to the named pipe | |
with open('py_to_julia', 'wb') as sink: | |
with pa.ipc.new_stream(sink, batch.schema) as writer: | |
writer.write_batch(batch) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment