Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove allocation in skipchars #50526

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

remove allocation in skipchars #50526

wants to merge 2 commits into from

Conversation

stevengj
Copy link
Member

@stevengj stevengj commented Jul 12, 2023

skipchars was using readline(io) to ignore line comments, which allocates a String that is immediately discarded. This PR switches it to copyline(devnull, io, keep=true) instead, exploiting #48273, which shouldn't allocate. (I pass keep=true because it's slightly more efficient in general, since it can skip the CRLF logic.)

(The existing tests cover this change.)

@stevengj stevengj added domain:io Involving the I/O subsystem: libuv, read, write, etc. performance Must go faster labels Jul 12, 2023
base/io.jl Outdated Show resolved Hide resolved
base/io.jl Outdated Show resolved Hide resolved
@stevengj
Copy link
Member Author

stevengj commented Jul 16, 2023

Unfortunately, despite removing allocations, this actually slows things down for the current version. A quick benchmark reading and discarding all of the lines of HISTORY.md gives:

julia> using_readline(io) = while !eof(io); readline(io); end;

julia> using_copyline(io) = while !eof(io); copyline(devnull, io, keep=true); end;

julia> io = open("HISTORY.md");

julia> @btime using_readline(seekstart($io));
  505.072 μs (5620 allocations: 480.03 KiB)

julia> @btime using_copyline(seekstart($io));
  17.041 ms (0 allocations: 0 bytes)

The basic problem is that copyline(::DevNull, ::IOStream) uses the fallback character-by-character method, which is slow for IOStream. To make this faster we'd need an optimized IOStream method for DevNull. Not sure if it's worth it. 😢

(Note that if someone really wants a faster skipchars, an optimized method for BufferedInputStream is implemented in JuliaIO/BufferedStreams.jl#77.)

@stevengj stevengj added the status:DO NOT MERGE Do not merge this PR! label Jul 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain:io Involving the I/O subsystem: libuv, read, write, etc. performance Must go faster status:DO NOT MERGE Do not merge this PR!
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants