Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Flow.read_all #596

Merged
merged 2 commits into from
Oct 11, 2023
Merged

Add Flow.read_all #596

merged 2 commits into from
Oct 11, 2023

Conversation

SGrondin
Copy link
Collaborator

@SGrondin SGrondin commented Aug 7, 2023

This is really basic, but it's something that I've found useful to have.

When I first started experimenting with Eio, the only slight pain point I encountered at the very beginning was trying to understand how Flows, Files, Fs, Flow.Buf_read, etc, worked together. Having this simple function would have helped me understand it all.

It can also be useful for debugging.

lib_eio/flow.ml Outdated
@@ -161,6 +161,12 @@ let buffer_sink =
let ops = Pi.sink (module Buffer_sink) in
fun b -> Resource.T (b, ops)

let load (t : _ source) =
let b = Buffer.create 4096 in

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to have the buffer size be configurable through an optional parameter? ?(size = 4096)?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possibly. I debated whether to do it or not. Ultimately I went with no because Buffer_sink.copy just 20 lines above also doesn't. Plus I figured that any situation where the buffer size would have an impact would not use a function that loads everything into a single large string; it would process the file in small chunks instead. It made sense to me to keep this "beginner function" as simple as possible.

Copy link
Collaborator

@talex5 talex5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having a function like this might be good. Going via Buffer probably isn't a good idea (because then we read into a cstruct, copy it into the buffer, then copy it out again for the final string).

It can be done at the moment with e.g.

let read_all flow =
  Eio.Buf_read.(parse_exn take_all) flow ~max_size:max_int

Having this in Flow might be more discoverable, but requires a little bit of extra work to avoid a cyclic dependency (because Buf_read depends on Flow).

@SGrondin
Copy link
Collaborator Author

I used Buffer to avoid the dependency issue since this function isn't meant to be ultra performant. I'll give it another shot this weekend and see what I can do.

@SGrondin
Copy link
Collaborator Author

SGrondin commented Sep 30, 2023

@talex5 I rewrote it using Buf_read. I tried a few different approaches to break the cyclic dependency and I think this is the simplest way. If a better way exists, feel free to close this PR.

Edit: I'm not convinced the performance is worth the extra complexity if this function is not intended as more than a debugging or discovery or small scale tool.

@talex5
Copy link
Collaborator

talex5 commented Oct 8, 2023

I tried a few different approaches to break the cyclic dependency and I think this is the simplest way. If a better way exists, feel free to close this PR

A simpler way might be to add it in eio.ml, e.g.

module Buf_read = Buf_read
module Flow = struct
  include Flow
  let read_all = ...
end

@SGrondin
Copy link
Collaborator Author

SGrondin commented Oct 9, 2023

Great idea, I like that a lot better.

EDIT: done. Much better 🎉

@SGrondin SGrondin force-pushed the add-flow-load branch 2 times, most recently from f940408 to 9d62a6e Compare October 9, 2023 13:03
@SGrondin SGrondin requested a review from talex5 October 9, 2023 13:09
Copy link
Collaborator

@talex5 talex5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good - I just made some minor changes to the doc-strings.

@talex5 talex5 changed the title Add Flow.load Add Flow.read_all Oct 11, 2023
@talex5 talex5 merged commit ee51b04 into ocaml-multicore:main Oct 11, 2023
4 of 5 checks passed
@SGrondin SGrondin deleted the add-flow-load branch October 11, 2023 16:04
talex5 added a commit to talex5/opam-repository that referenced this pull request Nov 2, 2023
CHANGES:

New features / API changes:

- Add `Flow.read_all` (@SGrondin ocaml-multicore/eio#596, reviewed by @talex5 @rbjorklin).

- Add `Path.stat` (@patricoferris @talex5 @avsm ocaml-multicore/eio#617 ocaml-multicore/eio#618 ocaml-multicore/eio#624 ocaml-multicore/eio#620, reviewed by @SGrondin).

- Add `Path.rmtree` (@talex5 ocaml-multicore/eio#627 ocaml-multicore/eio#628, reviewed by @SGrondin).

- Add `Path.mkdirs` and `Path.split` (@patricoferris @talex5 ocaml-multicore/eio#625).

- Add `Eio.File.{seek,sync,truncate}` (@talex5 ocaml-multicore/eio#626).

- Add `Eio.Path.{kind,is_file,is_directory}` (@patricoferris @talex5 ocaml-multicore/eio#623, reviewed by @avsm).

- Switch from CTF to OCaml 5.1 runtime events (@TheLortex @patricoferris @talex5 ocaml-multicore/eio#634 ocaml-multicore/eio#635, reviewed by @avsm).
  This is a minimal initial version.

Documentation:

- Document `File.Stat` record fields (@avsm @talex5 ocaml-multicore/eio#621).

- Update README section about `env` (@talex5 ocaml-multicore/eio#614, reported by @jonsterling).

Build and test changes:

- Add `File.stat` benchmark (@talex5 ocaml-multicore/eio#616).

- Add `Path.stat` benchmark (@patricoferris @talex5 ocaml-multicore/eio#630).

- eio_linux: mark as only available on Linux (@talex5 ocaml-multicore/eio#629).

- Make MDX tests idempotent (@SGrondin ocaml-multicore/eio#601, reviewed by @talex5).

- Allow trailing whitespace in CHANGES.md (@talex5 ocaml-multicore/eio#632).

- Update minimum OCaml version to 5.1 (@talex5 ocaml-multicore/eio#631).

- Generate prototypes for C stubs from ml files (@talex5 ocaml-multicore/eio#615).

- Don't try to compile uring support on centos 7 (@talex5 ocaml-multicore/eio#638, reported by @zenfey).
talex5 added a commit to talex5/opam-repository that referenced this pull request Nov 2, 2023
CHANGES:

New features / API changes:

- Add `Flow.read_all` (@SGrondin ocaml-multicore/eio#596, reviewed by @talex5 @rbjorklin).

- Add `Path.stat` (@patricoferris @talex5 @avsm ocaml-multicore/eio#617 ocaml-multicore/eio#618 ocaml-multicore/eio#624 ocaml-multicore/eio#620, reviewed by @SGrondin).

- Add `Path.rmtree` (@talex5 ocaml-multicore/eio#627 ocaml-multicore/eio#628, reviewed by @SGrondin).

- Add `Path.mkdirs` and `Path.split` (@patricoferris @talex5 ocaml-multicore/eio#625).

- Add `Eio.File.{seek,sync,truncate}` (@talex5 ocaml-multicore/eio#626).

- Add `Eio.Path.{kind,is_file,is_directory}` (@patricoferris @talex5 ocaml-multicore/eio#623, reviewed by @avsm).

- Switch from CTF to OCaml 5.1 runtime events (@TheLortex @patricoferris @talex5 ocaml-multicore/eio#634 ocaml-multicore/eio#635, reviewed by @avsm).
  This is a minimal initial version.

Documentation:

- Document `File.Stat` record fields (@avsm @talex5 ocaml-multicore/eio#621).

- Update README section about `env` (@talex5 ocaml-multicore/eio#614, reported by @jonsterling).

Build and test changes:

- Add `File.stat` benchmark (@talex5 ocaml-multicore/eio#616).

- Add `Path.stat` benchmark (@patricoferris @talex5 ocaml-multicore/eio#630).

- eio_linux: mark as only available on Linux (@talex5 ocaml-multicore/eio#629).

- Make MDX tests idempotent (@SGrondin ocaml-multicore/eio#601, reviewed by @talex5).

- Allow trailing whitespace in CHANGES.md (@talex5 ocaml-multicore/eio#632).

- Update minimum OCaml version to 5.1 (@talex5 ocaml-multicore/eio#631).

- Generate prototypes for C stubs from ml files (@talex5 ocaml-multicore/eio#615).

- Don't try to compile uring support on centos 7 (@talex5 ocaml-multicore/eio#638, reported by @zenfey).
nberth pushed a commit to nberth/opam-repository that referenced this pull request Jun 18, 2024
CHANGES:

New features / API changes:

- Add `Flow.read_all` (@SGrondin ocaml-multicore/eio#596, reviewed by @talex5 @rbjorklin).

- Add `Path.stat` (@patricoferris @talex5 @avsm ocaml-multicore/eio#617 ocaml-multicore/eio#618 ocaml-multicore/eio#624 ocaml-multicore/eio#620, reviewed by @SGrondin).

- Add `Path.rmtree` (@talex5 ocaml-multicore/eio#627 ocaml-multicore/eio#628, reviewed by @SGrondin).

- Add `Path.mkdirs` and `Path.split` (@patricoferris @talex5 ocaml-multicore/eio#625).

- Add `Eio.File.{seek,sync,truncate}` (@talex5 ocaml-multicore/eio#626).

- Add `Eio.Path.{kind,is_file,is_directory}` (@patricoferris @talex5 ocaml-multicore/eio#623, reviewed by @avsm).

- Switch from CTF to OCaml 5.1 runtime events (@TheLortex @patricoferris @talex5 ocaml-multicore/eio#634 ocaml-multicore/eio#635, reviewed by @avsm).
  This is a minimal initial version.

Documentation:

- Document `File.Stat` record fields (@avsm @talex5 ocaml-multicore/eio#621).

- Update README section about `env` (@talex5 ocaml-multicore/eio#614, reported by @jonsterling).

Build and test changes:

- Add `File.stat` benchmark (@talex5 ocaml-multicore/eio#616).

- Add `Path.stat` benchmark (@patricoferris @talex5 ocaml-multicore/eio#630).

- eio_linux: mark as only available on Linux (@talex5 ocaml-multicore/eio#629).

- Make MDX tests idempotent (@SGrondin ocaml-multicore/eio#601, reviewed by @talex5).

- Allow trailing whitespace in CHANGES.md (@talex5 ocaml-multicore/eio#632).

- Update minimum OCaml version to 5.1 (@talex5 ocaml-multicore/eio#631).

- Generate prototypes for C stubs from ml files (@talex5 ocaml-multicore/eio#615).

- Don't try to compile uring support on centos 7 (@talex5 ocaml-multicore/eio#638, reported by @zenfey).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants