[API Proposal]: Add File.ReadAllLinesAsync methods that return IAsyncEnumerable<string>

[eiriktsarpalis]

Background and motivation

The existing File.ReadAllLinesAsync method returns a Task<string[]> meaning that the entire contents will have to be loaded in-memory before the task completes. This might be unsuitable in cases where we need to lazily process very large files on a line-by-line basis. It seems like IAsyncEnumerable<string> might be the ideal abstraction to achieve this.

API Proposal

namespace System.IO
{
    public partial static class File
    {
        public static IAsyncEnumerable<string> ReadLinesAsAsyncEnumerable(string path, CancellationToken cancellationToken = default);
        public static IAsyncEnumerable<string> ReadLinesAsAsyncEnumerable(string path, Encoding encoding, CancellationToken cancellationToken = default);
    }
}

API Usage

await foreach(string line in File.ReadLinesAsAsyncEnumerable("myHugeFile.txt"))
{
     yield return JsonSerializer.Deserialize<MyPoco>(line);
}

Alternative Designs

Should we consider accepting a custom separator parameter? (instead of Environment.NewLine)

Risks

No response

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and motivation

The existing File.ReadAllLinesAsync method returns a Task<string[]> meaning that the entire contents will have to be loaded in-memory before the task completes. This might be unsuitable in cases where we need to lazily process very large files on a line-by-line basis. It seems like IAsyncEnumerable<string> might be the ideal abstraction to achieve this.

API Proposal

namespace System.IO
{
    public partial static class File
    {
        public static IAsyncEnumerable<string> ReadLinesAsAsyncEnumerable(string path, CancellationToken cancellationToken = default);
        public static IAsyncEnumerable<string> ReadLinesAsAsyncEnumerable(string path, Encoding encoding, CancellationToken cancellationToken = default);
    }
}

API Usage

await foreach(string line in File.ReadLinesAsAsyncEnumerable("myHugeFile.txt"))
{
     yield return JsonSerializer.Deserialize<MyPoco>(line);
}

Alternative Designs

Should we consider accepting a custom separator parameter? (instead of Environment.NewLine)

Risks

No response

Author:	eiriktsarpalis
Assignees:	-
Labels:	api-suggestion, area-System.IO, untriaged
Milestone:	-

[jozkee]

I think this is duplicate of #2214.

[eiriktsarpalis]

Oh, cool! For some reason it didn't turn up in my issue search.