Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support sync access by default #18

Closed
bionoren opened this issue Mar 12, 2019 · 5 comments
Closed

Support sync access by default #18

bionoren opened this issue Mar 12, 2019 · 5 comments

Comments

@bionoren
Copy link
Contributor

ParseLedger() is just a wrapper for ParseLedgerAsync, which means there is no public synchronous parsing API. If my program is going to be blocked anyway waiting for the results of a ledger parse, I don't want to have the overhead of channels and scheduler context switches on top of that.

Could we switch it around - so ParseLedger is where the work actually happens, and ParseLedgerAsync wraps it in a goroutine and channels? I would be happy to submit a pull request

@bionoren
Copy link
Contributor Author

Alternatively, if you're concerned about parsing large ledger files, could we switch to a callback model as recommend by David Cheney? https://dave.cheney.net/practical-go/presentations/qcon-china.html#_leave_concurrency_to_the_caller

@howeyc
Copy link
Owner

howeyc commented Mar 12, 2019

I don't understand why the internal methods used within the package would matter to the caller.

If you want to handle the ledger as a stream of transactions, processing them as they are parsed, call ParseLedgerAsync().
If you want to call a synchronous, blocking function that returns all transactions as a single array, call ParseLedger().

If you're advocating for a public API that allows for the caller to pass a callback function that is executed per transaction, I'm going to respectfully decline.

@bionoren
Copy link
Contributor Author

I realize I can call ParseLedger as a synchronous blocking API, my problem is that internally it’s asynchronous, and the memory allocations for those channels together with the context switching running a separate go-routine are a performance bottleneck for my application. I need a way to opt out of the goroutine and channels.

For context, benchmarking my application, ParseLedger takes 12% of total execution time (for a 15ms process). It’s not because ledger is inherently slow - I’ve looked at your benchmarks - it’s extra time in the scheduler and garbage collector due to, for me, unnecessary concurrency

@howeyc
Copy link
Owner

howeyc commented Mar 12, 2019

Interesting. Okay sure, go for it.

@howeyc
Copy link
Owner

howeyc commented Mar 15, 2019

LGTM

@howeyc howeyc closed this as completed Mar 15, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants