[🐛 Bug]: HeapProfiler.addHeapSnapshotChunk not fired in order

[zixinyin]

What happened?

For CDP session HeapProfiler.addHeapSnapshotChunk() event, Selenium is not returning the large chunk data in sequence and there is not way to place the returned data in order without low level access.

How can we reproduce the issue?

DevTools devTools = ((HasDevTools) driver).getDevTools();
        devTools.createSession();
        devTools.send(HeapProfiler.enable());
        Queue<String> queue = new LinkedList<>();
        devTools.addListener(HeapProfiler.addHeapSnapshotChunk(), queue::add);
        devTools.send(HeapProfiler.takeHeapSnapshot(Optional.of(false), Optional.of(false), Optional.of(false), Optional.of(false)));
        try {
            FileOutputStream fileOutputStream = new FileOutputStream("/tmp/output.heapsnapshot");
            while (!queue.isEmpty()) {
                fileOutputStream.write(queue.poll().getBytes());
            }
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
        devTools.clearListeners();
        devTools.close();



Another way to reproduce is to print the string size every time the event is fired.
final ReentrantLock lock = new ReentrantLock(true);
devTools.addListener(HeapProfiler.addHeapSnapshotChunk(), target -> {
            lock.lock();
            System.out.println(target.length());
            lock.unlock();
        });
And you would still see the output end up like:

102400
102400
102400
102400
102400
32441
102400
102400
102400
102400

Relevant log output

Usually the log output will start with {"snapshot":{"meta":{"node_fields": but not always. 
The log output should be a complete json file but you would often see the file ends with broken output.
Each large chunk data's length is 102400.

Operating System

maci and windows

Selenium version

open jdk 21.0.2

What are the browser(s) and version(s) where you see this issue?

Chrome 118

What are the browser driver(s) and version(s) where you see this issue?

ChromeDriver 118

Are you using Selenium Grid?

No response

[github-actions]

@zixinyin, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[pujagani]

Thank you for the details. Have you checked the logs to ensure that the events are returned in order from CDP itself? Can you please share the debug logs here. Thank you!

[joerg1985]

The CDP events are processed concurrently by different threads, i think this will destory the order.

Other areas rely on this, e.g. the NetworkInterceptor:
One thread will wait here for another thread to complete the future:
https://github.com/SeleniumHQ/selenium/blob/trunk/java/src/org/openqa/selenium/devtools/idealized/Network.java#L242
And this is the line where the other thread does complete the future:
https://github.com/SeleniumHQ/selenium/blob/trunk/java/src/org/openqa/selenium/devtools/idealized/Network.java#L226

[pujagani]

Makes sense. Thank you @joerg1985! So what should be done regarding this issue?

[joerg1985]

@pujagani this is a very special case, so i would empower the listener to handle this, e.g. like in the PR i will raise in the next minutes.

[pujagani]

Thank you for the PR! It looks good to me.

[joerg1985]

@zixinyin Starting with the next release it is possible to register a two argument listener, to sort the data:

TreeMap<Long, String> chunks = new TreeMap<>();
devTools.addListener(HeapProfiler.addHeapSnapshotChunk(), chunks::put);
devTools.send(HeapProfiler.takeHeapSnapshot(Optional.of(false), Optional.of(false), Optional.of(false), Optional.of(false)));
try (FileWriter fileWriter = new FileWriter("dump.json")) {
  for (String chunk : chunks.values()) {
     fileWriter.write(chunk);
  }
}