[BUG] Add-PnPDataRowsToSiteTemplate command makes additional, unnecessary calls to get field values from items one-at-a-time

[joshtransient]

Reporting an Issue or Missing Feature

I'm reporting a massive performance issue with the Add-PnPDataRowsToSiteTemplate command which makes it more or less unusable on large lists. Previously, I reported that the command didn't respect Scope="RecursiveAll" when using a CAML query. This was fixed with #3152, and the source now correctly shows that Scope="RecursiveAll" is being added.

To test this functionality, I ran the following command on a list with 205K+ items:

Add-PnPDataRowsToSiteTemplate -Path .\huge.xml -List '<Title of list>' -Query '<RowLimit>2500</RowLimit>'

Watching the Fiddler trace, I see the commands resolve the list, then make the GetItems query with the CAML in place. What I did not expect was that the command would then make individual requests to every single item to get each item's field value. This is the most inefficient thing that could be done with a query like this.

Enumeration of a large list can already take minutes, and I was throttled several times with 60-180 second Retry-Afters waiting for that enumeration. But then, come to find out, that response doesn't even contain any actual, usable list data, only the UniqueId and the ObjectIdentity values. Add more throttling every 100-200 items, and you can see how this command is completely unsustainable in its current form.

Please update this command so that it gets the items and their field values in bulk, instead of retrieving item field values one item at a time.

Expected behavior

Add-PnPDataRowsToSiteTemplate should complete in a timely fashion, now that it has the ability to page thanks to #3152.

Actual behavior

Add-PnPDataRowsToSiteTemplate takes hours (potentially days) for large lists because it is making individual requests for field values on every item in the list, even after retrieving them.

Steps to reproduce behavior

Start a Fiddler trace, then watch the requests that a command such as Add-PnPDataRowsToSiteTemplate -Path .\huger.xml -List '<Title of list>' -Query '<RowLimit>2500</RowLimit>' makes on any list.

What is the version of the Cmdlet module you are running?

2.1.84-nightly

Which operating system/environment are you running PnP PowerShell on?

• Windows
• Linux
• MacOS
• Azure Cloud Shell
• Azure Functions
• Other : please specify

[gautamdsheth]

Thanks @joshtransient for raising this issue.
Fair point, much appreciated your investigation into this.

Have removed the unnecessary loading of the individual list items as well as also managed to fix another bug in the same cmdlet.

It will be available in tomorrow's nightly build and the next major version.
Let us know if it helps.

[joshtransient]

Installed 2.2.0 and this went way faster. Thanks @gautamdsheth