We are getting a strange memory leak on latest version after upgrade

[DigitalTrading]

Hi,
We have a site that is several years old and gave a facelift and update which took us from DNN 8 and openstore 4.0.1 to latest version.
Since the upgrade, we have had an issue with intermittent CPU spikes to 100% for up to 2 minutes.
On the site, we see where the products are listed the thumbnail images are not all loading during the CPU spike. Once the page finally loads the remaining thumbnails load and it all works quickly again. This typically but not always is something we see if the site has not been visited for several hours even though we have a keep-alive on the home page maintaining the app pool every 5 minutes.

I have also noticed on a fresh install of the latest openstore versus the upgraded version we are having an issue with there is a new option in the admin > settings > store config called SEO Images. This option is not showing on the upgraded version.

We have tried turning off all schedules, custom urls, and removed all modules from the site with the exception of the OS_Product View and OS_CategoryMenu and still get the issue, so believe the issue is associated with one of these.
We have

We are using DNN 9.1-.2 and openstore 4.1.11.

Please let me know how to further troubleshoot the issue?

Regards
Roger

[DNNMonster]

It's possible that it's related to virus software or a bloated temp directory. You can try changing the web.config compilation tempDirectory to a subdirectory in your www folder which sometimes improves things especially if there is a lot of activity happening in the default temp dir. I've see this improve performance on shared hosting but it may not make any difference in other environments.

[DigitalTrading]

Thanks for the reply,
I am testing now to see if disabling the antivirus has any impact however I am not confident this is the cause.
The NBSTemp folder has had all the files removed from it several times and this has not made any impact, currently, it only has 35 files in it after 4 days since I cleared it.
The site is running on a dedicated server with only 2 other sites running on it so I am not sure the issue is with server resources.
What I have also observed is that when the list of products page is opened and there are thumbnails that don't load the cp usage increases based on the number of thumbnails waiting to load. If its just 1 image the CPU usage is at around 30% at 4 it is 100% and this lasts for around 30 seconds. Any more than 4 then the time the CPU is at 100% increases. This may just be coincident but it does look like images are the issue.
Also, can you confirm why in the back office the SEO images setting is not showing on the upgraded version?

[DNNMonster]

I am referring to the asp.net temp directory not the NBSTemp directory. https://docs.microsoft.com/en-us/previous-versions/ms366723(v=vs.140)?redirectedfrom=MSDN

[DigitalTrading]

I have created a different temp folder for IIS and it has not made any difference. I am still getting the issue where the CPU spikes to 100% and stays there for several minutes before the thumbnail images load.
I have several instances of OS and have never had any of these issues on any server. The only difference is the fact this is an upgrade from an earlier version.

[DNNMonster]

It will require someone to test and isolate the root cause then. If you're looking at it right now then it would be a great contribution! I am busy on another project but if I get a chance I will take a look,

[leedavi]

I hid the SEO image, because I don't believe it helps SEO and it was creating a BIG folder of multiple copies of images. A client paid for that enhancement, but it caused more problems than it solved. I think it was removed from all new versions after that... is it still there on the latest version?

As for the CPU spike... we had a problem with an upgraded site where the data was not correct. The category menu seemed to have a infinite loop. I think we found the problem in the DB and fixed it there.

Try removing the category menu and see if you have the same problem.

[DigitalTrading]

Ok so the top 4 longest-running processes during the 100% cpu time were:

36s NBStore trying to read a template file
32s Razor Engine using Reflection during compilation
31s Razor Engine using reflection during compilation
29s Razor Engine using reflection during compilation

So it looks like the issue is the compiling of NBstore templates using Razor engine.
I have found this article which may be relevant.

Antaris/RazorEngine#347

I will try just removing the OS_CategoryMenu from all the pages and advise once I have a result.

I can also confirm on the version I used from the download link on the main site is version 4.1.11 which still has this option.
Regards
Roger

[leedavi]

That problem seems unresolved. The first compile is slow... but 37 seconds is a problem.

Could it by a locked file ?

The reflection is probably the category menu. That's the only time I've seen the razorengine taking longer than I would expect.

[DigitalTrading]

I have deleted all the categories using the admin option and then deleted all fields in the database with a prefix of cat* from the typecode column.
I then recreated all the cats and reassigned the products.
This didn't make any difference.
I have attached the output from the debug we are running for the latest error. It is showing the result from a 25% cpu usage for over 15 seconds and refers once again to the System.Reflection.AssemblyName.
May be this will give you some more insights as to what our issue may be
dumpfile_printableparties.txt
.
Is there anything else you can suggest? As i am now at the point where I am going to have to do a full rebuild and add all the products again manually.

Thanks
Roger

[leedavi]

OK, so we can cross off category data error.

Is it only with the product view?

One thing I am sure is, is we don't run DNN9.1, we found it unstable for everything, not OpenStore. I think we upgraded all our installations to DNN9.4+
(I can't remember exact version we decided, I think it was the version that moved to .Net Framework v4.7.2 DNN was causing a problem with language mainly.)

I'm unsure if that could be the issue. But it does leave you with the unpleasant thought of upgrading.

What about any errors on the Windows Event log. In the past we have had a VPS having errors for no obvious reason. We reinstalled the same version of windows and the error disappeared. (Another horrible solution)

Is this installation running by itself on a VPS or with other OpenStore installations?

What plugins to OpenStore do you use?
What modules ?

[DNNMonster]

9.1-.2 is a typo and should read 9.10.2... Or so I presume. Roger is on the latest.

[DNNMonster]

I am not sure reducing the db activity will matter but there is a branch here which may help us in that area: faab48a

It needs to be tested by others but the idea is to eliminate the 2 queries per product which is happening presently when the list body template loads. I don't see issues in my local testing but others here may immediately see some problems that need fixing or make this approach unusable.

The idea is that we get all the data we need in our original list query GetList sproc so rather than newing up our body template ProductData the standard way (which hits the db) we can introduce a new ctor so we can just pass in our NBrightInfo from collection returned by the previous query.

These are all indexed queries and super fast so I'm not sure this will really matter but in general reducing db activity is a positive.

[leedavi]

It is a good idea, but I doubt the extra call causes much extra load. (Unless you have massive pages.) OpenStore also cache heavily, so I doubt you'll see a difference... but I'll be happy to be corrected on that thought.

[DNNMonster]

It's first render we're trying to improve so there will be no cache. In fact that is one more operation we'll be eliminating because each product hits the db twice and also writes to cache. But you're probably right and there may be little difference. If Roger is compiling then hopefully he can do a test for us and we'll know.

[leedavi]

OK

[leedavi]

@Roger, If you copy the website to your local PC, doe sit still have the same problem?

[DigitalTrading]

Morning all,
Sorry for the delay, we have been working on a complete rebuild of the site as we were not able to get to the bottom of the issue.
We did try it locally and on several different servers with the same result.

While testing what we found was that by removing the OS_Product View and the OS_Checkout module from all pages the issue went away.

For the rebuild, we have created a new install of the latest DNN and OS modules and set up each page, product, and category from the ground up. When we did this from data exported the problem persisted on the new site.
The new site we are working on now is using the skin and OS template files from the original site which have not been rebuilt.

One thing which is odd and possibly worth thinking about is the localisation. On the original site and site where we imported the data, we added the UK language, set it as default, and disabled the US language.
OS was creating blank empty articles and categories for the disabled local and the system was doing so very odd things such as when editing a category name after saving it would disappear. Adding a new category would remove the name of a different category. It was just really odd.

I am not sure if any of this helps but I am hoping that by the end of the week to have the new build live. I will then let you know if the issue has been resolved.
Regards

Roger

[leedavi]

Hi Roger,

Sorry I didn't fully get what was happening now. Does the new site work correctly, but fails when you import data?

As for the multiple records for language, usually they are never a problem. We also change en-US to en-GB and then disable the en-US, which creates fake/unused "en-US" records... they can be deleted from the DB, but usually they never cause a problem. They are only created on first setup.

As for the category name disappearing, that is strange. Could it have been the "language" cookie from DNN was still on en-US when editing?