NH-3530 - memory when using default_batch_fetch_size

[nhibernate-bot]

Kris Desmadryl created an issue — 12th September 2013, 20:45:09:

As since NH3+ setting the default_batch_fetch_size to something has very bad impact on memory as soon the SessionFactory is created. I'm using 3 session factories in an web app (I'm accessing 3 db's). It uses 250MB without the setting but 750MB (default_batch_fetch_size=16) with the setting. When I switch back to 2.1.2 the problem is gone. My webserver recycles because the private memory is reached!

grtz,
Kris

---

Filip Duyck added a comment — 13th September 2013, 7:29:56:

In addition to what Kris said, I'd like to share some results from our dotTrace profiling session.

Performance trace can be seen at -- notice how in NHibernate 3, EntityLoader..ctor is called about 10x as much as in 2.1.2 (even though the calls to BatchingEntityLoader.CreateBatchingEntityLoader are more or less of the same order of magnitude).

We also did memory profiling which showed similar results, which you can see at . In this trace, you can see that BatchingEntityLoader.CreateBatchingEntityLoader creates many, many more objects in 3.3.3 than it did in 2.1.2.

I have looked at the code of both methods but I can't really identify a change. Our setup for NH2.1.2 and NH3.3.3 use virtually the same configuration. The problem is, as Kris mentioned, amplified by the fact that we have defaultbatch_fetchsize set to 16 which causes more looping inside the CreateBatchingEntityLoader method, but this setting is the same during 2.1.2 and 3.3.3 testing.

Can anyone from the NH team shed a light on this? Especially the memory problem is killing us in production.

---

Filip Duyck added a comment — 16th July 2014, 13:13:52:

It's been almost a year. Is anyone ever going to look at this?

[hazzik]

It seems like this is the regression of following: a35aaa6

[LeMaciek]

We wanted to start using default_batch_fetch_size, but after enabling it, our tests started to throw OutOfMememory exceptions. Investigation led me to this issue... This is the difference in memory usage in our case, we are using NH 5.2.3

with default_batch_fetch_size=20...

• alocations
  
• memory
  

without

• allocations
  
• memory
  

Unfortunately this issue prevents us from using this feature...

[fredericDelaporte]

You may use instead discrete batch-size settings (hbm attributes) on classes and collections requiring it, instead of enabling it for all of them through default_batch_fetch_size. This will reduce the memory impact, but it is also more work on the developer side.
That is of course a work-around. This feature should not have such a memory impact.

[maca88]

The memory impact is due to the BatchingEntityLoader creating many EntityLoader instances which have pre-built sql statements for fetching the entity for different batch sizes. When batch fetch size 20 is used the BatchingEntityLoader will create eleven EntityLoader instances with batch sizes: 20, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, which is the reason for the high memory usage. One way to solve the issue is to port HHH-7746 which contains an additional setting hibernate.batch_fetch_style that affect how batching of entities is done. By using BatchFetchStyle .DYNAMIC the memory usage would not be affected as the sql statements are created dynamically, which would solve the OutOfMemoryException for those that have a limited amount of memory, but it will use more cpu due to dynamic sql generation.

[fredericDelaporte]

Related: #1904, which would be an intermediate solution, maybe insufficient for some applications (if they usually end up using most loaders during their lifetime).

[hazzik]

Additional details in #2427

[Aldomat]

We ran in this issue, too. We have got round about 300 hundreds web applications using NHibernate under the hood. The usage of the batch size would help us to avoid SELECT N+1-problems etc. Thus a solution similar to Hibernate would be really great.
What can I do to force the implementation?

[AlexanderMuylaert]

Hi All

We're a sponsor of the firebird project (we sponsor practically the complete .net driver for firebird). We migrated from LLBLGenPro to NHibernate. But, we're facing a severe issue with the problem described above. We have 50 databases on a server and hundreds of tables. Batching is set to 20 or 100 depending on table. We don't set default_batch_fetch_size, but still it requires a few GB just to start the project.

Is there somebody who can implement DYNAMIC batching on short term? We're willing to sponsor the project.

kind regards

Alexander

[Aldomat]

Hi Alexander,

I reviewed the code and to me it is obvious that the memory won't be released. I have a fix locally which I will provide next week.

Kind regards,
Carsten

[AlexanderMuylaert]

Hi Carsten

Any progress on this topic about dynamic sql generation as opposed of keeping all these "in ()" statements cached.

kind regards

Alexander

[AlexanderMuylaert]

Hi

Any luck on this? It is getting extremely urgent for us since it is actually freezing our servers all the time.

many thanks

Alexander

[fredericDelaporte]

As far as I know, no one works on this trouble currently. PR are welcome.

[cal-schleupen]

We have a solution on this. In our next sprint (three weeks) we need to prove that this has the expected behaviour. The problem with the solution above is that the memory will not be freed. Hopefully we can provide a solution in first week of october.

Kind regards
Carsten

[AlexanderMuylaert]

Hi all

How are you doing on this issue? Can I follow it up somewhere?

many thanks

Alexander

[cal-schleupen]

Hi Alexander,
currently a developer has reviewed the fix and veryfied that the memory won't grow. That has been successful on first stage. Now we are checking that against a process with bulk data. Than we try to upload that fix.

The fix provided in https://github.com/nhibernate/nhibernate-core/pull/1904/files was unsufficient because the memory still increases. So we needed to introduce a factory, so that the loaders will be created by a factory and the garbage collector can free memory.

The following shows the difference:

The fix will hopefully available during the next two weeks. Sorry for the delay. We are very busy at the moment.

Kind regards
Carsten

[bahusoid]

I guess it would work. But it's not very nice solution to re-create loaders on each load operation. So it's good only for custom forks. The proper way to solve it described here

Is there somebody who can implement DYNAMIC batching on short term? We're willing to sponsor the project

@AlexanderMuylaert If it's still a deal I can port DYNAMIC batching. I will have time to work on it in 2 weeks and I think I would need 1 week to have it implemented (though you would still need to stick to custom fork for some time till it's reviewed by other members)

[cal-schleupen]

I agree that the other implementation would be better. But our tests show that we can solve select N+1-problems and get better performance with that solution. Memory was also not increasing.
Unfortunately, we don't have the knowledge to implement it as described in #1316 (comment) and not the time to do it.
Sorry!