Code completion breaks when using compilation cache with --macro server.setModuleCheckPolicy(...)

[jeremyfa]

It's taken me a while to gather these informations, but here we are, these are the steps I went into:

1. I have a big project with a lot of classes, macros, import.hx files etc... which compiles fines and provides code completion fine

2. But even with haxe compilation server enabled, code completion is slow, probably because there are a lot of inter-dependencies between classes (inheritance, autoBuild macros etc...) and cache gets invalidated pretty quickly as soon as I start working on the project (with vshaxe).

3. Thanks to this page, I successfully made code completion MUCH faster by using --macro server.setModuleCheckPolicy(['package1', 'package2', ...], [NoCheckShadowing, NoCheckDependencies], true).

4. Unfortunately, as soon as I use setModuleCheckPolicy(), code completion is super fast and great but breaks when querying toplevel suggestion items on some files (not all, but many of them in the project). The completion server fails with error: Type name X redefined from module X (more specifically, I have have a Project.hx at the root package of my project, and the error is Type name Project redefined from module Project).

5. As it has been pretty complicated to provide a reproducible sample (smaller projects don't break, but the one I use daily does), I went a step further, downloaded and built haxe from source (took tag 4.0.0-preview.3), and commented out, in typeload.ml this line to see how it behaves. It turns out code completion works perfectly well after commenting out this line and is super fast thanks to the setModuleCheckPolicy() addition.

I understand that this "module A redefining module B" check is necessary when compiling a haxe project, but this check seems too agressive when using completion cache + setModuleCheckPolicy(), or somehow in that situation data gets invalid and triggers the error.

I really hope this issue can be solved because the use of --macro server.setModuleCheckPolicy() is mandatory to get an acceptable code completion speed in my project. Not being able to do so is so far the biggest issue I have with Haxe right now.

That said, now that I can compile a custom version of haxe compiler 🎉, I am open to any suggestion to help me find and fix the issue (like building haxe compiler with some more debug flags and provide more information?).

Meanwhile I am getting close to perfect code completion with a patched haxe compiler that got the commented-out line I mentioned above, but that doesn't look like a future-proof solution though ☹️

So, any help is welcome and will be very much appreciated...

[Simn]

To clarify, does this only occur for one specific file, and only for toplevel?

[jeremyfa]

What happens is usually this:

• I start editing some file, everything is fine, both toplevel and dot completion work

• Then I go to another file, start typing, toplevel completion fails with the Type name X redefined from module X error

• At this point, I can go to any file, toplevel completion doesn't work anymore until I reset the compiler cache (by reloading the vscode window). Dot completion still works fine however. (at each toplevel query, same Type name X redefined from module X error).

If needed, I can send to somebody of the HaxeFoundation the project reproducing the issue (and ensure it's easy to setup: just an hxml file with all related files). The project is pretty big but reproduces the problem consistently, so it may help a lot

[ncannasse]

Being able to fix the issue for real would be a great help. Simon can you look into it?

…

On Thu, May 3, 2018 at 11:15 AM, Jérémy Faivre ***@***.***> wrote: What happens is usually this: - I start editing some file, everything is fine, both toplevel and dot completion work - Then I go to another file, start typing, toplevel completion fails with the *Type name X redefined from module X* error - At this point, I can go to any file, toplevel completion doesn't work anymore until I reset the compiler cache (by reloading the vscode window). Dot completion still works fine however. If needed, I can send to somebody of the HaxeFoundation the project reproducing the issue (and ensure it's easy to setup: just an hxml file with all related files). The project is pretty big but reproduces the problem consistently, so it may help a lot — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6993 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA-bwCQn3UsvMUCRKWFopOqzaWO_FYT6ks5tusqigaJpZM4Tt-8u> .

[Simn]

I can take a look

[jeremyfa]

In case you don't succeed to reproduce it on your side, feel free to ping me and I will manage to send you a project that reproduces the error

[kLabz]

I have a similar issue, with a big project too, and commenting the same line (well it has moved since then; I am using haxe 4 rc2) solves it too.

The differences:

• I am not using server.setModuleCheckPolicy() at all
• It does not happen with a top-level type, but with one of my (macro helper) class
• It behaves exactly like issue Type name X redefined from module Y #5588 although it is not an enum

So my issue is different, but the symptoms are the same and I think there might be some kind of a link between this issue and #5588. I hope it can help you figure out what is going on; I still have no reduced example, though..

Edit:

• After turning the error into a print, I have a lot of modules "redefined" from themselves, including top level module (Any) and std modules (haxe.Log, haxe.Json, etc.). I'll try to find out what is going on based off this module list.

[kLabz]

Problem solved for me!

I had this in my build.hxml:

--macro addGlobalMetadata('', '@:build(module.importing.the.redefined.Module.build())')

I changed '' to specific packages, and it removed the error.

(This does seem similar to the issue with setModuleCheckPolicy somehow now)

[Gama11]

Hm, that's a problem, it's common to use addGlobalMetadata like that..

[kLabz]

Yeah, but I cannot reproduce the issue with simple projects.. I have no idea what I am doing wrong in this one =/

[jeremyfa]

Regarding the initial issue, @Simn helped me to nail the underlying problem a while ago: dependencies between haxe files where making server.setModuleCheckPolicy() settings break compilation server.
Tidying up how files depend on each other solved the issue for me (even if that was not an easy task).

However this issue specific to C# and Java targets still remains.

[Simn]

I guess this one is resolved as well.

[jeremyfa]

I recall still having hit similar issues, even after trying to tidy class dependencies, but at some point stopped using server.setModuleCheckPolicy() because compilation server was still breaking pretty often without knowing exactly why and couldn't completely change the project architecture. Cannot really say if the problem comes from the way the classes are put together or if there is really another issue.

Anyway, at the moment I am sticking with slower code completion, queries take more than 5 seconds on big projects, not so happy about it. So far, having no simple way to getting fast code completion on larger projects is my biggest and only serious issue with haxe. The rest is amazing and close to perfect!

[Simn]

Ok, in that case I'll reopen because I would like to figure this out.

We're currently working with our partner InnoGames to sort out various IDE-related problems they are having. There's a good chance that things will improve for you as a side-effect. Still, I'm interested in finding out why completion is so bad for you.

[jeremyfa]

I suspect too many classes are being invalidated in the server when I change a single one because of some remaining chain of dependency, making code completion slower and slower, or breaking completion when using setModuleCheckPolicy(). I remember having fixed the Type name X redefined from module X errors I had thanks to your help... until it appeared again later at some other place in the project without really understanding why.

Although I could probably try to fix more and more how classes are depending on each other to improve completion delay (and I may try to spend some time again on this), there is a limit to this approach as I cannot investigate every slow-down everytime I add new code in the project that risks to create some more interdependency in the code base. I also think that there may be other developers that may hit this issue as well because they accidentally created something the haxe server doesn't like without knowing it.

Ideally, one could hope that there is still room for improvement on haxe side as well, and that the server could cope better with situations like this.

I might be wrong but the behaviour I noticed with haxe server (through vshaxe) is:

• at start, one full compilation is done, caching all haxe files
• then when querying completion, every invalidated file is invalidated forever, and there is no way to get it "cached" again, unless restarting the server
• meaning: the server can only get slower and slower as more and more files need to be processed again at each query, even the ones not modified again on the following queries, and only restarting the server and performing a "full compilation" will "reset" the state to "all files cached"

An ideal expectation would be that any invalidated file in one completion query would be "cached" again so that a following query would use it and only have to re-process the files that got invalidated from this new query, but I guess there are reasons that this is non trivial to do.

That said, if at some point you want/need to investigate completion on this setup to improve haxe compiler, I can provide a "plain hxml" version of one of these projects again

I may try to spend some more time investigating on my side as well with verbose haxe server, but it's been pretty hard to really understand what is happening as I am not confortable with haxe compiler internals and what creates chains of invalidations

[Gama11]

but I guess there are reasons that this is non trivial to do.

Yeah, the compiler skips various things during completion requests so it's not safe to cache those results.

Compiling through the server does actually update the cache that's used for completion though, provided the same arguments are used (i.e. the same "context" in the cache).

[nadako]

Yeah, for big codebases it's advisable to build often through the same cache server as the one used for completion so the cache updates often...

But in general the problem of invalidation is pretty tricky. Currently a change anywhere in the module invalidates all the modules that directly or indirectly use anything from that module. We discussed two things that can be done to improve this:

• field level dependency, to precisely invalidate specific fields instead of whole modules. This should greatly reduce the amount of stuff being invalidated by a change in one field.
• signature-based dependencies: the the user of a module (or field in case of field-level dependencies) should only be invalidated if the "signature" changes (names, arguments, access, etc.), not when the implementation (expressions, imports, internal dependencies)

Both of these things are huge reworks in the cache system so that is going to take a long time, unfortunately.

[jeremyfa]

Yes, field level dependency and signature-based dependencies would be very good improvements for sure, but I can imagin how hard it would be to make these radical changes in the cache system indeed.

[kLabz]

I'm not sure what to do exactly here. At first glance, there doesn't seem to be anything more to it than the regular "Type X is redefined from X" one which would be #8368 I suppose.

[jeremyfa]

Didn't try setModuleCheckPolicy() for a long time, so no idea if this is still an issue or not, but I can take some time to try it again