cxx: incorrect huge typeref values

[techee]

It looks the compiler can get confused and here

https://github.com/universal-ctags/ctags/blob/master/parsers/cxx/cxx_tag.c#L323

it can generate tokens with huge strings. For instance, for

https://github.com/torvalds/linux/blob/master/lib/locking-selftest.c

I get typeref in one tag like

reset_locks lib/locking-selftest.c  /^static void reset_locks(void)$/;" f   typeref:typename:GENERATE_TESTCASE (ABBA_spin)GENERATE_TESTCASE (ABBA_wlock)GENERATE_TESTCASE (ABBA_rlock)GENERATE_TESTCASE (ABBA_mutex)GENERATE_TESTCASE (ABBA_wsem)GENERATE_TESTCASE (ABBA_rsem)GENERATE_TESTCASE (ABBCCA_spin)GENERATE_TESTCASE (ABBCCA_wlock)GENERATE_TESTCASE (ABBCCA_rlock)GENERATE_TESTCASE (ABBCCA_mutex)GENERATE_TESTCASE (ABBCCA_wsem)GENERATE_TESTCASE (ABBCCA_rsem)GENERATE_TESTCASE (ABCABC_spin)GENERATE_TESTCASE (ABCABC_wlock)GENERATE_TESTCASE (ABCABC_rlock)GENERATE_TESTCASE (ABCABC_mutex)GENERATE_TESTCASE (ABCABC_wsem)GENERATE_TESTCASE (ABCABC_rsem)GENERATE_TESTCASE (ABBCCDDA_spin)GENERATE_TESTCASE (ABBCCDDA_wlock)GENERATE_TESTCASE (ABBCCDDA_rlock)GENERATE_TESTCASE (ABBCCDDA_mutex)GENERATE_TESTCASE (ABBCCDDA_wsem)GENERATE_TESTCASE (ABBCCDDA_rsem)GENERATE_TESTCASE (ABCDBDDA_spin)GENERATE_TESTCASE (ABCDBDDA_wlock)GENERATE_TESTCASE (ABCDBDDA_rlock)GENERATE_TESTCASE (ABCDBDDA_mutex)GENERATE_TESTCASE (ABCDBDDA_wsem)GENERATE_TESTCASE (ABCDBDDA_rsem)GENERATE_TESTCASE (ABCDBCDA_spin)GENERATE_TESTCASE (ABCDBCDA_wlock)GENERATE_TESTCASE (ABCDBCDA_rlock)GENERATE_TESTCASE (ABCDBCDA_mutex)GENERATE_TESTCASE (ABCDBCDA_wsem)GENERATE_TESTCASE (ABCDBCDA_rsem)GENERATE_TESTCASE (double_unlock_spin)GENERATE_TESTCASE (double_unlock_wlock)GENERATE_TESTCASE (double_unlock_rlock)GENERATE_TESTCASE (double_unlock_mutex)GENERATE_TESTCASE (double_unlock_wsem)GENERATE_TESTCASE (double_unlock_rsem)GENERATE_TESTCASE (bad_unlock_order_spin)GENERATE_TESTCASE (bad_unlock_order_wlock)GENERATE_TESTCASE (bad_unlock_order_rlock)GENERATE_TESTCASE (bad_unlock_order_mutex)GENERATE_TESTCASE (bad_unlock_order_wsem)GENERATE_TESTCASE (bad_unlock_order_rsem)GENERATE_TESTCASE (init_held_spin)GENERATE_TESTCASE (init_held_wlock)GENERATE_TESTCASE (init_held_rlock)GENERATE_TESTCASE (init_held_mutex)GENERATE_TESTCASE (init_held_wsem)GENERATE_TESTCASE (init_held_rsem)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe1_hard_spin)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe1_hard_rlock)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe1_hard_wlock)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe1_soft_spin)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe1_soft_rlock)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe1_soft_wlock)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe2A_spin)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe2A_wlock)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe2A_rlock)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe2B_hard_spin)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe2B_hard_rlock)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe2B_hard_wlock)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe2B_soft_spin)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe2B_soft_rlock)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe2B_soft_wlock)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe3_hard_spin)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe3_hard_rlock)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe3_hard_wlock)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe3_soft_spin)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe3_soft_rlock)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe3_soft_wlock)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe4_hard_spin)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe4_hard_rlock)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe4_hard_wlock)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe4_soft_spin)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe4_soft_rlock)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe4_soft_wlock)GENERATE_PERMUTATIONS_3_EVENTS (irq_inversion_hard_spin)GENERATE_PERMUTATIONS_3_EVENTS (irq_inversion_hard_rlock)GENERATE_PERMUTATIONS_3_EVENTS (irq_inversion_hard_wlock)GENERATE_PERMUTATIONS_3_EVENTS (irq_inversion_soft_spin)GENERATE_PERMUTATIONS_3_EVENTS (irq_inversion_soft_rlock)GENERATE_PERMUTATIONS_3_EVENTS (irq_inversion_soft_wlock)GENERATE_PERMUTATIONS_3_EVENTS (irq_read_recursion_hard)GENERATE_PERMUTATIONS_3_EVENTS (irq_read_recursion_soft)void file:

Other good files to try are these from the boost library:

boost/phoenix/core/preprocessed/actor_50.hpp
boost/phoenix/core/preprocessed/actor_30.hpp
boost/phoenix/core/preprocessed/actor_20.hpp
boost/phoenix/core/preprocessed/actor_40.hpp
boost/phoenix/scope/preprocessed/lambda_50.hpp
boost/phoenix/scope/preprocessed/lambda_50.hpp
boost/phoenix/scope/preprocessed/lambda_50.hpp

Originally reported in #1111

[pragmaware]

Heh... the problem here is that the file uses the preprocessor to generate statements and we don't have a real preprocessor. All I can do, probably, is refuse to emit types that contain too many tokens.

Another thing we could attempt in the future would be to apply some preprocessing: capture macro names and either expand them properly (hard) or just replace them with a space (simple). It would probably fix this case. Obviously other cases with macros defined in different files would be still hopeless.

[techee]

Heh... the problem here is that the file uses the preprocessor to generate statements and we don't have a real preprocessor. All I can do, probably, is refuse to emit types that contain too many tokens.

The problem is that macros are skipped completely so here the cxx compiler thinks that
GENERATE_PERMUTATIONS_3_EVENTS is immediately before reset_locks which isn't the case. So IMO what the compiler should do is that the token chain that is used for typename should be cut off at the nearest macro definition and not to contain anything before it.

Another thing we could attempt in the future would be to apply some preprocessing: capture macro names and either expand them properly (hard) or just replace them with a space (simple). It would probably fix this case. Obviously other cases with macros defined in different files would be still hopeless.

Trying to do macro expansion doesn't probably make sense indeed as they can be stored in another file.

[masatake]

See #80.
We can give tags file as an input for ctags.

[masatake]

Is -I option helpful?

[pragmaware]

#1123 fixes this particular instance of bug. However a general fix for macros interference would be nice... it's just that I have no idea on how to do it.

[masatake]

Do you agree with this milestone setting?