[NSE-382]Fix Hadoop version issue

[weiting-chen]

What changes were proposed in this pull request?

1. Add hadoop.version for hadoop client in arrow-data-source and native-sql-engine
2. Add profile setting for hadoop version control, default use hadoop 2.7.4
3. Fixing one issue for the variable: hadoop.version does not apply to hadoop-client library in compiler phase before.
4. Move the order of profile section to higher for better reading

How was this patch tested?

Tested in local server.

[github-actions]

#382

[zhouyuan]

the failure seems due to ${hadoop.version} is not passed from parent pom if compile core only
"mvn package -pl native-sql-engine/core"

[zhouyuan]

looks a bit odd to me, this should be the same as original way?

[weiting-chen]

Not the same, the issue I found is ${hadoop.version} in arrow datasource and columnar plugin didn't catch the value from parent.
The solution can help to deliver ${hadoop.version} to both sub-projects.
My testing is to use mvn help:evaluate to confirm the difference.
The original way shows ${hadoop.version} as 2.7.4 when enabling -Phadoop-3.2.

[zhouyuan]

tried the same cmd here, my hadoop version is correct

[zhouyuan]

also add the maven version limit here?
https://github.com/apache/spark/blob/master/pom.xml#L118

[weiting-chen]

Yes, we can do this by using the same maven plugin.
Let me add it and submit a new PR.

[weiting-chen]

the failure seems due to ${hadoop.version} is not passed from parent pom if compile core only
"mvn package -pl native-sql-engine/core"

I can reproduce the error when adding -Pfull-scala-compiler, if remove the parameter, the compile can run successfully.
I am investigating the issue.

[weiting-chen]

For NullPointer issue,
The root cause is because of activeByDefault usage.
Based on the document,
https://maven.apache.org/guides/introduction/introduction-to-profiles.html
"This profile will automatically be active for all builds unless another profile in the same POM is activated using one of the previously described methods."

There is a potential issue for arrow-memory.artifact setting and we fix it in the PR as well.