[SPARKNLP-1031] Solves Dependency Parsers training issue

[danilojsl]

Description

This PR introduces critical enhancements and optimizations to the processing of the CoNLL-U format, which is instrumental in the training of Dependency Parsers. The key improvements include:

• Enhanced Multiword Token Handling: This update ensures proper processing of lines identified by id columns as multiword tokens (e.g., 2-3 no _ _ _ _ _ _ _ _). This adjustment guarantees that multiword tokens are accurately recognized and managed throughout the parsing process.

• Improved Handling of Missing uPos Values: Before this change, lines with unavailable uPos values could disrupt the parsing flow. With the current enhancements, the system gracefully handles such scenarios, ensuring uninterrupted parsing operations even in the absence of uPos values.

Beyond these functional enhancements, this PR undertakes a comprehensive refactoring of the underlying codebase. The refactoring efforts focus on enhancing code readability, cleanliness, and maintainability. These improvements pave the way for easier future modifications and debugging, aligning with best practices in software development.

Motivation and Context

Solves issue #14214

How Has This Been Tested?

Screenshots (if appropriate):

• Local Tests
• Google colab notebook

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• Code improvements with no or little impact
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING page.
• I have added tests to cover my changes.
• All new and existing tests passed.