Reject delete model request if model is in Training

[naveentatikonda]

Description

This PR includes the following changes:

• As we cannot safely cancel a job(when delete model request is sent) in which the model is training. We are failing that delete model request until the model completes training.
• When we receive a new Train model request, we are validating if the same modelId is already used by checking the model metadata. But, there is an edge case while performing a delete model request, the model is removed from system index after clearing the model metadata and there is a possibility that the train model request validation succeeds in this short duration between the above mentioned steps. To fix this, we are adding a locking mechanism to add that modelId into a blocked list (which is stored in cluster metadata) before deleting the model metadata. We will remove it from blocked list after deleting the model or even when an exception occurs during the process of delete model request.
• We are adding a new validation in Train Model Request to validate against this blocked list.

Issues Resolved

#365

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov-commenter]

Codecov Report

Merging #424 (4633db8) into main (714ddf3) will increase coverage by 0.29%.
The diff coverage is 89.39%.

@@             Coverage Diff              @@
##               main     #424      +/-   ##
============================================
+ Coverage     84.02%   84.31%   +0.29%     
- Complexity      911      945      +34     
============================================
  Files           130      134       +4     
  Lines          3880     4062     +182     
  Branches        359      370      +11     
============================================
+ Hits           3260     3425     +165     
- Misses          458      473      +15     
- Partials        162      164       +2     

Impacted Files	Coverage Δ
...main/java/org/opensearch/knn/indices/ModelDao.java	81.94% <78.26%> (-1.71%)	⬇️
...ava/org/opensearch/knn/indices/ModelGraveyard.java	93.33% <93.33%> (ø)
...n/transport/UpdateBlockedModelTransportAction.java	93.93% <93.93%> (ø)
...main/java/org/opensearch/knn/plugin/KNNPlugin.java	100.00% <100.00%> (ø)
.../knn/plugin/transport/GetModelTransportAction.java	100.00% <100.00%> (+20.00%)	⬆️
...rch/knn/plugin/transport/TrainingModelRequest.java	84.53% <100.00%> (+0.84%)	⬆️
...knn/plugin/transport/UpdateBlockedModelAction.java	100.00% <100.00%> (ø)
...nn/plugin/transport/UpdateBlockedModelRequest.java	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 714ddf3...4633db8. Read the comment docs.

[martin-gaievski]

Few general comments:

• did you check why test coverage went down? seems your changes are testable, maybe spend some time on adding/extending tests to at least keep it at the same level.
• I suggest make smaller PR in future, it makes review easier/quicker and with higher quality, for example here you could brake something like this: change the model in one PR and API changes in another.

[jmazanec15]

LGTM! Thanks @naveentatikonda great work!

[martin-gaievski]

Looks good, great work Naveen