Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improving log message when task vectors are close to zero #9

Open
bergr7 opened this issue Apr 26, 2024 · 0 comments
Open

Improving log message when task vectors are close to zero #9

bergr7 opened this issue Apr 26, 2024 · 0 comments
Labels
good first issue Good for newcomers

Comments

@bergr7
Copy link
Collaborator

bergr7 commented Apr 26, 2024

Description

If there is almost no difference between the tensor of the base model and the tensor of a model, task vector values are close to zero. If all the task vectors are close to zero, then the merge method of the merge method class just returns the base tensor. The current warning message "No task vectors. Returning the base model tensor." is not very helpful:

class TaskArithmetic(MergeMethod):
    def merge(
        self,
        weight: ModelWeight,
        base_model_tensor: torch.Tensor,
        models_tensors: Dict[Model, torch.Tensor],
        merge_method_settings: Union[TaskArithmeticSettings, TiesMergingSettings],
        base_model: Model,
    ) -> torch.Tensor:
        base_tensor_dtype = base_model_tensor.dtype

        task_vectors: Dict[Model, torch.Tensor] = self._get_task_vectors(
            base_model_tensor, models_tensors
        )

        if not task_vectors:
            logger.warning("No task vectors. Returning the base model tensor.")
            return base_model_tensor
            
            ...

It should provide a better explanation.

@bergr7 bergr7 self-assigned this Apr 26, 2024
@bergr7 bergr7 added the good first issue Good for newcomers label Apr 26, 2024
@bergr7 bergr7 removed their assignment Apr 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

1 participant