Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(metrics): add traditional methods for comparison #5

Merged
merged 3 commits into from
Aug 20, 2024

Conversation

calico-1226
Copy link
Member

@calico-1226 calico-1226 commented Aug 17, 2024

Description

Describe your changes in detail.

Motivation and Context

While some traditional methods may not precisely capture human value preferences, they are usually faster and cheaper to compute. Therefore, we will integrate some traditional metrics into the SafeSora library, expecting they will complement feedback-based methods and enable comparative analysis.

These methods include:

  • PSNR vs. Informativeness: PSNR measures the quality of a reconstructed or compressed image/video by comparing it to the original, and assessing their similarity. We evaluate dynamic changes in the video by analyzing PSNR between the first and subsequent frames.
  • HPSv2 vs. Aesthetic: HPSv2 predicts human preferences for image beauty. We use it to assess the aesthetics of each video frame, averaging the results for an overall aesthetic measure.
  • CLIP vs. Instruction Following: CLIP assesses instruction adherence by evaluating the similarity between the prompt and video frames.

More methods will be added gradually in the future.

  • I have raised an issue to propose this change (required for new features and bug fixes)

See #4

Types of changes

What types of changes does your code introduce? Put an x in all the boxes that apply:

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds core functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation (update in the documentation)

Checklist

Go over all the following points, and put an x in all the boxes that apply.
If you are unsure about any of these, don't hesitate to ask. We are here to help!

  • My change requires a change to the documentation.
  • I have updated the tests accordingly. (required for a bug fix or a new feature)
  • I have updated the documentation accordingly.
  • I have reformatted the code using make format. (required)
  • I have checked the code using make lint. (required)
  • I have ensured make test pass. (required)

Co-authored-by: linghaiCTL <2358044947@qq.com>
@calico-1226 calico-1226 added the enhancement New feature or request label Aug 17, 2024
@calico-1226 calico-1226 marked this pull request as ready for review August 20, 2024 08:10
@calico-1226 calico-1226 requested a review from linghaiCTL August 20, 2024 08:10
@calico-1226 calico-1226 merged commit 16a92fe into PKU-Alignment:main Aug 20, 2024
1 check passed
@calico-1226 calico-1226 changed the title feat: add traditional methods for comparison feat(metrics): add traditional methods for comparison Aug 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants