skill: Add reasoning skill about siblings

[russellb]

This skill was inspired by one of the tests included in mt-bench. It included some questions that the model answered incorrectly. When given the same question manually, I got the following incorrect answer:

Q: David has three sisters. Each of them has one brother. How many
brothers does David have?
A: David has three sisters, and each of them has one brother, so it
must be the same brother in all cases. Therefore, David has 1 brother.

[instruct-lab-bot]

Beep, boop 🤖, Hi, I'm @instructlab-bot and I'm going to help you with your pull request. Thanks for you contribution! 🎉

I support the following commands:

• @instructlab-bot precheck -- Check existing model behavior using the questions in this proposed change.
• @instructlab-bot generate -- Generate a sample of synthetic data using the synthetic data generation backend infrastructure.
• @instructlab-bot generate-local -- Generate a sample of synthetic data using a local model.
• @instructlab-bot help -- Print this help message again.

Note

Results or Errors of these commands will be posted as a pull request check in the Checks section below

Note

Currently only maintainers belongs to [[taxonomy-triagers taxonomy-approvers taxonomy-maintainers labrador-org-maintainers instruct-lab-bot-maintainers]] teams are allowed to run these commands.

[russellb]

Originally posted as #365

[russellb]

@instructlab-bot precheck

[instruct-lab-bot]

Beep, boop 🤖, Generating test data for your PR with the job type: precheck. Your Job ID is 180. The results will be presented below in the pull request status box. This may take several minutes...

[russellb]

@instructlab-bot generate

[instruct-lab-bot]

Beep, boop 🤖, Generating test data for your PR with the job type: sdg-svc. Your Job ID is 181. The results will be presented below in the pull request status box. This may take several minutes...

[instruct-lab-bot]

Results for job ID: 180 using the model merlinite-7b!

Results can be found here.

[instruct-lab-bot]

Results for job ID: 181 using the model sdg service backend!

Results can be found here.

[mingxzhao]

Based on SDG, seems model will have a bit of trouble coming up with good synthetic data. One problem that I can see beyond logical flaws is the gendering of the names. We assume David is male but model may not make same assumption, thus the question of # of brothers is more ambiguous.

[russellb]

Based on SDG, seems model will have a bit of trouble coming up with good synthetic data. One problem that I can see beyond logical flaws is the gendering of the names. We assume David is male but model may not make same assumption, thus the question of # of brothers is more ambiguous.

good feedback, thanks. I'll look at trying to improve this at some point, or I'm happy to take suggested improvements from anyone interested.

For reference, this was based on something the model didn't do well with mt-bench.

https://github.com/lm-sys/FastChat/blob/f22f2194c9152a25d2987e5118206e3bbb9efd5e/fastchat/llm_judge/data/mt_bench/question.jsonl#L24

[mingxzhao]

Ah I see, then I may just have to go argue with the mt-bench mods. Maybe try prefacing the question and confirming David is a male. I have doubts if this will allow the model to solve but it would at least tell us if the assumption of David's gender was the issue with the model reasoning.

[instruct-lab-bot]

Beep, boop 🤖, Hi, I'm @instructlab-bot and I'm going to help you with your pull request. Thanks for you contribution! 🎉

I support the following commands:

• @instructlab-bot precheck -- Check existing model behavior using the questions in this proposed change.
• @instructlab-bot generate -- Generate a sample of synthetic data using the synthetic data generation backend infrastructure.
• @instructlab-bot generate-local -- Generate a sample of synthetic data using a local model.
• @instructlab-bot help -- Print this help message again.

Note

Results or Errors of these commands will be posted as a pull request check in the Checks section below

Note

Currently only maintainers belongs to [[taxonomy-triagers taxonomy-approvers taxonomy-maintainers labrador-org-maintainers instruct-lab-bot-maintainers]] teams are allowed to run these commands.

[jjasghar]

@instructlab-bot precheck

[instruct-lab-bot]

Beep, boop 🤖, Generating test data for your PR with the job type: precheck. Your Job ID is 377. The results will be presented below in the pull request status box. This may take several minutes...

[instruct-lab-bot]

Results for job ID: 377 using the model instructlab/granite-7b-lab!

Results can be found here.

[bjhargrave]

I relocated the folder to the new taxonomy organization. I also updated the seed examples to be more specific about whether the subject is a brother or sister instead of making assumption from the name.

[mcorbin-ibm]

@bjhargrave

I relocated the folder to the new taxonomy organization (compositional_skills/philosophy/logic/induction/siblings/)

I think I would change "induction" to "inductive_reasoning" because induction has too many other connotations and inductive_reasoning makes it more apparent.

[instruct-lab-bot]

Beep, boop 🤖, Hi, I'm @instructlab-bot and I'm going to help you with your pull request. Thanks for you contribution! 🎉

I support the following commands:

• @instructlab-bot precheck -- Check existing model behavior using the questions in this proposed change.
• @instructlab-bot generate -- Generate a sample of synthetic data using the synthetic data generation backend infrastructure.
• @instructlab-bot generate-local -- Generate a sample of synthetic data using a local model.
• @instructlab-bot help -- Print this help message again.

Note

Results or Errors of these commands will be posted as a pull request check in the Checks section below

Note

Currently only maintainers belongs to [[taxonomy-triagers taxonomy-approvers taxonomy-maintainers instructlab-bot-triagers instructlab-bot-maintainers oversight-committee]] teams are allowed to run these commands.