OMI_personality proposal

[antpb]

I wanted to bring up an extension I have been working on under my own SXP namespace. Magick ML, 3OV and at least one other project that I know of want to use the extension I mocked up here: https://3ov.xyz/personalized-chatbots-in-the-metaverse/

I think since enough different platforms and people are working towards this it would make a good experimental extension within the OMI glTF group. I'm happy to champion this one.

Overview

The OMI_personality extension allows you to specify a personality for a glTF node representing a entity. This extension can be used in virtual worlds, where characters can have unique personalities, and interactions with them can be enhanced by providing additional information about their behavior and dialogue.

Example

To use the "OMI_personality" extension, you must first specify it in the extensionsUsed property of your glTF file.

{
	"extensionsUsed": ["OMI_personality"]
}

Next, apply the extension to a child node of the glTF file. The node's position and rotation data can be used to determine the location of the spawn point in the scene.

{
	"nodes": [
		{
			"name": "cat",
			"extensions": {
				"OMI_personality": {
					"agent": "cat",
					"personality": "#agent has a cheerful personality.",
					"defaultMessage": "nya nya!"
				}
			}
		}
	]
}

In the example above, the "OMI_personality" extension is applied to a node named "cat". The agent property is used to specify the type of agent associated with the node, in this case, it's a cat. The personality property describes the agent's personality, and the defaultMessage property is the message that the agent will send as a default.

Properties

The defaultMessage parameter is optional. The agent and personality options are required and provide context about the avatars name and default personality description.

	Type	Description
agent	string	The name of the agent or NPC.
personality	string	A default description of the personality of the agent allowing clients to inject that context into language model logic.
defaultMessage	string	A default message for this agent to initialize with.

JSON Schema

The OMI_personality extension is defined by the following JSON schema:

{
	"$schema": "http://json-schema.org/draft-07/schema#",
	"title": "OMI_personality",
	"description": "An extension for the glTF format that defines a personality for a node and an endpoint where additional information can be queried.",
	"type": "object",
	"properties": {
		"agent": {
			"type": "string",
			"description": "The name of the agent associated with the node.",
			"maxLength": 128
		},
		"personality": {
			"type": "string",
			"description": "A description of the agent's personality."
		},
		"defaultMessage": {
			"type": "string",
			"description": "The default message that the agent will send on initialization."
		}
	},
	"required": ["agent", "personality"]
}

In the example above, the "OMI_personality" extension is applied to a node named "cat". The agent property is used to specify the type of agent associated with the node, in this case, it's a cat. The personality property describes the agent's personality, and the defaultMessage property is the message that the agent will send as a default.

[mrmetaverse]

I am super interested in this proposal, and hope to see this advance. Please let me know if I can be of assistance.

[robertlong]

From our meeting today we discussed using this extension as structured metadata to interface with various chat bots. For example in Matrix / Discord the agent field could be the user id of the bot to send messages to. The personality and defaultMessage field would be used by the bot to determine its behavior.

We still should discuss how you might attach actions such as animation/audio queues to a personality.

[mrmetaverse]

Looks great to me

[lyuma]

The idea is that the personality could be the input to a text transformer large language model such as GPT.

We're in a space which is rapidly evolving, and I would also suggest that it would be helpful to consider a path forward for concepts such as soft-prompts which can define some of the environment or setting or story, or embeddings which can help refine the character's personality.

[antpb]

In the last few meetings it was discussed that there was a missing element of this proposal that would show how one might actually use this spec. I've added some implementation details for one example and aim to add more as this nears a possible stage 2.

[mrmetaverse]

Thank you for writing this up. Looks good, reads well.

[mrmetaverse]

@antpb all reviews have passed.

[avaer]

One of the main ways that users export and maintain their chatbots in practice is through a large corpus of example dialogue. E.g. this is how characters are exported from character.ai

This tends to work very well with modern NLP generators even if it's the only field. Perhaps there should be "quotes" field to handle this common case?

I am also wondering if quotes can be folded into one of the above fields instead -- the only problem is that a large quote block doesn't necessarily format naturally into JSON if it's just one block with a bunch of \n's.

Thoughts?

[lalalune]

+1 for this.

We have this integrated into Magick and can definitely push this along with some adjacent friendly web projects.

Obviously this has huge implications for those of use who are making context-aware experiences with LLMs and such.

[robertlong]