Skip to content

Commit

Permalink
feat: add forceSchemaConstrainedTokens to KurtSamplingOptions
Browse files Browse the repository at this point in the history
This maps to the new-ish `strict: true` feature of OpenAI which
enables constrained token sampling, but has certain caveats that
make it undesirable to turn on by default.

See issue #61 for more info.
  • Loading branch information
jemc committed Dec 6, 2024
1 parent 65a209d commit a0c8970
Show file tree
Hide file tree
Showing 2 changed files with 39 additions and 0 deletions.
19 changes: 19 additions & 0 deletions packages/kurt/src/Kurt.ts
Original file line number Diff line number Diff line change
Expand Up @@ -353,6 +353,25 @@ export const KurtSamplingOptionsDefault = {
* without any "top tokens" filtering being applied before sampling.
*/
topP: 0.95,

/**
* When true, structured data generation and/or tool calls will always use
* schema-aware constrained token sampling, rather than conditionally
* doing so, or falling back to relying only on the LLM's training for
* responding with structured data schemas.
*
* This gives a hard guarantee that the output will match the schema,
* but for some underlying LLM providers this may introduce more constraints
* on the kinds of JSON Schemas that are supported, which explains why
* it might not be the default behavior for a particular adapter.
*
* If this feature is unavailable for a particular adapter, the adapter
* should throw a `KurtCapabilityError` when this option is set to true.
*
* If this feature is available without downsides, the adapter should silently
* enable this feature regardless of the option being set to true or false.
*/
forceSchemaConstrainedTokens: false,
}

export interface KurtGenerateNaturalLanguageOptions {
Expand Down
20 changes: 20 additions & 0 deletions packages/kurt/src/KurtError.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,26 @@ export abstract class KurtError extends Error {
abstract readonly adapter: KurtAdapter
}

/**
* Base class for all Kurt errors thrown for rejected requests.
*/
export abstract class KurtRequestError extends KurtError {}

/**
* Thrown by a Kurt adapter when the request tried to use a capability or
* feature that isn't supported by this particular adapter (but might be
* supported by other adapters, or by the same adapter class if instantiated
* with different configuration, such as selecting a different LLM snapshot).
*/
export class KurtCapabilityError extends KurtError {
constructor(
readonly adapter: KurtAdapter,
readonly missingCapability: string
) {
super(`This adapter doesn't support: ${missingCapability}`)
}
}

/**
* Base class for all Kurt errors thrown when handling a result stream.
*/
Expand Down

0 comments on commit a0c8970

Please sign in to comment.