Criteria

Judge Criteria

Welcome

Introduction
Overview
Base Models

Getting Started

Quick Start
Installing the SDK

Features

Request Logs
Datasets
Fine Tuning
Direct Preference Optimization (DPO)
Evaluations
Criteria
Chat Completions
Caching
Updating Metadata Tags
Pruning Rules
Fallback
Mixture of Agents

API Reference

Logs
Chat Completions
Datasets
Models
Criteria
- POST
  Judge Criteria

Pricing

Pricing Overview

Criteria

Judge Criteria

Get a judgement of a completion against the specified criterion

POST

criteria

judge

Authorization

string

Bearer

Authorization

Required

string

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

object

criterion_id

string

criterion_id

Required

string

The ID of the criterion to judge.

input

object

input

object

messages

array

messages

array

All messages sent to the model when generating the output.

tool_choice

Option 1 · enum<string>

tool_choice

enum<string>

The tool choice to use when generating the output, if any.

tools

array

tools

array

The tools available to the model when generating the output, if any.

output

object

output

Required

object

The completion message of the model.

content

string

content

string

refusal

string

refusal

string

role

enum<string>

role

Required

enum<string>

function_call

object

function_call

object

name

string

name

string

arguments

string

arguments

string

tool_calls

array

tool_calls

array

Authorizations

Authorization

string

headerrequired

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Body

application/json

criterion_id

string

required

The ID of the criterion to judge.

output

object

required

The completion message of the model.

output.role

enum<string>

required

Available options:

assistant

output.content

string | null

output.refusal

string | null

output.function_call

object | null

output.function_call.name

string

default:

output.function_call.arguments

string

default:

output.tool_calls

object[] | null

output.tool_calls.id

string

required

output.tool_calls.function

object

required

output.tool_calls.function.name

string

required

output.tool_calls.function.arguments

string

required

output.tool_calls.type

enum<string>

required

Available options:

function

input

object

input.messages

object[]

All messages sent to the model when generating the output.

input.messages.role

enum<string>

required

Available options:

system

input.messages.content

default:

input.messages.name

string

input.tool_choice

The tool choice to use when generating the output, if any.

Available options:

none

input.tools

object[]

The tools available to the model when generating the output, if any.

input.tools.function

object

required

input.tools.function.name

string

required

input.tools.function.parameters

object

input.tools.function.parameters.{key}

any

input.tools.function.description

string

input.tools.function.strict

boolean | null

input.tools.type

enum<string>

required

Available options:

function

Response

200 - application/json

score

number

required

A score of 0 means the output failed this completion, and a score of 1 means it passed. A criteria may also return a decimal scores between 0 and 1, indicating the model's confidence or 'likelihood' that the criteria passed.

explanation

string

An explanation of the score including the model's reasoning, if applicable.

usage

object

usage.total_tokens

number

required

The total number of tokens used to generate the criterion judgement. Only returned for OpenPipe-trained reward models currently.

Delete Model Pricing Overview

twitter linkedin