Fallback options
Safeguard your application against potential failures, timeouts, or instabilities that may occur when using experimental or newly released models.
Fallback is a feature that ensures a seamless experience and guarantees 100% uptime when working with new or unstable models.
When fallback is enabled, any failed API calls will be automatically retried using OpenAI or any OpenAI-compatible client.
Fallback to OpenAI
To enable fallback to OpenAI, you can simply pass the fallback option to the openpipe object with the model property set to the OpenAI model you want to fall back to.
from openpipe import OpenAI
client = OpenAI()
completion = client.chat.completions.create(
model="openpipe:my-ft-model",
messages=[{"role": "system", "content": "count to 10"}],
openpipe={
"fallback": {
"model": "gpt-4-turbo"
}
},
)
Timeout Fallback
If a request takes too long to execute, you can set a timeout for the fallback. In the example below, the request will fall back to OpenAI after 10 seconds.
from openpipe import OpenAI
client = OpenAI(timeout=10) # initial OpenPipe call timeout in seconds
completion = client.chat.completions.create(
model="openpipe:my-ft-model",
messages=[{"role": "system", "content": "count to 10"}],
openpipe={
"fallback": {
"model": "gpt-4-turbo",
# optional fallback timeout. Defaults to the timeout specified in the client, or OpenAI default timeout if not set.
"timeout": 20 # seconds
}
},
)
Fallback to Custom OpenAI Compatible Client
If you want to use another OpenAI-compatible fallback client, you can pass a fallback_client to the openpipe object.
from openpipe import OpenAI
client = OpenAI(
openpipe={
"fallback_client": OpenAICompatibleClient(api_key="client api key")
}
);
completion = client.chat.completions.create(
model="openpipe:my-ft-model",
messages=[{"role": "system", "content": "count to 10"}],
openpipe={
"fallback": { "model": "gpt-4-turbo" }
},
)