Pruning Rules
Decrease input token counts by pruning out chunks of static text.
Some prompts have large chunks of unchanging text, like system messages which don’t change from one request to the next. By removing this static text and fine-tuning a model on the compacted data, we can reduce the size of incoming requests and save you money on inference.
Add pruning rules to your dataset in the Settings tab, as shown below and in our demo dataset.
To see the effect your pruning rules had on an individual training entry’s input messages, open the Dataset Entry drawer:
By default, fine-tuned models inherit pruning rules applied to the dataset on which they were trained (see demo model). These rules will automatically prune matching text from any incoming requests sent to that model. New pruning rules will not be associated with previously trained models, so you don’t need to worry about backwards compatibility when adding new rules to your dataset. Before training a new model, you can choose to disable any inherited pruning rules.
Warning: can affect quality!
We’ve found that while pruning rules always decrease latency and costs, they can also negatively affect response quality, especially with smaller datasets. We recommend enabling pruning rules on datasets with 10K+ training examples, as smaller datasets may not provide enough guidance for the model to fully learn the task.