LlamaLlmInferenceRequest

class oci.generative_ai_inference.models.LlamaLlmInferenceRequest(**kwargs)

Bases: oci.generative_ai_inference.models.llm_inference_request.LlmInferenceRequest

Details for the text generation request for Llama models.

Attributes

RUNTIME_TYPE_COHERE str(object=’’) -> str
RUNTIME_TYPE_LLAMA str(object=’’) -> str
frequency_penalty Gets the frequency_penalty of this LlamaLlmInferenceRequest.
is_echo Gets the is_echo of this LlamaLlmInferenceRequest.
is_stream Gets the is_stream of this LlamaLlmInferenceRequest.
log_probs Gets the log_probs of this LlamaLlmInferenceRequest.
max_tokens Gets the max_tokens of this LlamaLlmInferenceRequest.
num_generations Gets the num_generations of this LlamaLlmInferenceRequest.
presence_penalty Gets the presence_penalty of this LlamaLlmInferenceRequest.
prompt Gets the prompt of this LlamaLlmInferenceRequest.
runtime_type [Required] Gets the runtime_type of this LlmInferenceRequest.
stop Gets the stop of this LlamaLlmInferenceRequest.
temperature Gets the temperature of this LlamaLlmInferenceRequest.
top_k Gets the top_k of this LlamaLlmInferenceRequest.
top_p Gets the top_p of this LlamaLlmInferenceRequest.

Methods

__init__(**kwargs) Initializes a new LlamaLlmInferenceRequest object with values from keyword arguments.
get_subtype(object_dictionary) Given the hash representation of a subtype of this class, use the info in the hash to return the class of the subtype.
RUNTIME_TYPE_COHERE = 'COHERE'
RUNTIME_TYPE_LLAMA = 'LLAMA'
__init__(**kwargs)

Initializes a new LlamaLlmInferenceRequest object with values from keyword arguments. The default value of the runtime_type attribute of this class is LLAMA and it should not be changed. The following keyword arguments are supported (corresponding to the getters/setters of this class):

Parameters:
  • runtime_type (str) – The value to assign to the runtime_type property of this LlamaLlmInferenceRequest. Allowed values for this property are: “COHERE”, “LLAMA”
  • prompt (str) – The value to assign to the prompt property of this LlamaLlmInferenceRequest.
  • is_stream (bool) – The value to assign to the is_stream property of this LlamaLlmInferenceRequest.
  • num_generations (int) – The value to assign to the num_generations property of this LlamaLlmInferenceRequest.
  • is_echo (bool) – The value to assign to the is_echo property of this LlamaLlmInferenceRequest.
  • top_k (int) – The value to assign to the top_k property of this LlamaLlmInferenceRequest.
  • top_p (float) – The value to assign to the top_p property of this LlamaLlmInferenceRequest.
  • temperature (float) – The value to assign to the temperature property of this LlamaLlmInferenceRequest.
  • frequency_penalty (float) – The value to assign to the frequency_penalty property of this LlamaLlmInferenceRequest.
  • presence_penalty (float) – The value to assign to the presence_penalty property of this LlamaLlmInferenceRequest.
  • stop (list[str]) – The value to assign to the stop property of this LlamaLlmInferenceRequest.
  • log_probs (int) – The value to assign to the log_probs property of this LlamaLlmInferenceRequest.
  • max_tokens (int) – The value to assign to the max_tokens property of this LlamaLlmInferenceRequest.
frequency_penalty

Gets the frequency_penalty of this LlamaLlmInferenceRequest. To reduce repetitiveness of generated tokens, this number penalizes new tokens based on their frequency in the generated text so far. Values > 0 encourage the model to use new tokens and values < 0 encourage the model to repeat tokens. Set to 0 to disable.

Returns:The frequency_penalty of this LlamaLlmInferenceRequest.
Return type:float
static get_subtype(object_dictionary)

Given the hash representation of a subtype of this class, use the info in the hash to return the class of the subtype.

is_echo

Gets the is_echo of this LlamaLlmInferenceRequest. Whether or not to return the user prompt in the response. Applies only to non-stream results.

Returns:The is_echo of this LlamaLlmInferenceRequest.
Return type:bool
is_stream

Gets the is_stream of this LlamaLlmInferenceRequest. Whether to stream back partial progress. If set, tokens are sent as data-only server-sent events as they become available.

Returns:The is_stream of this LlamaLlmInferenceRequest.
Return type:bool
log_probs

Gets the log_probs of this LlamaLlmInferenceRequest. Includes the logarithmic probabilities for the most likely output tokens and the chosen tokens.

For example, if the log probability is 5, the API returns a list of the 5 most likely tokens. The API returns the log probability of the sampled token, so there might be up to logprobs+1 elements in the response.

Returns:The log_probs of this LlamaLlmInferenceRequest.
Return type:int
max_tokens

Gets the max_tokens of this LlamaLlmInferenceRequest. The maximum number of tokens that can be generated per output sequence. The token count of the prompt plus maxTokens cannot exceed the model’s context length.

Returns:The max_tokens of this LlamaLlmInferenceRequest.
Return type:int
num_generations

Gets the num_generations of this LlamaLlmInferenceRequest. The number of of generated texts that will be returned.

Returns:The num_generations of this LlamaLlmInferenceRequest.
Return type:int
presence_penalty

Gets the presence_penalty of this LlamaLlmInferenceRequest. To reduce repetitiveness of generated tokens, this number penalizes new tokens based on whether they’ve appeared in the generated text so far. Values > 0 encourage the model to use new tokens and values < 0 encourage the model to repeat tokens.

Similar to frequency penalty, a penalty is applied to previously present tokens, except that this penalty is applied equally to all tokens that have already appeared, regardless of how many times they’ve appeared. Set to 0 to disable.

Returns:The presence_penalty of this LlamaLlmInferenceRequest.
Return type:float
prompt

Gets the prompt of this LlamaLlmInferenceRequest. Represents the prompt to be completed. The trailing white spaces are trimmed before completion.

Returns:The prompt of this LlamaLlmInferenceRequest.
Return type:str
runtime_type

[Required] Gets the runtime_type of this LlmInferenceRequest. The runtime of the provided model.

Allowed values for this property are: “COHERE”, “LLAMA”

Returns:The runtime_type of this LlmInferenceRequest.
Return type:str
stop

Gets the stop of this LlamaLlmInferenceRequest. List of strings that stop the generation if they are generated for the response text. The returned output will not contain the stop strings.

Returns:The stop of this LlamaLlmInferenceRequest.
Return type:list[str]
temperature

Gets the temperature of this LlamaLlmInferenceRequest. A number that sets the randomness of the generated output. A lower temperature means a less random generations.

Use lower numbers for tasks with a correct answer such as question answering or summarizing. High temperatures can generate hallucinations or factually incorrect information. Start with temperatures lower than 1.0 and increase the temperature for more creative outputs, as you regenerate the prompts to refine the outputs.

Returns:The temperature of this LlamaLlmInferenceRequest.
Return type:float
top_k

Gets the top_k of this LlamaLlmInferenceRequest. An integer that sets up the model to use only the top k most likely tokens in the generated output. A higher k introduces more randomness into the output making the output text sound more natural. Default value is -1 which means to consider all tokens. Setting to 0 disables this method and considers all tokens.

If also using top p, then the model considers only the top tokens whose probabilities add up to p percent and ignores the rest of the k tokens. For example, if k is 20, but the probabilities of the top 10 add up to .75, then only the top 10 tokens are chosen.

Returns:The top_k of this LlamaLlmInferenceRequest.
Return type:int
top_p

Gets the top_p of this LlamaLlmInferenceRequest. If set to a probability 0.0 < p < 1.0, it ensures that only the most likely tokens, with total probability mass of p, are considered for generation at each step.

To eliminate tokens with low likelihood, assign p a minimum percentage for the next token’s likelihood. For example, when p is set to 0.75, the model eliminates the bottom 25 percent for the next token. Set to 1 to consider all tokens and set to 0 to disable. If both k and p are enabled, p acts after k.

Returns:The top_p of this LlamaLlmInferenceRequest.
Return type:float