Create Model Template

post https://api.egp.scale.com/v4/model-templates

Description

Creates a model template.

Details

Model templates serve 2 purposes. First, they provide common scaffolding that is static across multiple models. Second, they expose several variables that can be injected at model creation time to customize the model.

For example, a model template can define a docker image that contains code to run a HuggingFace or SentenceTransformers model. This docker image code also accepts environment variables that can be set to swap out the model weights or model name.

Two of the most important fields required to create a model template are the model_creation_parameters_schema and model_request_parameters_schema fields.

The model_creation_parameters_schema field defines the schema for parameters that can be injected at model creation time. For example, if the schema contains a model_weights_uri field, which expects a string, when a model is created from this template, the user can provide a URI to a model weights file that can be used to swap out the model weights used by the model.

The model_request_parameters_schema field defines the schema for parameters that can be injected by an end user at model execution time. For example, if the schema contains a model_request_parameters field, which expects a string, when a model is executed, the user can provide a string that will be passed to the model for inference.

Body Params

name

string

required

endpoint_type

string

required

An enum representing the different types of model endpoint types supported.

Attributes:
SYNC: Denotes that the model endpoint type is sync.
ASYNC: Denotes that the model endpoint type is async.
STREAMING: Denotes that the model endpoint type is streaming.
BATCH: Denotes that the model endpoint type is batch.

model_type

string

required

An enum representing the different types of models supported.

Attributes:
COMPLETION: Denotes that the model type is completion.
CHAT_COMPLETION: Denotes that the model type is chat completion.
AGENT: Denotes that the model type is agent.
EMBEDDING: Denotes that the model type is embedding.
RERANKING: Denotes that the model type is reranking.
GENERIC: Denotes that the model type is generic.
BUNDLE: "Not to be used directly - type to surface a model bundle in the UI - TODO: Explicitly type a bundle

vendor_configuration

object

required

Configuration for launching a model using the Launch service which is an internal and
self-hosted service developed by Scale that deploys models on Kubernetes.

Attributes:
vendor: The vendor of the model template
bundle_config: The bundle configuration of the model template
endpoint_config: The endpoint configuration of the model template

model_creation_parameters_schema

object

The field names and types of available parameter fields which may be specified during model creation

model_request_parameters_schema

object

The field names and types of available parameter fields which may be specified in a model execution API's model_request_parameters field.

endpoint_protocol

string

The name of the calling convention expected by the Launch model endpoint

account_id

string

required

The ID of the account that owns the given entity.

Headers

x-selected-account-id

string

Responses

Response body

object

name

string

required

endpoint_type

string

required

An enum representing the different types of model endpoint types supported.

SYNC ASYNC STREAMING BATCH

model_type

string

required

An enum representing the different types of models supported.

COMPLETION CHAT_COMPLETION AGENT EMBEDDING RERANKING GENERIC BUNDLE

vendor_configuration

object

required

Configuration for launching a model using the Launch service which is an internal and
self-hosted service developed by Scale that deploys models on Kubernetes.

Attributes:
vendor: The vendor of the model template
bundle_config: The bundle configuration of the model template
endpoint_config: The endpoint configuration of the model template

vendor

const

Defaults to LAUNCH

LAUNCH

bundle_config

object

required

endpoint_config

object

fine_tuning_job_bundle_config

object

model_creation_parameters_schema

object

The field names and types of available parameter fields which may be specified during model creation

parameters

array of objects

required

Parameters*

object

name

string

required

type

required

description

string

required

boolean

required

model_request_parameters_schema

object

The field names and types of available parameter fields which may be specified in a model execution API's model_request_parameters field.

parameters

array of objects

required

Parameters*

object

name

string

required

type

required

description

string

required

boolean

required

endpoint_protocol

string

The name of the calling convention expected by the Launch model endpoint

SGP COHERE VLLM

string

required

The unique identifier of the entity.

created_at

date-time

required

The date and time when the entity was created in ISO format.

account_id

string

required

The ID of the account that owns the given entity.

created_by_user_id

string

required

The user who originally created the entity.

Language

Credentials

Header

Request


xxxxxxxxxx
39
1
curl --request POST \
2
     --url https://api.egp.scale.com/v4/model-templates \
3
     --header 'accept: application/json' \
4
     --header 'content-type: application/json' \
5
     --data '
6
{
7
  "endpoint_type": "SYNC",
8
  "model_type": "COMPLETION",
9
  "vendor_configuration": {
10
    "vendor": "LAUNCH",
11
    "bundle_config": {
12
      "readiness_initial_delay_seconds": 120,
13
      "healthcheck_route": "/readyz",
14
      "predict_route": "/predict",
15
      "streaming_predict_route": "/generate_streaming"
16
    },
17
    "endpoint_config": {
18
      "cpus": 3,
19
      "memory": "8Gi",
20
      "storage": "16Gi",
21
      "gpus": 0,
22
      "min_workers": 0,
23
      "max_workers": 1,
24
      "per_worker": 10,
25
      "endpoint_type": "ASYNC",
26
      "high_priority": false
27
    },
28
    "fine_tuning_job_bundle_config": {
29
      "mount_location": "/workspace/launch_specific/config.json",

Click Try It! to start a request and see the response here! Or choose an example:

application/json

200Successful Response

422Validation Error