Getting Started

External applications provide a bridge between SGP and AI applications that exist off-platform. To get started, navigate to the "Applications" page on the SGP dashboard, click Create a new Application, and select External AI as the application template.

Integration

You can use the new SGP Python client to bridge your external AI to SGP. Doing this will allow you to run test cases from an Evaluation Dataset on your SGP account on your external AI and, subsequently, will enable you to run evaluations and generate a Scale Report.

Define an interface

In order to integrate your external AI with SGP, you'll have to create an interface to run your application locally. This interface is a Python callable that takes a single prompt and returns your external AI application's formatted output.

from scale_gp.lib.external_applications import ExternalApplicationOutputCompletion


def my_app_interface(prompt: str):
	
    output = ... # Call your application with the `prompt` here
  
    return ExternalApplicationOutputCompletion(
        generation_output=output,
    )

For example, an integration with a LangChain model might look like this:

from langchain_core.messages import HumanMessage
from .my_langchain_model import model


def my_langchain_app_interface(prompt: str):
  
    response = model.invoke([HumanMessage(content=prompt)])
  
    return ExternalApplicationOutputCompletion(
        generation_output=response.content,
    )

Custom Metrics

Optionally, you can include custom client-side metrics to display with your outputs. These values will surface on the details page for any evaluation run you create against your uploaded outputs.

from scale_gp.lib.external_applications import ExternalApplicationOutputCompletion
from scale_gp.types.evaluation_datasets.test_case import TestCase


def my_app_interface(
    prompt: str,
    test_case: TestCase, # Include a second parameter to access the full test case
):
	
    output = ...
  
    return ExternalApplicationOutputCompletion(
        generation_output=output,
        metrics={
            "completion_tokens": get_token_count(output),
            "meteor": compute_meteor_score(output, test_case.test_case_data.expected_output),
        }
    )

Advanced: Other Output Types

If your application uses RAG (or any other methodology) to generate additional context to inject into an LLM prompt, you can use one of our additional output types to include the context with the text response.

Context - String

You can include additional context as a single piece of text using ExternalApplicationOutputContextString

from scale_gp.lib.external_applications import ExternalApplicationOutputContextString
from .my_rag_pipeline import pipeline


def my_rag_app_interface(prompt: str):
  
    response = pipeline.invoke(prompt)
  
    return ExternalApplicationOutputContextString(
        generation_output=response.message,
        info=response.context,
    )

Context - Chunks

If your application retrieves a list of documents to use a additional context, you can use ExternalApplicationOutputContextChunks to include each piece of text as a separate Chunk

from scale_gp.lib.external_applications import Chunk, ExternalApplicationOutputContextChunks
from .my_rag_pipeline import pipeline


def my_rag_app_interface(prompt: str):
  
    response = pipeline.invoke(prompt)
  
    return ExternalApplicationOutputContextString(
        generation_output=response.message,
        chunks=[
            Chunk(text=chunk.context, metadata={...})
            for chunk in response.context
        ]
    )

Generate Outputs

Now that you've defined an interface for your external AI application, you can generate outputs for an evaluation dataset to use for a custom evaluation or to generate Scale Report. First, initialize an SGPClient with your API key, and an ExternalApplication with your external application variant ID and the interface you defined.

from scale_gp import SGPClient
from scale_gp.lib.external_applications import ExternalApplication

client = SGPClient(api_key=...)

external_application = ExternalApplication(client).initialize(
    application_variant_id=...,
    application=my_app_interface,
)

Then, find or create an evaluation dataset on the SGP dashboard, and copy the ID and latest version number to generate outputs for its test cases with your external application.

external_application.generate_outputs(
    evaluation_dataset_id=...,
    evaluation_dataset_version=...,
)

Uploading Precomputed Outputs

Optionally, you can bypass defining an ExternalApplication interface if you've already generated outputs for your evaluation dataset outside of the SGP SDK by creating a mapping of test case IDs to outputs, and using the batch upload function. This is what the ExternalApplication library class uses under the hood.

shared_params = {
    "account_id": ...,
    "application_variant_id": ...,
    "evaluation_dataset_version_num": ...,
}

my_outputs: List[Tuple[str, str, List[str]]] = [
    ("<SOME_TEST_CASE_ID_1>", "My example output 1"),
    ("<SOME_TEST_CASE_ID_2>", "My example output 2"),
    ...
] # Create a list of tuples here with the test case ID and output

client.application_test_case_outputs.batch(
    items=[
        {
            **shared_params,
            "test_case_id": test_case_id,
            "output": {
                "generation_output": output,
            },
        }
        for test_case_id, output in my_outputs
    ]
)

You can also include context chunks used by your application for a given output:

shared_params = {
    "account_id": ...,
    "application_variant_id": ...,
    "evaluation_dataset_version_num": ...,
}

my_outputs: List[Tuple[str, str, List[str]]] = [
    ("<SOME_TEST_CASE_ID>", "My example output", ["Text from chunk 1", "Text from chunk 2"])
] # Create a list of tuples here with the test case ID, output, and context chunks

client.application_test_case_outputs.batch(
    items=[
        {
            **shared_params,
            "test_case_id": test_case_id,
            "output": {
                "generation_output": output,
                "generation_extra_info": {
                    "schema_type": "CHUNKS",
                    "chunks": [
                        {
                            "text": text,
                            "metadata": {
                                ... # Include anything (or nothing)
                            },
                        }
                        for text in chunks
                    ],
                },
            },
        }
        for test_case_id, output, chunks in my_outputs
    ]
)

Upload interactions to SGP

If you wish to log interactions from your external application to utilize the SGP features like evaluations and application monitoring, you can do so by using the interactions.create method. If you have additional metadata emitted by the internal building blocks of your application (like a reranking or a completion component), you can attach them to the interaction as trace spans:

from scale_gp import SGPClient
from scale_gp.types.interaction_create_params import Input, Output, TraceSpan

client = SGPClient(account_id=..., api_key=...)

client.interactions.create(
    application_variant_id=...,
    input=Input(query="What is the speed of light in a vacuum?"),
    output=Output(response="The speed of light in a vacuum is approximately 299,792 (km/s)."),
    start_timestamp="2024-08-22T14:22:27.502Z",
    duration_ms=3200,
    operation_metadata={"user_id": "user-123"},
    trace_spans=[
        TraceSpan(
            node_id="RETRIEVAL-NODE-1",
            operation_type="RETRIEVAL", # Supported values are: COMPLETION, RERANKING, RETRIEVAL, CUSTOM
            start_timestamp="2024-08-22T14:22:28.502Z",
            duration_ms=2200
        ),
        TraceSpan(
            node_id="COMPLETION-NODE-1",
            operation_type="COMPLETION",
            start_timestamp="2024-08-22T14:22:30.502Z",
            duration_ms=1000,
            operation_metadata={"tokens_used": 420}
        )
    ],
)

The registered interactions will then show up on the Application Monitoring page that you can reach from the Applications page by clicking on your External Application instance, and selecting Monitoring Dashboard. From this page, you chan check out the details of your interactions, like the request latency, or the error rate. If you click on an interaction, you can also check out additional details about it, as well as the information emitted by the trace spans.

Creating threads for interactions

If your application supports multi-turn interactions, you might want to group certain interactions together, that are part of the same conversation. You can do this by assigning a UUID for the interaction, and then using the same UUID for any follow-up interaction.

from scale_gp import SGPClient
from scale_gp.types.interaction_create_params import Input, Output
import uuid

client = SGPClient(account_id=..., api_key=...)
  
application_variant_id = ...
thread_id = str(uuid.uuid4())
client.interactions.create(
    application_variant_id=application_variant_id,
    thread_id=thread_id,
    input=Input(query="What is the speed of light in a vacuum?"),
    output=Output(response="The speed of light in a vacuum is approximately 299,792 (km/s)."),
    start_timestamp="2024-08-22T14:22:27.502Z",
    duration_ms=3200,
)

client.interactions.create(
    application_variant_id=application_variant_id,
    thread_id=thread_id,
    input=Input(query="What is the capital of France?"),
    output=Output(response="The capital of France is Paris."),
    start_timestamp="2024-08-22T14:27:27.502Z",
    duration_ms=3200,
)

This will group them together in the interaction details page in the following way: