Orchestrating SageMaker AI endpoint auto scaling with AWS CDK

Deploying machine learning models to production often requires dynamic resource allocation to handle fluctuating traffic. When configuring production variants for a SageMaker AI endpoint, implementing Application Auto Scaling is a best practice to ensure high availability and cost efficiency. However, developers frequently encounter frustrating CloudFormation deployment failures when trying to update an endpoint that already has active scaling policies registered. This occurs because underlying CloudFormation resource constraints prevent modifying variants while external scaling targets hold locks on the endpoint capacity.

In this deep dive, we will explore why these deployment deadlocks happen and how to construct a robust, reusable solution using AWS CDK and CloudFormation custom resources. By decoupling the lifecycle of the scaling registration from the endpoint itself, you can achieve seamless, zero-downtime updates in a single cdk deploy execution.

The Root Cause of SageMaker Auto Scaling Deployment Deadlocks

When you define an CfnEndpoint alongside an ApplicationAutoScaling target in AWS CDK, CloudFormation manages them as independent resources. During an update—such as changing the instance type for DeepSeek-V3 or Claude 3.5 Sonnet hosting configurations—CloudFormation attempts to modify the SageMaker endpoint configuration concurrently with or immediately before updating the scaling targets.

Because the Application Auto Scaling service holds an active reference and execution lock on the variant's desired capacity, SageMaker rejects updates to the endpoint variant weights or instance counts. The CloudFormation stack rolls back, leaving your deployment in a failed state and requiring manual intervention in the AWS Management Console to deregister the scalable target before trying again.

To solve this architecturally, we need a mechanism that intercepts the CloudFormation lifecycle:

Pre-Update: Automatically deregister the SageMaker endpoint from Application Auto Scaling.
Update: Allow CloudFormation to successfully update the SageMaker endpoint and endpoint configuration.
Post-Update: Re-register the scaling target with the desired minimum and maximum capacity bounds.

While you could manage this manually, doing so defeats the purpose of Infrastructure as Code (IaC). Instead, we can encapsulate this logic inside a custom AWS CDK construct powered by AWS Lambda and CloudFormation custom resources.

Designing the Custom Construct Architecture

Our goal is to build a high-level construct called AutoScalingSageMakerEndpoint. This construct will wrap the standard CfnEndpoint, CfnEndpointConfig, and auto-scaling policies, while inserting a custom resource provider that listens to CloudFormation lifecycle events.

Here is how the components interact during a deployment lifecycle:

Lifecycle Phase	CloudFormation Action	Custom Resource Handler Action
Create	Provisions endpoint, then registers scaling	Calls Application Auto Scaling API to register the target.
Update	Detects config change	Deregisters target → Updates endpoint → Re-registers target.
Delete	Deletes scaling, then endpoint	Cleans up scaling policies and targets cleanly.

By orchestrating these steps sequentially within a Lambda-backed custom resource, we eliminate race conditions completely.

Implementing the CDK Construct in TypeScript

Let us write the complete, production-ready TypeScript code for this construct. First, ensure you have your CDK project initialized and your AWS SDK v3 dependencies configured.

import * as cdk from 'aws-cdk-lib';
import * as sagemaker from 'aws-cdk-lib/aws-sagemaker';
import * as appscaling from 'aws-cdk-lib/aws-applicationautoscaling';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as iam from 'aws-cdk-lib/aws-iam';
import * as cr from 'aws-cdk-lib/custom-resources';
import { Construct } from 'constructs';

export interface AutoScalingSageMakerEndpointProps {
  readonly endpointName: string;
  readonly endpointConfigName: string;
  readonly variantName: string;
  readonly minCapacity: number;
  readonly maxCapacity: number;
  readonly targetValue: number;
  readonly scaleInCooldown?: number;
  readonly scaleOutCooldown?: number;
}

export class AutoScalingSageMakerEndpoint extends Construct {
  public readonly endpoint: sagemaker.CfnEndpoint;

  constructor(scope: Construct, id: string, props: AutoScalingSageMakerEndpointProps) {
    super(scope, id);

    // 1. Define standard SageMaker endpoint configuration and endpoint
    const endpointConfig = new sagemaker.CfnEndpointConfig(this, 'EndpointConfig', {
      endpointConfigName: props.endpointConfigName,
      productionVariants: [
        {
          initialInstanceCount: props.minCapacity,
          instanceType: 'ml.m5.xlarge',
          variantName: props.variantName,
          modelName: 'your-model-name',
        },
      ],
    });

    this.endpoint = new sagemaker.CfnEndpoint(this, 'Endpoint', {
      endpointName: props.endpointName,
      endpointConfigName: endpointConfig.attrEndpointConfigName,
    });

    this.endpoint.addDependency(endpointConfig);

    // 2. Lambda handler for managing scaling registration lifecycle
    const scalingManagerLambda = new lambda.SingletonFunction(this, 'ScalingManagerLambda', {
      uuid: 'a1b2c3d4-e5f6-7890-abcd-ef0123456789',
      code: lambda.Code.fromInline(`
        const { ApplicationAutoScalingClient, RegisterScalableTargetCommand, DeregisterScalableTargetCommand, PutScalingPolicyCommand } = require('@aws-sdk/client-application-auto-scaling');
        const client = new ApplicationAutoScalingClient();

        exports.handler = async (event) => {
          const props = event.ResourceProperties;
          const resourceId = \`endpoint/\${props.EndpointName}/variant/\${props.VariantName}\`;
          
          if (event.RequestType === 'Delete') {
            await client.send(new DeregisterScalableTargetCommand({
              ServiceNamespace: 'sagemaker',
              ResourceId: resourceId,
              ScalableDimension: 'sagemaker:variant:DesiredInstanceCount',
            }));
            return { PhysicalResourceId: resourceId };
          }

          if (event.RequestType === 'Update' || event.RequestType === 'Create') {
            // Pre-emptively deregister to avoid locking conflicts during updates
            try {
              await client.send(new DeregisterScalableTargetCommand({
                ServiceNamespace: 'sagemaker',
                ResourceId: resourceId,
                ScalableDimension: 'sagemaker:variant:DesiredInstanceCount',
              }));
            } catch (e) {
              console.log('Target was not registered yet, proceeding.');
            }

            // Register scalable target
            await client.send(new RegisterScalableTargetCommand({
              ServiceNamespace: 'sagemaker',
              ResourceId: resourceId,
              ScalableDimension: 'sagemaker:variant:DesiredInstanceCount',
              MinCapacity: parseInt(props.MinCapacity),
              MaxCapacity: parseInt(props.MaxCapacity),
            }));

            // Put scaling policy
            await client.send(new PutScalingPolicyCommand({
              PolicyName: \`sagemaker-target-tracking-\${props.VariantName}\`,
              ServiceNamespace: 'sagemaker',
              ResourceId: resourceId,
              ScalableDimension: 'sagemaker:variant:DesiredInstanceCount',
              PolicyType: 'TargetTrackingScaling',
              TargetTrackingScalingPolicyConfiguration: {
                TargetValue: parseFloat(props.TargetValue),
                PredefinedMetricSpecification: {
                  PredefinedMetricType: 'SageMakerVariantInvocationsPerInstance',
                },
              },
            }));
          }

          return { PhysicalResourceId: resourceId };
        };
      `),
      handler: 'index.handler',
      runtime: lambda.Runtime.NODEJS_18_X,
      timeout: cdk.Duration.seconds(30),
    });

    scalingManagerLambda.addToRolePolicy(new iam.PolicyStatement({
      actions: [
        'application-autoscaling:RegisterScalableTarget',
        'application-autoscaling:DeregisterScalableTarget',
        'application-autoscaling:PutScalingPolicy',
        'application-autoscaling:DeleteScalingPolicy',
      ],
      resources: ['*'],
    }));

    // 3. Custom Resource Provider binding
    const provider = new cr.Provider(this, 'ScalingProvider', {
      onEventHandler: scalingManagerLambda,
    });

    const customResource = new cdk.CustomResource(this, 'ScalingCustomResource', {
      serviceToken: provider.serviceToken,
      properties: {
        EndpointName: this.endpoint.endpointName,
        VariantName: props.variantName,
        MinCapacity: props.minCapacity,
        MaxCapacity: props.maxCapacity,
        TargetValue: props.targetValue,
        // Force re-execution on config changes
        EndpointConfigName: endpointConfig.attrEndpointConfigName,
      },
    });

    customResource.node.addDependency(this.endpoint);
  }
}

Integrating with AI Workflows and External APIs

When operating large-scale AI applications, managing your infrastructure via CDK is only half the battle. You also need to manage high-throughput connections to LLM inference endpoints. Many engineering teams leverage aggregator platforms like n1n.ai to route traffic seamlessly between custom SageMaker deployments and hosted foundation models.

By combining automated infrastructure constructs in CDK with robust API routing layers through n1n.ai, developers can achieve a resilient architecture that scales dynamically both at the infrastructure layer and the application gateway layer. For instance, if your SageMaker endpoint experiences a scaling delay, n1n.ai can intelligently fallback or load-balance requests across alternative endpoints, ensuring zero downtime for your end users.

Testing and Verifying the Deployment

To verify that your newly created construct handles updates correctly, execute the following steps in your terminal:

# 1. Synthesize the CDK stack to check for template validity
cdk synth

# 2. Deploy the initial stack containing your SageMaker endpoint and auto scaling rules
cdk deploy --require-approval never

# 3. Modify the endpoint configuration (e.g., change instance count or model parameters) and re-deploy
cdk deploy --require-approval never

During the second deployment, CloudFormation will trigger the custom resource's update handler. The Lambda function will execute the DeregisterScalableTargetCommand, allowing CloudFormation to update the SageMaker endpoint successfully without throwing resource lock exceptions. Once updated, the custom resource re-establishes the scaling target and target tracking policy automatically.

Conclusion

Managing SageMaker AI endpoints with Application Auto Scaling inside AWS CDK no longer needs to be a source of deployment friction. By encapsulating the registration lifecycle inside a CloudFormation custom resource, you turn fragile infrastructure code into a reliable, reusable construct.

To further optimize your AI infrastructure pipelines and seamlessly integrate multiple LLM providers, explore the advanced routing features available at n1n.ai.

Get a free API key at n1n.ai

Source: https://aws.amazon.com/blogs/machine-learning/orchestrating-sagemaker-ai-endpoint-auto-scaling-with-aws-cdk/