AI Feature Architecture: Avoiding Vendor Lock-in and Maintaining Control

AI features are getting added to production systems quickly — often quickly enough that the usual architectural discipline gets skipped. The integration is direct: OpenAI SDK in the service, API calls scattered through business logic, prompts hardcoded in the calling code.

This works for a proof of concept. At production scale, with real operational requirements, it creates a class of problems that are expensive to fix after the fact.

The Vendor Lock-in Problem

Directly using the OpenAI SDK, the Anthropic SDK, or any provider’s client library throughout your codebase creates tight coupling to that provider’s interface. When you want to:

Switch to a different model (price, performance, capability)
Try a different provider
Add a fallback provider for reliability
Run a smaller model locally for cost reduction

…you have to find and change every call site. In a large codebase, this is a significant effort.

The standard solution: a model abstraction layer.

// Abstraction — your application depends on this
public interface LLMClient {
    
    CompletionResponse complete(CompletionRequest request);
    
    StreamingResponse completeStream(CompletionRequest request);
}

// Request/response types you control
public record CompletionRequest(
    String model,
    List<Message> messages,
    double temperature,
    int maxTokens,
    ResponseFormat responseFormat
) {}

public record CompletionResponse(
    String content,
    Usage usage,
    FinishReason finishReason
) {}

Adapters implement the interface for each provider:

@Component
public class OpenAIAdapter implements LLMClient {
    
    private final OpenAIClient client;
    
    @Override
    public CompletionResponse complete(CompletionRequest request) {
        var openAIRequest = ChatCompletionRequest.builder()
            .model(request.model())
            .messages(toOpenAIMessages(request.messages()))
            .temperature(request.temperature())
            .maxTokens(request.maxTokens())
            .build();
        
        var response = client.chat().completions().create(openAIRequest);
        return new CompletionResponse(
            response.choices().get(0).message().content(),
            new Usage(response.usage().promptTokens(), response.usage().completionTokens()),
            toFinishReason(response.choices().get(0).finishReason())
        );
    }
}

Switching providers or adding fallback is now a configuration change, not a codebase change:

@Bean
@ConditionalOnProperty(name = "llm.provider", havingValue = "anthropic")
public LLMClient anthropicClient(AnthropicProperties props) {
    return new AnthropicAdapter(props);
}

@Bean
@ConditionalOnProperty(name = "llm.provider", havingValue = "openai", matchIfMissing = true)
public LLMClient openAIClient(OpenAIProperties props) {
    return new OpenAIAdapter(props);
}

Prompt Management

Hardcoded prompts in application code are a maintenance problem:

Prompts evolve frequently in early stages
Prompt changes need to be tested and deployed
Multiple services may use similar prompts with slight variations

Manage prompts as first-class artifacts:

// Prompts loaded from versioned templates
@Component
public class PromptTemplates {
    
    private final Map<String, String> templates;
    
    public PromptTemplates(PromptTemplateLoader loader) {
        this.templates = loader.loadAll(); // From classpath, database, or config
    }
    
    public String render(String templateName, Map<String, Object> variables) {
        String template = templates.get(templateName);
        if (template == null) throw new PromptNotFoundException(templateName);
        return TemplateEngine.render(template, variables);
    }
}

# prompts/order-summary.txt (version controlled)
You are a customer service assistant for {{company_name}}.
Summarize the following order for the customer. 
Be concise and friendly. Include: order status, estimated delivery, item count.

Order data:
{{order_json}}

When prompts are in version control:

Changes are reviewed like code changes
Different environments can use different template sets
A/B testing between prompt versions is controlled

Token Budget Enforcement

Token costs compound. A service handling 10,000 requests/day at an average of 2,000 tokens each costs significantly more at 100 requests/minute of unexpected traffic or if prompt history accumulates without bounds.

Build token budget enforcement into your abstraction layer:

public class BudgetedLLMClient implements LLMClient {
    
    private final LLMClient delegate;
    private final TokenBudget budget;
    
    @Override
    public CompletionResponse complete(CompletionRequest request) {
        int estimatedPromptTokens = tokenCounter.estimate(request.messages());
        
        if (estimatedPromptTokens + request.maxTokens() > budget.remainingFor(request)) {
            throw new TokenBudgetExceededException(
                "Request would exceed token budget: " + estimatedPromptTokens + " tokens requested");
        }
        
        CompletionResponse response = delegate.complete(request);
        budget.record(request, response.usage());
        
        return response;
    }
}

Budget at multiple levels:

Per-request maximum (cap individual requests)
Per-user daily limit (prevent abuse)
Per-feature daily limit (contain runaway costs)
Overall daily limit (financial safety net)

Structured Outputs and Validation

LLM outputs should not be trusted directly. Parse them into structured types and validate:

public record CustomerIntentClassification(
    @NotNull IntentCategory category,
    @Min(0) @Max(1) double confidence,
    @NotBlank String reasoning
) {}

public CustomerIntentClassification classifyIntent(String customerMessage) {
    var request = CompletionRequest.builder()
        .messages(List.of(
            systemMessage(prompts.render("intent-classifier")),
            userMessage(customerMessage)
        ))
        .responseFormat(ResponseFormat.jsonSchema(CustomerIntentClassification.class))
        .maxTokens(200)
        .build();
    
    CompletionResponse response = llmClient.complete(request);
    
    try {
        CustomerIntentClassification result = objectMapper
            .readValue(response.content(), CustomerIntentClassification.class);
        
        // Validate constraints
        Set<ConstraintViolation<CustomerIntentClassification>> violations = 
            validator.validate(result);
        
        if (!violations.isEmpty()) {
            log.warn("LLM returned invalid structure: {}", violations);
            throw new LLMOutputValidationException(violations);
        }
        
        return result;
    } catch (JsonProcessingException e) {
        log.error("LLM returned invalid JSON: {}", response.content());
        throw new LLMOutputParseException(e);
    }
}

Structured output enforcement (available in OpenAI, Anthropic APIs) guarantees valid JSON matching your schema. Even with this, validate the content — a valid JSON object with an confidence value of 1.5 is schema-valid but semantically invalid.

Data Privacy Boundaries

Every token sent to an LLM API leaves your infrastructure. This has implications:

PII in prompts: customer names, email addresses, medical records, financial data sent to external LLM APIs may violate GDPR, HIPAA, or your customer contracts. Audit what goes into prompts.

Data retention: LLM providers have data retention policies. Understand them. OpenAI, Anthropic, and Azure OpenAI have different defaults. Zero-data-retention agreements exist but cost more.

Pseudonymization: for sensitive data that must be processed, pseudonymize before sending and de-pseudonymize on return:

public String summarizeWithPrivacy(CustomerRecord record) {
    // Pseudonymize
    String pseudoId = pseudonymizer.pseudonymize(record.customerId());
    String sanitized = record.notes()
        .replace(record.name(), "[CUSTOMER]")
        .replace(record.email(), "[EMAIL]");
    
    // Process with pseudonymized data
    String summary = llmClient.complete(buildRequest(sanitized));
    
    // Result doesn't contain PII — safe to use
    return summary;
}

On-premises models: for the highest privacy requirements, run models locally (Ollama, vLLM) or in your own cloud account. The tradeoff is infrastructure cost and model capability.

Observability for AI Features

Beyond standard application observability, AI features need:

Prompt logging (with PII scrubbing): the ability to reconstruct exactly what was sent to the LLM when something goes wrong.

Token usage by feature: which features are consuming most tokens? Where is the cost coming from?

Quality metrics: task success rate, user feedback, hallucination detection from your evaluation pipeline.

Provider availability: LLM API error rates, latency, and rate limit exhaustion.

@Aspect
@Component
public class LLMObservabilityAspect {

    @Around("@annotation(LLMCall)")
    public Object observe(ProceedingJoinPoint pjp) throws Throwable {
        Timer.Sample sample = Timer.start(registry);
        String featureName = getFeatureName(pjp);
        
        try {
            Object result = pjp.proceed();
            sample.stop(timer("llm.request", "feature", featureName, "result", "success"));
            recordTokenUsage((CompletionResponse) result, featureName);
            return result;
        } catch (Exception e) {
            sample.stop(timer("llm.request", "feature", featureName, "result", "error"));
            throw e;
        }
    }
}

The Architectural Boundary

AI features should have a clear boundary in your architecture. Business logic should not directly call LLM APIs — it should call an AI service interface:

// Business logic — doesn't know about LLMs
public class CustomerSupportService {
    private final TicketClassifier classifier;    // Interface
    private final ResponseDrafter drafter;         // Interface
    
    public SupportTicket processTicket(CustomerMessage message) {
        IntentCategory intent = classifier.classify(message);
        String draftResponse = drafter.draft(message, intent);
        return new SupportTicket(message, intent, draftResponse);
    }
}

// Implementation — knows about LLMs (behind the interface)
@Service
public class LLMTicketClassifier implements TicketClassifier {
    private final LLMClient llmClient;
    
    @Override
    public IntentCategory classify(CustomerMessage message) {
        // LLM-specific implementation
    }
}

This boundary provides:

Easy substitution of AI implementation for tests (use a fake classifier)
Ability to switch from LLM-based to rule-based implementation
Clear separation between business logic and AI plumbing
Testable business logic without LLM calls

AI is infrastructure, not domain logic. Treat it accordingly.