Engineering LLM Applications

Introduction

• Source: OpenAI Cookbook、Anthropic Building with the Claude API、Full Stack LLM Bootcamp
• Prerequisites: Python basics, understanding of LLM API calls, basic web development knowledge
• Programming language: Python
• Course difficulty: 🌟🌟🌟
• Estimated study hours: 20+

Calling an LLM API to build a demo is easy, but making an LLM application production-ready is a completely different story. This course covers engineering practices for LLM application development: how to design reliable API call layers, handle streaming output, implement error handling and retries, control costs, and evaluate output quality.

These skills are the key leap from "toy project" to "real product." Andrew Ng repeatedly emphasized at Buildathon: the core capability of an AI engineer isn't tuning models — it's building reliable AI systems.

Course Resources

• Documentation: OpenAI Cookbook — OpenAI's official code example collection covering various use cases
• Documentation: Anthropic API Documentation — Claude API official docs with rich engineering practices
• Course: Full Stack LLM Bootcamp (Stanford) — Stanford's free course covering full-stack LLM application development

Resource Summary

API call engineering:

• OpenAI Cookbook - API best practices — Streaming output, error handling, retry strategies, concurrent requests
• Anthropic - Streaming Messages — Streaming output implementation guide
• tenacity (Python retry library) — Add exponential backoff retries to API calls
• LiteLLM — Unified API interface for 100+ LLM providers, avoid vendor lock-in

Cost management:

• OpenAI Cookbook - Managing Costs — Rate limiting, budget control, token counting
• tiktoken (OpenAI Token Counter) — Precisely calculate token usage before API calls
• Token cost calculator — Unified token cost calculation across models

Output quality assurance:

• Structured Output (OpenAI) — Enforce specific JSON format from model output
• Instructor (Python) — Pydantic-based structured output library
• Guardrails AI — Framework for validating and filtering LLM output
• LMQL — Embed constraints and logic directly in prompts

Evaluation and monitoring:

• LangSmith (LangChain) — Debugging, testing, and monitoring platform for LLM applications
• Braintrust — LLM evaluation platform with automated testing support
• OpenEvals (LangChain) — Open-source LLM evaluation framework
• DeepEval — pytest-like testing framework for LLMs

Web framework integration:

• FastAPI — The go-to Python framework for building LLM backend services
• Streamlit — Quickly build front-end interfaces for LLM applications
• Gradio — Quickly create interactive demos for ML/AI models
• Next.js + Vercel AI SDK — Full-stack LLM application development

Notes

Engineering capability is the key differentiator between "can use AI" and "AI engineer." Recommended path: first build a simple backend with FastAPI + OpenAI API → add streaming output → add error handling and retries → add cost monitoring → add evaluation. With each addition, your understanding of production-grade AI systems deepens.

Don't neglect evaluation — an LLM application without evaluation is like software without tests. You'll never know when it breaks in production.