Advanced

LangSmith

LangSmith is the observability and evaluation platform for LLM applications. Trace every step, debug failures, evaluate quality, and monitor production systems.

What is LangSmith?

LangSmith is a DevOps platform built specifically for LLM applications. It provides:

🔎

Tracing

See every LLM call, tool invocation, and chain step with full input/output data, latency, and token counts.

🐛

Debugging

Drill into failed runs, inspect prompts, compare outputs, and replay runs with modified inputs.

📈

Evaluation

Create test datasets, run evaluations with custom metrics, and track quality over time.

📡

Monitoring

Track latency, error rates, token usage, and costs in production. Set up alerts for anomalies.

Setting Up LangSmith

Bash 
pip install langsmith
.env 
# Enable LangSmith tracing
LANGCHAIN_TRACING_V2=true
LANGCHAIN_API_KEY=lsv2_pt_your-key-here
LANGCHAIN_PROJECT=my-project  # Optional: organize runs into projects

Once these environment variables are set, all LangChain operations are automatically traced - no code changes needed.

Tracing Runs

Every chain invocation creates a trace that you can inspect in the LangSmith UI:

Python 
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

prompt = ChatPromptTemplate.from_template("Tell me about {topic}")
model = ChatOpenAI(model="gpt-4o-mini")
chain = prompt | model | StrOutputParser()

# This run is automatically traced in LangSmith
result = chain.invoke({"topic": "LangSmith"})

# Add metadata and tags for filtering
result = chain.invoke(
    {"topic": "LangSmith"},
    config={
        "metadata": {"user_id": "alice", "version": "1.0"},
        "tags": ["production", "v1"],
        "run_name": "topic-explainer",
    },
)

Debugging Chains

LangSmith shows a full tree view of each run, making it easy to find where things went wrong:

What You See in the LangSmith UI 
RunnableSequence (2.3s, 847 tokens)
  ChatPromptTemplate (0.1ms)
    Input:  {topic: "LangSmith"}
    Output: [SystemMessage(...), HumanMessage("Tell me about LangSmith")]
  ChatOpenAI (2.1s, 847 tokens, $0.0004)
    Input:  [messages...]
    Output: AIMessage("LangSmith is an observability platform...")
  StrOutputParser (0.1ms)
    Input:  AIMessage(...)
    Output: "LangSmith is an observability platform..."

Evaluation Datasets

Create test datasets and evaluate your chains systematically:

Python 
from langsmith import Client

client = Client()

# Create a dataset
dataset = client.create_dataset("qa-evaluation")

# Add examples
client.create_examples(
    inputs=[
        {"question": "What is LangChain?"},
        {"question": "How do chains work?"},
        {"question": "What is LCEL?"},
    ],
    outputs=[
        {"answer": "LangChain is an LLM application framework"},
        {"answer": "Chains compose components using the pipe operator"},
        {"answer": "LCEL is LangChain Expression Language"},
    ],
    dataset_id=dataset.id,
)

Custom Evaluators

Define custom evaluation metrics to measure chain quality:

Python 
from langsmith.evaluation import evaluate

# Custom evaluator function
def check_contains_keyword(run, example):
    """Check if the output contains key terms from the reference."""
    prediction = run.outputs["output"]
    reference = example.outputs["answer"]
    keywords = reference.lower().split()
    matches = sum(1 for kw in keywords if kw in prediction.lower())
    score = matches / len(keywords) if keywords else 0
    return {"key": "keyword_match", "score": score}

# Run evaluation
results = evaluate(
    chain.invoke,                  # Your chain
    data="qa-evaluation",           # Dataset name
    evaluators=[check_contains_keyword],
    experiment_prefix="v1-test",
)

print(results)

Prompt Playground and Hub

LangSmith includes a prompt playground for iterating on prompts and a Hub for sharing them:

Python 
from langchain import hub

# Pull a prompt from LangSmith Hub
prompt = hub.pull("rlm/rag-prompt")

# Push your own prompt to the Hub
hub.push("my-org/my-prompt", prompt)

Production Monitoring

Monitor your LLM application in production with LangSmith dashboards:

Python 
# Add feedback from users
from langsmith import Client

client = Client()

# Log user feedback on a specific run
client.create_feedback(
    run_id="run-uuid-here",
    key="user-rating",
    score=1.0,         # 0.0 to 1.0
    comment="Great answer!",
)

# Track costs and latency in dashboards
# LangSmith automatically tracks:
# - Total tokens (input + output)
# - Latency per step
# - Error rates
# - Cost estimates
✅
Free tier: LangSmith offers a generous free tier that includes tracing, debugging, and basic evaluation. This is sufficient for development and small production workloads. Paid plans add team features, higher limits, and advanced analytics.

What's Next?

The final lesson covers Best Practices - project structure, error handling, cost optimization, testing, and deploying LangChain applications to production.

← Previous LangGraph

Ready to Go Deeper?

Live instructor-led courses from our partners. Affiliate disclosure.

🧠

AI & ML Courses - 30% Off

Live instructor-led AI, machine learning, data science, and cloud courses for working professionals. Use code Limited30 at checkout.

Edureka
📊

DataCamp - AI & Data Science

Hands-on Python, machine learning, and AI courses with interactive exercises and real projects.

DataCamp
🎓

edX - Top AI Courses

University-level AI courses from MIT, Harvard, Stanford. Earn certificates that employers recognize.

edX