Working with LLM Reasoning / Thinking

Motivation

Sometimes you need to validate an LLM’s reasoning process in addition to obtaining a structured result.

Consider this scenario: a user wants to plan a vacation and specifies that their preferred destinations are Greece and Italy, with available travel dates in June, August, or September. They ask the LLM to find flight options with affordable tickets for a one-week stay. The LLM returns a structured Flight object containing departure and return dates, destinations, and prices. Even if the output adheres to the expected schema, you may want to verify that:

• The flight dates fall within the requested months
• The destinations are actually in Greece or Italy rather than somewhere else

If the flight details fall outside the user’s criteria, access to the LLM’s reasoning process helps you understand why it made those choices.

An even more important use case arises when the LLM cannot fulfill a request—for example, when it cannot create the requested object because the user’s criteria are ambiguous or contradictory. In this case, the thinking blocks explain what went wrong, even though no result was produced.

Concepts

• ThinkingBlock — An abstraction that carries details about LLM reasoning, including the tag type, tag value, and reasoning text content.
• ThinkingTagType — An enum defining the types of reasoning markers: TAG (XML-style tags like <think>), PREFIX (line prefixes like //THINKING:), and NO_PREFIX (untagged reasoning text before JSON output).
• ThinkingResponse<T> — A response wrapper that holds both the result object and a list of ThinkingBlock instances.
• ThinkingException — An exception that preserves thinking blocks when object instantiation fails, enabling debugging even in error scenarios.
• thinking() — The core PromptRunner API method that enables thinking extraction.

Example: Handling Objects and Thinking Blocks

Java

// Configure the PromptRunner with an LLM and optional tools
PromptRunner runner = ai.withDefaultLlm()  // Example uses claude-sonnet-4-5
                        .withToolObject(Tooling.class);

String prompt = """
    What is the hottest month in Florida and its average high temperature?
    Please provide a detailed analysis of your reasoning.
    """;

// Use thinking() to enable thinking block extraction
ThinkingResponse<MonthItem> response = runner
        .thinking()
        .createObject(prompt, MonthItem.class);

// Access the structured result
MonthItem result = response.getResult();

// Access the LLM's reasoning process
List<ThinkingBlock> thinkingBlocks = response.getThinkingBlocks();

// Inspect individual thinking blocks
for (ThinkingBlock block : thinkingBlocks) {
    System.out.println("Type: " + block.getTagType());   // TAG, PREFIX, or NO_PREFIX
    System.out.println("Tag: " + block.getTagValue());   // e.g., "think", "analysis"
    System.out.println("Content: " + block.getContent());
}

Kotlin

// Configure the PromptRunner with an LLM and optional tools
val runner = ai.withDefaultLlm()  // Example uses claude-sonnet-4-5
                .withToolObject(Tooling::class.java)

val prompt = """
    What is the hottest month in Florida and its average high temperature?
    Please provide a detailed analysis of your reasoning.
    """.trimIndent()

// Use thinking() to enable thinking block extraction
val response = runner
        .thinking()
        .createObject(prompt, MonthItem::class.java)

// Access the structured result
val result = response.result

// Access the LLM's reasoning process
val thinkingBlocks = response.thinkingBlocks

// Inspect individual thinking blocks
for (block in thinkingBlocks) {
    println("Type: ${block.tagType}")   // TAG, PREFIX, or NO_PREFIX
    println("Tag: ${block.tagValue}")   // e.g., "think", "analysis"
    println("Content: ${block.content}")
}

Example: Handling Failures Gracefully

Use createObjectIfPossible when the LLM might not be able to produce a valid result:

Java

ThinkingResponse<MonthItem> response = runner
        .thinking()
        .createObjectIfPossible(prompt, MonthItem.class);

MonthItem result = response.getResult();
if (result != null) {
    // Process the result
} else {
    // Object creation failed—examine the reasoning to understand why
    for (ThinkingBlock block : response.getThinkingBlocks()) {
        logger.info("LLM reasoning: {}", block.getContent());
    }
}

Kotlin

val response = runner
        .thinking()
        .createObjectIfPossible(prompt, MonthItem::class.java)

val result = response.result
if (result != null) {
    // Process the result
} else {
    // Object creation failed—examine the reasoning to understand why
    for (block in response.thinkingBlocks) {
        logger.info("LLM reasoning: {}", block.content)
    }
}