A New Benchmark for Embodied AI: Evaluating LLMs in Decision Making

In the world of embodied AI, where agents navigate and make decisions in digital or physical environments, a significant challenge has been evaluating the capabilities of large language models (LLMs).

Until now, research in this area has been fragmented: models have been tested under different conditions, using varying task specifications and success metrics, making it difficult to truly understand their strengths and weaknesses.

To address these issues, researchers have introduced a new evaluation framework called the Embodied Agent Interface. This framework seeks to standardize how we evaluate LLMs in embodied decision-making by unifying various tasks and creating a consistent benchmark for performance.

The Embodied Agent Interface breaks down the decision-making process into four fundamental modules: Goal Interpretation, Subgoal Decomposition, Action Sequencing, and Transition Modeling. Each module represents a distinct aspect of how an AI agent interprets instructions, formulates goals, sequences actions, and predicts the effects of its interactions with its environment.

One of the key innovations of this framework is the use of Linear Temporal Logic (LTL) to standardize goal specifications. LTL helps describe both state-based and extended goals over time, allowing for more expressive and flexible goal definitions.

This approach not only makes evaluation more consistent across tasks but also facilitates deeper insights into where LLMs excel or struggle—whether it's understanding goal nuances, breaking down tasks, or planning actions effectively.

The Embodied Agent Interface also introduces fine-grained metrics that go beyond simple success rates, identifying specific types of errors such as hallucination errors, affordance errors, and planning sequence errors. This provides a more nuanced understanding of how LLMs perform in complex environments and highlights areas that need improvement, like accurately predicting object relationships or handling preconditions for actions.

In testing multiple LLMs across well-known benchmarks like BEHAVIOR and VirtualHome, the researchers found that, while many models could successfully interpret basic instructions, their performance declined when tasked with more complex sequences or when goals involved intricate relationships between objects.

This new benchmark, therefore, not only shines a light on the current limitations of LLMs in embodied tasks but also provides a standardized path forward for researchers looking to enhance embodied AI capabilities.

The Embodied Agent Interface is a crucial step towards developing more capable AI systems that can understand, interpret, and act in the real world. By providing a unified and detailed assessment, it enables researchers to pinpoint specific areas for improvement, ultimately paving the way for more effective and versatile embodied agents.