0:00

Transforming 3D Gaming Environments with Generalist AI Agents 🚀

Introduction to SIMA Research

In the exciting realm of artificial intelligence, new findings are being revealed. Among them is the research on the Scalable Instructable Multiworld Agent (SIMA), a groundbreaking generalist AI agent capable of interpreting natural-language instructions and executing tasks within diverse video game environments.

The Significance of Video Games in AI Development

Video games serve as an exceptional testing ground for AI technologies. They provide rich learning environments that challenge AI systems with responsive, real-time settings and dynamic objectives. Google DeepMind has a longstanding presence in the intersection of AI and gaming, having evolved from initial projects with Atari games to the sophisticated AlphaStar system, which competes at a human-grandmaster level in StarCraft II. Today, the focus has shifted toward enhancing game-playing AI agents with broader applicability.

Introducing SIMA: A Generalist AI Agent for 3D Environments

We are thrilled to announce the launch of SIMA, as detailed in our technical report. This innovative generalist AI agent has been trained in collaboration with various game developers across multiple video game settings. SIMA represents a notable advancement in AI, demonstrating the ability to comprehend a wide range of gaming worlds and execute tasks based on natural-language commands, akin to human interaction.

Importantly, this research emphasizes not merely scoring points in games but the potential of AI agents to function across various environments by interpreting instructions. Our findings highlight the opportunity to evolve advanced AI models for real-world applications through a linguistic interface.

Renting Knowledge: Learning from Video Game Environments

To ensure SIMA is exposed to a range of environments, we partnered with eight game developers to train and evaluate it across nine different games, including No Man’s Sky by Hello Games and Teardown by Tuxedo Labs. Each game within SIMA’s diverse portfolio introduces unique interactive worlds and a variety of skills, ranging from basic navigation and menu handling to complex tasks like resource mining and constructing items.

Furthermore, we have integrated four specialized research environments, including a newly created platform with Unity, titled the Construction Lab. Here, agents engage in building sculptures from blocks, allowing us to evaluate their object manipulation skills and their intuitive grasp of physical interactions.

Through direct observation and analysis of human gameplay, the development of SIMA involved recording instances where human players followed instructions or engaged in gameplay independently, allowing us to derive meaningful instruction sets from their actions.

Understanding SIMA’s Architecture

SIMA is equipped with a robust architecture that enables it to perceive and comprehend various environments, responding effectively to user-instructed goals. The system is comprised of:

A precise image-language mapping model
A video model capable of predicting subsequent on-screen events

These models have been fine-tuned using training data specific to the diverse 3D settings within the SIMA portfolio. Notably, SIMA does not require access to the underlying source code of any game or specialized APIs, needing only two inputs: the visual display of the game and natural-language instructions relayed by the user. The simplicity of this interface mirrors human gameplay, allowing SIMA to engage with virtually any virtual environment.

The current iteration of SIMA has been assessed across 600 fundamental skills, covering domains such as navigation (e.g., “turn left”), object interaction (e.g., “climb the ladder”), and menu manipulation (e.g., “open the map”). Training has enabled SIMA to complete straightforward tasks that can be finished within roughly 10 seconds.

Aiming for Greater Complexity in Tasks

As we look to the future, our aim is for SIMA and similar agents to handle more sophisticated tasks requiring strategic planning and completion of multiple sub-tasks, such as “Find resources and build a camp.” This objective is critical in the broader landscape of AI development, as while Large Language Models have showcased their prowess in accumulating world knowledge and generating plans, they currently lack the capability to take proactive actions on behalf of users.

Generalization Across Diverse Game Environments

Our research indicates that SIMA, trained across various games, outperformed agents specialized in singular game experiences. When evaluated, SIMA agents developed from nine 3D games significantly surpassed those trained solely for individual titles. Notably, when tested in a game not included in its training, SIMA performed nearly as effectively as agents specifically trained on that game, showcasing a remarkable potential for generalization beyond its initial training parameters.

Furthermore, our tests elucidated that SIMA’s performance is fundamentally reliant on language comprehension. In control scenarios devoid of language training or instructions, the agent exhibited behavior that, while appropriate, was aimless. For instance, the agent might engage in resource gathering—a common task—rather than accurately following specific directional instructions.

We evaluated SIMA’s ability to follow instructions to complete nearly 1500 unique in-game tasks, in part using human judges. As our baseline comparison, we use the performance of environment-specialized SIMA agents (trained and evaluated to follow instructions within a single environment). We compare this performance with three types of generalist SIMA agents, each trained across multiple environments.

Future Directions in AI Agent Research

The promising results from SIMA pave the way for a new generation of generalist, language-driven AI agents. As this research is still in its early phases, we are enthusiastic about expanding SIMA’s capabilities and incorporating increasingly sophisticated models in future iterations.

By exposing SIMA to an extensive array of training environments, we anticipate enhancing its versatility and generalizability. Alongside the development of more advanced models, our goal is to improve SIMA’s understanding and competency in executing complex language commands to achieve broader objectives.

This research initiative is a stepping stone toward developing general AI systems and agents capable of comprehending and proficiently executing myriad tasks, benefitting users in both virtual and real-world experiences.

Credit Source: Revolutionizing 3D Environments with Generalist AI Agents