How the AI Dungeon Master Actually Works

Most AI "Dungeon Masters" are a system prompt wrapped around a chatbot. Type something, get a response, hope it remembers what happened last turn.

Infinite Realms works differently. Our DM is a multi-agent system — not a single model with a fantasy prompt, but a coordinated set of specialized AI agents that handle storytelling, rules interpretation, and world continuity as separate concerns. Here's how it actually works under the hood.

The Architecture: Two Agents, One DM

The AI DM runs on Google Gemini 2.5 Flash — chosen for its speed, context window, and instruction-following ability. But the magic isn't just the model. It's the system around it.

When you type an action, your input flows through two specialized agents:

The Storyteller Agent

This is the narrator. It receives your action, the current scene state, your character details, relevant memories, and any active combat context. Its job is to craft a response that:

Advances the narrative
Stays consistent with established world details
Matches the campaign's tone (gritty, whimsical, heroic)
Describes sensory details that make scenes feel real
Introduces meaningful choices

The Rules Interpreter Agent

This agent handles the mechanics. When you say "I try to shove the orc off the bridge," the rules interpreter:

Identifies the correct 5E mechanic (contested Athletics check)
Calculates modifiers from your character sheet
Determines advantage or disadvantage from conditions
Resolves the roll
Passes the mechanical result back to the storyteller for narration

The two agents collaborate. The storyteller never invents mechanics. The rules interpreter never writes prose. Each does what it's good at.

Memory: 13 Categories, Scored by Importance

The biggest gap in most AI RPG tools is memory. A chatbot forgets what happened three messages ago. Our system doesn't.

Every significant event, character, location, and item is stored as a structured memory object with:

Type — one of 13 categories: NPC, Location, Quest, Item, Event, Story Beat, Character Moment, World Detail, Dialogue Gem, Atmosphere, Plot Point, Foreshadowing, or General
Importance score — how narratively significant this memory is (1–5)
Recency — when it was last relevant
Embeddings — vector representations for semantic search

When the DM prepares a response, it queries this memory system. Relevant memories are injected into the AI's context, ordered by a combination of importance and recency. The NPC you angered three sessions ago surfaces when you enter their territory. The cursed item you picked up becomes relevant when the symptoms match.

This isn't keyword matching — it's semantic retrieval. The system finds memories that are conceptually related to the current scene, even if they use different words.

Combat: Real 5E, Not Theater of the Mind

When the AI detects a combat encounter, the system shifts modes. The DM doesn't just describe punches — it runs the actual 5E combat engine:

What's tracked:

Initiative order for all participants
Hit points, armor class, and conditions
Spell slots and concentration
Death saves (successes and failures)
Exhaustion levels (six tiers with escalating penalties)
Racial traits and class features

What's automated:

Attack rolls with correct modifiers
Advantage/disadvantage from conditions and positioning
Damage calculation including resistances
Opportunity attacks when movement triggers them
Fighting style bonuses (Defense, Archery, Dueling, Great Weapon Fighting, Protection, and more)

What you see:
The 3D battle map renders the fight in real time — tokens with health bars, condition icons, vision cones, fog of war, and area-of-effect templates. You can draw on the map, measure distances, and see exactly where everyone stands.

Combat resolves in seconds, not minutes. The AI handles the bookkeeping while you make tactical decisions.

The Prompt: 1,000+ Lines of D&D Education

Here's something most people don't think about: large language models are trained on the internet, and the internet is full of wrong D&D rules.

Our prompt builder is over 1,000 lines of carefully engineered instructions that teach the AI how D&D 5E actually works. For example:

Passive skills — The AI knows that passive Perception is 10 + Wisdom modifier + proficiency bonus. It never asks you to "roll a passive check" (a common AI mistake). It calculates the score and uses it automatically when you enter a room.

The Observant feat — If your character has it, the AI adds +5 to passive Perception and Investigation without being told.

Death saves — The AI tracks successes and failures correctly, knows that natural 20s restore 1 HP, and doesn't make the common mistake of resetting the count between rounds.

Every time we find the AI making a rules mistake, we add education to the prompt with forbidden patterns (what not to do) and correct examples (what to do instead). The system gets more accurate over time.

What the AI Can't Do (Yet)

Transparency matters. Here's what we haven't solved:

Multiplayer — This is a solo experience. The AI generates NPC companions, but you can't invite friends to join your campaign yet.
Custom rulesets — We run D&D 5E. Pathfinder, FATE, and homebrew systems aren't supported.
Perfect consistency — The memory system is good, but AI can still occasionally contradict itself. We're continuously improving retrieval quality.
Offline play — You need an internet connection. The AI runs on server-side infrastructure.

We'd rather be honest about limitations than promise features that don't exist.

The Stack (For the Curious)

AI Model: Google Gemini 2.5 Flash (with automatic fallback)
Frontend: React + TypeScript + Vite
Backend: Bun + Elysia (API server)
Database: PostgreSQL via Supabase
Battle Maps: React Three Fiber (3D rendering)
Voice: ElevenLabs text-to-speech
Memory Embeddings: OpenAI embedding model
Edge Functions: Deno (Supabase Edge Functions)
Hosting: Hetzner VPS (Nuremberg, Germany)

Try It Yourself

The best way to understand the AI DM is to play with it. Create a character, start a campaign, and see how it responds to your decisions.

Play at infiniterealms.app — free tier available, no credit card required.