Dwarf Fortress and Claude's ASCII Art Blindness

Claude has trouble playing Pokemon partially because it can't see the screen very well. This made me wonder if Claude would be better at an ASCII game like Dwarf Fortress, where it doesn't need to rely on image recognition.

To check this, I built an MCP server to let Claude control an interactive terminal, and installed a text version of Dwarf Fortress.

Screenshot from this video

Unfortunately this was a failure, since Claude has significant trouble interpreting the screen, and even when I switched to a much simpler roguelike game (Angband) it had trouble interpreting the 2D ASCII grid. To make this work well, an LLM would likely need to be trained with 2D position information and/or use character-level tokenization instead of BPE.

Terminal Game Tooling

Claude usually runs terminal commands until they end and then reads the output. This doesn't work for games since they expect commands while the game is running. Luckily, the Unix gods gifted us with tmux: A tool that runs a persistent terminal, with commands to send key presses and read the content of the screen.

I wrote an MCP server that wraps tmux, so Claude can use tools like:

• send_keys(["h", "e", "l", "l", "o", "Return"])
• read_output

Claude is extremely good at using this when the terminal output is easy to read.

Dwarf Fortress ASCII Mode

Dwarf Fortress was traditionally ASCII-based, but the newest version uses graphics, and the old terminal mode was removed. Luckily, there are still old versions floating around. The dwarffortress snap package uses an old enough version that ASCII mode still exists, although you'll need to install the latest/edge version to pick up a necessary library for text mode.

Then you can enable text mode by editing ~/snap/dwarffortress/current/data/init/init.txt to change PRINT_MODE to TEXT. See this recording for an example.

Color is Important

At this point, I ran into trouble because Claude could sort-of play the game, but was weirdly bad at it. It would say that it was selecting something in the menu that it clearly wasn't, or it would miss obvious rivers of blue on the map.

I eventually realized that Dwarf Fortress uses colors to indicate things like your current selection, and tmux doesn't show those by default.

Dwarf Fortress world creation screen showing that the History row is selected (note the bold) and that Medium is selected (the cyan highlight).

I added a mode to view color, and Claude got better at reading some of these screens...

Claude Doesn't See in 2D

And here's where the experiment starts to break down. With colorized output, Claude could see menu selections, but it became extremely hard for it to understand the 2D map output, which was a mix of meaningless symbols and color instructions.

LLMs like Claude are trained on 1D text. Where we see ASCII art like:

        (__)
`\------(oo)
  ||    (__)
  ||w--||

Claude sees¹:

        (__)\n`\------(oo)\n  ||    (__)\n  ||w--||

2D text games use ASCII art for their maps, and it turns out a Dwarf Fortress map including color codes is very hard to read in this format², and Claude was extremely confused.

Behold: A perfectly readable map

Let's Try Something Simpler

So, I decided to try another game: Angband. This is a roguelike game where you explore a dungeon. The map is much simpler, and doesn't use colors for anything important, so we don't need to pollute the tmux output with them.

To make things go more smoothly, I started by having Claude generate some documentation by reading the help menu³ so I could reference it in future sessions.

Then I told it to try to get through the dungeon... and it didn't work very well⁴.

Claude Sonnet 4 playing Angband
Later sections of this video get garbled because of asciinema's interaction with nested tmux sessions, but the raw tmux output was fine. I also recorded Claude Opus 4.1 having similar trouble.

It certainly works better than Dwarf Fortress, but it's clear that Claude can't tell where it is on this map either, or where the different rows in the output line up. It walks the wrong direction, runs into a lot of walls and can't catch up with the cat it wants to eat⁵. I added some tools to help it, like numbering the rows and giving it the coordinates of specific characters, but even with that, it gets lost if it doesn't ask for coordinates after every move.

This is the point where I gave up. The goal was to see if Claude could play 2D text games without assistance, and I think at this point it's clear that it would need significant assistance to use this input effectively.

Conclusion

In summary,

• Claude is pretty good at using tmux when it can understand the output.
• It has a lot of trouble understanding 2D ASCII-art output.
• Sometimes color is important, but colorized ASCII-art is even harder to read.

Making every token the same width instead of using BPE might help, but I think it would be really interesting if someone trained a model with 2D positional information. It's irrelevant for most text, but I imagine ASCII art is common enough in the training data that LLMs could learn to use it. Or it would be really easy to come up with synthetic data from terminal programs and games.