How we test an AI-made game before you ever play it
The worst thing an AI game maker can hand you is a game that opens to a black screen, or one where the first enemy can never be beaten. You find out in ten seconds and you lose faith in everything after.
So every game Tyrbo builds is played by machines before it reaches you. Not looked at, played. Here is what those checks do, in plain words, and why some things are measured but never allowed to fail a game.
First, does it build at all
The game's code has to compile cleanly and bundle into something a browser can load. That sounds basic, and it is, but it catches a surprising share of problems early, before anyone wastes a minute playing.
Then, does it boot and answer a thumb
Next the game is opened on three screens: a desktop window played with the keys, a phone held upright (390 by 844 points) and a phone held sideways (844 by 390), both played by a thumb. On each, three screenshots are saved: as the page opens, after an input, and mid-play.
A game is only played on the screens it was made for. A phone game is never failed for how it looks on a desktop, and a desktop game is never failed for lacking touch controls.
Then, two bots play it properly
This is the part we are proudest of. Two bots play every game, three times each, from three different seeds:
- A random bot mashes moves. It finds crashes and stuck states a careful player never would.
- A greedy bot looks a little ahead and picks the move that seems to help most. It shows whether the game can actually be made progress in.
Each run plays for up to 300 seconds of game time, stepped as fast as the computer allows rather than in real time. So a survival game whose first night only comes after a few minutes still gets played through to its end, and its end screen gets seen. Each bot's wins, losses, scores and play time are recorded.
What fails a game
The list is short and deliberate. A game fails if:
- its code does not compile or bundle
- the page throws an error
- it has not started within 15 seconds
- it plays differently twice from the same seed, so nothing you test can be trusted
- a screenshot comes back blank
- a run goes by in which nothing about the game changed
- the greedy bot makes no progress on any seed: no wins, and neither its score nor its sense of the goal rose
- the game hangs, with a call into it taking more than 10 seconds to return
When a check fails, the builder sees exactly which one and why, and fixes the game before you get it.
What is measured but never fails
Some things matter but are matters of taste, and a check that fails a game for style would push every game toward the same shape. So these are reported, never judged:
- Whether input changed anything, and how each bot did. A calm puzzle game might not care that random mashing never wins.
- Code quality, scored from 0 to 100 against the patterns we teach the builder, like keeping every sound in one place and every tuning number in one table. It helps us see trends. It never stops your game.
- How it runs on a slow phone. When asked, the game is opened at phone size with the processor slowed four times, as a phone speed test would, and its frames are counted while a thumb steers.
- What a change touched. After you ask for a change, a report lists every value in the game's data that changed or disappeared, so the builder can put back anything you did not ask it to remove. We never delete your work to make a check pass.
The tour: a play-through for the judge
Separately, the judge that scores a game against what you asked for is shown a tour. The greedy bot plays the best of its seeds in six short legs, with a frame saved after each. Then a careless bot, one that always picks the worst move, plays until it gets hit and until it loses, because a skilled bot never shows you the hit effect or the game over screen. In a survival game the wolves only come out at night, so the careless bot keeps trying from later points in the run until something actually hurts it.
Choosing the right seed matters too. In one test, an overworld game's first seed died in its forest, so the judge never saw the ruin at the end. The other two seeds won. Now the tour plays the seed that went furthest.
A small thing that made a big difference
Bots that look ahead have to try moves and then undo them. Our first bots undid a move by replaying the whole run from the start, which got slower and slower as the run went on. Games can now offer a save and restore of their whole state, and the bot jumps straight back. On our 2D starter held to a full 300-second run, a seed went from 15 seconds of real time to 0.15. Faster checks mean more of them, on every change.
What this means for you
When a build reaches you, it has already been opened on the screens it is for, played by a thumb, mashed by a random bot and played to an end by a greedy one, three times each. That does not make it fun; only you can judge that. It means your first ten seconds are spent on whether the game is good, not on whether it works. To get the most from that time, read how to write a game prompt before you start, and how to make a game with AI for what to do once you are playing.
Questions people ask
Does Tyrbo test the games it makes?
Yes. Every build is compiled, opened on a desktop and two phone shapes, and played by a random bot and a greedy bot from three seeds each, before you see it. A game that crashes, hangs, shows a blank screen or cannot be made progress in fails and is fixed first.
Do the checks make sure my game is fun?
No, and they do not try. They make sure it builds, boots, answers input and can be played. Whether it is fun is your call, which is why the checks never fail a game for style or structure.
Will a check ever delete part of my game?
No. After a change, a report lists every value that changed or was removed, so anything you did not ask to lose can be put back. Never deleting your work to pass a check is one of our rules.
Are phone games tested on a real phone?
The checks play phone games in a browser at real phone sizes, by touch, upright and sideways. A performance check can also run the game with the processor slowed four times. You should still play every build on your own phone, which is where you feel what a bot cannot.
Make the game this guide is about.
Make your gameSign in to get the plan and its concept art. Both are free.