The supply of interesting AI tools now exceeds anyone’s capacity to evaluate them. The scarce resource is not tools; it is the hours you can spend finding out whether one works.

What follows is a procedure for spending those hours well, biased toward reaching a “no” quickly.

Start with the failure modes, not the demo

Every project leads with what it does well. The useful question is what it does badly, and the answer is usually in the issue tracker rather than the README.

Sort open issues by most-commented and read the top ten. You will learn more in five minutes than from any amount of documentation.

Stars measure how many people thought something looked interesting once. They say nothing about whether it still works.

Better signals:

  • Date of the most recent commit that was not a dependency bump
  • Whether issues get responses, even short ones
  • Whether the last release was cut from a branch that still exists

A tool with 800 stars and a maintainer who answers questions is a safer bet than one with 30,000 and an inbox of two-year-old issues.

Run it on something real

The demo repository is chosen to make the tool look good. Your repository was not.

Give it a task you have already done yourself, so you can judge the output without a second investigation. If it cannot handle a problem you know the answer to, it will not handle one you don’t.

Establish the exit cost before you commit

The question that matters is not “is this good?” but “what happens when I stop using it?”

A tool that reads your existing files and writes ordinary output is nearly free to abandon. A tool that stores your data in its own format, or that everything else comes to depend on, is not. Prefer the reversible option when the two are close on merit — you are buying the ability to be wrong cheaply.

When to stop evaluating

Stop when you have an answer, not when you have a complete picture. Most evaluations should end within an hour, with a “no” and a one-line note about why, so that the next time it surfaces you do not repeat the work.

Write that note somewhere you will find it again. The most commonly wasted evaluation is the second evaluation of a tool you already rejected.