Same question, opposite answers

2026 · AI research as a draft · ChatGPT, Claude, Gemini Deep Research

In May 2026 I asked an AI whether a document I’d been sent had been written by AI. I asked in two separate chats, on the same day, with the same material.

One chat said it was probably written by a person. The other said, with complete confidence, that it was AI-generated.

Same assistant, same question, same evidence, opposite verdicts. If I’d only asked once, I’d have walked away with a certainty I had no right to.

A confident answer is not a finding

That moment put a name on something I’d been seeing for a while. AI research reads like a finished report: headings, citations, a clear conclusion. The format makes everything sound settled, even when the reasoning underneath is a coin toss.

When I published a Gemini Deep Research report on the state of software engineering, I left it unedited for exactly this reason. Some of its sources were Stanford, ACM and LinkedIn’s own newsroom. Others were Medium posts and Reddit threads. The report presented them all with the same confidence. Two of its patterns held up when I checked them against my own work. The rest I’d treat as starting points, not conclusions.

The job title nobody had

In 2025 I built a prompt to research companies before sales outreach. It began: “You are a B2B company research analyst… produce a factual brief suitable for sales outreach prep”. The briefs looked great.

Then one of them gave someone at a prospect a job title they didn’t have. It looked exactly like the correct facts around it. If I’d used it in an email, I’d have opened a conversation with a mistake about the person I was writing to.

I could have fixed that one field and moved on. I redesigned the prompt instead, so the same kind of error would be harder to make:

  • Cite a source for every fact.
  • Prefer first-party sources: the company’s own site and announcements.
  • Cross-check anything important against a second source.
  • Separate what was observed from what was inferred.
  • Flag conflicts instead of quietly picking one.
  • Say “Unknown” when something can’t be verified.

That last rule did more than all the others. A model that’s allowed to say “Unknown” stops inventing answers just to fill a gap.

When the research never ran

There’s a quieter failure, too. I set up an automated research call with a web-search tool attached. The answer came back as a request to run a search, not as research. The model had asked for the tool, and nothing in my setup actually ran the search and fed the results back.

Giving a model a tool isn’t the same as building the loop that uses it. It sounds obvious written down. It wasn’t obvious until a blank brief showed up.

Habits that came out of it

  • Ask twice for anything that matters. Different chats, or different models. If they disagree, you’ve learned how uncertain the question really is.
  • Give it the whole picture. The less context I give, the more an answer tends to lean toward whatever I seem to believe. With all the material in front of it, it’s far more willing to tell me where I’m wrong.
  • Bring your own knowledge. My best research sessions started with what I already knew about a business. The AI’s job was to challenge it and fill gaps, not replace it.
  • Let it correct you too. When I asked for help publishing an llms.txt file, the AI pointed out that llms.txt is a proposal, not an established standard. That saved me from presenting it as one.
  • Check the numbers yourself. On a long report early in 2025, AI drafted about 70 percent of the text. I checked every statistic by hand.

What stuck

AI research is a fast, well-formatted first draft. It gives you structure, leads and a list of things to check. It doesn’t give you the truth. The checking is still your job.

If you’re using AI for research today

  • Require sources, and open them. A citation you haven’t clicked is decoration.
  • Allow “Unknown” in every research prompt.
  • Run the important question twice, ideally in two different tools.
  • Mark what’s fact and what’s inference before anything goes into an email, a proposal or a decision.

Related: When the second AI just copies the first · State of Software Engineering 2026

Back to AI Journey Chronicles