Monday Question: What Does a Crawler See?
This Monday's question needs one free tool and about five minutes:
What does an AI crawler actually receive when it fetches your homepage?
Not what you see in a browser. What a fetch tool sees: raw HTML, no JavaScript executed, no rendering. Plenty of modern, good-looking sites return almost nothing this way, empty divs where the real content should be.
I found this out about my own site the hard way. It's the same failure mode behind why Copilot answers brilliantly in one company and not another and why five-year-old directory listings can outrank your own homepage: these systems only know what they can actually reach.
Paste your homepage URL into any raw-fetch tool this week. If what comes back is mostly blank, that's the finding, and it's fixable.
Checking whether a site is actually readable by these systems is where I start with every client. Get in touch if you want yours checked.
Rosemarie Withee has spent thirteen years helping operations teams get real work out of their software, first Microsoft 365, now AI. She’s written six books for Wiley and builds AI products at Portal Integrators.