Once upon a time, there was a bustling eCommerce store called "ShopNest," selling unique, handmade goods. The store had a gorgeous website, filled with high-quality images, detailed product descriptions, and a seamless shopping experience. But something strange happened one morning—sales suddenly dropped. It wasn't just a little drop. It was significant. Like a huge chunk of their audience had vanished.
Confused and frustrated, Maya, the store's digital marketing manager, sat down with her team. They’d run ads, worked on SEO, and had optimized the user experience. But the drop in sales wasn't making sense. That’s when she decided to look into Google Search Console (GSC) to see what was going on with their organic search traffic.
As she dove deeper into GSC, she stumbled upon a term that she hadn’t fully understood before Crawled – Not Indexed. According to the report, some of their key product pages were listed as "crawled but not indexed." But what did that mean?
The Mystery of "Crawled – Not Indexed"
To understand what had happened to ShopNest, Maya had to learn more about this mysterious "Crawled – Not Indexed" message. Here’s what she found:
When Googlebot crawls a website, it essentially means the search engine’s crawler (Googlebot) has visited and analyzed the page. This is an essential step in the process of getting a page indexed. Once Googlebot crawls a page, it can then index it, making it available in Google’s search results. But sometimes, Googlebot crawls a page but doesn't index it for various reasons. When this happens, the page remains invisible to Google’s search engine results.
So, why was this happening to ShopNest’s pages?
The Culprit: "Crawled – Not Indexed"
Maya reached out to her SEO expert, Raj, who had worked with many online stores over the years. He explained that "Crawled – Not Indexed" usually appears when Googlebot successfully crawls the page, but it decides that the page isn’t important enough to be included in its index. Raj likened it to a person who gets invited to a party (crawled) but isn't allowed to enter (indexed).
The causes could be numerous. Raj pointed out some common reasons:
- Thin or Duplicate Content: If the product page had too little original content or if it was too similar to other pages, Google might skip indexing it.
- Noindex Tag or Meta Tags: Sometimes, pages accidentally contain noindex tags or directives telling Google not to index them.
- Low-Quality or Low-Authority Pages: Google might choose not to index pages it considers of low value or relevance to search queries.
- Crawl Budget Issues: For larger eCommerce sites, Google allocates a "crawl budget" to pages. If a site has too many pages or low-quality pages, Google might not index all of them.
How to Fix "Crawled – Not Indexed" for eCommerce
Determined to get her pages back on track, Maya worked closely with Raj to fix the issue. Here are the steps they took to solve the "Crawled – Not Indexed" problem:
1. Improved Content Quality
Raj recommended that Maya focus on improving the content quality of the product pages. ShopNest had great products, but the descriptions were often too brief. They enriched the product descriptions with more details, added customer reviews, and included blog-style content around their products. By adding more valuable and unique content, they made sure that Google saw each page as important.
2. Checked for Noindex Tags
Maya made sure to audit the product pages for any accidental noindex tags or meta tags. It turned out a few pages had these tags set in their meta headers, which told Google not to index them. Once those were removed, Googlebot was able to revisit and re-crawl those pages with the intent to index them.
3. Boosted Internal Linking
Raj emphasized the importance of internal linking. If a page isn’t linked to other important pages on the site, it might be harder for Google to understand its relevance. Maya started linking to the newly improved product pages from her homepage, category pages, and even blog posts. This increased the authority and visibility of those pages in Google’s eyes.
4. Optimized for Crawl Budget
With a site like ShopNest’s, which had thousands of products, crawl budget was always a concern. Raj recommended prioritizing the most important pages by creating an XML sitemap and submitting it directly to Google Search Console. This ensured that Googlebot spent its time crawling the most valuable pages first.
5. Used Fetch as Google
Once all the fixes were in place, Maya used the "Fetch as Google" tool in GSC. This allowed her to request a re-crawl of the pages that were previously marked as "Crawled – Not Indexed." Googlebot revisited the pages and, with the improvements, started indexing them properly.
6. Checked for Technical Issues
Raj also suggested checking for any technical issues that might be blocking Google from indexing the pages. Maya ran a thorough audit using tools like Screaming Frog and SEMrush, fixing issues related to page speed, mobile-friendliness, and server errors.
The Results
It took a few weeks, but soon enough, the pages that had been marked "Crawled – Not Indexed" started showing up in the Google search results. Slowly but surely, ShopNest’s organic traffic began to rise again. The product pages were now visible to shoppers searching for unique handmade goods. The store’s rankings improved, and so did their sales.
Maya and Raj celebrated their victory. They had tackled the issue head-on and turned things around for ShopNest. The moral of the story? Understanding what "Crawled – Not Indexed" really means and taking the right steps to fix it can make a huge difference in an eCommerce business’s online visibility and success.
And so, ShopNest thrived once again—more visible than ever in the world of Google search.
