Reading server logs like a search engine does

If you only look at analytics, you are watching the audience and ignoring the projectionist. Logs are where you see how the search engine actually experiences your site.

Start with three questions

• Which URL patterns does Googlebot spend the most time on, and do they deserve it?
• How quickly does it return to your highest-value templates?
• What status codes is it actually getting, pattern by pattern?

Answer those and most "mysterious" indexation issues stop being mysterious.