I have been searching the forums and google and have been unable to find the solution for allowing google robots.txt to log into my site and scan it.
The issue is that my site basically doesn’t exist unless you log in with your character (every page requires some character data, to be displayed). Google suggest that I ‘disable the login page’ to allow it to scan. This is not an option as there will be nothing but broken pages.
The only idea I have had so far is to allow the robots to login (somehow) with my account to scan it (so it can see the site), which is far from ideal.
Does anyone have idea how I could allow google (for search, and adsense) to read my site without letting it log into my account to display the site?
(I don’t suppose there is a robots.txt refresh token available anywhere?)
I’m hoping that submitting a sitemap will assist google in indexing my site…
However, even if it does, if I wanted to incorporate adsense into my site, I think I may still need a solution for allowing the robots to access my site.
This isn’t my area of expertise but I doubt you would want authed pages of your application to be indexed? Also I’d say 99% of the people searching on google aren’t looking for your site, so since it’s for such a niche group is it even worth it?
Would probably get more traffic advertising on the forums or in game or something.
without knowing what your site is doing that it needs to be indexed, I doubt anyone will be able to offer you any feedback, as it seems absurd on its face to have a site that you need indexed for the public, want to adsense, and cant view without your own token all at the same time.
So… You guys don’t know how to do it. That’s okay. It would be much easier just to charge people ISK to use my site. I think I’ll go with that unless someone can answer my question; instead of questioning my question.
Regardless, Thank you for your replies!
(I’m a programmer; sometimes we do things not because we should, but because we can, just to see if we can do it.)