# EVE-Search and EVE-Online **URL:** \ **Category:** Forum Feedback & Requests\ **Created:** [June 20, 2017, 7:15pm UTC](https://forums.eveonline.com/t/eve-search-and-eve-online/4391 "2017-06-20T19:15:57Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Chribba](https://sea1.discourse-cdn.com/eveonline/user_avatar/forums.eveonline.com/chribba/32/396873_2.png) [@Chribba](https://forums.eveonline.com/u/Chribba)\ **Post date:** [June 20, 2017, 7:15pm UTC](https://forums.eveonline.com/t/eve-search-and-eve-online/4391/1 "2017-06-20T19:15:57Z")
PSA. EVE-Search. Might not mirror the new forums ahead. Reasoning, due to the nature of mirroring, Google has come down multiple times on me in the past for dick-pics/porn and general asshattery posting images that get mirrored. But we’ll see if I get the time to rewrite my bot and perhaps exclude images. Guess it comes down to popular demand/need of EVE-Search to continue its work when CCP changes forums. ---
**Author:** ![Yourmoney\_Mywallet](https://sea1.discourse-cdn.com/eveonline/user_avatar/forums.eveonline.com/yourmoney_mywallet/32/168387_2.png) [@Yourmoney\_Mywallet](https://forums.eveonline.com/u/Yourmoney_Mywallet)\ **Post date:** [June 20, 2017, 8:46pm UTC](https://forums.eveonline.com/t/eve-search-and-eve-online/4391/2 "2017-06-20T20:46:36Z")
Without eve-search how are we gonna remind all the bad-poasters with their empty “PLS DELETE” poasts of their bad-poasting? 😈 ---
**Author:** ![Chribba](https://sea1.discourse-cdn.com/eveonline/user_avatar/forums.eveonline.com/chribba/32/396873_2.png) [@Chribba](https://forums.eveonline.com/u/Chribba)\ **Post date:** [June 21, 2017, 5:52am UTC](https://forums.eveonline.com/t/eve-search-and-eve-online/4391/3 "2017-06-21T05:52:07Z")
It’s going to be hard. And we won’t find all the deleted posts and such. The biggest problem right now just looking at the layout of this new stuff is that there’s no last post timestamp and similar which makes it really really hard to determine if a thread has been updated or not, so I would have to crawl through threads over and over even if there’s nothing new in it since I can’t really see changes easily. Going by purely replies isn’t reliable since posts can be deleted and similar. But we’ll see if I can get time to re-write my indexer and adjust so I can merge the new into the old stuff. ---
**Author:** ![CCP\_Avalon](https://sea1.discourse-cdn.com/eveonline/user_avatar/forums.eveonline.com/ccp_avalon/32/410280_2.png) [@CCP\_Avalon](https://forums.eveonline.com/u/CCP_Avalon)\ **Post date:** [June 21, 2017, 1:38pm UTC](https://forums.eveonline.com/t/eve-search-and-eve-online/4391/4 "2017-06-21T13:38:09Z")
Would some [json](https://forums.eveonline.com/c/feedback/l/latest.json) per [category](https://forums.eveonline.com/categories.json) simplify your task? 😛 All I ask in return is that you keep your rate of checking sensible. ---
**Author:** ![ISD\_Stall](https://sea1.discourse-cdn.com/eveonline/user_avatar/forums.eveonline.com/isd_stall/32/12239_2.png) [@ISD\_Stall](https://forums.eveonline.com/u/ISD_Stall)\ **Post date:** [June 21, 2017, 1:49pm UTC](https://forums.eveonline.com/t/eve-search-and-eve-online/4391/5 "2017-06-21T13:49:27Z")
That’s no fun and you know it! ---
**Author:** ![Chribba](https://sea1.discourse-cdn.com/eveonline/user_avatar/forums.eveonline.com/chribba/32/396873_2.png) [@Chribba](https://forums.eveonline.com/u/Chribba)\ **Post date:** [June 22, 2017, 5:11am UTC](https://forums.eveonline.com/t/eve-search-and-eve-online/4391/6 "2017-06-22T05:11:29Z")
That’s pretty nice and that could potentially make it easier yeah. What is sensible? Thus far I’ve ran my indexer every 10 minute, so that would be the same rate most likely (and then indexing through the actual updated threads of course). I still need to make the decision about killing all images on my side though as google doesn’t appreciate our user’s dick pics 😉 ---
**Author:** ![yellow\_parasol](https://sea1.discourse-cdn.com/eveonline/user_avatar/forums.eveonline.com/yellow_parasol/32/10455_2.png) [@yellow\_parasol](https://forums.eveonline.com/u/yellow_parasol)\ **Post date:** [June 22, 2017, 6:59am UTC](https://forums.eveonline.com/t/eve-search-and-eve-online/4391/7 "2017-06-22T06:59:58Z")
hahaha they hate dickpics. i didnt test that feature yet, so here’s a pic of a Dick! ![image](https://global.discourse-cdn.com/eveonline/original/2X/7/7a8bb8257f964ae02775a96e8e9c7911b00d6eb0.jpg) Chribba, how do you scrape a site that’s not fully built? As, like, when you need to scroll down so it loads more posts? ---
**Author:** ![Chribba](https://sea1.discourse-cdn.com/eveonline/user_avatar/forums.eveonline.com/chribba/32/396873_2.png) [@Chribba](https://forums.eveonline.com/u/Chribba)\ **Post date:** [June 22, 2017, 4:03pm UTC](https://forums.eveonline.com/t/eve-search-and-eve-online/4391/8 "2017-06-22T16:03:59Z")
I’m still looking into that. What I’ve found so far is that one can make use of the “streams” value and poll a topic the stream ID’s (probably best to do in batches as a QueryString can only be so long). So at the moment what I might be going for is poll TOPIC\_ID.json to get the basics including streams, then counter-check ID’s in my database and query posts.json for ID’s I don’t have in a batch size of around 100 (I did a batch of 500 and it worked but it goes quick enough so I can pull 100 at a time without problem). ---
**Author:** ![yellow\_parasol](https://sea1.discourse-cdn.com/eveonline/user_avatar/forums.eveonline.com/yellow_parasol/32/10455_2.png) [@yellow\_parasol](https://forums.eveonline.com/u/yellow_parasol)\ **Post date:** [June 22, 2017, 4:10pm UTC](https://forums.eveonline.com/t/eve-search-and-eve-online/4391/9 "2017-06-22T16:10:28Z")
Ah, at least I’m not the only one. 🙂 thanks!