Valid character IDs?

Issues, Workarounds & Localization General Issues
1

Hello everyone!

I want to collect data from zkillboard using their stat API for a university project that would cluster players according to their behavior based on what types of ships they kill/lose, where, when, etc. The goal is to let students assess, whether Radoff’s motivational player types apply in EVE by having them cluster according to ships lost by type, ships destroyed, secStatus, etc.

However, to get information about a player in the API, I need to know their ID, and this is proving rather difficult so far. I haven’t come across a way to obtain character IDs that I could use, and the only piece of information that helped me further was what kind of range EVE character IDs are in:

But, I can’t just brute-force try every single possible ID, because the API does not allow for what seems to be around a few hundred requests per hour. Using this method, it would take years if not decades to gather the needed information. So my question is, is there a possibility to get my hands on character IDs anywhere? I don’t need the names of characters, because they aren’t used anyway and as for the amount, something around 10k-50k should be enough to achieve statistical significance.

Also, if there are (legal) issues with this, please let me know!

2

Well.

You are trying to find cluster, not all the players, but instead the km of the players based on what they fly.
You will have the victim ID in the KM . any other player Id is irrelevant, since by definition you have no data to cluster them.

3

I don’t understand.

I need the following info for each player: ships destroyed, ships lost (by type), security status, gangRatio.
No matter, from which direction I start, I can’t search for a player using the API if I dont have either a name or an ID.

4

No you don’t.
This information makes no sense for players that don’t have it.

So you need that information only for the players with it.

Therefore, you want the ids of the players which provide that information. Which is present in the KMs, as well as the information.

edit : I linked this in the first post but seems it did not work.

5

no, a killmail doesn’t solve any of the problems, because 1. the API only gives back 200 killmails at most, which means at least 500 iterations to get to 10k IDs, and 2. I’s still have to go over them manually to retrieve the ID and then use it.

EDIT: I know about this site, because that’s precisely what I am using. Could you elaborate on how exactly you think I should extract the information, because to say that it’s extremely badly documented would be an understatement.

6

Which API ? How is that an issue ?

No, you don’t.
Learn what a computer is.

By thinking about it.
Do it with small numbers first, and then you can scale up the thing.
Start with just a day worth of KMs.

7

Thank you for being absolutely unhelpful and condescending.!

8

I think you deserved it. You are not even trying to do anything.

OK I got it . You don’t care about the KMs. You just want the aggregated data from zkb. (which is fine, just not obvious). Still the only way that I know of it to fetch all the victims in all the kms.

9

Sure, assume that I didn’t spend the last 4 weeks searching through forums, documentations and coding my own solutions … why would anyone do that. Because, actually, I had a solution, just not a pretty one, and I was wondering if there was a way to do it with the API instead of scraping zkillboard’s HTML

No, I don’t need killmails specifically. The could be used for more in-depth analysis as in taking into account ship fitting, who exactly was involved in the kill, etc. but that would kinda blow everything out of proportions. People gotta program k-means, and then interpret the data they’re outputting and all that in less than 6 hours …

10

You definitely did not try what I linked you.
Also claiming that you “need to fetch all the data by yourself” is pretty much a stupidity when working with a computer.

As I wrote, it’s not a problem. The problem is that, it’s not explicit in your request. Typically

Implies that you use some analyzis on the kms.

11

As I understand it, you are trying to gather data on a subset of players, not all players. The players you’re looking for are selected based on the fact that they have a killmail uploaded to zkillboard, either by themselves, their corporation or their killer(s corporation).

It seems that you want to approach this problem from the wrong/inefficient way (player IDs first, then look if they have any killmails on zkillboard). You may find that lots of players do not show up on zkillboard at all, either due to inactivity, because they haven’t died in years or because they don’t allow zkillboard to read their killmails.

The nice thing about the subset of players that do show up on zkillboard is that their IDs are available as well in the list of data. So if I were to approach this problem, I’d start with the data from zkillboard itself and ignore the character IDs that do not show up on zkillboard.

And to start looking for kills you do not need player IDs or names. A list of recent kills is enough to give you the player IDs you want. Or a list of kills per ship type, region or other known value.

2 Likes
12

The simplest solution would be to ask the creators of Zkill and have them create a DB-dump with all the information that you require for your project. you can then recreate a DB with that information so that you may play with it the way that you need/want to.

13

well I gave him the API for historical kills, so he can decide on the range of data for his data.

14

Did you try asking for multiple pages?

https://zkillboard.com/api/kills/regionID/10000002/page/2/

Yes, you’ll have to make around 500 requests, but tbh, that’s nothing. I’m doing that for market data once every 30 minutes. unless you’re trying to do it by hand. At which point, umm, you’re a bit screwed.

This is a place to go learn automation. Python works well here, with the requests library to make it nicer to do web requests.

15

The zkill history API will dump every killmailID and hash for a given day into a json file if you don’t want to deal with the pagination. e.g.: https://zkillboard.com/api/history/20160816.json

Fairly easy to then cycle through them and pull the individual KMs and dump the relevant fields into a relational DB. This can be done with a reasonably simple script. You could then similarly cycle through the unique character IDs to get any relevant character information.

Having done this before, I think you’re overestimating how long it would take to pull ~100k killmails, which will contain the characterIDs.

If you’re trying to do this without writing any code… :confused:

Zkill also has the “redisq” endpoint, which delivers a full killmail. The advantage of using it is that it would limit how much code you need to pull full killmail data - literally just a request and something to parse the result.

The disadvantage is that it’s a realtime stream so you’ll only get kills as they happen, so it will take some time to catch a usable number of KMs that way.

Anyway if you want, I think I still have ~65K characterIDs in a sqlite db somewhere, I could just dump them to a csv and send it to you.

I also have partial data for about 250K killmails, but I only stored the values I cared about at the time. It contains killmailID, KM hash, KM date, victim ID, victim corp ID, victim ship type ID, systemID, whether or not it was an NPC kill, loss value, and number of attackers. I did not store fitting data or any data about the attackers apart from the count.

I will add, though, that the data is from the local blackout so it may be “tainted” for your purposes (blackout having a substantive impact on player behavior, etc).

Personally, I think your methodology has some flaws here. Relying on killmail data probably introduces a bias itself (some people don’t die with any real frequency, playstyles like station trading aren’t really going to track this way, etc.), and then there’s the alt problem. Some of my characters have no meaningful killmail history, others have high PvP activity, etc. You’ll end up categorizing characters, not players and in many cases, you’ll categorize a given player’s alts in completely different ways.

16

If you have those files, I’d be glad if you could send them to me, that would save me lots of time.

If not, that’s okay. However, what I don’t understand is, the killmail dump only returns the ID and their hashvalue, doesnt it? If that’s the case, how would I get the killmail itsaelf from the ID?

Also, yes I’m aware of the limits of this approach, but getting data about station trading is not nearly as simple as getting killmail info, at least from what I know. And, tbf, for Radoff’s player types you don’t need it. The classification of players goes as follows: Power, achievment, social and exploration as motivators. Trading could maybe fall under achievment, which be default is a very broad and wanky category. Besides, this is also a learning outcome: In games centered around combat/conflict, how do you differentiate between power and achievment as motivations? OR, in inherently social games, how do you distinguish the socially motivated group from all the others? I don’t think sufficient data could be gathered just accessing 3rd party websites…

As for alts, I am aware of this, too. But this isn’T even my greatest concern. My greatest concern would be player motivations changing over time, like, say someone started as an exploration person, but then shifted to PvP. If that person lost lots of exploration ships, but then lost lots of combat ships as well, they won’t be classified as either. As of now, I’m still thinking about fixes. But then, I’m not sure if it even needs to, because everything I add on top of it increased the danger of workload explosion. And ultimately, it’s all just to teach people k-means clustering…

17

I think we can easily have them. Still no reason to make your homework. From the beginning your post looked like a student trying to fraud : you have no knowledge about the topic, you have no idea about a program complexity and the need to reduce it.

My guess is, you are at the first year of university. You don’t even know what a computer is made of, what it’s made for.

18

How about you go about your own business if you don’t have anything reasonable to add to the discussion?

Oh, and in case you didn’t understand the task: I’m not a student trying to solve a clustering problem, I’m doing this for students so they have data they can use for clustering. Gathering this data is just a preprocessing stage and I could use any other kind of data, so this is not “homework”. Please, just go away and stop bothering me with your weird assumptions…

19

Please, stop claiming BS.

Anyone with knowledge of K-clustering but who does not know that computer are tools specifically designed to repeat common tasks, does not even have university’s first year in programming’s knowledge.

Stop claiming that you are not the student.

20

And there’s another stupid assumption you just made, none of this makes any sense. Do you even read what you write and think for one second? Never have I even hinted at not knowing what PCs can or can’t do. Misinterpreting my notion of not wanting to put in additional unnecessary work for not knowing anything about computers is either ignorance or malice, you decide which.

I’m just gonna ignore any crap you’ll fabricate from now on, so you might just as well stop wasting your and everyone else’s time here.

You can believe what you want, that I’m a student or not. It won’T change reality