EVE Technology Lab

 
^ Back to top

Topic is locked indefinitely.

 

Retribution 1.0.7 463858 SDE Conversions

Author
Vote Steve Ronuken for CSM
#1 Posted: 2012.12.19 13:40
http://www.fuzzwork.co.uk/dump/mysql55-retribution-1.0.7-463858.tbz2


XLS and CSV exports: http://www.fuzzwork.co.uk/dump/retribution-1.0.7-463858/

Sqllite (full database): http://www.fuzzwork.co.uk/dump/retribution-1.0.7-463858/eve.sqlite.bz2


And now read the following from CCP Prism X. Or else.



CCP Prism X wrote:


IMPORTANT NOTICE:
READ THIS OR YOU'LL GET ANGRY WHEN I MAKE FUN OF YOU FOR NOT HAVING READ THIS!

Now that I got your attention, reading this will also make you angry.
There is no iconID or graphicID in invTypes anymore. You will never find it there again. It has been moved into the types.YAML file we ship with the SDE zip file.
If you're using a third party conversion that does not include this data in one form or another that's your converter doing something wrong.
If you are doing a conversion and want this data back into a relationial database, you're going to have to get it from the YAML file.

Changelog:
invTypes:
Problem: table is missing the iconID column.
Solution: Get them, along with the graphicIDs, from types.YAML. They will not be found in the official backup.

eveUnits:
Problem: unitID 103, 122, 129 is missing.
Solution: These units do not have a display name, do include units without a display name.


dogmaTypeAttributes, dogmaAttributeTypes:
Problem: Missing slot attributes
Solution: Fixing eve units to include the units will retain the slot data rathre than sanitizing it out as a broken relation.

invMarketGroups, invTypes
Problem: Missing records.
Solution: These are unreleased DUST records which do not belong in the SDE.


ramTypeRequirements
Problem: Data Interfaces for invention have a damage per job of 1.
Solution: This is how the data is authored and always has been. Checked the history. I can't fix things that are by design.
Interfaces are special cased in our code. Maybe one day I get to rewrite the clusterintimacy that the S&I backend is.


invMarketGroups
Problem: Market groups exist with hasTypes = 0, but still have types. Market groups exist with hasTypes = 1 but have no types.
Solution: Check these relations and sanitize out bad data. I have no clue when GD will actually fix the data but at least I can fix your SDE
Amarr Empire
#2 Posted: 2012.12.21 22:44  |  Edited by: Cyrus Teymour
Thanks for doing this! It's immensely helpful.

One note: Unique indexes are showing up as regular indexes in the sqlite file.
Vote Steve Ronuken for CSM
#3 Posted: 2012.12.21 23:51
Cyrus Teymour wrote:
Thanks for doing this! It's immensely helpful.

One note: Unique indexes are showing up as regular indexes in the sqlite file.



I'm just running this through a converter script. I'll have a poke at it and see if there's something I can change.
Gallente Federation
#4 Posted: 2013.01.14 06:49  |  Edited by: Nykr Rognvaldr
Thanks for the conversions, I'm starting my first eve project and it's immensely helpful.

I have noticed what looks like a text encoding problem with the MySQL conversion. Some UTF-8 characters appear mangled. An example:

Item 33048 in the invTypes table should be "NEO YC 114: Asine Hitama‘s team". Instead, the file contains "NEO YC 114: Asine Hitama‘s team". I think the correct character is E28098 (left single quotation mark). The actual byte sequence in the FuzzWork file is C3A2E282ACCB9C.

I have verified that my DB table is using the utf8_general_ci collation, and that the bogus characters do appear in the raw SQL file before importing. (The command I used to test the file is fold invTypes.sql | grep 'NEO YC 114: Asine')

Thanks!
.
Vote Steve Ronuken for CSM
#5 Posted: 2013.01.14 10:48
That's odd. Looks fine in the copy that I'm using (which is where it was pulled from). And in a copy I just restored from the single file export.


That, however, looks like it's a UTF8 display issue, when you have 2 characters showing up where there should only be one.
Gallente Federation
#6 Posted: 2013.01.15 06:06
Thanks for checking. I'm still stuck on this, if you have a moment would you mind repeating this test for me? (I included the curl command this time to make sure we're talking about the same file).

#!/bin/bash
curl http://www.fuzzwork.co.uk/dump/retribution-1.0.7-463858/invTypes.sql.bz2 | bzip2 -d -c | fold -s | grep '114: Asine'

One other thing I found, the character sequence I'm seeing matches what would be expected if UTF-8 bytes were treated as latin (Windows-1252 or ISO-8859-1): http://www.i18nqa.com/debug/utf8-debug.html

I asked on #eve-dev, and two other people got the same result, so I don't think it's just my system.
.
Caldari State
#7 Posted: 2013.01.15 10:05
Nykr Rognvaldr wrote:
I asked on #eve-dev, and two other people got the same result, so I don't think it's just my system.

yup, that looks quite broken in the multi-encoded way.

"c3 a2 e2 82 ac cb 9c" instead of "e2 80 98".

looking at it in hex really helps there to rule out "display issues".
Vote Steve Ronuken for CSM
#8 Posted: 2013.01.15 10:32
Nykr Rognvaldr wrote:
Thanks for checking. I'm still stuck on this, if you have a moment would you mind repeating this test for me? (I included the curl command this time to make sure we're talking about the same file).

#!/bin/bash
curl http://www.fuzzwork.co.uk/dump/retribution-1.0.7-463858/invTypes.sql.bz2 | bzip2 -d -c | fold -s | grep '114: Asine'

One other thing I found, the character sequence I'm seeing matches what would be expected if UTF-8 bytes were treated as latin (Windows-1252 or ISO-8859-1): http://www.i18nqa.com/debug/utf8-debug.html

I asked on #eve-dev, and two other people got the same result, so I don't think it's just my system.



I get the same result for that.

Hmm. Sometimes I hate character encoding issues.

I can replicate both ways, however and I think I know what the problem is. It's the LANG setting. For some historical/hysterical reason, mine's set to en_GB. When it's that, it works on import. When it's en_GB.UTF-8, it doesn't.


I'll kick at it some more. See if I can work out appropriate options to make it work more generally. (redoing the export with a different LANG setting doesn't do it.)

Holy Frog
#9 Posted: 2013.01.15 11:09
Good Job!
Cheers.
..
Forum Jump