ESI failing on high volume requests?

[MJ_Maverick]

So I have a script running that checks 720 people for whether they should still have Temaspeak access or not. I know I’ve semi raised this before but I wasn’t so clear.

Using ESI (doesn’t occur using XML), the script falls silent when it’s ESI’s time to speak up. The script just stops dead. But the last line is always my debug print saying it’s making an ESI call.

I’ve been recording it and it always falls over without any error at the same region of calls. Can anyone comment on this?

Hourly attempt # | Result
#1 completed successfully. (1,412 calls)
#2 died without error at call 1,057
#3 died without error at call 1,310
#4 completed successfully. (1,412 calls)
#5 died without error at call 1,176
#6 completed successfully. (1,412 calls)
#7 died without error at call 1,269
#8 died without error at call 1,249
#9 died without error at call 1,259
#10 died without error at call 1,385
#11 died without error at call 1,291
#12 died with cURL timout, no response after 10 seconds at call 61
#13 died without error at call 1,333
#14 died without error at call 1,283
#15 died with cURL timout, no response after 10 seconds at call 401
#16 died without error at call 1,159
#17 died without error at call 1,191

The PHP function calling ESI can’t be to fault if it periodically works surely? If it works even once then that’s proof the function works, ESI isn’t even sending an error to be picked up. Just completely dead. Note that the function doesn’t report zero content being recieved either as that would be handled.

function ESIcall($url) {
    // Set constants for cURL
    $curlConstants = array(
						CURLOPT_CONNECTTIMEOUT => 10,
						CURLOPT_RETURNTRANSFER => 1,
						CURLOPT_SSL_VERIFYPEER => true,
						CURLOPT_SSL_VERIFYHOST => 2,
						CURLOPT_USERAGENT => "EVEOTS Auth",
						);
    // cURL
	$curl = curl_init();
	curl_setopt_array($curl, $curlConstants);
	curl_setopt($curl, CURLOPT_URL, $url);
	$result = curl_exec($curl);
	if ($result == false) {
		die(curl_error($curl)."[".__LINE__."]");
	}
	curl_close($curl);
	$response = json_decode($result);
	if (isset($response->error)) {
		die("<p><strong>ESI Error: ".$response->error."</strong> (".$response->error_description.") [".__LINE__."]</p>");
	} else if ($response == "") {
		die("<p><strong>ESI Error: Response was blank [".__LINE__."]</p>");
	}
	return $response;
}

[EveDataRules]

1. Are you sure this is CCP and not you?
2. You appear to be making a whole new client and connection per call which is wasteful. Esi supports http2 with concurrent requests.
3. Expect failures. You should build a retry and back off mechanism.

[MJ_Maverick]

1. No, but after extensive searching I can’t see why cURL would do this at exactly line “$result = curl_exec($curl);”.
2. I have no knowledge of this, do you have any documentation or better, examples?
3. Honestly have thoguht about it but not sure how to go about it, if I received an error that I could act on sure. But there is nothing, the script just drops dead.

[EveDataRules]

Unfortunately I do not code PHP much.

Simultaneous cURL requests using curl_multi_exec in PHP - Webkul Blog

This may help, you can batch requests. If all you want is what corp they are in, you can also use the bulk endpoint. https://esi.tech.ccp.is/ui/#/Character/post_characters_affiliation

[MJ_Maverick]

Hmm the limit from ESI of 1,000 isn’t great, but can be worked around with, just a pain in the arse.

Thanks for the link I’ll read up. I’d still like some input from CCP as to whether it is the API or not.

[Steve_Ronuken]

What are you doing when you’re checking? More than just a membership check? If not, look at https://esi.tech.ccp.is/ui/#/Character/post_characters_affiliation

[MJ_Maverick]

Yeah @EveDataRules already advised that
Just means I have to note down and store their character name now as that doesn’t provide it. But it’s something I’m looking to do with bulking cURL for Teamspeaks with membership of over 1,000.

[Steve_Ronuken]

Doh! Missed that bit.

[Franky_Saken]

function ESIcall($url) {
    // Set constants for cURL

    # XXX these should be stored globally no need to reset these every
    #     function call
    $curlConstants = array(
                        CURLOPT_CONNECTTIMEOUT => 10,
                        CURLOPT_RETURNTRANSFER => 1,
                        CURLOPT_SSL_VERIFYPEER => true,
                        CURLOPT_SSL_VERIFYHOST => 2,
                        CURLOPT_USERAGENT => "EVEOTS Auth",
                        );
    // cURL

    # XXX This should be globalled, you can re-use the same curl handle
    #     it's actually preferred because it means curl can keep the connection
    #     open provided it goes to the same hostname (it does)
    $curl = curl_init();
    curl_setopt_array($curl, $curlConstants);

    curl_setopt($curl, CURLOPT_URL, $url);

    $result = curl_exec($curl);

    # XXX needs === for identity comparison but is nitpicky
    if ($result == false) {
        die(curl_error($curl)."[".__LINE__."]");
    }

    # XXX Unecessary if you do the above
    curl_close($curl);

    # XXX This returns NULL on invalid json
    $response = json_decode($result);

    # XXX NULL Doesn't have a ->error attribute
    # XXX you need to use property_exists instead of isset, isset will return false if the property
    #     exists but is NULL
    if (isset($response->error)) {
        # XXX HTML injection, escape properly
        die("<p><strong>ESI Error: ".$response->error."</strong> (".$response->error_description.") [".__LINE__."]</p>");
    # XXX this makes no sense, check if it's null *before* you try to access properties, the above
    #     properties don't exist on "" either...
    } else if ($response == "") {
        die("<p><strong>ESI Error: Response was blank [".__LINE__."]</p>");
    }
    return $response;
}

XXX’s comments are mine.

You need to show caling code and how you handle errors for us to make any sense of what is happening but adjusting the code for the above errors will make it a bit more sane to reason about.

[EveDataRules]

Talking with some folks on slack, 1000 connections is around when the CDN starts ignoring requests. So it looks like you are hitting that limit and need to look at reusing connections.

[MJ_Maverick]

Hi guys, I’ve taken all your advice onboard and made some changes. The key of which being tweaking my database layout a little to support a call to /affiliation/.

The result has been the scripts performance is now 200x faster. Literally. With an increase in local execution time of just 1 second.

It would make sense if it was the CDN although I’ve never heard of that before. But that’s what I acted on nonetheless and the script will now work out how many batches need to be run prior, then build and execute them against /affiliation/.