In my opinion you only refresh when you need to, not just on a flat every 20 minute schedule. I store the expires time in epoch for each token which is tied to the specific character. When a request needs to be made I check if expires time is less than current time, and if so do a quick call to get a new access token.
Be prepared to catch 4xx errors, when a user revokes the token on the CCP side, your site will get an expired error with a 4xx code. I find it good practice to allow the “delete” option where you clear them from your DB, Cache and do the call to the revoke endpoint that removes the access for your site. That way, your site knows about the revoke, and you can remove them from your end.
Also 5xx code can be present when ESI is having issues.