I think the feature you’re talking about is called Thumbnail Previews. The Google search I would try would be “taskbar thumbnail preview not working windows 10”
If this doesn’t work, let me know, and I’ll keep trying to help you out. Also note that you might Eve-O Preview.
Finally, please come back and tell us how you got this sorted. Might help the next guy with the same problem.