In this context, brackets mean all the icons floating in space: friendly and enemy ship icons, wrecks, containers and such.
If you rightclick the hamburger menu (four horizontal lines) of your overview, you have the option to ‘show all brackets’, ‘hide all brackets’ or ‘stop showing/hiding all brackets’.
While these options won’t change what shows up on your overview, it will change how much you will see floating in space. When all brackets are hidden, you won’t see those icons in space (although they can still show up on your overview if your tab is set to show those icons). When all brackets are shown, you will see every icon in space regardless of your overview settings, so you will never be surprised with enemy ships right next to you that are invisible due to overview settings. The third option is the default option, which shows only brackets of the stuff that you selected to show on your overview.
I prefer showing all brackets regardless of overview settings. Unless your computer can’t handle it (because it’s an old computer or you’re in one of those huge battles) I recommend it.