This could be the most common overlay arrow icon, which indicates that the icon ... a blank or transparent .ico file of size, or you can download this one from our servers and use it.

Once all the icons are displayed, the Up/Down arrow will disappear. To hide some icons, just drag them to the left side and the arrow will appear. Then you can drop the icon to hide it.