Right-click on the Taskbar icon and further click on ‘Pin to Taskbar’ That’s all you have to do to permanently base your File Explorer icon on the Windows 11 Taskbar.