When waking my computer from sleep or I think sometimes when changing the video output mode (PC screen only, extend, etc.), other audio output sources aside from the currently selected one - including even the default device if it is not the current one in use - do not appear in the taskbar (see here: https://i.imgur.com/KUFKyqh.png). However, when I right click the volume icon and open the "Sounds" setting menu, all of them that should be there immediately become available as options (https://i.imgur.com/HSc6EXG.png?1).
I have tried the "playing audio" troubleshooter, and the audio drivers for all devices are up to date.