If you're on version 3.0 or newer of the Android or iOS app, you no longer have to go between Listen and Watch to switch to a video's audio-only version!
All you have to do is open the episode you want to listen to. From there, you can easily switch between audio and video at the top of the episode player.
If the episode in question does not have an audio and video version (ex, Morning Wire), you will not see this toggle.