To answer my own question.
I needed to set the baseURL field in the Media Source (which I did before posting the question in the forums), but then I needed to go back into the page that was using that template with the ClientConfig variable and resave it to update the cache.
Sometimes that cache can get you when troubleshooting.