Is it a problem if I mispronounce ㅗ for ㅛ or vise versa? Will the person I am speaking to still understand me?
In terms of combining sounds and real conversation people will understand what you mean
imagine you said, "안녕하세오" really fast, they might not catch the difference.
I'm my opinion however, I think it's important to perfect (or work really hard on) your pronunciation since ㅗ and ㅛ are easier to fix than ㅗ and ㅓ for example.