Is it a problem if I mispronounce ㅗ for ㅛ or vise versa? Will the person I am speaking to still understand me?
In terms of combining sounds and real conversation people will understand what you mean
imagine you said, "안녕하세오" really fast, they might not catch the difference.
I'm my opinion however, I think it's important to perfect (or work really hard on) your pronunciation since ㅗ and ㅛ are easier to fix than ㅗ and ㅓ for example.
there's a fairly big difference between the phonetical sounds of "ㅗ" and "ㅛ" - "ㅗ" is /oh/, while "ㅛ" is more like /yoh/. Therefore, if you somehow swapped the two vowels, I would probably misunderstand you.
Source: I'm korean
They might sound similar if preceded by another vowel, like the ㅔ in '안녕하세오'. They really don't sound similar as an initial sound, or if following a consonant - 고 and 교 are very distinct, for example.
If the syllable isn't the last of the sentence, it will be easily noticed. eg.
요구르트 vs 오구르트
전효성 vs 전호성
안녕하세요 vs 안녕하세오 (can pass if spoken fast)
Both of letters can be interchangeble in most cases. The person you are speaking to will definitely understand you. But using 'ㅗ'can be heard like commanding tone for some people, it depends.