I'm sure everyone will agree that they sound similar enough to their own counterparts to be interchangeable. If there are no tangible rules that separate their usage, what are some tips to spot their proper usage situations? Also, why does such distinction exist in the first place?
In the modern Seoul dialect, these are audibly "indistinguishable." I put that in quotations, since there is controversy over this. But, I believe that some dialects still distinguish them in an actually meaningful way. Not being an expert on southern 사투리, I cannot authoritatively comment on this, however it is my understanding that certain dialects certainly have distinct sounds for these pairs. This is evidence of there actually being a difference at one point, hence the distinction existing in the "first place." (Languages evolve of course).
Consider English for a moment. Do we really need the letter C? Every sound C makes can be replaced with a K or an S. Or take Q as another example. The mesh 'kw' provides a very nice replacement for Q. But we retain its use in English.
It could be argued that it is the same sort of thing for ㅐ, ㅒ as opposed to ㅔ,ㅖ. These letters are retained because they are still used. It is sort of circular logic to say that, but keep in mind that most languages are not changed for the sake of simplicity of spelling rules.
Furthermore, I met many Koreans that could not hear the difference between the words "play" and "pray." Words like "clown" and "crown" also sometimes gave them issues. But native speakers of English do not usually have these issues of auditory distinction. Even if non-natives cannot distinguish the sounds, native English speakers can tell the difference. Hence the reason we actually retain the use of R and L in English.
Based on everything I know about hanja and Korean linguistics, no Sino-Korean (Chinese character based) word uses the letter ㅒ.
I have seen 얘 used as a shortened form of 아이 (child), as in "얘들아" (hey kids!)
I believe that technically, 애 is pronounce with the mouth and throat more open (or "spread out" ["입을 벌리는 정도가 더 큽니다"], as dotVezz's link in the comments suggests). I have seen some people transcribe 애 as /ɛ/ as in "Larry" or "fairy" and 에 as /e/ as in "pest" or "men."
Not being a pure linguist, I am not going to try to delve into every little detail of the IPA transcription of the "true" pronunciation of these letters. Whether they want to admit it officially or not, Koreans under age 60 (especially those in Seoul) say 애 and 에 (and sometimes 예) pretty much identically. It is the whole "marry-merry-Mary" issue. English linguistics professors may say these differently, but we normal folk all say them pretty much identically.
Just be careful if you go to a restaurant and ask for "gey-gogi:" You might get 개고기 (Dog meat, which is quite good!), 게고기 (crab meat), or 계고기 (meat of the....lineage (?)...).
As a native Korean speaker, I use them based on my knowledge of grammar or rules or whatever you want to call it, rather than their sound.
It's because I totally agree that they are ALMOST completely indistinguishable by sound. For me, this can be compared to knowing the following difference:
I WRITE my journal everyday.
How would you know when to say RIGHT as opposed to WRITE while they sound so similar? They have different meaning, so you know when to use which.
Similarly, you would know when to use ㅔ/ㅖ or ㅐ/ㅒ because they have different places to be. It's more of a SPELLING or GRAMMAR issue rather than something that can be distinguished by their "supposed" sound differences.
For your question about why there is such a thing, I would say that languages are never perfect.
This would be like the question I used to have when I was learning English. I never understood why English would care so much about Singular or Plural nouns while the rules are so vague to be even called a rule, let alone vowel pronunciation. (If you disagree, I suggest you check out these words: fish, food, sheep, deer etc.)
I memorized them first and got used to them now. I got used to them so much that they just feel natural to me. And this is how I use ㅔ/ㅖ, ㅐ/ㅒ, too. I just got too used to them.
ㅐ and ㅔ are indistinguishable in normal conversation, and there is no need to distinguish it in a normal conversation. We understand using 'context' just like other languages.
Sometimes, you need to distinguish it when spelling names. In that situation, we say ㅏㅣ for ㅐ, and ㅓㅣ for ㅔ just to distinguish them.
However, there is a difference in theory.
We call the smallest unit of sound 음운(phoneme).
Then 음운 can be divided into 모음(vowel) and 자음(consonant).
(To be accurate, 본절 음운 can be divided into those two. Not going to go into details. The sentence above is still correct)
Then again 모음 can be divided into 단모음 and 이중 모음
단모음 is a vowel(모음) that doesn't change the position of your tongue or mouth while pronouncing it.
On the other hand, 이중 모음 is a vowel(모음) that changes the position of your tongue or mouth while pronouncing it.
Here is a table of 단모음
Now this is a map of 모음.
the quadrilateral shows where (or how) the tongue is when saying those vowels.
For example, ㅜ is said with your highest position of your tongue in the back and your tongue will be high with your mouth only slightly open. (also using the table above, you can find out that the shape of your mouth is round because ㅜ is 원순 모음)
Therefore, in theory, ㅔ and ㅐ are both pronounced with your highest position of your tongue in the front, the shape of your mouth not round. The only difference is that ㅔ is said with your tongue in the middle height, while ㅐ is said with your tongue low with your mouth more open.