I've read and been told divergent and confusing information regarding these three question ending styles. Note that I'm not asking about their neutral use in academic and other written contexts, just about the nuances and proper use when speaking.
Also, I'm not asking about the 나요 or (는/ㄴ)가요 forms either.
Thanks for any help.
Let's look at the dictionary definitions first.
-냐
(‘이다’의 어간, 용언의 어간 또는 어미 ‘-으시-’, ‘-었-’, ‘-겠-’ 뒤에 붙어)
해라할 자리에 쓰여, 물음을 나타내는 종결 어미.
-나
(주로 동사 어간이나 어미 ‘-으시-’, ‘-었-’, ‘-겠-’ 뒤에 붙어)
하게할 자리에 쓰여, 물음을 나타내는 종결 어미.
- 자네 언제 떠나나?
자기 스스로에게 묻는 물음이나 추측을 나타내는 종결 어미.
- 얼굴이 붉어서 난 자네가 술을 마셨나 했어.
-는가
(‘있다’, ‘없다’, ‘계시다’의 어간, 동사 어간 또는 어미 ‘-으시-’, ‘-었-’, ‘-겠-’ 뒤에 붙어)
하게할 자리에 쓰여, 현재의 사실에 대한 물음을 나타내는 종결 어미.
- 시간 좀 있는가?
자기 스스로에게 묻는 물음이나 추측을 나타내는 종결 어미.
- 과연 이 땅에도 봄은 오는가?
The biggest difference is in speech level. -냐 is 해라체, which is a impolite(talking down, 낮춤) speech level. Thing to note is that -냐 is a traditionally formal speech level, although most modern usages aren't. It goes with -아/어라, -자, -단다, etc. -느냐 and -으냐 is an old fashioned way of saying -냐.
-나 and -는가 are both 하게체, which is a strictly formal and moderately impolite(예사 낮춤) speech level. It's somewhat uncommon these days. They go together with -게, -세, -네. The difference between the two is: 나 is used in conjuction with tense endings, like 했었나 and 했겠나. -는가 is strictly present tense.
The second and more common usage of -나 and -는가 are the same: they're used to indicate self-questions and assumptions, and pretty much interchangable. This usage is only used in coda when you're talking to yourself. Otherwise they're used in the middle of a sentence, making auxillary verbs, used before verbs like 하다, 모르다, 궁금하다, etc.
Edit: There's also dialectal usages, quite different from standard usages above.
-나 is used as a yes/no question ending in 해라체 level in the southeastern reigon(경상). In contrast, -노 is used as a question ending asking for explanations. For example:
어디 가노? Where are you going?
- 슈퍼 갑니더. I'm going to the supermarket.어디 가나? Are you going somewhere?
- 예/아니오 Yes / No
This is a remnant of -오/-아 in Middle Korean, and a lost feature in Standard Korean.
On -나, we need to distinguish at least two different types of situation.
In type I, it is used between two equals.
A: 가 보았나?
B: 난 아직 못 가 보았네. A군, 자네는 가 보았나?
You see this type of use in the movies set against the sixties (perhaps) or earlier. It gives you the sense that both men are on terms of equality and mutual respect. (This does not contradict the other answer's characterization of -나 as 'moderately impolite.' There, 'polite' or 'impolite' is a grammatical term. I am using 'respect' in its ordinary sense.) Women are rarely seen talking this way even in the movies.
In type II, it is one way only.
Father-in-law (professor): 가 보았나?
Son-in-law (graduate student): 아니요. 아버님은 (교수님은) 가 보셨습니까?
This kind of talk is still common. Also women use it too, for example, a mother-in-law to her son-in-law (though not to a daughter-in-law).
Please note that I am not saying that -나 has two different senses or two different levels of 'impoliteness.' There is the same level of 'moderation of impoliteness,' as it were. The difference lies in the direction (bi- or unilateral).
If a young man today addressed a same-age friend in -나, then the addressee would be rather confused because neither type I nor II applies readily. (They are not in a movie, and the speaker is not an elder.)
-냐 and -니 imply total familiarity (again, not a grammatical term). You will use it to your childhood friends or a child or animal.
Notice that children who speak 'impolite' speech to their parents or grandparents will still stay clear of -냐 and -니 in addressing them.
-느냐 often (though not always) implies not only total familiarity but also assertion of authority or even contempt. In the movies, domestic servants, subordinates in the military, and captured criminals will be addressed this way.
-는가 as used in a question would seem to me equivalent to -나 in terms of respect. That is to say, whenever you could use -나, whether in type I or II, you could also use -는가, and vice versa. In fact, I cannot distinguish between these two in any respect:
(1) 자네 조간신문을 보나?
(2) 자네 조간신문을 보는가?
In some other context, -는가 may have a poetic tone not shared by -나. (For instance,'이 땅에도 봄은 오는가' quoted in the other answer.)
