I think 형제 is a bit of an outlier in Korean: it behaves like English words as "guy" or "man" or "dude" which are (to different degrees) usually a male person, especially when talking about a specific person, but can be applied to both genders (or non-gender-binary person or whoever).
So, for example, you could say:
형제가 있으세요? = Do you have siblings?
저는 형제가 없어요/셋이에요. = I have no/three siblings.
Although they could theoretically mean "Do you have brothers?" or "I have no/three brothers," in most cases, these sentences would include sisters as well.
남매 is more specific, and always refers to male-female sibling relation. Ditto for 자매 (sister-sister relation).
A word of caution: when talking about a specific person, try not to use 형제 or 남매. In many cases, the word 형제 is not used like English "brother": its meaning is closer to "sibling relation". E.g.,
(A) 우리는 형제예요. = We are siblings. (OK)
(B) 저 사람은 제 형제예요. = That person is my brother. (?????)
Although you could be understood, a real Korean will almost surely not say (B), because it's simply not how a Korean sees their siblings. In English, your sibling is either your brother or your sister; in Korean, your sibling is one of your 언니/오빠/형/누나/동생, depending on the gender and relative age.
So, just like an English speaker would almost never say "That person is my younger sibling," (unless they really want to emphasize the relative age), a Korean speaker would not say "저 사람은 제 형제예요," but instead say (for example) "저 사람은 제 (오빠예요/동생이에요)."
For the same reason:
제 형제가 전화했어요. (Wrong)
제 동생이 전화했어요. (OK)
In fact, you can easily speak an hour about your sibling, without ever using the word 형제/남매/자매 once. Just use the "right" word: one of 언니/오빠/누나/형/동생.