In a Korean book, I read a sentence like this:
최선을 다해 노력하는 사람이 성공하는 법이에요
In the phrase "다해 노력하는", I think the author used the grammar 아/어서 (다하다 + 아/어서 + 노력하다). Can I replace this with "다하려고 노력하다"? Thank you!
Your analysis of the structure is correct, but changing it to 다하려고 노력하는 will make it worse if anything.
-아/어서 and -려고 are distinctly different verb phrases. -아/어서 can express a reason or background information (피곤해서 일찍 잤다), or a means or manner of the action of the subsequent verb (경보를 울려서 위험을 알렸다 = Alerted people to the danger by sounding the alarm). It can also mean the first stage of a two-stage action (무를 썰어서 무 무침을 만들었다), but even in this case the actions are closely connected).
-려고 always indicates an intention or purpose of the action that follows, so it is usually not interchangeable with -아/어서. The -아/어서 verb happens before or at the same time as the other verb, whereas -려고 verb cannot have already occurred since it is the objective of the following verb.
To me, even though both are grammatically fine and might even sound alike, #2 is a little redundant because it is saying they're exerting their efforts into "doing their best". #1 is the more natural expression showing the manner ("doing their best") in which they are trying something. In the broad sense, though, #2 can also be used, just not as common as #1.
The meaning would change a bit if you'd replace it like that.
You're right about the grammar used; it's -아/어(서).
최선을 다해 노력하는 사람
Someone who does their very best ("while making an effort", which is superfluous/unnatural in English)
최선을 다하려고 노력하는 사람
Someone who [makes an effort/tries] to do their best