1) This is a big topic. I will start, bs'd, with a short answer.

The Tosfos Yom Tov (Berachos 5:4) proves that we do not learn Halachos from Agadeta from the Midrash Rabah (Bereshis Rabah 8:12) which cites the dispute between the Rabanan and Rebbi Yochanan ben Berokah regarding whether or not women have the Mitzvah of Piryah v'Rivyah. The Rabanan say that they do not have this Mitzvah, and Rebbi Yochanan says that they do. The Midrash concludes (end of 8:12) that the Halachah follows Rebbi Yochanan, so according to the Midrash women are commanded to bear children.

The dispute between the Rabanan and Rebbi Yochanan ben Berokah is cited in the Mishnah and Talmud Bavli, Yevamos 65b. We see from the conclusion of the Gemara there (65b and 66a) that the Halachah is that women are not obligated to bear children.

The Shulchan Aruch (EH 1:13) rules that a woman is not obligated in Piyrah v'Rivyah.

This is the proof of the Tosfos Yom Tov that if there is a dispute between the Gemara and the Midrash, we rule like the Gemara, because the rule is that we do not learn Halachah from the Midrash.

2) Let's continue with the example given by the above Tosfos Yom Tov.

The discussion there is about the Mishnah in Berachos 34a that the prayer leader does not answer Amen after the Berachah of the Kohanim lest he become confused. If he is the only Kohen he does not recite Birkas Kohanim, but the Mishnah adds that if he is confident that he can recite Birkas Kohanim and resume the prayer, he may do so.

The question now is: may the prayer leader answer Amen after Birkas Kohanim? The Lechem Chamudos #52 (which was written by the author of the Tosfos Yom Tov) on the Rosh to Berachos 5:17 writes that the Heter given if the leader is confidnet he can resume applies only if Birkas Kohanim will have to be missed entirely, but the leader is not allowed to say Amen to Birkas Kohanim even if he is confident he can resume effectively.

However, the Lechem Chamudos then writes that he was shown what the Midrash Rabah (beginning of Parshas Ki Savo) states -- that the prayer leader may not say Amen because he might get confused but "our Rabbis taught us that if he is able to answer Amen without getting confused then he may do so."

The Lechem Chamudos relies on this to permit the leader to say Amen.

The Tosfos Yom Tov cites what he wrote in Lechem Chamudos but then he writes, "I have retracted from this because of the rule that one may not learn Halachah from Midrash Rabos."

So this is a simple reading of the rule that one may not learn Halachah from Agadah; it means that one cannot rule Halachah from what it says in the Midrash Rabah.

The reasoning presumably is that the Midrash was not written with the aim of Halachic accuracy.

3) There is a widely cited statement on this subject in the Chidushei ha'Rashba to Megilah 15a.

The Gemara in Megilah 15a tells us that Esther said to Mordechai, "Just as I was lost from my father's home (orphaned), I am lost to you (as a wife, for now I go willingly to Achashverosh)."

Tosfos there (Megilah 15a, DH k'Shem) asks: Why did Mordechai not divorce her and afterwards he would be allowed to take her back as his wife?

Tosfos answers that if Mordechai would have written a Get, two witnesses would have been required to make the Get effective, but then the word might reach Achashverosh that Esther had been married to Mordechai, which he wanted to avoid. The Chidushei ha'Rashba writes that this answer is not correct because there is another way of divorcing her; Mordechai could have written a Get in his own handwriting which is effective mid'Oraisa.

However, the Rashba concludes: "Ela she'Divrei Agadah Hen, v'Ein Misyashvin Aleihen." We can translate this, "but these are words of Agadah and we do not try to reconcile them."

We see that there is a difference between Tosfos and the Rashba here concerning how much we may apply Halachah to accounts in Agadah. Tosfos considers it a valid question to ask why Mordechai did not give Esther a Get. The Rashba does not consider this a valid question.

According to the Rashba, therefore, one could not prove from the fact that Mordechai did not write a Get in his own handwriting that the Halachah is that such a method is not effective, because we cannot derive Halachah from Agadah accounts.

4) So far I have given examples where we do not learn Halachah from the Midrash. Now, bs'd, I will give an example where we do.

The Midrash (Vayikra Rabah 5:6) relates: "Rebbi Eivu said that there was an incident involving a Shochet from Tzipori who would feed meat to Jews from unslaughtered and Tereifah animals. One time, on the day before Yom Kippur, he ate, drank, became intoxicated and ascended to the rooftop, fell, and died. The dogs began licking his blood. They came and asked Rebbi Chanina, 'Is it permitted to move him away from the dogs?' He said to them that the verse (Shemos 22:30) states, 'You shall be holy men unto me. You shall not eat any flesh that is torn from beasts in the field but you shall cast it to the dogs.' This one robbed the dogs and fed Neveilos and Tereifos to Jews. Leave the body there because the dogs are eating what they deserve."

The Chochmas Shlomo, by Rav Shlomo Kluger zt'l, on the Shulchan Aruch OC 311 cites this Midrash and writes that we learn from this Midrash that one may not desecrate Shabbos or Yom Kippur to bury a Rasha who fed forbidden food to Jews.

The Chochmos Shlomo writes that even though the rule is that we may not learn Halachah from the Midrash, that rule applies only where the Midrash states a Halachah or learns it from a Derash, but if the Midrash cites an incident that actually happened, where Chazal gave a ruling which does not contradict anything written in the Gemara, we certainly can learn Halachah from the Midrash.

5) The only time we cannot learn from Agadah is when we try to use it to decide a dispute in Shas:

I will cite a final source, bs'd, before I close this topic. I hope I have managed to give a taste of this Sugya with the examples I have shown.

The present source is from Tosfos Rebbi Akiva Eiger #36, on Mishnayos Berachos 5:4, on the Tosfos Yom Tov with which we began. He cites the Pri Chadash (in his Sefer Mayim Chayim) who says that the rule that we cannot learn Halachah from Agadah applies only if there was already a dispute in Shas. Then we cannot use the Agadah to decide the Halachah according to a certain opinion. But if a matter was never mentioned in Shas, we may use Agadah to decide what the Psak is. This very much reduces the restriction on learning Halachah from Agadah.

We find a direct practical application of this Pri Chadash in the Shulchan Aruch, EH 119. The discussion there is about a wicked wife. The Rema (EH 119:6) writes that [before the Cherem of Rabeinu Gershom not to divorce a woman against her will] that he may divorce her even if he is not capable now of paying her Kesubah and Nedunyah and afterwards she can claim back from him whatever he owes her.

The Pri Chadash (printed on the page of the Shulchan Aruch) questions the Rema from Yevamos 63b where Rav Chisda says that the verse, "Hash-m put me in the hands of one from who I cannot get up," refers to a wicked wife who has a large Kesubah. We see from there that one cannot divorce a wicked wife without paying the Kesubah. The Be'er Heitev (EH 119:6) answers that one cannot learn from Hagados, as Tosfos Yom Tov (Berachos 5:4) writes.

The Pischei Teshuvah (119:5) answers the question of the Be'er Heitev on the Pri Chadash by noting that the Pri Chadash himself (cited by Rebbi Akiva Eiger mentioned above) writes that this rule applies only where there is a dispute in Shas and the Halachic conclusion is not stated, while, in our case, there is no explicit dispute in Shas about forcing a wicked woman to be divorced, so here we can prove from Rav Chisda in Yevamos that she cannot be divorced without first paying a Kesubah.

Dovid Bloom