Killing a LOT of animals in a short time, even with the sacrifice, seems to be a tipping factor. Cutting down a lot of trees, also. However, the Spirits seem to be very forgiving and it never has taken long before bad messages like 'the forest looks frightening' ceased.
When you build, cut down all the trees you need. Then the time spent building with them should get you past the 'angry spirit' period. If you've been very lucky hunting/trapping, well, I guess you have to take the bad with the good, eh?
Oh, and one usually gets that quest after achieving a very good opinion from the Spirit. Maybe you've taken a long time getting around to it and some of your actions have affected their opinions of you?