I'm not 100% sure as I raise honor solo, but I'm afraid it's the case.
What I'm 100% sure is that when I peace what I honor I drastically lower the honor I get.Way more than invis to give you an idea.
Area peace doesn't have that drawback though so you can use that as much as needed, but direct peace is a bad idea