Every player has the choice to either join VvV or not join VvV. If you opt not to do so, but insist on still being involved in a PvP battle - whether that be through direct combat engagement, healing/buffing, or merely just standing around taking advantage of Felucca push-through rules, you are now subject to the same consequences. The single biggest point of feedback when we set out on the VvV discussion nearly a year before we even started implementing was, "Fix the blue healers." As it stands now everyone who wishes to engage in PvP that has opted into VvV is on a level playing field. Stat loss effects those who wish to participate - one way or the other - evenly.
In your example scenario - if you are standing next to me in a non-consenual ruleset, I'm in VvV and you start casting on me, then attack me, the onus of situational awareness is on me to defend. Whether I respond in kind to the attack or get steam rolled relies on my innate skill - regardless of character stats, skills, and equipment.