In the modern MMO scene, group dynamics in most games is ruled by predefined roles, most often tank, healer, and DPS (damage dealer). This system gets a lot of criticism, but a look at the industry shows that it works for WoW, if not necessarily for a lot of other games. What I'm going to look at is the benefits, the drawbacks, the flaws in most proposed "solutions", and an idea of what I think good solutions could be.
For the sake of keeping discussion focused, this article specifically addresses semi-traditional MMOs (WoW, Everquest, Aion, etc.). Games like Eve are so different in most ways that discussing them is more likely to just take us off track.
At the core, the main benefit to using this system is to ease PuGs (Pick-up groups. In other words, playing with random strangers online). In an MMO that adheres to strict character roles, you can group with a few random people, and without a word of discussion, everybody knows what they're responsibilities are. You can do a difficult dungeon flawlessly without saying a single word to any other player.
Further, this allows you to roughly present your value to a team in a single number. "I deal 6000 DPS". "I have 5300 Gear Score".
Compare that to a more complex system. "I'm tough enough to take hits from 1-2 enemies, I can keep one enemy turned into a sheep, I set myself up to run faster so I can distract bad guys better, and I can call my Gnomish Battle Chicken when I need some extra power". Try forming an effective group from random people who describe themselves like this, and you've got a nice challenge cut out for you.
The big, obvious drawback, of course, is that it's shallow, but I want to be more precise than that. Specifically, it elminates decision-making from battles. Even complex battles such as raid bosses do not involve decision making because there's always a definite best solution. It's very rare that you actually strategize during a fight, rather than simply having everybody fulfill the pre-defined set of steps to make it through successfully.
The other major drawback is that this system doesn't support stylized characters. Once you've chosen your role, there are definite best stats, best equipment, etc. Any attempt to make a character that plays differently from others will almost always result in an inferior character.
Weak Solutions: Most conversations I see about this get the same solution offered: Add more roles. People will recommend adding a Crowd Control character or a Buffing character, but there's a reason why WoW moved away from that (Crowd Control used to be important). It's because it provided the worst of both worlds. If there are 5 mandatory roles instead of 3, it's hard to put a good group together. However, there is still no real decision-making during the fight. It's still following routine, but now the routine is harder to follow and getting a group together involves a lot more time hanging out in cities asking people to come.
More Effective Solutions: First off, you have to accept that if you're moving away from standard MMO roles, you're going to lose the benefits mentioned above (which are substantial). You're going for a different audience, and that may or may not be what you want for your game.
If you have decided that it's worth the cost, then the first two steps are to change the aggro system and change the class design. Classes must be created able to accomplish more than one thing and Aggro should have some trade-offs in its manipulation and possibly be more difficult to manipulate. Basically, either it has to be impossible to always keep aggro on the tank and still play effectively or it has to be a better choice to mix up who's being attacked.
The best example I know of with changing this while still maintaining a traditional fantasy MMO is the Guild Wars series. Guild Wars 1 had an aggro system that is very similar to what you expect from PvP (enemies prefer weak characters and high damage dealers. Healing creates much more aggro but only if they're attacking the person getting healed). Guild Wars 2 is breaking the traditional roles further, though I'd recommend the articles on the official website if you want more info there (too long to explain here).
Consider a "Glass Cannon" character under these rules. Instead of just constantly blasting the enemy as much as they can, they would look for targets of opportunity. They would prefer to attack an enemy who is almost dead, so that by the time they switched targets, they'd be finished. Likewise, if the game didn't have a "Taunt" ability, they would be some of the best at rescuing endangered party members because their damage output would be great for forcing a target switch. You would have to constantly look at the battle and make decisions instead of just firing away.
Moving to these types of systems is ideal for people who regularly play together, as there's much more room for ingenuity, and you learn to play off of each others' strengths. However, they're pretty harsh for people trying to play with strangers, as everybody will have a different idea of how the group should work, and they'll have a less clear picture of what their allies are capable of.
Too often, people either copy mechanics without considering why they're there or criticize mechanics without fully understanding the benefits or design decisions that brought them about. While I'm all about innovation (and I'm personally bored of routine MMO battles), it's important to understand both sides of each major mechanic before deciding whether you want to use it.