The first most obvious alternative is
Pathfinder
which is mostly the same rule set as DnD 3.5e but has a completely different world. I've only played first edition pathfinder which is very customizable but also very crunchy and can give too many options. 2nd Edition came out and is apparently a lot more streamlined, though it's also new so there's fewer resources out there for it.Another game I've played recently is
Deadlands: Reloaded
which is based off the Savage Worlds
system which has a ton of settings in the real of alternate history w/ real monsters. The setting of the old west was fun and the system is pretty simple and has support on Roll20. Also, exploding dice (i.e if you roll the highest number you roll another dice) are a major mechanic and are sooooo fun to deal with.There's also always
Call of Cthulu
though it's very far from normal DnD. Basically you guide your characters through a Lovecraftian mystery and it uses a D100 system for skill checks. Also instead of your characters getting stronger over time, your characters slowly lose sanity and luck, which vastly increases the odds of character death. The rules are also relatively easy to pick up so long as you read the rulebook, sometimes even easier than DnD with skill checks since there's only 3 levels of success (Normal, Hard and Extreme) and the number you need is just determined by your points in the relevant skill. A system very few have probably heard of is
Of Dreams and Magic
where is very customizable. The basic premise is magic is real but it's all derived from "dreamers" and dreams. Every time you use magic to solve a problem you increase the chance a monster from one of the dream worlds will enter the real world and cause a huuuge problem. The biggest issue is the rules are a bit harder to wrap your head around and combat is...not great. Definitely a lot better for a mystery based campaign. I've also played
Monster of the Week
which uses the Powered by the Apocalypse engine and basically allows you play out episodes of Supernatural "Monster of the Week" shows. It's very rules light, but you select your character archetype from the different types of characters in those story (Stuff like "The Chosen One" or "The Spooky") and while the flavoring is all up to you the rules also restrict the type of actions you can do, so a lot of working with the DM is necessary for figuring out what happens/what you can do.Last but certainly not least is
The One Ring
, a system I can not speak highly enough of. As you might guess, it's a campaign set in Middle Earth in the world of Tolkien. If you like LOTR you will like this game. You can play as a lot of different races from LOTR and more material is coming out. I believe my group is playing second edition, which is relatively new but does seem to have plenty of support in terms of being able to run it (all of the games I've mentioned my group has played on Roll20 btw, great website for this). Also while the skill system is quite different from DnD, it's not that hard to figure out, since once again the DC is determined by your character sheet, though the DM can make things harder by letting you roll fewer dice/roll with disadvantage. It also focuses a lot on travel time (cause Tolkien) and Shadow is a major mechanic where your character can just, turn evil if you aren't careful. It's really amazing.