RPH, Steam, and presumably the social club launcher (or whatever it is that launches the retail version) will always use the folder named 'Grand Theft Auto V', so if you want to play on your modded copy, rename the unmodded one to something else, and make sure the modded game folder is called 'Grand Theft Auto V'.
Example: if I want to play Online, I'll add 'SP' at the end of the currently used (modded) folder, and remove the MP bit from the unmodded one so it'll use that one