I think you should just wait and see what he does. It will say a lot about his character on how he handles this. You also have to watch out for him saying that he has dumped his girlfriend but, he never actually followed through with it. He could be planning that you are just going to be his girl on the side. If you do end up together try to meet his parents as soon as possible. If this happens then you know you are the only one.
You may be excited by the prospect of stealing someone’s boyfriend but I am glad to see you feel guilty about it. You never want to be the other women. It never works out and you will always be wondering if he will eventually cheat on you. Trust me I know, I have been there.