How can I get him to back off a little?

Well you get what you ask for. If you don't like him coming over everyday then tell him that. Let him know that he needs to call before he comes over because you might not be home or you might be busy.