It wont, so long as you don't get pressured into going. Bear in mind, though, once his ward finds out about your existence, they may be inclined to try and drag you into their little cult.
It sounds like he is just going to church to spend more time with his kids? If that's the case I wouldn't even worry about it. If he is a believing member, that's a whole other conversation in terms of what his real expectations and hopes might be for the relationship. If all he does is go to church on Sunday and otherwise doesn't practice I wouldn't think it would affect you that much.
The only thing you might want to be prepared for is people pestering him to have a calling or participate in a service project or visit you or otherwise be involved outside regular church hours. You might meet some Mormons, but they're generally nice enough and hopefully your guy will turn down the constant invitations to get more involved if he's just wanting to be where his kids are.