You definitely need to say something to him because in a way, if you don't you're leading him on. Because you're not saying no he probably still thinks he has a chance with you and that isn't fair on him really. Just tell him, you don't have to be nasty or anything about it just sit down and explain what you've told us, just say that a) your feelings towards him have changed, b) he went out with your friend so you couldn't get him with anyway because that's just being a **** friend and c) just say you're not even looking for a relationship.
I know you said you don't want to hurt him but sometimes in life you have to do stuff like that, I mean it's better than giving him false hope because that will hurt him more... if you tell him gently and choose your words carefully then he should be fine. If you find it tough speaking face to face or you think its gonna be awkward and you'll chicken out, just send him a text/facebook message like the user above suggested. But also, he sounds like a bit of a **** if he came running back to you after he split up with his ex (which happens to be your friend) and anyway, why did your friend get with him knowing you went out with each other? that's a bit of a **** friend, but anyway good luck to you!