"A momentary reaction, I assure you. I am mistrustful of many with odd abilities," a lie, but it's better than getting specific and saying 'your possibly demonic ones'.

And yes, the fear about her in specific is diminishing now that he's determining that he can't attack her. And she's attacked no one that he's heard of, which means Oliver is considering his life safe.

Fear of failing his God? Guilt at said failure? Frustration at the position he's been put in? Those are still continuing.