During their bath-time at Harrenhal, Brienne thinks Jaime is mocking her. This is his reaction to that:

“Are you thick as a castle wall? That was an apology.”

Leave it to a Lannister to think an insult is an appropriate follow-up when an apology doesn’t land very well.

Only it’s not necessarily the apology that raises her alarm. I think it’s the praise that she takes as mockery.