If someone’s temper or nerves fray, or if something frays them, they become annoyed.