Hay has had a tempestuous history, due to its borderlands position almost on the dividing line between England and Wales. In fact, at the time of the Norman conquest it was administered separately as English Hay (the town proper) and Welsh Hay (the countryside to the south and west).
Around 1200 William de Braose II, one of the Norman barons (marcher lords), built a castle here on the site of an earlier one. For the next three-and-a-half centuries Hay changed hands many times. Following the Tudor Acts of Union in 1536 and 1543 it settled down as a market town, and by the 18th century it had become a centre of the flannel trade.