Horsemen are the Cavalry of the Feudal Age, replacing Knights. Similar to Knights, they deal very high damage and have high speed and defence, but they also have a high price. They are best used by prosperous nations fighting infantry, as they strike before them but cost more.
In the Feudal Age, cavalry come into their own as able and efficient units, being much better in upkeep efficiency, stats per unit and ease of use; and retaining their high first-strike bonus against infantry.
It is important to remember that all advanced specialist and anti-unit roles in an army pay for their overall effectiveness, taking into account armament and armour equally to response times and manoeverability. With Cavalry this becomes especially important, as most of the unit cost is spent on these bonuses.
Per unit, they are cheaper to keep; 1.5x to 4x in the Classical Age. In gold efficiency, this is 0.67 g/a compared to 0.86 g/a.
However, proportionally their defense is still as low as the Knights, barely above 1 infantry attack for higher cost; and as their real-life counterpart, this makes Cavalry weak to the cheaper pikes and polearms of infantry. As they hit first, but get hit first as well, Cavalry are best used in numbers to crush through an attack force and neutralise this advantage.
They are best used as scouts or as a way to clear out any remaining units in a resource tile after having sent a larger army. For those that can afford to mass them in numbers, they can be a good infantry-killing army themselves.