Counts from other lineages came into power in The Hague, but not all lived in the castle of The Hague. One of the most significant counts of The Hague and Holland was Aalbrecht of Bavaria, who brought prosperity to Holland through his wise decisions. Aalbrecht waged a successful war against the city of Delft and ordered the people to raze its city walls and bring the stones back to The Hague castle to build an imposing wall. The Spui gate of this wall survived until 1861.

After Aalbrecht’s death, civil war erupted in Holland, with Holland becoming part of the Spanish Empire. During the late 13th Century, the castle looked about the same as it does today, with exception to the old castle’s thick walls, stables, gardens and a church. The large garden behind the castle became a public square in 1632. The Outer Court, today also a square, was located in front of the Inner Court. Before the 1920s, it was encircled by military buildings and stables, and could only be entered through the Prison Gate and the small street Halstreat.

Holland, with The Hague as its capital city, was ruled by foreign counts and kings between the 14th and 16th Centuries. King Philip II of Spain was the last foreign ruler. The locals were not too concerned about who ruled the city, evident from their saying ‘God is too high, and the king is too far’. Moreover, other cities, namely Dordrecht and Leiden, still wielded much power. Taxes were paid and there was peace. Representatives of the king lived in The Hague, as well as the administration.

Cities, such as Delft and Leiden, were in danger of potential rivalry with The Hague, so attempts were made to prevent it from building fortifications. Nevertheless, The Hague became rich from commerce, and in the 17th Century, the only European ships allowed to trade with Japan were Dutch.