Dieppe emerged as a fishing village in the 11th Century. It was bitterly fought over during the Hundred Years' War in the 14th-15th Centuries. In the 16th Century, it housed France's leading school of cartography. As Louis XIV revoked the Edict of Nantes in 1685, ordering the destruction of Huguenot churches and the closing of Protestant schools, some 3000 Huguenot citizens left Dieppe and moved abroad. Anglo-Dutch naval bombardment nearly leveled the town in 1694 but it was soon rebuilt and gained popularity as a seaside resort thanks to its attractive beach. Dieppe saw an unfortunate Allied attack known as Operation Jubilee in August 1942. Intended to seize the important port on the Channel, gather intelligence and lure the German air forces into a large planned battle, the raid proved a disaster, and more than two thirds of the landing infantry, predominantly Canadians, were killed, wounded or captured. It was again Canadian forces that liberated Dieppe in September 1944.

Luckily, three fine architectural monuments survived the 1694 bombardment: the castle and the churches of St. Jacques and St. Remy.

The Castle's early history is subject to controversies. Its oldest part, the Western Tower, dates back to the 14th century. Four other towers were built in the 15th Century. In 1589, Henri IV ordered the construction of a circular enclosure to strengthen the fortress. It served as the governors' home, fortunately escaped demolition after the French Revolution, and was finally transformed into a museum.

The castle has a total of sixteen showrooms. It features a rich maritime collection reflecting the town's centuries-long seafaring traditions: prints, charts, paintings, sketches, weapons and models. A whole room is dedicated to the life of the famous naval officer Marquis Abraham Duquesne, born in Dieppe, who fought valiantly the Spanish and Mediterranean pirates.