Overlooking the city of Tripoli is the imposing Citadel which has been renovated and changed many times during history. Today the castle’s main features are an octagonal Fatimid construction converted to a church by the Crusaders, some Crusader structures of the12th-13th centuries, a number of 14th century Mamluk additions, as well as additions made by the Ottoman in the 16th century. The present state of this huge fortress (140 meters long and 70 meters wide) is largely the result of extensive restoration work by Mustafa Barbar Agha, governor of Tripoli at the beginning of the 19th century.
The first Crusade, called by the Pope Urban II in 905, seized Jerusalem in 1099. The city of Tripoli itself remained in Muslim hands. To maintain a secure foothold and to isolate the town more effectively, Raymond built a fort on a nearby hill, called Mount Peregrinus by the Crusaders; from his fort, later known as Saint Gilles, he began his siege of the city of Tripoli in 1101. The siege lasted nine years, and in the course of its original encampment of St.-Gilles gradually assumed the form of a built-up suburb.

The Citadel of Tripoli was built by Esendemir al-Kurji, governor of Tripoli, in 1308 on the emplacement of the Castle of Saint-Gilles. This Mamluk emir was also responsible for several works of public utility in the city such as public bath and a large market place. When the Mount Pelerin quarter was set ablaze by the Mamluks in 1289, the castle of Saint-Gilles suffered from the holocaust and stood abandoned on the hilltop for the next eighteen years. It was essential to have an adequate stronghold in Tripoli for the Sultan’s troops, temporarily garrisoned in Hisn al-Akrad, as the distance was too great in case of enemy attack. The governor therefore chose the emplacement of the gutted Crusader castle on the hill, incorporating what he could in his citadel, and made use of Roman column shafts and other building material he found nearby. Many of the interior walls, ramps and terraces of the citadel seen today were built in his time.

In 1516 Syria and Egypt fell to the Ottoman Sultan Selim I. His son and successor Suleiman I, called the Magnificent (1520-1566), soon after his accession made an inspection tour of his newly conquered lands. He gathered about him in Damascus all his provincial governors and on this occasion took the decision to rebuild the great citadel of Tripoli. Over the entrance portal, the sultan commemorated this important restoration work with an inscription: " in the name of Allah, it has been decreed by the royal sultan’s order, al Malik al-Muzzafar Sultan Suleiman Shah, son of Sultan Selim Shah, may his orders never cease to be obeyed by the emirs, that this blessed citadel be restored so as to be a fortified stronghold for all time". Its construction was completed in the blessed month of Sha’ban of the year 927 (July 1521).

In the years that followed, various Ottoman governors of Tripoli did restoration work on the Citadel to suit their needs and with time the medieval crenelated battlements were destroyed in order to open sally ports for cannons. Very little of the original Crusader structure has survived until this day. The graves of a number of nameless Franklish knights, here and there, are the only bits of evidence today evocative of their presence on the heights of Tripoli’s "Pilgrim’s Mountain" many centuries ago.