The castle of Porto Palermo was built by Ali Pasha Tepelena in 1804. At this time this ruler was distinguished for the reconstruction and construction of fortifications, he created a whole system of castles and separate towers as predicting a possible conflict with Turkey. The castles are built on important strategic points, close to the main roads and ports of the country, the position of building the castle of Panormi clearly speaks by itself.
In his military constructions, Ali Pasha used a large number of Italian and French architects and engineers, but his work from 1800 to 1812 was chaired by Petro Korçari, a stone master from Korça. The castle has a triangular shape, with three bastions in its corners, built on the roof and two bastions built in the western courtyard of the castle. On the terrace parapet there are turret for balls.
This castle is covered in all its surface with stone arc. Inside there are halls, storage rooms and numerous environments. In the central part there is a pair of stone stairs leading to the terrace. The main entrance is protected, it is located between one of the ribs, between the two bastions. From inside it is controlled by a flip-flop wall of a clutter, also a turret wall is built on the inner side facing the entrance. On the outer side of the entrance gate comes a console of a hot stone balcony for the time at which it was built had decorative purpose.
An inscription was embedded in its doorway, but it was destroyed in 1995. Something not too many people know is that the castle was built on top of the ruins of St. Nicola’s monastery and in respect of religion and the locals he built a church dedicated to Aghios Nikolaos at the entrance of the island.
Ali Pasha's Castle carries an old story. Ali Pasha built this castle for his wife Vasiliki (a young woman from the villages of Himara). It is said that her treasure was buried under the catacombs and after the exile of Ali Pasha and his wife, the treasure was not found, indicating that the legend was more of a folk tale.