The village was first settled during Roman times, between the I and IV centuries. The Romans developed a fish conservation industry in the area and the resulting product was very highly valued in Rome.

After this period, the region was occupied between the IV and VIII centuries by the Visigoths, who gave way to the Moors (VIII Century). Prince Omeya Abd-el-Rhaman settled in Torrox and used it as a staging post to found the Independent Emirate in Cordoba.

Later, there was discontent among the Mozarab population in Torrox, and at the end of the IX Century they joined the Mozarab rebellion led by Omar ben Hafsun against the Caliphate of Córdoba. Caliph Abd el-Rahman III laid siege to Torrox Castle in 914, defeated the rebels, took prisoners and burnt the ships that had come to the coast to help the besieged, leaving Torrox without a Mozarab population. The area had a completely Muslim population by the XI Century, ruled by the Taha of Frigiliana.

During the Nazarite period in the XIV and XV Centuries, Torrox was strongly influenced by the silk trade. It became a silk collection centre to meet the demand from Málaga and Granada, where it was a highly sought-after product. The Nazarites also grew nuts and sugar cane in the region.

The reconquest of Torrox by the Catholic Kings happened on the 29 April 1487, but this victory was not consolidated until the following year, as both the village itself and Nerja were re-conquered by El Zagal.