He never told us - my brothers, sisters and me - that we were not allowed to swim at the other beaches because, I suppose, he was trying to protect us against the realities of apartheid.
Or maybe, like so many people at the time, he just found it difficult to explain what apartheid was about.
It was customary for working-class people to go to the beach during the festive season, but for many reasons, mainly economic, most poor people, including my parents, could only go to the beach once or twice during this period.
The popular days were Boxing Day and Tweede Nuwe Jaar.
We ended up at beaches such as Kalk Bay, Soetwater, Mnandi, Strandfontein, Harmony Park, Long Beach in Simon's Town and Maiden’s Cove, Clifton.
Beaches like the rest of Clifton, Gordon’s Bay, the Strand, Camps Bay and others were out of bounds for people like me, who were classified non-white under apartheid.
Non-white: what a disgusting term to describe people, but maybe I will deal with that at a later stage.
Years later, I participated in protests at beaches reserved for whites, insisting that all God’s beaches should be for all God’s children.
Granted, we already saw the dead signals for apartheid, even though the regime fought hard to maintain this vile and inhuman system.
Our beaches - or rather the beaches we were not allowed to frequent - became sites of struggle during that period, leading to the Nationalist Party eventually agreeing to open all beaches to everyone.
I thought about this this week because of the reports of people being told by private security to leave Clifton beach at 8 pm on December 23, apparently on the instructions of the City of Cape Town, which has now denied giving such an instruction.
I felt angry that 24 years into our democracy there are still people trying to implement divisive apartheid tactics in order to keep black and poor people from enjoying the natural beauty of our city.
It is possible that the City did not give an instruction to the private security company.
It is likely that they were acting on behalf of some of the rich people who live in Clifton and who think that they deserve to have privacy on “their” beaches.
They probably do not realise how sensitive we still are about being deprived of using these beaches until apartheid was on its deathbed.
To this day, I still sometimes get angry when I visit Gordon’s Bay or Fish Hoek, two of my favourite beaches, and realise that the first time I could go there legally was when I was already over 30 years old.
I grew up in Cape Town but could not go to some of its beaches until I was already married with two children.
If it is true that residents of Clifton instructed the private security to remove people from the beach at 8pm, then they need to remember that the beaches do not belong to them.
In fact, it is highly possible that the property they are living in used to belong to a so-called coloured family who was forcibly removed from the area under the Group Areas Act.
Sometimes I deal with white people who ask me why I cannot forget about what happened under apartheid.
They would never ask Jewish people to forget what happened during the Holocaust.
But when I hear reports like what reportedly happened this week - blatant racist and economic discrimination - it brings back all the worst memories of apartheid.
We should not allow anyone to go back there, especially not in a divided city such as Cape Town.
* Fisher is an independent media professional. Follow him on Twitter: @rylandfisher
** The view expressed here are not necessarily those of Independent Media.