But he’s not ordinary by any measure.
Well known as an anti-apartheid activist and respected high court judge, he is also a gardener, cook and an avid golfer.
While politics and his role in the Struggle was all he wanted to chat about, he also revealed his love for family, music, sports and plants.
“Ever since I retired seven years ago, I find myself having more time to do the things I would not necessarily have had the chance to do,” said Pillay, who recently turned 82.
Living in a luxurious three-bedroom flat with his wife, Dolly, Pillay’s home encompasses everything he holds dear. As you walk in, you are immediately greeted with a breathtaking view of the suburb and a wooden bookshelf boasting rows of his favourite reads. A canvassed family picture also hangs on a soft-beige wall while more family pictures are on display next to the dinner table. A spotless, dark wood kitchen, the sunlit terrace filled with pot plants and a cozy lounge area is where he loves to spend most of his time.
“I have three children and six grandchildren, two of whom are in Durban, and they keep me very occupied, I’ll have you know,” he gushed.
His grandchildren, Saurav Matai, 13, a Northwood pupil, and Sahil, 11, from Chelsea Preparatory School, are involved in sports and their proud granddad attends as many cricket, hockey and soccer matches as he can. He is also a whizz in the kitchen, he playfully boasts, adding that Saurav and Sahil loves his crab curry and lamb shank.
“I started cooking after I retired and have quite a collection of recipes. Crab and lamb shank are two of my favourite dishes to make and my grandchildren love it. They say my crab curry is the best.”
Sahil, who was standing close by, nodded in agreement. “My granddad’s crab is tasty. It’s my favourite.”
Pillay is also fond of writing and has made much progress penning his memoirs in the form of letters to his grandchildren.
It currently sits at 300 pages and with 44 chapters it will include stories on his family, early political life, Struggle days and his thoughts on the country.
“It has coverage of my former glory days, my human rights work and the various protest meetings and rallies I addressed. Protests that eventually led to my banning.”
He hopes his life story will be published. “Maybe when I’m no longer around, but we’ll see.”
Pillay said a typical day started off with a cup of tea in one hand and the newspaper in the other.
“I’m fanatical about reading the newspapers and watching the news in the morning.”
He is also a bookworm. He is currently reading Purple Hibiscus by Nigerian author Chimamanda Ngozi Adichie on post-colonial Nigeria. It centres around 15-year-old Kambili Achike, a member of a wealthy family dominated by her devoutly Catholic father.
“I also try to gym at least three times a week and enjoy listening to music.”
He often attends the jazz sessions at the University of KwaZulu-Natal on Wednesday evenings.
But gardening, he said, is one of his greatest passions.
“I had a huge house in Greenwood Park with a magnificent garden, but unfortunately we had to downsize as it was just my wife and I. Now I have a little herb garden with lettuce, mint, thyme, rosemary, parsley and curry leaves,” said Pillay as he showed off his perfectly manicured plants, neatly sectioned into categories, at the edge of his terrace.
He also supervises the residential gardeners and helps with the selection of plants bedded around the flat.
Pillay, the former president of the Durban Golf Club for three years during the 1980s, said he enjoyed playing a round of golf with his four chums, three of whom he matriculated with - Vivian Persadh, Rathilal Gordhan and Willie Soupan.
Asked if he had ever considered emigrating, he shook his head.
“We made a decision as a family to never leave. We know our place is here in South Africa. Lots of family have left but we took a principled step not to emigrate. I wanted my children to be educated here.”
His children have stayed true to this decision.
“My eldest son, Udesh, was the deputy chief executive officer at the Human Sciences Research Council. My first daughter, Nirvana, who is currently doing her PhD, is a qualified industrial psychologist and research analyst, and my youngest, Sue, is an attorney.”
Pillay said he stayed in the country despite the severe restrictions on him, which almost prevented him from attending his own wedding.
“My wedding took place at a hall in Isipingo in August 1965. At the time I was banned from attending any public gatherings but was granted permission from the court to attend only the ceremony and then leave.”
Asked about whether his wife was upset about this, he laughed.
“She knew what she was getting into when she agreed to marry me.”
Despite his retirement, Pillay still believes in implementing change in the country and as a result is involved in many civil society movements. This includes the Active Citizens Movement, which was launched last year to develop a society free of corruption as well as hold government and public figures to account.