It was the ghosts' turn to have their say.
Over the past months their skeletons have been unearthed and some of their bones removed. Building at the Prestwich Street site, where they have lain for so long, has been halted, at least until their descendants decide what to do with them.
Some of their voices, however, are asking to be heard and have persistently been calling to psychic Belinda Silbert recently, she says.
And so it is that on a hot summer's day we're at the site, with permission from the South African Heritage Resources Agency, to hear what the voices have to say.
Silbert steps right in, keeping to the paths, speaking constantly as she relays the information she's getting.
It's not a comfortable place, she says, unlike graveyards that are peaceful and timeless. There's been a lot of pain here, and many souls have not been able to move on.
There are layers of burials, and some people were buried elsewhere and then brought here. Many were buried without dignity.
There's also no uniformity - different areas seem to have different groupings - to the north lie slaves who were Muslim, but were forced to convert to Christianity, and then buried as neither. They want to be buried as Muslims.
Some were just thrown into graves, many died of smallpox. Some, from different religions, want to be separate.
As she walks around she picks up names like Maria Adriana van Vuuren, Anna Februarie, Johan van Rensburg. So many stories. She wants historians to check the names she's come up with.
"There aren't so many old people here," she says.
Anna Februarie was only 17 when she was buried here with her son, Klonkie, the slavemaster's son. The child was not baptised because the slavemaster's wife prevented it, and so Anna is earthbound because she believes Klonkie can't enter heaven and doesn't want to leave him.
While many have passed on, some souls are earthbound and stuck in a timezone.
"They can see me and are confused by me. But they're willing to listen," she says.
"There isn't a nice feeling here," she says, at another spot. "He spoke back to the slavemaster's wife. He was publicly flogged and his back broken on a wheel. He died humiliated because all he had done was ask for water for a second time on a hot day.
"Here's a Johan van Rensburg, and Lettie? Or is it Letta? They aren't from this area or the slave lodge ... they were buried on a farm and later brought here. She doesn't rest."
In the northwest corner she pauses, head down. "I see such darkness here. They didn't die in an easy way. Dampness and a difficult winter... and they weren't dead when they were thrown here."
Maria Adriana, she says, was a Dutchwoman who was an outcast because her father was a slave. "Her mother isn't buried here, she had a very hard life."
Silbert says there are layers of bones on the site. "There are just so many people here."
She stops, as if to listen: "Hannah van Rinsburg, not Rensburg, was very beautiful, but her family discarded her when her face became eaten by the pox. In desperation she was buried on a beach, and later brought here.
"The message she's getting, she says, is that these people are not unhappy they've been unearthed - it's a chance to be acknowledged.
"The majority do not object to their bones being moved, they don't expect this place to be a memorial," she says. "And if their bones are not moved, it's okay - but there should be a ceremony and they'd like to be a symbol to others buried all over the city.
"The city was built on their backs. They don't want to stop progress, but they have an issue with how the land is used. There must be provision for a place for children to play, issues of freedom have to be addressed," she says.
She points to a section where they want a garden, with a plaque, as a testament to them. "They want a ceremony to be done. There has to be honour and dignity.
"The spirits are asking to be laid to rest, and by telling their story this will happen.
"You won't get all the people out. Those deeply embedded want to be left alone, don't want to be hacked out the rock. But the Muslims say that if you take the bones, you must take the entire body - if you cannot take entire skeletons, leave them. They want prayers said for them."
Silbert is confident that if all this is done, there will be a feeling of peace, and this would be a symbol for all the others buried in the area. She's also willing to help the earthbound "cross over".
Maybe then peace will return to the area, she says.
- Silbert's submission will be considered by the South African Heritage Resources Agency in the decision-making process.