The Ndifuna Ukwazi Centre, which is assisting the 27 residents, said in a statement on Wednesday that the recusal application would be argued on May 17 and 18.
The families had been living in their Bromwell street houses for generations when they were served with eviction notices in March 2016.
The City of Cape Town's plan to relocate them after the houses were sold for development was met with strong resistance and accusations of apartheid style forced removals.
The residents turned to the Western Cape High Court to stay the eviction and have rejected the City of Cape Town's proposal to move them to Wolwerivier.
The Ndifuna Ukwazi press statement said the evictions would leave them homeless.
"On 20 September 2016, they brought an application before the Western Cape High Court to compel the City of Cape Town to provide them with temporary alternative accommodation in a location as near as possible to their home community of Woodstock. The City proposes to relocate the residents to the Wolwerivier relocation area, some 29 kms north of Woodstock."
It further states that residents believe Acting Judge Weinkove "impugned their dignity and violated their right to fair access to the courts".
Weinkove reportedly made a number of "disparaging remarks": "These included insinuations that unemployed people do not need to be close to opportunities and services". "Where judges make comments which degrade a person because of their socio-economic and employment status this adds a more insidious barrier to access to the courts. It serves to further exclude and alienate the most vulnerable persons in our society who often are the most in need of the court’s assistance," the statement read.
Attorney for the Ndifuna Ukwazi Law Centre, Disha Govender, said the matter was argued on January 31 and February 1.
She said that Weinkove reserved judgment, but in the interim he required additional information from both the residents and the City of Cape Town.
The matter will return to court on May 17 and 18 where the points Weinkove raised will be argued as well as arguments for the recusal application.