
They started marching from in front of the Government House, an area they have they have occupied since last week, to Prem's residence, which is on Sri Ayutthaya road, about two kilometres from the Government House.
The protesters then set up stages and the leaders took turns to speak through megaphones criticising Prem.
Dozens of soldiers and police have been deployed at Prem's residence since last night, Police have stepped up security at the Privy Councillor's residence, with concrete blocks, barbed wire and fire brigade trucks in place to prevent protester access.
More troops were deployed to the area on Wednesday morning after the red shirts announced that their target of the protest would be Prem.
Fugitive ex-Prime Minister Thaksin Shinawatra has repeatedly accused Prem and other privy councillors of being behind the coup that toppled him from the premiership in September 2006.
It is reported that Prem is inside his residence.
The protesters, known as the United Front for Democracy against Dictatorship (UDD), hoped that tens of thousands of their supporters from the provinces and metropolitan Bangkok will join what they claim is a demonstration larger than any previously held in Bangkok.
The event is focused on three locations - Government House, the Royal Plaza and Prem's residence.
Earlier in the morning, about 1,000 red-clad supporters of ousted former premier Thaksin Shinawatra gathered at Metropolitan Police headquarters, waiting for the demonstrators to march from the Government House to the home of the Privy Council President.
Thousands of red-clad demonstrators have been rallying at the Government House since March 26, demanding Prime Minister Abhisit Vejjajiva step down, and calling for fresh elections.
On Tuesday, Prime Minister Abhisit Vejjajiva said his government may have to take drastic action against protesters if their mass protest in Bangkok turned violent.
He said his government is trying to avoid confrontation with the protesters demonstrating at the gates of his offices. They have now been blocked the entrances for 12 days - since March 26.
The government has been most lenient and will not initiate any violence against the UDD protesters, but "drastic action may have to be used", which if taken, will be intended to have "minimal losses or no loss at all," said the prime minister.
The prime minister said he did not believe special laws such as using an emergency decree would enable state officials to do more than under normal circumstances.