David Atkins uses the centre which is currently based at North Cambs Hospital.
In a letter to the Citizen, Mr Atkins said: "The Bowthorpe Centre has for a long time offered Wisbech mental health sufferers a refuge and a safe haven from a society which is often uncaring and dismissive of them. Now it is under threat.
"A move from its current premises has been used as an opportunity for the management to introduce wholesale changes to the way the centre operates, along with government directives to move sufferers out into the community. Reintegration into the community is a valid objective but it does not suit everyone. Many will continue to need the safe haven."
Mr Atkins claims drop-in times have been severely cuts and groups who meet at the centre have been told to find other premises.
He also said the loss of the garden will be a particular blow, as sufferers meet there and chat to others with similar problems.
Mr Atkins added: "Future generations of Wisbech sufferers will be denied the caring, compassionate services provided in the past, all in the name of 'back to the community' - the very community which has misunderstood and failed to help mental health sufferers in the first place.
"Those responsible for those changes, from uncaring Government agencies to the centre management, should be ashamed of themselves."
Kevin Tunnard, Operations Manager for the Richmond Fellowship who run the Bowthorpe Centre, said Mr Atkins had some valid points but he did not want service users to see the move in a negative light.
He said: "People who are dealing with mental health issues need to be out in the community and supported in the community, not in hospital settings. Obviously, when people need hospitalisation, we will do it, but there are a lot of issues with people being stigmatised. If they are hidden away, they are not dealt with. Society needs to come to terms with mental health issues."
Mr Tunnard admitted the move was driven by the government, but was not a cost-cutting exercise. He said they had in fact been allocated more funding from Cambridgeshire County Council to make it happen.
A new site has not been confirmed for the Bowthorpe Centre but the Richmond Fellowship want to keep it close to Wisbech town centre, as North Cambs Hospital is, so it will not be more difficult to access.
Mr Tunnard said the town centre location meant it was a challenge to find premises that would work for the centre and unfortunately no site they had found could provide a garden. He stressed the Fellowship worked with service users and got their thoughts on potential sites.
He said: "We accept this will cause stress because we are asking people to go through a period of change and some people find it more stressful than others."
Mr Tunnard said the traditional day service did not work for many people and they wanted to tailor services to people's individual needs and interests.