Exit polls suggest her centre-right Christian Democratic Union has dropped to third behind the centre-left Social Democrats and the anti-immigrant Alternative for Germany, or AfD, party.
The elections were seen as a test for Ms Merkel, coming exactly one year after her decision to open Germany's borders to hundreds of thousands of refugees.
Suppporters of the populist, anti-migrant Alternative for Germany party, or AfD, cheered as they watched results come in from a region of the former East Germany.
The party's main candidate, Leif-Erik Holm, says history is being made after his party became the second most powerful in the state.
"Finally, a real opposition again in Schwerin, and, in a year's time, hopefully in the Bundestag in Berlin. And maybe this is the beginning of the end for the chancellorship of Angela Merkel."
The AfD party was formed three years ago.
Earlier this year, it declared Islam was not part of Germany and called for bans on minarets and burqas.
Its chairwoman, Frauke Petry, says the voters' discontent was clear.
"The established parties have lost people's approval because they fool the people over and over again. We do not have an asylum crisis, we have a problem with illegal immigration. We have a euro crisis because the German government ignores the fact that the European project has died. And it does not look any better if it comes to regional policy objectives. The police force has been budgeted to death. They try to secure freedom and citizens' rights with open borders and confuse freedom with anarchy."
The elections were held in the state of Mecklenburg-Western Pomerania, a small coastal area in north-eastern Germany.
Until re-unification in 1990, it was part of communist East Germany.
The election result is a stinging defeat for Angela Merkel in her home constituency one year ahead of federal elections.
The Chancellor, herself, was in China attending the G20 summit.
Members of her party are calling the result a catastrophe.
The Christian Democratic Union's main candidate, Lorenz Caffier, acknowledges the issue of refugees dominated the elections.
"The positive regional developments that we have achieved here in the state didn't register in the slightest with the population. Instead, there was only one topic, and that topic was refugee policy, even though refugees don't play a real role here politically."
The AfD has increased its presence in state governments across the country to nine out of 16 states.
But despite that success, other parties say they will not form a coalition with it.
The state's premier, Erwin Sellering, had urged voters not to vote for the party.
"Very important now is how we now deal with the AfD. I think that we were right to say, 'Don't waste your vote. Don't waste your vote on people who are just critical and who just spread frustration, but who then in parliament don't want to take up a constructive role.' And now we have to make sure that we push our own political program further forward."
Next September, Germans will vote nationally in parliamentary elections, where Angela Merkel will be seeking a fourth term.
