Over 93% percent of voters cast ballots opposing the £3.5bn deal to compensate the British and the Dutch state, initial results showed.
"This result is no surprise," Prime Minister Johanna Sigurdardottir said.
"Now we must turn to the task of finishing the negotiations on Icesave."
The propsed deal would require each person to pay around £90 a month for eight years, and many Icelandic taxpayers say they cannot afford it.
However, the rejection of the deal could jeopardise Iceland's credit ratings, making it harder to access much-needed funding to fuel an economic recovery.
The island is grappling with a 9% unemployment rate, a 7% annual inflation rate and an economy that is still shrinking.
It was unclear how many of the 230,000 registered voters cast a ballot, but public television reported a turnout rate of around 55% an hour before polls closed.
Meanwhile, around 1,000 protesters gathered in downtown Reykjavik to demand a better say over the plans.
Last-minute talks broke down this week, despite Britain and the UK saying they had offered more favourable terms, including a significant cut on the interest rate in the original deal.
Despite stalled talks and the rejection in the referendum, Iceland's government said it still believed it could strike a deal.
"The government of Iceland is confident that a solution acceptable to all parties can be achieved," it said in a statement.