The game sprang into life just before the half-hour mark when a 30-yard wonder strike from Saulius Mikoliunas dipped under the crossbar in the swirling wind and put Hearts into the lead.
Celtic came back and Jan Vennegoor of Hesselink came close five minutes later when Craig Gordon stumbled but Christophe Berra cleared his shot off the line.
Just as it looked as if Celtic would dominate the closing minutes of the half, Andy Driver made a break to the other end.
His shot was cleared by Artur Boruc but the ball fell to Roman Bednar who thought that his effort had gone over the line but was denied by the linesman.
In first-half injury-time Maciej Zurawski tried his luck in front of goal but his powerful volley was well saved by Gordon.
The second half started at the same fast pace and Celtic took the early initiative with Gordon denying Vennegoor of Hesselink and Panagiotis Fyssas doing well to clear Shunsuke Nakamura's follow up off the line.
The visitors' pressure paid off on 59 minutes when Gordon failed to hold on to a Craig Beattie cross and Vennegoor of Hesselink was on hand to tap in from inside the six-yard box.
The Dutchman was the hero at the other end minutes later as he was perfectly placed to block Fyssas' effort.
Ten minutes from time, with Celtic continuing to pressure the Hearts goal, Jiri Jarosik found the back of the net for the visitors to snatch victory with a spectacular 20-yard strike.