The Edinburgh side needed a victory against Ferencvaros at Murrayfield and required Feyenoord to do them a favour against Basel in order to qualify, but the scenario did not materialise.
Hearts lost out to a 29th-minute goal from Denes Rosa and never looked like salvaging it in the second half.
John Robertson's side started well and Paul Hartley forced a fine save from Lajos Szucs after good work from Jamie McAllister.
Hartley had the ball in the net after 13 minutes but had clearly handled the ball as he brought it under control and the strike did not count.
McAllister then tested Szucs with a free-kick which the goalkeeper did well to tip over his crossbar.
There was a clear warning for Hearts after 17 minutes when Aleksandar Bajevski presented Marek Penska with an easy chance but he shot weakly wide of target.
A minute later, Michael Stewart sent Hartley clear again but Szucs was again alert to the situation and came out to block the midfielder's shot with his legs.
Hearts were then punished for some slack defending as they fell behind in the 29th minute.
Craig Gordon was left horribly exposed by his defence as Rosa beat the offside trap to accept Peter Lipcsei's pass and, after the goalkeeper twice blocked his efforts, he scored at the third attempt.
It was a poor goal to lose with the Hearts defence looking on and not getting back to help their goalkeeper.
Hearts tried to come back and a 20-yard shot from Mark de Vries took a deflection and was heading for the net until Szucs reacted swiftly to push the ball past for a corner.
The Hungarians remained dangerous on the break and a mistake by Steven Pressley almost let Bajevski in, but Gordon sprinted out of his penalty area to make the crucial intervention.
Stewart came close to equalising for Hearts early in the second half but Szucs beat out his shot and the Hungarians managed to clear before Graham Weir took advantage of the loose ball.
But the Jambos struggled to break down the Hungarians who were well organized at the back and broke the game up with a series of fouls.
Hearts threw on Ramon Pereira in a bid to change the flow of the game but it took a fingertip save from Gordon to prevent Leandro from adding to Ferencvaros' lead.
From the resultant corner, Bajevski saw his flick strike the crossbar and Hearts wilted in the closing stages.