The court ruled that the 550-seat parliament should have convened with at least 367 deputies -- or the two-thirds majority needed for a president to be elected in the first round -- for the voting to have begun, the court's vice president Hasim Kilic told reporters.
According to parliamentary minutes, 361 lawmakers were present in the assembly on Friday.
The petition challenging the procedure was filed by the main opposition party, the social-democratic Republican People's Party.
"It is now up to parliament to decide how to proceed," Kilic said, adding that parliament could begin the election process anew.
Early elections urged
Gul urged speedy early legislative polls if parliament fails to elect him in a repeated vote.