After succesful attack on players capital who stole my key in a battle, i didnt get my capital key back.

Is it a bug or what?

it may be possible that key retrival has, like the key stealing itself, a probability. keep attacking and see if you get it with 10 attacks in a row on that planet.
(it doesn't matter how much units you send)

Let us know the result.

To steal a key from a player you must have lower score points from him.
No matter how many times you attack, if you have higher points you will never "steal" back your key.
If it is not used in any other way, it will return to you in 3 days.