You can only decline a swap when it is shown under the "Awaiting Your Response" tab.
If the swap is shown under the "Awaiting Other Party's Response" tab, you cannot decline it.
Make sure to check the details of swaps carefully before making any requests or responses.