If an order is returned then the balance for the relevant affiliate is reduced by that amount.
When we calculate the balance we only take into account "fulfilled" and "unfulfilled" (not yet shipped) orders. All other order statuses are ignored for affiliate commission balances.
There is an edge case to be aware of: If you pay an affiliate for a given month and then an order is returned the following month, then affiliates may be overpaid. This is because the calculation is made by individual order over a given time period. We do not currently create negative balances looking forward after an affiliate payout has been initiated.