This is true of lots of foods. What tends to be the case is that, when times are bad and the unemployed need to subsist directly on their own labour, it makes sense for them to do things like go oystering because time is something they have, but when times are better, catching oysters is time consuming so the food is expensive.