The problem here was, as the title says, that the two kernels were swapped. Since shader evaluation is only used for building the samling map when World MIS is enabled, rendering without it would still work fine, although baking also was broken.