CFLAGS=-m32 doesn't seem ideal, as that implies dropping "-g -O2". I'd suggest CC="gcc -m32", or perhaps something like CFLAGS="-m32 -g -O2".
Does -g matter for a CI build?
Potentially, when attempting to debug test crashes, I'd say. More generally though, I don't think it matters less for the 32-bit build than it does for the 64-bit build; if we're convinced that we don't want debug information and are deliberately dropping those flags, we should probably drop them for the 64-bit build as well.