Without codec_data set, mp4mux element fails. And according to my test, setting codec_data to MF_MT_MPEG_SEQUENCE_HEADER is OK, becuase I finally successfully generate a mp4 file which is able to be played by video player.
Yeah, but mp4mux also requires stream-format=avc(3). That's not a coincidence; besides the difference in start codes I mentioned in 3810, the Annex B and AVC formats differ in whether the "codec data" is contained in-band or out-of-band (Annex B has it in-band; AVC has it out-of-band). If you use h264parse to convert Annex B to AVC, it should also supply the correct codec data.
I can't explain why the file gets played back correctly in that case—perhaps GStreamer (or whatever player you're using) detects that the data is actually in the Annex B format and plays it back as such, ignoring the incorrect out-of-band codec data?