Should this be in wg_format_to_caps_video_h264()? Currently wg_format_to_caps_video_h264 set "stream-format" to "byte-stream". Is it OK to change the "byte-stream" to "avc" in wg_format_to_caps_video_h264()?
Not per se, no.
According to my understanding, there are two variants of H.264 around (well, actually three, but let's set aside AVC3 for the moment).
One is Annex B, which GStreamer calls "byte-stream", and which uses start codes (a fixed sequence of three bytes that identify the start of a packet). The other seems to be usually referred to as "AVC"; it does not use start codes, but instead prefixes packets with their length. Obviously the two are not compatible, but fortunately there is a very simple way to convert between them: prepend an h264parse element.