https://bugs.winehq.org/show_bug.cgi?id=45151
Bug ID: 45151 Summary: patches: 'git send-email' through Zoho SMTP causes patch queue to go berserk Product: WineHQ.org Version: unspecified Hardware: x86-64 OS: Linux Status: UNCONFIRMED Severity: normal Priority: P2 Component: www-unknown Assignee: wine-bugs@winehq.org Reporter: kyle.auble@zoho.com Distribution: ---
Created attachment 61339 --> https://bugs.winehq.org/attachment.cgi?id=61339 I'm starting to get embarrassed
So I'll probably just start using my Gmail account for submissions to Wine anyways, but I've been having problems sending patches through Zoho.
I've followed the instructions on https://wiki.winehq.org/Submitting_Patches and when running 'git send-email', there are no issues on my computer. I'm prompted for my SMTP password, git does its thing, and I've received an "OK / 250" return code every time.
Small, one-off patches seem to make it through fine, but beyond that, then the wine-devel or the patch-list doesn't get them. Furthermore, Zoho delivers me a message saying that wine-devel can't be reached and that they'll keep retrying the delivery. And they do, which is where I think part of the problem might be coming from.
---
In the first case, I tried sending a series of 7 patches. The first patch was a very short text change, the next five were larger batch substitutions on different strings, and the last one was a relatively short binary patch to delete five images.
The first patch made it through pretty quickly, but the rest never showed up. To make things more confusing, I received Zoho notices that patches 5-7 couldn't reach the list and would be retried, but nothing about 2-4. Then a few hours later, the mailing list confirmed that patch 7 had made it through fine, but no sign of the middle 5.
Since I had no notification from wine-devel or Zoho for patches 2-4, I tried resending patch 2 a couple times. Everything besides patches 1 & 7 seemed to just be falling into a black-hole though... until the next morning, every last patch I had sent (including the repeats of patch 2) was on both wine-devel and the patch-list, showing the original send dates, as if they had been there all along.
---
In the second case, I had three large binary patches for deleting old images that have since been moved onto the wiki. I tried sending them one-at-a-time, 'git send-email' gave me an OK / 250, and they never made it to the list. This time Zoho sent me a notice saying that there was a fatal error with wine-devel's address, specifically "ResponseCode 421, Read timed out"
The notice said they would keep retrying the delivery for 4 days, but then I didn't hear anything... until all of a sudden, today I see this on the patch queue (screenshot attached). I know I didn't resend either of these patches, much less 17 times. Weirdly enough, the wine-devel mailing list seems to understand what's going on a little better and only shows the message once.
---
The one other pattern I have noticed is that it almost seems like the mailing list is clocking out for business hours. The first was on a Wed evening, but it wasn't until the next morning that everything suddenly appeared. The second time, I tried sending in my patches before Friday evening, but there was still no sign of them when I checked yesterday (Sun). Sure enough, they're all there today though.
TL/DR: I know very little about how mail protocols work, and it could very well be on my mail service's end, but it almost seems like the mailing-list is actually receiving the emails. It then puts them in quarantine outside of normal work-hours, which doesn't seem like a problem in isolation. Somehow the sender is getting "delivery failed" codes though, and I only seem to get delivery receipts sporadically, despite turning on delivery receipts in my pipermail settings for wine-devel.
https://bugs.winehq.org/show_bug.cgi?id=45151
--- Comment #1 from Kyle Auble kyle.auble@zoho.com --- A couple of updates...
1) Today, I received a Zoho notice (after the 4 days mentioned above) saying that there had been a permanent error with my transmission. So at least from Zoho's end, they still think the message that spammed the list never actually made it.
2) As an experiment, I followed the exact same procedure as before, only sending the patches to my Gmail address. I tried it once around lunch-time, then again this evening (around 7:45 PM EDT) just in case the time is involved somehow.
Everything went through fine at both times, including a massive binary patch and a series sent in a batch. So whatever's going on, it appears to be happening somewhere between Zoho and the mail-list (though some setting of mine could definitely still be a contributing factor).
https://bugs.winehq.org/show_bug.cgi?id=45151
Kyle Auble kyle.auble@zoho.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Summary|patches: 'git send-email' |wine-devel: 'git |through Zoho SMTP causes |send-email' via Zoho SMTP |patch queue to go berserk |makes WineHQ mail queue go | |berserk
--- Comment #2 from Kyle Auble kyle.auble@zoho.com --- So Jeremy let me know that the same patch I sent last Friday started blasting wine-devel again. He mentioned that it was stuck in the exim retry queue. I've sent a support request to Zoho to see if they can tell me anything from their end; once they get back to me, I'll report any useful information.
I don't know details about how the mail system is set up, but if a queue in exim is where the message was lingering, my guess is the SMTP connection between the MTA and Zoho is specifically where things go haywire. Wouldn't that also explain why the patch queue sees all the repeats while mailman understands it's just one message? I've edited the bug name to reflect the new hypothesis.
I also tried googling for similar bug reports, and I really couldn't find anything. The one that seemed possibly related was that apparently using 'format-patch' and 'send-email' in tandem can sometimes duplicate headers: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=593542
I know Jeremy mentioned at one point he thought the messages themselves might be garbled. If this turns out to be the problem, I'll just add a note on the Submitting Patches page.
The only other things that jumped out were some discussions of how greylisting works on exim, which reminded me of the 4xx messages I got. For now though, that's all I have.
https://bugs.winehq.org/show_bug.cgi?id=45151
Kyle Auble kyle.auble@zoho.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Severity|normal |major
--- Comment #3 from Kyle Auble kyle.auble@zoho.com --- I have enough info for one last update, but I don't think I can add anything else beyond this point. I did go ahead and up the priority for this report. After thinking about it a bit more, I realized this bug potentially leaves the mailing list open to DoS attacks.
After the most recent blow-up (on May 10th), I found out that Zoho had blocked outgoing mail service on my account. I was able to get my account unblocked, but they weren't able to give me any debug information or transmission logs. All they said was that my recent activity triggered an anti-abuse filter so they lifted the block once I explained everything to them. Just to be safe, I'll be submitting patches as attachments through Gmail from here on out though.
According to their website, a burst of enough emails or possibly a data limit can cause a block. The block didn't start after the SNAFU with the 7 patch series though. The other dysfunctional patch was a larger binary one (~3.5M), but that didn't close down my SMTP service until after it reappeared on the patch queue Wednesday. Plus the overall message size isn't larger than, say, a photo I'd email to family. The one other possibility they mention is that if they get enough repeated bounces, they'll also lockdown the account.
I also finally tracked down some reports of the same problem, with multiple copies of a message flooding an SMTP server, on various forums. It's apparently not very common, but every discussion I found converged on a diagnosis of an email loop, possibly between the mail queue and the spam filter within the receiving server.
https://bugs.winehq.org/show_bug.cgi?id=45151
--- Comment #4 from Ken Sharp imwellcushtymelike@gmail.com --- Is this still an issue?