Title
#fleet
a

Artem

04/27/2022, 1:27 PM
Hello everyone! Did anybody previously see such error
535 5.7.8 Username and Password not accepted
when setting up SMTP settings in Fleet UI? We use Google Workspace with our custom domain, I’ve enabled 2FA on service email account and generated application-specific password. When I set up SMTP config on Settings page, I don’t see any problems. Using Chrome Dev Console I see such SMTP settings:
"smtp_settings": {
    "enable_smtp": true,
    "configured": true,
    "sender_address": "<mailto:user@ourdomain.com|user@ourdomain.com>",
    "server": "<http://smtp.gmail.com|smtp.gmail.com>",
    "port": 587,
    "authentication_type": "authtype_username_password",
    "user_name": "<mailto:user@ourdomain.com|user@ourdomain.com>",
    "password": "********",
    "enable_ssl_tls": true,
    "authentication_method": "authmethod_plain",
    "domain": "",
    "verify_ssl_certs": true,
    "enable_start_tls": true
  }
But when I try to invite new user to Fleet I see such error:
535 5.7.8 Username and Password not accepted. Learn more at\n5.7.8  <https://support.google.com/mail/?p=BadCredentials> i27-20020a1c541b000000b003928e866d32sm1520496wmb.37 - gsmtp
2:38 PM
The funny stuff related to this problem is that I receive an email from this service account about successful SMTP configuration, but still can not add new users because of error described earlier.
Kathy Satterlee

Kathy Satterlee

04/27/2022, 6:14 PM
@Artem Sorry SMTP is giving you trouble! Have you tried Google's Captcha Unlock?
a

Artem

04/27/2022, 6:15 PM
Hi @Kathy Satterlee! I didn’t try it yet, will do it asap, thanks for advice!
zwass

zwass

04/27/2022, 6:17 PM
btw on newer versions of Fleet you can still add users without email.
a

Artem

04/27/2022, 6:17 PM
@zwass yea, that’s our current single option 😉
zwass

zwass

04/27/2022, 6:18 PM
Hopefully Kathy's suggestion helps get this working
a

Artem

04/27/2022, 6:18 PM
But it’s really interesting to find and fix the reason of the problem
6:20 PM
@Kathy Satterlee could you please explain any details about this mechanism? I didn’t have experience with it and currently don’t properly understand how to user it with Fleet.
zwass

zwass

04/27/2022, 6:22 PM
Do you have 2FA enabled on this account? Might need to use app specific password.
a

Artem

04/27/2022, 6:24 PM
@zwass yes, I have 2FA enabled and I use 16-char app specific password. With general password I see another type of error during SMTP configuration
Could not update settings. sending mail: client auth error: 534 5.7.9 Application-specific password required. Learn more at 5.7.9 <https://support.google.com/mail/?p=InvalidSecondFactor> i27-20020a1c541b000000b003928e866d32sm2111167wmb.37 - gsmtp
Kathy Satterlee

Kathy Satterlee

04/27/2022, 6:28 PM
As far as that Captcha unblock goes, Google has some built-in protections against SPAM senders/scammers that can be a little overprotective. By doing the captcha unblock, you're essentially confirming that you're actually to do the thing it blocked you from doing. You should (hopefully) only need to do it once after the initial setup.
a

Artem

04/27/2022, 6:29 PM
As I can see, I can not use Google Captcha with 2FA enabled 😞
6:29 PM
I see such message, when try to follow https://accounts.google.com/b/0/DisplayUnlockCaptcha
Kathy Satterlee

Kathy Satterlee

04/27/2022, 6:30 PM
It looks like it unblocked successfully. Try adding a new user and we'll see if that was the issue.
a

Artem

04/27/2022, 6:33 PM
Sorry, but nothing is changed. I tried to invite new user right now and saw same error.
6:37 PM
I also tried to create another app specific passwords, but had same results with them. This problem looks really strange for me because of successful email about SMTP configuration, but no opportunity to invite new users…
Kathy Satterlee

Kathy Satterlee

04/27/2022, 6:39 PM
What version of Fleet are you running? I can test adding Google SMTP settings to see what happens.
a

Artem

04/27/2022, 6:40 PM
Fleet 4.13.0 • Go go1.17.8
Kathy Satterlee

Kathy Satterlee

04/27/2022, 6:42 PM
Give me a few minutes to test things out.
a

Artem

04/27/2022, 6:45 PM
I think there is small probability about google workspace based reason of problem. Found such article https://stackoverflow.com/questions/63562314/535-b5-7-8-username-and-password-not-accepted-with-g-suite-and-django, but without solution
6:58 PM
Is it posiible to debug actial connection between google and Fleet somehow? Maybe any kind of verbose logging? I would like to compare connections at first time (when I get email about finished SMTP configuration) and at user invitation time
zwass

zwass

04/27/2022, 7:00 PM
I think you'd have to use some sort of packet capture tool for that. The strange thing is that the same email sending code is used everywhere in the app: https://github.com/fleetdm/fleet/blob/main/server/mail/mail.go#L143
Kathy Satterlee

Kathy Satterlee

04/27/2022, 7:00 PM
I'm getting the same error, so I'm digging in to see if I can get it resolved on my end.
a

Artem

04/27/2022, 7:03 PM
@Kathy Satterlee thank you for such deep diving, hope you will find how to solve it!
Kathy Satterlee

Kathy Satterlee

04/27/2022, 7:42 PM
Happy to help! Looking at one last setting and then we may need to call this one a bug.
7:43 PM
This one might take a few to filter through, so I'm going to step away for a few minutes.
8:15 PM
I'm going to loop our engineering team in on this one. I'll let you know as soon as I have an update!
a

Artem

04/27/2022, 8:20 PM
@Kathy Satterlee thank you!
Kathy Satterlee

Kathy Satterlee

04/27/2022, 9:17 PM
Happy to help 🙂
9:19 PM
I may not have an answer for you today, but I'll make sure I check in on this first thing in the morning as well.
4:35 PM
Testing this out with everything all fresh and new now!
a

Artem

04/28/2022, 6:15 PM
@Kathy Satterlee good luck with it! Hope you will find exact reason!
Kathy Satterlee

Kathy Satterlee

04/28/2022, 7:39 PM
Me too! I ran into some unrelated technical issues I'm needing to clean up (went a little to rogue with some file permissions), so things are taking a while.
a

Artem

05/01/2022, 9:58 AM
Hello @Kathy Satterlee! Did you manage to achieve any results in researching the problem?
Kathy Satterlee

Kathy Satterlee

05/02/2022, 5:00 PM
No, and I've thrown everything I've got at it. I'll get a bug report filed to get some more eyes on it, but it may be something odd with Google.
10:59 PM
We have a bug report filed for this. Hopefully we can track down where the breakdown in communication is happening with the SMTP. If anything new develops, just let me know!
a

Artem

05/03/2022, 7:32 PM
Did I understand you correctly that I need to create an issue for this problem?
Kathy Satterlee

Kathy Satterlee

05/03/2022, 7:33 PM
I've already gotten it filed!
a

Artem

05/03/2022, 8:06 PM
@Kathy Satterlee okay, thanks! I didn’t find anything related to our problem in https://github.com/fleetdm/fleet/issues, so decided to ask you.
Kathy Satterlee

Kathy Satterlee

05/03/2022, 8:07 PM
That's odd. I'll double check in a few minutes and file again if I need to.
9:26 PM
I did have to recreate the issue, thanks for bringing that back to me! Here's the active link: https://github.com/fleetdm/fleet/issues/5542
a

Artem

05/03/2022, 9:52 PM
@Kathy Satterlee thanks!
Kathy Satterlee

Kathy Satterlee

05/17/2022, 7:12 PM
Hey @Artem, wanted to drop a quick line here so that you knew we were still on it! Engineering is still looking into this but it's definitely looking more and more like it's something on the Google end blocking the smtp.
a

Artem

05/18/2022, 8:30 AM
Hi @Kathy Satterlee! Got it, thanks! Currently we found small “workaround” via adding user with SAML access, but I hope you will find the way to solve this bug!