Verify if cloud.gov catalog can send email

[jbrown-xentity]

User Story

In order to get harvest reports, data.gov data providers wants email setup.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN catalog is in dev on cloud.gov
  WHEN a harvest is completed
  THEN we should be able to verify whether an email is sent to the organization email or not
  AND datagovhelp is sent a copy of the harvest report

Background

Just need to add https://github.com/GSA/datagov-brokerpak-smtp as a service to catalog.

Security Considerations (required)

None

Sketch

Should be as simple as adding the smtp service similar to others: https://github.com/GSA/catalog.data.gov/blob/main/create-cloudgov-services.sh.
The credentials will need to be parsed and placed in the config appropriately for CKAN to use.

This should also be documented in setup readme steps, along with the note that this shouldn't be added to dev environments due to confusion it may cause in the email receivers.

Validate that this can be done in dev:

• Create SMTP service manually
• Add service to catalog manifest and attach credentials appropriately
• Deploy to develop
• Create organization with email address for testing
• Create harvest source and run
• Update emails in staging db in group_extra table
• Update emails in staging db in user table
• Run harvester
• Validate email received

Final step: create new ticket for sanitizing staging and documenting, and create release PR

[mogul]

The SMTP service is already registered and available in the appropriate spaces, so there's no need to mess with the broker deployment to complete this.

That said there might still be a step to move a provisioned SES instance into the approved status on the AWS side (via console or in future via Terraform); can't remember where I left that!

[nickumia-reisys]

Successfully created SMTP service and bound to catalog. Need help with the following,

• How to configure ckan with SMTP credentials
• How to change all of the production emails in staging to dummy emails so real emails don't go out.

[jbrown-xentity]

Email settings can be set in the .profile:

• http://docs.ckan.org/en/2.9/maintaining/configuration.html#email-settings
• https://github.com/GSA/catalog.data.gov/blob/main/.profile#L76

As to replacing the URL's in staging, it would be something like:

• Connect to catalog DB (either with SSH or with tunneling)
• Examine the organizations table, there should be an extras field that contains emails
• Set all emails to datagovhelp, or some email that we can verify without spamming others

[nickumia-reisys]

Going to follow the instructions here... https://docs.ckan.org/en/2.9/maintaining/email-notifications.html

[nickumia-reisys]

Is it just me or should the smtp service output what the actual email is? (outside of the domain_arn)

  "datagov-smtp": [
    {
      "credentials": {
        "domain_arn": "<fancyawssutff>/ses-2e4380c03ce8656f.ssb.data.gov",
        "email_receipt_error": "datagovhelp@gsa.gov",
        "region": "blahblahblah",
        "smtp_password": "blahblahblah",
        "smtp_server": "blahblahblah",
        "smtp_user": "blahblahblah"
      },

[nickumia-reisys]

Was able to get error from SMTP setup... long story short, we have to figure out what the real email address is 😞

[nickumia-reisys]

Okay.. doing <anything>@domain_name seems to have worked, but there's a formatting issues somewhere in our pipeline,

   2022-05-03T21:09:30.89-0400 [APP/TASK/harvester/0] ERR Please do not reply to this email as it was sent from a non-monitored address.
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR 2022-05-04 01:09:31,124 ERROR [ckanext.harvest.logic.action.update] 'NoneType' object has no attribute 'strip'
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR Traceback (most recent call last):
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/bin/ckan", line 8, in <module>
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR sys.exit(ckan())
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/click/core.py", line 829, in __call__
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR return self.main(*args, **kwargs)
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/click/core.py", line 782, in main
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR rv = self.invoke(ctx)
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR return _process_result(sub_ctx.command.invoke(sub_ctx))
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR return _process_result(sub_ctx.command.invoke(sub_ctx))
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR return ctx.invoke(self.callback, **ctx.params)
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/click/core.py", line 610, in invoke
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR return callback(*args, **kwargs)
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR return f(get_current_context(), *args, **kwargs)
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/src/ckanext-harvest/ckanext/harvest/cli.py", line 264, in run
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR utils.run_harvester()
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/src/ckanext-harvest/ckanext/harvest/utils.py", line 364, in run_harvester
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR tk.get_action("harvest_jobs_run")(context, {})
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/ckan/logic/__init__.py", line 504, in wrapped
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR result = _action(context, data_dict, **kw)
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/src/ckanext-harvest/ckanext/harvest/logic/action/update.py", line 715, in harvest_jobs_run
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR send_summary_email(context, job_obj.source.id, status)
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/src/ckanext-harvest/ckanext/harvest/logic/action/update.py", line 810, in send_summary_email
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR send_mail(recipients, subject, body)
   2022-05-03T21:09:31.12-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/src/ckanext-harvest/ckanext/harvest/logic/action/update.py", line 828, in send_mail
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR mailer.mail_recipient(**email)
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/ckan/lib/mailer.py", line 125, in mail_recipient
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR body_html=body_html, headers=headers)
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/ckan/lib/mailer.py", line 107, in _mail_recipient
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR smtp_connection.sendmail(mail_from, [recipient_email], msg.as_string())
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/smtplib.py", line 877, in sendmail
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR (code, resp) = self.rcpt(each, rcpt_options)
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/smtplib.py", line 547, in rcpt
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR self.putcmd("rcpt", "TO:%s%s" % (quoteaddr(recip), optionlist))
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/smtplib.py", line 153, in quoteaddr
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR if addrstring.strip().startswith('<'):
   2022-05-03T21:09:31.13-0400 [APP/TASK/harvester/0] ERR AttributeError: 'NoneType' object has no attribute 'strip'

[nickumia-reisys]

Okay, I know, this is a lot of logs, but maybe python isn't the error, I found this in the logs too... Definitely an SMTP broker issue and I don't know where aaron's email came from... @mogul any insights?

   2022-05-03T21:09:30.89-0400 [APP/TASK/harvester/0] ERR 2022-05-04 01:09:30,896 ERROR [ckan.lib.mailer] SMTPDataError(554, b"Access denied: User `arn:aws:iam::821341638715:user/cf/ses-b7a1cd34ddf3b0cf-csb--e3a67950-0aa6-4cb0-b272-76cfcf93cb8a' is not authorized to perform `ses:SendRawEmail' on resource `arn:aws:ses:us-west-2:821341638715:identity/aaron.borden@gsa.gov'")
   2022-05-03T21:09:30.89-0400 [APP/TASK/harvester/0] ERR Traceback (most recent call last):
   2022-05-03T21:09:30.89-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/ckan/lib/mailer.py", line 107, in _mail_recipient
   2022-05-03T21:09:30.89-0400 [APP/TASK/harvester/0] ERR smtp_connection.sendmail(mail_from, [recipient_email], msg.as_string())
   2022-05-03T21:09:30.89-0400 [APP/TASK/harvester/0] ERR File "/home/vcap/deps/1/python/lib/python3.7/smtplib.py", line 893, in sendmail
   2022-05-03T21:09:30.89-0400 [APP/TASK/harvester/0] ERR raise SMTPDataError(code, resp)
   2022-05-03T21:09:30.89-0400 [APP/TASK/harvester/0] ERR smtplib.SMTPDataError: (554, b"Access denied: User `arn:aws:iam::821341638715:user/cf/ses-b7a1cd34ddf3b0cf-csb--e3a67950-0aa6-4cb0-b272-76cfcf93cb8a' is not authorized to perform `ses:SendRawEmail' on resource `arn:aws:ses:us-west-2:821341638715:identity/aaron.borden@gsa.gov'")

P.S. This is a freshly created service today. So there's nothing tying this particular instance to anything aaron was doing before.

P.S.2. This is the code that adds the SendRawEmail permission, https://github.com/GSA/datagov-brokerpak-smtp/blob/main/terraform/bind/main.tf#L15-L35

[mogul]

Not a clue and the HCL is very simple. This is really surprising!

[nickumia-reisys]

I'll try and investigate then. Thanks for taking a look @mogul!

[nickumia-reisys]

• It was determined that the datagov-brokerpak-smtp service could not actually send email. Which lead to Debug and create test for datagov-smtp broker #3810

[FuhuXia]

Error sending emails:

[APP/TASK/05ccfb8b/0] OUT 
INFO  [ckanext.datagovcatalog.harvester.notifications] Extra recipients for source found: [{'name': 'fuhu.xia@gsa.gov', 'email': 'fuhu.xia@gsa.gov'}]                                                                                                                         
[APP/TASK/05ccfb8b/0] OUT 
ERROR [ckanext.harvest.logic.action.update] 'NoneType' object has no attribute 'strip'          
[APP/TASK/05ccfb8b/0] ERR 
ERROR [ckanext.harvest.logic.action.update] 'NoneType' object has no attribute 'strip'          
ERR Traceback (most recent call last):                                                                                      
ERR File "/home/vcap/deps/1/bin/ckan", line 8, in <module>                                                                  
ERR sys.exit(ckan())                                                                                                        
ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/click/core.py", line 829, in __call__                        
ERR return self.main(*args, **kwargs)                                                                                       
...
ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/ckan/logic/__init__.py", line 504, in wrapped                
ERR result = _action(context, data_dict, **kw)                                                                              
ERR File "/home/vcap/deps/1/src/ckanext-harvest/ckanext/harvest/logic/action/update.py", line 715, in harvest_jobs_run      
ERR send_summary_email(context, job_obj.source.id, status)                                                                  
ERR File "/home/vcap/deps/1/src/ckanext-harvest/ckanext/harvest/logic/action/update.py", line 810, in send_summary_email    
ERR send_mail(recipients, subject, body)                                                                                    
ERR File "/home/vcap/deps/1/src/ckanext-harvest/ckanext/harvest/logic/action/update.py", line 828, in send_mail             
ERR mailer.mail_recipient(**email)                                                                                          
ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/ckan/lib/mailer.py", line 125, in mail_recipient             
ERR body_html=body_html, headers=headers)                                                                                   
ERR File "/home/vcap/deps/1/python/lib/python3.7/site-packages/ckan/lib/mailer.py", line 107, in _mail_recipient            
ERR smtp_connection.sendmail(mail_from, [recipient_email], msg.as_string())                                                 
ERR File "/home/vcap/deps/1/python/lib/python3.7/smtplib.py", line 877, in sendmail                                         
ERR (code, resp) = self.rcpt(each, rcpt_options)                                                                            
ERR File "/home/vcap/deps/1/python/lib/python3.7/smtplib.py", line 547, in rcpt                                             
ERR self.putcmd("rcpt", "TO:%s%s" % (quoteaddr(recip), optionlist))
ERR File "/home/vcap/deps/1/python/lib/python3.7/smtplib.py", line 153, in quoteaddr                                        
ERR if addrstring.strip().startswith('<'):                                                                                  
ERR AttributeError: 'NoneType' object has no attribute 'strip'                                                              
OUT Exit status 1  

[nickumia-reisys]

To reiterate: the instructions in this comment needs to be checked/done everytime there's a questionable email added to the DB or if the DB is restored with bad emails.

[FuhuXia]

This is a harvest extension bug. Did a PR upstream to fix it, checking email address exist before sending notifications, so we don't have to run any manual steps.

[Update]
Upstream PR merged.