email¶
Pepipost
mailchimp
ets say an email bounces so the address gets blacklisted, but its a legit email address from a real customer. customer emails in and asks why they arent getting their emails, and customer support should have the tools to handle. sendgrid: search email address, find the bounces, click the button that says allow and user gets emails.
ses: set up an sqs queue. Pipe all bounces into a db. manage said db. Make a service with ui to search for emails. include a ui to unblock address.
There's at least one big difference between Mailgun and SendGrid that I know of: Latency. I implemented an email-only authentication library based on random tokens for Django0, and that's very sensitive to latency (you don't want the user to have to wait ten minutes for a login link). Unfortunately, Mailgun is pretty slow to deliver emails, so I use SendGrid to send emails for that.
What is the missing sauce? My understanding is that once your sending IP has some history and has developed a reputation as not sending spam, there isn't anything offered by Sendgrid that improves upon that. If you set up SPF and Domainkeys/DKIM that is an added vote of confidence but Sendgrid doesn't even require it, so I think most of the value of Sendgrid is simply offering a stable IP address and a convenient API. Am I wrong?
https://news.ycombinator.com/item?id=18225068 At a previous company I worked at we used to manage our own email infrastructure before finally switching over to a dedicated service. There are a few problems if you are sending over bulk email that is customized per user when running your own SMTP service.
-
IP address reputation - Keeping your IP addresses reputable is not a simple task. It requires balancing your emails for popular destination domains (gmail.com, aol.com, yahoo.com, etc.) across multiple external IP address. It requires you to deal with many different conflict resolution departments, who don't care about email, when a dispute comes up. It's practically a requirement to use a service like ReturnPath to maintain your reputation.
-
Throttling - When doing it yourself you need to throttle yourself. This is problematic on "big" days, especially when your marketing department wants to send many millions of emails for a big product push, promotion, or on days like black friday/cyber monday.
-
Hiring - A lot of people think sending email is easy. When you get up to the multiple million per day mark things start to fall apart. Do you have someone(s) on staff who really know sendmail/postfix/qmail inside and out?
-
Monitoring - sendmail/postfix/qmail are often times hard to monitor. You have to put together all of your stats. You have to put together all of your alerts. If you aren't really experienced with bulk email, you won't know what to look for and that can impact your reputation. Also consider your logging infrastructure. sendmail/postfix/qmail are noisy.
-
Cost - All of the points above play into the cost aspect of it. Is it cheaper to run it yourself, pay for all of the services and salaries, etc. Or is it actually cheaper to just use sendgrid/mailgun/etc. IP address reputation services are not cheap. Infrastructure cost is also something to consider. AWS IPs all have pretty terrible reputations so running this in AWS (and maybe other cloud providers) is a non-starter since no one will accept your email.
If you've got the expertise and you are sending a massive amount of emails then it might be worth it to run your own infrastructure, but at the end of the day, a single developer consuming an API is often easier and less problematic.
CAN SPAM Act https://www.ftc.gov/tips-advice/business-center/guidance/can-spam-act-compliance-guide-business
My favorite features:
- SAAS web app to manage the service with logs, and configuration as well as multi-user management with various roles. 2. HTTP Rest-like api for sending mail from your app 3. API keys for authentication, managed through web app 4. They do monitor black-lists and removals from blacklists. They also manage the warm-up time for your IP to make sure it is ready to send the volume of email you are sending, and can queue emails to avoid sending too many at once to the same destination mail service. 5. Spam control, in case you use the key in an app that turns out to have a relay vulnerability. 6. Return-path management to catch blocks and smtp errors.
I've managed my own SMTP services with PostFix and SendMail, and I will never go back to doing that myself again. It would take a lot more than 1M emails/month to make that worthwhile.
Please note that the appearance of an Amazon SES IP address on a blacklist does not always affect your sending as an Amazon SES customer. If your sending is affected, you will most commonly get an indication via a bounce message that your mail is being rejected due to a listing on a blacklist. In most of these cases, you will see the name or URL of the blacklist in the bounce message (for example, "Message rejected due to IP http://0.0.0.0 listed on RBL [X]"). If you do not see these types of messages in your bounces, it is unlikely that a listing is affecting your sending.
IP address reputation is only one signal in several hundred that sophisticated ISPs use. That said, we do work closely with mailbox providers and blacklist operators, and carefully monitor the reputation of our IP address space. Whenever possible, we work directly with these providers to identify and mitigate such threats in parallel with our own routine compliance processes to remove offending users from our system. Unfortunately, we cannot provide a guarantee or timeframe, because the list operators set their own listing and delisting policy.
Most of the IP addresses are listed in SpamCannibal blacklist, Our metrics indicate that the impact of the SpamCannibal listing to email delivery is extremely limited, because SpamCannibal is not commonly used by receivers (ISPs) to make decisions about accepting mail. In particular, we are not aware of any large ISPs that use SpamCannibal.
In general, the better the quality of your mail (DKIM, DMARC, SPF, formatting, content, valid addresses etc) the less likely it is to be rejected by mail servers. You can also check the quality/deliver-ability of your emails by using a free service such as www.mail-tester.com
We've been also experiencing the same problem for 2 weeks now. In our case email is send by our application and it is no possible to auto interpret or even do anything (e.g. auto retry) about bouncing back emails of that sort: 550 rbl-reject: Message rejected because 54.240.27.23 is blacklisted
Trying to understand where SES IPs are blacklisted I found http://mxtoolbox.com/blacklists.aspx. You can easily see that pretty much any 54.240.27.xxx IP exists on one or more blacklists.
One other data point you may want to look at in your analysis -- overall SES complaint rates. My hunch is that complaint rates would have decreased globally starting sometime last week as more SES emails landed in spam folders (and thus never got opened).
I know you know more about this than I do, but this is an observation from our side I thought might share -- because I'd imagine it's much harder to examine IP reputation when it doesn't manifest itself as explicitly as a SORBS blacklist entry or outright rejection.
https://mxtoolbox.com/SuperTool.aspx?action=blacklist:54.240.10.126&run=toolpage
https://wordtothewise.com/2015/11/increase-in-cbl-listings/ http://www.abuseat.org/ http://www.sorbs.net/lookup.shtml?54.240.8.29 http://www.uceprotect.net/en/rblcheck.php?ipr=54.240.8.97
If you go this route, I would recommend a separate domain for any emails that are are bulk in nature. Keep the one off transactional ones on a different domain. Then, if your bulk emails cause a false positive, you haven't burned essential things like receipts, password resets, shipping notices, etc.
Test email in lower environments:
Local Server: maildev/maildev
Validate email address:
https://blog.mailtrap.io/python-validate-email/
Mialjet/sendgrid/mailchimp
EC2 instance with bring your own IP and run something like SparkPost Momentum.
sendgrid - Very expensive in our experience as well as lacking key features for enterprise and documentation being hard to find key limits like maximum api call size which is useful when dealing with mad emails with tens of thousands of recipients and many tag replacements.
- Yes you can BYOIP - https://docs.aws.amazon.com/ses/latest/DeveloperGuide/dedicated-ip-byo.html
-
I can recommend SendGrid for this scale. I know at least 2 enterprises that use it at similar scale. (although I do not know the details of their operations nor problems)
-
Consider using multiple accounts. This doesn't even need to be done at the application level I guess. You can probably have a funneling application of some sort which splits the calls 50/50 across two accounts or more. There would be various possible solutions to achieve it I imagine. We considered this as a solution but ended up not requiring this.