[INCIDENT] - 12/20/23 - Investigating Email Spam

[INCIDENT] - 12/20/23 - Investigating Email Spam

tl;dr

A threat actor compromised an application running on a dev server which was improperly configured with production SMTP credentials. This application, an earlier version of the Kbin fediverse software, was running with Symfony Debug Mode turned on, which allowed the attacker to inappropriately dump environment variables by intentionally triggering application errors. Contained within these environment variables were the impacted SES keys, as well as an IAM credential for uploading images to S3.

They then used the compromised SES credentials at approximately 7:30 AM PST to send out roughly 10,000 spam emails to Hotmail / Outlook customers (presumably from a pre-compromised list). We noticed this anomaly at 7:00 PM PST, 11 hours and 30 minutes after the incident. 

We promptly disabled the compromised credentials to prevent further spam, and began investigating the incident. The affected application has been decommissioned and additional alerting has been added to SES to detect any future incidents. 

The Credentials in question were only ever given access to the SES sending identity, and the S3 bucket attached to kbin, so there were no wider implications to the AWS account. Likewise, the dev application contained no customer data (it was a testing server).

We apologize for this incident and if you are one of the people who received an inappropriate email from our support account, we encourage you to not click any links therein.

12/21/23 1:13 PM

We continue to monitor the AWS account and affected server and have not detected any further compromises.

Additional changes have been made:
  1. SES will now alert if more than 200 emails are sent in a 6 hour period (well over our typical email traffic)
  2. Rotated credentials were reviewed for access, and a dedicated dev email sending identity has been established.

12/20/23 8:53 PM

The source of the leaked credentials has been located. A test server running a dev version of the Fediverse social network software kbin was incorrectly configured with production SMTP credentials configured to allow email sends from the production support@cthonicstudios.com. Domain. A pair of AWS credentials with S3 write permissions to a bucket dedicated to Kbin image uploads was also compromised and promptly rotated.

Account creation was always disabled on the Kbin server, and the only accounts present belonged to developers. No customer data was exposed, and none of the AWS credentials had access to any other resources in the account.

The Kbin server has also been decommissioned, and its DNS removed from Cloudflare.

12/20/23 8:10 PM

We're investigating an issue where an unusual amount of spam / phishing emails were sent out from the support@cthonicstudios.com email address to a large number of Outlook.com users via our SES credentials. The impacted SES credentials have been disabled preventing further spam, and at this time there does not appear to be any further compromise of the AWS account, and no other unusual activity in the logs. We will continue to investigate.