On July 29 we became aware that a cache database instance was exposed to the public internet after a maintenance task completed on July 28. Due to this, data temporarily in this database may have been compromised. After thorough investigation we have concluded that user data was most likely not compromised. But, in the best interest of security and transparency we have revoked any secrets that may have been exposed and have published notice of this incident. Since Nightbot does not store passwords, there is no need to change any passwords.
Who was potentially affected?
Users in any of the following conditions between Jul 28 2019 18:41:29 GMT-0700 and Jul 29 2019 09:04:00 GMT-0700
- Active chats (meaning messages being sent or received to their chat) on Twitch, YouTube, and Discord
- Joined Nightbot to their chat on Twitch, YouTube, and Discord
- Parted Nightbot from their chat on Twitch, YouTube, and Discord
- Deleted their Nightbot account (which parted Nightbot from their chat)
- Sent a message through Nightbot’s API
What was potentially exposed?
- User Nightbot channel settings
- User Twitch and YouTube API access tokens used by some Nightbot commands
- User Email addresses (affecting only users who registered prior to Feb. 20, 2016)
- Side note: If we had your email stored we have emailed you a notice of this incident and your email has been purged from our database.
What do I need to do?
No action should be required on your behalf. We have expired all secrets, and any email addresses still stored in our database have been deleted.
If you have a want to know the technical details, below is the timeline of events.
Technical Timeline of Events
- Jul 28 2019: Infrastructure work for Nightbot was occurring throughout the day to integrate private networking between servers in preparation for some relocation of services to multiple servers. Part of this work was upgrading and relocating MongoDB, the primary database used by Nightbot, to new infrastructure in this type of configuration.
- Jul 28 2019 19:41:29 GMT-0700: Nightbot’s master node, which used to host both redis and a MongoDB instance, had its firewall configuration updated to remove MongoDB from its firewall. Due to misconfiguration this inadvertently opened up Redis, the caching database used by Nightbot, to the public internet too.
- Jul 29 2019 05:40:00 GMT-0700: Something connected to Redis.
- Jul 29 2019 05:40:00 GMT-0700: “FLUSHALL” was executed on the Redis database, which effectively deletes all the data from it.
- Jul 29 2019 05:40:00 GMT-0700: Redis key(s) inserted for a cronjob
- Jul 29 2019 05:40:00 GMT-0700: Redis data directory reconfigured to /var/spool/cron/
- Jul 29 2019 05:40:00 GMT-0700: Redis data file reconfigured to root
- Jul 29 2019 05:40:00 GMT-0700: “SAVE” was executed, which attempted to persist the key(s), but fails since Redis does not have permissions to write to cron spool.
- Jul 29 2019 05:40:00 GMT-0700: Redis begins to fail to save repeatedly
- Jul 29 2019 05:42:00 GMT-0700: Redis is configured to stop accepting new keys
- Jul 29 2019 06:42:00 GMT-0700: Redis keys from the period of 05:40:00 to 05:42:00 are evicted from cache. Cache is now empty apart from attacker’s cronjobs
- Jul 29 2019 08:45:00 GMT-0700: Investigation begins into Nightbot outage (Nightbot has been offline since 05:42:00 for most users, though some still had access until 06:42:00 due to cache existence, including our monitoring account).
- Jul 29 2019 09:00:00 GMT-0700 (approx): Redis is determined to be the cause of outage. Logs indicated failure to save database. It is determined Redis must have been compromised.
- Jul 29 2019 09:04:00 GMT-0700: Redis is stopped. Firewall configuration is repaired. Redis configuration is validated.
- Jul 29 2019 09:09:00 GMT-0700: Redis is started. Nightbot is recovering. Investigation begins to determine scope of data breach.
Jul 29 2019 11:00:00 GMT-0700 (approx): Scouring of caching logic and logs ensued to determine what data could have been compromised, and whose it was.
- It was determined that channels that, starting from an hour before this timeline of events, had chat flowing through them, as well as users that joined Nightbot to a chat (on Twitch, YouTube, or Discord), parted Nightbot to a chat (on Twitch, YouTube, or Discord), deleted their account (which triggers Nightbot to part their chat), or sent a message through Nightbot’s API could have been in the cache.
- It was determined that data in the cache included: temporary ratelimits, chat warnings/timeouts/cooldowns, and channel records.
- Channel records include base Nightbot configuration for a channel, plus user information including account provider credentials (which powers some chat commands).
- It was determined that a handful of user data included an email, and this only affected really old Nightbot accounts.
- It was determined that Nightbot’s credentials for Twitch and YouTube were also exposed in the cache.
- Jul 29 2019 12:00:00 GMT-0700 (approx): Start resetting Nightbot tokens + secrets and reconfiguring Nightbot to use them.
- Jul 29 2019 13:00:00 GMT-0700 (approx): Start researching into user account provider credentials scope.
- Jul 29 2019 13:15:00 GMT-0700 (approx): It is determined that no Discord user account provider credentials were in the cache.
- Jul 29 2019 13:30:00 GMT-0700 (approx): It is determined that YouTube user account provider credentials were in the cache, and scope was limited to YouTube channel data. YouTube credentials expire hourly, and any that were in the cache were long since expired.
- Jul 29 2019 14:00:00 GMT-0700 (approx): It is determined that Twitch user account provider credentials were in the cache, and scope included user information like email address, as well as channel information. Twitch credentials do not expire, and any that were in the cache are still valid.
- Jul 29 2019 14:15:00 GMT-0700 (approx): It is determined that there exists no setting to mass revoke Twitch access tokens, only individual tokens.
- Jul 29 2019 14:38:00 GMT-0700: First ping to Twitch regarding this security incident to ask for them to mass revoke access tokens.
- Jul 29 2019 15:15:00 GMT-0700: Confirmation that Twitch is working towards mass revocation of the credentials.
- Jul 29 2019 16:37:00 GMT-0700: Notification that Twitch will not be able to complete this until Jul 30.
Jul 29 2019 17:00:00 GMT-0700 (approx): Metrics begin to explain the timeline of events around
05:40. Slowlog metrics report
keymetrics report the decrease in keys from flushing and cache eviction. Metrics do not show a
SCANcommand on the slowlog during the window of vulnerability, and do not show an abnormal amount of commands being executed nor large amounts of data being transferred out of the server, which leads us to believe that there was no attempt to “dump” redis contents. Without
SCANbeing ran, an attacker would have no way of knowing what content they could pull from Redis. It is determined at this point that user data was most likely not compromised.
- July 30 2019 15:39:00 GMT-0700: Twitch confirmed they began mass revocation of credentials.
- July 31 2019 09:49:00 GMT-0700: Twitch confirmed the mass revocation had completed.