Subscribe to RSS
Follow me on Twitter
Join me on Facebook
On Jan. 11, Ubiquiti Inc. [NYSE:UI] — a major vendor of cloud-enabled Internet of Things (IoT) devices such as routers, network video recorders and security cameras — disclosed that a breach involving a third-party cloud provider had exposed customer account credentials. Now a source who participated in the response to that breach alleges Ubiquiti massively downplayed a “catastrophic” incident to minimize the hit to its stock price, and that the third-party cloud provider claim was a fabrication.
A security professional at Ubiquiti who helped the company respond to the two-month breach beginning in December 2020 contacted KrebsOnSecurity after raising his concerns with both Ubiquiti’s whistleblower hotline and with European data protection authorities. The source — we’ll call him Adam — spoke on condition of anonymity for fear of retribution by Ubiquiti.
“It was catastrophically worse than reported, and legal silenced and overruled efforts to decisively protect customers,” Adam wrote in a letter to the European Data Protection Supervisor. “The breach was massive, customer data was at risk, access to customers’ devices deployed in corporations and homes around the world was at risk.”
Ubiquiti has not responded to repeated requests for comment.
According to Adam, the hackers obtained full read/write access to Ubiquiti databases at Amazon Web Services (AWS), which was the alleged “third party” involved in the breach. Ubiquiti’s breach disclosure, he wrote, was “downplayed and purposefully written to imply that a 3rd party cloud vendor was at risk and that Ubiquiti was merely a casualty of that, instead of the target of the attack.”
In its Jan. 11 public notice, Ubiquiti said it became aware of “unauthorized access to certain of our information technology systems hosted by a third party cloud provider,” although it declined to name the third party.
In reality, Adam said, the attackers had gained administrative access to Ubiquiti’s servers at Amazon’s cloud service, which secures the underlying server hardware and software but requires the cloud tenant (client) to secure access to any data stored there.
“They were able to get cryptographic secrets for single sign-on cookies and remote access, full source code control contents, and signing keys exfiltration,” Adam said.
Adam says the attacker(s) had access to privileged credentials that were previously stored in the LastPass account of a Ubiquiti IT employee, and gained root administrator access to all Ubiquiti AWS accounts, including all S3 data buckets, all application logs, all databases, all user database credentials, and secrets required to forge single sign-on (SSO) cookies.
Such access could have allowed the intruders to remotely authenticate to countless Ubiquiti cloud-based devices around the world. According to its website, Ubiquiti has shipped more than 85 million devices that play a key role in networking infrastructure in over 200 countries and territories worldwide.
Adam says Ubiquiti’s security team picked up signals in late December 2020 that someone with administrative access had set up several Linux virtual machines that weren’t accounted for.
Then they found a backdoor that an intruder had left behind in the system.
When security engineers removed the backdoor account in the first week of January, the intruders responded by sending a message saying they wanted 50 bitcoin (~$2.8 million USD) in exchange for a promise to remain quiet about the breach. The attackers also provided proof they’d stolen Ubiquiti’s source code, and pledged to disclose the location of another backdoor if their ransom demand was met.
Ubiquiti did not engage with the hackers, Adam said, and ultimately the incident response team found the second backdoor the extortionists had left in the system. The company would spend the next few days furiously rotating credentials for all employees, before Ubiquiti started alerting customers about the need to reset their passwords.
But he maintains that instead of asking customers to change their passwords when they next log on — as the company did on Jan. 11 — Ubiquiti should have immediately invalidated all of its customer’s credentials and forced a reset on all accounts, mainly because the intruders already had credentials needed to remotely access customer IoT systems.
“Ubiquiti had negligent logging (no access logging on databases) so it was unable to prove or disprove what they accessed, but the attacker targeted the credentials to the databases, and created Linux instances with networking connectivity to said databases,” Adam wrote in his letter. “Legal overrode the repeated requests to force rotation of all customer credentials, and to revert any device access permission changes within the relevant period.”
If you have Ubiquiti devices installed and haven’t yet changed the passwords on the devices since Jan. 11 this year, now would be a good time to care of that.
It might also be a good idea to just delete any profiles you had on these devices, make sure they’re up to date on the latest firmware, and then re-create those profiles with new [and preferably unique] credentials. And seriously consider disabling any remote access on the devices.
Ubiquiti’s stock price has grown remarkably since the company’s breach disclosure Jan. 16. After a brief dip following the news, Ubiquiti’s shares have surged from $243 on Jan. 13 to $370 as of today. By market close Tuesday, UI had slipped to $349.
I am curious to know how the unauthorized access to the LastPass account of an IT admin occurred. If you glean more information about that particular initial compromise, please let us know.
Susan – I agree. I wonder if they hacked the LastPass account, or what?
LastPass doesn’t require 2fa on your account so phishing the master password is possible.
LastPass does requires 2FA only if you’ve enabled it in your browser.
Must be via browser or local privileged access; and the user storing the master for ease of use (LastPass does have that function I recall; but warns when enabled).
I love how the employee that works in security has all the answers AFTER the hack…how about he thinks about this stuff before the attack and saves himself and us all form his vitriol!
The company wasn’t interested in mitigation after the attack.
Doesn’t sound like a company that would spend money on security before the attack, does it?
You sir are silly.
This: “I love how the employee that works in security has all the answers AFTER the hack”
is the same as this: I love how the Doctor that works in viruses has all the answers AFTER the sickness”
You understand that he is a contractor who was contracted after the breach was detected right? That’s what the article says. Stop speed reading.
“A security professional at Ubiquiti who helped the company respond to the two-month breach beginning in December 2020”
Doesn’t say or imply either contractor or perm. staff.
In big corporation, it’s all about money. One security administrator yelling around about security holes and how to patch them, usually is not loud enough. Sad but true for many Fortune 500. It is indeed harder to steer Titanic.
thats what happens when you are pushing security from the bottom. Security needs to be a top level spot. Officer or equivalent
The Krebs ran a story where he suggests doing that, after going into how most of the top companies are still making IT security a 2nd-rate priority
Can you say “Jail” baby!
I doubt that stock price most recent cited by NK in the article isn’t gonna hold up for long. Indeed it is down 6+% intoday’s session on NYSE
It is even worse: Ubiquiti forced all users to use cloud-based authentification even for accessing your controller software on a local network with a local client. This was not even properly communicated but deployed by one of the regular maintenance updates. A lot of people raised security concerns within the community already.
I hope there is a class action lawsuit anytime soon.
That’s not true at all. I run 2 controllers neither of which have to be connected to their cloud in order to be accessed or function.
I’ve been hypercritical of Ubiquiti lately with the development of their UniFi product line as they have a significant firmware development problem going on right now. But what you mentioned simply is not true.
That being said, I do not use their UDM or CloudKey product lines. I have stood up one UDM-Pro as a side project for a church, but I don’t recall if you had to utilize a cloud account for that 100%. I know they had local users, I just don’t recall if a cloud account was REQUIRED on setup. But I know for a fact that if you roll your own controller, you do not have to attach it to one of their cloud accounts.
You have to have to use a cloud account to set up the newer line UDM/UDMP, but you can disconnect the controller from their cloud after the initial setup to my knowledge.
This is also true of the CloudKey Gen2. After the update there is no way to setup the device with only local accounts, even when doing a factory reset. Your only option is to associate it with a Unifi account, and then disable remote access. But even then, this leaves you with an “Owner” account that still authenticates to the Unifi servers and grants access. There is no way to disable it. Ubiquiti is dead to me.
This is also true of the CloudKey Gen2+. After the update, there is no way to setup the device without logging into a Unifi account. Your only option is to use the cloud account, then add a local admin, then disable remote access. This removes the device from the Unifi cloud service. But even then, this leaves you with the Unifi “Owner” account permanently entrenched on the device, and that account can be used to log into the controller by authenticating against the Unifi account used to setup the device, and there is no way to disable it. Ubiquiti is dead to me.
Brian, Many companies use Ubiquiti wireless access points for their wireless networks. I imagine these were also affected. This would have serious implications.
I would only see that being a problem if the company was attaching their controller to their cloud system. In my organization, it is not cloud connected. I’m very confused from the article stating that all device passwords need to be changed as that information shouldn’t be in the UBNT cloud. The devices communicate with an on-premise controller, the controller can be accessed by the cloud, but there shouldn’t be any association between the device passwords themselves and the UBNT cloud.
I purchased a Unified Dream Machine (UDM) from Ubiquiti when it was first released. I was horrified that the setup procedure REQUIRED the device to configured with cloud based service run by Ubiquiti.
After using the UDM for about 2 months I was never comfortable with the way their system needed to communicate with their service to function. I performed a factory reset and sold the device on eBay because it was past their allowed return date. I’m glad to have gotten rid of the device.
I also have their Edge Router X, a nice device that has no such requirement.
It was the case, IIRC, that you could NOT disable to cloud access for a while on the Dream Machines, but now you can, once you’ve created a privileged local account.
That’s absolutely the first thing you should do.
I HOPE that Ubiquiti learns from this, and allows these things to be set up without requiring any internet access, or allows you to set up a strong password locally before connecting to the internet.
Ubiquiti used to be be a REALLY great product range between consumer and enterprise, but they caught a bad case of Marketing, and went down the crapper. It was one the case that relatively unsophisticated folks could do a simple, default install of their Unifi kit, and be secure. Not so much now.
Whenever I set up a new Unifi system for my customer, I always have my handy Edge Router X available…
I got rid of it before they changed that option, apparently.
Even so, the fact it was that way at all gave me pause. They must have had enough complaints about that ridiculous requirement.
Additionally, I found the device to be overly confusing and difficult to configure. I went back to my old, but easy to use Apple AirPort Extreme. I wanted 2 independent wireless networks and found it much simpler to just add a 2nd used AirPort Extreme attached to a separate vlan on the trusty Edge Router X.
Much simpler and less expensive too! Probably more secure as well.
> but now you can, once you’ve created a privileged local account.
How do you do this? Just been looking through the interface and can’t see the option, nor the option to disable remote access.
I just bought a UDM five days ago to trial as we’re moving into a new office and I’ve heard good things, but I was stunned to discover this requirement to use an online account to login. If I can’t easily disable it along with remote access, this story is enough to make me return it.
Roberto,
At last check (a few months ago) you can still setup and provision the controller and APs without having to use the cloud-based account. They don’t make it obvious, but the local option is still there, or at least it was quite recently.
Using their cloud based service should be opt-in. I lost their trust long before this problem was discovered. They’re not doing security right.
Not on their CloudKey Gen2 and UDM devices. It is not possible to configure the devices without using a cloud account. It also leaves a permanent Unifi “Owner” account enabled on the device that can authenticate against the Unifi servers and grant access, EVEN IF you disable Remote Access.
Not on their CloudKey Gen2 and UDM devices. It is not possible to configure the devices without using a cloud account. It also leaves a permanent Unifi “Owner” account enabled on the device that can authenticate against the Unifi servers and grant access, EVEN IF you disable Remote Access. There is no way to disable this account.
How are individuals supposed to keep their data safe when corporations won’t be accountable?
Reading some of these people commenting on defense… it’s easy to comment on some people’s ability to defend and find out what happened after the fact when you don’t know the politics of the organization, or how advanced the attackers actually were. Sounds like the attackers knew what they were looking for and what to do. Attacking is easy. You find one hole and get it. Defense is a lot harder because you have to plug all the holes and you have to prove why you need security mechanisms, and management doesn’t always agree because it’s an “acceptable risk” and they have “insurance”. The fact that the team was able to find out what happened after the fact is actually amazing. Some teams aren’t even that good.
Attacking is easy? But enabling multi-factor authentication on a root level AWS administrator account is hard?
Exactly! Did everyone miss this fundamental?
“Ubiquiti had negligent logging (no access logging on databases) so it was unable to prove or disprove what they accessed, but the attacker targeted the credentials to the databases, and created Linux instances with networking connectivity to said databases”. I think this alone is damning. I don’t think you can say they had good logging when they don’t log who accessed their data, which most companies will tell you, is their most valuable asset.
Also some of the wording such as “targeted” and “attacking is easy” is disingenuous. The only thing we know is an IT person got popped and they used those creds for initial access and the AWS root account didnt have MFA.
The creds is par for the course of attacks, but no MFA on an internet accessible highly privileged account, like AWS root, is not acceptable by any organization.
Ubiquiti is well-named: breaches to its products are everywhere. “countless Ubiquiti cloud-based devices around the world”
https://finance.yahoo.com/news/shareholder-alert-ubiquiti-inc-investigated-184800904.html
Mission accomplished
As usual, companies that manage to get all of hardware, software *and* security right are few and far between.
Their hardware is actually quite nice, so taking these devices offline and installing OpenWRT (or something similarly reasonable and non-cloudy) on them is likely to be a very good idea.
Ha ha ha ha.
This on Ubiquiti Wiki:
Other
In 2015, Ubiquiti revealed that it lost $46.7 million when its finance department was tricked into sending money to someone posing as an employee.[16]
It’s interesting that no one is talking about the need for a whistleblower to inform the public. If the security community or even the public accepts zero transparency from a major company like Ubiquiti, then we are all to blame for their failures.
I’m not someone who likes a lot of “rules”, but it seems to me that Ubiquiti and a lot of other large corporations would benefit from regulation around data breaches.
For those who own Ubiquiti devices and/or run Ubiquiti networks and have yet to take action, at the very least you should:
a) Change your Ubiquiti cloud account password (accounts.ui.com)
b) Enable 2FA on your cloud account
Ubiquiti has fairly comprehensive documentation on the different sets of credentials used in Ubiquiti products and services. https://help.ui.com/hc/en-us/articles/204909374-UniFi-Accounts-and-Passwords-for-Controller-Cloud-Key-and-Other-Devices
Unfortunately I don’t think there’s going to be a one-size-fits-all approach to mitigation. It’s going to depend on which devices you are managing and what your environment looks like. Again, at the very least fix-up your UI account and then take a look at that document to see which credentials sets are applicable.
Another great article, keep them coming !
You know, I always looked longingly at those excessively expensive cloud devices Ubiquiti put out, when all I could afford were EdgeRouters and the like, which all use local accounts and local configuration.
Now I don’t feel so bad.
I hope the people who got my passwords on ubiquiti appreciate that those passwords are unique to each website so gfl using them on another website.
tiny typo – missing “not”.
The attackers also provided proof they’d stolen Ubiquiti’s source code, and pledged to disclose the location of another backdoor if their ransom demand was met.
should read:
The attackers also provided proof they’d stolen Ubiquiti’s source code, and pledged to disclose the location of another backdoor if their ransom demand was NOT met.
No. The attackers said they’d tell Ubiquiti where the backdoor was if they paid.
I wonder if anyone at Ubiquiti was fired in November 2020….hmmmmm….
I was immediately suspicious of this company and it’s products as soon as it “hit the scene.” Convenience is often the number 1 selling feature, even if it’s not really there. You either understand what you’re doing or you don’t, lots of white space in your config pages doesn’t change anything.