Yeah. I don’t think anyone else at Signal has ever tried, at least so vocally, to emphasize this definition of Signal as the opposite of everything else in the tech industry, the only major communications platform that is not a for-profit business.
Yeah, I mean, we don’t have a party line at Signal. But I think we should be proud of who we are and let people know that there are clear differences that matter to them. It’s not for nothing that WhatsApp is spending millions of dollars on billboards calling itself private, with the load-bearing privacy infrastructure having been created by the Signal protocol that WhatsApp uses.
Now, we’re happy that WhatsApp integrated that, but let’s be real. It’s not by accident that WhatsApp and Apple are spending billions of dollars defining themselves as private. Because privacy is incredibly valuable. And who’s the gold standard for privacy? It’s Signal.
I think people need to reframe their understanding of the tech industry, understanding how surveillance is so critical to its business model. And then understand how Signal stands apart, and recognize that we need to expand the space for that model to grow. Because having 70 percent of the global market for cloud in the hands of three companies globally is simply not safe. It’s Microsoft and CrowdStrike taking down half of the critical infrastructure in the world, because CrowdStrike cut corners on QA for a fucking kernel update. Are you kidding me? That’s totally insane, if you think about it, in terms of actually stewarding these infrastructures.
…
So you’re saying that AI and surveillance are self-perpetuating: You get the materials to create what we call AI from surveillance, and you use it for more surveillance. But there are forms of AI that ought to be more benevolent than that, right? Like finding tumors in medical scans.
I guess, yeah, although a lot of the claims end up being way overhyped when they’re compared to their utility within clinical settings.
What I’m not saying is that pattern matching across large sets of robust data is not useful. That is totally useful. What I’m talking about is the business model it’s contained in.
OK, say we have radiological detection that actually is robust. But then it gets released into a health care system where it’s not used to treat people, where it’s used by insurance companies to exclude people from coverage—because that’s a business model. Or it’s used by hospital chains to turn patients away. How is this actually going to be used, given the cost of training, given the cost of infrastructure, given the actors who control those things?
AI is constituted by this mass Big Tech surveillance business model. And it’s also entrenching it. The more we trust these companies to become the nervous systems of our governments and institutions, the more power they accrue, the harder it is to create alternatives that actually honor certain missions.
Just seeing your Twitter commentary, it seems like you’re calling AI a bubble. Is it going to self-correct by imploding at some point?
I mean, the dotcom bubble imploded, and we still got the Big Tech surveillance business model. I think this generative AI moment is definitely a bubble. You cannot spend a billion dollars per training run when you need to do multiple training runs and then launch a fucking email-writing engine. Something is wrong there.
But you’re looking at an industry that is not going to go away. So I don’t have a clear prediction on that. I do think you’re going to see a market drawdown. Nvidia’s market cap is going to die for a second.
Security
On 'Simulated Worlds'
in AI, OpenAI, Security, Behavior, Generative AI
OpenAI’s video generation model Sora, in its current iteration, is incredible even though it’ll undoubtedly get better in the coming months. The post they have on their website makes for fascinating reading.
Extending generated videos. Sora is also capable of extending videos, either forward or backward in time.
Long-range coherence and object permanence. A significant challenge for video generation systems has been maintaining temporal consistency when sampling long videos. We find that Sora is often, though not always, able to effectively model both short- and long-range dependencies. For example, our model can persist people, animals and objects even when they are occluded or leave the frame. Likewise, it can generate multiple shots of the same character in a single sample, maintaining their appearance throughout the video.
Interacting with the world. Sora can sometimes simulate actions that affect the state of the world in simple ways. For example, a painter can leave new strokes along a canvas that persist over time, or a man can eat a burger and leave bite marks.
These advancements, alongside how far LLMs and other transformer-based technologies have come in the past few months, have been quite something to behold. While equal parts exciting and terrifying, it’s hard not to think about how and how much they’ll impact industries and society at large. It will likely become harder (and more time-consuming) to sift through what is a genuine advancement and not just another grift (NFTs, anyone?). Art, music, technology, video games, programming, editing, writing, law, disinformation, misinformation, capital markets, cybersecurity, democracy, and medicine will all invariably see some impact. A small part of me thinks that, as amazing as all this is right now, ‘AI’ (quotes intentional) may not be immune to enshittification—not just from the pressure to monetise but also from the unstoppable deluge of low-quality and unimaginative generated content.
On Worldcoin, DAOs and digital identity black markets
in Identity, Blockchain, Crypto, Security
Molly White has a great essay on Sam Altman’s iris scanning orb and its purported use cases.
Much of Worldcoin’s promises are predicated on the questionable idea that highly sophisticated artificial intelligence, even artificial general intelligence, is right around the corner. It also hinges on the “robots will take our jobs!” panic — a staple of the last couple centuries — finally coming to bear. Worldcoin offers other use cases for its product too, like DAO voting, but it is not the promise to solve DAO voting that earned them a multi-billion dollar valuation from venture capitalists.
Other use cases that Worldcoin has offered seem to assume that various entities — governments, software companies, etc. — would actually want to use the Worldcoin system. This seems highly dubious to me, particularly given that many governments have established identification systems that already enjoy widespread use. Some even employ biometrics of their own, like India’s Aadhaar. There’s also the scalability question: Worldcoin operates on the Optimism Ethereum layer-2 blockchain, a much speedier alternative to the layer-1 Ethereum chain to be sure, but any blockchain is liable to be a poor candidate for handling the kind of volume demanded by a multi-billion user system processing everyday transactions.
…
What will happen when you promise people anywhere from $10 to $100 for scanning their eyeball? What if that’s not dollars, but denominated in a crypto token, making it appealing to speculators? And what if some people don’t have the option to scan their own eyeballs to achieve access to it?
A black market for Worldcoin accounts has already emerged in Cambodia, Nigeria, and elsewhere, where people are being paid to sign up for a World ID and then transfer ownership to buyers elsewhere — many of whom are in China, where Worldcoin is restricted. There is no ongoing verification process to ensure that a World ID continues to belong to the person who signed up for it, and no way for the eyeball-haver to recover an account that is under another person’s control. Worldcoin acknowledges that they have no clue how to resolve the issue: “Innovative ideas in mechanism design and the attribution of social relationships will be necessary.“ The lack of ongoing verification also means that there is no mechanism by which people can be removed from the program once they pass away, but perhaps Worldcoin will add survivors’ benefits to its list of use cases and call that a feature.
Relatively speaking, scanning your iris and selling the account is fairly benign. But depending on the popularity of Worldcoin, the eventual price of WLD, and the types of things a World ID can be used to accomplish, the incentives to gain access to others’ accounts could become severe. Coercion at the individual or state level is absolutely within the realm of possibility, and could become dangerous.
On Facial Recognition and Identity Proofing
Wired has a good piece on the IRS in the US caving to public outcry and ditching its integration with ID.me - a service that was supposed to verify identities (by matching video selfies to existing records). It’s understandable why this would cause concerns given that facial recognition is rife with false matches, biases and a reputation for invasiveness. With fraud being a pressing issue now when a majority of us (at least in Australia) access nearly every civic service online, governments are going to want to think about how they balance policy, privacy and messaging.
Unfortunately, the landscape at the moment is messy and populated by a number of third-party vendors still finding their feet in an area where privacy and policy concerns are outweighed by sexier usability and convenience use cases.
“The fact we don’t have good digital identity systems can’t become a rationale for rushing to create systems with Kafkaesque fairness and equity problems.” - Jay Stanley, ACLU
It’ll be interesting to see how Australia’s Trusted Digital Identity Framework (TDIF) will look to address some of these inherent problems through a continuous expansion of its standards.
On web3
in Security, Blockchain, web3, Crypto, NFT
(Note to self)
Moxie has some great thoughts on web3 even though what it really means depends on who you ask.
NFTs:
Instead of storing the data on-chain, NFTs instead contain a URL that points to the data. What surprised me about the standards was that there’s no hash commitment for the data located at the URL. Looking at many of the NFTs on popular marketplaces being sold for tens, hundreds, or millions of dollars, that URL often just points to some VPS running Apache somewhere. Anyone with access to that machine, anyone who buys that domain name in the future, or anyone who compromises that machine can change the image, title, description, etc for the NFT to whatever they’d like at any time (regardless of whether or not they “own” the token). There’s nothing in the NFT spec that tells you what the image “should” be, or even allows you to confirm whether something is the “correct” image.
web3 and Platforms (web2):
The people at the end of the line who are flipping NFTs do not fundamentally care about distributed trust models or payment mechanics, but they care about where the money is. So the money draws people into OpenSea, they improve the experience by building a platform that iterates on the underlying web3 protocols in web2 space, they eventually offer the ability to “mint” NFTs through OpenSea itself instead of through your own smart contract, and eventually this all opens the door for Coinbase to offer access to the validated NFT market with their own platform via your debit card.
At the end of the stack, NFT artists are excited about this kind of progression because it means more speculation/investment in their art, but it also seems like if the point of web3 is to avoid the trappings of web2, we should be concerned that this is already the natural tendency for these new protocols that are supposed to offer a different future.
Clients and Servers in the Decentralised Web:
One thing that has always felt strange to me about the cryptocurrency world is the lack of attention to the client/server interface. When people talk about blockchains, they talk about distributed trust, leaderless consensus, and all the mechanics of how that works, but often gloss over the reality that clients ultimately can’t participate in those mechanics. All the network diagrams are of servers, the trust model is between servers, everything is about servers. Blockchains are designed to be a network of peers, but not designed such that it’s really possible for your mobile device or your browser to be one of those peers.
With the shift to mobile, we now live firmly in a world of clients and servers – with the former completely unable to act as the latter – and those questions seem more important to me than ever. Meanwhile, ethereum actually refers to servers as “clients,” so there’s not even a word for an actual untrusted client/server interface that will have to exist somewhere, and no acknowledgement that if successful there will ultimately be billions (!) more clients than servers.
The increasing complexity of creating software:
At this point, software projects require an enormous amount of human effort. Even relatively simple apps require a group of people to sit in front of a computer for eight hours a day, every day, forever. This wasn’t always the case, and there was a time when 50 people working on a software project wasn’t considered a “small team.” As long as software requires such concerted energy and so much highly specialized human focus, I think it will have the tendency to serve the interests of the people sitting in that room every day rather than what we may consider our broader goals. I think changing our relationship to technology will probably require making software easier to create, but in my lifetime I’ve seen the opposite come to pass. Unfortunately, I think distributed systems have a tendency to exacerbate this trend by making things more complicated and more difficult, not less complicated and less difficult.
On the SolarWinds Breach
Where to begin. It’s almost impossible to comprehend what the fallout of this breach will be in the immediate to medium term; in fact, there isn’t enough information out there yet to conduct an effective post-mortem so to speak.
One thing is for certain - organisations are going to be wary of trusting ‘technology solutions’ from vendors. This isn’t to say that SolarWinds (and FireEye) did not have adequate measures in place; just that breaches are inevitable and organisations that rely on technology vendors are also dependent on these vendors having adequate controls in place alongside stringent self-audits. Organisations out there that trusted SolarWinds to push Orion to their networks in effect trusted that SolarWinds had a strong handle on their security posture. In Australia, CPS 234 mandates that APRA-regulated entities will need to have information security measures in place and includes cybersecurity assessments by independent assessors. Obviously, the effectiveness will boil down to the thoroughness of the assessor.
All this gets more complicated when you start to look into how the breach occurred in the first instance. It is starting to increasingly seem like hackers leveraged widely-used protocols and solutions. Ars Technica has a fascinating report referencing security firm Volexity, who encountered the same attackers in 2019; at the time, they bypassed MFA protections for Microsoft Outlook Web App (OWA) users.
Toward the end of the second incident that Volexity worked involving Dark Halo, the actor was observed accessing the e-mail account of a user via OWA. This was unexpected for a few reasons, not least of which was the targeted mailbox was protected by MFA. Logs from the Exchange server showed that the attacker provided username and password authentication like normal but were not challenged for a second factor through Duo. The logs from the Duo authentication server further showed that no attempts had been made to log into the account in question. Volexity was able to confirm that session hijacking was not involved and, through a memory dump of the OWA server, could also confirm that the attacker had presented cookie tied to a Duo MFA session named duo-sid.
Krebs on Security has a great write-up with a sobering quote from the DHS’s Cybersecurity and Infrastructure Security Agency.
CISA’s advisory specifically noted that “one of the principal ways the adversary is accomplishing this objective is by compromising the Security Assertion Markup Language (SAML) signing certificate using their escalated Active Directory privileges. Once this is accomplished, the adversary creates unauthorized but valid tokens and presents them to services that trust SAML tokens from the environment. These tokens can then be used to access resources in hosted environments, such as email, for data exfiltration via authorized application programming interfaces (APIs).”
The CISA goes on to advise that if an org identifies ‘SAML abuse’, mitigating individual issues might not be enough; you’ll need to consider the entire identity store as compromised. And unfortunately, the only remedy to that is building back identity and trust services from the ground up.
Additional Reading:
VMware Flaw a Vector in SolarWinds Breach?
SolarWinds hackers have a clever way to bypass multi-factor authentication
It Takes Two (To Thwart Data Breaches)
Some interesting insight from Gemalto's 2017 Data Breaches and Customer Loyalty Report:
- Of the 10,000 consumers interviewed, only 27% feel businesses take customer data security seriously
- 70% would take their business elsewhere following a breach
- 41% fail to take advantage of available security measures available such as multi-factor authentication
- 56% use the same password for multiple online accounts
While consumers are rightfully skeptical of the security hygiene of businesses they interact with, there is certainly a role for consumers to play here.
Krebs on IoT Vulnerabilities
Brian Krebs has some interesting insight into this past weekend's DDoS attack on Dyn, an internet infrastructure company that provides services for some of the web's biggest destinations including Twitter, Amazon, Reddit and Netflix.
At first, it was unclear who or what was behind the attack on Dyn. But over the past few hours, at least one computer security firm has come out saying the attack involved Mirai, the same malware strain that was used in the record 620 Gpbs attack on my site last month. At the end September 2016, the hacker responsible for creating the Mirai malware released the source code for it, effectively letting anyone build their own attack army using Mirai.
Mirai scours the Web for IoT devices protected by little more than factory-default usernames and passwords, and then enlists the devices in attacks that hurl junk traffic at an online target until it can no longer accommodate legitimate visitors or users.
...
The wholesalers and retailers of these devices might then be encouraged to shift their focus toward buying and promoting connected devices which have this industry security association seal of approval. Consumers also would need to be educated to look for that seal of approval. Something like Underwriters Laboratories (UL), but for the Internet, perhaps.
Until then, these insecure IoT devices are going to stick around like a bad rash — unless and until there is a major, global effort to recall and remove vulnerable systems from the Internet. In my humble opinion, this global cleanup effort should be funded mainly by the companies that are dumping these cheap, poorly-secured hardware devices onto the market in an apparent bid to own the market. Well, they should be made to own the cleanup efforts as well.
The upside here is that IoT manufacturers and vendors will now have to wisen up to the fact that they have more to gain from secure devices and a lot to lose from a repeat of this weekend's events.