When are verified words unverifiable?
On Facebook's revenue streams, secret family recipes, and the treacherous territory of verifying chatbots as "official"
Facebook is encouraging users to "chat" with an artificial intelligence model in the character of Jane Austen. It's part of a fairly transparent effort by the site to get users to spend as much time as possible engaging with the platform, which is important to a company that derives about $50 in revenue per North American user per quarter.
■ The first obvious question is: Why Austen in particular? But close behind it is: Why does this AI model have a verification symbol that is the same as what Facebook applies to real, living celebrities? It could easily be designated differently, but it is not.
■ What is the purpose of a verification mark for an AI model representing a real historical figure, if not to create at least some synthetic appearance of (unearned) authority? That's a bold and dangerous move for Facebook. Today, perhaps it's merely Jane Austen. But what's to stop them from doing the same thing tomorrow with the synthesized words and likeness of George Washington? Or Aristotle? Or Jesus?
■ AI models based upon real people have been a fairly evident "next step" for at least a decade now. The entire history of biography -- and family lore -- is about reaching into the past to seek answers for the present. It's one of the most obvious use cases for artificial intelligence. Like search engines, these models retrieve and reconstruct material from databases, so they really ought to be called "personality engines". And they most likely will prove too irresistible to have around in one form or another, indefinitely.
■ What people don't fully appreciate yet is just how little source content it will really take to form a personality engine for just about anyone. With enough willingness to let computers fill in the blanks and make forecasts based upon incomplete information, one could probably synthesize enough material to look like a defensible worldview from about 200 to 300 pages of written text.
■ That just isn't very much source text to ask! But it's treacherous territory, particularly if the model-builders aren't extremely careful about what they use for input material, how they label it, and what they do to make sure that newly-synthesized material generated in the "voice" of a particular individual doesn't become the source material for another personality engine that doesn't know the difference between the original material and what's post-canonical.
■ And it truly begs us to consider the ramifications of co-opting someone's "voice" without their consent. Jane Austen never said Facebook could use her as a chatbot -- she died in 1817. Modern audiences may be greedy to get her advice today, but whatever we get shouldn't be considered "official" in the sense that most audiences would consider material published by a "verified" account.
■ For now, it may seem harmless to use Jane Austen like this. And it will probably seem mostly harmless to take Grandma's private diaries after she passes away and submit them as source material for a Grandma-AI (this is absolutely certain to be someone's business model sooner rather than later). But who controls whether Grandma-AI is released for public distribution, and who rakes in the earnings from her words and likeness? Families already go to legal battle with one another over real property, nest eggs, and secret recipes. Who controls Grandma's synthesized words and likeness when they could be worth money in the commercial market?
■ We've entered extraordinary territory -- completely uncharted -- and it's not clear that significant participants have even begun to duly consider the consequences. A great deal of good could come from plumbing the sources of the past for answers in the present, but we need to put as much energy into the boundaries as we invest in charging for the frontiers.
