Episode 291 - Publishers vs Librarians, Artificial Digital Scarcity, and Revisiting Bertrand
Max discusses the legal troubles of the Internet Archive as it get pummeled by publishers and copyright lawyers. A new paper on Bertrand's Paradox claims to resolve it.
Links
New York Times - The Case of the Internet Archives vs. Book Publishers
MDPI - Bertrand’s Paradox Resolution and Its Implications for the Bing–Fisher Problem Returns with Priors
Digitopoly - "Information Wants to be Free": The history of that quote
Fight For The Future - 1000+ AUTHORS FOR LIBRARIES
The Authors Guild - AG Sends Open Letter Demanding “National Emergency Library” Shut Down
New York Times - Publishers Sue Internet Archive Over Free E-Books
Archive.Org - Announcing a National Emergency Library to Provide Digitized Books to Students and the Public
DNYUZ - The Dream Was Universal Access to Knowledge. The Result Was a Fiasco.
Related Episodes
Episode 260 - New Year Predictions: Generative AI, Self Driving Cars, and Web3
Episode 207 - Max Returns with Priors
Transcript
Max Sklar: You're listening to the Local Maximum episode 291.
Narration: Time to expand your perspective. Welcome to the Local Maximum. Now here's your host, Max Clar.
Max: Welcome everyone. Welcome. You have reached another Local Maximum. Sorry for the late hour and late timing of this particular episode. We’re gonna first we're gonna talk about an article that made it into the New York Times about the Internet Archive, which seems a little obscure, but it's very interesting to me, as you'll see. And then we're gonna move on to Bertrand's Paradox in math. And the reason why I was delayed is because I was trying to get through this paper, which is an academic paper. It's always tough trying to get through an academic paper for the podcast. Sometimes very much worth it though. So let's get started. The article was the dream was universal access to knowledge. The result was a fiasco in the Pandemic emergency. Brewster Kale's Internet Archive freely lent out digital scans of its library. Publishers sued. Owning a book means something different now.
This is particularly interesting to me because I love this service of the Internet Archive. It's a site that it should be more well known, honestly. It should be in your repertoire. Maybe not on the level of Google or Twitter, but somewhere around there. Maybe on the level of Wolfram Alpha or something like that, which is computational engine.
It's extremely useful and it also has a physical location. It's close to where I stayed with relatives when I went out to San Francisco periodically, back when I worked for Foursquare in the inner Richmond district of San Francisco, which is on Clement Street, right next to the Park Presidio Highway. Very nice area.
Come to think of it, the last time that I was in San Francisco at all and probably the last time I walked by this place was April 2018. So over five years ago. Time really does fly post COVID I've heard that parts of San Francisco have gone downhill quite a bit. So we'll see what I observe if and when I go back, though I have no plans to. There were some areas that were a bit sketchy when I was there. When you're not dealing with the drugs and the crime though, it's a beautiful city. When you look up, they can't take that away from you. At least not for a while.
Anyway, back to our discussion on the Internet Archive. I remember first passing by that really interesting looking building. Well, in 2012, the first time I went there, but in 2013 was the second time I went to San Francisco a year later. And the building is really interesting. I think it used to be a Christian Science church. It's like kind of looks like what's the style of the White House? Old columns and all that. And there was this memorial for the death of Aaron Schwartz and Aaron was a young internet entrepreneur. He founded Reddit, one of the founders of Reddit, he was very active in the Free Information movement and the push to stop some of these egregious acts by Congress and they're lobbyists to close off the Internet.
So I actually met him in a meeting about the Stop Online Piracy Act in 2011, which they effectively blocked, that was going to give the copyright lawyers so much more power. So I believe his efforts delayed these measures quite a bit. But now if you delay something ten years, it's great, but it's over ten years later now, in 2023, it's post COVID, post heavy Internet regulation that really started overseas with GDPR, but came for all of us, ultimately. Post conglomerate, big tech social media with their widespread censorship and their rough propaganda campaigns, which I believe you could probably put the start on that in 2018.
We talked about this on the show when they kind of stomped out Alex Jones and everyone's like, Alex Jones, who cares? But when you got to 2020, you couldn't say anything. You could say some things, but it was very heavily censored. I think we've hit the maximum on this, and I think that the turning point was, in retrospect, will be the takeover of Twitter by Elon, but we'll see.
We're also post politicization of the justice system, intelligence apparatus, all this in America. Let's be honest, we're kind of living through an authoritarian moment in history, although we can still talk and do podcasts, so that's kind of nice. But a lot of it is like, hey, the people be damned. So all of the stuff that we want to do with Stop Online Piracy Act, even though nobody wanted it and the people didn't want it, who cares? Stuff has just returned with a vengeance as the powerful have kept pushing and pushing and pushing in Washington and the Internet Archive.
So it is kind of disappointing that a lot of this stuff has been I think that what the powerful wanted to introduce, they got to introduce. Anyway, I haven't kept up on the laws of this country that much, so if someone wants to correct me, let me know. But that's what it seems like. So the Internet Archive. What's that? The Internet Archive saves the state of the Internet by taking snapshots of websites periodically. At least it's one of the things they do. So you look at different websites and you could see what they look like in the past, and you could see what certain websites who have been around all the way going back to the 90s, even localmaxradio.com the website for the show, or stickymap.com, which was my first kind of location based site. I'm sure it all has it.
So in some sense, stuff on the Internet does live forever, particularly if it's on static sites that can easily be crawled and it's not behind a paywall and it gets archived. So the stuff on Internet Archive might not be searchable through Google or anything like that. You can't search for a phrase that has been taken off the site, and you wouldn't get the link to the Wayback Machine, which is what they call it at Internet Archive. But you can see what a certain website said in the past. Very useful sometimes for websites that are changing all the time. Like you can see what the cover of the New York Times was or whatever.
So Brewster Kale runs the Internet Archive, a venerable tech nonprofit. Let's quote from the article here in that miserable, frightening first month of the COVID pandemic, he had the notion to try to help students, researchers and general readers. He unveiled the National Emergency Library, a vast trove of digital books mostly unavailable elsewhere, and made access to it a breeze. This good deed backfired spectacularly, four publishers claimed willful mass copyright infringement and sued. They won.
On Friday, the publishers said through their trade association that they had negotiated a deal with the archive that would remove all their copyright books from the site. The proposed judgment is an appropriately serious bookend to the court's decisive finding of liability, said Maria Palente, chief executive of the association of American Publishers. We feel very good about it. The archive had a muted response, saying it expected there would be changes to its lending program, but their full scope was unknown.
There was also an undisclosed financial payment if the archive loses on appeal. The case has generated a great deal of bitterness, and the deal, which is immediately approved by the judge, is likely to generate more. Each side accuses the other of bad faith and calls its opponents well funded zealots who won't listen to reason and want to destroy the culture. In the middle of this mess are writers whose job is to produce the books that contain much of the world's best information. Despite that central role, they are largely powerless, a familiar position for most writers.
Emotions are running high. 6000 writers signed a petition supporting the lawsuit, and a thousand names are on a petition denouncing it. The Romance Writers of America and the Western Writers of America joined brief in favor of the publishers, while the Authors Alliance blah, blah, blah. It's rarely this nasty. But free versus expensive is a struggle that plays out continuously against all forms of media entertainment. Neither side has the upper hand forever, even if it's sometimes it seems it might.
Quote: “The more information is free, the more opportunities for it to be collected, refined, packaged and made expensive,” said Stuart Brand, the technology visionary who first developed the formulation. “The more it is expensive, the more workarounds to make it free. It's a paradox. Each side makes the other true.”
That last quote is very interesting. I want to explore what Stuart Brand says a lot more, and it's something to think about as we head into the world of large language models and generative AI, which in some ways you could say it's not creating new information, but it's reorganizing old information. So what's the right business model here? I personally hope that someone finds a model that makes good content and information highly available, and it looks like the collection of free stuff is the way to do it.
But how do you compensate the original content creators? Very interesting question. I personally, I used to love using these desktop apps like Napster and Kazaa way back 20 years ago, like when I was in college, as I'm sure the people who are my age will remember, you could download free music, lots of free music. And apparently I still have my music collection from that. Apparently that wasn't supposed to be allowed. The music industry didn't like that.
But today all music is highly available and the prices are extremely reasonable. So I wouldn't call the loss of these things a total loss. But some other types of content, non music just hasn't worked out in the same way. If I want a book or a textbook and I want to check something in it, do I have to pay the $20? Do I have to pay hundreds of dollars for a textbook just to look something up? Just to look at a particular chapter? It seems crazy. If I wanted to peek at a chapter, maybe I could do it. Maybe I'd do it if I could pay fifty cents and get it. But that's just not available.
So the models they have, I think, are not effective. That whole micro payment thing is just one example. But something is wrong here. So continuing with the article, universal access to all knowledge was a dream of the early internet. It's an idea that Mr. Kale has long championed. As the United States lurched to a halt in March 2020, he saw an opportunity.
The Internet Archive would be a temporary bridge between beleaguered readers and the volumes shut away in libraries and schools who weren't allowed to go to libraries, weren't allowed to go to schools. There are a lot of places we weren't allowed to go to. Don't even bring that up right now. It didn't turn out that way, not a bit. The emergency library shut down in June, 2023.
Years later, Mr. Kale remained angry and frustrated. So skipping ahead, librarians are custodians. Mr. Kale has spent his career working in tech, but he wants the future to behave a little more like the past. Quote: “If I pay you for an ebook, I should own that book,” he said. “Companies used to sell things, media companies now rent them instead. It's like they have tentacles. You pull a book off the shelf and say, I think I'll keep this, and then the tentacle yanks it back.”
So that's interesting. I got really into ebooks. I had an ebook reader going back to 2013, even earlier, probably 2010 during grad school, and it really got me into reading. It really got me into books, particularly when I had a really tiny apartment in New York City. That one room in the East Village.
I had a small bookshelf, but I didn't have a whole lot of room for extra books. And if I did put an extra book, it would kind of be a burden for me. That's not so much true anymore. But this might be why I've been a binge on buying physical books recently. Not just a little more space, but when you buy a physical book, you own it. You could give it away. You could mark it up. You could do whatever you want with it. You can do some of that with digital books temporarily. Hey, when you own a digital physical book, you even make copies of individual pages, take photos or whatever.
Maybe it's not technically legal, but if you do it for individual pages, nobody's going to bother you. I don't know, maybe that's not technically allowed. All right, scratch that. The digital medium, though, is different because the ownership is not quite there. And it's never quite there unless you literally own the file. But then it's harder to maintain. I mean, I have files that I've kept for many, many years, but there's sort of a cost to maintaining that that doesn't exist for a book. So you're creating artificial scarcity of a good with these ebooks that might have a very high demand.
Not only that, I'm also worried that they're going to change digital books. They're going to give it updates. Okay, it's fine to correct some spellings some errors, but there's also a portion of it where it's like erasing the past because it's inconvenient to our current ideology. So keep some physical books.
The article that I'm reading here also has an amazing statistic, which I had no idea. This one came out of the blue, and it kind of blew my mind. When Netflix was giving out DVDs, I would think that Netflix has more content available now in digital because DVDs, it's like, how many DVDs you have lying around.When Netflix was giving out DVDs that had 100,000 movies to choose from, now they have 6000. In other words, they lost 94% of their content. That's because of all the laws surrounding digital media. That's an interesting statistic. It kind of seems suboptimal if they produce far less content than the system before with physical medium, with DVDs that no one was complaining about in terms of content, property ownership. So that's kind of fascinating.
Although, well, no, you didn't have to pay per DVD, but you could only get a few per month. On the other hand, when Netflix was giving out DVDs, it's not like people watch more movies now. How many movies can you watch? I probably watched more movies then. How many movies can you watch in a month? I mean, maybe some people watch movies every day when they have Netflix. And then because you had to mail the movie back in and get a new one. They were watching less.
But three, four movies a month, that seems kind of high anyway. So, I don't know, it seems like they're artificially restricting the amount of movies and digital videos that people can watch because of this. Okay. The Times continues. This is a very long article. I would suggest, if you're interested, to read the whole thing, but I'm going to read some clips of it here.
Some necessary background for The New York Times when a physical book is sold, the first sale provision of copyright law says the author and publisher have no control over that volume's fate in the world. It can be resold. They don't get a cut. It could be lent out as many times as the readers demand. The information in the text flows freely through society without leaving a trace. Religions and revolutions have been built on this.
Thanks to their digital nature, ebooks are treated much differently. They can't be resold or given away. A library that wants to lend ebooks must buy a license from the copyright holder. These subscriptions can be limited to a number of reads or by periods of a year or two. Everything is tracked. Libraries own nothing.
Skipping ahead. The Internet Archives Lending program, developed long before the Pandemic, involved scanning physical books and offering them to readers in its open library, a practice called controlled digital lending, you're artificially creating scarcity. One reader at a time could borrow each scanned book. If the library or one of its partners had two copies, then two readers could borrow it. So you're kind of making kind of a fake digital scarcity based on the physical scarcity that exists. The archive defended making its own ebooks by citing fair use, a broad legal concept that permits copyrighted materials to be quoted in excerpt. And the first sale doctrine? It could do what it wanted with its own books.
No dice, wrote Judge John Cotle of the US. District Court in Manhattan. His decision granting a summary judgment for the publishers in March, went far beyond the Pandemic library. Any benefit for research and cultural participation, he said, was outweighed by harm to the publisher’s bottom line.
So interestingly, you can't even lend out these books in a very constrained way based on the physical scarcity. That's not good enough for them. So my question is, hey, how does the judge weigh that? How does he know? Like, oh, the benefit for research and cultural participation is clearly less than the harm to the publisher. How does he measure that? How does he know that? And why does the judge have the authority to determine who benefits more? You'd have to read the decision. He might be just finding justifications for existing laws, but who knows?
This is an amazing quote that I found in this article, very deep in the article. Listen to this. Quote: “Digital is different than print because it is infinitely copyable and unprotectable,” said Mary Rasenberger, the chief executive of the Guild and copyright lawyer. Quote again: “If anyone could call themselves a library, set up a website and do the exact same thing the archive did, writers would have absolutely no control over their work anymore. Traditional libraries promote discovery, but publishers perennially worry that they cost sales. Most publishers are not purely profit driven,” Miss Rasenberger said. “If one were, you could imagine it might not allow libraries to have e-books at all.”
Oh! Oh, thank you very much, kind publishers, for not shutting down all the public libraries in the country. Truly out for the little guy. And I know she was talking about just digital books here. I think they're like, okay, fine, paper books, but the arguments that they're making here goes for paper books as well. It's like, oh, we're not purely out for money! So long as we leave one library standing, we're being nice to you. Because I could read stuff without paying for it at the library. I do that all the time. I don't want to buy a book I read at the library. Sometimes I skim a book at a bookstore, like Barnes and Noble or something like that. There's nothing you could do about it.
I mean, I suppose Barnes and Noble could do something about it, but they tend not to because they tend to like having people like me in that library because occasionally I buy something. So you just want to make me get up and walk four blocks to the library to look at a book rather than having it right here at my computer. You probably prefer I don't have access to the library at all, but you are such nice people that you're going to allow the library to stand.
Okay, interesting quote. What do you guys think about it? maximum.ocals.com, I'd like to know. The article goes I thought that was the most interesting quote of the article. You can tell me if you agree or not.
The article goes on to quote a variety of opinions from authors, and there's, like, several paragraphs on this. Some of the authors are very pro-Archive and anti-publisher. Some are pro publisher, anti archive. And I get it. As a writer, you want to be able to earn a living through writing books that people want. If you write a book that's in high demand, you want to be able to make a lot of money on that, which I think you should be able to. I think you should be able to earn money that way. I think even if the books are easily available for free, you could still earn money that way.
But that's not necessarily the way you have to do it. I'm not saying that your book needs to be everywhere for free, but there does have to be some allowance for free trade of information, and there has to be the ability to innovate on the system for monetizing these books, because the system is clearly crazy. The fact that you have to pay for the whole thing digitally to just look up a single section of it, the fact that there's kind of like an artificial scarcity being imposed that sort of measures physical scarcity, which seems like, why would that be the optimal thing to do?
You're just kind of trying to mimic what was true before based on the physical properties of books, which doesn't make sense. And some of this could be very good for writers. Writers have a monopoly on their own content. Maybe this is a little bit of price discrimination. Students and such who don't have a lot of money to begin with could get it for free, and then when they're older, they can pay for it, while older and wealthier readers maybe can pay extra to get it in a more convenient form.
I mean, I know I'd pay to have a book. If I really wanted a book, I want a physical copy and I want a digital copy both to be available, because then I can have the physical copy on my shelf, but then if I'm traveling or whatever, I can have the digital copy available for me always. That seems like a good plan.
So right off the bat, there are plausible business models to experiment with, which sounds like the publishing industry is not going to allow it's. Not really the publishing industry. It's also like our legal system and our government and all that. So they've also gone after some of the music recently stored on the Internet Archive. I should say not music that has recently been stored on the Internet Archive. Music that has been stored in the Internet Archive for a long time, but lawyers have recently gone after that. Quoting from the article again, in the wake of the publisher's success, other parts of the Internet Archive have become a tempting target.
Universal, Sony, Arista and other music companies sued the archive in New York on Friday, saying it unabashedly seeks to provide free and unlimited access to music for everyone, regardless of copyright. The plaintiffs cite 2749 violations, 2749 violations, all recorded with an antiquated format used before 1959, for which they are asking $150,000 each.
Quote: “Now the Washington lawyers want to destroy a digital collection of scratchy 78 rpm records, 70 to 120 years old, built by dedicated preservationists in 2006,” Mr. Kale said. “Who benefits?” So, interesting article. This is a fitting one for me to talk about today because I was just in New Haven, and I happened to walk into Binicky library of rare books and manuscripts, beautiful library. And in it I saw the Gutenberg Bible, the first surviving book published with Movable type, with the Gutenberg Machine. I saw books that were handwritten by scribes from the 1400s,old encyclopedias, the works of Seneca, all that.
Now you can't go in and touch the physical books. You see them behind glass, which I'm thankful for because this stuff is so important. And I'm always worried for some reason, I have this anxiety. I'm always going to spill stuff on something, I'm going to get water on it. I don't know, I get because I destroyed so much of my own stuff that way when I see very rare and precious and expensive stuff, I have this anxiety. So I'm kind of happy they put it behind glass.
But it turns out because I was thinking, like, what's actually in these books? Turns out Yale actually makes digital copies of many of these works accessible to the public on their website. And you could just go and look at them. Maybe you can get some commentary on them. Some of them are written in very old languages, so we can't read them, but you have digital copies of them available.
Now fortunately, there's no Scribe Guild from the 14th century who are going to show up and sue everyone. But it kind of makes you think, I'd love to be presented with some good solutions to our problems with digital and physical print media today. I know we're not doing it right. So even if you want to get rid of all these laws, which I know many libertarians support, I do too, get the lawyers out of it. You still need a business model. What is that? Hopefully a good topic for another day. Or a good ongoing topic on that.
All right, so real interesting, localmaxradio@gmail.com if you want to weigh in through email, check out the locals, maximum.locals.com.
Now, I recently reviewed, or I'm going to review now, a new paper on the Bertrand Paradox that came across my desk. It's on MDPI, a math journal. Bertrand's Paradox resolution and its implications for the Bing-Fisher problem. The author is Richard Chechile. Richard, I hope I'm pronouncing this right. So we've spoken about the Bertrand Paradox on here several times. Episode 207 Max Returns With Priors, that was a good episode because it was the one where I was back after being away for or after being sick for a while, and it turns out really well.
It was like with Aaron at one in the morning, and we talked about the problem of assigning probabilities to things where you don't have a whole lot of data or any data. What's the uniform distribution? And we talked about this again in episode 260 where we did probability distribution of the week, and we literally talked about the uniform distribution and we talked about the Bertrand Paradox. So it's a really interesting question. When it comes to assessing prior probabilities, how likely do I think different events are?
So to review what Bertrand was trying to do, he wasn't trying to do this because he really needed to do this. He was trying to do this because he was trying to prove a point, was that he was trying to ask, well, what is I want a random chord from a circle. A chord is a line between straight line between two points on a circle. I want to just take a random one. What does it mean to have a random one?
Well, he gave three methods for picking a random chord. That all seem reasonable, but they're all different. They all give you different answers. They'll make different chords more likely or less likely, depending on which method you use. The first method is the radial method, which you kind of start in the center and then you can look 360 degrees all around you and you pick a random radius. And so that's a random direction from the center. Then that kind of creates a line from the center to the edge of the circle. Okay, great.
By the way, as we've also said on the program, usually a distribution around a circle, the uniform distribution around a circle is not the contentious one because a uniform distribution around a circle is pretty well founded. So, okay, so that's good. So you have got this radial method and then you have a line between the center of the circle and the edge of the circle. Then you pick a random point on that line that's kind of uniform. All right? And then you have a random point inside the circle and that becomes the midpoint of your chord. You kind of draw a line that's perpendicular to that, that becomes your chord, that's method one.
The second method is you basically pick a random angle. You basically pick two random points on the circle and then connect them. That's the chord. Then the third one is you actually pick a random point in the circle and it's going to be kind of like, how should I put it? Uniform within the circle.
So it could be any point in the circle uniformly based on the rules of two dimensional space. And then that point becomes the midpoint of the chord that you draw. So these are three very different methods. Interestingly, the random point in the circle method, the third one is related to the first one. It's just the radial method, but it's using a different coordinate system. So for those of you who remember coordinate systems from high school, one is using the Cartesian coordinates and the other is using the polar coordinates. And you're randomizing, you're taking like uniform randomness from each one.
But the issue with that is that because you've rewritten the coordinates, your uniform distribution on each one are going to be different from each other because the whole number system is off. That's kind of the point here. That what is uniform depends on your point of view.
So what does this new paper say? I've been trying to follow it and I'm hoping to get some follow up on this because I can't quite follow the argument yet. I'd have to talk to the author. But they want to choose a. Random chord length, and they kind of find a fourth way for Bertrand to build this chord. So what they do is first they start by picking a radius or point on the circle, just like what they did before.
But instead of picking a point between the center and that point on the circle, that random point in the circle, is the start of your chord. Then where does the chord end? Well, you have a random point in the circle right now. It could be a very small chord, it could be like right next to it, or it could be a very long chord. In fact, the longest the chord can be is a length two if it's a unit circle. So that would be the chord that is actually a diameter. It goes through the center of the circle. So they say, okay, well, the chord is between length zero and two. And they argue that they actually want a uniform distribution between the length of the chord.
So it could be anywhere between zero and two. We're going to choose uniformly between that. And then, of course, if you have a chord that's of length one, there's one that you could have. It go in two directions. You pick randomly based on that.
So this is an interesting question. I think this is indeed a fourth logical solution to the other three Bertrand results. But part of me is thinking maybe not, because there's no reason why I can think of why the chord length should be the main variable, why the chord length should be the one that should be the uniform. Because it seems to me there's a good argument to be made that you're much less likely to get the full chord near two, the full diameter, than somewhere in the middle.
However, first, I'm going to declare this to be a reasonable answer, for sure, but this paper goes beyond that. It argues that this is the best uninformative answer. The paper argues that, yes, chord length is the variable you want to apply maximum entropy to. Chord length is the variable that you want to be indifferent to. That is the one that you want to have a uniform distribution on, because that's the coordinate system we care about, essentially.
So, unfortunately, I don't quite understand the argument. Maybe I'll have to have the author on, but for the same reason, the author, I believe, wants uniform distribution on probabilities. So when you're picking a probability between zero and one, what's a random one? They want a uniform one. That I think that's what he's talking about with the Bing-Fisher problem. We've gone over on the show reasons not to do that.
So what do you think? Do you think this random chord length idea solves the problem of Bertrand Paradox? For those of you who are mathematically inclined, check out the paper and let me know on maximum.locals.com or localmaxradio@gmail.com.
All right, that's all we have for this week. I'm going to have Aaron on later this week and we're going to put it out on Monday. Finally we're going to get back to Monday and we're finally going to do a news update, which we haven't done in a while because we have a lot of that. And I'm going to solicit some more guests. And I have a lot of guests in mind that have come up, but I'm a little behind my emailing of them, so we'll see how that goes.
But we have a lot of news to catch up on and a lot of topics to catch up on, so I hope I get in touch hope I can get in touch with Aaron soon. And of course, we've got a lot of those are the three things articles, current events, topics, and guests. Always nothing in particular. Well, there are certain things in particular in the pipeline, but nothing to report as of now, so I'm going to leave it at that and look forward to Aaron next week. Have a great week, everyone.
That's the show. To support the Local Maximum, sign up for exclusive content and our online community at maximum.locals.com. The Local Maximum is available wherever podcasts are found. If you want to keep up, remember to subscribe on your podcast app. Also check out the website with show notes and additional materials at localmaxradio.com. If you want to contact me, the host, send an email to localmaxradio@gmail.com. Have a great week.