Yale Image Finder is a search engine for searching medical articles in PubMed Central for images. YIF is notable because it searches for text that is contained in images, many of which are charts and graphs with embedded “text” describing the data being presented. The “text” in these images, as in the example from YIF below, is converted to searchable OCR text.

What especially strikes me about this project is how similar it is to several initiatives from Google — For several years, Google has been working on image-to-text conversion in various of its facets, starting with Google Catalogs (now defunct) and Google Book Search. More recently, in 2008, several patents were published which extend the potential use of this sort of technology to a variety of possibilities, some of which include use in Google Maps street view, labels in museums and stores, and YouTube videos. Also showing Google’s continuing interest in this area is the announcement in Oct, 2008 that scanned PDF documents in Google Web Search are being converted to OCR text format.

Yale Image Finder was first announced in August, 2008, so it’s surprising that I have not been able to find anywhere (including a scholarly description by the developers) that it’s been connected to the initiatives by Google, which seem to be so similar. The same sorts of expressions of awe and amazement that have been expressed about the Google initiatives apply equally well to the Yale project, so I’m excerpting several of these commentaries below, all written in January, 2008, when the latest patents from Google inventors Luc Vincent and Adrian Ulges were published …

Bill Slawski, who has written several articles on Google image-to-text patents – Google on Reading Text in Images from Street Views, Store Shelves, and Museum Interiors :

One of the standard rules of search engine optimization that’s been around for a long time is that “search engines cannot read text that is placed within images.” What if that changed?

Here’s more from Slawski – Googlebot In Aisle Three: How Google Plans To Index The World? :

It’s been an old sawhorse for years that Google couldn’t recognize text that was displayed in images while indexing pages on the Web. These patent filings hint that Google may be able to do much more with images than we can imagine.

Duncan RileyGoogle Lodges Patent For Reading Text In Images And Video :

I may be stating the blatantly obvious when I say that if Google has found a way to index text in static images and video this is a great leap forward in the progression of search technology. This will make every book in the Google Books database really searchable, with the next step being YouTube, Flickr (or Picasa Web) and more. The search capabilities of the future just became seriously advanced.

Of course — sorry to keep harping on it! — as much as recognizing text in pictures would be a great advance, the REAL advance, of recognizing the actual objects in pictures, the philosopher’s stone of image search, still seems far from happening.

Please comment here or Twitter @ericrumsey

The slides and data from Jon Orwant’s presentation on Google Book Search at TOC, that were not available when I wrote previously, have now been put up on the O’Reilly site. [these have been removed, see comment below] This is made up of 59 PDF slides, covering a range of recent developments with Google Books, including the recent release of GBS mobile, and a discussion of the Oct 2008 Publisher settlement. The part I’m most interested in is the data on GBS usage that had been mentioned by Orwant in various venues before, but with few details. The details in the TOC presentation are mostly in three “Case studies” of publishers that participate in the GBS Partner Plan — McGraw-Hill, Oxford University Press, and Springer. I’ve chosen one slide for each of these publishers that show various long-tail effects for usage of their books that are in GBS, and one slide that has data for a more extensive grouping from GBS.

McGraw-Hill case study is presented in slides 21-23. Below is slide 24. Note that this is a small sample of only the top 30 titles.

Oxford University Press – Slides 26-31. Below is slide 27. Note the long tail of visits for pre-1990 books.

Springer – Slides 32-36. Below is slide 35, showing clicks for Buy this Book. Note again the very long tail of clicks for pre-1995 books.

Slide 37 below shows “Share of books with more than 10 pages viewed”, apparently for all books in GBS. The coloring of the data lines looks ambiguous to me – The lowest line is undoubtedly for Snippet View books. It looks like the top line is for Limited Preview, which are presumably higher than Full View books, apparently the middle line, because Limited Preview books are more current.


Please comment here or Twitter @ericrumsey

Jon Orwant, from Google Book Search, made a presentation at the O’Reilly Tools Of Change (TOC) for Publishing Conference in New York last week, which I did not attend. Apparently Orwant presented some numeric data about the use of Google Books, but the data has yet to be spread to the world (See my comment on Peter Brantley’s blog about this). I’ve been searching in the week since TOC, to see what discussion there is of Orwant’s talk, and have found little. So I’m excerpting the three pieces that I have found. Only the first has any numeric data at all.

First, a piece by Jackie Fry, on the BookNet Canada publishers’ Blog. This is notable, and I’m putting it first, because it’s the only report I’ve found that has any numeric data at all from Orwant’s talk:

Conversion rates from Google Book Search results have been great for their partner publishers, mostly in the Textbook, Reference and STM channels, particularly in the shallow backlist (2003-2005 pubdates) with the highest Buy the Book clickthrus on 2004 titles. For some publishers, conversion to buy is as high as 89% for the titles they have made available.

30% of viewers looked at 10 or more pages when viewing the content of the book to make a buy decision.

The future is analytics! Google is thinking about what data they can pull out of their logs and provide anonymous aggregate data around consumer behaviour like what books were purchased that were like this one, search terms used most often for a category, most effective discounts, most effective referral sites etc.

More research [is needed] – Saw some good presentations with quantifiable research included – Brian O’Leary from Magellan, Joe Orwent (sic) from Google, and Neelan Choksi from Lexcycle were some of the few presenters who were able to quantify in any way what is going on in the marketplace. We need more  …

James Long’s report, on thedigitalist.net (Pan Macmillan Publishing):

Jon Orwant, from Google Book Search, stated at TOC that ‘the ultimate goal of Google Book Search is to convert images to “original intent” XML’. He explained the post-processing Google runs to continuously improve the quality of the scanned books, and to convert images to structured content. Retro-injecting structure accurately is no mean feat but when it’s done, Google will be able to transform the books into a variety of formats. The content becomes mutable and transportable, in a sense it isn’t yet, even though it is scanned, online and searchable. Orwant also presented three case studies – McGraw Hill, OUP, Springer – that demonstrated the benefits publishers can gain from having their books in GBS.

Highlighting the theme of discovery (to my mind), Tim O’Reilly interjected, at the end of these case studies, and made the point that O’Reilly used to own the top links to their own books in Google search results, but have now lost those links to GBS. Orwant, somewhat simplistically, responded that O’Reilly needed to improve their website to regain the top ranked link per title, as this spot was determined by Google’s search algorithms. This was not a convincing response, and dodged the issue, which I understood to be that the scale and in-house-ness of GBS could seriously inhibit the ability of the publisher to represent their own products online at the most common point of entry by the consumer, Google search results. There are many compelling reasons for publishers to own the top search result link, the most obvious being: offer unique additional content around the title, start a conversation with the reader, control the brand.

Edward Champion’s comments on his blog:

Thanks to a concept called blending, Google Book Search options remain in the top search results. An effort to direct traffic GBS’s way. …

There are 1.5 million free books, all public domain titles, available on Google. But if you want to access them, well, you’ll have to go to Google. Or you’ll have to have Google generate results at your site. Because the Google team are specialists in latency. They can do things with milliseconds that you couldn’t work out in your dreams.

As an experiment, Google recently unleashed Google Books Mobile, which means that you can nose search Google Book Search from your smartphone … Orwant was careful to point out that Google is not in the handset manufacturing or carrier business. But he anticipated, just as many of the seer-like speakers at Tools of Change did based on sketchy inside information, a “rapid evolution.”

Tim O’Reilly tried to badger Orwant too. You see, O’Reilly used to have good webpage placement for many of his titles. But those days are gone, replaced by Google Book Search results above the O’Reilly pages. And that hardly seems fair …

There’s some comfort in knowing that 99% of the books at GBS have been viewed at least once. Even the sleep-inducing textbooks. Which is really quite something. Which brings us to the future, which is based on the past …

That snippet view you see with some titles? Orwant‘s official position, pressed by Cory Doctorow, is that it’s fair use. But once the October 2008 settlement in Authors Guild v. Google is approved by the court, you’re going to see that snippet view jump to 20% of the book.

Please comment here or Twitter @ericrumsey

When a link is clicked to a specific page in GBS Mobile, the page that always opens is the entry page for the book. There doesn’t seem to be a way to link successfully to specific pages. I’ve tried this in several examples, and have had the same experience in all of them. An example below illustrates.

In this example, I’m trying to link to a group of pages starting with page 31. But when the link below is clicked, it goes to the entry page, which is page 21, with the same URL as below except that the page number is 21 instead of 31.

http://books.google.com/googlebooks/mobile/#Read?id=yb8UAAAAYAAJ&page_num=31
[This link and the link in the image below are the same]

After this link is clicked, and it goes to page 21, then it does work to change the number from 21 to 31, and it goes to page 31. The right > next to “Pages 21-30″ also works.

When the link is clicked to go to page 31, and ends up on page 21, clicking the Back button goes to page 31. And the address bar initially initially reads 31, but then changes to 21 – So when the link is initially clicked, it does “pass through” page 31, but apparently there’s some signal on page 31 that tells it to redirect to page 21.

Does anyone see what’s happening here? Any help would be much appreciated! Please post suggestions in comments, or in Twitter.

.

.

.

.

.

There’s been a lot of buzz about the announcement last week of mobile access to Google Book Search public-domain books. I’ve been looking hard for nitty-gritty details of how it works, though, and haven’t found much. The best is in comments by bowerbird on an announcement article on toc.oreilly.com. It’s easy for comments to get lost, so I’m excerpting most of bowerbird’s words here:

this offering is very good. extremely good. the interface is quite nice…

it was great to see google is serving digital text, rather than scans, since text is a lot more nimble. however, a tap on a paragraph brings up the scan of that paragraph, which is nice. and another tap restores the text. so if you want to verify the o.c.r., it’s simple to do. as i said above, this is nicely done.

curiously, in the one book i checked (roughing it), the text was extremely accurate as well, which is a pleasant discovery. i found only one o.c.r. error — “firty” for “fifty”, due to a blotch on the page …

this quality text is _not_ typical of google’s raw o.c.r., so they’ve evidently run some clean-up routines on it. i’m curious to see if they share this cleaned-up text with their library partners, or keep it to themselves… (no, the libraries weren’t smart enough to ask for it, as far as i know, let alone write it into the contracts.)

I’ve bolded what I take to be the most interesting point here, that Google has done an extra-special job of OCR’ing text for GBS mobile. As bowerbird notes, hopefully Google will share more about this process, sooner or later.

In a recent NY Times article that I blogged on, Dan Clancy, the engineering director for Google book search, is cited as saying “every month users view at least 10 pages of more than half of the one million out-of-copyright books that Google has scanned into its servers.” Remarkably, this classic long tail description of Google Books seems not to have been noticed by anyone — I’ve searched in Google (web and blogs) for various word combinations in the quote combined with “Dan Clancy,” and have found nothing at all except the original NYT article.

The long tail idea, which was first described by Chris Anderson in 2004, is that when a very large number of users are given a very large number of items to choose from, especially in an online environment with virtually unlimited “shelf space” and easy access, a very wide variety of items will be chosen. Anderson proposed the idea especially to describe commercial sites such as Amazon and Netflix, but it has also been seen as a good fit for libraries, and especially online library/book sources, such as Google Books.

So — Yes — There has been discussion of Google Books and the long tail. For the most part, though, this has been on a conceptual, non-numeric level. The statement by Clancy is valuable because it’s the first time there have been actual numbers provided by Google sources to back up the conceptual ideas. And, indeed, striking numbers they are — every month, half of the out-of-copyright books — i.e. old books — in Google Books are getting significant use. The long tail will certainly be even longer when newer books are made available after the October 2008 settlement goes into effect.

The best numeric data that I’ve found on Google Books and the long tail is given in an article by Tim O’Reilly in 2006, which compares sales of O’Reilly Media book titles, as reported by Nielsen Bookscan, with page views from Google Books. As the graph (at left) from that article shows, the Google Books page views (in red) have a very long, almost flat, tail, in contrast with the relatively short tail for actual sales of book titles (in blue). Incidentally, the graph shown here has a bad link in the O’Reilly article, so all that displays is the file name; I did some digging on the O’Reilly site to find it here. (Feb 11: Bad link for this image and others in O’Reilly article are fixed, after I noted them in a comment.)

The closest thing I have found to other long tail numeric data relating to online books is reported in a 2006 article by Jason Epstein:

According to Mark Sandler of the University of Michigan Library, in an essay in Libraries and Google, an experiment by the library involving the digitization of 10,000 “low use” monographs offered on the Web produced “between 500,000 and one million hits per month.”

I suspect the realization of the “power of the long tail” shown in this experiment contributed to the University of Michigan opting to be one of the original library partners in the Google Books project.

Clancy cites high usage of out-of-copyright books

This article is generally unremarkable, although it does have some good quotes from prominent players. Otherwise, just another article in NY Times on Google Books. But it has two notable features — The first is the quote from Google’s Dan Clancy, in the second paragraph, stating a remarkably high volume of usage of out-of-copyright books. The second notable feature, which is why I’m excerpting the article at some length, is that it was given surprisingly little attention in the blogosphere/twittersphere when it was published a month ago.

Google hopes to open a trove of little-seen books [IHT version]
by Motoko Rich, New York Times, Jan 5, 2009

Ever since Google began scanning printed books four years ago, scholars and others … have been able to tap a trove of information that had been locked away on the dusty shelves of libraries and in antiquarian bookstores.

[boldface added] According to Dan Clancy, the engineering director for Google book search, every month users view at least 10 pages of more than half of the one million out-of-copyright books that Google has scanned into its servers.

The agreement, pending approval by a judge this year, also paved the way for both sides to make profits from digital versions of books. Just what kind of commercial opportunity the settlement represents is unknown, but few expect it to generate significant profits for any individual author. Even Google does not necessarily expect the book program to contribute significantly to its bottom line. … “We did not think necessarily we could make money,” said Sergey Brin, a Google founder and its president of technology, in a brief interview at the company’s headquarters. “We just feel this is part of our core mission. There is fantastic information in books. Often when I do a search, what is in a book is miles ahead of what I find on a Web site.”

Users are already taking advantage of out-of-print books that have been scanned and are available for free download. Mr. Clancy was monitoring search queries recently when one for “concrete fountain molds” caught his attention. The search turned up a digital version of an obscure 1910 book, and the user had spent four hours perusing 350 pages of it.

“More students in small towns around America are going to have a lot more stuff at their fingertips,” said Michael A. Keller, the university librarian at Stanford. “That is really important.”

Some librarians privately expressed fears that Google might charge high prices for subscriptions to the book database as it grows. … David Drummond, Google’s chief legal officer, said the company wanted to push the book database to as many libraries as possible. “If the price gets too high,” he said, “we are simply not going to have libraries that can afford to purchase it.”

Authors view the possibility of readers finding their out-of-print books as a cultural victory more than a financial one. … “Our culture is not just Stephen King’s latest novel or the new Harry Potter book,” said James Gleick, a member of the board of the Authors Guild. “It is also 1,000 completely obscure books that appeal not to the one million people who bought the Harry Potter book but to 100 people at a time.”

Some scholars worry that Google users are more likely to search for narrow information than to read at length. “I have to say that I think pedagogically and in terms of the advancement of scholarship, I have a concern that people will be encouraged to use books in this very fragmentary way,” said Alice Prochaska, university librarian at Yale.

“There is no short way to appreciate Jane Austen …,” said Paul Courant, university librarian at the University of Michigan. “But a lot of reading is going to happen on screens. One of the important things about this settlement is that it brings the literature of the 20th century back into a form that the students of the 21st century will be able to find it.”

Adam Hodgkin, in Google Pictures and Google Books, wonders why Google has chosen to put Prado paintings in Google Earth rather than in Google Images. In December I asked a similar question about Google’s putting Life Magazine pictures in Google Images, but putting other picture-laden magazines in Google Books. And, in another recent launch they’ve put newspapers, which also have many pictures, in Google News.

Once again I come back to the theme of this blog — Pictures are just different — They don’t fit neatly into our categories. Pictures are an important part of several different media — books, magazines, newspapers, and (of course) art — So what slot do we put them in?

Even before the recent questions arose with Life Magazine pictures, Google Magazines, Google Newspapers, and Prado paintings, there’s the ongoing, but little-noted question of pictures in the growing collection of public domain books in Google Books. In my experience, these are completely absent from Google Image Search — When will Google make this connection?

Figuring out what category to put them into, of course, is a relatively minor problem compared to the BIG PROBLEM with pictures, which is making them searchable! If there was one category to put them into that was searchable, then of course that would be the place for Google to put them!

Excerpts from Peter Brantley’s eloquent words on the Google Book settlement, in A fire on the plain (bold added).

With recent back and forth over the proposed Google Book Search settlement (e.g., Robert Darnton’s essay in The New York Review of Books; Tim O’Reilly’s response; and James Grimmelman’s litany of proposed corrections predating both at The Labortorium), I’ve been cast again into thinking about aspects of the agreement.

It is difficult to credit that frustrating access is ever able to delay or stem fundamental social trends – for example, the increasing importance of visual and interactive media. … Or the possibility that searching and reading networked books for anyone under the age of 40 might be an inherently social activity that generally increases enthusiasm for all forms of reading.

Let us consider a far more basic, more fundamental concern: the proposed Google Book Search settlement is embedded in a set of conceptions about books, reading, and information access which is as profoundly obsolescent as the printed Encyclopedia.

This is a world where young children carry around in the palm of their hands gaming consoles that have more networked computing capacity than a moderately powerful Sun workstation of five years back. Where increasingly I think about printed books with as much fondness as large cinder blocks, …  And yet authors and publishers worry that a fair level of access to digitized books … might reduce their profits. Truly, this should not be their worry. Their eyes remain cast on a horizon which has fallen from the earth, while a new sun is rising.

The settlement describes a world of time past, not a world of possibilities. Can we not imagine a redrafting of the settlement’s terms with libraries? … let us envision an alternative world where children routinely carry Alexandria in their hands. Where they experience works of literature as games, pushing at the borders of their knowledge and experience by engaging the library with others as a festschrift.

The people served by our libraries – let them show us how to re-make literature in a world where it fits in the circle of many hands, caressed by fingers, shared between minds. Libraries are laboratories for the future of reading, and with this, we have the key to it. … We stride into a world where books are narratives in long winding rivers; drops of thought misting from the sundering thrust of great waterfalls; and seas from which all rivers and rain coalesce, and which carry our sails to continents not yet imagined.

[concluding paragraph] Digital books are sparkles of magic untapped. The settlement proposes a bold path from darkness. But it is a trail that circles back to an old forest, abandoned. Our people have left, ventured onto a flat savannah, strewn with rocks, thorny shrubs, windblown trees, beasts. We can see it all now. And we are starting fires, with wood from fallen trees. Burning down the forest.

Related articles:

Eric Rumsey is at @ericrumsey

In a brief response letter, author and publisher Marc Aronson writes about the copyright status of pictures that are in publisher partner books in Google Books. Aronson suggests that the rights for pictures are separate from the rights for text. I’ve corresponded with Aronson to expand on this idea, and he says that in his experience as an author and editor, he has been told that he needs to obtain rights to pictures and text separately. I’ve searched for other commentary on this issue, and have found very little. It’s a subject that needs exploration. Anyone have ideas?

All books in the publisher partner program, of course, are under copyright, and are available only in Limited Preview, with the publisher giving Google the rights to display a specific number of pages. In some cases of books containing pictures, however, the pages are available, but without the pictures. Is this because the publisher has gotten the rights for limited preview of the text, but not the pictures, as Aronson suggests? The three examples below show a variety of Limited Preview options. The first two are especially pertinent, because they are for books from the same publisher (Macmillan), in the same series, that have a different picture preview status, possibly indicating that the illustrator has given permission to display pictures in the first case, but not in the second.

In this example, the first 39 pages* are available for preview, with all pictures displaying. There are about 30 thumbnail images for pages with pictures on the About this Book page.
Birds of North America (Golden Field Guides)
By Chandler S. Robbins et al, Illustrated by Arthur Singer, Published by Macmillan, 2001

In this book, from the same publisher, the first 37 pages* are available for preview, but almost all pictures do not display, replaced with the message “Copyrighted image.” There are no thumbnail images on the About page.
Wildflowers of North America (Golden Field Guides)
By Frank D. Venning, Illustrated by Manabu C. Saito, Published by Macmillan, 2001

This book follows the most common, fairly liberal, pattern of publishers in Limited preview books, with the first 50 pages* available including all pictures. A full complement of 30 thumbnails is on the About page.
Central Rocky Mountain Wildflowers
By H. Wayne Phillips, Illustrated, Published by Globe Pequot, 1999

* The number of pages available for preview varies from session to session — The number given here is the maximum I experienced.