@devhead: To import the data into ES we used a custom application to extract the text from the OCR'd documents.
This is required to support our bookreader software. A complete ingestion takes a few days; we rate-limit indexing in order not to overload the cluster, and maintain reasonable search performance.
Great article man. Am always super exited when i find articles that talk about information retrieval systems. I have a lot of questions for you. Been working on a search engine project www.cognifly.com for a year and its inverted index is still very small. Like 4gig now. So is ok if i send you an email?
It was a great article. On a side note, holy shit is it rare to even hear of other guys named Giovanni, much less as similar a last name (I'm Giovanni d'Amelio).
@devhead: To import the data into ES we used a custom application to extract the text from the OCR'd documents. This is required to support our bookreader software. A complete ingestion takes a few days; we rate-limit indexing in order not to overload the cluster, and maintain reasonable search performance.