can I extract native text from .pdf's without generating pages and do so with either the API or an SDK script?

asked on September 9

I recently migrated a large number of .pdf documents into an on prem Laserfiche 12 system. There were over 25 million documents so the .pdf's were imported into Laserfiche via a script but without generating pages. Now I need to be able to full text search on the .pdf documents but am unable to because the text was not extracted. Generating pages using the DCC has proven to be very slow and has also shown to increase the file sizes tenfold.

What is the best method of extracting the native .pdf text on these files? In the client, I can generate OCR text manually but with that many documents, not a very efficient way of doing things. What would be the equiv method using either the API or the SDK??

0 0

replied on September 9

That's a big operation! I'm not a LF dev, but I can say with some confidence that you could probably set up DCC with multiple worker nodes to OCR thousands of page generations per hour. Let's say you're using 2 workers + your scheduler, you could probably get 5k/hour. If the workflow is running continuously without errors or DCC queue issues, it would take you at the minimum 208 days. Such big expensive operations are prone to getting stuck though, so if you're not checking in on it daily, you could lose progress.

It's certainly possible to use the SDK to call the API's Entries.../Pages or Entries.../GenerateText nodes to handle the work on their end, but it would probably still take a very long time. Additionally you'd have to manage which ones were asynchronously sent over, plus you will very likely experience throttling for such high volumes (it would be crazy if Laserfiche didn't throttle this kind of action).

This would probably be something to submit a ticket to Laserfiche about, they may have some official recommendations and strategies.

0 0

Question

Question

can I extract native text from .pdf's without generating pages and do so with either the API or an SDK script?

Replies

Sign in to reply to this post.