You are viewing limited content. For full access, please sign in.

Question

Question

Full Text Index queue size...

Search and Retrieval Laserfiche Version 9 Administration
Updated March 26, 2015
asked on February 9, 2015

Since upgrading to 9.2 SP 1 (from 9.0.2) we are having serious issues with the creation of our full text index.  We currently have just under 1 million documents in our repository and our text size is roughly 25GB, but the queue size of the index is showing nearly 10 million documents and the lfqueueN.idx folders under the index directory are already consuming nearly 500GB of disk space.  We keep running out of disk space on the partition storing the index and I have to allocate more.

 

Anyone else having such issues when creating a new FT index?
 

0 0

Replies

replied on February 9, 2015 • Show version history

Hello Brian,

1. It is possible that the queue size is larger than the count of documents, for a): Laserfiche full text search engine also index metadata, such as entry name, which also is calculated in queue size. You may also notice that the queue size will go up and down, it is because that LFS expands the document index request to metadata index requests and queue size is increased during index.
b). If you modify a document (which will trigger an index), queue size will be increased too.

2. In your case, from the info in your support case, you have ~25GB text documents and ~400GB electronic documents. Thus, it is possible that the lfqueueN.idx folders are consuming 500GB, since you have that size of text and electronic documents. You may need to reserve 700GB for the catalog. 

For other users who don't have many electronic documents, you need to reserve: 4 * size of text documents, or 2.5 * size of text documents + 3 * size of lflink.idx in 9.2.0, for whole search folder(including lfqueueN.idx folders and idx files). E.g. if you have 25GB text documents and no electronic document, you need to reserve 4 * 25GB = 100 GB disk space.

3. It is consuming the disk space during index. After indexing finished, the disk space will be released. If it is still consuming disk space, you can restart Laserfiche full text search engine to free it.

Since electronic documents have large size, you can extract text pages from Client, this may save the disk space in Laserfiche full text search engine.

In the next minor release, you can filter extensions to not index specified types of electronic documents, such as video, which can save disk space in Laserfiche full text search engine too.

2 0
View 2 previous replies
replied on March 24, 2015

Hello Cangfei,

How can i calculate the sufficient size for the queue size?, I have to import 265 GB to laserfiche , and I am worry if i have enough space in my HD where Full text search is installed.

 

Thanks for the help.

Best Regards,

Vitor

0 0
replied on March 25, 2015

Hello Vitor,

What is the exact version of LFS and LFFTS you are using?

If you are using versions before 9.2.0, the disk space estimation would be different. There is no index queue before 9.2.0, so I suppose this is not your case.

If you are using version 9.2.0, I suggest you to upgrade both LFS and LFFTS to 9.2.1. There are several fixes related to index queue size, which is helpful in saving disk space.

If you are using version 9.2.1, in theory, you need to reserve 4 * 265 GB = 1 TB disk space. But usually 500GB is sufficient. If you have electronic documents, then it can be less. If you have lots of electronic documents which you don't want to index, you can set BlockedExtensions and files with such extensions won't be included in the index queue. It can save disk space. Please refer to this answers thread https://answers.laserfiche.com/questions/50692/LFFTS-BlockedExtensions-List-registry-key

Finally, the disk space we estimated here are for .IDX files and index queue only. When importing files to laserfiche, you also need to consider the disk space for volume files on LFS machine.

1 0
replied on March 25, 2015

Hello Cangfei,

Thank you very much for the clarification.

I already updated to 9.2.1. This documents that i will import it is eletronic documents(pdf's), and they will be attached to a document in laserfiche through SDK.

If i choose to not index them, can i index them in the future?

Thank you very much for the help.

Have a nice day.

regards
 

0 0
replied on March 26, 2015

Hello Vitor,

Yes, you can index them in the future.

There are several ways to not index documents:

1. If you don't want to index all documents during importing, you may disable the "Automatically Indexing New Documents" option, see here. After importing finished, enable the option back so that new documents imported later can be indexed automatically.
2. If you just don't want to index specified extensions of electronic documents, then you can set BlockedExtensions. In future, if you want to index electronic documents of blocked extensions, remove the extensions from the BlockedExtensions. Note: After removing extensions from BlockedExtensions, the un-indexed electronic documents won't be indexed until you index them explicitly.

There are several ways to index documents explicitly:

1. If you want to index all the documents in a repository or volume, then you can choose to reindex the repository or that volume, see here.
2. Also, you can select documents and index them in LFClient, see here.

You can generate text pages for electronic documents in LFClient, see here. This can save disk space for index queue (since text size is usually smaller than the edoc size) and index will be faster (since LFFTS don't need to extract text from electronic documents).

1 0
replied on March 26, 2015

Hello Cangfei,

Thank you very much for all the information that you shared!

Have a great day!

 

Regards,

Vitor

0 0
replied on February 9, 2015

FYI. A case has been opened for this issue with your VAR.  He should be contacting you shortly, if he has not already, with the troubleshooting steps.

0 0
replied on February 9, 2015

Thank you.  I opened a case this morning with our VAR but thought I'd throw it out here as well seeking guidance, especially in case others have experienced the same or similar impacts.  Thanks again.

0 0
You are not allowed to follow up in this post.

Sign in to reply to this post.

Details
Related Posts
Loading ...