You are viewing limited content. For full access, please sign in.

Question

Question

using ILFSearchResultListing, what is the fastest way to get the metadata?

SDK
asked on January 8, 2014

 How can I get the meta data from LaserFiche the fastest.

 

I have been using the following code and the second line takes over 3 minutes for 15000 document's metadata:

 

ILFSearchResultListing srchHits = (ILFSearchResultListing)search.GetSearchResultListing(_parms,0);   srchHits.GetRowData(1, srchHits.RowCount, out RowData, out RowDataAsString);

 

 

0 0

Replies

replied on January 10, 2014

Coming late to the party, but I also want to add that if you develop an app that does go direct to the SQL, there is nothing to guarantee the SQL Schema will not change with an update (breaking your application).  By using the Laserfiche SDK, your application will continue to function even with server upgrades.  For long term stability, you should avoid direct interaction with the SQL DB if possible.

 

Just my 2 cents...

2 0
replied on January 8, 2014 Show version history

Jean,

 

Depending on what you plan to do with the returned metadata one option is to do a SQL query back into the Laserfiche database instead of using the native LF search capabilities.  I haven't done any metrics but I am confident that a SQL query will return a recordset much faster than the native search capabilities.  (The LF engineers can chime in on that one).

 

If you provide specifics on your search parameters then  perhaps I can help with the query syntax.

 

The obvious downside is that if Laserfiche changes the underlying SQL data structures then you would have to modify your query accordingly, but I don't believe that would happen often.

1 0
replied on January 9, 2014 Show version history

The SQL query is faster because it does not enforce security (nor does it get audited since the LF Server has no idea what you're doing). The Laserfiche Server caches data, so querying SQL directly is not guaranteed to give the same results as you would get through the SDK.

 

I don't know if this is possible at this point in your project, but if you use RepositoryAccess instead of LFSO, you can specify fields as columns to be returned in the search results entry listing. See the SearchListingSettings class in your SDK .Net documentation.

 

Edit: LFSO's LFSearchListingParams.AddTemplateFieldColumn allows you to specify fields to retrieve with your search results.

2 0
replied on January 9, 2014

Ah, I didn't think about LF caching any recent metadata activity but I would also assume that there is not much latency.  What would be a ballpark on how long LF will cache metadata changes before writing them to SQL?

 

As far as enforcing security and recording activity in Audit Trail; I assumed that the framework for the search was Workflow, not an external application so those would be valid points in certain deployments.

0 0
replied on January 9, 2014

The Laserfiche server caches data to avoid reads, but always writes through changes to SQL Server immediately. While this could potentially change in a future version, the above behavior has been the case for all versions of Laserfiche. As long as you don't modify the database directly, you shouldn't have to worry about caching effects.

0 0
replied on January 9, 2014

Ok, that makes sense.  The points Miruna brings up about SQL queries bypassing audit trail and access rights are appropriate when developing an external application though.

0 0
replied on January 9, 2014

Thanks so much for your help.  I'll first try RepositoryAccess. Where is this located?

0 0
replied on January 9, 2014

Cliff, Can you help with some sql syntax? I have a ton of fields but something to get me started would be great... here are my fields:

 

            Name, 
            EntryType, 
            ID, 
            ParentID, 
            TemplateName, 
            DocType, 
            Sub_DocType, 
            Entity Name,
            Tags, 
            Extension, 
            Mime Type, 
            OCR'ed Pages, 
            Page Count, 
            Indexed, 
            Create Date LF, 
            Created By LF, 
            LastModified, 
            Volume, 
            Linked, 
            Version, 
            //GSCO Template Fields
            All Fields, 
            Date Range Type, 
            Doc - Date Range - Start, 
            Doc - Date Range - End,     
            Comments 1, 
            Entity Type, 
            Doc Category, 
            Doc Source, 
            Doc - Date Rec'd by GSCO, 
            Bates # - Start, 
            Bates # - End, 
            Department, 
            Client Name, 
            File Description, 
            Created by, 
            Date Printed, 
            Time Printed, 
            Barcode, 
            GSID, 
            //Correspondence Fields
            Subject, 
            Sender Name, 
            Sender Email, 
            To, 
            CC, 
            Comments1, 
            Received Date, 
            Received Time, 
            Dept

0 0
replied on January 9, 2014 Show version history

Jean,

 

As to your first question on Repository Access (RA); it is a available via the SDK by including the appropriate RA references in Visual Studio.  (Depending on what you want to do within Laserfiche it could be as simple as adding a reference to the 'Laserfiche.RepositoryAccess'  and/or the 'Laserfiche.DocumentServices' assemblies.)

 

As to the syntax for the SQL query.  Hopefully your search is limited to just a couple of metadata fields and not the entire list?  wink 

 

I guess a good place to start would be what are you going to do with the resulting recordset from the search?  Do you want to end up with a list of entryID's that can be used to reference the actual Laserfiche entries that meet the search criteria? Or do you want to end up with a list of column data (like the native search function) that you can use to populate something like a listbox?

 

0 0
replied on January 9, 2014

If i did the sql query, I would need that whole list.. sad

 

In the end I need all of those columns with the entryIDs. I am basically creating a client that works like an excel sheet allowing more customized filtering.

0 0
replied on January 9, 2014

Hmmm, trying to wrap my head around that one.  How would you display a list of entries in a table like that if the entries had different fields and/or templates assigned? 

 

Would it be something like each row would be an entry that matched the search criteria and then there would be a column for each metadata field that exists in the system.  If a particular field was assigned to the entry and contained a value then that value would be displayed in the appropriate cell?

 

0 0
replied on January 9, 2014

Yes, that is correct! 

0 0
replied on January 9, 2014

I can give it some thought if you are not in too much of a hurry.  My first thought would be to look at a SQL query that could return the appropriate data but it wouldn't be very efficient as you would have to step through the returned recordset to populate the grid.

 

The second thought (if you are using Visual Studio) would be a bit cleaner.  I would try to try to tie a Visual Studio datagrid to an appropriate datasource that would autopopulate the grid based on the appropriate search criteria.  That would probably be more efficient as the datagrid only pulls back displayed data (plus some overhead)  instead of the entire recordset.

 

Unfortunately in either case I don't think it is going to be simple.  Hopefully I haven't taken you down a rabbit hole with the SQL query idea!
 

0 0
replied on January 9, 2014

haha.. thanks.. Before I take up your time with the Sql route.. Let me try to use the repository access. I will get back to you.

0 0
replied on January 9, 2014

With the risk of sounding like a broken record, what about security? If this is a one-user application and that user is allowed to see absolutely everything in the repository, then by all means ignore me.

 

But if you plan on distributing this app to different users who should only be seeing parts of the repository, then going the SQL query route is not the way to go.
 

0 0
replied on January 9, 2014

Not everything will be pulled out.. they have a choice of folders based on there access rights.

0 0
replied on January 9, 2014

Miruna - I agree, bypassing field level and document level access rights is a valid concern when developing an external application.  Hopefully moving to RA will allow enough flexibility and address any latency issues.

 

Jean - One final thought is that perhaps you could rethink your UI?  Instead of one large datagrid displaying all of the data, perhaps there is a way to design a UI that reduces the amount of data retrieved at any one time.  Maybe a combination of grids; one parent grid displaying generic entry data matching your search parameters and a child grid displaying field level data for selected parent records?

0 0
replied on January 10, 2014 Show version history

Jean,

 

It looks like you might be mixing RA, LFSO, and some SQL all in one script.  Here is a quick code snippet that uses RA to do a search similar to your second SQL SELECT statement

 

(Sorry, it is in VB, I don't do C# wink)

 

The search is looking for a value of "Drama, Johnny" in the 'Client Name' field.  Once the search has completed you can step through the results and do something with them.  In this quick example I am just adding the entryID and Document Name to a multi-line string variable (which is not very useful!)

 

        Dim rr As New RepositoryRegistration("Laserfiche", "LFVM")
        Dim sess As New Session

        sess.LogIn(rr)

        Dim mySearch As New Search(sess)

        mySearch.Command = "{[]:[Client Name]=""Drama, Johnny""}"
        mySearch.Run()

        Dim settings As SearchListingSettings = New SearchListingSettings
        Dim results As SearchResultListing = mySearch.GetResultListing(settings)
        Dim x As String = ""

        For i As Integer = 1 To results.RowCount

            x &= results.GetDatumAsString(i, SystemColumn.Id) & " " & results.GetDatumAsString(i, SystemColumn.Name) & vbCrLf

        Next

 

1 0
replied on January 10, 2014

I am afraid that code is similar to what I was doing in the first place using getdatum which is what is taking so much time.

0 0
replied on January 10, 2014 Show version history

Here is a short SQL query to get some entry data that is somewhat based on your earlier query (This query is just returning the entryID, name, dateModified, and dateCreated.  Many other fields are available in the 'toc' table);

SELECT toc.tocid, toc.name, toc.modified, toc.created FROM propval
LEFT JOIN toc ON propval.tocid = toc.tocid  
LEFT JOIN propdef ON propval.prop_id = propdef.prop_id
WHERE propdef.prop_name = 'Client Name' AND propval.str_val = 'Drama, Johnny' ORDER BY toc.created

Without going into a lot of detail the 'toc' table is where most of the 'entry' data resides, the 'propdef' table contains the metadata field definition data (Field name, type, length, etc), and the 'propval' table contains the actual metadata values.

 

Speaking of the 'propval' table; the actual metadata value is only in the 'str_val' column for text type fields.  If the metadata field is a numeric then the field value is in the 'num_val' column and if it is a date then those values are in the 'date_val' column.  (Then there is the 'bin_val' column for large blob data like images!)

 

So even using the direct SQL query route you are going to have to go through a lot of gyrations to end up with an Excel type spreadsheet that you mentioned earlier.

 

That and all of the earlier warning about access rights, audit trail, and schema changes.

 

You sure you don't want to rethink your user interface? smiley

2 0
replied on January 13, 2014

Thanks Cliff!.. This works so fast and I think we can use it.. I will update you later.

0 0
replied on January 13, 2014

Trying to determine how the location of the file is queried.  Could you provide a sample joining the file path?

 

Also do you know of any documentation I could reference that describes the

overall database structure?

 

Thanks.

0 0
replied on January 13, 2014

Figured out a way recursing the parentIds via recursive CTEs.

 

ALTER FUNCTION [dbo].[GetFullPath](
    @tocId int
)
RETURNS Varchar(1000) AS
BEGIN

    DECLARE @tmpPath varchar(1000);

    WITH parentCTE( parentId, name )
    AS 
    (
        --Anchor member definition
        SELECT     
            parentId,
            name 
        FROM 
            toc AS t 
        WHERE 
            t.tocId = @tocId
        UNION ALL
        --Recursive member definition
        SELECT 
            t.parentId,
            CAST(t.name + '\' + 
                d.name AS NVARCHAR(255))
        FROM 
            toc AS t, 
            parentCTE AS d 
        WHERE 
            t.tocId = d.parentId
    )    
    SELECT @tmpPath = name from parentCTE OPTION (MAXRECURSION 0)

    RETURN @tmpPath

END

 

 

2 0
replied on January 13, 2014

You beat me to it!  The path of an entry is not stored in a single field (unfortunately) so you will have to do some type of recursion to get the path.  I am going to copy your answer and keep it on file for future reference!  wink

 

As far as a reference to the actual data schema I don't believe I have seen one.  All my exposure is just poking around until I found what I was looking for.

 

Perhaps one of the LF Engineers can chime in on any schema reference?

 

0 0
replied on January 10, 2014

Bert - Yes, I would agree that reading the underlying data structures in SQL requires that you understand the inherent risks but I also have no problem doing that if the benefits outweigh the drawbacks.  Going directly to SQL would not be my first choice for the reasons you mention.  wink

 

0 0
replied on January 10, 2014

So I am trying to use RepositoryAccess.. Straight select works. But, my join doesn't work.. Am I missing something?

 

 RepositoryRegistration rr = new RepositoryRegistration("LFVM", "Laserfiche");

                Session sess = new Session();

                sess.LogIn(rr);

                LfConnection conn = new LfConnection(sess);

                conn.Open();

                LfCommand lfqlCommand = conn.CreateCommand();

             

  //so basic selects work, it's just when joins are included
                //lfql doesn't like the result set...
          

            //    srch = "SELECT entry_id, entry_name FROM lf.Entry";

               
              
                 srch = @"SELECT 
                            e.entry_id, e.entry_name 
                        FROM lf.Entry e 
                        INNER JOIN lf.Prop_value p on e.entry_id = p.entry_id
                        WHERE 
                            p.prop_name = 'Client Name' AND 
                            p.string_value = 'Drama, Johnny'";
                

                lfqlCommand.CommandText = srch;

                LfDataReader reader = lfqlCommand.ExecuteReader();

                DataTable dt = new DataTable();

            
                dt.Load(reader);

0 0
You are not allowed to follow up in this post.

Sign in to reply to this post.