You are viewing limited content. For full access, please sign in.

Discussion

Discussion

How to import CSV into SQL with Workflow

posted on October 29, 2018 Show version history

From time to time I run across a customer with a system that does not allow ODBC connections for Look-ups.  We need the data accessible by Workflow (or Quick Fields), so we need to get it into a data source that can handle the ODBC connections.  If the 3rd party software will export to CSV, then our customer can export the metadata for their batch and now they have data they can look-up against.  While CSV does allow for ODBC connections, it is not the best for production and you must always make sure the name is the same and it is in the same location for the ODBC configuration, so I like to have them import (drag and drop) the CSV into Laserfiche and have Workflow import it into a SQL table.

You must have a Text ODBC driver that matches the architecture of your Workflow services (32 bit Workflow requires 32 bit driver and 64 bit Workflow requires 64 bit drive).  I like to use the Microsoft Access Database Engine 2010 Redistributable.

Also, you must connect to SQL with a user that has permissions to run the "OPENROWSET" command and your SQL must be configured to allow the Ad-Hoc Distributed Queries by running the below SQL code:

EXEC sp_configure 'show advanced options', 1
RECONFIGURE
GO
EXEC sp_configure 'ad hoc distributed queries', 1
RECONFIGURE
GO

Then here is an example of the workflow I set up

I start with a conditional decision to only process documents that have csv extension.  If it is a CSV file, I use a SDK Script to export the CSV file. (make sure to add reference and Imports statement for Laserfiche.DocumentServices)

Imports System
Imports System.Collections.Generic
Imports System.ComponentModel
Imports System.Data
Imports System.Data.SqlClient
Imports System.Text
Imports Laserfiche.RepositoryAccess
Imports Laserfiche.DocumentServices

Namespace WorkflowActivity.Scripting.ExportCSVSDKScript
    '''<summary>
    '''Provides one or more methods that can be run when the workflow scripting activity is performed.
    '''</summary>
    Public Class Script1
        Inherits RAScriptClass102
        '''<summary>
        '''This method is run when the activity is performed.
        '''</summary>
        Protected Overrides Sub Execute()
            'Write your code here. The BoundEntryInfo property will access the entry, RASession will get the Repository Access session
            Try
                Using doc As DocumentInfo = DirectCast(BoundEntryInfo, DocumentInfo)
                    Dim docExporter As DocumentExporter = New DocumentExporter()
                    Dim fi As System.IO.FileInfo = New System.IO.FileInfo( "E:\DATA\Work\temp\Check Run.csv")
                    If fi.Exists Then
                        fi.Delete
                    End If
                    docExporter.ExportElecDoc(doc, fi.FullName)
                End Using
            Catch ex As Exception

            End Try

        End Sub
    End Class
End Namespace

After that, I run a custom Query to import the CSV into a SQL Table.  If your CSV has a header row, you use code like this:

TRUNCATE TABLE [ExternalTables].[dbo].[CheckData];
INSERT INTO [ExternalTables].[dbo].[CheckData] (
    [Document Type],
    [Check Number],
    [Check Date],
    [Period],
    [Vendor],
    [Check Amount])
SELECT A.[Document Type],
       A.[Check Number],
       A.[Check Date],
       A.[Period],
       A.[Vendor],
       A.[Check Amount]
FROM OPENROWSET ( 'Microsoft.ACE.OLEDB.12.0', 'Text;IMEX=1;Database=E:\DATA\Work\temp\', 'select * from [Check Run.csv]') AS A;

And if your CSV does not have a header row, you run it like this:

TRUNCATE TABLE [ExternalTables].[dbo].[CheckData];
INSERT INTO [ExternalTables].[dbo].[CheckData] (
    [Document Type],
    [Check Number],
    [Check Date],
    [Period],
    [Vendor],
    [Check Amount])
SELECT A.F1,
       A.F2,
       A.F3,
       A.F4,
       A.F5,
       A.F6
FROM OPENROWSET ( 'Microsoft.ACE.OLEDB.12.0', 'Text;HDR=No;IMEX=1;Database=E:\DATA\Work\temp\', 'select * from [Check Run.csv]') AS A;

After the CSV is imported into SQL, we need to remove the CSV file from Windows and I use a Script activity for that:

Imports System
Imports System.Collections.Generic
Imports System.ComponentModel
Imports System.Data
Imports System.Data.SqlClient
Imports System.Text

Namespace WorkflowActivity.Scripting.RemoveTempCSVScript
    '''<summary>
    '''Provides one or more methods that can be run when the workflow scripting activity is performed.
    '''</summary>
    Public Class Script1
        Inherits ScriptClass90
        '''<summary>
        '''This method is run when the activity is performed.
        '''</summary>
        Protected Overrides Sub Execute()
            'Write your code here.
            Try
                Dim fi As System.IO.FileInfo = New System.IO.FileInfo( "E:\DATA\Work\temp\Check Run.csv")
                If fi.Exists Then
                    fi.Delete
                End If
            Catch ex As Exception
                WorkflowApi.TrackError(ex.Message)
            End Try
        End Sub
    End Class
End Namespace

Finally, I delete the CSV file from the repository.

 

Once the workflow is built, I create starting rules to trigger if a document with csv extension is created  in or moved into a specific folder.  Now anyone in the organization can export batch data, import it into the folder in Laserfiche, and have updated data for Workflow or Quick Fields to Look-up against.

3 0
replied on September 28, 2020

Would you use a CSV Data Source for the custom query pointed to the temp CSV file that is being generated in the first script?

0 0
replied on September 29, 2020

Yes, you must set up an ODBC System DSN for the CSV file with the export file name and location that is exported by the first SDK Script activity.  Then the Custom Query activity uses that System DSN to connect to the exported CSV file.

0 0
replied on September 29, 2020

Thanks for the reply Bert. I'm getting a new error now. 

And this is my code.

TRUNCATE TABLE VendorMasterTable_TEMP;
INSERT INTO VendorMasterTable_TEMP (
       [Company_ID]
      ,[Vendor_Account]
      ,[Record_type]
      ,[GApps_WW_num]
      ,[Our_AccountNum]
      ,[Vendor_name]
      ,[Search_name]
      ,[First_name]
      ,[Middle_name]
      ,[Last_name]
      ,[Vendor_group]
      ,[AP_code]
      ,[Buyer_group]
      ,[Mode_of_delivery]
      ,[Delivery_terms]
      ,[Method_of_payment]
      ,[Terms_of_payment]
      ,[cash_discount]
      ,[Report_1099]
      ,[W9_received]
      ,[Federal_TaxID]
      ,[TaxID_type]
      ,[z1099_box]
      ,[ForeignEntityIndicator]
      ,[Biz_Addr_Name]
      ,[BizCountry]
      ,[BizState]
      ,[BizCity]
      ,[BizStreet]
      ,[BizZipCode]
      ,[BizIsPrimary]
      ,[RemitTo_Addr_Name]
      ,[RTCountry]
      ,[RTState]
      ,[RTCity]
      ,[RTStreet]
      ,[RTZipCode]
      ,[RTIsPrimary]
      ,[ContactName1]
      ,[ContactType1]
      ,[ContactLocator1]
      ,[ContactExt1]
      ,[ContactName2]
      ,[ContactType2]
      ,[ContactLocator2]
      ,[ContactExt2]
      ,[ContactName3]
      ,[ContactType3]
      ,[ContactLocator3]
      ,[ContactExt3]
      ,[ContactName4]
      ,[ContactType4]
      ,[ContactLocator4]
      ,[ContactExt4]
      ,[ContactName5]
      ,[ContactType5]
      ,[ContactLocator5]
      ,[ContactExt5]
      ,[VendorAccountVendorName]
      ,[RemitToFullAddress])
SELECT A.[Company_ID]
      ,A.[Vendor_Account]
      ,A.[Record_type]
      ,A.[GApps_WW_num]
      ,A.[Our_AccountNum]
      ,A.[Vendor_name]
      ,A.[Search_name]
      ,A.[First_name]
      ,A.[Middle_name]
      ,A.[Last_name]
      ,A.[Vendor_group]
      ,A.[AP_code]
      ,A.[Buyer_group]
      ,A.[Mode_of_delivery]
      ,A.[Delivery_terms]
      ,A.[Method_of_payment]
      ,A.[Terms_of_payment]
      ,A.[cash_discount]
      ,A.[Report_1099]
      ,A.[W9_received]
      ,A.[Federal_TaxID]
      ,A.[TaxID_type]
      ,A.[z1099_box]
      ,A.[ForeignEntityIndicator]
      ,A.[Biz_Addr_Name]
      ,A.[BizCountry]
      ,A.[BizState]
      ,A.[BizCity]
      ,A.[BizStreet]
      ,A.[BizZipCode]
      ,A.[BizIsPrimary]
      ,A.[RemitTo_Addr_Name]
      ,A.[RTCountry]
      ,A.[RTState]
      ,A.[RTCity]
      ,A.[RTStreet]
      ,A.[RTZipCode]
      ,A.[RTIsPrimary]
      ,A.[ContactName1]
      ,A.[ContactType1]
      ,A.[ContactLocator1]
      ,A.[ContactExt1]
      ,A.[ContactName2]
      ,A.[ContactType2]
      ,A.[ContactLocator2]
      ,A.[ContactExt2]
      ,A.[ContactName3]
      ,A.[ContactType3]
      ,A.[ContactLocator3]
      ,A.[ContactExt3]
      ,A.[ContactName4]
      ,A.[ContactType4]
      ,A.[ContactLocator4]
      ,A.[ContactExt4]
      ,A.[ContactName5]
      ,A.[ContactType5]
      ,A.[ContactLocator5]
      ,A.[ContactExt5]
      ,A.[VendorAccountVendorName]
      ,A.[RemitToFullAddress]
FROM OPENROWSET ( 'Microsoft.ACE.OLEDB.10.0', 'Text;IMEX=1;Database=E:\LF_CSV\', 'select * from [Check Run.csv]') AS A;

I'm still using your temp csv filename btw.

0 0
replied on September 29, 2020

Start by opening your SQL Management Studio, opening a new Query Window and just run the "Truncate Table" command and see if that works.  Aslo, it is a good habit/practice to include the DB name and schema before the table name and encase the names in square brackets

TRUNCATE TABLE [DBName].[dbo].[VendorMasterTable_TEMP];

If you can successfully truncate the table, then see if you can read the CSV by using the

SELECT A.[Company_ID]
      ,A.[Vendor_Account]
      ,A.[Record_type]
      ,...
      ,A.[VendorAccountVendorName]
      ,A.[RemitToFullAddress]
FROM OPENROWSET ( 'Microsoft.ACE.OLEDB.10.0', 'Text;IMEX=1;Database=E:\LF_CSV\', 'select * from [Check Run.csv]') AS A;

By braking it down into pieces, you can get a better grip on what is causing the issues.

0 0
replied on September 30, 2020

Is there a way to denote using pipes as a delimiter instead of a comma? I know you can do it with a format file but I'd rather not have to do all that. I haven't found any parameter that would let me tell it to use pipes instead of commas.

0 0
replied on September 30, 2020

Try adding "FORMAT=Delimited(|)" to the parameters

OPENROWSET ( 'Microsoft.ACE.OLEDB.10.0', 'Text;IMEX=1;Database=E:\LF_CSV\;FORMAT=Delimited(|)', 'select * from [Check Run.csv]') AS A;

I know this works with Microsoft.ACE.OLEDB.12.0 but am not sure about Microsoft.ACE.OLEDB.10.0

0 0
replied on April 7, 2021

Hi Bert, 

I am getting this error "The OLE DB provider "Microsoft.ACE.OLEDB.10.0" has not been registered."

Any fixes that I can try out?

Thank you.

0 0
replied on April 7, 2021

@████████ what version of the Access Database Engine Redistributable did you install? Also, what architecture is your workflow and what architecture version of the Access Database Engine did you install?

0 0
replied on June 23, 2021

Bert, is there a way to have the file append to an existing file? The issue I'm running into is that my client will sometimes generate multiple files and right now the script is just deleting the old file instead of appending the new information to the existing file.

0 0
replied on June 23, 2021

You can remove the first line of the SQL custom Query

TRUNCATE TABLE [ExternalTables].[dbo].[CheckData];

This is what clears out and resets the table.

0 0
replied on June 23, 2021

I was referring to the actual CSV file that gets created. I'm not using the query to insert the values into the table. I'm using an SSIS package because the delimiter I'm using "|" is not working in the query. You tried helping me some months back but I couldn't get it to work.

0 0
replied on June 24, 2021 Show version history

This really should be its own post to deal with combining multiple csv files into a single file.  You will have to build your own workflow logic to export, but the way I would do it is search for the CSV files and then export them to a temp location in a for each entry loop.  After all the CSV files are exported (1 or more), then I would run a Script Activity to merge them into a single CSV file in the location your SSIS package is expecting it.

To merge all the CSV files in a given Windows path, something like the following should work (tested in Visual Studio but not in Workflow).

        Try
            Dim lines As HashSet(Of String) = New HashSet(Of String)()
            Dim diInput As System.IO.DirectoryInfo = New System.IO.DirectoryInfo("C:\Temp\TextCombine")
            If diInput.Exists Then
                For Each fi As System.IO.FileInfo In diInput.GetFiles("*.csv", System.IO.SearchOption.TopDirectoryOnly)
                    Using reader As System.IO.StreamReader = New System.IO.StreamReader(fi.FullName, System.Text.Encoding.Default)
                        Do While reader.Peek() >= 0
                            lines.Add(reader.ReadLine())
                        Loop
                    End Using
                    fi.Delete()
                Next
            End If
            Dim diOutput As System.IO.DirectoryInfo = New System.IO.DirectoryInfo("C:\Temp\TextCombine\Combine")
            If Not diOutput.Exists Then
                diOutput.Create()
            End If
            Using writer As System.IO.StreamWriter = New System.IO.StreamWriter(System.IO.Path.Combine(diOutput.FullName, "CombineOutput.csv"), True, System.Text.Encoding.Default)
                For Each line As String In lines
                    writer.WriteLine(line)
                Next
            End Using
        Catch ex As Exception

        End Try

 

1 0
replied on October 29, 2018

Also want to note that if your Amount field is a money (or other decimal type) you may need to use a try_parse to convert from string.

TRUNCATE TABLE [ExternalTables].[dbo].[CheckData];
INSERT INTO [ExternalTables].[dbo].[CheckData] (
    [Document Type],
    [Check Number],
    [Check Date],
    [Period],
    [Vendor],
    [Check Amount])
SELECT A.[Document Type],
       A.[Check Number],
       A.[Check Date],
       A.[Period],
       A.[Vendor],
       TRY_PARSE(REPLACE(REPLACE(A.[Check Amount], '"', ''), ',', '') AS money) AS [Check Amount]
FROM OPENROWSET ( 'Microsoft.ACE.OLEDB.12.0', 'Text;IMEX=1;Database=E:\DATA\Work\temp\', 'select * from [Check Run.csv]') AS A;

 

 

1 0
You are not allowed to follow up in this post.

Sign in to reply to this post.