How to use JS to parse a csv file

Following is an example on how can we use Regular Expression to parse a CSV file in Java script

function CSVToArray( strData, strDelimiter ){
	// Check to see if the delimiter is defined. If not,
	// then default to comma.
	strDelimiter = (strDelimiter || ",");

	// Create a regular expression to parse the CSV values.
	var objPattern = new RegExp(
		(
			// Delimiters.
			"(\\" + strDelimiter + "|\\r?\\n|\\r|^)" +

			// Quoted fields.
			"(?:\"([^\"]*(?:\"\"[^\"]*)*)\"|" +

			// Standard fields.
			"([^\"\\" + strDelimiter + "\\r\\n]*))"
		),
		"gi"
		);


	// Create an array to hold our data. Give the array
	// a default empty first row.
	var arrData = [[]];

	// Create an array to hold our individual pattern
	// matching groups.
	var arrMatches = null;


	// Keep looping over the regular expression matches
	// until we can no longer find a match.
	while (arrMatches = objPattern.exec( strData )){

		// Get the delimiter that was found.
		var strMatchedDelimiter = arrMatches[ 1 ];

		// Check to see if the given delimiter has a length
		// (is not the start of string) and if it matches
		// field delimiter. If id does not, then we know
		// that this delimiter is a row delimiter.
		if (
			strMatchedDelimiter.length &&
			strMatchedDelimiter !== strDelimiter
			){

			// Since we have reached a new row of data,
			// add an empty row to our data array.
			arrData.push( [] );

		}

		var strMatchedValue;

		// Now that we have our delimiter out of the way,
		// let's check to see which kind of value we
		// captured (quoted or unquoted).
		if (arrMatches[ 2 ]){

			// We found a quoted value. When we capture
			// this value, unescape any double quotes.
			strMatchedValue = arrMatches[ 2 ].replace(
				new RegExp( "\"\"", "g" ),
				"\""
				);

		} else {

			// We found a non-quoted value.
			strMatchedValue = arrMatches[ 3 ];

		}


		// Now that we have our value string, let's add
		// it to the data array.
		arrData[ arrData.length - 1 ].push( strMatchedValue );
	}

	// Return the parsed data.
	return( arrData );
}

Camel – Unmarshal Csv to Java POJO

This document shows how to do following:

  1. Get a csv from a local folder,
  2. Unmarshal it to a Java pojo object array
  3. Split the array to each object
  4. Handle each pojo object by a customized code

Following is the pojo definition with camel csv format annotations


package com.zzyan.domain;

import org.apache.camel.dataformat.bindy.annotation.CsvRecord;
import org.apache.camel.dataformat.bindy.annotation.DataField;

@CsvRecord(separator = ",", skipFirstLine = true)
public class Item {

    @DataField(pos=1)
    private String transactionType;

    @DataField(pos=2)
    private String sku;

    @DataField(pos=3)
    private String itemDescrition;

    @DataField(pos=4)
    private String price;

/*
    here ignores all the getter and setters
*/

    @Override
    public String toString() {
        return "Item{" +
                "transactionType='" + transactionType + '\'' +
                ", sku='" + sku + '\'' +
                ", itemDescrition='" + itemDescrition + '\'' +
                ", price='" + price + '\'' +
                '}';
    }
}

We also define a customized processor, the processor will take in 1 Item object, and do some handling. Here we do nothing but just print it out.


package com.zzyan.processor;

import com.zzyan.domain.Item;
import lombok.extern.slf4j.Slf4j;
import org.apache.camel.Exchange;
import org.springframework.stereotype.Component;

@Component
@Slf4j
public class ItemProcessor implements org.apache.camel.Processor {
    @Override
    public void process(Exchange exchange) throws Exception {
        Item item = (Item)exchange.getIn().getBody();
        System.out.println(item.getItemDescrition());
    }
}

Finally we define the Camel Route to link these together.


package com.zzyan.route;

import com.zzyan.domain.Item;
import com.zzyan.processor.ItemProcessor;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.dataformat.bindy.csv.BindyCsvDataFormat;
import org.apache.camel.spi.DataFormat;
import org.springframework.stereotype.Component;

//@Component
public class UnmarshalCsvRoute extends RouteBuilder {
    @Override
    public void configure() throws Exception {

        DataFormat bindy = new BindyCsvDataFormat(Item.class);

        from("timer:hello?period=10s")
                .log("Timer Invoked and the body is ${body}")
                .pollEnrich("file:c:/data/input?noop=true&readLock=none")
                .log("body: ${body}")
                .unmarshal(bindy)
                .log("The unmarshaled object is ${body}")
                .split(body())
                  .log("new Item ${body}")
                  .process(new ItemProcessor())
                .end();
    }
}

Put the csv file with the same header to c:/data/input folder, and run the spring boot project, the Route will pickup the file and do the transformation.

 

 

Camel – Fetch file from sftp and unzip

In real time integration, especially file based integration, we normally starts from fetching file from a sftp server.

It’s common that the files put on the sftp server is a zipped file with multiple files compressed together.

Following shows how to use a Camel route to fetch a zip file from the sftp server, and after that unzip the file and put to local folders.

Please refer to Spring boot + Camel hello world to see how to setup the project before creating the Route.

The code contains both getting file from a sftp, and fetch from a local folder(commented). It also includes a Choice to do the file filtering and switch to different sub-Routes if required.


package com.zzyan.route;

import org.apache.camel.Predicate;
import org.apache.camel.builder.PredicateBuilder;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.model.dataformat.ZipFileDataFormat;
import org.springframework.stereotype.Component;

import java.util.Iterator;

//@Component
public class UnzipRoute extends RouteBuilder {
    @Override
    public void configure() throws Exception {

        // Define a Zip File format, since our Zip file have multiple zip files
        ZipFileDataFormat zipFile = new ZipFileDataFormat();
        zipFile.setUsingIterator(true);

        from("timer:hello?period=10s") // Triggered by Timer every 10 seconds
                .log("Timer Invoked and the body is ${body}")
                .pollEnrich("sftp://USERNAME@HOST/FILEPATH?password=PASSWORD&noop=true")
				// following shows pull the file from local c:/data/zipInput
                //.pollEnrich("file:c:/data/zipInput?noop=true&readLock=none")
                .log("${file:name}")
                .unmarshal(zipFile)
                .split(bodyAs(Iterator.class)).streaming()
                  .log("${file:name}")
                  .choice()
                    .when(header("CamelFileName").startsWith("Headers"))
                      .log("This is the Header")
                      .to("file:c:/data/output")
                    .otherwise()
                      .to("file:c:/data/output")
                  .endChoice()
                .end();
    }
}

 

How to use Python to connect to sftp and download file

We can use pysftp to connect to python and get the files from an sftp server.

Before that, we need to get the sftp server’s key using following command.

$ ssh-keyscan example.com
# example.com SSH-2.0-OpenSSH_5.3
example.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQAB...

When fetched the key, put it to file  C:/Users/username/.ssh/known_hosts
– You can also save the key in a folder and pass in using the cnopts variable shown below in the code

import pysftp
import io
import csv

myHostname = "HOST"
myUsername = "USERNAME"
myPassword = "PASSWORD"

cnopts = pysftp.CnOpts() # C:/Users/USERNAME/.ssh/known_hosts
cnopts.hostkeys.load('keys/testSftp.key')

with pysftp.Connection(host=myHostname, username= myUsername, password=myPassword, cnopts = cnopts) as sftp:
    print("Connection successfully stablished...")
    flo = io.BytesIO()
    sftp.getfo('/PATH_TO_DATA_FILE',flo)
    fileStr=flo.getvalue()
    textStr = fileStr.decode('UTF-8')
    print(textStr)

If everything works fine, you should see your file print in the command line

How can Java/Groovy connect to Oracle Always Free Autonomous Database

Thanks to Oracle’s Cloud Always Free services, now we can have a free VM and a free Oracle Autonomous Database.

Isn’t it excited? And then you want to do some thing with it.

1 constraint of the Always Free Oracle Autonomous Database is that, you need to use it.

If you don’t use the database within 7 days, Oracle will shut the database down, you need to start it manually later. And if you don’t use it for 3 months after that, Oracle deletes the whole instance.

So, how to avoid that? Or, how to auto do something in the database to keep it alive?

An easy way to do this is to insert some data into the database periodically.

We got a VM, and we got the database, so why don’t use the VM to insert some data into the database?

So here I come up with the “ping” groovy script, which get triggered by cron job, and inserts a dummy record every hour to the Database.

Here’s how it works.

Firstly, we install Java and Groovy into the database

How to install Java and Groovy in Oracle Always free Ubuntu

Secondly, we download the Oracle Wallet, and then, referring to Oracle document on how to use JDBC thin to connect to Database, we can download all the required jar files from Oracle Database 18c (18.3) JDBC Driver and UCP Downloads. (We will need ojdbc8.jar, ucp.jar, oraclepki.jar, osdt_core.jar and osdt_cert.jar)

Put the downloaded Jar file to your Groovy /lib directory
And put the wallet to an folder, unzipped it, we need the unzipped path later in the Groovy script.

Create a table using following script

create table ping(creation_date date)

Thirdly, we create the Groovy Script as following:

//Please note the wallet path is the unzipped path
//Replace DBNAME, USERNAME, PASSWORD to your own ones
url = "jdbc:oracle:thin:@DBNAME_high?TNS_ADMIN=PATH_TO_THE_Wallet/Wallet_DBNAME"
username = "USERNAME"
password = "PASSWORD"

println url

driver = "oracle.jdbc.driver.OracleDriver"

import groovy.sql.*
sql = Sql.newInstance(url,username,password,driver)
try{
  sql.execute("insert into ping(creation_date) values (sysdate)")
} finally{
  sql.close()
}

Finally, we create a Cron job in the crontab, and let Linux schedule and trigger the Groovy script every hour

 0 */1 * * * /home/ubuntu/.sdkman/candidates/groovy/current/bin/groovy /home/ubuntu/groovy/PingDB.groovy 

If everything works fine, you should see the Ping table been populated every hour.

Now you can enjoy a none turned off Free database and VM.

How to install Java and Groovy in Oracle Always free Ubuntu

Refer to following URL to install Java and Groovy in Ubuntu

https://computingforgeeks.com/how-to-install-apache-groovy-on-ubuntu-18-04-ubuntu-16-04/

How to Install Java on Ubuntu

http://groovy-lang.org/install.html
Install Java to Ubuntu
You need to upgrate your Ubuntu to the latest version
sudo apt update
sudo apt -y upgrade
sudo reboot
sudo apt-get install default-jdk
The jdk will be installed at /usr/lib/jvm

After this, we need to setup the JAVA_HOME environment parameter
go to folder /etc

change the “environment” file and add a new line of the JAVA_HOME
you need to use sudo vi environment to open the file and edit, since this folder needs root access

JAVA_HOME=/usr/lib/jvm/default-java

after saving the file, use to load the new profile and double check the setting
source environment
echo $JAVA_HOME

Install Groovy
before installing Groovy, we need to install zip and unzip tool
sudo apt-get install -y zip
sudo apt-get install -y unzip

and after this, we refer to following Groovy official document to install Groovy
http://groovy-lang.org/install.html

We need to use sudo to install the sdkman
sudo curl -s get.sdkman.io | sudo bash

we need to change user to root to execute the sdkman-init.sh
sudo -s
source “$HOME/.sdkman/bin/sdkman-init.sh”

now install groovy using root account
sudo -s
sdk install groovy
groovy -version

 

The groovy was installed under following directory
/home/ubuntu/.sdkman/candidates/groovy/current/bin

 

need to add the same to environment so it will take in the setting in PATH
/etc/environment

now we can try to deploy the groovy script in the ubuntu server and try to run them

How to get the PO import Errors

Use following sql script to get the PO import errors.

Same sql can be created as a bi publisher report, and schedule to deliver the output to the key user daily.

SELECT
a.batch_id,
a.INTERFACE_HEADER_KEY,
a.BILLTO_BU_NAME,
a.PRC_BU_NAME,
a.REQ_BU_NAME,
a.DOCUMENT_NUM,
a.AGENT_NAME,
a.VENDOR_NAME,
a.VENDOR_SITE_CODE,
a.LOAD_REQUEST_ID,
b.COLUMN_NAME,
b.COLUMN_VALUE,
b.ERROR_MESSAGE,
b.ERROR_MESSAGE_NAME,
b.TABLE_NAME
FROM  PO_Headers_Interface a,
PO_Interface_Errors b
WHERE 1 = 1
AND a.INTERFACE_HEADER_ID = b.INTERFACE_HEADER_ID
AND b.ERROR_MESSAGE  !=  'The order number must be unique in a sold-to legal entity. '
and a.request_id = :request_id

How to get Suppliers Import Errors

When triggers the supplier import for Procurement Cloud, we need to check the import errors.

It’s tedious to check the standard pdf report generated by the import process.

Instead, we can create following reports and schedule it to run daily, to delivery the errors to a key user to check the detail issue.


-- Rejection for Supplier Header
select rej.REJECT_LOOKUP_CODE, sup.* from POZ_SUPPLIER_INT_REJECTIONS rej, POZ_SUPPLIERS_INT sup
where rej.request_id = :request_id
and rej.reject_lookup_code not like 'Supplier Number%already exists. Review the supplier numbering setup.%'
and rej.reject_lookup_code not like 'A record with the value%already exists. Enter a unique value.%'
and rej.PARENT_ID = sup.VENDOR_INTERFACE_ID

-- Rejection for Supplier Addresses
select rej.REJECT_LOOKUP_CODE, adr.* from POZ_SUPPLIER_INT_REJECTIONS rej, POZ_SUP_ADDRESSES_INT adr
where rej.request_id = :request_id
and rej.reject_lookup_code not like 'The value already exists. You must provide a unique value.'
and rej.PARENT_ID = adr.address_interface_id

-- Rejection for Supplier Sites
select rej.reject_lookup_code, site.* from POZ_SUPPLIER_INT_REJECTIONS rej, POZ_SUPPLIER_SITES_INT site
where rej.request_id = :request_id
and rej.PARENT_ID = site.VENDOR_SITE_INTERFACE_ID
and rej.reject_lookup_code not like 'The site name already exists in the given procurement business unit for the supplier indicated in the record.%'

-- Rejection for Site Assignments
select rej.REJECT_LOOKUP_CODE, ass.* from POZ_SUPPLIER_INT_REJECTIONS rej, POZ_SITE_ASSIGNMENTS_INT ass
where rej.request_id = :request_id
and rej.reject_lookup_code not like 'The assignment for the Client BU already exists for the given supplier, site, and procurement business unit combination indicated in the record%'
and rej.PARENT_ID = ass.ASSIGNMENT_INTERFACE_ID

-- IBY_TRANSACTION_ERRORS contains the detail error message for the Bank account import
-- Bank Account Import Errors
select * from IBY_TRANSACTION_ERRORS where creation_date > sysdate -1 and error_message not like 'The external bank account already exists. The details included in the interface table are ignored%'
and error_message not like 'The external payee already exists. The details included in the interface table are ignored%'

How can Java run Groovy script dynamically 2

This is a continue with the previous blog “How can Java run Groovy script dynamically”.

We try to do some complex things inside the groovy script, this time we tried to define a data structure directly inside the script, and see if the GroovyShell can identify it at run time and run correctly

Following are the sample script that passed to the GroovyShell

The code refers to “How to use a Groovy trait to output any class as csv?”

import java.io.ByteArrayInputStream
import java.io.Reader
import java.io.InputStreamReader
import static com.xlson.groovycsv.CsvParser.parseCsv

def sb = StringBuilder.newInstance()

/*
// TEST1 - simple CSV parsing
Reader inputReader = new InputStreamReader(new ByteArrayInputStream(input));
for(line in parseCsv(inputReader,separator:",")){
  // println line
  sb.append(line.attribute1+";"+line.attribute2)
  sb.append("\n")

}

output = sb.toString().getBytes()
*/

// TEST2 - Define data structure inside the dynamic script and gets output
def m = new MeetupMember()
m.name = "Sergio del Amo"
m.locality = "Guadalajara"
m.twitter = "https://twitter.com/sdelamo"
m.facebook = null
m.tumblr = null
m.imageUrl = "http://photos4.meetupstatic.com/photos/member/c/d/6/b/member_254392587.jpeg"
m.website = "http://www.meetup.com/es-ES/Warsaw-Groovy-User-Group/members/200767921/"

output = m.asCSV().getBytes()

// Following are the data structure definition
// You can see that this part is at the end of the script
// But GroovyShell handle it correctly
trait TraitAsCSV {
// manually provide the header to make sure the csv output is in the expected sequence
   List<String> propertyNames() {
        return ["name","locality","twitter","facebook","tumblr","background","imageUrl","website"]
    }

    String csvHeaders() {
        propertyNames().join(delimiter())
    }

    String asCSV() {
        def str = ""
        def arr = []
        for(def propName in propertyNames()) {
            def v = this."$propName"
            arr <<  (v ?: "")
        }
        arr.join(delimiter())
    }

    static String delimiter() {
     ";"
    }
}
class MeetupMember implements TraitAsCSV {
    String name
    String locality
    String twitter
    String facebook
    String tumblr
    String background
    String imageUrl
    String website
}

How to download a file by Code in VBCS?

Following the previous blog of uploading file to VBCS, this time, we handle the download issue.

We can fetch file contents from external web service – etc, fetch a pdf binary string from DB or other report services, or manually construct a csv file content, and put to a VBCS page variable.

OK, and then, how can we download the file from the variable to local PC?

You can add the following function code to page function of VBCS, and hook it up in action chain, pass in the variable with file content, it will directly download the file in your browser.

Please remember to change the file type to the one you required, following code is generating a pdf file

Cheers!

// following is downloading a pdf,
// you can download other files, you just need to change the file type
PageModule.prototype.downloadFile = function(fileBytes){
var blob = new Blob([fileBytes],{type:'application/x-pdf'});
var filename = "test.pdf";

if(navigator.msSaveBlob){ // IE 10+
  navigator.msSaveBlob(blob,filename);
} else {
  var link = document.createElement("a");
  if(link.download !== undefined){ // feature detection
	// Browsers that support HTML5 download attribute
	var url = URL.createObjectURL(blob);
	link.setAttribute("href",url);
	link.setAttribute("download",filename);
	link.style.visibility='hidden';
	document.body.appendChild(link);
	link.click();
	document.body.removeChild(link);
  }
}

};