Never been to DZone Snippets before?

Snippets is a public source code repository. Easily build up your personal collection of code snippets, categorize them with tags / keywords, and share them with the world

« Newer Snippets
Older Snippets »
Showing 1-10 of 10 total  RSS 

How to produce daily database table dumps in CSV format

The following code demonstrates how to produce CSV files with dynamic
file name pattern based on a current day. The produced files have the following naming format:
TABLE_NAME_MM_DD_YYYY.csv

<!DOCTYPE etl SYSTEM "http://scriptella.javaforge.com/dtd/etl.dtd">
<etl>
    <properties> <!-- Configure table name -->
        table_name=test
    </properties>
    <connection id="in" driver="auto" url="jdbc:oracle:thin:@localhost:1521:ORCL" 
      classpath="ojdbc14.jar" user="scott" password="tiger"/>
    <connection id="out" driver="csv" url="${table_name}_${etl.date.now('MM_dd_yyyy')}.csv" />
    <query connection-id="in"> <!-- Query table rows -->
        SELECT * FROM ${table_name}
        <script connection-id="out"> <!-- Export each row into a CSV -->
            $ID, $Name, $Surname <!-- Use column names from selected table -->
        </script>
    </query>
</etl>

Use Scriptella ETL to run the example.

See How to execute an ETL file from command line, Ant or directly from Java .

Importing XML into a database with Scriptella ETL

The following Scriptella ETL simple usage example imports RSS file into a database table.
<!DOCTYPE etl SYSTEM "http://scriptella.javaforge.com/dtd/etl.dtd">
<etl>
    <connection id="in" driver="xpath" url="http://snippets.dzone.com/rss"/>
    <connection id="db" driver="hsqldb" url="jdbc:hsqldb:db/rss" user="sa" classpath="hsqldb.jar"/>
classpath="hsqldb.jar"/>
    <query connection-id="in">
        /rss/channel/item
        <script connection-id="db">
            INSERT INTO Rss (ID, Title, Description, Link) 
            VALUES (?rownum, ?title, ?description, ?link);
        </script>
    </query>
</etl>


Here is the full version of the example described above. It creates an RSS table, downloads rss file, inserts rss records into a database, converts rss.xml to a plain text file and saves it to rss.txt.
<!DOCTYPE etl SYSTEM "http://scriptella.javaforge.com/dtd/etl.dtd">
<etl>
    <connection id="in" driver="xpath" url="http://snippets.dzone.com/rss"/>
    <connection id="out" driver="text" url="rss.txt"/>
    <connection id="db" driver="hsqldb" url="jdbc:hsqldb:db/rss" user="sa" classpath="hsqldb.jar"/>
    <script connection-id="db">
       CREATE TABLE Rss (
           ID Integer,
           Title VARCHAR(255),
           Description VARCHAR(255),   
           Link VARCHAR(255)

       )
    </script>
    <query connection-id="in">
        /rss/channel/item
        <script connection-id="out">
            Title: $title
            Description: [
            ${description.substring(0, 20)}...
            ]
            Link: $link
            ----------------------------------
        </script>
        <script connection-id="db">
            INSERT INTO Rss (ID, Title, Description, Link) 
            VALUES (?rownum, ?title, ?description, ?link);
        </script>
    </query>
</etl>

Script to insert BLOB from file into a database

Scriptella ETL allows inserting files into a database. This is achieved by a simple bind variables extension syntax ?{file ...}.
The following sample initializes table of music tracks. Each track has a DATA field containing a file loaded from an external location. File song1.mp3 is stored in the same directory as etl.xml and song2.mp3 is loaded from the web:
    <!DOCTYPE etl SYSTEM "http://scriptella.javaforge.com/dtd/etl.dtd">
    <etl>
        <connection driver="hsqldb" url="jdbc:hsqldb:file:tracks" user="sa" classpath="hsqldb.jar"/>
        <script>
            CREATE TABLE Track (
              ID INT,
              ALBUM_ID INT,
              NAME VARCHAR(100),
              DATA LONGVARBINARY
            );
            <!-- Inserts file with path relative to ETL script location -->
            INSERT INTO Track(id, album_id, name, data) VALUES
                   (1, 1, 'Song1.mp3', ?{file 'song1.mp3'});
            <!-- Inserts file from an external URL-->
            INSERT INTO Track(id, album_id, name, data) VALUES
                   (2, 2, 'Song2.mp3', ?{file 'http://musicstoresample.com/song2.mp3'});
        </script>
    </etl>

Copy table from one database to another

This Scriptella ETL script copies all rows from Src_Table to Dest_Table.
Src_Table contains the following columns: id, first_name, last_name
Dest_Table contains the following columns: id, name
The name column of the Dest_Table is produced by a concatenation of first_name and last_name from the Src_Table
This example demonstrates HSQLDB-To-Oracle copy procedure, although it works between virtually any databases.
<!DOCTYPE etl SYSTEM "http://scriptella.javaforge.com/dtd/etl.dtd">
<etl>
  <connection id="in" driver="hsqldb" url="jdbc:hsqldb:file:demo" 
              classpath="hsqldb.jar" user="sa"/>
  <connection id="out" driver="oracle" url="jdbc:oracle:thin:@localhost:1521:ORCL" 
              classpath="ojdbc14.jar" user="scott" password="tiger"/>
  <!-- Copy all table rows from one to another database -->
  <query connection-id="in">
      SELECT * FROM Src_Table --Selects all rows
      <!-- For each row executes insert -->  
      <script connection-id="out"> 
          INSERT INTO Dest_Table(ID, Name) 
          VALUES (?id,?{first_name+' '+last_name})
      </script>
  </query>
</etl>

Load CSV data into a database (Scriptella ETL tool)

This example demonstrates usage of Scriptella ETL Tool to load CSV data into a database table.

Input CSV file data.csv:
id,priority,summary,status
1,Critical,NullPointerException in Main class,Open
5,Low,"Checkstyle, PMD, Findbugs issues",Reopened
7,Low,Maven integration,Open
10,High,SPI API,Closed

The CSV loading script has the following content:
<!DOCTYPE etl SYSTEM "http://scriptella.javaforge.com/dtd/etl.dtd">
<etl>
  <connection id="in" driver="csv" url="data.csv"/>
  <connection id="out" driver="oracle" url="jdbc:oracle:thin:@localhost:1521:ORCL" 
      classpath="ojdbc14.jar" user="scott" password="tiger"/>
  <!-- Copy all CSV rows to a database table -->
  <query connection-id="in">
      <!-- Empty query means select all columns -->
      <script connection-id="out">
          INSERT INTO Table_Name VALUES (?id,?priority, ?summary, ?status)
      </script>
  </query>
</etl>


Use RegEx to filter CSV data:
<query connection-id="in">
    <!--Select bugs with status open or reopened.-->
    ,,,open|reopened
    <!--Inserts imported rows into a database-->
    <script connection-id="out">
       INSERT INTO Table_Name VALUES (?id, ?priority, ?summary, ?status);
    </script>
</query>

JDBC 101

// Note the use of a prepared statement. You can gain a couple of different benefits
// from using a prepared statement versus a conventional statement. One is that the
// database can compile the statement once and simply insert different values into the
// variables each time you call it. For frequently used functions this can result in
// improved performance. The second advantage is where the values to be passed in might
// be changable by hostile users (e.g. a search term taken from a webpage). In those
// cases assembled SQL strings are suseptible to attack by cleverly phrased inputs but
// prepared statements are not because they are _never_ interpreted as actual SQL, only
// as values for variables.
//
// See my other code snippets "Getting A Data Source From Tomcat" and "Data Source 101"
// for more information on how to get the data source you pass into a routine like this.

    /**
     * In this case the Record object is something we are saving to the database.
     */
    public void persistRecord(DataSource dataSource, 
        Record record) throws Exception {

        Connection conn = null;
        PreparedStatement st = null;
        
        int retVal = 0;
        
        try {
            conn = dataSource.getConnection();
            
            st = conn.prepareStatement("insert into test(a, b) values(?, ?)");

            st.setInt(1, record.getA());
            st.setString(2, record.getB());
            
            int records = st.executeUpdate();
        } catch (SQLException se) {
            log.error(se, se);
            throw se;
        } finally {
            UtilityJDBC.closeSQLClasses(conn, st, null);   
        }
    }

    /**
     * NOTE!! if you close a statement the associated ResultSet
     * is closed too so you should copy the contents to some other
     * object. The result set is invalidated also if you recycle a
     * Statement and try to execute some other query before the result
     * set has been completely examined.
     * 
     * @param conn
     * @param st
     * @param rs
     * @throws SQLException
     */
    public static void closeSQLClasses(Connection conn, Statement st,
        ResultSet rs) {

        try {
            if (rs != null) {
                rs.close();
            }

            if (st != null) {
                st.close();
            }

            if (conn != null) {
                conn.close();
            }
        } catch (SQLException se) {
            log.error(se, se);
        }
    }

Open a JDBC connection

This opens a JDBC connection givine a driver class name, database URL, user name and password. It is possible this method is deprecated and some newer procedure is preferred. I need to look into that.

    /*
    ** driverClass is the JDBC driver class name as a String.
    */

    try
    {
	Class.forName (driverClass);
	connection = DriverManager.getConnection (url,
	    userName, password);
	connection.setAutoCommit (false);
    }
    catch (SQLException ex)
    {
	/*
	** "Connect error"...
	*/
    }
    catch (java.lang.ClassNotFoundException ex)
    {
	/*
	** "Driver error"...
	*/
    }

Execute arbitary SQL in JDBC & get column names etc from the meta data

This will execute an arbitary SQL string in JDBC and extract the column names. I extracted this code from a much larger module I wrote years ago, so it isn't complete and hasn't been tested. You have been warned.

    /*
    ** connection is a java.sql.Connection obtained in the usual way.
    */

    PreparedStatement statement =
	connection.prepareStatement (sqlString);
    statement.setMaxRows (configModel.getMaxRows ());

    if (statement.execute ())
    {
	ResultSet resultSet = statement.getResultSet ();
	ResultSetMetaData metaData = resultSet.getMetaData ();

	/*
	** Get the column names.
	*/

	for (int i = 0 ; i < metaData.getColumnCount () ; i++)
	{
	    int columnType = metaData.getColumnType (i + 1);
	    String columnName = metaData.getColumnLabel (i + 1);

	    /*
	    ** Do something with columnType & columnName.
	    */

	}

	/*
	** Fetch the rows.
	*/

	while (resultSet.next ())
	{
	    String value;

	    for (int i = 0 ; i < metaData.getColumnCount () ; i++)
		value = resultSet.getString (i + 1);
	}
    }
    else
    {
	/*
	** Query was probably update/insert/delete.
	*/

	int rowCount = statement.getUpdateCount ();
    }

    statement.close ();

Getting A Data Source From Tomcat

// You can setup a data source in tomcat using a context file
// or you can set one up using the Administration web pages
// as well. Either way you do it, here is the simple code to
// get the data source from Tomcat so you can start pulling
// out database connections.

// Obtain our environment naming context
Context initCtx = new InitialContext();
Context envCtx = (Context) initCtx.lookup("java:comp/env");

// Look up our data source by the name we gave it when we created it. 
// In this case that's "jdbc/EmployeeDB".
DataSource ds = (DataSource) envCtx.lookup("jdbc/EmployeeDB");

Data Source 101

// Way too many people hard code their JDBC connections to
// the database server(s). Use data sources to connect to
// your databases. They can be setup easily in any major
// Java application server as well as JSP/servlet engines
// like Tomcat and Spring.
//
// If you aren't running inside a container that provides
// you with easy access to a data source though, you can
// include the Jakarta Commons DBCP library and create a
// data source as shown in the code below. The result is
// a data source complete with pooling, caching, etc. that
// you can pass to other code that just expects to receive
// a data source.

BasicDataSource dataSource = new BasicDataSource();

dataSource.setDriverClassName(System.getProperty("driverClassName"));
dataSource.setUsername(System.getProperty("username"));
dataSource.setPassword(System.getProperty("password"));
dataSource.setUrl(System.getProperty("url"));
« Newer Snippets
Older Snippets »
Showing 1-10 of 10 total  RSS