CoSort Speeds Valdero Supply Chain Data Integration
Reviewer
Krishanu Lahiri, integration manager for Valdero Corporation.
Background
Valdero Corporation is the leading provider of enterprise-class software applications for real-time supply and demand chain execution. Valdero creates high-velocity supply and demand chains through real-time visibility, collaboration and execution. Applicable across manufacturing companies in the high technology, medical device and consumer markets, Valdero solutions are currently in use at industry-leading companies. Valdero's management team has deep supply chain experience with premier technology companies and a demonstrated track record in building and managing customer-focused organizations. Valdero's real-time supply chain execution applications give manufacturers visibility across multi-tier supply and demand chains, enabling them to rapidly respond to fluctuations in demand and supply as they occur.
Platforms
Solaris, Linux, Windows NT/2000.
Problem Solved
A major strength of the Valdero solution lies in its ability to process large amounts of supply chain data in near-real time. This data is sourced from both internal systems as well as from the various nodes of the supply and demand chain. Typically, within the same implementation, data is received over different transport mechanisms varying from bulk file transfers to message queues. Integrating these varied sets of data involves identifying changes, performing data quality checks and transforming incoming data to match application standards before loading the active repository. One of the main constraints to this entire process is that it must occur with minimal impact to online activity. We use CoSORT's sort control language (sortcl) program to sort, transform and join files outside the database to accomplish large parts of our integration processing. We use our proprietary Application Integration Manager (AIM) to load data into our Active Repository. We also use CoSORT extensively within AIM to identify and filter the data that has changed in the current ETL batch (internally known as the Netchange) process. Because the Valdero application sources data in different formats from multiple locations, it is very critical to our daily batch loads to narrow the records to only those that have changed since the last successful load. CoSORT utilities improve the performance of the Netchange process and provide an easily configurable framework to perform the Netchange on different files.
Product Functionality
Among the CoSORT package's many user interfaces and API calls is a 4GL for data definition and manipulation (integration, staging and reporting) called sortcl. By wrapping Python script around CoSORT's sortcl program, we can maintain multi-iplatform support for AIM. This has been a very important factor in selling the Valdero AIM process to our customer IT departments, as they can maintain the same code base and performance levels across multiple platforms. In the future, we plan to use CoSORT's native support of different data types, as well as sortcl's filtering capabilities to enhance the data transformations supported by the Valdero AIM process. We can also count on CoSORT's superior sort-join performance for surrogate key lookups within AIM.
Strengths
The primary strength of the CoSORT sortcl utility that we leverage in the Valdero AIM process is the enhanced performance on multi-CPU machines. The ability to process hundreds of thousands of input rows and quickly filter them based on various criteria helps in significantly improving the overall time it takes to refresh the Valdero repository with minimum impact to the application users that are online.
Weaknesses
sortcl does not allow more than one character per field to be used as an internal separator. This has caused us to "escape" special characters before processing them with CoSORT.
Selection Criteria
We required a product that has stood the test of time, is available on all of our target platforms and has a price point that would enable us to bundle it with our solution. After researching the open systems sorting and ETL tool markets, CoSORT was the best choice.
Deliverables
We use CoSORT primarily as a data processing tool with minimal reporting usage. CoSORT's sortcl program is invoked from within our application on dynamically generated transform information.
Vendor Support
Vendor support has been excellent. We have had quick responses from CoSORT's support team via e-mail and phone. The best part is that our contacts are always abreast of the issues.
Documentation
While the documentation is excellent, the sample programs and examples that are bundled with the product are an invaluable pointer to develop programs using CoSORT. The product is very easy to install, and the documents guide us through all the different functionality provided by CoSORT.
