TurnOver PDQ
As seen in
SoftLanding Systems Keeps Logistics Supply Chain Moving
by Stephen Elsemore
 Menlo Logistics is a contract logistics firm specializing in cost-effective management of the entire supply chain for such major clients as Hewlett-Packard, Nike, IBM, Dow Chemical, and Frigidaire. While logistics is a complex science that uses leading edge technology to oversee and integrate all phases of the product cycle from raw materials procurement to manufacturing, storage, and distribution, Menlo’s mission is simply stated: "Our goal is to make the world’s most complex supply chains more efficient."
 Menlo's IT department teamed with Softlanding to convert 169 physical files containing 112 million records and rebuild more than 300 logical files. Since it was founded in 1990, Menlo has grown rapidly, doubling its business volume in the past three years. Menlo’s 4,000 employees manage 9 million square feet of warehouse space, more than 2 million shipments annually, and worldwide daily freight and inventory valued at about $1.15 billion. In one day, Menlo oversees the movement of 34 million pounds of freight. For Menlo, the development of the innovative technologies that give its customers a competitive edge is an ongoing process. This quest for creative new solutions allows Menlo to maintain its leadership position in the logistics industry, but also presents challenges, particularly that of implementing these enhancements with minimal business disruption for its clients.
Menlo recently completed development and testing of a year-long project that introduced significant new functionality to its systems. Richard Atwater, senior analyst, is responsible for AS/400 administration and support at Menlo. He described the dilemma that Menlo faced when it came time to go live with the enhancements.
"The AS/400 is a critical part of our transportation management, EDI, and shipment tracking," Atwater said. "This project required modifications and conversions of all our major AS/400 databases. We needed to convert 169 physical files containing 112 million records. Over 300 logical files needed to be rebuilt, and more than 1,100 programs needed to be modified or recompiled."
Using traditional methods, the conversion would have made mission-critical applications unavailable while millions of database records were copied and converted, hundreds of access paths were rebuilt, and the new files and programs were moved into production. Atwater estimated the potential downtime to be about 85 hours. "I don’t know how to put a value on the business disruption that four days of downtime would cause, but it would be substantial," Atwater said. It wasn’t an option for Menlo.
Fortunately, Menlo was aware of the challenge from the outset. Along with Atwater, Menlo development managers Tom Williams and Desiree Barrett and project team leader Scott Farestrand were charged with finding a solution to the problem.
As the software development was underway, the Menlo team worked in parallel to find a way to reduce the downtime required to update production. They discovered that there were plenty of tools on the market to facilitate the development effort, such as cross-reference software, conversion utilities, and testing tools. But, once it came time to put the new version into production, nothing promised to reduce downtime to an acceptable level.
Menlo Logistics uses the AS/400 to manage 9 million square feet of warehouse space and more than 2 million shipments annually. PDQ Reduces Downtime
A strategy began to emerge for the team when they learned that two software vendors, Lakeview Technology and SoftLanding Systems, Inc. were working jointly to provide a solution. Menlo was familiar with both companies because SoftLanding’s TurnOver change management and Lakeview’s MIMIX availability management software were already well-established elements of Menlo’s IS strategy. The companies were a good fit for this venture, bringing together Lakeview’s experience with high availability and SoftLanding’s expertise in change management. The resulting product, PDQ, closely integrated with SoftLanding’s TurnOver and promised to reduce downtime by as much as 98 percent. The choice was simple for Atwater. "We reviewed strategies and didn’t find any other product that fit our needs."
The heart of PDQ is the CPYACTF command used to copy a file while it is in use and even being updated. This is accomplished by using journaling to capture updates made to the file during the copy, and merging those transactions with the copied data so no updates are lost. Once the copy is complete, it’s necessary to allocate the file only for the time it takes to apply any outstanding journal transactions (there are very few, as PDQ continually processes the transactions during the copy), and then move the old versions of the files out and the new versions in. It doesn’t matter if it takes days to copy the file; users of the application will not be affected until it’s time to move the new objects to production, a process that can take minutes instead of hours.
Although powerful, the CPYACTF command is not a complete solution. Menlo had to convert not one, but 169 files. Not only did all the files need to be copied and converted, but it was essential that all the copies complete at the same time so they could be moved into production together. Even though some of the files contained thousands of records, while others contained millions, PDQ successfully allowed Menlo to synchronize the conversion process for all 169 files.
Menlo followed the recommended PDQ procedures. Before beginning to copy records, Menlo IT staff used the TCRTPDQF command to create a control file representing a list of files to be copied simultaneously. The TADDPDQFE command was then used to add entries to the control file. Each entry identified the from-file, the target-file, and the conversion method to use.
The conversion method can be *MAP, *NOCHK, or any conversion method that’s available to the CPYF command. A conversion program or service program can be used as well. Regardless of the conversion method, it’s applied to the records that are copied, as well as to any updates that are made based on the journal transactions captured during the copy.
Next, the TCPYACTF command was submitted to batch. The parameters to this command included the name of the control file and the name of a lock management exit program. The TCPYACTF job then submitted two jobs for each file: one that did the copying, and a "journal scraper" job that simultaneously processed the updates that happened while it was copied. Once all the copy and journal scraper jobs were submitted, the TCPYACTF job acted as a monitor, periodically checking the status of the other jobs. As each file copy finished, the job idled until all the other copies also finished.
Meanwhile, the journal scraper jobs continued to apply transactions to the target files. When all the files had been copied, TCPYACTF passed control to the lock program, which is responsible for checking for locks and taking the necessary steps to eliminate them before passing control back to TCPYACTF. Only then did PDQ allocate the files exclusively so that any outstanding transactions could be processed and the objects moved to production.
The Proof Is in the PDQ
Before using PDQ to convert a production database, Atwater and Farestrand checked out the product using test databases. During the first phase of testing, they used the commands described to run scenarios to prove that the PDQ technology worked as advertised. "We helped identify a few issues in the early release, but we found SoftLanding’s support to be excellent and any initial problems were quickly resolved," Atwater said.
Next, the conversion team tested PDQ with SoftLanding’s TurnOver, the change management system used at Menlo. Since TurnOver already included a PDQ interface, they created and submitted a TurnOver promotion request, including all the files and related objects to be promoted using PDQ. TurnOver handled creating and populating a PDQ control file and running the TCPYACTF command before moving the objects to production. "The integration of PDQ with TurnOver is seamless, and that helped us immensely," Atwater said.
Farestrand was also impressed by PDQ’s TGENCNVPGM command, which compares two versions of a file, shows the user the differences, and generates a file conversion program based on the user’s selections. PDQ allows data to be converted using a service program that is dynamically bound by the copy job and does no I/O but simply converts each record by updating a buffer identified by a pointer, resulting in excellent performance. Farestrand said, "We think the process is faster than copying a file using *MAP."
Within weeks, Menlo had promoted all the database changes and related program objects into staging libraries, and PDQ had been thoroughly tested. The conversion team then used PDQ and TurnOver to convert the data from the production libraries to the staging libraries, then to move the converted files and program objects into production.
During the several days that the copy jobs were active, all of the applications were available to Menlo’s users. The team estimated that 60,000 updates took place during this time, all of which were applied to the staging files. Once all the copies had completed, the lock program caused the jobs to idle until it was convenient to shut down the applications in order to apply outstanding transactions and move the objects to production. The result was that the applications were only offline for one off-peak hour, compared to the original estimate of four days; good for Menlo Logistics, and good for its clients.
SoftLanding Systems, Inc. can be reached at:
SoftLanding Systems, Inc.
84 Elm St.
Peterborough, NH 03458
Tel: 800-545-9485 or 603-924-8818
Fax: 603-924-8508
Web: www.softlanding.com
Stephen Elsemore, vice president of development at SoftLanding Systems, Inc., specializes in helping AS/400 shops manage changes to software and data files. He can be reached at stevee@softlanding.com .
This material has been reprinted with permission from the April 1999 issue of AS/400 Technology SHOWCASE published by IIR Publications, Carlsbad, CA © 1999. All rights reserved.
| 
