A MVS SYSTEM SOFTWARE MANAGEMENT METHODOLOGY
By Bob Krause

The author is a senior systems programmer for Simpson Paper Co.
in Anderson, Calif. He has 10 years of systems programming
experience, specializing in MVS system software installation and
maintenance.

Getting Started

I recently proposed and justified a system software installation
and maintenance methodology for an installation new to MVS. I was
forced to commit my years of knowledge and experience installing
and maintaining MVS onto paper, and to try to sell my ideas to
management. This article is taken from the result.

Most of the technical details and techniques are left to the
systems programmer's imagination. This overview is designed to
give perspective to the various processes many of us engage in,
thus providing some insight into how we might describe them to
non-systems programmers.

The methodology was to be simple, comprehensible, accomplished
with a minimum of human resources, and not require a large amount
of DASD.

Basically, the methodology consists of building a target system,
installing preventive maintenance, installing updated products,
testing the system thoroughly, moving the updated system into
production and providing a RESCUE system.

System Software Management

Maintaining MVS/ESA and its component system software at current
maintenance and release levels is the foundation of a sound
systems software management strategy. By keeping system software
current, the installation protects its total software and
hardware investment, and ensures that new software capabilities
and hardware advances can be capitalized upon to gain a
competitive advantage.

Failure to maintain software currency increases the likelihood
that costly software upgrades or conversions will become
necessary to meet new competitive business challenges.
Installations that intend to use leading edge software and
hardware in their strategic business systems must keep their
system software current.

Installations that fall behind in software release and
maintenance levels find that discovering and fixing system
software problems is tedious and time-consuming process, often
requiring major maintenance upgrades to correct minor problems.
Help from IBM in supporting outdated IBM system software can be
minimal or non-existent.

An installation can avoid the problems associated with aging and
obsolete software resources by setting up a solid installation
and maintenance methodology, setting a regular schedule for
applying maintenance, and installing updated software releases.

Post Express: Maintaining and Updating System Software

MVS/ESA EXPRESS is the delivery vehicle for implementing MVS/ESA
in an installation new to MVS. Once an installation has
installed, the EXPRESS system, other installation and maintenance
delivery options such as CBIPO, CBPDO or PUT must be used to keep
the system up-to-date. The new MVS installation should not delay
in setting up a regular maintenance schedule. As the system ages,
the options for updating it become more limited, and the task of
updating it becomes more complex and costly.

SMP/E is the software tool used for updating system software with
maintenance or new releases. SMP/E provides change management and
an audit trail for these processes. A prerequisite for the
implementation of any system software update process is the
restructuring of the EXPRESS delivered system into a form that is
easily copied to a duplicate target system.

The Target System

A target system is essential for an installation that intends to
keep its software up-to-date. A target system is a copy of the
operating system that is used for applying updates. A copy of the
production system is used so it doesn't interfere with or disrupt
the production environment. Software maintenance or upgrades
should never be applied to a running production system.
Maintenance and upgrades should always be applied to a test
target system. All changes must first be made to this target
system, then tested and then moved into production.

Having and using a target system requires additional DASD. Space
must be reserved for duplicate copies of all system software. To
allow for the movement of the test system into production in a
timely manner, dedicated volumes must be used. High density DASD
devices are recommended: The fastest and easiest method of
updating and moving a target system into production is by having
the system occupy as few logical volumes as possible.

Building the Initial Target System

A system properly structured on one or two volumes can easily be
cloned or copied using full volume or data set copy utilities. A
test system clone of a production system can easily be tested and
moved into production. This approach ensures a fast and reliable
fallback to the previous system should problems, not discovered
during testing, appear soon after cutover. The initial build or
restructuring of the target system after an EXPRESS or CBIPO
install is a key task and should be accomplished with care and
precision.

There must not be any data sets on the system volume that are
updated outside of the SMP/E update processes. Changes to data
sets made by the system or other users and systems programmers
will be lost in the system cutover process. This includes system
data sets such as PARMLIB, PROCLIBs, IMAGELIB, LOGREC, VTAMLST,
BRODCAST, DAE, UADS, the RACF data base, SMF data sets, IPCS data
sets, JES data sets, catalogs, page and STGINDEX data sets, or
any user data sets. For data sets that are updated by both SMP/E
and users, dummy uncataloged copies of these data sets should be
maintained on the target volume for use by SMP/E. CICS, DB and
NCP software should be kept on separate volumes that are also
duplicative. Non-IBM software should be kept on another
duplicative volume.

The Cloning Process

The cloning process not only includes making a copy of the
production system libraries, but also making copies of the SMP/E
zones that describe the system. SMP/E target and DLIB zones are
used by the SMP/E processes to install and update system
software. These zones contain descriptions of the components of
the system software and information on how they are to be
installed or updated.

SMP/E zones must be kept strictly synchronized with the system
libraries they describe. Once a system's integrity has been
compromised by out of sync target libraries and SMP/E zones, it
can become unusable. Installation and maintenance procedures must
ensure that this does not occur, and provide reliable backup in
the event that it does.

The SMP/E cloning process consists of copying the SMPE zones to
test zones. The duplicated zones must then be updated to point to
the duplicated libraries on the target volumes. Once a completely
separate test environment has been built, it can be updated by
maintenance or new product installs.

The Vanilla System: Keeping Software Maintenance Do-able

A key to success when maintaining system software is to
accomplish it in a timely and cost-effective manner. To complete
software maintenance in a reasonable time frame and with a
minimum impact on the production system, user installed system
modifications should be kept to a minimum and be carefully
managed. Unmanaged and undocumented system modifications can
cause program failures and system outages when attempting to
upgrade or maintain software.

When user modifications are unavoidable, they should be packaged
in SMP/E format and applied using SMP/E. SMP/E packaged USERMODs
are capable of warning when USERMOD elements will be overlaid by
maintenance or new product installs. When preventive service or
updated releases are applied to the system, SMP/E installed
USERMODs can easily be removed from the system and then
reinstalled after the system has been updated.

Installing Preventive Service

A stable system can turn unstable when external conditions
change, such as the number of users, addition of new or upgraded
hardware, addition of new software, or by exercising previously
unused system components. All of these conditions can occur in an
installation that is rapidly growing and evolving. Most system
software errors have already been discovered by IBM or other
users, and fixes have been provided as preventive or corrective
service Program Temporary Fixes (PTF).

To avoid discovering and debugging problems already solved, and
to be prepared for any possible environment changes, preventive
service should be installed on a regularly. The more backlevel a
system is, the harder it will be to apply the corrective fixes
that will be required in the future.

Service Delivery Options

IBM service is distributed via several methods: PUT tapes are
provided at intervals during the year and CBPDO is available upon
request. Both contain preventive service since the last service
update in the form of PTFs. CBPDO also contains corrective reach
ahead service (not available on the most recent PUT tape) and PUT
Preventive Service Planning (PSP) information.

CBPDO cannot be used if the system has not been updated via CBPDO
or PUT within the last two years. For an installation that
intends to keep its systems software reasonably current, CBPDO is
the preferred alternative, and is recommended by IBM.

The Maintenance Process

The process of applying software maintenance uses SMP/E, one of
the above delivery options and the cloned target system.
Preventive service PTFs are RECEIVEd from the delivery tape.
HOLDDATA describing PTFes in error are also RECEIVEd. The desired
service level is then determined. PSP information about the
maintenance levels to be applied must be reviewed and the latest
HOLDDATA obtained from IBM.

The most aggressive approach is to install all of the maintenance
on the CBPDO or PUT tape. A more conservative approach is to
install service to a PUT level one or two months behind service
of the current PUT tape. This helps ensure that fewer of the PTFs
that are applied will become PTFs in Error (PEs). By providing
this buffer, time is allowed for other installations to find and
fix the problems. Some products, however, require that
maintenance be kept up-to-date (i.e., IBM recommends this for
DB2). Regardless of the approach used, PTFs that become PEs after
the maintenance is applied should be tracked and resolved as part
of an ongoing maintenance review process.

The maintenance process consists of APPLY CHECK and APPLY steps
directed at the target system. Exception conditions must be
resolved, and USERMODs and APARS must be RESTORed before the
APPLY, then reapplied afterward. Detailed maintenance install
tasks should be documented in a task checklist, and backups
should be taken at strategic points to checkpoint the process.

Installing New Product Releases

Updated releases of system software products are provided on a
regularly. These updates may introduce new functions, support for
new devices, or correct software deficiencies and performance
problems. If these updates are not installed, and IBM Central
Service support has been discontinued for the older release of
the product, the user is responsible for the support of the
product and for fixing problems. The result can be increased
maintenance costs, unmaintainable software or obsolete systems.
Software releases should be kept current to avoid these problems
and to position the installation to take advantage of new tools
and technology soon after they become available.

New Product Delivery Options

IBM products are distributed by several methods: CBPDO, CBIPO and
individual product orders.

CBPDO delivers the product and service together on one tape and
includes PSP information, program directories and HOLDDATA.

CBIPO is a complete system or subsystem replacement that includes
all products and integrated service.

Individual product orders provide the product and a separate
cumulative service tape.

CBPDO is the recommended alternative for adding new products,
while CBIPO is preferred when doing a complete system
replacement. Depending on the degree of user customization
required, the number and complexity of installed vendor products,
and the simplicity of the install path, CBPDO may be more
cost-effective even when replacing or updating major system
components. Newer products may not be available via CBIPO, while
older releases may not be available via CBPDO.

The Software Product Install Process

The install process for the CBIPO system replacement option
consists of downloading the CBIPO system from tape and then
recustomizing it to production requirements. The system will need
to be restructured for future maintenance or installs if they are
to done via the other delivery options.

The process of applying new or updated software products with the
other delivery options utilizes SMP/E, the delivery tapes and the
cloned target system. Products and their service are RECEIVEd
from the delivery tape or tapes. HOLDDATA is also RECEIVEd. PSP
information about the software product levels to be applied must
be reviewed and the latest HOLDDATA obtained from IBM.

The install process continues with an APPLY CHECK followed by an
APPLY directed at the target system. Exception conditions must
resolved and USERMODs and APARS must be RESTORed before the
APPLY. After the APPLY, the product is ACCEPTed into the system
DLIBS. An APPLY CHECK is then run for any maintenance to the
product, exception conditions again resolved, and the maintenance
APPLY run. Any restored USERMODs and APARs must then be reapplied
if still applicable. Once again, detailed product install tasks
should be documented in a task checklist, and backups should be
taken at strategic points to checkpoint the process.

Testing the Updated System

Once the software and or maintenance has been installed on the
target system, it must be thoroughly tested before it can be
cutover into production. Test time must be scheduled on the
production machine or a separate test machine used. Time and care
must be taken to test as many system functions as possible.
Prudent testing is the critical success factor for the entire
methodology.

After a sufficient period of testing (determined by the
complexity of the changes and the potential impact on the end
users), the system is scheduled to be cutover into production.

Moving the Updated System Into Production

At the scheduled cutover time, the test system is IPLed in
production mode. Additional testing may be required at this time,
perhaps to test devices not available on the test system. If all
goes well, the system is put first into limited and then full
production. The original system volume is still available as a
live fallback. If unsolvable problems arise with the updated
system, the old system can be reIPLed from the original system
volume.

After the system has run without incident for an acceptable
period of time, the old system volume can be backed up and the
device used in the next cloning process. All outstanding service
should be ACCEPTed before the next maintenance or product install
process begins.

The RESCUE System

The concept of a RESCUE system is an extremely important part of
the software management plan. The ability to get out of
unexpected difficulties resulting from hardware failure, software
failure or human error as quickly as possible is invaluable. The
RESCUE system is a completely self-contained operating system
that exists on a single volume specifically for the purpose of
recovery.

The RESCUE system should be kept current, particularly if major
changes such as updated configuration or new hardware support are
made to the environment. A properly implemented and maintained
RESCUE volume will prove its worth many times over on the first
instance that it is needed.

/*

Was this article of value to you? If so, please let us know by
circling Reader Service No. 00.

2624