Real World NetBackup 7.5 Upgrade Catalog Migration Speed Comparison

During the NetBackup 7.5 Upgrade, part of the catalog is moved from the flat files into the EMM database. This may give you cause for concern as this seems like it may take a long time to complete. In my experience, it has not taken very long and conforms to the specs that Symantec has outlined in the Upgrade Guide. I have done a few upgrades and wanted to share some real world experience. Your results may vary, but at least you have something to draw on from my experiences:

Original Version	Upgraded Version	Master Make/Model	# Images Converted	Images per Second	Time Elapsed HH:MM:SS
7.1.0.4	7.5.0.4	Oracle T2000	22,673	21	00:22:51
7.1.0.4	7.5.0.4	Oracle T5220	69,399	37	00:36:01
7.1.0.4	7.5.0.4	Oracle T2000	64,931	14	01:23:38*
7.1.0.4	7.5.0.4	NBU 5220 Appliance	256	128	00:00:16
7.1.0.4	7.5.0.4	NBU 5220 Appliance	4,559	55	00:49:40
7.1.0.4	7.5.0.4	NBU 5220 Appliance	176	1	00:04:54

* We found that this server’s catalog disk was on 7200rpm SATA which contributed to the poor performance.

Note1 – The time shown is the total time it took for the Image Cleanup job to run and complete. Obviously, the catalog migration portion is faster, and in some it is significantly faster. The remaining time is the image deletion on the media servers. I wanted to show the entire time it took so that you can have a feel for how long this may take. You can take steps in shortening the Image Cleanup run time by running it manually before the upgrade.

Note 2 – This only concerns the actual image migration after the server was upgraded to 7.5 and had the latest patch applied. The actual time to install the base upgrade and patch is not accounted for here.

 

NetBackup Appliance Patch Upgrade Fails Self Test

The NetBackup Appliances have a self test that it runs through to check various components of your appliance to ensure it is ready for the patch. I ran into a problem where the appliance was fine, but the self test failed.

My symptom was a self test output that looked like the following:

Running Sanity tests on Media Server Appliance.

Checking if installation is successful ...                       [OK]
Checking if SSH is enabled ...                                   [OK]
Checking if hostname is set ...                                  [OK]
Checking if IP address is set ...                                [OK]
Checking if appliance user accounts exists ...                   [OK]
Checking if required packages are installed ...                  [OK]
Checking if NetBackup is configured and running ...              [FAIL]
Checking if storage units and diskpools have been created ...    [FAIL]
Checking if SLPs have been created ...                           [NA]
Checking if required NetBackup policies have been created ...    [NA]
Checking if OpsCenter is configured and running ...              [OK]
Checking test backup and restore operations ...                  [OK]
Expiring test backup images and deleting test policy ...         [OK]
Checking IPsec setup on the box ...                              [NOT CONFIGURED]

Completed running tests on the appliance.
SELF TEST RESULT :                                               [FAIL]

I discovered that there are scripts that perform this self test and it looks for the disk pools in use by the appliance and somehow it expects PureDisk pools. In my case, I have an Advanced Disk pool and the command it was using failed since it was expecting PureDisk. The scripts are located in /opt/NBAppliance/scripts and are named self_test.pm and self_test.conf. I was able to edit these so that it would skip portions of the test to make it valid and continue with the patch. I do NOT recommend this if there are really some problems with your appliance.

Your first step should always be to make a backup copy in case your changes cause more issues, you can easily back them out by copying the original script back.

In /opt/NBAppliance/scripts/self_test.pm, find the following lines and comment them out as shown below. This will cause the script to not check for any PureDisk pools and storage units:

# @names = getStuNames( $master, $host, 0, "PureDisk" );
# if( scalar (@names) > 0 )   {
#  $backup_stu = $names[0];
#  $error->logDbg("PureDisk Storage Unit already exists. Will use $backup_stu for test backup.",__FILE__,__LINE__);
#} else {
{    <------Note, I had to add this curly bracket since it was commented out in the line above.
@names = getStuNames( $master, $host, 0, "AdvanceDisk" );
if( scalar (@names) > 0 ) {
$backup_stu = $names[0];
$error->logDbg("AdvancedDisk Storage Unit already exists.Will use $backup_stu for test backup.",__FILE__,__LINE__);
}
}

In /opt/NBAppliance/scripts/self_test.conf, find the following lines and comment them out as shown below. This will cause the script to not check for any PureDisk processes:

Begin_PureDisk_Process_Section
#/usr/openv/pdde/pddb/bin/postmaster -D /disk/databases/pddb/data
#/usr/openv/pdde/pdcr/bin/spad -q
#/usr/openv/pdde/pdcr/bin/spoold
End_PureDisk_Process_Section

After making these changes and saving them, I tested it by running test software from the support menu. The output should be all OK and a final result of PASS if it all worked correctly and I was able to patch the appliances.

 

NetBackup Appliance Hangs During Boot or Reimage Process

If you have a new NetBackup 5220 appliance and it hangs at boot, or hangs during a reimage process  it could be due to the NetBackup license having expired.

To resolve this, reboot the device and go into the BIOS and set the date to one of the following:

• Appliance version 2.0.1 – prior to March 10, 2012
• Appliance version 2.0.2 – prior to January 11, 2013 (01/09/2013 was verified as working)

If you can, make sure your /tmp directory is clean as when I ran into this, it also filled up the /tmp directory causing space issues on the / volume, leading to other problems.

Once the date has been changed, save and exit the BIOS and either continue with the reimage process or boot into the normal CLISH. Change the NetBackup license key (using the get_license_key command from the command line) to a valid license. You will then need to reboot, enter setup once more, and change the date back to the correct date and time.

Symantec technote for further details: http://www.symantec.com/business/support/index?page=content&id=TECH183939

 

NetBackup Appliance Patch Log

If you are patching your NetBackup appliance and are interested in more details about what is going on than what is shown on the screen, follow these steps:

1. Patches should be applied through the IPMI/console connection
2. Open a connection to your normal in-band  management IP address
3. From the CLISH, navigate to the OS root shell
4. Once the patch is applying, the progress log can be found in the /log directory with file format of patch_log_<version>_<dateandtime> – an example is patch_log_2.5_030913020924

Here is a sample of some of the log output from the file:

[2013-03-09 02:41:10] Dropping/recreating database...
Database server is NB_master01
Dropping the database: NBDB
Creating the NetBackup database.
Starting the NetBackup database.
Authenticating the NetBackup database.
VXDBMS_NB_DATA entry in bp.conf updated successfully.
Setting of database authentication for NBDB successful.
Change of dba password for NBDB successful.
Creating the NetBackup database files.
Creating the NetBackup database schema.

Also, if you had a problem with the install, you would want to look here first.

 

Trouble with NetBackup Appliance Callhome Error

With some 2.0.x versions of the Symantec NetBackup 5220 Appliances, a known issue exists with the callhome script when using it in conjunction with some web proxies. The symptom is that when you enable callhome proxy, you get a failure returned and then every 15 minutes you get notified via email that the callhome is not working.

The error ends up being a bunch of html code which basically says there was an error reported from your proxy server and it couldn’t reach the symappmon.com callhome website.

The fix is to update a couple of lines in the callhome script. The script is located in /opt/NBUappliance/scripts and is file SymAppMon.pm. Open that in your favorite editor and scroll almost to the end.
Look for this block of text:

}
$self->{ua}->proxy(['http','https'] => $p_str);
}


Replace that with these two lines:

}
$self->{ua}->proxy(['http'] => $p_str);
$ENV{HTTPS_PROXY} = $p_str;
}


Once those are replaced, save the file and then run callhome enable, and callhome proxy enable and you should have success with the callhome script through a proxy.

 

Do you do backup reporting?

If you’ve used NetBackup, or any other backup tool for that matter, for any length of time, you know how lacking the base product’s reporting has been until the past couple of years. There are several products that fill the void that Veritas/Symantec has, until recently, not addressed very well.

At first glance, you might not think backup reporting tools are worth the money. However, their ability to quickly present readable, well organized charts, graphs, and spreadsheets, even from the most complicated of data sets, starts making up for the cost. Add in the time it would take a normal admin to cobble together requests from management, if it would even be possible, and these products are looking even better.

Let’s face it, management likes pretty reports, but more to the point, management values pretty reports that contain facts.

With a tool of this nature, you can take a top down approach to resolving widespread issues that may have escaped the admins. Further, since the data is historical, you can prove that, in fact, client A has always been slow to backup. If you are a service provider or bill your customers, chances are you are not billing accurately for the backups you are performing. A tool of this nature, either with a built-in chargeback report or just exporting the raw backup usage, can recover these lost dollars. This in turn increases the ROI on the product, turning backups into a profit center. The possibilities are endless, even more so with the ability to create custom reports outside of what is already offered.

If you are at a site of any size, you should definitely consider a product that offers historical reporting and trending of your backup environment.

Here is a listing of the popular products (in alphabetical order):

Aptare Storage Console
Bocada
Solar Winds/Tek-Tools Profiler
Symantec OpsCenter/Backup Reporter (VBR)
WysDM/EMC Data Protection Advisor

Most of the products will support NetBackup, BackupExec, TSM, Networker, and Legato, among others. Many of these also have additional modules for storage and virtualization reporting that makes them even more attractive.

I invite you to answer a quick question on what reporting product you use (if any):

 

NetBackup Documentation Bundle

I figure most of you NetBackup admins are like me and don’t want to download the manuals one by one everytime a new release of NBU comes out. Here’s a link to a zip file of PDF documentation:

NetBackup 7.1 Documentation Zip Bundle

I can’t find any zips for other bundles, so if anyone has these, leave it in the comments or email me!

 

6.5.3.1 Security Update

In March, 2009, Symantec provided a security update to NetBackup. Reportedly, this update fixes a vulnerability with vnetd and also addresses issues with calendar based scheduling and DST. The entire bulletin describing the fixes and known issues can be found here.

 

Media In Use When It Really Isn’t

In NetBackup 6.x, the Resource Broker (nbrb) handles assignment of resources. A known bug exists, which was supposed to be fixed in 6.5.1, where media would not have this assignment released. Most times this is noticed during a restore when a particular tape is required, yet the restore details state that it is waiting on the media. To resolve this, do the following:

1. Verify NBU inventory and library inventory are synced
2. Verify NBU inventory shows tape as in library
3. Verify tape is not already mounted or stuck in a tape drive via robtest or some other method
4. If the above show the tape is in a normal, expected state, continue below; otherwise, resolve the issues above and retry the job
5. Note: running these commands will cancel any job you have running that is requesting this media; be sure to suspend/cancel them beforehand
6. Run /usr/openv/netbackup/bin/admincmd/nbrbutil -dump | grep <mediaID>
7. If nbrb still has an open assignment for this tape, you will see quite a few lines of information, but look for the line containing “MdsAllocation“
8. Next to this will be a number so that it looks like this: MdsAllocation allocationKey=4838
9. This is the assignment that needs to be released using this command: /usr/openv/netbackup/bin/admincmd/nbrbutil -releaseMDS 4838
10. Run the dump command again (step 6) to verify the assignment has been released from the tape
11. Resume/restart the job

 

6.5.4 Estimated Release Date

NetBackup 6.5.4 is estimated to be released early June.