You are currently browsing the tag archive for the ‘bash’ tag.

Here is the main script I use to parse the gpfs.tmp files for which I/O nodes have which dm’s associated with which file system and then using that data create the multitude of graphs.

I make graphs for 12 hour (one Navy watch), 24 hours, 48 hours, a week, and a month. There are two main graphs I’m creating right now. The Average Wait graph and the % utilization graph. Also, if you delve into the code you will see I search for data in the lun name so that I don’t add the metadata luns into the charts. It just keeps it cleaner.

FYI. I’ve modified the scripts to remove any reference to any system where I work. So, I don’t “think” I’ve introduced any errors into the scripts, but it’s definitely possible that I have.

# written by Richard Hickey
# 20 March 2014
# This script will read the lun layout files /gpfs/scratch/*.tmp
# and then create the utilization and average wait graphs in /var/www/html/iostats

import re
import sys
import rrdtool

# Set up an array with all of the file systems to parse through
# Set up a dictionary called filesystem for human readable names
myGPFSArray = [“gpfs_alpha”, “gpfs_beta”, “gpfs_ops”, “gpfs_scratch”]
filesystem = {‘gpfs_alpha’:’Alpha’, ‘gpfs_beta’:’Beta’, ‘gpfs_ops’:’Ops’, ‘gpfs_scratch’:’Scratch’}

# This function opens the gpfs lun mapping configuration file
# and fills in a data array with LUN, host, dm, and state (state isn’t used)
def getData(GPFSFileSystem):
        myFile=open(‘/gpfs/scratch/’ + GPFSFileSystem + ‘.tmp’, ‘r’) # open the config file
        myConfigArray = [] # initialize the array
        for line in myFile: # walk through the file line by line
            line = line.strip() # remove the newline character
            myline = line.split( ) # break the line into pieces using space
        myFile.close() # close the config file
        return(myConfigArray) #return an array consisting of the data from the config file

    except IOError:
        print ‘Could not open file ‘, myFile

# Create the Graph routine
def GraphCreate(lunData, areaData, graphtype):
    title = [’12 Hours’,’One Day’,’2 Days’,’One Week’,’One Month’]
    subpath = [’12’,’24’,’48’,’week’,’month’]
    path = ‘/var/www/html/iostats/’
    start = [‘-12h’, ‘-24h’,’-48h’,’-1w’,’-1m’]
    horizontalRule = ‘HRULE:90#000000:’

    # set some perameters based on the graph type
    if graphtype == ‘await’:
        verticalLabel = ‘Milliseconds’
        subtitle = ‘ Average Wait ‘
        filename = GPFSFileSystem + ‘_await.png’
        upperLimit = ’80’
        lowerLimit = ‘0’
    if graphtype == ‘util’:
        verticalLabel = ‘%’
        subtitle = ‘  % Utilization ‘
        filename = GPFSFileSystem + ‘_data.png’
        upperLimit = ‘100’
        lowerLimit = ‘0’
    # Create the Graph
    for count in range(5): # walk through the five chart types
        fullpath = path + ‘/’ + subpath[count] + ‘/’ + filename
        fulltitle = ‘/gpfs/’ + filesystem[GPFSFileSystem] + subtitle + title[count]
            ‘–title’, fulltitle,
            ‘–imgformat’, ‘PNG’,
            ‘–width’, ‘800’,
            ‘–height’, ‘400’,
            ‘–vertical-label’, verticalLabel,
            ‘–start’, start[count],
            ‘–upper-limit’, upperLimit,
            ‘–lower-limit’, lowerLimit,
            areaData )   

# Main routine
for GPFSFileSystem in myGPFSArray:
    myConfigArray = getData(GPFSFileSystem)
    print ‘Doing ‘ + GPFSFileSystem

    # Pull the individual components out of each line of the config file
    utilData = []
    awaitData = []
    areaData = []
    for line in myConfigArray: # the line is an array with LUN HOST DM STATE
        lunType =’data’,line[0])
        if lunType:
            tmplun = line[0] 
            lun = tmplun.split(‘_’)
            node = line[1]
            tmpdm = line[2]
            dm = tmpdm.split(‘/’)
            x = ‘DEF:’ + lun[0]+ ‘_’ + lun[1] + ‘=/gpfs/scratch/’ + node + ‘/’ + dm[2] + ‘.rrd:util:AVERAGE’
            utilData.append(x) # this creates the utilData array with the DEF lines of the rrdgraph
            y = ‘DEF:’ + lun[0]+ ‘_’ + lun[1] + ‘=/gpfs/scratch/’ + node + ‘/’ + dm[2] + ‘.rrd:await:AVERAGE’
            awaitData.append(y) # this creates the awaitData array with the DEF lines of the rrdgraph

            # The following populates the AREA portion of the rrdgraph array named areaData
            # The primary reason to break these apart is just to set the colors differently
            if node == ‘frodo-io3’:
                z = ‘AREA:’ + lun[0]+ ‘_’ + lun[1] + ‘#421c52:’ + lun[0]+ ‘_’ + lun[1]   
            if node == ‘frodo-io4’:
                z = ‘AREA:’ + lun[0]+ ‘_’ + lun[1] + ‘#005500:’ + lun[0]+ ‘_’ + lun[1]   
            if node == ‘frodo-io5’:
                z = ‘AREA:’ + lun[0]+ ‘_’ + lun[1] + ‘#21b6a8:’ + lun[0]+ ‘_’ + lun[1]   
            if node == ‘frodo-io6’:
                z = ‘AREA:’ + lun[0]+ ‘_’ + lun[1] + ‘#3300ff:’ + lun[0]+ ‘_’ + lun[1]   
        # Call the function that creates the graphs
    GraphCreate(utilData, areaData, ‘util’) # call the graph creating function
    GraphCreate(awaitData, areaData, ‘await’) # call the graph creating function


Now it’s time to get to the meat of things. Here is a bash script that will create some tmp files containing which dm’s on which I/O nodes go with which file systems.


/usr/lpp/mmfs/bin/mmlsconfig|grep /dev/|awk -F\/ ‘{print $3}’|while read fs
    echo “Creating the tmp file for ${fs}”
    /usr/lpp/mmfs/bin/mmlsdisk ${fs} -M |grep frodo > ${fs}.tmp

So a few notes to make this easier to understand. The first major line is

/usr/lpp/mmfs/bin/mmlsconfig|grep /dev/|awk -F\/ ‘{print $3}’|while read fs

mmlsconfig gives way more data than just a list of file systems. I just want the file system names to input into a different command. I could make a static list, but then if something changed it would take manual intervention to get it correct again. Better to do a few extra steps right now and automate it. So, mmlsconfig gives to much informations, so I grep for /dev which gives me just the file systems (/dev/gpfs_scratch), I then awk -F\/ to split the line up using the / (the \ is so that awk doesn’t think the / is a special character) as my splitter. I then grab the third item, which is just the file system name (gpfs_scratch).

Now that I have just the file system name I push that into the mmlsdisk command. The -M option will display the underlying disk name on the I/O server node. I then output that information into a temp file. IE gpfs_scratch.tmp

/usr/lpp/mmfs/bin/mmlsdisk ${fs} -M |grep frodo > ${fs}.tmp

Easy peasy. Now I have my configuration files containing which dm on which I/O node goes with which gpfs file system. It’s now time to write a script to pull all this information together and make a nice pretty graph out of it.

Blog Stats

  • 60,475 hits
August 2019
« Jun