OCI Teleport

Here's a fun lil project, configure Gravitational Teleport on Oracle Cloud Infrastructure Free Tier

Teleport is a "Unified Access Plane" for your infrastructure. In this example I'm going to show how I configured Web SSH Access to OCI via Teleport.

Setup OCI

• Sign up for a free account here: https://www.oracle.com/cloud/free/
• Verify email
• In creating an account you will need to give them a credit card. You shouldn't ever be charged anything as long as you don't use any non-free resources.
• Once you create an account you will get an email after the setup completes.

Create Virtual Cloud Network (VCN)

• OCI Cloud Console > Set up a network with a wizard
• Select root compartments from Comartments drop down on left side of screen.
• This should start the VCN Wizard
• You want "VCN with Internet Connectivity" > Start VCN Wizard
• VCN Name: vcn-demo
• Compartment: username(root)
• Defaults for the rest:
• VPN CIDR Block: 10.0.0.0/16
• Public Subnet CIDR Block: 10.0.0.0/24
• Private Subnet CIDR Block: 10.0.1.0/24
• Use Hostnames in this VCN: Checked
• Next
• Create
• View Virtual Cloud Network

Create Teleport Host

• Settings > Compute > Instances
• Create Instance
• Name: instance-teleport
• Compartment: username(root)
• Placement and Hardware > Edit
• Change Image to Oracle Linux 8
• Make sure Always Free AD & Shape selected
• Networking: Make sure vcn-demo is selected, and in public subnet
• Add SSH Keys
• - Generate & Save Private Key
• Click Create
• Wait a few minutes for Instance to be created

Verify Connecitivity

• Settings > Compute > Instances
• Make a note of the Public IP, use it in the following as $node_ip_addr
• Go to noip.com
• Sign up for free teletele.ddns.net hostname
• Setup hostname teletele.ddns.net to point to $node_ip_addr
• nslookup teletele.ddns.net
• chmod 600 ssh-key-2021-03-12.key
• ssh -i ssh-key-2021-03-12.key opc@teletele.ddns.net

Install Teleport

• sudo su -
• sudo yum-config-manager --add-repo https://rpm.releases.teleport.dev/teleport.repo
• sudo yum install teleport

Setup TLS

• Install Certbot
• Prereq install snapd, docs here https://snapcraft.io/docs/installing-snap-on-red-hat
• sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
• sudo dnf upgrade
• sudo yum install snapd
• sudo systemctl enable --now snapd.socket
• sudo ln -s /var/lib/snapd/snap /snap
• Log out and back in to update paths
• sudo snap install core
• sudo snap refresh core
• sudo snap install --classic certbot
• sudo ln -s /snap/bin/certbot /usr/bin/certbot
• # Fix firewall issue on  next command by opening port 80
• sudo firewall-cmd --zone=public --permanent --add-port=80/tcp
• sudo firewall-cmd --reload
• Update network firewall
• OCI Settings > Networking > VCNs > Root Compartment > vcn-demo
• Security Lists > Default Security List for vcn-demo
• Add Ingress Rules
• Source CIDR: 0.0.0.0/0
• Destination Port Range: 80
• sudo certbot certonly --standalone
• Enter email address
• Read TOS and Agree to share email address
• Enter domain name: teletele.ddns.net
• Delete Network firewall rule allowing traffic on port 80

Configure & Launch Teleport

• vi /etc/teleport.yaml which should be a new file and consist of the following:

• teleport:
      data_dir: /var/lib/teleport
  auth_service:
      enabled: true
      cluster_name: "teleport"
      listen_addr: 0.0.0.0:3025
      tokens:
      - proxy,node,app:f7adb7ccdf04037bcd2b52ec6010fd6f0caec94ba190b765
  ssh_service:
      enabled: true
      labels:
          env: staging
  app_service:
      enabled: true
      debug_app: true
  proxy_service:
      enabled: true
      listen_addr: 0.0.0.0:3023
      web_listen_addr: 0.0.0.0:3080
      tunnel_listen_addr: 0.0.0.0:3024
      public_addr: teletele.ddns.net:3080
      https_keypairs:
      - key_file: /etc/letsencrypt/live/teletele.ddns.net/privkey.pem
        cert_file: /etc/letsencrypt/live/teletele.ddns.net/fullchain.pem

• Start Teleport
• teleport start --config=/etc/teleport.yaml
• Open firewall to port 3080
• Host firewall
• sudo firewall-cmd --zone=public --permanent --add-port=3080/tcp
• sudo firewall-cmd --reload
• Network firewall
• OCI Settings > Networking > VCNs > Root Compartment > vcn-demo
• Security Lists > Default Security List for vcn-demo
• Security Lists > Default Security List for vcn-demo
• Add Ingress Rules
• Source CIDR: 0.0.0.0/0
• Destination Port Range: 3080
• Verify teleport is accessible
• https://teletele.ddns.net:3080/

Running teleport

• Cancel the session we previously started teleport and start via
• service teleport start
• Create a user
• adduser devuser
• /usr/local/bin/tctl users add devuser --logins=devuser,root --roles=admin
• Should see something like the following:
• User "devuser" has been created but requires a password. Share this URL with the user to complete user setup, link is valid for 1h:
  https://teletele.ddns.net:3080/web/invite/sometoken
  
  NOTE: Make sure teletele.ddns.net:3080 points at a Teleport proxy which users can access.
• Visit the tokenized URL to configure a password & MFA for devuser

This was a fun exercise to tinker with a new app. The next iteration of this install would be to include OCI VPN options so the app is never intentionally exposed at any point, and spend a little more time with certs to see if I can simplify the install & config process.

Finding product key for sql server 2008

Great help from this site in finding the product key for an existing sql server 2008 install.

Paste the following into powershell:

function Get-SQLserverKey {
    ## function to retrieve the license key of a SQL 2008 Server.
     param ($targets = ".")
    $hklm = 2147483650
    $regPath = "SOFTWARE\Microsoft\Microsoft SQL Server\100\Tools\Setup"
    $regValue1 = "DigitalProductId"
    $regValue2 = "PatchLevel"
    $regValue3 = "Edition"
    Foreach ($target in $targets) {
        $productKey = $null
        $win32os = $null
        $wmi = [WMIClass]"\\$target\root\default:stdRegProv"
        $data = $wmi.GetBinaryValue($hklm,$regPath,$regValue1)
        [string]$SQLver = $wmi.GetstringValue($hklm,$regPath,$regValue2).svalue
        [string]$SQLedition = $wmi.GetstringValue($hklm,$regPath,$regValue3).svalue
        $binArray = ($data.uValue)[52..66]
        $charsArray = "B","C","D","F","G","H","J","K","M","P","Q","R","T","V","W","X","Y","2","3","4","6","7","8","9"
        ## decrypt base24 encoded binary data
        For ($i = 24; $i -ge 0; $i--) {
            $k = 0
            For ($j = 14; $j -ge 0; $j--) {
                $k = $k * 256 -bxor $binArray[$j]
                $binArray[$j] = [math]::truncate($k / 24)
                $k = $k % 24
         }
            $productKey = $charsArray[$k] + $productKey
            If (($i % 5 -eq 0) -and ($i -ne 0)) {
                $productKey = "-" + $productKey
            }
        }
        $win32os = Get-WmiObject Win32_OperatingSystem -computer $target
        $obj = New-Object Object
        $obj | Add-Member Noteproperty Computer -value $target
        $obj | Add-Member Noteproperty OSCaption -value $win32os.Caption
        $obj | Add-Member Noteproperty OSArch -value $win32os.OSArchitecture
        $obj | Add-Member Noteproperty SQLver -value $SQLver
        $obj | Add-Member Noteproperty SQLedition -value $SQLedition
        $obj | Add-Member Noteproperty ProductKey -value $productkey
        $obj
    }
}

Then enter the following:

Get-SQLserverKey

Boom, should tell you the version of the OS, of SQL Server, and the SQL Server Product Key.

Create java keystore using wallet for Digicert wildcard cert

Had a heck of a time getting a DigiCert wildcard cert converted to a Java Keystore for use with Oracle Fusion Middleware 10.3.6.

Here's the procedure I ended up using to get this working:

1. Create wallet
1. mkdir $MW_HOME/owm
2. Start OWM
3. Create new wallet, store it in $MW_HOME/owm
4. Enter password for wallet (make note of this for later)
5. Create certificate request
1. CN: *.domain.com
2. OU: YourDept
3. Org: YourCo
4. Locality: City
5. State: MI
6. Key Size 2048 << important, our CSR will reject without keysize at least 2048
6. Click on certificate, select operation >> export certifciate request
1. Enter request.csr in name
7. Send request.csr to your unix admins
8. Save wallet, back it up in case you make a mistake with the following
9. Unix admins send back the following
1. star_domain_com
2. DigiCERTCA.crt
10. Get root.crt from https://ev-root.digicert.com/info/index.html
1. Copy every thing starting with -----BEGIN CERTIFICATE----- to -----END CERTIFICATE----- in a text file root.crt
11. In OWM, import certificates
1. Operations, import trusted certificate, select root.crt
2. Operations, import trusted certificate, select DigiCertCA.crt
3. Operations, import user certificate, select star_domain_com
12. Do not check auto login
13. Wallet > save wallet
2. Convert wallet to keystore
1. export ORACLE_HOME=$MW_HOME/oracle_common
2. export PATH=$ORACLE_HOME/bin:$PATH
3. export JAVA_HOME=$MW_HOME/Oracle_WT1/jdk
4. orapki wallet pkcs12_to_jks -wallet $MW_HOME/owm -pwd notapassword -jksKeyStoreLoc $MW_HOME/owm/keystore.jks -jksKeyStorepwd notapassword
3. Verify keystore
1. keytool -list -v -keystore keystore.jks -storepass password
1. make note of alias for your specific cert
4. Implement in Fusion Middleware
1. Login to FMW console
1. base_domain > environment, servers, click on server name
2. Keystores tab 
1. Custom Identity and Java Standard Trust
2. Custom Identity Keystore: $MW_HOME/owm/keystore.jks
3. Custom identity keystore type: jks
4. Custom identity keystore passphrase: notapassword
5. Save
3. SSL Tab
1. Identity and Trust Locations: change to keystores
2. Private Key Alias, alias from step 3.1.1 above
3. Private Key Passphrase: notapassword
4. Confirm Private Key Passphrase: notapassword
2. Go to general tab
1. Enable SSL listen port enabled
2. Set listen port to port 7003
3. Telnet from another machine to servername:7003 to verify connectivity
5. You should now be able to connect to servername:7003 with a browser and not get an ugly cert error message.

Hopefully this will help whoever stumbles for this in the future.

Useful notes:

1. Convert Wallet To Keystore for WebLogic. (Doc ID 1363979.1)
2. http://docs.oracle.com/cd/E16340_01/core.1111/e10105/walletmgr.htm#CJGGFCGC

Shoring up SQL Skills

Goal for 2014:  Revisit weak points in my knowledge of SQL.  Going to try http://www.sql-ex.ru/ to help improve those skills.  I'll report back on how it goes.

Unable to register database with DBCA

Attempting to register a database with DBCA and got the following error message:

TNS-04409: Directory service error caused by: oracle.net.config.DirectoryServiceException: TNS-04405: General error caused by: oracle.net.ldap.NNFLException

Thought thought perhaps the problem was something like this issue with password policys, but it turns out it was simply that I needed to change how anonymous binds are handled. Per Doc ID 947285.1, check the value of orclanonymousbindsflag. I had it set to "2", which didn't work. Setting it to "1" allowed me to register the database, and then I set it back to 2 as it is a more secure setting.

RMAN cannot restore from backupset copied to disk from tape

Attempting to work around an issue with our backup system, I copied the backup pieces from a known-good backup from tape to disk. Then I attempted to catalog those pieces for a recovery attempt. Cataloging those pieces failed.

RMAN> catalog backuppiece '/u01/app/oracle/admin/scripts/restore/filetest/DB_3rn2tp0j_1_1';

using target database control file instead of recovery catalog
ORA-19870: error while restoring backup piece /u01/app/oracle/admin/scripts/restore/filetest/DB_3rn2tp0j_1_1
ORA-19587: error occurred reading 0 bytes at block number 1
ORA-27067: size of I/O buffer is invalid
Additional information: 2

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of catalog command at 02/17/2012 09:37:20
RMAN-06209: List of failed objects
RMAN-06211: ==========================
RMAN-06212:   Object Type   Filename/Handle
RMAN-06213: --------------- ---------------------------------------------------
RMAN-06214: Backup Piece    /u01/app/oracle/admin/scripts/restore/filetest/DB_3rn2tp0j_1_1

Eventually found Oracle Support Doc ID 1111603.1 on the subject. Essentially, no, you can't catalog a backup piece that's been restored from tape to disk because the header information will be different than the file at rest on tape.

A bad use of find in a nightly script

I was going through cleaning up files on one of the database servers. I happened to find this script.

> cat /home/oracle/scripts/rm_old_alogstrc.shl
cd /u03/oradata/SID/archivelog
find . -mtime +2 -exec rm {} \; 
cd /u01/app/oracle/admin/SID/adump
find . -mtime +7 -exec rm {} \;
cd /u01/app/oracle/admin/SID/bdump
find . -mtime +7 -exec rm {} \;
cd /u01/app/oracle/admin/SID/cdump
find . -mtime +7 -exec rm {} \;
cd /u01/app/oracle/admin/SID/udump
find . -mtime +7 -exec rm {} \;
cd /home/oracle/datapump/SID
find . -mtime +7 -exec rm {} \;
cd /home/oracle/exports/SID
find . -mtime +7 -exec rm {} \;

For the benefit of those not cringing as you read that. Two major things I see wrong with it. The first is that someone had entrusted the OS to clean up old archivelog files older than 2 days. While it's not so common with all the safeguards we put in place for backups, but I can still imagine scenarios where archivelogs don't get backed up off the server in 2 days. This script runs and poof there goes recoverability.

And the second which makes me cringe is the two step file removal. Specifically cd to some directory and then rm all files in that directory. Just imagine what would happen if that script didn't change directory. Next you know whatever the working directory was is effectively wiped out. It was likely the home directory.

The much better way is to combine it into one command, that way there's no accidental deletions.

find /home/oracle/exports/SID -mtime +7 -exec rm {} \;

In a way I wish I didn't see this. Now I wonder what other gems are out there on this server that someone left me.

Manually remove oracle server software on Microsoft Windows Platforms

What happens if Oracle deinstall.bat doesn't cleanly remove all the windows components? Try using doc id 1069034.1: 11.2: How to Manually Remove Oracle Server Software on Microsoft Windows Platforms [ID 1069034.1]

Nice try

Tired of looking this up... so I'm posting it. I don't use the unix nice command very often, but when I do, the man page seems a little confusing. Here's the format I typically use the command:

nice -n +19

That runs the command at the lowest possible priority, helpful if you're going to do some big file op and don't want to cause anyone else any grief.

What service has oci.dll open?

Attempting to use deployment procedure's to install patch bundles on windows hosts, sometimes you run into issues with services keeping dll's (especially oci.dll) open.

Here's a trick to determining what service is keeping a specific dll open.

From a command prompt:

tasklist /m > tasks.txt 

notepad tasks.txt 

The look for the specified dll in tasks.txt and find the process. From there it should be somewhat obvious which service to shut down in order to get the patch to continue.

ORA-01406: fetched column value was truncated

I just got a dump file of our data from a vendor that is hosting an application for us and I was making it available to my developers and since we will be getting files on a somewhat frequent basis, initially I created an external table to read the data.

I had to get over the first hurdle of version incompatibility of the datapump file, which wasn't too bad, but after I cleared up those issues (lots of references on google, this chart helped the most) I received the following error in Toad:

ORA-01406: fetched column value was truncated

with further details

fetched column value was truncated

Cause: In a host language program, a FETCH operation was forced to truncate a
character string. The program buffer area for this column was not large enough to
contain the entire string. The cursor return code from the fetch was +3. 

Action: Increase the column buffer area to hold the largest column value or perform
other appropriate processing.

Google wasn't as helpful with this (hence the post here), but essentially this is a client side setting that needs to be adjusted. In version 10.6.1.3 of Toad there is one setting that needs to be changed. Search for "OCI Array Buffer size" or find it under "Oracle - General". You need to change it to be large enough to fetch the largest column in the query. In my case I had a column of size 4000, so setting this to 4000 returned results.

Fun with Logon Triggers

Sometimes a process happens too fast to be traced effectively, but it's happening slowly enough to make your users angry.

Here's an example of a logon trigger setup to make a trace happen, we're using it to diagnose an issue with slow response from an ODBC connection, using user account ADMIN_JOSH:

DROP TRIGGER SYSTEM.JOSH_LOGINTRIGGER;
 
CREATE OR REPLACE TRIGGER SYSTEM.Josh_LoginTrigger AFTER LOGON ON ADMIN_JOSH.SCHEMA
BEGIN
   execute immediate 'ALTER SESSION SET tracefile_identifier = MYTRACEFILE';
   execute immediate 'alter session set sql_trace=true';
END;
/

Note if you get the following error, the owner of the trigger needs the alter session priv (in this case system).

ORA-00604: error occurred at recursive SQL level 1
ORA-01031: insufficient privileges
ORA-06512: at line 2

It's the env, stupid

While performing the following...

$ORACLE_HOME/forms/lib32/make -f ins_forms.mk install

Got the following error:

ld: fatal: library -ljvm: not found
ld: fatal: File processing errors. No output written to frmbld
make: *** [frmbld] Error 1

Apparently no one else has ever reported this problem before...

Turns out the user's env didn't include LD_LIBRARY_PATH. Fixed that, and problem was solved.

It pays to read the documentation

Had an issue where I couldn't get forms builder to work.

Here's the error code:

# frmbld
FRM-91111: Internal Error: window system startup failure.
FRM-10039: Unable to start up the Form Builder.

It should work right out of the box, looks like the build config had changed much, here's the env file used:

ORACLE_HOME=/u01/app/oracle/product/ias10gds
export ORACLE_HOME
LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/jdk/jre/lib/sparc:$ORACLE_HOME/jdk/jre/lib/sparc/native_threads
export LD_LIBRARY_PATH
PATH=$ORACLE_HOME/bin:$ORACLE_HOME/opmn/bin:/usr/local/bin:/usr/ccs/bin:/bin:/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/usr/ucb:/usr/ccs/bin:/usr/sbin:/usr/X/bin:$ORACLE_HOME/OPatch
export PATH
FORMS_PATH=/u01/app/sct/baninbr2/QUAL/forms/fmb
export FORMS_PATH

Eventually found this documentation post about how to solve the problem. Basically since the last time we used forms builder on this server, the other Oracle homes required a change to NLS_LANG with a UTF8 characterset.

Simply added the following to the above forms builder env file fixed the issue with forms builder:

export NLS_LANG=

Problem solved.

Before you set global_names=true

For all the years that I've been working with Oracle I have not had to deal with the oracle streams capture/apply processes. Other DBA's on my team have had to work with it, but I didn't get the hands on experience. That is until past weekend when I had installed a streams backed process for our identity management product for creation of downstream accounts.

Part of the installation process was to set global_names=true, which as I have now found out is the recommendation from oracle. From what little I know about it, it is for additional security as well as preventing bad things from happening during clones (or so I've read).

It turns out that one of the results from enabling global_names is that you end up getting "ORA-02085: database link FOO.DB.COM connects to BAR.DB.COM" when the database link name does not match the target.

I tried "fooling" it with everything I could think of that didn't involve rewriting any of the code that calls the database link, but nothing in fact worked. The name itself has to be changed to match the target database name (which is clearly stated in oracle documentation...I hoped it was a lie).

This was new to me.

I currently have set global_names=false until I can figure out how much code has to be reworked reflecting all the database link name changes.

So the one thing I should have checked into before setting global_names to true was which database links were going to fail after the switch. And had I done that before, I would have realized the answer would be... all of them. At least now I can push for a new standard for everyone to follow when they create database links.

Fun with ancient clients

User was reporting an error with ODBC after a recent upgrade to 11g:

Unable to connect
SQLState=28000
[Oracle][ODBC][Ora]ORA-01017: invalid username/password; login denied

The user reported, and the tech verified that on this machine they could login with sqlplus but not ODBC on this PC. Multiple accounts were tried.

Interesting test case illuminated the issue: Created user with ALL UPPERCASE PASSWORD. Using this, could connect through ODBC.

The ancient 9i ODBC connection will not send a case sensitive password.

Made the following change, problem solved, user is happy though we probably could use this as an opportunity to force upgrading ODBC clients:

alter system set sec_case_sensitive_logon=FALSE scope=both

Empty variables can be troublesome

Upgrading a 10.2.0.4 ERP test database to 11.2.0.2 today, and shortly after starting the upgrade via DBUA, got the following error:

ORA-16024: parameter LOG_ARCHIVE_DEST_1 cannot be parsed

I had chosen to disable archiving as one of the options for DBUA, but was really surprised to see this error. After a little bit of research I found Metalink Doc 96278.1.

Turns out this error was happening because we had log_archive_dest along with log_archive_dest_1 set -- the kicker was it was set to be blank. Here's the relevant info from the pfile:

*.log_archive_dest=''
*.log_archive_dest_1='LOCATION=/u01/app/oracle/admin//arch'

So the trick was to completely remove the entire *.log_archive_dest='' line from the pfile and DBUA was able to proceed with the upgrade without the ORA-16024 error.

Death by a thousand trace files

Found a large number of trace files generated in the /udump admin directory.

Each file looked like the following:

$ cat orcl_ora_12191.trc 
/u01/app/oracle/admin/ORCL/udump/orcl_ora_12191.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORACLE_HOME = /u01/app/oracle/product/10.2.0/db_1
System name:    SunOS
Node name:      hostname
Release:        5.10
Version:        Generic_144488-14
Machine:        sun4u
Instance name: ORCL
Redo thread mounted by this instance: 1
Oracle process number: 389
Unix process pid: 12191, image: oracle@hostname

opiino: Attach failed! error=-1 ifvp=0

Not much help on support.oracle.com or google in diagnosing this issue.

Oracle support was helpful in doing some tracing which proved that these were failures to connect rather than existing connections that were lost.

With that information in hand, we correlated the timestamps of the trace files to timestamps of listener connections. This narrowed our issue down to a specific IP address, and by that a specific application.

Our lead application administrator was able to determine the root cause of the failed connections, an outdated library used in development of the application:

From what I can tell CoreLab Oracle is now dotConnect for Oracle from Devart. They resolved this issue with version 5.70.140.

Whew.

UDM for Banner Jobsub and other subsystems

Over the weekend we had a SungardHE Banner subsystem offline. This particular subsystem uses Oracle Pipes to manage user job submission. The downtime could have been prevented if this subsystem had been represented as a target within Grid Control.

So, to prevent future outages, we're going to make this process a User Defined Metric, and have grid control fire off a notification if the service is offline. At the end of every maintenance window, if Grid Control says all systems are online, then my team can have more confidence that all systems are actually online.

Found a great article by Sagar Patil titled Oracle Grid Tracking OS Process Using a Custom User Defined Matrix udmshell script, which is the basis for what follows.

I created the following script that determines if ARG1 has ARG2 processes running and returns either Online or Offline:

#!/usr/local/bin/bash

if [ $# -ne 2 ]
then
  echo "Usage: `basename $0` {ORACLE_SID} {Expected Count}"
  exit 1
fi
toCheck=$1
testEm=`/bin/ps -ef | /usr/local/bin/grep -v grep | /usr/local/bin/grep -v $0 |
/usr/local/bin/grep $toCheck | /usr/local/bin/wc -l`

if [ $testEm -eq $2 ]
then
  echo "em_result=Online"
else
  echo "em_result=Offline"
fi

Next I create the UDM.

From the target Host screen, click on "User-Defined Metrics", then click create.

Here's the parameters I used:

Metric Name: GURJOBS DEVL

Metric Type: String

Command Line: /banner/u01/app/oracle/product/agent11g/sysman/emd/custom/checkSubsystems.sh gurjobDEVL 1

User Name: Oracle

Comparison Operator: Match

Critical: Offline

The rest I left as default. After hitting OK, it took about 15 minutes to show that gurjobDEVL was in fact online. Shutting down gurjobs in DEVL returned an error in the UDM screen but did not send a notification.

To enable notifications, I had to do the following:

Grid Control User Preferences -> Rules -> Select "Host Availability and Critical States", then click Edit. In the Metrics tab, click Add, enter "User" in the search and click Go.

To enable the metric, check the box next to User Defined Metric, and then choose "Critical" and "Clear" severity states, then click continue, and then OK.

Whoever is subscribed to that notification rule will be sent a notification per their preferences.

In this simple case, knowing that Banner Jobsub is offline is great. Another example would be with CAPP Pipes, the next step is to build in automatic restart scripts. That will have to wait for another blog post.

VI trick of the year

Sometimes the faster you type the slower you are.

Yesterday I was working on a configuration file, commenting and uncommenting about 10 different lines. Being a somewhat fast typist, the keyboard gymnastics required to add "--" to the beginning of 10 lines of code wasn't daunting enough to make me consider better ways to get the job done.

Those keyboard gymnastics consisted of the following 6 keypresses per line commented out:
i to insert
-- to comment the line out
ESC to go back to command mode
down arrow
then zero to go to the beginning of the line

I suppose if this had been 25 or 50 or whatever that unknowable mental threshold of "too many lines", I would have used awk or sed to get the job done.

My new coworker saw what I was doing, and asked why I didn't use the repeat command. Que? I asked?

Apparently hitting a period while in command mode will repeat the last insert/append that took place in edit mode. So simply made the change once in edit mode, then hit "." to repeat on the next line.

Ok, maybe not the "trick of the year". Maybe trick of the day is more appropriate, but I'm sure this is one of those tips that I'll be using on a daily basis -- a somewhat year changing event.