Blog

It’s time for the fourth post in my series about performing regular server reviews. In the first post, we took a look at setting up your local environment. The second post reviewed the workflow and quirks of RDP and VM environments, as well as getting SSMS set up on your machine. Part three led you through the first steps of creating a bio for your clients so that you and your backup can have all the information you need to get connected and working at your fingertips.

Today let’s take a look at the importance of building a server inventory and why it’s a critically important addition to your client bios.

NOTE: I’m using the word “client” throughout this post because my experience as a DBA is as an outside consultant. For full-time employees, different departments, teams, or groups within your place of business are essentially the same as clients. You can mentally replace the word “client” for one that is relevant to your circumstances and needs.

Build Your Registered Servers List

For starters, there’s sort of a chicken-and-egg situation when dealing with a new client. For the purposes of this post, we’re going to imagine you’re working with an amazing client that has given you a complete list of all the servers you’re expected to maintain and has already ensured you have access and permissions to all of those boxes.

Assuming you have all of this, now you’re ready to set up your Registered Servers list in SSMS. I’m not going to teach you how to set up your list in this post. To learn more I recommend reading “Registering SQL Server instances for easier management” from mssqltips.com. You can also learn more about Registered Servers at Microsoft Docs.

One tip I do have is to group your servers into subfolders according to their status or usage. For instance, we often separate the production servers from non-production servers in our lists. Many clients are not concerned with regular checks of development servers, but they do want us to have access in the event of some sort of problem they need help resolving.

For other clients we have not only created subfolders for Prod/Non-prod, but also nested subfolders for each set of nodes in an availability group.

What’s the point of a Registered Servers list? First, it gives you an easy-access listing of all the servers you’re responsible for maintaining. Second, the main advantage is that it gives you the ability to run your scripts against multiple instances all at once rather than one-by-one.

No matter how you set it up, be sure to craft your Registered Servers list in a way that meets your client’s needs and allows you to be a more efficient DBA.

Collect Server Info

Discovering the information we need to start building our server inventory is quite simple. Connect to your client’s servers and start up SSMS. From there, click the appropriate subfolder in the Registered Servers list and open a new query.

In the new query window, run the following simple script:

SELECT @@Version AS Version

This will return results similar to what you see below:

Some DBAs like to also use @@ServerName in their script, but as you can see it’s unnecessary since SSMS will return the server names along with the version information.

You can see that the script results provide the SQL Server version running on each server—along with the build number. The screenshot above is from one of my test boxes so the results are quite sparse. We’ll imagine we’ve got more to work on going forward for the purposes of this post.

From here, you can start building your Server Inventory List. My advice is to create a spreadsheet of some sort and file it with your client bio docs, or embed it in the doc itself if you can. Info that you’ll probably want to add to your spreadsheet includes the server name, the environment (Production, Development, Reporting, Archive, etc.), the current SQL Server version—including the build number, and the purpose and/or details relevant to the box.

Server Inventory List

Server Name	Environment	Version	Purpose
PRODSQL01\2016SP	Prod	2016 (RTM) - 13.0.1601.5 (X64)
PRODSQL02\REPORTSERVER	Prod	2016 (RTM) - 13.0.1601.5 (X64)	Reporting
PRODSQL03\SP2012PROD	Prod	2012 (SP3) (KB3072779) - 11.0.6020.0 (X64)	Website
PRODSQL04\AG1	Prod	2016 (SP1) (KB3182545) - 13.0.4001.0 (X64)	AG Primary
PRODSQL05\AG2	Prod	2016 (SP1) (KB3182545) - 13.0.4001.0 (X64)	AG Secondary
PRODSQL06	Prod	2016 (SP2-GDR) - 13.0.5102.14	Transactional replication Publisher. Log shipping primary.
PRODSQL07	Prod	2016 (SP2-GDR) - 13.0.5102.14	Transactional replication Subscriber from PRODSQL06.
PRODSQL08	DR	2016 (SP2-GDR) - 13.0.5102.14	Log shipping secondary.
PRODSQL09\Monitor	Prod	2012 (SP3-GDR) (KB4019092) - 11.0.6251.0 (X64)	Monitoring/DBA jumpbox
DEVSQL10\Sandbox	Dev	2012 (SP3-GDR) (KB4019092) - 11.0.6251.0 (X64)
PRODSQL11	QA	2016 (SP1) (KB3182545) - 13.0.4001.0 (X64)

You’ll want to keep this list up-to-date so you know what’s going on in your client’s environment. I like to update my list quarterly at minimum. Of critical importance is keeping the version build number fresh. I’ll explain some reasons why further below.

Putting Your Server Inventory List to Work

Once you have a complete server inventory list, take some time to study the list and see if any critical updates need to be applied to your servers. Some clients like to keep their installations up-to-date with the latest Service Packs and Cumulative Updates, while others take an “it’s working fine, so don’t touch it” approach.

At Dallas DBAs, we like to take a measured approach to updates. We don’t recommend applying the newest updates immediately unless there’s a security patch or hotfix that addresses a specific critical issue in SQL Server. We typically recommend against using an RTM version, and recommend only upgrading to a newer version of SQL Server after a couple of CUs are out and stable. Otherwise, we like to wait and see if a new update has bugs in it before recommending installation on our client’s servers.

Case in point, just look at how buggy Cumulative Updates for SQL Server 2019 have been: Microsoft pushed out the 2019 CU7 only to soon remove it and push CU8 as a fix what CU7 had broken.

Keeping up with patching can be critical since there are some absolutely deadly builds of SQL Server out there. Some builds are known to cause corruption during index rebuilds and should be patched immediately. (Example 1; Example 2). And wildly enough, sometimes you can run into a corruption bug when upgrading from one version of SQL Server to another.

Unsupported builds of SQL Server should also be updated. If there’s a problem with your servers bad enough to get Microsoft support involved, the first thing they’ll tell you to do is apply the requisite updates if necessary.

Ensuring your servers are up-to-date can also introduce additional features for free. Stop missing out on the best tools available to yourself and your clients.

If you’re not sure how to find out if your servers could benefit from an update, I highly recommend you grab Brent Ozar’s sp_Blitz script from his First Responder Kit.

Here are two great sources to find the latest information about available SQL Server updates:

• SQL Server Updates
• SQL Server Builds

If you want to take your Server Inventory List to the next level, check out Brent Ozar’s recommendations for your list.

In the next post we’ll take a look at adding a Maintenance Job list and Backup Frequency List to your client bios.

If you’d like to know more about SQL Server Maintenance, check out Kevin’s Getting Started with SQL Server Maintenance course at Pluralsight.

Follow me on Twitter at @SQLandMTB, and if you’re into mountain bikes come over and check out my site NTX Trails.

Follow @Dallas_DBAs

Welcome back to my ongoing series about how we perform regular server reviews here at Dallas DBAs, and the tips and tricks I’ve picked up in my first 1.5 years of work as a DBA. If you’re new to the series, go back and check out the Setting Up Part 1 and Part 2 posts where I step through the most basic beginnings from setting up your work environment to getting your script library in order.

Today, we’re moving forward with getting familiar with your client, their servers, and their specific needs. Keep in mind that this is all written from an outside consultant’s point of view. If you work in-house as a DBA for a company or corporation, some of what you read here will be different than your experience. I challenge you to take what you read and see if you can apply it to your work environment and become a more efficient DBA.

Create a Client Bio

I’ll admit from the beginning of this post that much of what you’ll read below may not apply to you if you work on-prem. For those of us who jump from client to client throughout the day, getting a baseline of information at the beginning can save you from many headaches in the future.

Here at Dallas DBAs, we have set up client bio documents in Microsoft Teams for each client we serve. This is a very convenient way for us to not only access all of a client’s vital information, but we can also share and edit the information for our entire team quickly and easily.

We feel like the best approach to writing a client bio is to think about what another DBA would need to know in order to do your job in case you’re unavailable. How can you continue to provide your clients the best service possible in the event you have a car accident or get sick and are unable to work for a while? How can you best help a backup DBA cover for you effectively?

With that in mind, here’s a rundown of what we collect for our client bios.

Contact Information

Be sure to collect the various names, email addresses, and phone numbers of relevant team members from your client. To whom do you send daily/weekly server reports? Who do you contact if you discover an issue on a server? How do you contact them? Who will contact you in an emergency?

Connectivity Information

How do you connect to the client’s servers? What VPN client is used and where can it be downloaded? Are there any special notes or connectivity quirks another DBA needs to know about? Do they connect to a jumpbox or directly to production servers?

Put together a list of VPN URLs and RDP IP addresses as necessary.

What about credentials? Can the backup DBA share your credentials, or do they need their own? What is your client’s policy on that score? Does your backup have all the permissions they need to get the job done? Make sure your backup has all the credentials and clearances they need in your absence.

This is especially important if your client uses some sort of two-factor authentication. If you haven’t gotten your backup set up ahead of time, they’re going to need to jump through extra hoops to get the work done.

Environment Features

Make note of things like whether the servers are on-prem or VMs or a mix of the two. Is the client using any of SQL Server’s “special features” such as replication, log shipping, or availability groups?

What about third party monitoring tools such as Manage Engine, DPA, SQL Sentry, or Redgate SQL Monitor, etc? Does your backup even need to bother with checking these? If yes, then what are you looking for and what are the reporting procedures? (For instance, one of our clients likes to see screengrabs of some of the graphs on a weekly basis.) Also if yes, make sure your backup has the credentials they need to access these additional tools.

Task List

What, specifically, needs to be checked on the servers and how often? Write up a list of tasks that should be performed and their frequency. Depending on your SLA, some clients may get daily checks while others will only receive weekly checks. How often are deeper health checks performed and when?

Detailed Procedures

Once you’ve got the task list put together, be sure to create a detailed explanation of procedures for your client’s server check. It may seem sort of pedantic to type up a list of what scripts need to be run and how the results should be reported, but this is actually the most vital portion of your job when it comes to performing regular server checks. This is how are able to discover and remedy potential problems before they become serious issues.

And remember, you’re writing this as if your backup knows nothing about the client’s environment. Which scripts need to be executed? Where are the scripts located? Against which servers? How frequently should the scripts be run?

This is where you will likely want to create a repository of scripts and other files you use for your client. Create an archive of your custom scripts that are particular to your client. If you’re using password management software, add the master file to this archive. Are you using Remote Desktop Connection Manager for this client? If yes, have you provided an RDG file? What about a REGSRVR file for the Registered Servers list in SSMS?

Think about gathering everything you need to get stuff done, not only for your backup, but yourself. Imagine some strange scenario where you couldn’t access your work computer during a client emergency? Could you borrow a laptop from someone and get your work done?

But wait! There’s More

There are a few more things you should consider including in your client bio such as a server inventory, maintenance job list, and backup frequency chart. I’ll cover these in the next few posts.

You’ll find that gathering all this information is extremely helpful not only for your potential backup, but for yourself as well. I can’t tell you how many times something weird has happened on my end that prompts me open up the client bio and find the information I need. Yes, write the bio as if your backup needs to start from a position of complete ignorance of the client, but also write it in such a way to help yourself remember what to do for your client and how to get it accomplished.

I’ve found that the greatest challenge is keeping client bios updated. Very often I find myself in the situation where things have slowly changed in tiny increments over time, so it’s easy to make personal adjustments as necessary without updating the bio. Eventually all these changes add up and I’ll discover that my client bios are very out of date and it takes some time to update everything.

In fact, as I write this, I see that I’ve got some work to do on my own client bios…

If you’d like to know more about SQL Server Maintenance, check out Kevin’s Getting Started with SQL Server Maintenance course at Pluralsight.

Follow me on Twitter at @SQLandMTB, and if you’re into mountain bikes come over and check out my site NTX Trails.

Follow @Dallas_DBAs

Welcome back to my Server Review Essentials for Accidental and Junior DBAs series. Last week, we took a closer look at the first steps of getting your environment set up, including an overview of password management and VPN clients. This week we’ll address RDP issues, SSMS installation, building a script library, and more.

Let’s get started.

RDP Clients and Jumpboxes

I’m not going to go very deep into how to remote into another computer via RDP. In short, depending on your environment, you’ll need to have RDP set up on your computer (which is already part of the Windows operating system) and possibly need to also set up the Remote Desktop Connection Manager or the Universal Remote Desktop Client.

The most common workflow as a consultant is to connect to your client’s specific VPN, then RDP into your client’s server.

Once you’ve got your remote access client all set up, hopefully you’ll be connecting from your local machine to a jumpbox rather than directly to a production server. What’s a jumpbox, aka terminal server or Citrix server? Simply put, it’s a computer (either bare metal or VM) that servers as an intermediary between your local hardware and the servers you’re administrating.

It’s typically a bad idea to connect directly to a production server because…well…accidents and mistakes happen to the best of us. It’s somewhat safer to make mistakes on a jumpbox instead of a prod machine. One of the most common mistakes you can make is accidentally clicking “Shut Down” instead of “Sign Out” or “Disconnect” when you’re logging out of a server. At least if you shut down a jumpbox, you’re not taking down production.

Jumpboxes also serve as the perfect place to keep all your client-specific scripts and tools. Some clients may choose to set up the box that runs their monitoring software as the jumpbox. This is a cost-effective solution.

A best practice is to log into a jumpbox and use SSMS on that box to connect to the servers you administer. Yes, it’s still possible to mess up production, but as a junior DBA you’re likely logging in to run basic check script. Here at Dallas DBAs, if I’m doing something beyond running daily checks or periodic HealthChecks, Kevin is looking over my shoulder virtually via Zoom to make sure I don’t miss anything.

Even so, it’s still best to be careful and pay attention to what you’re doing. Depending on what you’re digging into with SSMS, it’s easy to let your mouse or attention slip and click the wrong thing. If you’re not paying attention, you could find yourself disabling a mission-critical job or worse!

If you’re in an environment that doesn’t use jumpboxes, see if you can push for one to get implemented. Your company/client may not want to spend the extra expense of spinning up a VM, but I can assure you the costs of accidentally taking down production is much higher.

Keep in mind that there may still be valid reasons to log directly into a particular production server, but we should aim to minimize our direct footprint in production as much as possible.

VM Environments

There’s not a lot to say about using virtual machines other than—if your work remotely like I do—you’ll likely be logging into a VM on a regular basis. Even if you’re administrating on-prem servers, a jumpbox in a VM is the safest approach.

Note from Kevin: On-prem folks will typically work with SSMS on their laptop—which is effectively their jumpbox—unless they have multiple internal domains.

For most usages, a VM isn’t all that different from physical hardware from an operational standpoint, but it wouldn’t be a bad idea to familiarize yourself with the basics of dealing with VMs. You never know when you’ll need to troubleshoot an issue unique to a VM.

Another consideration is to set up your own personal VM as a backup jumpbox of sorts. This allows you to have emergency access to your tools when you need them. Imagine being away from your office or work machine that has all your logins and tools—hopefully, a rare occurrence. If you have access to a computer with a secure internet connection you can get logged into your VM and jump from there to your client’s servers. If you’ve set up your VM to have all the login credentials and scripts you need to serve your clients, you’re (hopefully) good to go.

Note from Kevin: This doesn’t work in all situations. Some VPNs will kick you out of your VM when you fire them up.

Another scenario I’ve personally encountered is one that forces me to use a VM as a jumpbox. We have one particular client that uses both VMWare and another cloud provider for their environments. I have zero issues connecting to the VMWare boxes via RDP, but when I attempt to RDP into the cloud provider’s boxes I just get a black square on my screen. My workaround is to fire up my VM and connect via RDP from there. It’s a few extra steps, but necessary since I haven’t yet found a solution for my laptop.

SSMS Setup

SQL Server Management Studio (SSMS) is the main tool you’ll be using for the foreseeable future as a database administrator. Yes, Azure Data Studio (ADS) has come a long way recently, and the concept of using Jupyter Notebooks for running daily scripts is highly intriguing and something we plan to look into very soon, but there are enough missing pieces in ADS that keep it from being the defacto admin tool.

Note that, beginning with SSMS 18.7, Azure Data Studio is included in the install. Many of us aren’t too happy with that decision by Microsoft, and hopefully, they’ll listen to their user base and reverse that decision.

At one time, SSMS was part of the installation of a SQL Server instance, but beginning with SQL Server 2016, SSMS became a separate install. This post isn’t going to go through the ins-and-outs of installing or using SSMS.

Depending on your company or client’s policies, you may not have much say in how you get SSMS set up. You’ll need to discover for yourself how much freedom you have. Some policies will dictate the specific version of SSMS you’re allowed to use, others won’t care what you use as long as you don’t break their servers and get your work done.

Granted, I currently have less than two years of experience at this point, but my personal recommendation is to install the latest version of SSMS when allowed. SSMS is backward compatible with older versions of SQL Server, and newer versions of SSMS give you access to new and updated tools that help you get your work done. There’s really no reason to not use the latest version of SSMS unless you’re on hardware that simply can’t handle it for some reason.

Script Library

As we work our way through this series on server reviews, I’ll share several different scripts we use to check various things on client servers and show you how we use them. I’ve learned that it’s important to not only build your own library of useful scripts on your local machine, but to also create individual folders of scripts for each client.

Why? Because over time you’ll customize some of these scripts for the specific needs and quirks of your client’s servers. Having a backup of those specific customizations is going to be a lifesaver. As I stated above, you’ll hopefully have scripts available on individual client jumpboxes for your daily use, but keeping a local copy that you periodically update is a best practice.

I learned this the hard way a few months ago. One of our clients has a habit of quietly decommissioning servers without warning. We often don’t find out the server is gone until we’re running daily checks against all the boxes in a Registered Servers list and get a connection error of some sort. When we inform the client that the server is unavailable, they’ll often respond, “Oh, we don’t use that box anymore so we decommissioned it.”

One of the boxes it seems they had decided to dump was the jumpbox we use. Suddenly, I could no longer use the jumpbox to check their servers and was forced to log directly into a production machine to get my work done. I learned—to my frustration—that I had to spend a couple of hours rebuilding my tools and customized scripts. That’s on me since I hadn’t bothered to create a local copy of those resources.

A few days after we inquired about the decommissioned jumpbox—after receiving no response about the issue from the client—the jumpbox magically reappeared, complete with all our custom scripts and tools. I immediately pulled copies of everything possible in case the box disappeared again, and now have backup copies of every clients’ custom stuff. All of this is backed up to the cloud, and I’m also working on building a library of these files in our company Teams environment.

We’re DBAs. We should know how to back up our own stuff. Just get it done.

What should your script library look like? That’s up to you unless your employer has a specific protocol for such things. I know some folks like to have an application to manage their scripts and code snippets, while others use a private GitHub repository or Jupyter Notebooks. So far, simply organizing my scripts into folders and subfolders on my local drive—synced to OneDrive—has been sufficient for my needs.

Bonus: Slack

I can’t emphasize enough how much we use Slack on a daily basis. I’m not going to spend time selling it to you or talking about how to use it. Just check it out and see how you can best implement Slack usage in your team’s environment. You’ll be glad you did.

If you’d like to know more about SQL Server Maintenance, check out Kevin’s Getting Started with SQL Server Maintenance course at Pluralsight.

Follow me on Twitter at @SQLandMTB, and if you’re into mountain bikes come over and check out my site NTX Trails.

Follow @Dallas_DBAs

Welcome to a new year, and a new look at what I’ve learned in my time as a beginner and junior DBA.

So, I’ve been in this DBA gig for just over 1.5 years, and it’s been quite a fun and interesting ride. Many readers have read my Ironic DBA Files posts where I regularly talked about what I was learning as a new DBA, and then my four-part series on reviewing my first full year. If you haven’t read them, you can go back and see everything I’ve written so far.

One of the things I mentioned in my four-part series was how I had begun working daily in Production, checking out some of our client’s servers on a daily basis. I shared a super-basic overview of what I looked for each day, and how I reported my findings to Kevin and/or our clients.

I thought 2021 was a great time to revisit these ideas in greater detail. There may be accidental or junior DBAs out there who could benefit from seeing how I go about my daily tasks, so I whipped up a series of posts to share the basics of performing server reviews. Along the way, Kevin will chime in with separate comments and posts using his 20+ years of knowledge and experience to take us to a deeper level in some of these issues.

Welcome to the party. Let’s get started.

Environment Setup

Keep in mind during these first two posts—and likely many others—that the specifics of your environment will vary depending upon your personal workflow and the policies/procedures of your employer. Some of you are working from home like me, and will find a lot of what I’m sharing practical to implement, while others who are working in an office environment may need to alter what you read to fit your specific environment.

To get started performing server reviews, you’ll need to get five basic sets of tools set up in your environment:

• Password management software
• VPN clients
• RDP clients and Jumpboxes
• SQL Server Management Studio (SSMS)
• Script library

This week we’ll take a closer look at the first two: password management and VPN clients.

Password Management

Again, depending upon the requirements of your employer and/or circumstances, what you use to accomplish some things will vary. For instance, here at Dallas DBAs we use KeePass for our password management. Simply put, it does everything we need it to do flawlessly and allows us to share credential files with one another (when clients approve this). As long as we know the master password, we’re ready to connect to any client at any time.

Whatever you do, get something to manage your login credentials. Don’t try to memorize all your different passwords—that’s actually less secure than using a password manager.

VPN Considerations

If you work in an office environment, you may not need to use a VPN since it’s likely your local machine is connected to the same network as your servers, but this isn’t always the case. Sure, it’s probably a fair bet that anyone who’s been working in any sort of IT for any amount of time has used a VPN, so I’m not going to go into a lot of detail. My main purpose here is to simply share some of our experiences.

First up, if you have to opportunity to be a consultant, you’re going to run into half a dozen or more VPN clients, and you’ll need to learn the quirks of each one. Some of the VPNs we see most often include Cisco AnyConnect, Citrix Workspace and Citrix Storefront, Pulse Secure, Barracuda Network Access Client, and SecureLink. You’ll have to learn how to deal with and troubleshoot each one you use.

For instance, there’s a 50/50 chance that any given Citrix update will take away my ability to copy/paste directly out of a VPN window into a local text document. Thankfully, there’s a way to fix this, but I had to put on my search engine hat to find it—and sometimes this fix doesn’t work when Citrix pushes out a new update. No, it’s not hard, but it means I can’t just be a lazy DBA.

We also run into connection quirks with VPNs. Sometimes we’ve found ourselves unable to connect to a VPN if another VPN’s software is still running in the background. Pulse often had trouble connecting unless we killed off both Cisco and Citrix clients—even though they were not actively connected. Kevin had an issue for a while where he could not send or receive email while connected to a particular VPN client. A frustrating limitation when you’re trying to troubleshoot a problem while communicating with a client.

One odd quirk I have with the Barracuda client is that I have to unplug my ethernet cable to make a connection—the client only works over wifi. It took several minutes of troubleshooting the first time I used this client to get going. Now, when I have VPN client trouble, one of the first things I try is unplugging my ethernet.

Lastly, Kevin and I sometimes have to use shared VPN login credentials (not optimal!), and we’ve learned that there can be trouble connecting to a VPN if the other user was recently connected. We’re not absolutely sure of the cause, but we suspect that it has something to do with IP addresses since we live in different locations. The VPN seems to possibly reject an IP address if the other user had been connected (again, when clients give us a shared account to use). It can sometimes take several attempts and computer restarts before the VPN will accept a new connection.

Come back next week and we’ll cover RDP client issues, VM environments, and script libraries.

If you’d like to know more about SQL Server Maintenance, check out Kevin’s Getting Started with SQL Server Maintenance course at Pluralsight.

Follow me on Twitter at @SQLandMTB, and if you’re into mountain bikes come over and check out my site NTX Trails.

Follow @Dallas_DBAs