Emergency

Does skipping a DBA save money? Wait until your system grinds to a halt, your backups fail, or your cloud bill skyrockets.

The “DBA? We Don’t Need No Stinkin’ DBA” Mindset

Common Excuses That Lead to Expensive Problems:

• “Our system runs fine,”
• “The cloud takes care of it”
• “Our sysadmin is really smart”
• “DBAs are expensive”

 

SQL Server is an Enterprise level database product with so many facets to it you could make 5 different careers developing and supporting it (DBA, Developer, BI Stack, Analyst and more). Terabytes of data, thousands of transactions per second.

I have seen repeated instances of someone not fully trained as a DBA reaching out to Google (faster than vendor support) for answers only to find a 15-year-old blog post and start throwing ancient answers at the server. More than once, a client has reached out to me while on hold with the vendor!

The Hidden Costs of Hiring a Junior DBA

Junior DBAs are generally wonderful people. I don’t bash people, pretty much ever. Almost all of them come from other parts of the overall IT ecosystem. The strongest DBA skills come from a LOT of study and a LOT of practice. You don’t get that in 2 years.

Your shiny new DBA (if you have one) might not realize the nuances of the following critical items:

• Index design at the table level – too many, not enough, duplicates, etc.
• Query tuning – When is a cursor fine, and when is it bad? What’s killing tempdb? What the heck is RBAR?
• Inefficient server level configurations. As I typed this, I found 2 major systems at a new client with some horrible default settings that are causing them very high CPU utilization. They are paying monthly for Enterprise licenses they shouldn’t need!
• Security – Poorly managed permissions = open door for ransomware you may never fully recover from.
• A Junior DBA may not be test restoring your databases. You MUST know that your backups are valid if you want to meet management expectations.
• I did this early in my SQL career – ASK FOR HELP sooner rather than later! Junior “anything” people will frequently try too hard to figure something out on their own before seeking assistance.

 

How Not Having a DBA Hurts Your Business

• Longer downtimes – lets face reality – having a seasoned veteran on speed-dial or in the office means a faster response and resolution. Opening a ticket with your vendor or hunting for a consultant available “now” is difficult, at best.
• Slow performance leads to lost productivity, angry employees, and customers buying elsewhere. When they STOP complaining, it has gone way too far.
• Compliance failures if security and audits aren’t handled properly can be costly in fines or remediation expenses.
• Expensive cloud bills due to inefficient configurations. The cloud isn’t free, or even cheap. Don’t take that nonsense from the cloud vendors. It’s far worse if you are paying for double the hardware of what you really need. Having access to DBA skills can help you right-size!

 

What’s the Right Balance?

A full-time mid to senior DBA makes a lot of sense when there are multiple SQL Servers (25+), various projects happening, tight RPO/RTO requirements, etc.

I ran a poll on LinkedIn asking where the line is drawn on how many SQL Servers a 5-7 year DBA can support:

With a small number of servers, a full-time Sr. DBA is probably overkill. They are going to get bored. I spent a year baby-sitting 4 servers for one of the Big 4 accounting firms. I worked 30 minutes a day and drank a lot of coffee. Until I couldn’t take it anymore and started Dallas DBAs.

With a limited footprint, you are probably better off outsourcing. You might look for a Senior DBA that freelances on the side (availability might be an issue), or a firm that offers a mix of maintenance and off-hours. Dallas DBAs is the best in the world of course (😊):

The Bottom Line: What Should You Do Next?

Get a SQL Server Health Check done. Periodically.
Look for signs of database neglect: Slow queries, long downtimes, too many people with sysadmin rights.
Don’t wait until you’re in emergency mode! Get ahead of the chaos today.

Thanks for reading!

Kevin

My Newsletter (full version)

New Pocket DBA® clients get the first month FREE!

Join my ‘Accidental SQL DBA‘ group here

Get a Free SQL Server Health Check!

Common Reasons for Emergency SQL calls

If you are a production DBA (or Accidental prod DBA) you’ve gotten that frantic call in the middle of the night. Or maybe during little Suzy’s soccer game? Something broke, alerts are firing, nobody can order your widgets and the help desk tech has no idea where to start.

I’ve been on call since 2006. These are the most common things I’ve been called for at the worst times:

Tempdb growth

When tempdb grows out of control, it can easily fill a drive and stop the SQL Server in its tracks. Every database (potentially) uses the same tempdb to do its sorting, aggregating, maintenance and spilling of large transactions. That is the purpose of it.

Nothing in the system will stop a really bad query from filling up the tempdb data files to consume the entire drive. Once it fails the system should be ok/online, but you still have a full drive firing alerts at the sysadmins.

How to prevent: cap the tempdb files to maximum size that will keep alerts from firing for disk space. Also, set up alerts for errors 1105 (data file is full) and 9002 (log file is full). These go to the DBA.

You cannot stop bad queries if you allow users to run Ad Hoc, but you can limit the blast radius.

Disk Space

I don’t have enough toes to count the number of times I’ve been hit with an on-call because disk space was causing issues in SQL Server. Most of the time I search for all files, sort by size descending and look at the biggest files. VERY frequently someone dumped a backup file there, or created a new DB on the wrong drive.

How to prevent: Log the output of exec master..xp_fixeddrives and keep an eye on it. Set permissions for just admins and SQL Service accounts to limit Joe’s ability to break things.

Backup failures

Tend to these quickly, as your RPO might be in jeopardy if backups are missing. Easily 80% of the backup failures I see are related to disk space (see above). 19% are network related. 1% is everything else.

How to Prevent: see Disk Space above. For T-log backups I set the job step to retry once before failing. Weird network blips shouldn’t cause someone to wake me up.

Patching
This is probably the hardest to deal with, but it comes up often. If you are not mostly current on Cumulative Updates and Security Patches, expect this call. Especially during a monthly Windows patch cycle – which doesn’t include SQL Server patches.

How to Prevent: see Disk Space above. For T-log backups I set the job step to retry once before failing. Weird network blips shouldn’t cause someone to wake me up.

The Bottom Line:
An ounce of prevention prevents your phone from ringing. Pay close attention to the setup and maintenance of your SQL Server to stop these issues before they start!