What do you do when you stuff up?

I don’t think there is a single sysadmin on this planet who hasn’t made a mistake significant enough to cost their employer or customers cash or reputation. So, what do you do when faced with such a situation?

I’ve read some ‘I stuffed up, should I ‘fess up?’ posts on Reddit recently. I and others I’ve worked with are no strangers to this question. Should one come clean with mistakes, and why?

Some years ago I was part of a team migrating some back-end storage to a shiny new SAN. The migration included moving the data of some not-so-shiny (read ‘legacy’) servers to the new SAN. These servers were hosting critical functions and were becoming quite long in the tooth. However the cost of replacing them was turning out to be far greater than the cost of keeping them limping along, so we were stuck with them for the present. It’s a common enough story in the field.

The customer brief was to migrate the legacy servers’ data to the new SAN with limited impact to services – i.e, online. We did our research and planned the migration as carefully as was possible given the age of the environment and the limited vendor support available. We decided to mitigate possible trouble ahead by breaking the work up into manageable stages.

Finally using a carefully reviewed work plan and change management approvals, at a time of low user activity (i.e, around midnight!), the first stage of the migration began. But part-way through, it all went wrong. One server metaphorically kicked another one in the guts, forcing a reset, and the hosted services ground to a halt.

The team was onto service recovery like a flash. But the work that had been planned to be online had now become offline. Critical services were disrupted. It was tempting to pretend that the outage had been caused by something other than the work I’d been doing. The servers had a history of instability and it would be easy to point the finger there.

I chose to be up-front and let my manager and the customer know what had caused the outage. Why? Your reasons might be different, but some of mine were:

1. There was an agreed process that needed to be followed.

The organisation I worked for had clear engagement rules in place, internally and externally, around change and incident management. This included things like what to do if scheduled work went wrong. If I didn’t follow these processes, I would be breaking faith with both my employer and the customer. To me this is the most compelling reason to ‘fess up when we stuff up. However I have other reasons too.

2. Stakeholders shouldn’t be kept in the dark.

My manager and the customer’s critical incident manager needed to know what had really happened so they could do their job. If I didn’t let them know, they wouldn’t have the information they needed. For instance, my manager needed to have full disclosure from me in order to act appropriately if the incident got escalated. Similarly, the customer’s incident manager needed to know what was going on so he could manage escalations at his end.

This might sound like stating the proverbial obvious, but non-stakeholders didn’t need to know. It would have been completely inappropriate for me to share what happened with, say, a competitor of my customer! Likewise, I have withheld identifying details from you as a reader of this post. 🙂

3. ‘Fessing up is a trust-building exercise.

  • My manager knew I wasn’t hiding facts to cover my behind. Why? Because I have a track record of making full disclosures of my mistakes, and by doing so again now, I was confirming our relationship of trust. A trusted relationship with one’s boss is of great value!
  • The customer’s incident manager didn’t yell down the phone at me or start a witch-hunt afterwards – both of which he could have done. Instead he listened to the facts and did his best to help me manage the incident. Why? Because I’d built up trust with this customer over time, by being honest when I didn’t know something, being willing to help when I could, and being upfront with them when I’d made mistakes before. They knew they would get a straight story from me. My honest communication in this situation helped maintain that trust. The incident manager and I later worked on a report, which fed into a modified work plan for the rest of the migration after we identified that yet another unresolved bug had caused our work plan to go awry.

4. The truth is easier to sustain than a lie!

Always. It’s much less exhausting to tell the truth than keep up a lie. Even in fiction. Agatha Christie’s Hercule Poirot believed that people found it a relief to tell the truth. A lie is too much effort to sustain. The great Sheldon Cooper would agree, spinning his “un-unravelable web” in Season 1 Episode 10 of The Big Bang Theory (‘The Loobenfield Decay’)!

5. Last but not least, it’s about reputation.

I am not really such an unselfish character as this story may make me sound. There is something in it for me. Making short-term choices like this helps my reputation. And that benefits me too in the long run.

Growth happens when we acknowledge our mistakes, don’t make excuses, and instead make a plan to move forward.

Unfortunately in today’s litigation-and-blame culture we are not encouraged to come clean. It can be difficult to straddle the line between transparency and liability. I think we have gone overboard with this, to the point that most tech professionals work in an atmosphere of blame management. I’m pointing the finger at me too here. Blame management isn’t healthy, and it doesn’t produce the kind of relationships that in turn produce dividends in the long run.

What do you think? Please feel free to leave a comment below.

Featured image by Andrea Piacquadio from Pexels.com.

Python Turtle Art

I was inspired to make art with Python today. Yes, really.

It came out of some coding lessons I have been preparing for two high school students. As an aside, I’ve been using Runestone Academy’s Foundations of Python Programming, modifying it to suit students who have learnt English as a second or third language.

Python Turtle module

After messing around with an example Python script for my students, I used Python’s random module to create some cool effects using a simple script. You can find the code for the script TurtleTriangles.py on GitHub. Feel free to download and run it – as long as you have a Python environment to run it in.

TurtleTriangles.py uses randomised values, so each invocation will produce a unique result – and one-of-a-kind artwork. Yes, really! 😀

If you code in Python and have never used Turtle before, I highly recommend it for the sheer visual rewards it gives. Read all about what you can do with Turtle at the official Python documentation site.

Feel free to post any artwork you make using Turtle in the comments below. I would love to see it!

TurtleTriangles.py

You can download TurtleTriangles.py and modify it to suit your Python environment and your preferences. My coding is by no means perfect, elegant, or efficient, so this script has much room for improvement.

For instance, you may need to change the first line which is the ‘shebang’ header, as it was written assuming it would be run on Linux using Python 3:

#!/usr/bin/python3

You may also like to change the colour of the background away from bright orange! 

window.bgcolor("orange")

The ranges of the randomised values can be adjusted too: 

draw_triangle(random.choice(sizelist),  \
   random.choice(colorlist), random.randint(-175, 175), \
   random.randint(-175, 175), random.randrange(0, 360, 10))

Basically you can go to town with it! After all, TurtleTriangles.py is open source. GNU General Public License v3.0 to be exact.

If you find any issues with it, please let me know in the comments below, or by logging an issue at the GitHub repository itself.

Happy creating!

find: a cool command for sysadmins

One of my all-time favourite Linux commands as a sysadmin is ‘find‘. It’s flexible, powerful and it’s got me out of trouble many times.

find works by traversing a directory tree for files and folders that meet a specified search criterion.

For example, if you want to find a JPEG file called Cats.JPG under /home/suuze_linux:

$ find /home/suuze_linux -name Cats.JPG

What if you know the JPEG file’s name has the word ‘Cats’ in it, but you aren’t sure exactly what the rest of the name is? You can use wildcards:

$ find /home/suuze_linux -name "*Cats*.JPG"

The searches above are case-sensitive. If you’re not sure if the filename has ‘Cats’, ‘cAts’, ‘cats’, or any other combination of the letters, you could try:

$ find /home/suuze_linux -iname "cats.jpg"

find can search using many other kinds of criteria. To search for items that have changed in the last day:

$ find /home/suuze_linux -mtime -1

The ‘-‘ indicates we’re looking for all files with an inode modification time less than 1 day. If the ‘-‘ is left off, find will return matches whose inodes were modified exactly 1 day ago.

How about files that were created in the last week? No problem:

$ find /home/suuze_linux -ctime -7

In order to fully understand how find works, bear a couple of things in mind: One of the fundamental (and over-simplified) precepts of Linux and UNIX-like operating systems is that everything is a file. This is not strictly true. But the precept works for our purposes here. find uses inode data to look for matches. Everything in a Linux filesystem is associated with an inode (Index Node). Therefore (unless otherwise specified), find will return both files and folders that match.

Here’s another example. Perhaps your root filesystem is close to capacity. You want to find all files that are greater than 1 GiB in size:

$ sudo find / -size +1G

Again, if the ‘+’ was left off, find will return all files that are exactly 1GiB in size.

As an aside, GiB stands for gibibytes (i.e, 1024 * 1024 * 1024 bytes). This is different to GB which stands for gigabytes (i.e, 1000 * 1000 * 1000 bytes). Why do we care? Among other things, apparently a whole lot of lawsuits have been brought against hardware manufacturers over the difference between 10243 and 10003! You can read all about this at MASV’s article, GB vs GiB: What’s the Difference Between Gigabytes and Gibibytes?

The power of find is in its flexibility. Multiple search criteria can be combined. Let’s say your filesystem is filling up, and you know you have old ISO files lying around that can be got rid of. You could search for them with something like this:

$ sudo find / -iname "*.iso" -type f -mtime +365 -size +4G

Breaking it down:

  • –iname specifies a case-insensitive search.
  • ‘*‘ is a wildcard which will match a string of length 0 or more.
  • -type specifies the file type – in this case, ‘f’ will constrain the search to regular files only.
  • –mtime specifies all inodes that were modified over 1 year ago.
  • –size specifies 4 gibibytes or larger.

The above command would give you a list of files and folders that match the search criteria – if any exist. The resulting list will be fully qualified. I.e, find will give you the complete path to each matching file or folder. You can then navigate to that file or folder, query it using ls, remove it using rm, etc.

However, find can make our lives even easier with the ‘-exec’ flag. Like so:

$ sudo find / -iname "*.iso" -type f -mtime +365 -size +4G -exec ls -l {} \;

Instead of returning a list of matching files, find will run the command that follows ‘-exec’ on each matching file. In this case, it will do a long listing of each. Neat, huh?

Note that the command above can be shortened as follows:

$ sudo find / -iname "*.iso" -type f -mtime +365 -size +4G -ls

The ‘-ls’ flag effectively is a ‘-exec ls -lids {} \;’ – it does a long listing on matching files and directories, displaying inode numbers and sizes in blocks.

A sequence of find commands such as the following is a handy addition to a Linux sysadmin’s toolbox. 

$ sudo find / -iname "*.iso" -mtime +365 -size +4G -exec ls -ld {} \;

followed by: 

$ sudo find / -iname "*.iso" -mtime +365 -size +4G -exec rm {} \;

or:

$ sudo find / -iname "*.iso" -mtime +365 -size +4G -delete

However, use this with caution! ‘rm‘ and the ‘-delete’ option both remove files permanently – there is no recycle bin option here. Once rm has removed a file on a Linux filesystem, the only way to get it back is to restore from a backup (if one exists!).

A cool thing about Linux is how the output of one command can be used in another. For instance…

  • if you want to change directory into a folder called ‘scripts’,
  • you know scripts is under your home directory somewhere but you can’t recall exactly where, and
  • you know there is only one ‘scripts’ folder sitting under your home directory tree somewhere,

… you could run something like this from your home directory: 

$ cd "`find /home/suuze_linux -name scripts`"

A ‘pwd‘ should now show that your current working directory is the directory scripts and the path to it.

I used double quotes in the example above because I know the path to my scripts folder is likely to have folders with spaces in their names. If you knew for sure there are no folder names with spaces in the directory tree, you could leave out the double quotes. But leave the back ticks in – they are what causes the Linux shell to evaluate the find statement first, before passing the result to cd.

This post just scratches the surface of what find can do! To find out more:

$ man find

or

$ find --help

What do you think? What is your favourite find option? Feel free to comment below – and happy finding!

New to Docker?

Docker is a containerization system that has been around since 2013. For the uninitiated, it can be a complex beast to understand and conquer. Containerization systems like Docker and Kubernetes are key to rapidly deploying applications and services and lend themselves to DevOps and models like the increasingly popular continuous integration / continuous deployment.

If you are a new Docker user, you might find the following articles helpful:

Happy reading!

“Love is chemistry” – Inma Martinez on AI

I’m fascinated by AI’s potential and promise.

And so I enjoyed this interview in EL PAÍS. Inma Martínez, an expert in artificial intelligence, chats about natural language processing (NLP), smart cars and whether technology can predict the future. Enjoy. 😊

https://english.elpais.com/usa/2021-12-07/love-is-chemistry-algorithms-fail-the-more-abstract-and-complicated-a-person-is.html

TaskJuggler: an open source project management tool

So my random Linux-related inspiration for the week has been to try out TaskJuggler.

My husband and I will be launching our older child into the world later this year, and there are a few convoluted tasks involved in making this happen. It’s doing my head in. So I went hunting for a Linux-based project management tool that I would find relatively clean and useful to work with.

TaskJuggler is an open source project management tool. I came across it in UbuntuPIT’s post “The 20 Best Project Management Tools for Linux Desktop” from February this year. So I’m hoping all its recommendations are up to date, though TaskJuggler itself doesn’t seem to have been updated since 2020.

TaskJuggler’s latest version doesn’t have a graphical user interface. Apparently version 2.x did, but the GUI hasn’t been ported across to 3.x. You can use a command shell, a plaintext editor (no fancy word processing software allowed) and a web browser. This might therefore only appeal to command-line nerds like me.

Installation

To install, I followed the documentation at https://taskjuggler.org. Basically one installs Ruby Gems (if not already done), then installs the Ruby Gems package TaskJuggler. If you like reading transcripts, here’s mine. Note that I didn’t install taskjuggler system-wide, just for my user account.

$ sudo dnf install rubygems
Last metadata expiration check: 1:09:23 ago on Fri 01 Apr 2022 14:39:29.
Dependencies resolved.
==============================================================================
 Package                 Arch        Version               Repository    Size
==============================================================================
Installing:
 rubygems                noarch      3.2.22-149.fc34       updates      247 k
Installing dependencies:
 ruby                    x86_64      3.0.2-149.fc34        updates       41 k
 ruby-libs               x86_64      3.0.2-149.fc34        updates      3.2 M
 rubygem-json            x86_64      2.5.1-201.fc34        fedora        64 k
 rubygem-psych           x86_64      3.3.0-149.fc34        updates       51 k
 rubypick                noarch      1.1.1-14.fc34         fedora       9.9 k
Installing weak dependencies:
 ruby-default-gems       noarch      3.0.2-149.fc34        updates       32 k
 rubygem-bigdecimal      x86_64      3.0.0-149.fc34        updates       54 k
 rubygem-bundler         noarch      2.2.22-149.fc34       updates      367 k
 rubygem-io-console      x86_64      0.5.7-149.fc34        updates       25 k
 rubygem-rdoc            noarch      6.3.1-149.fc34        updates      400 k

Transaction Summary
==============================================================================
Install  11 Packages

Total size: 4.4 M
Total download size: 4.3 M
Installed size: 16 M
Is this ok [y/N]: y
Downloading Packages:
:
:
Installed:
  ruby-3.0.2-149.fc34.x86_64                                                  
  ruby-default-gems-3.0.2-149.fc34.noarch                                     
  ruby-libs-3.0.2-149.fc34.x86_64                                             
  rubygem-bigdecimal-3.0.0-149.fc34.x86_64                                    
  rubygem-bundler-2.2.22-149.fc34.noarch                                      
  rubygem-io-console-0.5.7-149.fc34.x86_64                                    
  rubygem-json-2.5.1-201.fc34.x86_64                                          
  rubygem-psych-3.3.0-149.fc34.x86_64                                         
  rubygem-rdoc-6.3.1-149.fc34.noarch                                          
  rubygems-3.2.22-149.fc34.noarch                                             
  rubypick-1.1.1-14.fc34.noarch                                               

Complete!
$ 
$ gem install taskjuggler
Successfully installed taskjuggler-3.7.1
Parsing documentation for taskjuggler-3.7.1
Done installing documentation for taskjuggler after 3 seconds
1 gem installed
$ which tj3
~/bin/tj3
$  ruby -e "puts Gem::Specification.find_by_name('taskjuggler').gem_dir"
/home/<obscured>/.local/share/gem/ruby/gems/taskjuggler-3.7.1
$ tj3 --version
tj3 (TaskJuggler) 3.7.1
$ sudo find / -name taskjuggler
/home/<obscured>/.local/share/gem/ruby/gems/taskjuggler-3.7.1/lib/taskjuggler
$

Using it

TaskJuggler 3.x basically works like a compiler. The user sets up the project details in a source file and then passes them to the Task Juggler tj3 binary.

Taking TaskJuggler for a small test drive, I first created a test project using a plaintext editor. I used vi, but you could use anything – except a word processor. 

$ cat test-project.tjp 
### Test project

project test_project "My Test Project 2022"  2022-01-01 +12m {
  timezone "Australia/Melbourne"
  currency "AUD"
  timeformat "%Y-%m-%d"
  numberformat "-" "" "," "." 1
  currencyformat "-" "" "," "." 0
}

### Define resources

resource parent1 "Parent1" {
  email "blah@blahs.com"
}

resource parent2 "Parent2" {
  email "blahmore@blahs.com"
}

### Tasks and subtasks

task uni_app "University application" {
}

task travel_oz "Travel to Oz" {

  task accom_transport "Oz accommodation and transport" {
  }

  task flights "Flights to Oz" {
  }

} 

### Reporting and exports. Taken from https://taskjuggler.org example. 
taskreport breakdown "ProjectBreakdown" {
  formats html
  caption "This is the project breakdown"
  headline "Project Breakdown"
  columns name, start, end, daily
  # Don't hide any resource, meaning show them all.
  hideresource 0
}

# Export the project as fully scheduled project. Taken from https://taskjuggler.org example.
export "FullProject" {
  definitions *
  taskattributes *
  hideresource 0
}
$

The next step is to compile the source code file using TaskJuggler: 

$ ~/bin/tj3 test-project.tjp 
TaskJuggler v3.7.1 - A Project Management Software

Copyright (c) 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020
              by Chris Schlaeger <cs@taskjuggler.org>

This program is free software; you can redistribute it and/or modify it under
the terms of version 2 of the GNU General Public License as published by the
Free Software Foundation.

Reading file test-project.tjp                                [      Done      ]
Preparing scenario Plan Scenario                             [      Done      ]
Scheduling scenario Plan Scenario                            [      Done      ]
Checking scenario Plan Scenario                              [      Done      ]
Report ProjectBreakdown                                      [      Done      ]
Report FullProject                                           [      Done      ]
$
$ ls -la
total 52
drwxrwxr-x. 1 <obscured> <obscured>   186 Apr  1 17:47 .
drwxrwxr-x. 1 <obscured> <obscured>    62 Apr  1 17:41 ..
drwxrwxr-x. 1 <obscured> <obscured>    48 Apr  1 17:45 css
-rw-rw-r--. 1 <obscured> <obscured>  1515 Apr  1 17:47 FullProject.tjp
drwxrwxr-x. 1 <obscured> <obscured>   284 Apr  1 17:45 icons
-rw-rw-r--. 1 <obscured> <obscured> 17200 Apr  1 17:47 ProjectBreakdown.html
drwxrwxr-x. 1 <obscured> <obscured>    26 Apr  1 17:45 scripts
-rw-rw-r--. 1 <obscured> <obscured>  1102 Apr  1 17:47 test-project.tjp
$

Opening the file ProjectBreakdown.html in a browser shows this:

And that’s it.

The blurb from TaskJuggler’s home page says:

TaskJuggler is project management software for serious project managers. It covers the complete spectrum of project management tasks from the first idea to the completion of the project. It assists you during project scoping, resource assignment, cost and revenue planning, risk and communication management.

https://taskjuggler.org/index.html

It’s usefulness as a collaborative resource would be limited. There are far fancier and WYSIWYG tools around. But it has potential for someone like me who is looking for a reasonably simple text-based personal project management solution without all the overhead that a WYSIWYG solution brings.

Image credits TaskJuggler.org.

Coding: What if you don’t know English?

So I’ve been working on a little project recently. I’m creating an introductory Python course based on the Foundations of Python Programming from Runestone Academy, but modifying it for high school students whose primary language isn’t English.

As I ran a pilot of the course, I realised that my test pupil is somewhat disadvantaged. She knows English, but it isn’t her first language. I began to wish I could present her with a programming language whose tokens look like those from her first language. How much easier it would be for her to learn to code!

And then I realised how biased the world of programming is towards English.

The article “Coding Is for Everyone—as Long as You Speak English” by Gretchen McCulloch on Wired.com mirrors my frustrations.

Me and my soapbox

There needs to be a shift. I wonder what it might look like for this shift to be implemented. It might need a completely new tool we haven’t dreamed of yet. Or something we already have that can be repurposed.

Perhaps something akin to natural language processing that can translate source code into language tokens that align with different natural languages?

That article again from Wired.com is “Coding Is for Everyone—as Long as You Speak English.” Have a read and let me know your thoughts!

#BreakTheBias: Happy IWD 2022!

This International Women’s Day, we are being encouraged to Break The Bias.

I’m a woman who loves working in the tech industry, but often thinks I am less competent than male colleagues. Heck – I even think I’m less competent in tech than men who don’t work in the field, but can spout confident techie lingo.

So I love organisations like Girls In Tech whose mission is to encourage and support women working in tech roles. Girls In Tech run courses and provide resources for women in tech – and I’ve benefited from their training.

Head over to their article on #BreakTheBias to see what they have to say about IWD 2022.

I fell into open source by accident

So how did I start on this open source journey?

In the late 1990s, I was a fresh Computer Science graduate. Australia had recently come out of a recession, and the market was starting to be glutted with graduates like me. I was expecting to become a software developer. But about a hundred job applications and five interviews later, I got an entry-level sysadmin role. I became part of a team managing Windows file and print servers, one midrange server that ran an operating system that was neither Windows nor UNIX… and one brand spanking new Sun Microsystems midrange computer running SunOS 4. It became my first UNIX love.

Not long after, the I.T. infrastructure management was outsourced to a managed services provider (MSP). The new organisation had a strictly silo’ed approach. Our team of sysadmins was split across different specialities. I ended up in the Midrange/UNIX team.

And that’s how my specialisation in the UNIX and Linux space began.

International Women’s Day 2022’s theme is #BreakTheBias

We might get tired of hearing the same old report of how under-represented women are in science and technology. And is it really true, and why does it matter anyway?

Well, here’s a real life example. I worked for almost 25 years in tech. At least 24 of those years were focused on UNIX and, later, Linux. In all those years, I’ve had a grand total of 3 other female colleagues working in UNIX and Linux with me. I’ve had more female managers than colleagues.

That’s not to say there weren’t many women in the organisations I worked for. Just that the majority of them were in non-technical roles. Even database and app-development teams I’ve worked with have had a significantly large male majority, though they had more female representation than the UNIX and Linux space had.

There are many theories out there on this gender disparity in tech. Future post coming up on this – and also why this gender disparity matters anyway.

For now, ’nuff said. Bring on #BreakTheBias.

Women in Open Source

On 8 March, we celebrate International Women’s Day

IWD 2022’s theme is #BreakTheBias. It would be helpful in its lead-up to look at gender diversity in tech. Today, I’m taking a look at open source in particular.

Back in March 2021, /* Academy Software Foundation posted an article titled “The Truth about Women in Open Source” by Nithya Ruff. I found this encouraging and informative. The article details some stereotypes we have about women’s involvement in open source, and goes on to evaluate their truth and where some more work needs to be done to bridge gender gaps.

The article finishes with several resources that aspiring open source contributors might find helpful.

I also found it helpful to read up on Nithya’s biography here in a write-up on Behind the Tech.

If like me you identify as female and work (or used to work) in tech, what would you agree with, or disagree with, in Nithya’s article? Also, have you ever worked in open source, and what was your experience like?

Image credit ThisIsEngineering from Pexels.com.