Linux Development:Detecting STDOUT Escape Char Support
======================================================
:author: Aaron Ball
:email: nullspoon@oper.io


If you've spent much time in the command line, especially if you script, you
will have noticed that some programs output colored output. This output, when
piped to another program such as 'less', is no longer colored.

I personally have written quite a few scripts that have colored or styled
(bold) output so the end user of my scripts has an easier time reading things.
However, when I want to output to a program such as 'less', my output is
mangled.

For example, a script like this...

----
#!/usr/bin/env bash

echo -e "\e[32mINFO\e[0m  This is a useful info message."
echo -e "\e[33mWARN\e[0m  This is an ominous warning message."
echo -e "\e[31mERROR\e[0m This is a scary error message."
----

Outputs something like this in the terminal

[.terminal]
--
[green]#INFO#  This is a useful info message.

[yellow]#WARN#  This is an ominous warning message.

[red]#ERROR# This is a scary error message.
--


...But something like this when piped to less

[role="terminal"]
----
^[[32mINFO^[[0m  This is a useful info message.
^[[33mWARN^[[0m  This is an ominous warning message.
^[[31mERROR^[[0m This is a scary error message.
----


As you can see, the escape codes are sent verbatim to the program on the other
side of the pipe. If it doesn't properly render them by default (which newer
versions of gnu less do now), you'll see those escape codes on your output.
This is particularly annoying in the cases of log files. The same behavior
occurs also if stdout is redirected into a file.

So how do we detect that the end user has redirected output with a pipe or a
redirection so we can remove the escape codes and just ouput plain text?


The Solution
------------

Bash provides this really useful 'if switch', *-t*. From the bash man page...

[quote, Bash(1)]
____
...
*-t* fd  True if file descriptor fd is open and refers to a terminal.
...
____

Let's give this a try.

.test.sh
----
#!/usr/bin/env bash

if [[ -t 1 ]]; then
  echo -e "\e[32mYay! Colored text is cool!\e[0m"
else
  echo -e "Boo! Plain text is lame."
fi
----


If we execute this with './test.sh', we'll see the green colored text "Yay!
Colored text is cool!".

If we then execute this with './test.sh | less', we'll see the plain text "Boo,
Plain text is lame." This also outputs plain text if you redirect the output
using './test.sh > test.out'.



What -t Really Does
-------------------

When I saw this functionality in bash, I immediately wanted to know how it
worked. Sure, I'd read the documentation, but there's no understanding like the
understanding that comes from writing it in C yourself.


----
#include <stdio.h>
#include <sys/stat.h>

/**
 * Detects if the specified file descriptor is a character device, or something
 * else.
 * Useful for determiniing if colored output is supported.
 * 
 * @param FILE* fd File descriptor to check
 *
 * @return int FD is char dev (1) or not (0)
 */
int ischardev(FILE* fd) {
  struct stat d;
  // stat the file descriptor
  fstat(fileno(fd), &d);

  // Check st_mode (see "man 2 stat" for more information about this)
  if(S_ISCHR(d.st_mode))
    return 1;

  return 0;
}

int main(int argc, char* argv[]) {
  if(ischardev(stdout)) {
    printf("Character dev. Colors supported\n");
  } else {
    printf("Something else. Colors not supported\n");
  }

  return 0;
}
----

In this code, we can see how to detect if stdout will support formatting escape
characters.

In the ischardev function, we stat the stdout file descriptor and check if it
is a character device (IS_CHR). Only character devices support escape character
styling, so if the stdout file descriptor is anything other than a character
device, it is safe to assume it does not support escape character formatting.



[role="datelastedit"]
Last edited: {docdate} {doctime}
// vim: set syntax=asciidoc:
