Immutable global variables

• Try to keep globals to minimum
• UPPER_CASE naming
• readonly decleration
• Use globals to replace cryptic $0, $1, etc.
• Globals I allways use in my programs:

1
2
3

readonly PROGNAME=$(basename $0)
readonly PROGDIR=$(readlink -m $(dirname $0))
readonly ARGS="$@"

Everything is local

All variable should be local.

1
2
3
4
5
6
7

change_owner_of_file() {
    local filename=$1
    local user=$2
    local group=$3

    chown $user:$group $filename
}

1
2
3
4
5
6
7
8
9
10
11

change_owner_of_files() {
    local user=$1; shift
    local group=$1; shift
    local files=$@
    local i

    for i in $files
    do
        chown $user:$group $i
    done
}

• self documenting parameters
• Usually for loop use i variable, so it is very important that you declare it as local.
• local does not work on global scope.

1
2

kfir@goofy ~ $ local a
bash: local: can only be used in a function

main()

• Help keep all variables local
• Intuitive for functional programming
• The only global command in the code is: main

1
2
3
4
5
6
7
8
9
10

main() {
    local files="/tmp/a /tmp/b"
    local i

    for i in $files
    do
        change_owner_of_file kfir users $i
    done
}
main

Everything is a function

• The only code that is running globaly is:
  • Global declerations that are immutable.
  • main
• Keep code clean
• procedures become descriptive

1
2
3

main() {
    local files=$(ls /tmp | grep pid | grep -v daemon)
}

1
2
3
4
5
6
7
8
9
10
11

temporary_files() {
    local dir=$1

    ls $dir \
        | grep pid \
        | grep -v daemon
}

main() {
    local files=$(temporary_files /tmp)
}

• Second example is much better. Finding files is the problem of temporary_files() and not of main()’s. This code is also testable, by unit testing of temporary_files().
• If you try to test the first example, you will mish mash finding temporary files with main algorithm.

1
2
3
4
5
6
7
8
9
10
11
12

test_temporary_files() {
    local dir=/tmp

    touch $dir/a-pid1232.tmp
    touch $dir/a-pid1232-daemon.tmp

    returns "$dir/a-pid1232.tmp" temporary_files $dir

    touch $dir/b-pid1534.tmp

    returns "$dir/a-pid1232.tmp $dir/b-pid1534.tmp" temporary_files $dir
}

As we see, this test does not concern main().

Debugging functions

• Run program with -x flag:

1

bash -x my_prog.sh

• debug just a small section of code using set -x and set +x, which will print debug info just for the current code wrapped with set -x … set +x.

1
2
3
4
5
6
7
8
9

temporary_files() {
    local dir=$1

    set -x
    ls $dir \
        | grep pid \
        | grep -v daemon
    set +x
}

• Printing function name and its arguments:

1
2
3
4
5
6
7
8

temporary_files() {
    echo $FUNCNAME $@
    local dir=$1

    ls $dir \
        | grep pid \
        | grep -v daemon
}

So calling the function:

1

temporary_files /tmp

will print to the standard output:

1

temporary_files /tmp

Code clarity

What this code do?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

main() {
    local dir=/tmp

    [[ -z $dir ]] \
        && do_something...

    [[ -n $dir ]] \
        && do_something...

    [[ -f $dir ]] \
        && do_something...

    [[ -d $dir ]] \
        && do_something...
}
main

Let your code speak:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

is_empty() {
    local var=$1

    [[ -z $var ]]
}

is_not_empty() {
    local var=$1

    [[ -n $var ]]
}

is_file() {
    local file=$1

    [[ -f $file ]]
}

is_dir() {
    local dir=$1

    [[ -d $dir ]]
}

main() {
    local dir=/tmp

    is_empty $dir \
        && do_something...

    is_not_empty $dir \
        && do_something...

    is_file $dir \
        && do_something...

    is_dir $dir \
        && do_something...
}
main

Each line does just one thing

• Break expression with back slash \
  For example:

1
2
3
4
5

temporary_files() {
    local dir=$1

    ls $dir | grep pid | grep -v daemon
}

Can be written much cleaner:

1
2
3
4
5
6
7

temporary_files() {
    local dir=$1

    ls $dir \
        | grep pid \
        | grep -v daemon
}

• Symbols at the start of the line indented
  Bad example of symbols at the end:

1
2
3
4
5
6
7

temporary_files() {
    local dir=$1

    ls $dir | \
        grep pid | \
        grep -v daemon
}

Good example where we clearly see the connection between lines and the connecting rods:

1
2
3
4
5
6
7

print_dir_if_not_empty() {
    local dir=$1

    is_empty $dir \
        && echo "dir is empty" \
        || echo "dir=$dir"
}

Printing usage

Don’t do this:

1
2
3

echo "this prog does:..."
echo "flags:"
echo "-h print help"

It should be a function:

1
2
3
4
5

usage() {
    echo "this prog does:..."
    echo "flags:"
    echo "-h print help"
}

echo is repeated in each line. For that we have Here Document:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

usage() {
    cat <<- EOF
    usage: $PROGNAME options
    
    Program deletes files from filesystems to release space. 
    It gets config file that define fileystem paths to work on, and whitelist rules to 
    keep certain files.

    OPTIONS:
       -c --config              configuration file containing the rules. use --help-config to see the syntax.
       -n --pretend             do not really delete, just how what you are going to do.
       -t --test                run unit test to check the program
       -v --verbose             Verbose. You can specify more then one -v to have more verbose
       -x --debug               debug
       -h --help                show this help
          --help-config         configuration help

    
    Examples:
       Run all tests:
       $PROGNAME --test all

       Run specific test:
       $PROGNAME --test test_string.sh

       Run:
       $PROGNAME --config /path/to/config/$PROGNAME.conf

       Just show what you are going to do:
       $PROGNAME -vn -c /path/to/config/$PROGNAME.conf
    EOF
}

Pay attention that there should be real tab ‘\t’ in the start of the line for each line.
In vim you can use this replace command if your tab is 4 spaces:

1

:s/^    /\t/

Command line arguments

Here is an example to complement the usage function above. I got this code from Kirk’s blog post - bash shell script to use getopts with gnu style long positional parameters:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

cmdline() {
    # got this idea from here:
    # http://kirk.webfinish.com/2009/10/bash-shell-script-to-use-getopts-with-gnu-style-long-positional-parameters/
    local arg=
    for arg
    do
        local delim=""
        case "$arg" in
            #translate --gnu-long-options to -g (short options)
            --config)         args="${args}-c ";;
            --pretend)        args="${args}-n ";;
            --test)           args="${args}-t ";;
            --help-config)    usage_config && exit 0;;
            --help)           args="${args}-h ";;
            --verbose)        args="${args}-v ";;
            --debug)          args="${args}-x ";;
            #pass through anything else
            *) [[ "${arg:0:1}" == "-" ]] || delim="\""
                args="${args}${delim}${arg}${delim} ";;
        esac
    done

    #Reset the positional parameters to the short options
    eval set -- $args

    while getopts "nvhxt:c:" OPTION
    do
         case $OPTION in
         v)
             readonly VERBOSE=1
             ;;
         h)
             usage
             exit 0
             ;;
         x)
             readonly DEBUG='-x'
             set -x
             ;;
         t)
             RUN_TESTS=$OPTARG
             verbose VINFO "Running tests"
             ;;
         c)
             readonly CONFIG_FILE=$OPTARG
             ;;
         n)
             readonly PRETEND=1
             ;;
        esac
    done

    if [[ $recursive_testing || -z $RUN_TESTS ]]; then
        [[ ! -f $CONFIG_FILE ]] \
            && eexit "You must provide --config file"
    fi
    return 0
}

You use it like this, using the immutable ARGS variable we defined at the top:

1
2
3
4

main() {
    cmdline $ARGS
}
main

Unit Testing

• very important in higher level languages
• Use shunit2 for unit testing

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

test_config_line_paths() {
    local s='partition cpm-all, 80-90,'

    returns "/a" "config_line_paths '$s /a, '"
    returns "/a /b/c" "config_line_paths '$s /a:/b/c, '"
    returns "/a /b /c" "config_line_paths '$s   /a  :    /b : /c, '"
}

config_line_paths() {
    local partition_line="$@"

    echo $partition_line \
        | csv_column 3 \
        | delete_spaces \
        | column 1 \
        | colons_to_spaces
}

source /usr/bin/shunit2

Here is another example using df command:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

DF=df

mock_df_with_eols() {
    cat <<- EOF
    Filesystem           1K-blocks      Used Available Use% Mounted on
    /very/long/device/path
                         124628916  23063572 100299192  19% /
    EOF
}

test_disk_size() {
    returns 1000 "disk_size /dev/sda1"

    DF=mock_df_with_eols
    returns 124628916 "disk_size /very/long/device/path"
}

df_column() {
    local disk_device=$1
    local column=$2

    $DF $disk_device \
        | grep -v 'Use%' \
        | tr '\n' ' ' \
        | awk "{print \$$column}"
}

disk_size() {
    local disk_device=$1

    df_column $disk_device 2
}

Here I have exception, for testing, I declare DF in the global scope not readonly. This is because of shunit2 not allowing to change global scope functions.

I want to block program from running twice simultaneously. Why would I want such a thing? Lets say you have a program that creates the directory /tmp/prog and update files inside this directory. If you run the program twice at the same time, files inside this directory will get clobbered. Searching the net for solution, flock comes up as a good way to solve the problem. flock is part of the util-linux package.

Basic solution

Here is an example posted at stackoverflow:

1
2
3
4
5
6
7
8
9
10
11
12

#!/bin/bash

# Makes sure we exit if flock fails.
set -e

(
    # Wait for lock on /var/lock/.myscript.exclusivelock (fd 200) for 10 seconds
    flock -n 200

    # Do stuff

) 200>/var/lock/.myscript.exclusivelock

At the heart of the locking mechanizm is the last line, which create the file /var/lock/.myscript.exclusivelock using file descriptor 200. The syntax 9>textfile, is the way in BASH to create textfile using file descriptor 9. Then we check if the file is locked in line 8, flock -n 200 using the file descriptor 200 we used to create the lock file with. So while the critical code after line 8, that runs in the sub shell, is running, the file /var/lock/.myscript.exclusivelock stays locked, and will be released when the sub shell exits.

Next step

My first rule of programming in BASH, is to have all the code functional and the only line that is globally executed is main like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

#!/bin/bash

do_A() {
    ...
}

do_B() {
    ...
}

main() {
    do_A
    do_B
}
main

So using this example, to lock the program we can do this:

1
2
3
4
5
6
7
8
9
10

main() {
    do_A
    do_B
}

(
    flock -n 200
    main

) 200>/var/lock/.myscript.exclusivelock

or:

1
2
3
4
5
6
7
8
9

main() {
    (
        flock -n 200
        do_A
        do_B

    ) 200>/var/lock/.myscript.exclusivelock
}
main

First example breaks my rule, and main is not the only global code, but is not too bad. The second example, does not look good to me. It seems that main have more syntax complexity then it should. What we really want, in order to keep the code clean, is creating a lock function that will behave like this:

1
2
3
4
5

main() {
    lock || exit 1
    do_A
    do_B
}

Creating a locking function

To move locking to a function, we’ll need to get rid of this sub shell syntax.

1
2
3
4
5

(
    flock -n 200
    ...

) 200>/var/lock/.myscript.exclusivelock

The answer is to be found in jdimpson’s blog post using flock to protect critical sections in shell scripts. Using exec to create the lock file:

exec 200>/var/lock/.myscript.exclusivelock

Now we can crate the lock file, and use flock to acquire it:

1
2
3
4
5
6
7
8
9
10

main() {
    exec 200>/var/lock/.myscript.exclusivelock

    flock -n 200 \
        || exit 1

    do_A
    do_B
}
main

Lets create the lock function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

lock() {
    exec 200>/var/lock/.myscript.exclusivelock

    flock -n 200 \
        && return 0 \
        || return 1
}

main() {
    lock || exit 1

    do_A
    do_B
}
main

So now we have hard coded function ;-) What can we do next? Lets get rid of the magic number 200 and the lock filename, and turn them to a variables.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

#!/bin/bash

readonly PROGNAME=$(basename "$0")
readonly LOCKFILE_DIR=/tmp
readonly LOCK_FD=200

lock() {
    local prefix=$1
    local fd=${2:-$LOCK_FD}
    local lock_file=$LOCKFILE_DIR/$prefix.lock

    # create lock file
    eval "exec $fd>$lock_file"

    # acquier the lock
    flock -n $fd \
        && return 0 \
        || return 1
}

eexit() {
    local error_str="$@"

    echo $error_str
    exit 1
}

main() {
    lock $PROGNAME \
        || eexit "Only one instance of $PROGNAME can run at one time."

    do_A
    do_B
}
main

So now we have a lock function that does what we want. It gets the prefix of the lock filename, and then variable fd is 200 by default, but you can specify different file descriptor number like this lock $PROGNAME 500. The important change in the lock function is the turning of 200 to a variable $fd, and to be able to execute the lock file creation, we use eval. Try it without eval and see what happens. You will get error that there is no command 200.

Conclusion

We arrived from script like locking mechanism to a well closed lock function, that will create exclusivity of the program running. I would go further and paste this lock command to a library of its own. Then source it. i.e. source /usr/lib/lock.sh. Look again at the main function and see how elegant and expressive it is. i.e. lock program or exit the program, because another instance of the program is already running.

nJoy,
Kfir

In the preparations for getting my first smart phone, mainly to use it as a heart rate monitor and GPS for my workouts, I opted for buying a bluetooth heart rate chest strap. Until I’ll get new phone, I want to connect the chest strap to my laptop, for logging heart rate when exercising at home.

Searching the net for Linux HRM (heart rate monitor) implementation, I have found an interesting post Listening to a Polar Bluetooth HRM in Linux. The author wrote a Ruby script that listens to the Polar bluetooth device. So I opted to buy the chest strap he uses in the blog post - Polar WearLink+ transmitter with Bluetooth. Pay attention this is not the Polar H7 which is the newer and support Bluetooth 4.0, but have a lot of issues connecting to android devices. I paid 80$ include shipping to Israel.

After getting the Polar strap, I notice problem connecting the device. Searching Google, I found that there are issues with the battery, that drains too fast. Here is the thread about this problem. I took the battery out, and tried to replace it with other batteries, but nothing. My computer couldn’t find the device. Next day I tried again, and replaced the battery again to the original. Then by magic, the computer did found the device. I don’t know how or why this happened, but I’ll keep track of the battery life and post back.

My heart rate on the screen ;-)

In his blog post Daniel created a small program that listen to the heart rate belt, then prints the data to the screen. Here is a sample output of the program, and the column :hr=> shows my heart rate:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

{:len=>10, :chk=>245, :seq=>9, :status=>209, :hr=>74, :rr=>[793], :time=>2012-10-27 17:21:25 +0200}
{:len=>8, :chk=>247, :seq=>10, :status=>209, :hr=>74, :rr=>[811], :time=>2012-10-27 17:21:26 +0200}
{:len=>10, :chk=>245, :seq=>11, :status=>209, :hr=>73, :rr=>[845], :time=>2012-10-27 17:21:27 +0200}
{:len=>8, :chk=>247, :seq=>12, :status=>209, :hr=>73, :rr=>[816], :time=>2012-10-27 17:21:29 +0200}
{:len=>10, :chk=>245, :seq=>13, :status=>209, :hr=>73, :rr=>[810], :time=>2012-10-27 17:21:30 +0200}
{:len=>8, :chk=>247, :seq=>14, :status=>209, :hr=>74, :rr=>[800], :time=>2012-10-27 17:21:31 +0200}
{:len=>10, :chk=>245, :seq=>15, :status=>209, :hr=>74, :rr=>[779], :time=>2012-10-27 17:21:32 +0200}
{:len=>10, :chk=>245, :seq=>0, :status=>209, :hr=>75, :rr=>[776], :time=>2012-10-27 17:21:34 +0200}
{:len=>8, :chk=>247, :seq=>1, :status=>209, :hr=>76, :rr=>[769], :time=>2012-10-27 17:21:36 +0200}
{:len=>10, :chk=>245, :seq=>2, :status=>209, :hr=>75, :rr=>[833], :time=>2012-10-27 17:21:36 +0200}
{:len=>8, :chk=>247, :seq=>3, :status=>209, :hr=>74, :rr=>[837], :time=>2012-10-27 17:21:38 +0200}
{:len=>10, :chk=>245, :seq=>4, :status=>209, :hr=>75, :rr=>[800], :time=>2012-10-27 17:21:39 +0200}
{:len=>10, :chk=>245, :seq=>5, :status=>209, :hr=>75, :rr=>[783], :time=>2012-10-27 17:21:40 +0200}
{:len=>8, :chk=>247, :seq=>6, :status=>209, :hr=>75, :rr=>[814], :time=>2012-10-27 17:21:41 +0200}

Few steps to get it working

Compile kernel Bluetooth drivers

make menuconfig

and in Networking support -> Bluetooth subsystem support (NEW) I just marked everything as module. After you understand what driver you need, you can unmark the drivers that are not needed by your system. This procedure is for Gentoo users, or people that use custom kernels. For other OSs Bluetooth is usually provided with the standard kernel.

Install software

You will need to install bluez for handling the Bluetooth service:

emerge -av net-wireless/bluez

Next, if you have Broadcom BCM203x and STLC2300 Bluetooth chips, you should install their firmware:

emerge -av net-wireless/bluez-firmware 

For debugging the connection you can use hcidump similar to tcpdump:

emerge -av net-wireless/bluez-hcidump

And last, but very important for providing PIN to the Bluetooth device:

emerge -av net-wireless/blueman 

Enable Bluetooth

I have an Lenovo X201 laptop, so I enable and disable the Bluetooth with this command:

echo enable > /proc/acpi/ibm/bluetooth
echo disable > /proc/acpi/ibm/bluetooth

If you don’t have a Thikpad, this method will not work, as this command relays on thinkpad.ko kernel module. Please Google to find out how to enable Bluetooth on your setup.

Now start the bluez service:

/etc/init.d/bluetooth start

Search the Polar device

In order to search the chest strap, you will need to run:

hcitool scan

and the output should look similar to this:

Scanning ...
        00:22:D0:00:95:31       Polar iWL

The mac address is important, as you need to supply it as a parameter to polar.rb.

I found that my laptop will not find the Polar device on the first scan, so I ran the following command to scan few times:

while true ; do hcitool scan ; sleep 1 ; done

It took a while, but it finds the Polar device eventually.

Pairing laptop with Polar strap

Ok, I must admit, this was quit challenging. I didn’t install the bluman package at first, and saw that running

ruby polar.rb 00:22:D0:00:95:31

result with connection refused. Using hcidump, I saw that the Bluetooth on the chest strap is aborting the connection due to lack of PIN code. The manual states that the PIN is 0000 (4 times zero), so I searched for a way to add this to the program. It wasn’t straight forward task, so I read again Daniel’s post and noticed that he uses the blueman package. I installed it. Then ran the command:

blueman-applet

Hit the Search button and it should find the Polar device, then hit the Pair button to allow the device to request the PIN. This provided the initial pairing of the device with the laptop.

After this procedure, I was able to run the polar.rb using the command:

ruby polar.rb 00:22:D0:00:95:31

and my heart rate beats started appearing on the laptop screen.