linux

Chinese BitTorrent DDoS Mitigation

Here lately there's been lots of talk about the Great Firewall of China DNS posioning and inadvertently (?) DDoS'ing a servers all over..

The typical requests look similar to this:

125.77.141.10 - - [17/Jan/2015:04:51:21 +0000] "GET /announce.php?info_hash=%D6%BA%3E%2B%BC%86%5D%16%EED%D7%ADSFi%A5%13b%14%82&peer_id=%2DSD0100%2D%C6%D5%87%2F%0C%C1y%3E%F6%F5%14%1E&ip=192.168.1.104&port=13423&uploaded=820039352&downloaded=820039352&left=712758596&numwant=200&key=2204&compact=1 HTTP/1.0" 403 214 "tracker.thepiratebay.org"  
211.100.51.150 - - [17/Jan/2015:04:51:21 +0000] "GET /474x/0c/3c/1a/0c3c1ab61ea01f0ed7ddcab06daf4cd9.jpg HTTP/1.1" 403 252 "media-cache-ec0.pinimg.com"  
182.38.2.198 - - [17/Jan/2015:04:51:21 +0000] "GET /announce?info_hash=%BD%F4%CC%EC%A2%21L%B7X%E9%99%CA%22%DCE%7D%C1%92%2D%C9&peer_id=%2DSD0100%2D%EB%A4%D5%40%0C%FE%E1%9571%0F%CC&ip=192.168.1.101&port=8571&uploaded=776994816&downloaded=776994816&left=1800404992&numwant=200&key=3921&compact=1 HTTP/1.0" 403 210 "a.tracker.thepiratebay.org"  
183.12.184.251 - - [17/Jan/2015:04:51:21 +0000] "GET /announce?info_hash=%D8%21Vd%B5%B8d%D4%17%D4%D7g%F6i%3B%B6%2D%E7lN&peer_id=%2DSD0100%2Da%20%91l%F2B%EF%BD%8C%D3%7B%B1&ip=192.168.0.112&port=10141&uploaded=3291119737&downloaded=3291119737&left=235912704&numwant=200&key=32232&compact=1 HTTP/1.0" 403 210 "vip.tracker.thepiratebay.org"  
223.104.5.155 - - [17/Jan/2015:04:51:21 +0000] "GET /searchbox/alba.js HTTP/1.1" 403 219 "api.popin.cc"  
183.224.92.81 - - [17/Jan/2015:04:51:21 +0000] "GET /crossdomain.xml HTTP/1.1" 403 217 "livepassdl.conviva.com"  
113.57.186.42 - - [17/Jan/2015:04:51:21 +0000] "GET /announce.php?info_hash=%7F6%E9%12%E3%13%AEk%3C%03%15I%9A%028%E1%81%88%1D%11&peer_id=%2DSD0100%2DX%23%CFW%A7%9A%FF%14%91%F7%15%3E&ip=182.83.160.218&port=28388&uploaded=883389509&downloaded=883389509&left=330836393&numwant=200&key=15941&compact=1 HTTP/1.0" 403 214 "vip.tracker.thepiratebay.org"  
122.90.95.227 - - [17/Jan/2015:04:51:21 +0000] "GET /banners/108499/livedealer_ca_sc_300x250%20short.gif HTTP/1.1" 403 251 "static.exoclick.com"  
183.252.230.114 - - [17/Jan/2015:04:51:21 +0000] "GET /announce?info_hash=z%21%1A%89%DAn%BFU2%81%23O%A5%00cf%3Fw%9BY&peer_id=%2DSD0100%2D%0DH%1F%87%EB%D7%9F%E5%10%0C%A70&ip=183.252.230.114&port=12021&uploaded=319029248&downloaded=319029248&left=524288&numwant=200&key=13387&compact=1 HTTP/1.0" 403 210 "vip.tracker.thepiratebay.org"  
106.112.224.119 - - [17/Jan/2015:04:51:21 +0000] "GET /crossdomain.xml HTTP/1.1" 403 217 "livepassdl.conviva.com"  

I whipped up the following IPtables rules to drop the BitTorrent announce traffic so it never hits Apache/nginix/etc...

iptables -I INPUT -p tcp --dport 80 -m string --to 70 --algo bm --string 'GET /announce.php?info_hash=' -j DROP  
iptables -I INPUT -p tcp --dport 80 -m string --to 70 --algo bm --string 'GET /announce?info_hash=' -j DROP  
iptables -I INPUT -m state --state INVALID -j DROP  

Ugly one-liner for gzip'ed gzip files

Had to rig up this one-liner to un-screw a situation were logrotate was configured to rotate and compress all files in a directory, including ones that had already been rotated and compressed; this sort of configuration lead to gzip'ed gzip files like this:

error_log-20141109.gz-20141111.gz-20141113.gz-20141115.gz-20141117.gz-20141119.gz-20141121.gz-20141123.gz-20141125.gz-20141127.gz-20141129.gz-20141201.gz-20141203.gz-20141205.gz-20141207.gz-20141209.gz-20141211.gz-20141213.gz-20141215.gz  

The one-liner I put together to loop through and gunzip the many gzip'ed layers is:

for i in $(ls *.gz*); do loops=$(echo $i | awk 'END{print _}{_+=NF-3}' FS=".gz"); filename=$(echo $i | cut -d '-' -f 1,2); zcat $i > $filename; for a in $(seq $loops); do gunzip -c $filename > $filename-tmp; mv -f $filename-tmp $filename; done; rm -f $i; done  

Very ugly but you do end up with all the files in your directory named and working the way they should.

Converting VMAX NAA Values

Quick bash script I put together to convert VMAX NAAs into the values seen within PowerPath on Linux hosts. There's a PowerShell script floating around the internet to do the same thing but nothing in bash.

#!/bin/bash
# (C) 2014 Derek Morton
# Licensed under GPL 2.0

naa=$1  
naalen=${#naa}

if [ $naalen == 32 ]; then  
        device1=${naa:24:2}
        device2=${naa:26:2}
        device3=${naa:28:2}
        device4=${naa:30:2}
        echo "Frame/Symmetrix ID: ${naa:16:4}/${naa:8:12}"
        echo -e "Logical Device: \x$device1\x$device2\x$device3\x$device4"

elif [ $naalen == 36 ]; then  
        device1=${naa:28:2}
        device2=${naa:30:2}
        device3=${naa:32:2}
        device4=${naa:34:2}
        echo "Frame/Symmetrix ID: ${naa:20:4}/${naa:12:12}"
        echo -e "Logical Device: \x$device1\x$device2\x$device3\x$device4"

else  
        echo "NAA Value must be 32 or 36 characters"
        echo "ex: 6090a038f0cd4e5bdaa8248e6856d4fe"
        echo "or: naa.6090a038f0cd4e5bdaa8248e6856d4fe"
fi  

Sample output:

$ ./naa_to_powerpath.sh 60000970000295700173533034373635
Frame/Symmetrix ID: 0173/000295700173  
Logical Device: 4765

# powermt display dev=emcpowera
Pseudo name=emcpowera  
Symmetrix ID=000295700173  
Logical device ID=4765  
state=alive; policy=SymmOpt; queued-IOs=0  

HBA/bfa bio.c panic on boot

Came across a server with dual Broadcom/QLogic 815 HBAs that was failing to boot with the following error and dropping into debug mode:

LOG Firmware heartbeat failure at 0
LOG bfa panic bio.c:1233: 0

hba-boot-error

Entering the HBA BIOS with CTRL+B or ALT+B was also failing with the same error; I was able to get the server to boot by holding down the 'x' key when the HBA BIOS appeared.

Once actually SSH'ed into the server I discovered that the HBAs had different versions of the HBA Boot Code/Option ROM (3.0.1.0 vs. 3.0.3.1) which caused the error message on boot:

[root@db1 ~]# bcu adapter --query 1
Adapter Information:  
    model info:        Brocade-815
    OEM info:          N/A
    num ports:         1
    hw path:           0000:06
    Serial Num:        xxxxxxxxxx
    name:
PCI Information:  
    vendor id:         0x1657
    device id:         0x0017
    ssvid:             0x1657
    PCIe Gen:          Gen2
    PCIe lanes:        8(Initial number of lanes = 8)
    PCI function0:
            ssid:      0x0014
            port:      0
            type:      FC
Port Information:  
    Port 0:
            name:
            pwwn:      xxxxxxxxxx
            nwwn:      xxxxxxxxxx
            hwpath:    0000:06:00.0
Flash Information:  
    status:            good
    option ROM version:
            current:   3.0.1.0
            flashed:   3.0.1.0
    fw version:        3.2.1.0
[root@db1 ~]# bcu adapter --query 2
Adapter Information:  
    model info:        Brocade-815
    OEM info:          N/A
    num ports:         1
    hw path:           0000:07
    Serial Num:        xxxxxxxxxx
    name:
PCI Information:  
    vendor id:         0x1657
    device id:         0x0017
    ssvid:             0x1657
    PCIe Gen:          Gen2
    PCIe lanes:        8(Initial number of lanes = 8)
    PCI function0:
            ssid:      0x0014
            port:      0
            type:      FC
Port Information:  
    Port 0:
            name:
            pwwn:      xxxxxxxxxx
            nwwn:      xxxxxxxxxx
            hwpath:    0000:07:00.0
Flash Information:  
    status:            good
    option ROM version:
            current:   3.0.3.1
            flashed:   3.0.3.1
    fw version:        3.2.1.0

Once the HBA boot code was updated to the current versions on both cards the server is now booting properly again.

raxmon - Malformed Response

From Github Issues Page

When using the latest version of raxmon (a.k.a. Rackspace Monitoring CLI) you might get the following error:

$ raxmon-entities-list
Traceback (most recent call last):  
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/raxmon-entities-list", line 23, in <module>
    run_action(None, None, 'entities', 'list', callback)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/raxmon_cli/common.py", line 119, in run_action
    instance = get_instance(username, api_key, api_url, auth_url)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/raxmon_cli/common.py", line 136, in get_instance
    instance = driver(username, api_key, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/rackspace_monitoring/drivers/rackspace.py", line 173, in __init__
    self._initialize_connection_base_url(constructor_kwargs=kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/rackspace_monitoring/drivers/rackspace.py", line 187, in _initialize_connection_base_url
    self.connection._populate_hosts_and_request_paths()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/libcloud/common/openstack.py", line 602, in _populate_hosts_and_request_paths
    osa.authenticate()  # may throw InvalidCreds
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/libcloud/common/openstack.py", line 155, in authenticate
    return self.authenticate_2_0_with_apikey()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/libcloud/common/openstack.py", line 240, in authenticate_2_0_with_apikey
    return self.authenticate_2_0_with_body(reqbody)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/libcloud/common/openstack.py", line 264, in authenticate_2_0_with_body
    driver=self.driver)
libcloud.common.types.MalformedResponseError: <MalformedResponseException in None 'Malformed response'>: 'code: 405 body: {"methodNotAllowed":{"code":405}}'  

The simple fix is to add '/tokens' to your auth_api section in your .raxrc file or on your command arguments:

[auth_api]
url=https://identity.api.rackspacecloud.com/v2.0/tokens  

apachectl Alternate Location

Quick snippet for when Apache isn't listening on 127.0.0.1 or on an alternative port.

On RHEL/CentOS add the following to /etc/sysconfig/httpd:

STATUSURL=http://<IP>:<PORT>/server-status  

On Ubuntu add to /etc/apache2/envars:

STATUSURL=http://<IP>:<PORT>/server-status