Tag Archives: linux

Headless RaspPi: Tell Me Where You Are

I like several of the little SBC (Single Board Computer) systems that are available, despite having a love/hate relationship with my favourite – the ODROID C2. For the minute I’m stuck with just a Raspberry Pi 3 as a little SBC, so needs must when the devil, etc etc…

Anyway, when you control your own network, as I used to, it doesn’t really matter what IP DHCP decides to assign to each system. I could look it up on the router easily enough, or even fix an IP address based on MAC address.

OK, fine… but what if you don’t control your router?

Well, you could brute-force scan the entire known IP range of the network using NMAP… (nmap -sS xxx.yyy.zzz.0/24) but that seems a little extreme. It works in a pinch, though, and might be the only way of finding it that all important first time.

Once you know where it is, however, how can you keep it? On a network you don’t control, you can get jumped around the IP range depending on how the router is feeling and how many others are connected. Not very useful.

Here’s a little script modified from another one that didn’t work for me… (requires Python 2, not Python 3…)

Ah, and I found where the mods come from…!

This will send you an e-mail telling you your headless systems’ IP when the RPi boots.

Just add this to a file with vi bootupEmail.py, then chmod +x it:

import subprocess
import smtplib
import socket
from email.mime.text import MIMEText
import datetime
# Change to your own account information
to = 'me@example.com'
gmail_user = 'test@gmail.com'
gmail_password = 'yourpassword'
smtpserver = smtplib.SMTP('smtp.gmail.com', 587)
smtpserver.ehlo()
smtpserver.starttls()
smtpserver.ehlo
smtpserver.login(gmail_user, gmail_password)
today = datetime.date.today()
# Very Linux Specific
arg='ip route list'
p=subprocess.Popen(arg,shell=True,stdout=subprocess.PIPE)
data = p.communicate()
split_data = data[0].split()
ipaddr = split_data[split_data.index('src')+1]
my_ip = 'Your ip is %s' %  ipaddr
msg = MIMEText(my_ip)
msg['Subject'] = 'IP For RaspberryPi on %s' % today.strftime('%b %d %Y')
msg['From'] = gmail_user
msg['To'] = to
smtpserver.sendmail(gmail_user, [to], msg.as_string())
smtpserver.quit()

You’ll also need to add python /home/user/bin/scripts/bootupEmail.py to your rc.local, just before the exit 0 at the end.

IMPORTANT: Obviously, since you just put your Gmail password in a plain text file (really, really bad security practice!) just set up a gmail account to only do this one thing. Or, if you want, you could set up a mailserver on your Pi.

That may become the topic of a future post. In fact, it probably will, because I can’t believe I just suggested putting a password in plaintext.

Linux Ping Script… (Finally…)

Anyone reading this may remember I was having intermittent – but regular and extremely annoying – network dropouts in my accommodation.

Well, I finally got twenty minutes (and a network switch so I can have more than one thing plugged in to the LAN at once) to get my act together and update my little Raspberry Pi 3 so it can sit and (low power) ping a server and let me know when the network drops out.

Just like the Windows version, this only writes to the log file when it can’t connect – so if your network never drops out, you’ll never even see the log file get created. You can test it by changing the sleep time and what it greps for (if you change sleep to 1 and grep unknown to grep PING, you can check it works…)

Just save it somewhere sensible, like /home/user/bin/scripts or similar, chmod +x it, and run it from the terminal. Force it to the background and you can log out and leave it ticking over.

#!/bin/bash
while true; do
     x=`ping -c1 www.google.com 2>&1 | grep unknown`
     if [ ! "$x" = "" ]; then
         date >> brokentimes-pingtest.txt
     fi
sleep 30
done

ORCA & OpenMPI: File Descriptor Limits

I’ve just been tinkering around with an interesting issue: ORCA, the computational chemistry program I’ve been using because I can’t afford Gaussian, crashes during geometry optimisation of a (moderately) complex molecule because of OpenMPI.

OpenMPI is complaining about running out of file descriptors. Eh? Seriously? OK…

Turns out that Ubuntu 16.04 (even the server version) sets the open file limit at what is frankly a little on the low side – 1024 open files. That sounds like a lot, until you think that when running something via MPI it can be crunching across a lot of temporary files and so on… and it suddenly doesn’t seem so many. Interestingly, I never had this problem before because I was running Ubuntu 14.04 previously which (from what the internet says) had a limit of 4096. I checked with the latest release (14.04.5) which had a limit of 1024, so I’ll assume for now that the 4096 limit was in an older release…

I’ll be honest, since this is the first time I’ve encountered this issue, I’ve never actually checked previously…

Anyway, there appear to be two fixes that work, one on a per-user basis, one on a system-wide basis. Pick your poison.

The user-level fix is super easy, add the following to your .bashrc:

if [ $USER = "paramagnetic" ]; then
ulimit -n [pick-a-big-number, eg; 32768]
fi

Can also be added to /etc/profile

At the system level, it’s a little more difficult, but still totally doable. Edit /etc/security/limits.conf

* soft nofile 32768
* hard nofile 32768
root soft nofile32768
root hard nofile 32768

Then add:

session required pam_limits.so

To /etc/pam.d/common-sessions and /etc/pam.d/common-sessions-noninteractive

And reboot in all cases. If you have another form of system management software running, it might overrule you, which is annoying, but outside of the scope of this post.

NWChem: Headaches

Been struggling to get NWChem to compile today. I think I’ve nearly got it, but I’m calling it a day for now because otherwise I’ll still be working on it at 0500…

For the record, it compiles fine if I don’t want CUDA or OpenBLAS (the internal BLAS libraries are horribly slow, by the developers’ own admission) but it’s getting CUDA and a faster MKL in place that is causing me grief.

Basically, I’m hoping that it can provide me some speedup over a CPU computational chemistry package – I’ve had the chance to test TeraChem, which I actually think is awesome… it’s just that it doesn’t like Pascal generation GPUs, so it’s fine on my laptop with a 980m (well, if you call the GPU sitting at frighteningly high temperatures “fine”) but doesn’t want to know about the twin GTX1080’s I’ve got in a sort-of-but-not-workstation.

So, anyway, as soon as I’ve figured out what compile flags I need to get it working properly, I’ll update this post.

How To: Intel GPU for monitor, nVidia GPUs for CUDA

There are lots of methods for getting this working spread over the internet. Some work, but are a mess (and involve faking monitor connections to the nVidia GPUs using some Xorg trickery) or just flat out don’t work.

This method works great, provided you don’t want fan control over the GPUs. So just get some decent case fans and you’re golden.

I used the nVidia Ubuntu PPA for drivers; you can use the binary blob .run file, and frankly it’ll probably save you headaches in the long run (like, for example, when the drivers update and freakin’ break everything! /rant)

First, I got a more up-to-date kernel on Ubuntu 16.04…

apt-get install linux-headers-4.10 linux-headers-4.10-generic linux-image-4.10-generic linux-image-extras-4.10-generic

You should have dkms installed, but if not, pull it in with:

apt-get install dkms

reboot

Add nVidia drivers PPA:

add-apt-repository ppa:graphics-drivers/ppa

apt-get update

apt-get install nvidia-375

apt-get install nvidia-settings nvidia-prime

reboot

at this point, the graphics may work, or they may not. If they do, great. if they don:t, well, it:s easy to fix by reinstalling nvidia-settings and nvidia-prime.

Set the focus GPU to intel via the system tray nvidia settings app. log out, log back in again. 3d accel should still work.

Download the CUDA repo .deb (I used network, but local works too) and install using dpkg. there should not be any dependencies unmet or conflicting.

dpkg -i [cudarepofile.deb]

apt-get update

apt-get install cuda

This will pull in CUDA8.0. Again, like I said, you can always use the local .run file, which will save you updating headaches.

Now, at this point, CUDA wants to reinstall the gpu driver. I let it. It’ll break 3d acceleration, but reinstall nvidia-settings and nvidia-prime to fix it again.

At this point, nvidia-smi (the nvidia system management interface) will stop working. This is because it can’t cope with the idea of you not using the nVidia GPUs for Xorg. Dumb, given the drive nVidia has been putting into GPGPU, but nevertheless true.

To get it working again, do the following; you can either remove or rename the link in /usr/bin. I rename, others might remove:

mv /usr/bin/nvidia-smi /usr/bin/nvidia-smi.backup

then, place the following script in a file called “nvidia-smi”

vi /usr/bin/nvidia-smi

#!/bin/bash
LD_PRELOAD=/usr/lib/nvidia-375/libnvidia-ml.so /etc/alternatives/x86_64-linux-gnu_nvidia_smi "$@"

OK. Now, you need to change the /nvidia-375/ bit to whatever driver version you are using. If it’s 375, awesome. If it’s 378, you’ll get an error until you change it.

The “$@” bit is necessary as nvidia-smi for the correct parsing of the command; without it, it’ll throw a nasty error.

So, nvidia-smi works, but some things will still throw up wobblers. To fix this, add;

# CUDA stuff
export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib:$LD_LIBRARY_PATH

# Fixing Intel/nVidia conflict (nVidia for compute, Intel for display)
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/lib/nvidia-375:$LD_LIBRARY_PATH

To your .bashrc

To get persistence working (that is, power state management, which will allow the card to clock down when it’s not being used) edit the following:

vi /etc/rc.local

and add

/usr/bin/nvidia-smi -i 0,1 -pm ENABLED

above the “exit 0”

This tells nvidia-smi to select both GPUs and enable persistence mode so that GPUs actually freakin’ work properly. This has the side effect of making the nvidia-smi command instant again (it wasn’t when switching to Intel iGPU). Obviously, if you’ve only got a single GPU, you just tell it -i 0… if you’ve got three or four, you can add accordingly. The GPU count is always from 0.

Whenever you have a driver update occur, it’ll replace the nvidia-smi script with the actual program again. Let it. CUDA will break when you reboot (so long as drivers don’t update, it’ll survive any kernel updates without breaking) and it’s easy enough to remove the symlink in /usr/bin/nvidia-smi and replace with that little script again.

I’m using this to drive a 4K@60Hz monitor via the iGPU while using a pair of GTX1080’s for compute.

The only thing I can’t get working is fan control. I’d rather have my GPUs loud, than I would have them cooking. But I compensate with some fairly chunky intake fans pointing straight across the GPUs.

I’ve seen people (some fairly high-ranking Professors, as well) suggest watercooling. Well, yes, that is a possible solution. It’s just a shame that I’m a little shy about watercooling 24/7 systems that are largely unattended, because… I’ve had two watercooling pumps, of different types, die on me recently. I was fortunate that one system was idle when it happened, but it still reached “idle” at 66 degrees C. That was with full-copper (read: expensive) waterblocks, and CPU only. I don’t doubt that GPUs would be dead if that had happened to them.

So, yeah, I’m a little leery of watercooling systems for high compute loads unless backed by manufacturer warranty. Haha.

nVidia GPUs: Fixing Power State Woes in Linux

This one is easy. I mean, embarrassingly easy.

A little background, however; in Linux, nVidia GPUs don’t appear to clock down properly, even when idle and not connected to a monitor. This is frustrating for several reasons:

  1. P0 (full power) state when ‘idle’ wastes power, which wastes money
  2. The cards idle hot (my GTX 1080’s, without this tweak, idle around 60°C, with tweak, idle around 35-40°C)
  3. The P0 state downshifts to P2 (approximately a 20% performance drop!) when under compute load (which is stupid, but apparently to protect consumer grade cards under sustained load)
  4. I expect my computers to do what I tell them

Fortunately, nvidia-smi, the system management interface, can come to your rescue, without needing Xorg or nvidia-settings or a GUI.

Pick your favourite text editor (VI(m), Emacs, nano, pico, Xed… Leafpad (*shudder*) and point it, with root privileges, to :

/etc/rc.local

Then add:

/usr/bin/nvidia-smi -i 0,1 -pm ENABLED

To the file, just above exit 0, then save and quit. Log out and back in, and your GPUs will have persistence enabled, which will make sure they clock down appropriately.

The -i flag indicates which GPUs you want to instruct with this command, and the -pm flag just sets persistence mode to enabled.

There’s another trick to get the cards to sit at P0 state when in compute mode, but I’ll admit I’ve not tried it because mine already get plenty warm enough.

UEFI & Linux

I’ll do a longer post on this another time, going into the details of how to persuade Windows 10 Pro and Ubuntu 16.04/Mint 18(.1) (or other distro of choice that supports SecureBoot) to play nicely together with SecureBoot enabled and third party drivers so CUDA can actually… y’know, work.

But for right now, a quick PSA (that’s Public Service Announcement, not psa as in the gene family…):

I’ve not had Ubuntu get it quite right yet with the USB stick I used to install it from; it wants to format the USB stick, which naturally enough doesn’t work.

For the purposes of this, I’m going to assume you want to run an SSD (solid state drive) as you main system drive with a larger spinning rust HDD for bulk data storage. I’ve done various setups over the years, but this one seems to work the best.

Partitioning must be set manually (I’ll fire up Virtualbox and take some screenshots to demonstrate later) to the following…

EFI System Partition (512MB) Beginning of SSD
Swap Partition (16GB) End of SSD
/ (that’s ‘root’ and holds everything else) the rest of the SSD
/data all the HDD

You also need to set the installation target of the GRUB bootloader to /dev/sda1 (or whatever partition the EFI System Partition is) (not the MBR!) or the system won’t boot with EFI/SecureBoot enabled.