ENABLE_PERSISTENT_CONFIG = TRUE PERSISTENT_CONFIG_DIR = /etc/condor/ral STARTD_ATTRS = $(STARTD_ATTRS) StartJobs, RalNodeOnline STARTD.SETTABLE_ATTRS_ADMINISTRATOR = StartJobs StartJobs = False RalNodeOnline = FalseI use the prefix "Ral" here because I inherited some of this material from Andrew Lahiffe at RAL! Basically, it's just to de-conflict names. I should have used "Liv" right from the start, but I'm not changing it now. Anyway, the first section says to keep a persistent record of configuration settings; it adds new configuration settings called "StartJobs" and “RalNodeOnline”; it's sets them initially to False; and it makes the START configuration setting dependant upon them both being set. Note: the START setting is very important because the node won't start jobs unless it is True. I also need this. It tells the system (startd) to run a cron script every three minutes.
STARTD_CRON_JOBLIST=TESTNODE STARTD_CRON_TESTNODE_EXECUTABLE=/usr/libexec/condor/scripts/testnodeWrapper.sh STARTD_CRON_TESTNODE_PERIOD=300s # Make sure values get over STARTD_CRON_AUTOPUBLISH = If_ChangedThe testnodeWrapper.sh script looks like this:
#!/bin/bash
MESSAGE=OK
/usr/libexec/condor/scripts/testnode.sh > /dev/null 2>&1
STATUS=$?
if [ $STATUS != 0 ]; then
MESSAGE=`grep ^[A-Z0-9_][A-Z0-9_]*=$STATUS\$ /usr/libexec/condor/scripts/testnode.sh | head -n 1 | sed -e "s/=.*//"`
if [[ -z "$MESSAGE" ]]; then
MESSAGE=ERROR
fi
fi
if [[ $MESSAGE =~ ^OK$ ]] ; then
echo "RalNodeOnline = True"
else
echo "RalNodeOnline = False"
fi
echo "RalNodeOnlineMessage = $MESSAGE"
echo `date`, message $MESSAGE >> /tmp/testnode.status
exit 0
This just wraps an existing script which I reuse from out TORQUE/MAUI cluster. The existing script just returns a non-zero code if any error happens. To add a bit of extra info, I also lookup the meaning of the code. The important thing to notice is that it echoes out a line to set the RalNodeOnline setting to false. This is then used in the setting of START. Note: on TORQUE/MAUI, the script ran as “root”; here it runs as “condor”. I had to use sudo for some of the sections which (e.g.) check disks etc. because condor could not get smartctl settings etc.
Right, so I think that's it. When a node fails the test, START goes to False and the node won't run more jobs.
Oh, there's another thing to say. I use two settings to control START. As well as RalNodeOnline, I have the StartJobs setting. I can control this independently, so I can turn a node offline whether or not it has an error. This is useful for stopping the node to (say) rebuild it. It's done on the server, like this.
condor_config_val -verbose -name r21-n01 -startd -set "StartJobs = false" condor_reconfig r21-n01 condor_reconfig -daemon startd r21-n01
# crontab -l | grep spacemonc.py 50 18 * * * /root/bin/spacemonc.pyBecause it's going to be an (almost) single threaded server, I use puppet to make it run at a random time on each system (I say "almost" because it actually uses method level locking to hold each thread in a sleep state, so it's actually a queueing server, I think; it won't drop simultaneous incoming connections, but it's unwise to allow too many of them to occur at once.)
cron { "spacemonc":
#ensure => absent,
command => "/root/bin/spacemonc.py",
user => root,
hour => fqdn_rand(24),
minute => fqdn_rand(60),
}
And it's pretty small:
/usr/bin/python
import xmlrpclib
import os
import subprocess
from socket import gethostname
proc = subprocess.Popen(["df | perl -p00e 's/\n\s//g' | grep -v ^cvmfs | grep -v hepraid[0-9][0-9]*_[0-9]"], stdout=subprocess.PIPE, shell=True)
(dfReport, err) = proc.communicate()
s = xmlrpclib.ServerProxy('http://SOMESERVEROROTHER.COM.ph.liv.ac.uk:8000')
status = s.post_report(gethostname(),dfReport)
if (status != 1):
print("Client failed");
The strange piece of perl in the middle is to stop a bad habit in df of breaking lines that have long fields (I hate that; ldapsearch and qstat also do it.) I don't want to know about cvmfs partitions, nor raid storage mounts.
#!/usr/local/bin/python2.4
import sys
from SimpleXMLRPCServer import SimpleXMLRPCServer
from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler
import time
import smtplib
import logging
if (len(sys.argv) == 2):
limit = int(sys.argv[1])
else:
limit = 90
# Maybe put logging in some time
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s %(levelname)s %(message)s',
filename="/var/log/spacemon/log",
filemode='a')
# Email details
smtpserver = 'hep.ph.liv.ac.uk'
recipients = ['sjones@hep.ph.liv.ac.uk','sjones@hep.ph.liv.ac.uk']
sender = 'root@SOMESERVEROROTHER.COM.ph.liv.ac.uk'
msgheader = "From: root@SOMESERVEROROTHER.COM.ph.liv.ac.uk\r\nTo: YOURNAME@hep.ph.liv.ac.uk\r\nSubject: spacemon report\r\n\r\n"
# Test the server started
session = smtplib.SMTP(smtpserver)
smtpresult = session.sendmail(sender, recipients, msgheader + "spacemond server started\n")
session.quit()
# Restrict to a particular path.
class RequestHandler(SimpleXMLRPCRequestHandler):
rpc_paths = ('/RPC2',)
# Create server
server = SimpleXMLRPCServer(("SOMESERVEROROTHER.COM", 8000), requestHandler=RequestHandler)
server.logRequests = 0
server.register_introspection_functions()
# Class with a method to process incoming reports
class SpaceMon:
def post_report(address,hostname,report):
full_messages = []
full_messages[:] = [] # Always empty it
lines = report.split('\n')
for l in lines[1:]:
fields = l.split()
if (len(fields) >= 5):
fs = fields[0]
pc = fields[4][:-1]
ipc = int(pc)
if (ipc >= limit ):
full_messages.append("File system " + fs + " on " + hostname + " is getting full at " + pc + " percent.\n")
if (len(full_messages) > 0):
session = smtplib.SMTP(smtpserver)
smtpresult = session.sendmail(sender, recipients, msgheader + ("").join(full_messages))
session.quit()
logging.info(("").join(full_messages))
else:
logging.info("Happy state for " + hostname )
return 1
# Register and serve
server.register_instance(SpaceMon())
server.serve_forever()
And now I get an email if any of my OS partitions is getting too full.
It's surpising how small server software can be when you use a framework like XMLRPC. In the old days, I would have needed 200 lines of parsing code and case statements. Goodbye to all that.
 | |
| Timeline for Cern Voms Server Changes |
| VO | Vomses Port | Old Server | Is admin? | New Server | IsAdmin? |
|---|---|---|---|---|---|
| atlas | 15001 | lcg-voms.cern.ch | No | lcg-voms2.cern.ch | Yes |
| atlas | 15001 | voms.cern.ch | Yes | voms2.cern.ch | Yes |
| alice | 15000 | lcg-voms.cern.ch | No | lcg-voms2.cern.ch | Yes |
| alice | 15000 | voms.cern.ch | Yes | voms2.cern.ch | Yes |
| cms | 15002 | lcg-voms.cern.ch | No | lcg-voms2.cern.ch | Yes |
| cms | 15002 | voms.cern.ch | Yes | voms2.cern.ch | Yes |
| lhcb | 15003 | lcg-voms.cern.ch | No | lcg-voms2.cern.ch | Yes |
| lhcb | 15003 | voms.cern.ch | Yes | voms2.cern.ch | Yes |
| ops | 15009 | lcg-voms.cern.ch | No | lcg-voms2.cern.ch | Yes |
| ops | 15009 | voms.cern.ch | Yes | voms2.cern.ch | Yes |
http://wiki.nikhef.nl/grid/Argus_Global_Banning_Setup_Overview
To avoid killing everything but still putting a barrier against the
memory leaks what I'm going to do in Manchester is to limit for mem to 4GB and a limit for vmem to 5GB.  |
| New Cisco blades, engines and power supplies |
 |
| DELLs boxes among which new switches |
 |
| Aerial view of the old cabling |
 |
| Frontal view of the mess |
 |
| Cables unplugged from the cisco |
 |
| Cisco old blades with services racks still connected |
 |
| New cat6a cisco cabling aerial view nice and tidy |
 |
| Frontal view of the new cisco blades and cabling nice and tidy |
 |
| Old and new rack switches front view |
 |
| Old and new rack switches rear view |
 |
| Emptying and reorganising the racks |
 |
| Empty racks ready to be filled with new machines |
 |
| Old DELLs cemetery |
 |
| Old cables cemetery. All the cat5e cables going under the floor from the racks to the cisco half of the cables from the rack switches to the machines and all the patch cables in front of the cisco shown above have gone. |
 | ||||||||
| All the racks but two have now the new switches but the machines are still connected with cat5e cables. Upgrading the network cards will be done in Phase two one rack at the time to minimize service disruption. |
