Sheffield in downtime

Sheffield has been put in downtime until Monday 12/11/2007 at 5 pm.


Reason: Power cut affecting much of central sheffield. Substation exploded. Not even allowed inside the physics building.

Matt is also back in the GOCDB now as site admin.

Sheffield accounting

From Matt:

/opt/glite/bin/apel-pbs-log-parser
is trying o contact the ce on 2170, I think expecting the site bdii to be there.
I changed ce_node</GII> to mon_node in /opt/glite/etc/glite-apel-pbs/parser-config-yaim.xml
and now thing seem much improved.

However, I am getting this

Fri Nov 2 13:48:39 UTC 2007: apel-publisher - Record/s found: 8539
Fri Nov 2 13:48:39 UTC 2007: apel-publisher - Checking Archiver is Online
Fri Nov 2 13:49:40 UTC 2007: apel-publisher - Unable to retrieve any response while querying the GOC
Fri Nov 2 13:49:40 UTC 2007: apel-publisher - Archiver Not Responding: Please inform apel-support@listserv.cclrc.ac.uk
Fri Nov 2 13:49:40 UTC 2007: apel-publisher - WARNING - Received a 'null' result set while querying the 'LcgRecords' table using rgma, this probably means the GOC is currently off-line, will therefore cancel attempt to re-publish

running /opt/glite/bin/apel-publisher on the mon box.

I the goc machine is really off-line, I'll have to wait to publish the missing data for sheffield.

Sheffield

Quiet night for sheffield after reimaged nodes where taken offline in PBS. Matt also increased the number of ssh connections allowed on the CE from 10 to 100 to reduce the time outs between the WN and CE and reduce the incidence of Maradona errors.

Sheffield latest

Trying to stabilize Sheffield cluster.
After the scheduled power outage the nodes didn't restart properly and some of the old jobs needed to be cleaned up. After that the cluster was ok apart from the BDII dropping out. We have applied the famous Kostas patch

https://savannah.cern.ch/bugs/?16625

which is getting into the release after 1.5 years Hurray!!!

The stability of the BDII has improved and DPM seems stable. The SAM tests have been stable over the weekend and the today Steve Atlas tests showed a 96% availability which is a big improvement. However the cluster filled up this morning and the instability reappeared, sign that there is still something to fix on the worker nodes and in the scheduling. Added a reservation for ops and looking at the WNs some of which were re-imaged this morning.

Its alive !

Sheffield problems reviewed

During the last update to the DPM I ran in to several problem.

1. DPM update failed due to changes in the way the password are
stored in mysql
2. A miss understandind with the new version of yaim that rolled out
at the same time
3. Config errors with the sBDii
4. mds-vo-name
5. Too many roll outs in one go for me to have a clue which broke and where to start
looking.

DPM update fails
I would like to thank Graeme for the great update instructions, they
helped lots. The problems came when the update script used a
different hashing method to the one used by mysql problem found here
http://<>. This took some finding, it also means every
time we run yaim config on the SE we have to go back and fixs the
passwords again, this is because yain still uses the old hash not the
new one.

Yaim update half way and congig errors
This confused the hell out of me one minute I'm using yaim scripts to
run updates. Next I have an updated version of yaim that I had to
pass flags to and is where I guess I started to make the mistakes that
lead to me setting the SE as a sBDii. After getting lost with the new
yain I told the wrong machine that it was a sBDii and never relised.

mds-vo-name
With the help of Henry, we found out that our information was wrong ie we had

mds-vo-name=local it is now mds-vo-name=resource

Once this was changed in the site-info.def and yaim was re ran on our mon box which is
also out sBDii it alll seamed to work.

Sheffield

Hi all

Sorry it been so quiet on the Sheffield front. I've been out of the country, and it currently registration here.

Whats the state of the LCG here. I feel like I'm chasing my tail hence there will shortly be a back of email asking for help from TB Support. I have notice there have been several updates while I've been away so I will add them before getting back to the main problems of why our SE seam to not have an entry in the BDii

Week 33 start of 34

Main goings on have been the DMP update this was hampered by password version problems in MySQL, was resolved with help from here http://www.digitalpeer.com/id/mysql . More problem came with the change to BDii new firewall ports open and yet there was still no data coming out.

The BDii was going to be fixed today, however Sheffield has suffered several power cut over the last 24 hours. This has affected the hole of the LCG here, recovery work is ongoing.

Sheffield Jully looking back

July had to point of outage the worst being at the start of the month and just after a gLite 3.0 upgrade it did take a bit of time to find the problem and the solution.

Error message: /opt/glue/schema/ldap/Glue-CORE.schema: No such file or directory
ldap_bind: Can't contact LDAP server
Solution was found here: http://wiki.grid.cyfronet.pl/Pre-production/CYF-PPS-gLite3.0.2-UPDATE33

At the end of the month we has a strange error that was spotted quickly and turn out to be the result of a DNS server crash on the LGC here at Sheffield not resolving the worker nodes IPs

Sheffield hosted the monthly North grid meeting and all in all it was a good event.

Yesterday the LCG got it's own dedicated 1gig link to YHman and beyond we also now have our own firewall which will make changes quicker and easier.

Sheffield week 28

Now begins the trying to unpick what wrong here at Sheffield.
So far main problem seams to be with the CE

On the plus the DPM patching went well

sheffield week 27

Arrrr it all gone wrong things very broke and not enough time to fix them.

Power outages

Sheffield week26

Wet wet and did I mention it rained here.

Not much to report to do with the cluster it is still up and running although we started failing lloyds tests yesterday afternoon. I will look into this when I get time

The University's power is in a state of "At Risk" until midday Friday. As a result Sheffield might go off line with out warning.

Sheffield week 24

I think I'm slowly getting my head round all this now {but don't test me ;)}

Technicaly there is not much new to report some down workers have had new disks put in them. Plans are bing made to upgrade the worker and to finish sorting out Andy's legacy.

Main problem the building where the machine room is housed is a no access building site and I have been warned about a power outage in July.

Sheffield Update

I go away for a long weekend and we start failing SAM tests again. After a few email from Greig and some time waiting for the next tests we are now passing all the tests.

Our failings over the past few weeks seam to be down to one of 2 things cert upgrades not automatically working on all machines and me not knowing when and how to change the DN.

We have fixed the gridice information about disk sizes on the SE, as well as looking into adding more pools.

back to my day job