tag:blogger.com,1999:blog-4670756400590062347.post5897519016438381877..comments2020-05-08T10:03:03.398+00:00Comments on Northgrid-tech: RFIO tuning for Atlas analysis jobsAlessandra Fortihttp://www.blogger.com/profile/11973932320387024088noreply@blogger.comBlogger6125tag:blogger.com,1999:blog-4670756400590062347.post-51433179075830454472011-05-25T08:21:28.924+00:002011-05-25T08:21:28.924+00:00Hi,
Where do you place the /etc/shift.conf file ?...Hi,<br /><br />Where do you place the /etc/shift.conf file ? On the WNs or the SEs ?<br /><br />Did you saw any modification in the transfer rate since last October ?Jerome Pansanelhttps://www.blogger.com/profile/15017006910955783009noreply@blogger.comtag:blogger.com,1999:blog-4670756400590062347.post-33527785963387229002009-02-10T12:30:00.000+00:002009-02-10T12:30:00.000+00:00Simon,The commands used are a simple athena job (1...Simon,<BR/><BR/>The commands used are a simple athena job (14.2.10) run over RFIO, on some atlas FDR08 datasets (~1.5G/file). A voms proxy is made available for GSI access and we have a local hack that forces libshift.so to point to libdpm.so.<BR/><BR/>The wall time and cpu time are recorded with 'time' on that athena job.<BR/><BR/>Throughput of data is taken from ganglia (bandwidth is steady enough to take a good estimate).<BR/><BR/>All tests are performed on otherwise idle nodes directly, not through grid/batch.John Blandhttps://www.blogger.com/profile/16051241409269392358noreply@blogger.comtag:blogger.com,1999:blog-4670756400590062347.post-49973629773732255772009-02-09T14:55:00.000+00:002009-02-09T14:55:00.000+00:00hi John,can you say more about the commands you us...hi John,<BR/>can you say more about the commands you used to make these measurements?<BR/>Thanks,<BR/>SimonSimon Georgehttps://www.blogger.com/profile/10363160113556218890noreply@blogger.comtag:blogger.com,1999:blog-4670756400590062347.post-38922138129110007012008-12-02T12:45:00.000+00:002008-12-02T12:45:00.000+00:00ps I tried a 256kB buffer out of interest, as this...ps I tried a 256kB buffer out of interest, as this should be enough for one whole event, but the result was even worse than 128kB.John Blandhttps://www.blogger.com/profile/16051241409269392358noreply@blogger.comtag:blogger.com,1999:blog-4670756400590062347.post-91548000094085096932008-12-02T12:33:00.000+00:002008-12-02T12:33:00.000+00:00I've run a quick retest at the extremes, with buff...I've run a quick retest at the extremes, with buffer sizes of 4kB, 128kB and 128MB.<BR/><BR/>Buffer, Efficiency, Throughput<BR/>4kB, 60%, 0.66GB<BR/>128kB, 46%, 13.95GB<BR/>128MB, 71%, 4.50GB<BR/><BR/>The much smaller buffer has clear gains in efficiency and bandwidth. I assume this to be that even though the buffer is clearly not big enough for a single event, each read is only for a very small amount. This does harm efficiency however as it is hitting the latency of the network/rfio interface.<BR/><BR/>The large 128MB buffer uses more bandwidth but gets even better efficiency. Note that these tests are on a file size of 1.5GB (ie 12x the buffer size).<BR/><BR/>The 4kB route should give a predictable, low bandwidth solution but leaving you at the mercy of your network latency. The 128MB route gives a less predictable (but still significant) increase to cpu efficiency at the expense of some bandwidth.<BR/><BR/>File sizes less than 128MB give a very similar bandwidth usage to the 4kB setup (as seen during atlas analysis challenge 1) with close to 100% efficiency.<BR/><BR/>My opinion is that if the bandwidth is there it might as well be sacrificed to give a gain in cpu efficiency (and hence useful analysis/day) but I agree that the base problem is the file layout. If the data was sequential none of this would matter and we could tune BUFSIZE to best match our network and disks.<BR/><BR/>As for local disk, even though data is only read once the device should be reading ahead by some amount (and so over time building up the whole file), but again as the data is random this gains nothing, particularly if the files/datasets are larger than the cache. Setting a large readahead as per the RFIO buffer might give similar gains but that could seriously affect the rest of the system performance.<BR/><BR/>All of these findings are on relatively old hardware. Lots of factors such as bus speed, network interface efficiency etc could greatly affect the results (eg the 128kB cpu efficiency is so low partly because the system is spending more system cpu time on moving data across the LAN), so this really needs testing on other sites.John Blandhttps://www.blogger.com/profile/16051241409269392358noreply@blogger.comtag:blogger.com,1999:blog-4670756400590062347.post-69453271591532591382008-12-01T21:57:00.000+00:002008-12-01T21:57:00.000+00:00Care to check with smaller buffers, going down to ...Care to check with smaller buffers, going down to 4K if possible? <BR/><BR/>The most likely reason why local IO is slower is that you never use the pagecache, you only read the data once and since your IO is more or less random you can't expect good performance. <BR/><BR/>A dd (or something similar) before the jobs starts will load the file in the pagecache but since most files aren't small you might not have enough memory to keep them there and you are back to square one.<BR/><BR/>The real problem is that the file format isn't efficient and the IO pattern is random, not entirely random for the files you tested since it seems that it's done in blocks of around 128MB but nothing really tells you that other files will be similar so you might need a much bigger buffer there :(Anonymoushttps://www.blogger.com/profile/03503328263670563368noreply@blogger.com