<div dir="ltr"><div>I'm honestly not sure David, I'm a bit confused as to if NUMA is supposed to be something implemented at a die level or at a system level. If at a die level, This box is running 2x Xeon E5-2650. I've never gotten involved in the architecture side of things so this is a bit foreign to me. <br>
<br>The bulk of the computation for a run is done on the GPUs, if it all ran on the CPU a single run would take months. The GPUs have cut the runtime way down, but only part of the process is GPU accelerated, and this file creation phase is one of the parts that is not and is still being processed by the CPU. <span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:20px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);display:inline!important;float:none"><br>
</span></div><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:20px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);display:inline!important;float:none"></span></div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Dec 12, 2013 at 3:40 PM, David <span dir="ltr"><<a href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Sometimes, the use of DMA with the newest SATA III controllers
actually slows it down. Only a live test will show which is faster.<br>
Good point about the random access versus linear write, though. My
suspicion from his overview is that it is linear.<br>
<br>
Linux is pretty good at managing disk optimization but if tweaking
is necessary, it can get very problematic because as you shift the
dynamics, the OS file system handler shifts it's management
algorithm. I believe that came from the Cray world.<br>
<br>
Stephan, does your computer use the NUMA architecture? There is a
newer, slightly faster design but I can't remember what it is. The
reason I bring that up is that in a former life I had to deal with
data the size you mentioned. As a test, I ran some benchmarks
several times with the best PC available at the time against a
smallish SGI pizza box. The program used was one I wrote and had in
production for quite a while. The PC would massage the data and
finish in about 12 1/2 hours. The SGI box did it in 2 hours 45. I
ran that test several times using data that changed somewhat each
month and timing results were consistent. Just something to think
about if run-times are killing you at the moment.<div><div class="h5"><br>
<br>
<br>
<div>Arthur wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Here's something else to think about. Is the
program writing out the data in sequential chuncks, or is it
writing to random parts of the file?
<div><br>
</div>
<div>With buffered writes the best speed up you can get with a
raid 0 array is if one disk is writing something while the
other disk is going to the place where the next thing is going
to be written. If you're dealing with a bunch of random
writes, then ponying up for a few SSDs or refactoring the code
might be worth it.</div>
<div><br>
</div>
<div>Assuming that your raid controller isn't the bottleneck.
Some motherboard based raid controllers use the CPU to do the
work, and can cause everything to slow down. (Side note.
Does anyone know if DMA works with those kind of
controllers?)</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Thu, Dec 12, 2013 at 11:26 AM,
Stephan Henning <span dir="ltr"><<a href="mailto:shenning@gmail.com" target="_blank">shenning@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>The program will write out a file of variable
size, it's based on the problem being run.
Currently, it writes out approximately 1.5TB for the
benchmark problem, most of that contained in a
single file, much too large for a ramdisk.
Unfortunately, the problems have grown so large that
they can't be run in ram any more. This is a GPU
accelerated program so this file gets modified very
heavily during the process of a run. <br>
<br>
</div>
Current testing is being done on a Raid0 of 5c Crucial
960G SSDs. This has proven to be significantly faster
than the old array, but I am trying to determine
exactly how hard the disks are being hammered so I can
try and optimize the hardware configuration. <br>
<br>
</div>
The program is compiled from source, but I'm not
involved in that process, I'd much rather try and
piggyback something and monitor the process than try and
go in and have something added to source. <br>
<br>
</div>
I'll add parted and gparted to my list of things to read
up on, thanks. <br>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Thu, Dec 12, 2013 at 12:29
AM, David <span dir="ltr"><<a href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Excellent
approach.
<div>
<div><br>
<br>
<br>
<div>Arthur wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">How big are the files that
you're dealing with?
<div>If they're small you can just make
a ramdisk and try running everything
in there.</div>
<div>It's not a final solution, but
between that and strace you should be
able to see if that's really the issue
or not.</div>
<div><br>
</div>
<div>Are you compiling from source? If
you are, then there are a bunch of
debugging tools you can use as well as
doing things like timing individual
commands, and seeing how many times
each line of code is run.</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Dec 11,
2013 at 10:48 PM, Stephan Henning <span dir="ltr"><<a href="mailto:shenning@gmail.com" target="_blank">shenning@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">This is a RedHat6
Enterprise install.
<div><br>
</div>
<div>I don't think htop has the
data I need, but I'll check. I'm
not familiar with ntop and I
didn't consider using trace for
this, I'll check that as well.</div>
<div><br>
</div>
<div>The goal is to record
read/write rates and block
sizes. I'm pretty sure I am
bottlenecking against the drive
array, I'm hoping I can get some
definitive answers from this. </div>
</div>
<div>
<div>
<div class="gmail_extra"> <br>
<br>
<div class="gmail_quote">On
Wed, Dec 11, 2013 at 6:01
PM, David <span dir="ltr"><<a href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> ntop
might do the trick, but
not available in Fedora.
<div>
<div><br>
<br>
<br>
<div>David wrote:<br>
</div>
<blockquote type="cite"> Can
'htop' show open
files?<br>
<br>
For intensive live
net data, look at
WireShark for
linux.<br>
<br>
<br>
<div>David wrote:<br>
</div>
<blockquote type="cite"> If
that's what
you're looking
for, there are
several (free)
programs you
could run from
the command line
in a separate
window/screen
while your
program is
running that
give you all
you're asking
about. Sort of
an equivalent to
Winblows "System
Explorer." What
flavor or Linux
are you using? <br>
<br>
David M.<br>
<br>
<br>
<div>Devin Boyer
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Try
something like
"strace -T
myapp" or
"strace -T -c
myapp";
they'll show
the system
calls being
made and the
amount of time
spent in each.
It's slightly
different
information
than iostat,
but it may be
useful in
figuring out
what and where
your program
is performing
io access.</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On
Wed, Dec 11,
2013 at 3:37
PM, Stephan
Henning <span dir="ltr"><<a href="mailto:shenning@gmail.com" target="_blank">shenning@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>No,
iostat will
normally just
dump to the
terminal
window, but
I'd like to
pipe it's
output to a
file so I can
parse it
later. <br>
<br>
</div>
My end goal
here is to be
able to
generate a log
of iostat
output while I
run this
program, I'm
trying to
determine
exactly how
hard this
program is
hitting my
harddrive and
at what points
during it's
run does it
access the
drive the most
frequently. <br>
<br>
</div>
I've done
something
similar in
bash before,
but it is
rather clunky.
<br>
<br>
</div>
I'll take a
look at exec
and see if I
can use it. <br>
</div>
<div>
<div>
<div class="gmail_extra">
<br>
<br>
<div class="gmail_quote">On
Wed, Dec 11,
2013 at 4:46
PM, David <span dir="ltr"><<a href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Do you need to
do anything
with the
results or
just need them
displayed?<br>
If you need to
manipulate the
results,
consider using
Perl, <br>
or if C or
C++,<br>
in your 'exec'
call, pipe the
output to a
file, then
just read that
file into your
program.<br>
Ain't UNIX
great?
<div><br>
<br>
David M.<br>
<br>
<br>
<div>Stephan
Henning wrote:<br>
</div>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">
<div>I'd like
to take some
metrics with
iostat while I
have a
specific
program
running, is
there a way to
wrap iostat
around another
program (it is
called from
the command
line) so that
iostat ends
when the
program
finishes
running? <br>
<br>
</div>
I know I can
do it with a
bash script,
but I'm hoping
for a more
elegant
solution. <br>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
<div>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</div>
</blockquote>
<br>
</div>
<br>
_______________________________________________<br>
General
mailing list<br>
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General
mailing list<br>
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
Sincerely,<br>
Arthur Moore<br>
<a href="tel:%28256%29%20277-1001" value="+12562771001" target="_blank">(256)
277-1001</a><br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
Sincerely,<br>
Arthur Moore<br>
<a href="tel:%28256%29%20277-1001" value="+12562771001" target="_blank">(256) 277-1001</a><br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</div></div></div>
<br>_______________________________________________<br>
General mailing list<br>
<a href="mailto:General@lists.makerslocal.org">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br></blockquote></div><br></div>