<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Stephan,<br>
<br>
The system architecture will determine how quickly your huge tasks
get finished. (See SGI versus PC in previous email.)<br>
<br>
NUMA refers to design of the computer and for our interests,
determines how fast data moves around the buses.<br>
It is one of the reasons that, back in the 90's, SGI would whup
Sun's butt in execution times for data intensive tasks, even though
the Sun's were more expensive (Sun just had more advertising
money.) Anyway, a lot of that technology finally made it's way from
the workstation/minicomputer world into the PC world. <br>
<br>
GPU's are fantastic for this class of tasks. FPGA's would be much
faster still, but I doubt you could even get one big enough.<br>
<br>
While we're on the subject of optimizations :)<br>
Is the algorithm linear or can it be broken up such that it could
run several times faster if run on a cluster? It might be a
candidate, depending upon the algorithm. A PS3 can sometimes rival
GPU's as well but it is severely limited in onboard memory. A
cluster is ideal if, for example, your task is image analysis and
processor 1 can work on the upper left corner of the image, while
processor 2, the upper right, and so on, with each computer in the
cluster having it's own GPU. But if during runtime, each successive
calculation depends entirely on the one preceding it, then a cluster
would just slow you down.<br>
<br>
Yeah, you don't want the GPU's doing mundane work like file i/o,
just data crunching because that's what they're best at. That,
though, is entirely compiler dependent, unless your language allows
you to specify what runs where.<br>
<br>
May I ask what your runtimes are now?<br>
<br>
if you installed 2 terabytes of memory (mortgaging the state of
Alabama to do so), you could get unbelievable runtimes. :)<br>
<br>
David M.<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">Stephan Henning wrote:<br>
</div>
<blockquote
cite="mid:CACu1UD4+6FkDn9eHE9=gX_3Pbg-V8O6EfNOpLXm42R2ZpvNwUQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>I'm honestly not sure David, I'm a bit confused as to if
NUMA is supposed to be something implemented at a die level or
at a system level. If at a die level, This box is running 2x
Xeon E5-2650. I've never gotten involved in the architecture
side of things so this is a bit foreign to me. <br>
<br>
The bulk of the computation for a run is done on the GPUs, if
it all ran on the CPU a single run would take months. The GPUs
have cut the runtime way down, but only part of the process is
GPU accelerated, and this file creation phase is one of the
parts that is not and is still being processed by the CPU. <span
style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:20px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);display:inline!important;float:none"><br>
</span></div>
<span
style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:20px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);display:inline!important;float:none"></span></div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Thu, Dec 12, 2013 at 3:40 PM, David
<span dir="ltr"><<a moz-do-not-send="true"
href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Sometimes, the use of
DMA with the newest SATA III controllers actually slows it
down. Only a live test will show which is faster.<br>
Good point about the random access versus linear write,
though. My suspicion from his overview is that it is
linear.<br>
<br>
Linux is pretty good at managing disk optimization but if
tweaking is necessary, it can get very problematic because
as you shift the dynamics, the OS file system handler
shifts it's management algorithm. I believe that came
from the Cray world.<br>
<br>
Stephan, does your computer use the NUMA architecture?
There is a newer, slightly faster design but I can't
remember what it is. The reason I bring that up is that
in a former life I had to deal with data the size you
mentioned. As a test, I ran some benchmarks several times
with the best PC available at the time against a smallish
SGI pizza box. The program used was one I wrote and had
in production for quite a while. The PC would massage the
data and finish in about 12 1/2 hours. The SGI box did it
in 2 hours 45. I ran that test several times using data
that changed somewhat each month and timing results were
consistent. Just something to think about if run-times
are killing you at the moment.
<div>
<div class="h5"><br>
<br>
<br>
<div>Arthur wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Here's something else to think about.
Is the program writing out the data in sequential
chuncks, or is it writing to random parts of the
file?
<div><br>
</div>
<div>With buffered writes the best speed up you
can get with a raid 0 array is if one disk is
writing something while the other disk is going
to the place where the next thing is going to be
written. If you're dealing with a bunch of
random writes, then ponying up for a few SSDs or
refactoring the code might be worth it.</div>
<div><br>
</div>
<div>Assuming that your raid controller isn't the
bottleneck. Some motherboard based raid
controllers use the CPU to do the work, and can
cause everything to slow down. (Side note.
Does anyone know if DMA works with those kind
of controllers?)</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Thu, Dec 12, 2013 at
11:26 AM, Stephan Henning <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:shenning@gmail.com"
target="_blank">shenning@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>The program will write out a file
of variable size, it's based on the
problem being run. Currently, it
writes out approximately 1.5TB for the
benchmark problem, most of that
contained in a single file, much too
large for a ramdisk. Unfortunately,
the problems have grown so large that
they can't be run in ram any more.
This is a GPU accelerated program so
this file gets modified very heavily
during the process of a run. <br>
<br>
</div>
Current testing is being done on a Raid0
of 5c Crucial 960G SSDs. This has proven
to be significantly faster than the old
array, but I am trying to determine
exactly how hard the disks are being
hammered so I can try and optimize the
hardware configuration. <br>
<br>
</div>
The program is compiled from source, but
I'm not involved in that process, I'd much
rather try and piggyback something and
monitor the process than try and go in and
have something added to source. <br>
<br>
</div>
I'll add parted and gparted to my list of
things to read up on, thanks. <br>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Thu, Dec 12,
2013 at 12:29 AM, David <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:ainut@knology.net"
target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF"
text="#000000"> Excellent
approach.
<div>
<div><br>
<br>
<br>
<div>Arthur wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">How big are
the files that you're
dealing with?
<div>If they're small you
can just make a ramdisk
and try running
everything in there.</div>
<div>It's not a final
solution, but between
that and strace you
should be able to see if
that's really the issue
or not.</div>
<div><br>
</div>
<div>Are you compiling
from source? If you
are, then there are a
bunch of debugging tools
you can use as well as
doing things like timing
individual commands, and
seeing how many times
each line of code is
run.</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On
Wed, Dec 11, 2013 at
10:48 PM, Stephan
Henning <span dir="ltr"><<a
moz-do-not-send="true" href="mailto:shenning@gmail.com" target="_blank">shenning@gmail.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div dir="ltr">This is
a RedHat6 Enterprise
install.
<div><br>
</div>
<div>I don't think
htop has the data
I need, but I'll
check. I'm not
familiar with ntop
and I didn't
consider using
trace for this,
I'll check that as
well.</div>
<div><br>
</div>
<div>The goal is to
record read/write
rates and block
sizes. I'm pretty
sure I am
bottlenecking
against the drive
array, I'm hoping
I can get some
definitive answers
from this. </div>
</div>
<div>
<div>
<div
class="gmail_extra">
<br>
<br>
<div
class="gmail_quote">On
Wed, Dec 11,
2013 at 6:01
PM, David <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
ntop might do
the trick, but
not available
in Fedora.
<div>
<div><br>
<br>
<br>
<div>David
wrote:<br>
</div>
<blockquote
type="cite">
Can 'htop'
show open
files?<br>
<br>
For intensive
live net data,
look at
WireShark for
linux.<br>
<br>
<br>
<div>David
wrote:<br>
</div>
<blockquote
type="cite">
If that's what
you're looking
for, there are
several (free)
programs you
could run from
the command
line in a
separate
window/screen
while your
program is
running that
give you all
you're asking
about. Sort
of an
equivalent to
Winblows
"System
Explorer."
What flavor or
Linux are you
using? <br>
<br>
David M.<br>
<br>
<br>
<div>Devin
Boyer wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">Try
something like
"strace -T
myapp" or
"strace -T -c
myapp";
they'll show
the system
calls being
made and the
amount of time
spent in each.
It's slightly
different
information
than iostat,
but it may be
useful in
figuring out
what and where
your program
is performing
io access.</div>
<div
class="gmail_extra"><br>
<br>
<div
class="gmail_quote">On
Wed, Dec 11,
2013 at 3:37
PM, Stephan
Henning <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:shenning@gmail.com" target="_blank">shenning@gmail.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>No,
iostat will
normally just
dump to the
terminal
window, but
I'd like to
pipe it's
output to a
file so I can
parse it
later. <br>
<br>
</div>
My end goal
here is to be
able to
generate a log
of iostat
output while I
run this
program, I'm
trying to
determine
exactly how
hard this
program is
hitting my
harddrive and
at what points
during it's
run does it
access the
drive the most
frequently. <br>
<br>
</div>
I've done
something
similar in
bash before,
but it is
rather clunky.
<br>
<br>
</div>
I'll take a
look at exec
and see if I
can use it. <br>
</div>
<div>
<div>
<div
class="gmail_extra">
<br>
<br>
<div
class="gmail_quote">On
Wed, Dec 11,
2013 at 4:46
PM, David <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
Do you need to
do anything
with the
results or
just need them
displayed?<br>
If you need to
manipulate the
results,
consider using
Perl, <br>
or if C or
C++,<br>
in your 'exec'
call, pipe the
output to a
file, then
just read that
file into your
program.<br>
Ain't UNIX
great?
<div><br>
<br>
David M.<br>
<br>
<br>
<div>Stephan
Henning wrote:<br>
</div>
</div>
<blockquote
type="cite">
<div>
<div dir="ltr">
<div>I'd like
to take some
metrics with
iostat while I
have a
specific
program
running, is
there a way to
wrap iostat
around another
program (it is
called from
the command
line) so that
iostat ends
when the
program
finishes
running? <br>
<br>
</div>
I know I can
do it with a
bash script,
but I'm hoping
for a more
elegant
solution. <br>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
<div>
<pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</div>
</blockquote>
<br>
</div>
<br>
_______________________________________________<br>
General
mailing list<br>
<a
moz-do-not-send="true"
href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a
moz-do-not-send="true"
href="http://lists.makerslocal.org/mailman/listinfo/general"
target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General
mailing list<br>
<a
moz-do-not-send="true"
href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a
moz-do-not-send="true"
href="http://lists.makerslocal.org/mailman/listinfo/general"
target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General
mailing list<br>
<a
moz-do-not-send="true"
href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a
moz-do-not-send="true"
href="http://lists.makerslocal.org/mailman/listinfo/general"
target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a
moz-do-not-send="true"
href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a
moz-do-not-send="true"
href="http://lists.makerslocal.org/mailman/listinfo/general"
target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
Sincerely,<br>
Arthur Moore<br>
<a moz-do-not-send="true"
href="tel:%28256%29%20277-1001" value="+12562771001" target="_blank">(256)
277-1001</a><br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a moz-do-not-send="true"
href="mailto:General@lists.makerslocal.org"
target="_blank">General@lists.makerslocal.org</a><br>
<a moz-do-not-send="true"
href="http://lists.makerslocal.org/mailman/listinfo/general"
target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a moz-do-not-send="true"
href="mailto:General@lists.makerslocal.org"
target="_blank">General@lists.makerslocal.org</a><br>
<a moz-do-not-send="true"
href="http://lists.makerslocal.org/mailman/listinfo/general"
target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
Sincerely,<br>
Arthur Moore<br>
<a moz-do-not-send="true"
href="tel:%28256%29%20277-1001"
value="+12562771001" target="_blank">(256)
277-1001</a><br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a moz-do-not-send="true"
href="mailto:General@lists.makerslocal.org">General@lists.makerslocal.org</a><br>
<a moz-do-not-send="true"
href="http://lists.makerslocal.org/mailman/listinfo/general"
target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
General mailing list
<a class="moz-txt-link-abbreviated" href="mailto:General@lists.makerslocal.org">General@lists.makerslocal.org</a>
<a class="moz-txt-link-freetext" href="http://lists.makerslocal.org/mailman/listinfo/general">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</body>
</html>