<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Clustering is no longer difficult. There are several versions for
Linux already out there. All of them allow clustering via ethernet
so you -could- get, say, 5 pc's each with a dual-core GPU (what,
$500 per computer?) and let-er-rip. If ethernet is too slow a link,
others are available but I am not personally familiar with any of
them.<br>
See Linux Beowulf cluster. To fully optimize a task, it's best to
have the source so you can tweak it, but with Beowulf clusters, it's
not necessary. Just feed it your existing binary and the OS cluster
manager will do it's best to divide the task among the nodes.<br>
<br>
Shoot, might be fun just to set one up and run your task to see how
it compares to the Xeon box.<br>
<br>
Does the state of Alabama still have it's supecomputer up? It's
here in Huntsville. Cheap, too. I don't remember their hardware.<br>
<br>
David M.<br>
<br>
<br>
<div class="moz-cite-prefix">Stephan Henning wrote:<br>
</div>
<blockquote
cite="mid:CACu1UD6yawMK-57Ybo7rfGi1rdtXrzFuvZ34g49-yq=QOhuB0A@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>Very much drool worthy, until you realize that it's still
not fast enough. Then you just want to smack your head against
the table. <br>
<br>
</div>
I actually called Oak Ridge at one point to see what's invovled
in getting time on their cluster for a lark. This program would
not scale well to a distributed environment though, the way it's
coded it's designed to fully utilize every resource it can in a
single node. <br>
<div>
<div>
<div><br>
Nope, it's solid data. There is some compression that is
being done to eliminate redundant data (I think), but that
is black magic to me. <br>
<br>
</div>
<div>Hopefully I can get the metrics I need out of one of
the tools mentioned here. This is a test box, we've stolen
the gpu's out of it's twin machine to try and stress these
SSDs to see if we can observe a new bottleneck in the
system. On a mechanical array we couldn't feed data to
more than 2 of the K10 cards before we saturated what the
drive array. Mechanical drives just can't sustain the
read/write speeds the GPUs need.<br>
<br>
</div>
<div>The 'dream' here would be if could get a single host
node connected to several PCI expansion boxes, each one
containing 4 or 6 k10 cards, with between 16 and 24 k10's
total per node. To make that work I think the entire
problem would need to be GPU accelerated and the node
would need to have at least 512Gb of ram to buffer enough
data to keep up with the cards and I think we might get
close to saturating the PCIe bus with that...<br>
<br>
</div>
<div>Anyway, enough daydreaming, one step at a time. <br>
</div>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Thu, Dec 12, 2013 at 4:18 PM, David
<span dir="ltr"><<a moz-do-not-send="true"
href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <drool> Ok,
I've gotten control of myself. <br>
<br>
This huge file that holds temp data, it's not by chance a
sparse matrix, is it? You might could reduce the file
size (and times) if it is. <br>
<br>
David M.<br>
grasping at straws.
<div>
<div class="h5"><br>
<br>
<br>
<div>Stephan Henning wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>Well, the only issue there is that the
file size changes based on the problem size.
Every problem creates a file of a different
size, and the block sizes and the timings
for when the blocks are requested are not
necessarily consistent between problems.<br>
<br>
</div>
Ya, the write life is something I've been
paying very careful attention to. Thankfully,
even under this very heavy use case, they
should last at least a year, probably closer
to 5 before we have to pull them out. This
really large file is not mission critical,
it's effectively a giant lookup table that is
used to perform the calculations. Once the
'answer' is generated, that table is basically
useless. The final 'answer' file is usually
less that 100megs, so backing that up is a
trivial challenge :). <br>
<br>
</div>
Ya, the SSDs are nice. I'm trying to collect
this data to get a better idea if it's worth it
to look at the SAN flash arrays that are on the
market. The bad thing is this use case is very
unique compared to what these SAN arrays are
designed for (typically high end Oracle
databases), so it's difficult for their
engineers to fully grasp what our need is. I was
hoping that being able to hand them a chart
package that shows how all these metrics vary
over the course of the problem would give them a
better idea of what our need is, but generating
these metrics is becoming an adventure in it's
own right. <br>
<br>
</div>
The box that has the SSDs in it has 4 K10's doing
the computation. Hence the need for fast data
delivery :). <br>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Thu, Dec 12, 2013 at
3:14 PM, David <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:ainut@knology.net"
target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> You
might write a separate program whose only
function is to write files of that size and
do the timings for you.<br>
<br>
Btw, you are aware that there are only so
many writes you can do to SSD before they're
used up, right? If this is mission critical
data, you may want to keep frequent backups
on other media.<br>
<br>
Still, having huge SSD's, and several of
them to boot, makes me jealous! :)
<div>
<div><br>
<br>
David M.<br>
<br>
<br>
<div>Stephan Henning wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>The program will write out
a file of variable size, it's
based on the problem being
run. Currently, it writes out
approximately 1.5TB for the
benchmark problem, most of
that contained in a single
file, much too large for a
ramdisk. Unfortunately, the
problems have grown so large
that they can't be run in ram
any more. This is a GPU
accelerated program so this
file gets modified very
heavily during the process of
a run. <br>
<br>
</div>
Current testing is being done on
a Raid0 of 5c Crucial 960G SSDs.
This has proven to be
significantly faster than the
old array, but I am trying to
determine exactly how hard the
disks are being hammered so I
can try and optimize the
hardware configuration. <br>
<br>
</div>
The program is compiled from
source, but I'm not involved in
that process, I'd much rather try
and piggyback something and
monitor the process than try and
go in and have something added to
source. <br>
<br>
</div>
I'll add parted and gparted to my
list of things to read up on,
thanks. <br>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Thu, Dec
12, 2013 at 12:29 AM, David <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:ainut@knology.net"
target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF"
text="#000000"> Excellent
approach.
<div>
<div><br>
<br>
<br>
<div>Arthur wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">How big
are the files that
you're dealing with?
<div>If they're small
you can just make a
ramdisk and try
running everything
in there.</div>
<div>It's not a final
solution, but
between that and
strace you should be
able to see if
that's really the
issue or not.</div>
<div><br>
</div>
<div>Are you compiling
from source? If you
are, then there are
a bunch of debugging
tools you can use as
well as doing things
like timing
individual commands,
and seeing how many
times each line of
code is run.</div>
</div>
<div class="gmail_extra"><br>
<br>
<div
class="gmail_quote">On
Wed, Dec 11, 2013 at
10:48 PM, Stephan
Henning <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:shenning@gmail.com" target="_blank">shenning@gmail.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0
0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div dir="ltr">This
is a RedHat6
Enterprise
install.
<div><br>
</div>
<div>I don't
think htop has
the data I
need, but I'll
check. I'm not
familiar with
ntop and I
didn't
consider using
trace for
this, I'll
check that as
well.</div>
<div><br>
</div>
<div>The goal is
to record
read/write
rates and
block sizes.
I'm pretty
sure I am
bottlenecking
against the
drive array,
I'm hoping I
can get some
definitive
answers from
this. </div>
</div>
<div>
<div>
<div
class="gmail_extra">
<br>
<br>
<div
class="gmail_quote">On
Wed, Dec 11,
2013 at 6:01
PM, David <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
ntop might do
the trick, but
not available
in Fedora.
<div>
<div><br>
<br>
<br>
<div>David
wrote:<br>
</div>
<blockquote
type="cite">
Can 'htop'
show open
files?<br>
<br>
For intensive
live net data,
look at
WireShark for
linux.<br>
<br>
<br>
<div>David
wrote:<br>
</div>
<blockquote
type="cite">
If that's what
you're looking
for, there are
several (free)
programs you
could run from
the command
line in a
separate
window/screen
while your
program is
running that
give you all
you're asking
about. Sort
of an
equivalent to
Winblows
"System
Explorer."
What flavor or
Linux are you
using? <br>
<br>
David M.<br>
<br>
<br>
<div>Devin
Boyer wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">Try
something like
"strace -T
myapp" or
"strace -T -c
myapp";
they'll show
the system
calls being
made and the
amount of time
spent in each.
It's slightly
different
information
than iostat,
but it may be
useful in
figuring out
what and where
your program
is performing
io access.</div>
<div
class="gmail_extra"><br>
<br>
<div
class="gmail_quote">On
Wed, Dec 11,
2013 at 3:37
PM, Stephan
Henning <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:shenning@gmail.com" target="_blank">shenning@gmail.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>No,
iostat will
normally just
dump to the
terminal
window, but
I'd like to
pipe it's
output to a
file so I can
parse it
later. <br>
<br>
</div>
My end goal
here is to be
able to
generate a log
of iostat
output while I
run this
program, I'm
trying to
determine
exactly how
hard this
program is
hitting my
harddrive and
at what points
during it's
run does it
access the
drive the most
frequently. <br>
<br>
</div>
I've done
something
similar in
bash before,
but it is
rather clunky.
<br>
<br>
</div>
I'll take a
look at exec
and see if I
can use it. <br>
</div>
<div>
<div>
<div
class="gmail_extra">
<br>
<br>
<div
class="gmail_quote">On
Wed, Dec 11,
2013 at 4:46
PM, David <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
Do you need to
do anything
with the
results or
just need them
displayed?<br>
If you need to
manipulate the
results,
consider using
Perl, <br>
or if C or
C++,<br>
in your 'exec'
call, pipe the
output to a
file, then
just read that
file into your
program.<br>
Ain't UNIX
great?
<div><br>
<br>
David M.<br>
<br>
<br>
<div>Stephan
Henning wrote:<br>
</div>
</div>
<blockquote
type="cite">
<div>
<div dir="ltr">
<div>I'd like
to take some
metrics with
iostat while I
have a
specific
program
running, is
there a way to
wrap iostat
around another
program (it is
called from
the command
line) so that
iostat ends
when the
program
finishes
running? <br>
<br>
</div>
I know I can
do it with a
bash script,
but I'm hoping
for a more
elegant
solution. <br>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
<div>
<pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</div>
</blockquote>
<br>
</div>
<br>
_______________________________________________<br>
General
mailing list<br>
<a
moz-do-not-send="true"
href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a
moz-do-not-send="true"
href="http://lists.makerslocal.org/mailman/listinfo/general"
target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General
mailing list<br>
<a
moz-do-not-send="true"
href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a
moz-do-not-send="true"
href="http://lists.makerslocal.org/mailman/listinfo/general"
target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General
mailing list<br>
<a
moz-do-not-send="true"
href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a
moz-do-not-send="true"
href="http://lists.makerslocal.org/mailman/listinfo/general"
target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing
list<br>
<a
moz-do-not-send="true"
href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a
moz-do-not-send="true"
href="http://lists.makerslocal.org/mailman/listinfo/general"
target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
Sincerely,<br>
Arthur Moore<br>
<a
moz-do-not-send="true"
href="tel:%28256%29%20277-1001" value="+12562771001" target="_blank">(256)
277-1001</a><br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a moz-do-not-send="true"
href="mailto:General@lists.makerslocal.org"
target="_blank">General@lists.makerslocal.org</a><br>
<a moz-do-not-send="true"
href="http://lists.makerslocal.org/mailman/listinfo/general"
target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a moz-do-not-send="true"
href="mailto:General@lists.makerslocal.org"
target="_blank">General@lists.makerslocal.org</a><br>
<a moz-do-not-send="true"
href="http://lists.makerslocal.org/mailman/listinfo/general"
target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a moz-do-not-send="true"
href="mailto:General@lists.makerslocal.org">General@lists.makerslocal.org</a><br>
<a moz-do-not-send="true"
href="http://lists.makerslocal.org/mailman/listinfo/general"
target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
General mailing list
<a class="moz-txt-link-abbreviated" href="mailto:General@lists.makerslocal.org">General@lists.makerslocal.org</a>
<a class="moz-txt-link-freetext" href="http://lists.makerslocal.org/mailman/listinfo/general">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</body>
</html>