<div dir="ltr"><div>Very much drool worthy, until you realize that it's still not fast enough. Then you just want to smack your head against the table. <br><br></div>I actually called Oak Ridge at one point to see what's invovled in getting time on their cluster for a lark. This program would not scale well to a distributed environment though, the way it's coded it's designed to fully utilize every resource it can in a single node. <br>
<div><div><div><br>Nope, it's solid data. There is some compression that is being done to eliminate redundant data (I think), but that is black magic to me. <br><br></div><div>Hopefully I can get the metrics I need out of one of the tools mentioned here. This is a test box, we've stolen the gpu's out of it's twin machine to try and stress these SSDs to see if we can observe a new bottleneck in the system. On a mechanical array we couldn't feed data to more than 2 of the K10 cards before we saturated what the drive array. Mechanical drives just can't sustain the read/write speeds the GPUs need.<br>
<br></div><div>The 'dream' here would be if could get a single host node connected to several PCI expansion boxes, each one containing 4 or 6 k10 cards, with between 16 and 24 k10's total per node. To make that work I think the entire problem would need to be GPU accelerated and the node would need to have at least 512Gb of ram to buffer enough data to keep up with the cards and I think we might get close to saturating the PCIe bus with that...<br>
<br></div><div>Anyway, enough daydreaming, one step at a time. <br></div></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Dec 12, 2013 at 4:18 PM, David <span dir="ltr"><<a href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<drool> Ok, I've gotten control of myself. <br>
<br>
This huge file that holds temp data, it's not by chance a sparse
matrix, is it? You might could reduce the file size (and times) if
it is. <br>
<br>
David M.<br>
grasping at straws.<div><div class="h5"><br>
<br>
<br>
<div>Stephan Henning wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>Well, the only issue there is that the file size
changes based on the problem size. Every problem creates a
file of a different size, and the block sizes and the
timings for when the blocks are requested are not
necessarily consistent between problems.<br>
<br>
</div>
Ya, the write life is something I've been paying very
careful attention to. Thankfully, even under this very heavy
use case, they should last at least a year, probably closer
to 5 before we have to pull them out. This really large file
is not mission critical, it's effectively a giant lookup
table that is used to perform the calculations. Once the
'answer' is generated, that table is basically useless. The
final 'answer' file is usually less that 100megs, so backing
that up is a trivial challenge :). <br>
<br>
</div>
Ya, the SSDs are nice. I'm trying to collect this data to get
a better idea if it's worth it to look at the SAN flash arrays
that are on the market. The bad thing is this use case is very
unique compared to what these SAN arrays are designed for
(typically high end Oracle databases), so it's difficult for
their engineers to fully grasp what our need is. I was hoping
that being able to hand them a chart package that shows how
all these metrics vary over the course of the problem would
give them a better idea of what our need is, but generating
these metrics is becoming an adventure in it's own right. <br>
<br>
</div>
The box that has the SSDs in it has 4 K10's doing the
computation. Hence the need for fast data delivery :). <br>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Thu, Dec 12, 2013 at 3:14 PM, David
<span dir="ltr"><<a href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> You might write a
separate program whose only function is to write files of
that size and do the timings for you.<br>
<br>
Btw, you are aware that there are only so many writes you
can do to SSD before they're used up, right? If this is
mission critical data, you may want to keep frequent
backups on other media.<br>
<br>
Still, having huge SSD's, and several of them to boot,
makes me jealous! :)
<div>
<div><br>
<br>
David M.<br>
<br>
<br>
<div>Stephan Henning wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>The program will write out a file of
variable size, it's based on the problem
being run. Currently, it writes out
approximately 1.5TB for the benchmark
problem, most of that contained in a single
file, much too large for a ramdisk.
Unfortunately, the problems have grown so
large that they can't be run in ram any
more. This is a GPU accelerated program so
this file gets modified very heavily during
the process of a run. <br>
<br>
</div>
Current testing is being done on a Raid0 of 5c
Crucial 960G SSDs. This has proven to be
significantly faster than the old array, but I
am trying to determine exactly how hard the
disks are being hammered so I can try and
optimize the hardware configuration. <br>
<br>
</div>
The program is compiled from source, but I'm not
involved in that process, I'd much rather try
and piggyback something and monitor the process
than try and go in and have something added to
source. <br>
<br>
</div>
I'll add parted and gparted to my list of things
to read up on, thanks. <br>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Thu, Dec 12, 2013 at
12:29 AM, David <span dir="ltr"><<a href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Excellent approach.
<div>
<div><br>
<br>
<br>
<div>Arthur wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">How big are the files
that you're dealing with?
<div>If they're small you can just
make a ramdisk and try running
everything in there.</div>
<div>It's not a final solution, but
between that and strace you should
be able to see if that's really
the issue or not.</div>
<div><br>
</div>
<div>Are you compiling from source?
If you are, then there are a
bunch of debugging tools you can
use as well as doing things like
timing individual commands, and
seeing how many times each line of
code is run.</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Dec
11, 2013 at 10:48 PM, Stephan
Henning <span dir="ltr"><<a href="mailto:shenning@gmail.com" target="_blank">shenning@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">This is a RedHat6
Enterprise install.
<div><br>
</div>
<div>I don't think htop has
the data I need, but I'll
check. I'm not familiar with
ntop and I didn't consider
using trace for this, I'll
check that as well.</div>
<div><br>
</div>
<div>The goal is to record
read/write rates and block
sizes. I'm pretty sure I am
bottlenecking against the
drive array, I'm hoping I
can get some definitive
answers from this. </div>
</div>
<div>
<div>
<div class="gmail_extra"> <br>
<br>
<div class="gmail_quote">On
Wed, Dec 11, 2013 at
6:01 PM, David <span dir="ltr"><<a href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> ntop
might do the trick,
but not available in
Fedora.
<div>
<div><br>
<br>
<br>
<div>David
wrote:<br>
</div>
<blockquote type="cite">
Can 'htop'
show open
files?<br>
<br>
For intensive
live net data,
look at
WireShark for
linux.<br>
<br>
<br>
<div>David
wrote:<br>
</div>
<blockquote type="cite">
If that's what
you're looking
for, there are
several (free)
programs you
could run from
the command
line in a
separate
window/screen
while your
program is
running that
give you all
you're asking
about. Sort
of an
equivalent to
Winblows
"System
Explorer."
What flavor or
Linux are you
using? <br>
<br>
David M.<br>
<br>
<br>
<div>Devin
Boyer wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Try
something like
"strace -T
myapp" or
"strace -T -c
myapp";
they'll show
the system
calls being
made and the
amount of time
spent in each.
It's slightly
different
information
than iostat,
but it may be
useful in
figuring out
what and where
your program
is performing
io access.</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On
Wed, Dec 11,
2013 at 3:37
PM, Stephan
Henning <span dir="ltr"><<a href="mailto:shenning@gmail.com" target="_blank">shenning@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>No,
iostat will
normally just
dump to the
terminal
window, but
I'd like to
pipe it's
output to a
file so I can
parse it
later. <br>
<br>
</div>
My end goal
here is to be
able to
generate a log
of iostat
output while I
run this
program, I'm
trying to
determine
exactly how
hard this
program is
hitting my
harddrive and
at what points
during it's
run does it
access the
drive the most
frequently. <br>
<br>
</div>
I've done
something
similar in
bash before,
but it is
rather clunky.
<br>
<br>
</div>
I'll take a
look at exec
and see if I
can use it. <br>
</div>
<div>
<div>
<div class="gmail_extra">
<br>
<br>
<div class="gmail_quote">On
Wed, Dec 11,
2013 at 4:46
PM, David <span dir="ltr"><<a href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Do you need to
do anything
with the
results or
just need them
displayed?<br>
If you need to
manipulate the
results,
consider using
Perl, <br>
or if C or
C++,<br>
in your 'exec'
call, pipe the
output to a
file, then
just read that
file into your
program.<br>
Ain't UNIX
great?
<div><br>
<br>
David M.<br>
<br>
<br>
<div>Stephan
Henning wrote:<br>
</div>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">
<div>I'd like
to take some
metrics with
iostat while I
have a
specific
program
running, is
there a way to
wrap iostat
around another
program (it is
called from
the command
line) so that
iostat ends
when the
program
finishes
running? <br>
<br>
</div>
I know I can
do it with a
bash script,
but I'm hoping
for a more
elegant
solution. <br>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
<div>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</div>
</blockquote>
<br>
</div>
<br>
_______________________________________________<br>
General
mailing list<br>
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General
mailing list<br>
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
Sincerely,<br>
Arthur Moore<br>
<a href="tel:%28256%29%20277-1001" value="+12562771001" target="_blank">(256) 277-1001</a><br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</div></div></div>
<br>_______________________________________________<br>
General mailing list<br>
<a href="mailto:General@lists.makerslocal.org">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br></blockquote></div><br></div>