<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<drool> Ok, I've gotten control of myself. <br>
<br>
This huge file that holds temp data, it's not by chance a sparse
matrix, is it? You might could reduce the file size (and times) if
it is. <br>
<br>
David M.<br>
grasping at straws.<br>
<br>
<br>
<div class="moz-cite-prefix">Stephan Henning wrote:<br>
</div>
<blockquote
cite="mid:CACu1UD4tmkqfivbNbCGX1Hu=KP8-6dHbexEyBzvg33UwEJMU9Q@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>
<div>Well, the only issue there is that the file size
changes based on the problem size. Every problem creates a
file of a different size, and the block sizes and the
timings for when the blocks are requested are not
necessarily consistent between problems.<br>
<br>
</div>
Ya, the write life is something I've been paying very
careful attention to. Thankfully, even under this very heavy
use case, they should last at least a year, probably closer
to 5 before we have to pull them out. This really large file
is not mission critical, it's effectively a giant lookup
table that is used to perform the calculations. Once the
'answer' is generated, that table is basically useless. The
final 'answer' file is usually less that 100megs, so backing
that up is a trivial challenge :). <br>
<br>
</div>
Ya, the SSDs are nice. I'm trying to collect this data to get
a better idea if it's worth it to look at the SAN flash arrays
that are on the market. The bad thing is this use case is very
unique compared to what these SAN arrays are designed for
(typically high end Oracle databases), so it's difficult for
their engineers to fully grasp what our need is. I was hoping
that being able to hand them a chart package that shows how
all these metrics vary over the course of the problem would
give them a better idea of what our need is, but generating
these metrics is becoming an adventure in it's own right. <br>
<br>
</div>
The box that has the SSDs in it has 4 K10's doing the
computation. Hence the need for fast data delivery :). <br>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Thu, Dec 12, 2013 at 3:14 PM, David
<span dir="ltr"><<a moz-do-not-send="true"
href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> You might write a
separate program whose only function is to write files of
that size and do the timings for you.<br>
<br>
Btw, you are aware that there are only so many writes you
can do to SSD before they're used up, right? If this is
mission critical data, you may want to keep frequent
backups on other media.<br>
<br>
Still, having huge SSD's, and several of them to boot,
makes me jealous! :)
<div>
<div class="h5"><br>
<br>
David M.<br>
<br>
<br>
<div>Stephan Henning wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>The program will write out a file of
variable size, it's based on the problem
being run. Currently, it writes out
approximately 1.5TB for the benchmark
problem, most of that contained in a single
file, much too large for a ramdisk.
Unfortunately, the problems have grown so
large that they can't be run in ram any
more. This is a GPU accelerated program so
this file gets modified very heavily during
the process of a run. <br>
<br>
</div>
Current testing is being done on a Raid0 of 5c
Crucial 960G SSDs. This has proven to be
significantly faster than the old array, but I
am trying to determine exactly how hard the
disks are being hammered so I can try and
optimize the hardware configuration. <br>
<br>
</div>
The program is compiled from source, but I'm not
involved in that process, I'd much rather try
and piggyback something and monitor the process
than try and go in and have something added to
source. <br>
<br>
</div>
I'll add parted and gparted to my list of things
to read up on, thanks. <br>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Thu, Dec 12, 2013 at
12:29 AM, David <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:ainut@knology.net"
target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Excellent approach.
<div>
<div><br>
<br>
<br>
<div>Arthur wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">How big are the files
that you're dealing with?
<div>If they're small you can just
make a ramdisk and try running
everything in there.</div>
<div>It's not a final solution, but
between that and strace you should
be able to see if that's really
the issue or not.</div>
<div><br>
</div>
<div>Are you compiling from source?
If you are, then there are a
bunch of debugging tools you can
use as well as doing things like
timing individual commands, and
seeing how many times each line of
code is run.</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Dec
11, 2013 at 10:48 PM, Stephan
Henning <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:shenning@gmail.com"
target="_blank">shenning@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr">This is a RedHat6
Enterprise install.
<div><br>
</div>
<div>I don't think htop has
the data I need, but I'll
check. I'm not familiar with
ntop and I didn't consider
using trace for this, I'll
check that as well.</div>
<div><br>
</div>
<div>The goal is to record
read/write rates and block
sizes. I'm pretty sure I am
bottlenecking against the
drive array, I'm hoping I
can get some definitive
answers from this. </div>
</div>
<div>
<div>
<div class="gmail_extra"> <br>
<br>
<div class="gmail_quote">On
Wed, Dec 11, 2013 at
6:01 PM, David <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF"
text="#000000"> ntop
might do the trick,
but not available in
Fedora.
<div>
<div><br>
<br>
<br>
<div>David
wrote:<br>
</div>
<blockquote
type="cite">
Can 'htop'
show open
files?<br>
<br>
For intensive
live net data,
look at
WireShark for
linux.<br>
<br>
<br>
<div>David
wrote:<br>
</div>
<blockquote
type="cite">
If that's what
you're looking
for, there are
several (free)
programs you
could run from
the command
line in a
separate
window/screen
while your
program is
running that
give you all
you're asking
about. Sort
of an
equivalent to
Winblows
"System
Explorer."
What flavor or
Linux are you
using? <br>
<br>
David M.<br>
<br>
<br>
<div>Devin
Boyer wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">Try
something like
"strace -T
myapp" or
"strace -T -c
myapp";
they'll show
the system
calls being
made and the
amount of time
spent in each.
It's slightly
different
information
than iostat,
but it may be
useful in
figuring out
what and where
your program
is performing
io access.</div>
<div
class="gmail_extra"><br>
<br>
<div
class="gmail_quote">On
Wed, Dec 11,
2013 at 3:37
PM, Stephan
Henning <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:shenning@gmail.com" target="_blank">shenning@gmail.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>No,
iostat will
normally just
dump to the
terminal
window, but
I'd like to
pipe it's
output to a
file so I can
parse it
later. <br>
<br>
</div>
My end goal
here is to be
able to
generate a log
of iostat
output while I
run this
program, I'm
trying to
determine
exactly how
hard this
program is
hitting my
harddrive and
at what points
during it's
run does it
access the
drive the most
frequently. <br>
<br>
</div>
I've done
something
similar in
bash before,
but it is
rather clunky.
<br>
<br>
</div>
I'll take a
look at exec
and see if I
can use it. <br>
</div>
<div>
<div>
<div
class="gmail_extra">
<br>
<br>
<div
class="gmail_quote">On
Wed, Dec 11,
2013 at 4:46
PM, David <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
Do you need to
do anything
with the
results or
just need them
displayed?<br>
If you need to
manipulate the
results,
consider using
Perl, <br>
or if C or
C++,<br>
in your 'exec'
call, pipe the
output to a
file, then
just read that
file into your
program.<br>
Ain't UNIX
great?
<div><br>
<br>
David M.<br>
<br>
<br>
<div>Stephan
Henning wrote:<br>
</div>
</div>
<blockquote
type="cite">
<div>
<div dir="ltr">
<div>I'd like
to take some
metrics with
iostat while I
have a
specific
program
running, is
there a way to
wrap iostat
around another
program (it is
called from
the command
line) so that
iostat ends
when the
program
finishes
running? <br>
<br>
</div>
I know I can
do it with a
bash script,
but I'm hoping
for a more
elegant
solution. <br>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
<div>
<pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</div>
</blockquote>
<br>
</div>
<br>
_______________________________________________<br>
General
mailing list<br>
<a
moz-do-not-send="true"
href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a
moz-do-not-send="true"
href="http://lists.makerslocal.org/mailman/listinfo/general"
target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General
mailing list<br>
<a
moz-do-not-send="true"
href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a
moz-do-not-send="true"
href="http://lists.makerslocal.org/mailman/listinfo/general"
target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a
moz-do-not-send="true"
href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a
moz-do-not-send="true"
href="http://lists.makerslocal.org/mailman/listinfo/general"
target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a moz-do-not-send="true"
href="mailto:General@lists.makerslocal.org"
target="_blank">General@lists.makerslocal.org</a><br>
<a moz-do-not-send="true"
href="http://lists.makerslocal.org/mailman/listinfo/general"
target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
Sincerely,<br>
Arthur Moore<br>
<a moz-do-not-send="true"
href="tel:%28256%29%20277-1001"
value="+12562771001"
target="_blank">(256) 277-1001</a><br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a moz-do-not-send="true"
href="mailto:General@lists.makerslocal.org"
target="_blank">General@lists.makerslocal.org</a><br>
<a moz-do-not-send="true"
href="http://lists.makerslocal.org/mailman/listinfo/general"
target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a moz-do-not-send="true"
href="mailto:General@lists.makerslocal.org">General@lists.makerslocal.org</a><br>
<a moz-do-not-send="true"
href="http://lists.makerslocal.org/mailman/listinfo/general"
target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
General mailing list
<a class="moz-txt-link-abbreviated" href="mailto:General@lists.makerslocal.org">General@lists.makerslocal.org</a>
<a class="moz-txt-link-freetext" href="http://lists.makerslocal.org/mailman/listinfo/general">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</body>
</html>