<div dir="ltr"><div><div><div>Well, the only issue there is that the file size changes based on the problem size. Every problem creates a file of a different size, and the block sizes and the timings for when the blocks are requested are not necessarily consistent between problems.<br>
<br></div>Ya, the write life is something I've been paying very careful attention to. Thankfully, even under this very heavy use case, they should last at least a year, probably closer to 5 before we have to pull them out. This really large file is not mission critical, it's effectively a giant lookup table that is used to perform the calculations. Once the 'answer' is generated, that table is basically useless. The final 'answer' file is usually less that 100megs, so backing that up is a trivial challenge :). <br>
<br></div>Ya, the SSDs are nice. I'm trying to collect this data to get a better idea if it's worth it to look at the SAN flash arrays that are on the market. The bad thing is this use case is very unique compared to what these SAN arrays are designed for (typically high end Oracle databases), so it's difficult for their engineers to fully grasp what our need is. I was hoping that being able to hand them a chart package that shows how all these metrics vary over the course of the problem would give them a better idea of what our need is, but generating these metrics is becoming an adventure in it's own right. <br>
<br></div>The box that has the SSDs in it has 4 K10's doing the computation. Hence the need for fast data delivery :). <br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Dec 12, 2013 at 3:14 PM, David <span dir="ltr"><<a href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
You might write a separate program whose only function is to write
files of that size and do the timings for you.<br>
<br>
Btw, you are aware that there are only so many writes you can do to
SSD before they're used up, right? If this is mission critical
data, you may want to keep frequent backups on other media.<br>
<br>
Still, having huge SSD's, and several of them to boot, makes me
jealous! :)<div><div class="h5"><br>
<br>
David M.<br>
<br>
<br>
<div>Stephan Henning wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>The program will write out a file of variable size,
it's based on the problem being run. Currently, it writes
out approximately 1.5TB for the benchmark problem, most of
that contained in a single file, much too large for a
ramdisk. Unfortunately, the problems have grown so large
that they can't be run in ram any more. This is a GPU
accelerated program so this file gets modified very
heavily during the process of a run. <br>
<br>
</div>
Current testing is being done on a Raid0 of 5c Crucial 960G
SSDs. This has proven to be significantly faster than the
old array, but I am trying to determine exactly how hard the
disks are being hammered so I can try and optimize the
hardware configuration. <br>
<br>
</div>
The program is compiled from source, but I'm not involved in
that process, I'd much rather try and piggyback something and
monitor the process than try and go in and have something
added to source. <br>
<br>
</div>
I'll add parted and gparted to my list of things to read up on,
thanks. <br>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Thu, Dec 12, 2013 at 12:29 AM, David
<span dir="ltr"><<a href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Excellent approach.
<div>
<div><br>
<br>
<br>
<div>Arthur wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">How big are the files that you're
dealing with?
<div>If they're small you can just make a ramdisk
and try running everything in there.</div>
<div>It's not a final solution, but between that
and strace you should be able to see if that's
really the issue or not.</div>
<div><br>
</div>
<div>Are you compiling from source? If you are,
then there are a bunch of debugging tools you
can use as well as doing things like timing
individual commands, and seeing how many times
each line of code is run.</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Dec 11, 2013 at
10:48 PM, Stephan Henning <span dir="ltr"><<a href="mailto:shenning@gmail.com" target="_blank">shenning@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">This is a RedHat6 Enterprise
install.
<div><br>
</div>
<div>I don't think htop has the data I need,
but I'll check. I'm not familiar with ntop
and I didn't consider using trace for
this, I'll check that as well.</div>
<div><br>
</div>
<div>The goal is to record read/write rates
and block sizes. I'm pretty sure I am
bottlenecking against the drive array, I'm
hoping I can get some definitive answers
from this. </div>
</div>
<div>
<div>
<div class="gmail_extra"> <br>
<br>
<div class="gmail_quote">On Wed, Dec 11,
2013 at 6:01 PM, David <span dir="ltr"><<a href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> ntop might do the
trick, but not available in
Fedora.
<div>
<div><br>
<br>
<br>
<div>David wrote:<br>
</div>
<blockquote type="cite"> Can
'htop' show open files?<br>
<br>
For intensive live net data,
look at WireShark for linux.<br>
<br>
<br>
<div>David wrote:<br>
</div>
<blockquote type="cite"> If
that's what you're looking
for, there are several
(free) programs you could
run from the command line
in a separate
window/screen while your
program is running that
give you all you're asking
about. Sort of an
equivalent to Winblows
"System Explorer." What
flavor or Linux are you
using? <br>
<br>
David M.<br>
<br>
<br>
<div>Devin Boyer wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Try
something like "strace
-T myapp" or "strace
-T -c myapp"; they'll
show the system calls
being made and the
amount of time spent
in each. It's
slightly different
information than
iostat, but it may be
useful in figuring out
what and where your
program is performing
io access.</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On
Wed, Dec 11, 2013 at
3:37 PM, Stephan
Henning <span dir="ltr"><<a href="mailto:shenning@gmail.com" target="_blank">shenning@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>No,
iostat will
normally just
dump to the
terminal
window, but
I'd like to
pipe it's
output to a
file so I can
parse it
later. <br>
<br>
</div>
My end goal
here is to be
able to
generate a log
of iostat
output while I
run this
program, I'm
trying to
determine
exactly how
hard this
program is
hitting my
harddrive and
at what points
during it's
run does it
access the
drive the most
frequently. <br>
<br>
</div>
I've done
something
similar in
bash before,
but it is
rather clunky.
<br>
<br>
</div>
I'll take a look
at exec and see
if I can use it.
<br>
</div>
<div>
<div>
<div class="gmail_extra">
<br>
<br>
<div class="gmail_quote">On
Wed, Dec 11,
2013 at 4:46
PM, David <span dir="ltr"><<a href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Do you need to
do anything
with the
results or
just need them
displayed?<br>
If you need to
manipulate the
results,
consider using
Perl, <br>
or if C or
C++,<br>
in your 'exec'
call, pipe the
output to a
file, then
just read that
file into your
program.<br>
Ain't UNIX
great?
<div><br>
<br>
David M.<br>
<br>
<br>
<div>Stephan
Henning wrote:<br>
</div>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">
<div>I'd like
to take some
metrics with
iostat while I
have a
specific
program
running, is
there a way to
wrap iostat
around another
program (it is
called from
the command
line) so that
iostat ends
when the
program
finishes
running? <br>
<br>
</div>
I know I can
do it with a
bash script,
but I'm hoping
for a more
elegant
solution. <br>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
<div>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</div>
</blockquote>
<br>
</div>
<br>
_______________________________________________<br>
General
mailing list<br>
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing
list<br>
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
Sincerely,<br>
Arthur Moore<br>
<a href="tel:%28256%29%20277-1001" value="+12562771001" target="_blank">(256)
277-1001</a><br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</div></div></div>
<br>_______________________________________________<br>
General mailing list<br>
<a href="mailto:General@lists.makerslocal.org">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br></blockquote></div><br></div>