<div dir="ltr"><div><div>Depending on the phase the problem is in, there can be a lot of sequential or a lot of random writes. During the initial part of the problem it is mostly sequential, through the rest of the process it is relatively random. <br>
<br></div>It is currently running on an array of 5 SSDs in raid0, there was a marked improvement in this over a raid0 of mechanical drives, shaved about half the time of the disk intensive portion of the operation. <br><br>
</div><div>It's possible the controller could become a bottleneck, but it's an 8 port LSI card so I doubt we are maxing it out right now. <br><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Thu, Dec 12, 2013 at 1:37 PM, Arthur <span dir="ltr"><<a href="mailto:Arthur@cd-net.net" target="_blank">Arthur@cd-net.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Here's something else to think about. Is the program writing out the data in sequential chuncks, or is it writing to random parts of the file?<div><br></div><div>With buffered writes the best speed up you can get with a raid 0 array is if one disk is writing something while the other disk is going to the place where the next thing is going to be written. If you're dealing with a bunch of random writes, then ponying up for a few SSDs or refactoring the code might be worth it.</div>
<div><br></div><div>Assuming that your raid controller isn't the bottleneck. Some motherboard based raid controllers use the CPU to do the work, and can cause everything to slow down. (Side note. Does anyone know if DMA works with those kind of controllers?)</div>
</div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Dec 12, 2013 at 11:26 AM, Stephan Henning <span dir="ltr"><<a href="mailto:shenning@gmail.com" target="_blank">shenning@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>The program will write out a file of variable size, it's based on the problem being run. Currently, it writes out approximately 1.5TB for the benchmark problem, most of that contained in a single file, much too large for a ramdisk. Unfortunately, the problems have grown so large that they can't be run in ram any more. This is a GPU accelerated program so this file gets modified very heavily during the process of a run. <br>
<br></div>Current testing is being done on a Raid0 of 5c Crucial 960G SSDs. This has proven to be significantly faster than the old array, but I am trying to determine exactly how hard the disks are being hammered so I can try and optimize the hardware configuration. <br>
<br></div>The program is compiled from source, but I'm not involved in that process, I'd much rather try and piggyback something and monitor the process than try and go in and have something added to source. <br>
<br>
</div>I'll add parted and gparted to my list of things to read up on, thanks. <br></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Dec 12, 2013 at 12:29 AM, David <span dir="ltr"><<a href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Excellent approach.<div><div><br>
<br>
<br>
<div>Arthur wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">How big are the files that you're dealing with?
<div>If they're small you can just make a ramdisk and try
running everything in there.</div>
<div>It's not a final solution, but between that and strace you
should be able to see if that's really the issue or not.</div>
<div><br>
</div>
<div>Are you compiling from source? If you are, then there are
a bunch of debugging tools you can use as well as doing things
like timing individual commands, and seeing how many times
each line of code is run.</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Dec 11, 2013 at 10:48 PM,
Stephan Henning <span dir="ltr"><<a href="mailto:shenning@gmail.com" target="_blank">shenning@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">This is a RedHat6 Enterprise install.
<div><br>
</div>
<div>I don't think htop has the data I need, but I'll
check. I'm not familiar with ntop and I didn't consider
using trace for this, I'll check that as well.</div>
<div><br>
</div>
<div>The goal is to record read/write rates and block
sizes. I'm pretty sure I am bottlenecking against the
drive array, I'm hoping I can get some definitive
answers from this. </div>
</div>
<div>
<div>
<div class="gmail_extra">
<br>
<br>
<div class="gmail_quote">On Wed, Dec 11, 2013 at 6:01
PM, David <span dir="ltr"><<a href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> ntop might
do the trick, but not available in Fedora.
<div>
<div><br>
<br>
<br>
<div>David wrote:<br>
</div>
<blockquote type="cite"> Can 'htop' show
open files?<br>
<br>
For intensive live net data, look at
WireShark for linux.<br>
<br>
<br>
<div>David wrote:<br>
</div>
<blockquote type="cite"> If that's what
you're looking for, there are several
(free) programs you could run from the
command line in a separate window/screen
while your program is running that give
you all you're asking about. Sort of an
equivalent to Winblows "System
Explorer." What flavor or Linux are you
using? <br>
<br>
David M.<br>
<br>
<br>
<div>Devin Boyer wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Try something like
"strace -T myapp" or "strace -T -c
myapp"; they'll show the system
calls being made and the amount of
time spent in each. It's slightly
different information than iostat,
but it may be useful in figuring out
what and where your program is
performing io access.</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Dec
11, 2013 at 3:37 PM, Stephan
Henning <span dir="ltr"><<a href="mailto:shenning@gmail.com" target="_blank">shenning@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>No, iostat will
normally just dump to
the terminal window, but
I'd like to pipe it's
output to a file so I
can parse it later. <br>
<br>
</div>
My end goal here is to be
able to generate a log of
iostat output while I run
this program, I'm trying
to determine exactly how
hard this program is
hitting my harddrive and
at what points during it's
run does it access the
drive the most frequently.
<br>
<br>
</div>
I've done something similar
in bash before, but it is
rather clunky. <br>
<br>
</div>
I'll take a look at exec and
see if I can use it. <br>
</div>
<div>
<div>
<div class="gmail_extra"> <br>
<br>
<div class="gmail_quote">On
Wed, Dec 11, 2013 at
4:46 PM, David <span dir="ltr"><<a href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Do
you need to do
anything with the
results or just need
them displayed?<br>
If you need to
manipulate the
results, consider
using Perl, <br>
or if C or C++,<br>
in your 'exec' call,
pipe the output to a
file, then just read
that file into your
program.<br>
Ain't UNIX great?
<div><br>
<br>
David M.<br>
<br>
<br>
<div>Stephan
Henning wrote:<br>
</div>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">
<div>I'd like
to take some
metrics with
iostat while I
have a
specific
program
running, is
there a way to
wrap iostat
around another
program (it is
called from
the command
line) so that
iostat ends
when the
program
finishes
running? <br>
<br>
</div>
I know I can
do it with a
bash script,
but I'm hoping
for a more
elegant
solution. <br>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
<div>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</div>
</blockquote>
<br>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
General mailing list<br>
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
Sincerely,<br>
Arthur Moore<br>
<a href="tel:%28256%29%20277-1001" value="+12562771001" target="_blank">(256) 277-1001</a><br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
General mailing list
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
</blockquote>
<br>
</div></div></div>
<br>_______________________________________________<br>
General mailing list<br>
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
General mailing list<br>
<a href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br>Sincerely,<br>Arthur Moore<br>
<a href="tel:%28256%29%20277-1001" value="+12562771001" target="_blank">(256) 277-1001</a><br>
</div>
</div></div><br>_______________________________________________<br>
General mailing list<br>
<a href="mailto:General@lists.makerslocal.org">General@lists.makerslocal.org</a><br>
<a href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br></blockquote></div><br></div>