<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <drool>  Ok, I've gotten control of myself.  <br>
    <br>
    This huge file that holds temp data, it's not by chance a sparse
    matrix, is it?  You might could reduce the file size (and times) if
    it is.  <br>
    <br>
    David M.<br>
    grasping at straws.<br>
    <br>
    <br>
    <div class="moz-cite-prefix">Stephan Henning wrote:<br>
    </div>
    <blockquote
cite="mid:CACu1UD4tmkqfivbNbCGX1Hu=KP8-6dHbexEyBzvg33UwEJMU9Q@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div>
          <div>
            <div>Well, the only issue there is that the file size
              changes based on the problem size. Every problem creates a
              file of a different size, and the block sizes and the
              timings for when the blocks are requested are not
              necessarily consistent between problems.<br>
              <br>
            </div>
            Ya, the write life is something I've been paying very
            careful attention to. Thankfully, even under this very heavy
            use case, they should last at least a year, probably closer
            to 5 before we have to pull them out. This really large file
            is not mission critical, it's effectively a giant lookup
            table that is used to perform the calculations. Once the
            'answer' is generated, that table is basically useless. The
            final 'answer' file is usually less that 100megs, so backing
            that up is a trivial challenge :). <br>
            <br>
          </div>
          Ya, the SSDs are nice. I'm trying to collect this data to get
          a better idea if it's worth it to look at the SAN flash arrays
          that are on the market. The bad thing is this use case is very
          unique compared to what these SAN arrays are designed for
          (typically high end Oracle databases), so it's difficult for
          their engineers to fully grasp what our need is. I was hoping
          that being able to hand them a chart package that shows how
          all these metrics vary over the course of the problem would
          give them a better idea of what our need is, but generating
          these metrics is becoming an adventure in it's own right. <br>
          <br>
        </div>
        The box that has the SSDs in it has 4 K10's doing the
        computation. Hence the need for fast data delivery :). <br>
      </div>
      <div class="gmail_extra"><br>
        <br>
        <div class="gmail_quote">On Thu, Dec 12, 2013 at 3:14 PM, David
          <span dir="ltr"><<a moz-do-not-send="true"
              href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div bgcolor="#FFFFFF" text="#000000"> You might write a
              separate program whose only function is to write files of
              that size and do the timings for you.<br>
              <br>
              Btw, you are aware that there are only so many writes you
              can do to SSD before they're used up, right?  If this is
              mission critical data, you may want to keep frequent
              backups on other media.<br>
              <br>
              Still, having huge SSD's, and several of them to boot,
              makes me jealous!  :)
              <div>
                <div class="h5"><br>
                  <br>
                  David M.<br>
                  <br>
                  <br>
                  <div>Stephan Henning wrote:<br>
                  </div>
                  <blockquote type="cite">
                    <div dir="ltr">
                      <div>
                        <div>
                          <div>The program will write out a file of
                            variable size, it's based on the problem
                            being run. Currently, it writes out
                            approximately 1.5TB for the benchmark
                            problem, most of that contained in a single
                            file, much too large for a ramdisk.
                            Unfortunately, the problems have grown so
                            large that they can't be run in ram any
                            more. This is a GPU accelerated program so
                            this file gets modified very heavily during
                            the process of a run. <br>
                            <br>
                          </div>
                          Current testing is being done on a Raid0 of 5c
                          Crucial 960G SSDs. This has proven to be
                          significantly faster than the old array, but I
                          am trying to determine exactly how hard the
                          disks are being hammered so I can try and
                          optimize the hardware configuration. <br>
                          <br>
                        </div>
                        The program is compiled from source, but I'm not
                        involved in that process, I'd much rather try
                        and piggyback something and monitor the process
                        than try and go in and have something added to
                        source. <br>
                        <br>
                      </div>
                      I'll add parted and gparted to my list of things
                      to read up on, thanks. <br>
                    </div>
                    <div class="gmail_extra"><br>
                      <br>
                      <div class="gmail_quote">On Thu, Dec 12, 2013 at
                        12:29 AM, David <span dir="ltr"><<a
                            moz-do-not-send="true"
                            href="mailto:ainut@knology.net"
                            target="_blank">ainut@knology.net</a>></span>
                        wrote:<br>
                        <blockquote class="gmail_quote" style="margin:0
                          0 0 .8ex;border-left:1px #ccc
                          solid;padding-left:1ex">
                          <div bgcolor="#FFFFFF" text="#000000">
                            Excellent approach.
                            <div>
                              <div><br>
                                <br>
                                <br>
                                <div>Arthur wrote:<br>
                                </div>
                                <blockquote type="cite">
                                  <div dir="ltr">How big are the files
                                    that you're dealing with?
                                    <div>If they're small you can just
                                      make a ramdisk and try running
                                      everything in there.</div>
                                    <div>It's not a final solution, but
                                      between that and strace you should
                                      be able to see if that's really
                                      the issue or not.</div>
                                    <div><br>
                                    </div>
                                    <div>Are you compiling from source?
                                       If you are, then there are a
                                      bunch of debugging tools you can
                                      use as well as doing things like
                                      timing individual commands, and
                                      seeing how many times each line of
                                      code is run.</div>
                                  </div>
                                  <div class="gmail_extra"><br>
                                    <br>
                                    <div class="gmail_quote">On Wed, Dec
                                      11, 2013 at 10:48 PM, Stephan
                                      Henning <span dir="ltr"><<a
                                          moz-do-not-send="true"
                                          href="mailto:shenning@gmail.com"
                                          target="_blank">shenning@gmail.com</a>></span>
                                      wrote:<br>
                                      <blockquote class="gmail_quote"
                                        style="margin:0 0 0
                                        .8ex;border-left:1px #ccc
                                        solid;padding-left:1ex">
                                        <div dir="ltr">This is a RedHat6
                                          Enterprise install. 
                                          <div><br>
                                          </div>
                                          <div>I don't think htop has
                                            the data I need, but I'll
                                            check. I'm not familiar with
                                            ntop and I didn't consider
                                            using trace for this, I'll
                                            check that as well.</div>
                                          <div><br>
                                          </div>
                                          <div>The goal is to record
                                            read/write rates and block
                                            sizes. I'm pretty sure I am
                                            bottlenecking against the
                                            drive array, I'm hoping I
                                            can get some definitive
                                            answers from this. </div>
                                        </div>
                                        <div>
                                          <div>
                                            <div class="gmail_extra"> <br>
                                              <br>
                                              <div class="gmail_quote">On
                                                Wed, Dec 11, 2013 at
                                                6:01 PM, David <span
                                                  dir="ltr"><<a
                                                    moz-do-not-send="true"
href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
                                                wrote:<br>
                                                <blockquote
                                                  class="gmail_quote"
                                                  style="margin:0 0 0
                                                  .8ex;border-left:1px
                                                  #ccc
                                                  solid;padding-left:1ex">
                                                  <div bgcolor="#FFFFFF"
                                                    text="#000000"> ntop
                                                    might do the trick,
                                                    but not available in
                                                    Fedora.
                                                    <div>
                                                      <div><br>
                                                        <br>
                                                        <br>
                                                        <div>David
                                                          wrote:<br>
                                                        </div>
                                                        <blockquote
                                                          type="cite">
                                                          Can 'htop'
                                                          show open
                                                          files?<br>
                                                          <br>
                                                          For intensive
                                                          live net data,
                                                          look at
                                                          WireShark for
                                                          linux.<br>
                                                          <br>
                                                          <br>
                                                          <div>David
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote
                                                          type="cite">
                                                          If that's what
                                                          you're looking
                                                          for, there are
                                                          several (free)
                                                          programs you
                                                          could run from
                                                          the command
                                                          line in a
                                                          separate
                                                          window/screen
                                                          while your
                                                          program is
                                                          running that
                                                          give you all
                                                          you're asking
                                                          about.  Sort
                                                          of an
                                                          equivalent to
                                                          Winblows
                                                          "System
                                                          Explorer." 
                                                          What flavor or
                                                          Linux are you
                                                          using? <br>
                                                          <br>
                                                          David M.<br>
                                                          <br>
                                                          <br>
                                                          <div>Devin
                                                          Boyer wrote:<br>
                                                          </div>
                                                          <blockquote
                                                          type="cite">
                                                          <div dir="ltr">Try

                                                          something like
                                                          "strace -T
                                                          myapp" or
                                                          "strace -T -c
                                                          myapp";
                                                          they'll show
                                                          the system
                                                          calls being
                                                          made and the
                                                          amount of time
                                                          spent in each.
                                                           It's slightly
                                                          different
                                                          information
                                                          than iostat,
                                                          but it may be
                                                          useful in
                                                          figuring out
                                                          what and where
                                                          your program
                                                          is performing
                                                          io access.</div>
                                                          <div
                                                          class="gmail_extra"><br>
                                                          <br>
                                                          <div
                                                          class="gmail_quote">On

                                                          Wed, Dec 11,
                                                          2013 at 3:37
                                                          PM, Stephan
                                                          Henning <span
                                                          dir="ltr"><<a
moz-do-not-send="true" href="mailto:shenning@gmail.com" target="_blank">shenning@gmail.com</a>></span>
                                                          wrote:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0
                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>
                                                          <div>No,
                                                          iostat will
                                                          normally just
                                                          dump to the
                                                          terminal
                                                          window, but
                                                          I'd like to
                                                          pipe it's
                                                          output to a
                                                          file so I can
                                                          parse it
                                                          later. <br>
                                                          <br>
                                                          </div>
                                                          My end goal
                                                          here is to be
                                                          able to
                                                          generate a log
                                                          of iostat
                                                          output while I
                                                          run this
                                                          program, I'm
                                                          trying to
                                                          determine
                                                          exactly how
                                                          hard this
                                                          program is
                                                          hitting my
                                                          harddrive and
                                                          at what points
                                                          during it's
                                                          run does it
                                                          access the
                                                          drive the most
                                                          frequently. <br>
                                                          <br>
                                                          </div>
                                                          I've done
                                                          something
                                                          similar in
                                                          bash before,
                                                          but it is
                                                          rather clunky.
                                                          <br>
                                                          <br>
                                                          </div>
                                                          I'll take a
                                                          look at exec
                                                          and see if I
                                                          can use it. <br>
                                                          </div>
                                                          <div>
                                                          <div>
                                                          <div
                                                          class="gmail_extra">
                                                          <br>
                                                          <br>
                                                          <div
                                                          class="gmail_quote">On


                                                          Wed, Dec 11,
                                                          2013 at 4:46
                                                          PM, David <span
                                                          dir="ltr"><<a
moz-do-not-send="true" href="mailto:ainut@knology.net" target="_blank">ainut@knology.net</a>></span>
                                                          wrote:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0
                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div
                                                          bgcolor="#FFFFFF"
                                                          text="#000000">
                                                          Do you need to
                                                          do anything
                                                          with the
                                                          results or
                                                          just need them
                                                          displayed?<br>
                                                          If you need to
                                                          manipulate the
                                                          results,
                                                          consider using
                                                          Perl, <br>
                                                          or if C or
                                                          C++,<br>
                                                          in your 'exec'
                                                          call, pipe the
                                                          output to a
                                                          file, then
                                                          just read that
                                                          file into your
                                                          program.<br>
                                                          Ain't UNIX
                                                          great?
                                                          <div><br>
                                                          <br>
                                                          David M.<br>
                                                          <br>
                                                          <br>
                                                          <div>Stephan
                                                          Henning wrote:<br>
                                                          </div>
                                                          </div>
                                                          <blockquote
                                                          type="cite">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>I'd like
                                                          to take some
                                                          metrics with
                                                          iostat while I
                                                          have a
                                                          specific
                                                          program
                                                          running, is
                                                          there a way to
                                                          wrap iostat
                                                          around another
                                                          program (it is
                                                          called from
                                                          the command
                                                          line) so that
                                                          iostat ends
                                                          when the
                                                          program
                                                          finishes
                                                          running? <br>
                                                          <br>
                                                          </div>
                                                          I know I can
                                                          do it with a
                                                          bash script,
                                                          but I'm hoping
                                                          for a more
                                                          elegant
                                                          solution. <br>
                                                          </div>
                                                          <br>
                                                          <fieldset></fieldset>
                                                          <br>
                                                          </div>
                                                          <div>
                                                          <pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          </div>
                                                          <br>
_______________________________________________<br>
                                                          General
                                                          mailing list<br>
                                                          <a
                                                          moz-do-not-send="true"
href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
                                                          <a
                                                          moz-do-not-send="true"
href="http://lists.makerslocal.org/mailman/listinfo/general"
                                                          target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <br>
_______________________________________________<br>
                                                          General
                                                          mailing list<br>
                                                          <a
                                                          moz-do-not-send="true"
href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
                                                          <a
                                                          moz-do-not-send="true"
href="http://lists.makerslocal.org/mailman/listinfo/general"
                                                          target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          <br>
                                                          <fieldset></fieldset>
                                                          <br>
                                                          <pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
                                                          </blockquote>
                                                          <br>
                                                          <br>
                                                          <fieldset></fieldset>
                                                          <br>
                                                          <pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
                                                          </blockquote>
                                                          <br>
                                                          <br>
                                                          <fieldset></fieldset>
                                                          <br>
                                                          <pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
                                                        </blockquote>
                                                        <br>
                                                      </div>
                                                    </div>
                                                  </div>
                                                  <br>
_______________________________________________<br>
                                                  General mailing list<br>
                                                  <a
                                                    moz-do-not-send="true"
href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a><br>
                                                  <a
                                                    moz-do-not-send="true"
href="http://lists.makerslocal.org/mailman/listinfo/general"
                                                    target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
                                                </blockquote>
                                              </div>
                                              <br>
                                            </div>
                                          </div>
                                        </div>
                                        <br>
_______________________________________________<br>
                                        General mailing list<br>
                                        <a moz-do-not-send="true"
                                          href="mailto:General@lists.makerslocal.org"
                                          target="_blank">General@lists.makerslocal.org</a><br>
                                        <a moz-do-not-send="true"
                                          href="http://lists.makerslocal.org/mailman/listinfo/general"
                                          target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
                                      </blockquote>
                                    </div>
                                    <br>
                                    <br clear="all">
                                    <div><br>
                                    </div>
                                    -- <br>
                                    Sincerely,<br>
                                    Arthur Moore<br>
                                    <a moz-do-not-send="true"
                                      href="tel:%28256%29%20277-1001"
                                      value="+12562771001"
                                      target="_blank">(256) 277-1001</a><br>
                                  </div>
                                  <br>
                                  <fieldset></fieldset>
                                  <br>
                                  <pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
                                </blockquote>
                                <br>
                              </div>
                            </div>
                          </div>
                          <br>
_______________________________________________<br>
                          General mailing list<br>
                          <a moz-do-not-send="true"
                            href="mailto:General@lists.makerslocal.org"
                            target="_blank">General@lists.makerslocal.org</a><br>
                          <a moz-do-not-send="true"
                            href="http://lists.makerslocal.org/mailman/listinfo/general"
                            target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
                        </blockquote>
                      </div>
                      <br>
                    </div>
                    <br>
                    <fieldset></fieldset>
                    <br>
                    <pre>_______________________________________________
General mailing list
<a moz-do-not-send="true" href="mailto:General@lists.makerslocal.org" target="_blank">General@lists.makerslocal.org</a>
<a moz-do-not-send="true" href="http://lists.makerslocal.org/mailman/listinfo/general" target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
                  </blockquote>
                  <br>
                </div>
              </div>
            </div>
            <br>
            _______________________________________________<br>
            General mailing list<br>
            <a moz-do-not-send="true"
              href="mailto:General@lists.makerslocal.org">General@lists.makerslocal.org</a><br>
            <a moz-do-not-send="true"
              href="http://lists.makerslocal.org/mailman/listinfo/general"
              target="_blank">http://lists.makerslocal.org/mailman/listinfo/general</a><br>
          </blockquote>
        </div>
        <br>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
General mailing list
<a class="moz-txt-link-abbreviated" href="mailto:General@lists.makerslocal.org">General@lists.makerslocal.org</a>
<a class="moz-txt-link-freetext" href="http://lists.makerslocal.org/mailman/listinfo/general">http://lists.makerslocal.org/mailman/listinfo/general</a></pre>
    </blockquote>
    <br>
  </body>
</html>