[General] Any linux gurus?

Thu Dec 12 16:32:04 CST 2013

Very much drool worthy, until you realize that it's still not fast enough.
Then you just want to smack your head against the table.

I actually called Oak Ridge at one point to see what's invovled in getting
time on their cluster for a lark. This program would not scale well to a
distributed environment though, the way it's coded it's designed to fully
utilize every resource it can in a single node.

Nope, it's solid data. There is some compression that is being done to
eliminate redundant data (I think), but that is black magic to me.

Hopefully I can get the metrics I need out of one of the tools mentioned
here. This is a test box, we've stolen the gpu's out of it's twin machine
to try and stress these SSDs to see if we can observe a new bottleneck in
the system. On a mechanical array we couldn't feed data to more than 2 of
the K10 cards before we saturated what the drive array. Mechanical drives
just can't sustain the read/write speeds the GPUs need.

The 'dream' here would be if could get a single host node connected to
several PCI expansion boxes, each one containing 4 or 6 k10 cards, with
between 16 and 24 k10's total per node. To make that work I think the
entire problem would need to be GPU accelerated and the node would need to
have at least 512Gb of ram to buffer enough data to keep up with the cards
and I think we might get close to saturating the PCIe bus with that...

Anyway, enough daydreaming, one step at a time.

On Thu, Dec 12, 2013 at 4:18 PM, David <ainut at knology.net> wrote:

>  <drool>  Ok, I've gotten control of myself.
>
> This huge file that holds temp data, it's not by chance a sparse matrix,
> is it?  You might could reduce the file size (and times) if it is.
>
> David M.
> grasping at straws.
>
>
>
> Stephan Henning wrote:
>
>   Well, the only issue there is that the file size changes based on the
> problem size. Every problem creates a file of a different size, and the
> block sizes and the timings for when the blocks are requested are not
> necessarily consistent between problems.
>
>  Ya, the write life is something I've been paying very careful attention
> to. Thankfully, even under this very heavy use case, they should last at
> least a year, probably closer to 5 before we have to pull them out. This
> really large file is not mission critical, it's effectively a giant lookup
> table that is used to perform the calculations. Once the 'answer' is
> generated, that table is basically useless. The final 'answer' file is
> usually less that 100megs, so backing that up is a trivial challenge :).
>
>  Ya, the SSDs are nice. I'm trying to collect this data to get a better
> idea if it's worth it to look at the SAN flash arrays that are on the
> market. The bad thing is this use case is very unique compared to what
> these SAN arrays are designed for (typically high end Oracle databases), so
> it's difficult for their engineers to fully grasp what our need is. I was
> hoping that being able to hand them a chart package that shows how all
> these metrics vary over the course of the problem would give them a better
> idea of what our need is, but generating these metrics is becoming an
> adventure in it's own right.
>
>  The box that has the SSDs in it has 4 K10's doing the computation. Hence
> the need for fast data delivery :).
>
>
> On Thu, Dec 12, 2013 at 3:14 PM, David <ainut at knology.net> wrote:
>
>>  You might write a separate program whose only function is to write files
>> of that size and do the timings for you.
>>
>> Btw, you are aware that there are only so many writes you can do to SSD
>> before they're used up, right?  If this is mission critical data, you may
>> want to keep frequent backups on other media.
>>
>> Still, having huge SSD's, and several of them to boot, makes me jealous!
>> :)
>>
>>
>> David M.
>>
>>
>> Stephan Henning wrote:
>>
>>   The program will write out a file of variable size, it's based on the
>> problem being run. Currently, it writes out approximately 1.5TB for the
>> benchmark problem, most of that contained in a single file, much too large
>> for a ramdisk. Unfortunately, the problems have grown so large that they
>> can't be run in ram any more. This is a GPU accelerated program so this
>> file gets modified very heavily during the process of a run.
>>
>>  Current testing is being done on a Raid0 of 5c Crucial 960G SSDs. This
>> has proven to be significantly faster than the old array, but I am trying
>> to determine exactly how hard the disks are being hammered so I can try and
>> optimize the hardware configuration.
>>
>>  The program is compiled from source, but I'm not involved in that
>> process, I'd much rather try and piggyback something and monitor the
>> process than try and go in and have something added to source.
>>
>>  I'll add parted and gparted to my list of things to read up on, thanks.
>>
>>
>> On Thu, Dec 12, 2013 at 12:29 AM, David <ainut at knology.net> wrote:
>>
>>>  Excellent approach.
>>>
>>>
>>>
>>> Arthur wrote:
>>>
>>> How big are the files that you're dealing with?
>>> If they're small you can just make a ramdisk and try running everything
>>> in there.
>>> It's not a final solution, but between that and strace you should be
>>> able to see if that's really the issue or not.
>>>
>>>  Are you compiling from source?  If you are, then there are a bunch of
>>> debugging tools you can use as well as doing things like timing individual
>>> commands, and seeing how many times each line of code is run.
>>>
>>>
>>> On Wed, Dec 11, 2013 at 10:48 PM, Stephan Henning <shenning at gmail.com>wrote:
>>>
>>>> This is a RedHat6 Enterprise install.
>>>>
>>>>  I don't think htop has the data I need, but I'll check. I'm not
>>>> familiar with ntop and I didn't consider using trace for this, I'll check
>>>> that as well.
>>>>
>>>>  The goal is to record read/write rates and block sizes. I'm pretty
>>>> sure I am bottlenecking against the drive array, I'm hoping I can get some
>>>> definitive answers from this.
>>>>
>>>>
>>>> On Wed, Dec 11, 2013 at 6:01 PM, David <ainut at knology.net> wrote:
>>>>
>>>>>  ntop might do the trick, but not available in Fedora.
>>>>>
>>>>>
>>>>>
>>>>> David wrote:
>>>>>
>>>>> Can 'htop' show open files?
>>>>>
>>>>> For intensive live net data, look at WireShark for linux.
>>>>>
>>>>>
>>>>> David wrote:
>>>>>
>>>>> If that's what you're looking for, there are several (free) programs
>>>>> you could run from the command line in a separate window/screen while your
>>>>> program is running that give you all you're asking about.  Sort of an
>>>>> equivalent to Winblows "System Explorer."  What flavor or Linux are you
>>>>> using?
>>>>>
>>>>> David M.
>>>>>
>>>>>
>>>>> Devin Boyer wrote:
>>>>>
>>>>> Try something like "strace -T myapp" or "strace -T -c myapp"; they'll
>>>>> show the system calls being made and the amount of time spent in each.
>>>>>  It's slightly different information than iostat, but it may be useful in
>>>>> figuring out what and where your program is performing io access.
>>>>>
>>>>>
>>>>> On Wed, Dec 11, 2013 at 3:37 PM, Stephan Henning <shenning at gmail.com>wrote:
>>>>>
>>>>>>   No, iostat will normally just dump to the terminal window, but I'd
>>>>>> like to pipe it's output to a file so I can parse it later.
>>>>>>
>>>>>>  My end goal here is to be able to generate a log of iostat output
>>>>>> while I run this program, I'm trying to determine exactly how hard this
>>>>>> program is hitting my harddrive and at what points during it's run does it
>>>>>> access the drive the most frequently.
>>>>>>
>>>>>>  I've done something similar in bash before, but it is rather clunky.
>>>>>>
>>>>>>  I'll take a look at exec and see if I can use it.
>>>>>>
>>>>>>
>>>>>> On Wed, Dec 11, 2013 at 4:46 PM, David <ainut at knology.net> wrote:
>>>>>>
>>>>>>>  Do you need to do anything with the results or just need them
>>>>>>> displayed?
>>>>>>> If you need to manipulate the results, consider using Perl,
>>>>>>> or if C or C++,
>>>>>>> in your 'exec' call, pipe the output to a file, then just read that
>>>>>>> file into your program.
>>>>>>> Ain't UNIX great?
>>>>>>>
>>>>>>>
>>>>>>> David M.
>>>>>>>
>>>>>>>
>>>>>>> Stephan Henning wrote:
>>>>>>>
>>>>>>>  I'd like to take some metrics with iostat while I have a specific
>>>>>>> program running, is there a way to wrap iostat around another program (it
>>>>>>> is called from the command line) so that iostat ends when the program
>>>>>>> finishes running?
>>>>>>>
>>>>>>>  I know I can do it with a bash script, but I'm hoping for a more
>>>>>>> elegant solution.
>>>>>>>
>>>>>>>
>>>>>>>  _______________________________________________
>>>>>>> General mailing listGeneral at lists.makerslocal.orghttp://lists.makerslocal.org/mailman/listinfo/general
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> General mailing list
>>>>>>> General at lists.makerslocal.org
>>>>>>> http://lists.makerslocal.org/mailman/listinfo/general
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> General mailing list
>>>>>> General at lists.makerslocal.org
>>>>>> http://lists.makerslocal.org/mailman/listinfo/general
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> General mailing listGeneral at lists.makerslocal.orghttp://lists.makerslocal.org/mailman/listinfo/general
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> General mailing listGeneral at lists.makerslocal.orghttp://lists.makerslocal.org/mailman/listinfo/general
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> General mailing listGeneral at lists.makerslocal.orghttp://lists.makerslocal.org/mailman/listinfo/general
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> General mailing list
>>>>> General at lists.makerslocal.org
>>>>> http://lists.makerslocal.org/mailman/listinfo/general
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> General mailing list
>>>> General at lists.makerslocal.org
>>>> http://lists.makerslocal.org/mailman/listinfo/general
>>>>
>>>
>>>
>>>
>>>  --
>>> Sincerely,
>>> Arthur Moore
>>> (256) 277-1001
>>>
>>>
>>> _______________________________________________
>>> General mailing listGeneral at lists.makerslocal.orghttp://lists.makerslocal.org/mailman/listinfo/general
>>>
>>>
>>>
>>> _______________________________________________
>>> General mailing list
>>> General at lists.makerslocal.org
>>> http://lists.makerslocal.org/mailman/listinfo/general
>>>
>>
>>
>>
>> _______________________________________________
>> General mailing listGeneral at lists.makerslocal.orghttp://lists.makerslocal.org/mailman/listinfo/general
>>
>>
>>
>> _______________________________________________
>> General mailing list
>> General at lists.makerslocal.org
>> http://lists.makerslocal.org/mailman/listinfo/general
>>
>
>
>
> _______________________________________________
> General mailing listGeneral at lists.makerslocal.orghttp://lists.makerslocal.org/mailman/listinfo/general
>
>
>
> _______________________________________________
> General mailing list
> General at lists.makerslocal.org
> http://lists.makerslocal.org/mailman/listinfo/general
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.makerslocal.org/pipermail/general/attachments/20131212/232d80fa/attachment-0001.html>