[ML-General] Cluster Computing
david
ainut at knology.net
Tue Feb 3 16:43:17 CST 2015
Think of the Challenge!
On 02/03/2015 04:28 PM, Shae wrote:
> The PS3 has a number of downsides, which can mostly be summarized with
> the lack of support since Sony removed the OtherOS option.
>
> Secondarily, it's difficult to write a compiler smart enough to take
> good advantage of the PSUs on the Cell Broadband Engine.
>
> If you decide you want to learn about doing highly parallel computing
> on the Cell, I can bring my BladeCenter to the shop and fire up one
> (or many) of my QS20 blades for experiments.
> But honestly, the Cell is lots of trouble without so much benefit.
>
> On Tue Feb 03 2015 at 4:14:46 PM david <ainut at knology.net
> <mailto:ainut at knology.net>> wrote:
>
> IF and it's a big IF, your problem lends itself to a pure
> distributed processing paradigm (think Cray...), a very low cost
> setup with phenomenal compute speeds is the Sony Playstation 3,
> believe it or not. You can find them really cheap nowadays,.
> Network a few of them together, install LINUX/UNIX on them (might
> be available out there) and setup the Cray-type compiler (from
> SGI) and you'll have a honking system. In the right problem
> domain, 5 of those would outperform hundreds of the pico-computers.
>
>
>
>
>
> On 02/03/2015 03:57 PM, Stephan Henning wrote:
>> There was a group that did it a while back. I want to say they
>> did it with Atom processors. Ended up with 400+ nodes in a 10U
>> rack I think.
>>
>> On Tue, Feb 3, 2015 at 3:55 PM, Erik Arendall
>> <earendall at gmail.com <mailto:earendall at gmail.com>> wrote:
>>
>> This would be a cool project to develop a module board that
>> contains the cpu/gpu of choice and required ram for use. then
>> the modules could plug in to a supervisory control node.
>>
>> On Tue, Feb 3, 2015 at 3:50 PM, Stephan Henning
>> <shenning at gmail.com <mailto:shenning at gmail.com>> wrote:
>>
>> Hey Hunter,
>>
>> Well, with the Edison, it wouldn't be 27 devices, it
>> would be closer to 400 :)
>>
>> I /think/ I can fit 27 mini-itx motherboards in a 4U
>> chassis (maybe only 21-24, depending on heatsink height).
>> For the raspi's or the Edisons to be viable they would
>> need to beat that baseline on a flop/watt vs $$
>> comparison. Even in that case, the low RAM amount limits
>> their usefulness.
>>
>> On Tue, Feb 3, 2015 at 3:44 PM, Hunter Fuller
>> <hfuller at pixilic.com <mailto:hfuller at pixilic.com>> wrote:
>>
>> 27 devices in a metal box will work fine, provided
>> there is also a fairly robust AP in that box. I would
>> personally still lean toward USB Ethernet though. But
>> that increases your devices size and complexity... Hm.
>>
>> As far as PXE boot, since there is no wired Ethernet
>> available, I doubt that is a thing. However, you can
>> Mount the internal storage as /boot, and have a
>> script run that rsyncs the /boot fs between the boxes
>> and a server. The rest can be achieved by using an
>> NFS volume as your root partition. This setup is
>> commonly done on armies of raspberry pis.
>>
>> There wouldn't be much difference between original
>> prep on this and originally preparing several SD
>> cards. In one case, you have to connect each device
>> to a provisioning station. In the other case,you
>> connect each SD card to the same station. Not much
>> different, and once you boot one time, you can do the
>> maintenance in an automated fashion across all nodes.
>>
>> On Jan 23, 2015 9:36 AM, "Michael Carroll"
>> <carroll.michael at gmail.com
>> <mailto:carroll.michael at gmail.com>> wrote:
>>
>> Stephan,
>>
>> I didn't realize that the Edison was wifi-only.
>> I'm interested to hear how 27 wifi devices in a
>> metal box will work?
>>
>> Also, do you know if the edison can pxeboot? I
>> think that's the best approach for booting a
>> whole bunch of homogeneous computers, it would
>> certainly be more maintenance overhead without
>> that capability.
>>
>> ~mc
>>
>>
>> On Thu, Jan 22, 2015 at 11:04 PM, Stephan Henning
>> <shenning at gmail.com <mailto:shenning at gmail.com>>
>> wrote:
>>
>> @Erik
>> Well, the raspi and beaglebone have less ram
>> than the Edison. I'll have to take a look at
>> the Rock, the Pro version offers 2gb, but
>> since the Edison is an x86 platform it is
>> advantageous in many ways.
>>
>> @Tim
>> Ya, that looks very similar. I'll give it a
>> read through in the morning. I'll make sure
>> to keep you updated.
>>
>> On Thu, Jan 22, 2015 at 10:11 PM, Erik
>> Arendall <earendall at gmail.com
>> <mailto:earendall at gmail.com>> wrote:
>>
>> Not sure of your ram requirements, but
>> there are options in the RasPI,
>> beaglebone black, and check out Radxa Rock.
>>
>> http://radxa.com/Rock
>>
>> Erik
>>
>> On Jan 22, 2015 10:07 PM, "Tim H"
>> <crashcartpro at gmail.com
>> <mailto:crashcartpro at gmail.com>> wrote:
>>
>> This sounds like a fun project!
>> Reminds me of this guy:
>> http://www.pcworld.idg.com.au/article/349862/seamicro_cloud_server_sports_512_atom_processors/
>> (cluster of low power processors in a
>> single box)
>>
>> I'd also been kicking a similar idea
>> around for the last year, but no real
>> ability to do it, so I'd love to see
>> your progress!
>> -Tim
>>
>> On Thu, Jan 22, 2015 at 9:10 PM,
>> Stephan Henning <shenning at gmail.com
>> <mailto:shenning at gmail.com>> wrote:
>>
>> In some ways, yes. The biggest
>> limitation with the Edison for me
>> is the ram. While there is a lot
>> that we could run on it, it's
>> restricts them enough that I
>> don't think it would be as
>> useful, which changes alters the
>> true 'cost' of the setup.
>>
>> Granted, you could probably fit a
>> few hundred of them in a 4U
>> chassis. It would be an
>> interesting experiment in
>> integration though since they
>> have no ethernet interface, only
>> wireless.
>>
>> On Thu, Jan 22, 2015 at 9:02 PM,
>> Erik Arendall
>> <earendall at gmail.com
>> <mailto:earendall at gmail.com>> wrote:
>>
>> I've often kicked the idea
>> around doing this with
>> Arduinos and FPGAs. I guess
>> you could also do it with
>> Intel Edison modules. Cost
>> wise the Edison modules would
>> better than a PC.
>>
>> Erik
>>
>> On Jan 22, 2015 6:44 PM,
>> "Stephan Henning"
>> <shenning at gmail.com
>> <mailto:shenning at gmail.com>>
>> wrote:
>>
>> @mc
>> Both. If I start to scale
>> this to a large number of
>> nodes I can foresee many
>> headaches if I can't
>> easily push modifications
>> and updates. From the job
>> distribution side, it
>> would be great to
>> maintain compatibility
>> with condor, I'm just
>> unsure how well it will
>> operate if it has to hand
>> jobs off to the head node
>> that then get distributed
>> out further.
>>
>> @ Brian
>> Our current cluster is
>> made up of discrete
>> machines only about 20
>> nodes. Many of the nodes
>> are actual user
>> workstations that are
>> brought in when inactive.
>> There is no uniform
>> provisioning method.
>> Every box has a slightly
>> different hardware
>> configuration. Thankfully
>> we do a pretty good job
>> keeping all required
>> software aligned to the
>> sam version.
>>
>> The VM idea is
>> interesting. I hadn't
>> considered that. I will
>> need to think on that and
>> how I might be able to
>> implement it.
>>
>> @david
>> Yup, I'm fully aware this
>> level of distributed
>> computing is only good
>> for specific cases. I
>> understand your position,
>> thanks.
>>
>> -stephan
>>
>> ---———---•---———---•---———---
>> Sent from a mobile
>> device, please excuse the
>> spelling and brevity.
>>
>> On Jan 22, 2015, at 5:54
>> PM, Brian Oborn
>> <linuxpunk at gmail.com
>> <mailto:linuxpunk at gmail.com>>
>> wrote:
>>
>>> I would be tempted to
>>> just copy what the
>>> in-house cluster uses
>>> for provisioning. That
>>> will save you a lot of
>>> time and make it easier
>>> to integrate with the
>>> larger cluster if you
>>> choose to do so.
>>> Although it can be
>>> tempting to get hardware
>>> in your hands, I've done
>>> a lot of work with
>>> building all of the
>>> fiddly Linux bits
>>> (DHCP+TFTP+root on
>>> NFS+NFS home) in several
>>> VMs before moving to
>>> real hardware. You can
>>> set up a private VM-only
>>> network between your
>>> head node and the slave
>>> nodes and work from there.
>>>
>>> On Thu, Jan 22, 2015 at
>>> 5:31 PM, Michael Carroll
>>> <carroll.michael at gmail.com
>>> <mailto:carroll.michael at gmail.com>>
>>> wrote:
>>>
>>> So is your concern
>>> with provisioning
>>> and setup or with
>>> actual job distribution?
>>>
>>> ~mc mobile
>>>
>>> On Jan 22, 2015, at
>>> 17:15, Stephan
>>> Henning
>>> <shenning at gmail.com
>>> <mailto:shenning at gmail.com>>
>>> wrote:
>>>
>>>> This is a side
>>>> project for the
>>>> office. Sadly, most
>>>> of this type of
>>>> work can't be
>>>> farmed out to
>>>> external clusters,
>>>> otherwise we would
>>>> use it for that. We
>>>> do currently
>>>> utilize AWS for
>>>> some of this type
>>>> work, but only for
>>>> internal R&D.
>>>>
>>>> This all started
>>>> when the Intel
>>>> Edison got
>>>> released. Some of
>>>> us were talking
>>>> about it one day
>>>> and realized that
>>>> it /might/ have
>>>> /just
>>>> enough/ processing
>>>> power and ram to
>>>> handle some of our
>>>> smaller problems.
>>>> We've talked about
>>>> it some more and
>>>> the discussion has
>>>> evolved to the
>>>> point where I've
>>>> been handed some
>>>> hours and a small
>>>> amount of funding
>>>> to try and
>>>> implement a
>>>> 'cluster-in-a-box'.
>>>>
>>>> The main idea being
>>>> to rack a whole
>>>> bunch of mini-itx
>>>> boards on edge into
>>>> a 4U chassis (yes,
>>>> they will fit).
>>>> Assuming a 2"
>>>> board-board
>>>> clearance across
>>>> the width of the
>>>> chassis and 1"
>>>> spacing
>>>> back-to-front down
>>>> the depth of a box,
>>>> I think I could fit
>>>> 27 boards into a
>>>> 36" deep chassis,
>>>> with enough room
>>>> for the power
>>>> supplies and
>>>> interconnects.
>>>>
>>>> Utilizing embedded
>>>> motherboards with
>>>> Atom C2750 8-core
>>>> CPU's and 16gb of
>>>> ram per board, that
>>>> should give me a
>>>> pretty substantial
>>>> cluster to play
>>>> with. Obviously I
>>>> am starting small,
>>>> probably with two
>>>> or three boards
>>>> running Q2900
>>>> 4-core cpus until I
>>>> can get the
>>>> software side
>>>> worked out.
>>>>
>>>> The
>>>> software-infrastructure
>>>> side is the part
>>>> I'm having a hard
>>>> time with. While
>>>> there are options
>>>> out there for how
>>>> to do this, they
>>>> are all relatively
>>>> involved and there
>>>> isn't an obvious
>>>> 'best' choice to me
>>>> right now.
>>>> Currently our
>>>> in-house HPC
>>>> cluster utilizes
>>>> HTCondor for it's
>>>> backbone, so I
>>>> would like to
>>>> maintain some sort
>>>> of connection to
>>>> it. Otherwise, I'm
>>>> seeing options in
>>>> the Beowulf and
>>>> Rocks areas that
>>>> could be useful,
>>>> I'm just not sure
>>>> where to start in
>>>> all honesty.
>>>>
>>>> At the end of the
>>>> day this needs to
>>>> be relatively easy
>>>> for us to manage
>>>> (time spent working
>>>> on the cluster is
>>>> time spent not
>>>> billing the
>>>> customer) while
>>>> being easy enough
>>>> to add notes to,
>>>> assuming this is a
>>>> success and I get
>>>> the OK to expand it
>>>> to a full 42U racks
>>>> worth.
>>>>
>>>>
>>>> Our current cluster
>>>> is almost always
>>>> fully utilized.
>>>> Currently we've got
>>>> about a 2 month
>>>> backlog of jobs on it.
>>>>
>>>>
>>>> On Thu, Jan 22,
>>>> 2015 at 4:55 PM,
>>>> Brian Oborn
>>>> <linuxpunk at gmail.com <mailto:linuxpunk at gmail.com>>
>>>> wrote:
>>>>
>>>> If you can keep
>>>> your
>>>> utilization
>>>> high, then your
>>>> own hardware
>>>> can be much
>>>> more cost
>>>> effective.
>>>> However, if you
>>>> end up paying
>>>> depreciation
>>>> and maintenance
>>>> on a cluster
>>>> that's doing
>>>> nothing most of
>>>> the time you'd
>>>> be better off
>>>> in the cloud.
>>>>
>>>> On Thu, Jan 22,
>>>> 2015 at 4:50
>>>> PM, Michael
>>>> Carroll
>>>> <carroll.michael at gmail.com
>>>> <mailto:carroll.michael at gmail.com>>
>>>> wrote:
>>>>
>>>> Depending
>>>> on what you
>>>> are going
>>>> to do, it
>>>> seems like
>>>> it would
>>>> make more
>>>> sense to
>>>> use AWS or
>>>> Digital
>>>> Ocean these
>>>> days,
>>>> rather than
>>>> standing up
>>>> your own
>>>> hardware.
>>>> Maintaining
>>>> your own
>>>> hardware
>>>> sucks.
>>>>
>>>> That being
>>>> said, if
>>>> you are
>>>> doing
>>>> something
>>>> that
>>>> requires
>>>> InfiniBand,
>>>> then
>>>> hardware is
>>>> your only
>>>> choice :)
>>>>
>>>> ~mc
>>>>
>>>> On Thu, Jan
>>>> 22, 2015 at
>>>> 4:43 PM,
>>>> Joshua
>>>> Pritt
>>>> <ramgarden at gmail.com
>>>> <mailto:ramgarden at gmail.com>>
>>>> wrote:
>>>>
>>>> My
>>>> friends
>>>> and I
>>>> installed
>>>> a
>>>> Beowulf
>>>> cluster
>>>> on a
>>>> closet
>>>> full of
>>>> Pentium
>>>> 75 Mhz
>>>> machines we
>>>> were
>>>> donated
>>>> just
>>>> for fun
>>>> many
>>>> years
>>>> ago
>>>> back
>>>> when
>>>> Beowulf
>>>> was
>>>> just
>>>> getting
>>>> popular.
>>>> We
>>>> never
>>>> figured
>>>> out
>>>> anything to
>>>> do with
>>>> it
>>>> though...
>>>>
>>>> On Thu,
>>>> Jan 22,
>>>> 2015 at
>>>> 5:31
>>>> PM,
>>>> Brian
>>>> Oborn
>>>> <linuxpunk at gmail.com
>>>> <mailto:linuxpunk at gmail.com>>
>>>> wrote:
>>>>
>>>> In
>>>> my
>>>> previous
>>>> job
>>>> I
>>>> set
>>>> up
>>>> several
>>>> production
>>>> Beowulf
>>>> clusters,
>>>> mainly
>>>> for
>>>> particle
>>>> physics
>>>> simulations
>>>> and
>>>> this has
>>>> been an
>>>> area of
>>>> intense
>>>> interest
>>>> for
>>>> me.
>>>> I
>>>> would
>>>> be
>>>> excited
>>>> to
>>>> help you
>>>> out
>>>> and
>>>> I
>>>> think
>>>> I
>>>> could
>>>> provide
>>>> some good
>>>> assistance.
>>>>
>>>>
>>>> Brian
>>>> Oborn
>>>> (aka bobbytables)
>>>>
>>>>
>>>> On
>>>> Thu, Jan
>>>> 22,
>>>> 2015 at
>>>> 4:25 PM,
>>>> Stephan
>>>> Henning
>>>> <shenning at gmail.com
>>>> <mailto:shenning at gmail.com>>
>>>> wrote:
>>>>
>>>> Does
>>>> anyone
>>>> on
>>>> the
>>>> mailing
>>>> list
>>>> have
>>>> any
>>>> experience
>>>> with
>>>> setting
>>>> up
>>>> a
>>>> cluster
>>>> computation
>>>> system?
>>>> If
>>>> so
>>>> and
>>>> you
>>>> are
>>>> willing
>>>> to
>>>> humor
>>>> my
>>>> questions,
>>>> I'd
>>>> greatly
>>>> appreciate
>>>> a
>>>> few
>>>> minutes
>>>> of
>>>> your
>>>> time.
>>>>
>>>>
>>>> -stephan
>>>>
>>>> _______________________________________________
>>>> General
>>>> mailing
>>>> list
>>>> General at lists.makerslocal.org
>>>> <mailto:General at lists.makerslocal.org>
>>>> http://lists.makerslocal.org/mailman/listinfo/general
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> General
>>>> mailing
>>>> list
>>>> General at lists.makerslocal.org
>>>> <mailto:General at lists.makerslocal.org>
>>>> http://lists.makerslocal.org/mailman/listinfo/general
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> General
>>>> mailing
>>>> list
>>>> General at lists.makerslocal.org
>>>> <mailto:General at lists.makerslocal.org>
>>>> http://lists.makerslocal.org/mailman/listinfo/general
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> General
>>>> mailing list
>>>> General at lists.makerslocal.org
>>>> <mailto:General at lists.makerslocal.org>
>>>> http://lists.makerslocal.org/mailman/listinfo/general
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> General mailing
>>>> list
>>>> General at lists.makerslocal.org
>>>> <mailto:General at lists.makerslocal.org>
>>>> http://lists.makerslocal.org/mailman/listinfo/general
>>>>
>>>>
>>>> _______________________________________________
>>>> General mailing list
>>>> General at lists.makerslocal.org
>>>> <mailto:General at lists.makerslocal.org>
>>>> http://lists.makerslocal.org/mailman/listinfo/general
>>>
>>> _______________________________________________
>>> General mailing list
>>> General at lists.makerslocal.org
>>> <mailto:General at lists.makerslocal.org>
>>> http://lists.makerslocal.org/mailman/listinfo/general
>>>
>>>
>>> _______________________________________________
>>> General mailing list
>>> General at lists.makerslocal.org
>>> <mailto:General at lists.makerslocal.org>
>>> http://lists.makerslocal.org/mailman/listinfo/general
>>
>> _______________________________________________
>> General mailing list
>> General at lists.makerslocal.org
>> <mailto:General at lists.makerslocal.org>
>> http://lists.makerslocal.org/mailman/listinfo/general
>>
>>
>> _______________________________________________
>> General mailing list
>> General at lists.makerslocal.org
>> <mailto:General at lists.makerslocal.org>
>> http://lists.makerslocal.org/mailman/listinfo/general
>>
>>
>>
>> _______________________________________________
>> General mailing list
>> General at lists.makerslocal.org
>> <mailto:General at lists.makerslocal.org>
>> http://lists.makerslocal.org/mailman/listinfo/general
>>
>>
>>
>> _______________________________________________
>> General mailing list
>> General at lists.makerslocal.org
>> <mailto:General at lists.makerslocal.org>
>> http://lists.makerslocal.org/mailman/listinfo/general
>>
>>
>> _______________________________________________
>> General mailing list
>> General at lists.makerslocal.org
>> <mailto:General at lists.makerslocal.org>
>> http://lists.makerslocal.org/mailman/listinfo/general
>>
>>
>>
>> _______________________________________________
>> General mailing list
>> General at lists.makerslocal.org
>> <mailto:General at lists.makerslocal.org>
>> http://lists.makerslocal.org/mailman/listinfo/general
>>
>>
>>
>> _______________________________________________
>> General mailing list
>> General at lists.makerslocal.org
>> <mailto:General at lists.makerslocal.org>
>> http://lists.makerslocal.org/mailman/listinfo/general
>>
>>
>> _______________________________________________
>> General mailing list
>> General at lists.makerslocal.org
>> <mailto:General at lists.makerslocal.org>
>> http://lists.makerslocal.org/mailman/listinfo/general
>>
>>
>>
>> _______________________________________________
>> General mailing list
>> General at lists.makerslocal.org
>> <mailto:General at lists.makerslocal.org>
>> http://lists.makerslocal.org/mailman/listinfo/general
>>
>>
>>
>> _______________________________________________
>> General mailing list
>> General at lists.makerslocal.org
>> <mailto:General at lists.makerslocal.org>
>> http://lists.makerslocal.org/mailman/listinfo/general
>>
>>
>>
>>
>> _______________________________________________
>> General mailing list
>> General at lists.makerslocal.org <mailto:General at lists.makerslocal.org>
>> http://lists.makerslocal.org/mailman/listinfo/general
>
> --
> This headspace for rent
>
> _______________________________________________
> General mailing list
> General at lists.makerslocal.org <mailto:General at lists.makerslocal.org>
> http://lists.makerslocal.org/mailman/listinfo/general
>
>
>
> _______________________________________________
> General mailing list
> General at lists.makerslocal.org
> http://lists.makerslocal.org/mailman/listinfo/general
--
This headspace for rent
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.makerslocal.org/pipermail/general/attachments/20150203/33e3146b/attachment.html>
More information about the General
mailing list