# Xgrid Project



## Krevinek (Jan 7, 2004)

Well, now that we have Xgrid... which does nothing, why not do something just because it is there? I remember helping someone attempt the factorial of 1 million, but failing because we didn't have enough time or computing power. Here is the trick I am attempting... I want to get an Xgrid setup of a few dozen machines from MacOSX.com users and see how quickly we can crack the factorial of 1 million! It would be an interesting attempt, and if it goes smoothly, we can attempt the factorial of larger numbers and see what type of speed record for amateur clusters for processing of large numbers we can set.

I am currently writing the code needed to attempt this along with a nice clean plug-in for the attempt.

Anyone interested in contributing a machine or two to the cluster? Right now I mostly need machines that can run 24/7 on broadband so I can run some speed tests using the Mandlebrot example that comes with Xgrid to see performance levels we can achieve over the internet.

To contribute to the project:

1) Download and install Xgrid
2) Open System Preferences and the new Xgrid panel
3) In the Agent tab, select 'Connect to specific host' and fill in the text field with: mzed.homeip.net
4) You can configure when your machine can accept tasks, although selecting 'Always' will definitely help out more.
5) Open the Agent Security tab
6) Set your password to: megamacgrid
7) Back in the Agent tab, start the agent, it will automagically join the cluster.

This password is simply to ensure you want to join the cluster, and don't worry about security over the network, since SSL is used for communications. You can select the 'Xgrid' screen saver if you wish, which will devote all your resources to Xgrid while you are asleep.


----------



## Arden (Jan 7, 2004)

I suppose I could turn sleep off for my iMac for a while... would you be able to run stuff while it's got a screen saver going?

I don't know how much power you can squeeze out of a 400 Mhz G3...


----------



## Krevinek (Jan 7, 2004)

Yeah, there is actually a screensaver for Xgrid that allows you to do just that... and don't be afraid of the computing power, so far I am running a 400Mhz G4 8600 and my Lombard 333 in a cluster fairly successfully... a 400Mhz G3 would get similar performance as my 8600 since the factorial code isn't going to be optimized for altivec. I am going to post up instructions in the first post soon so people can setup Xgrid to help this project.

And don't worry about the security risk, Xgrid cluster apps are run as 'nobody', which means they have no access on your system, only read-access to the files needed to do the computation.


----------



## Captain Code (Jan 7, 2004)

I'll give it a try.  1.4GHz G4.

XGrid looks cool but other than this I wouldn't have any use for it.

The thing is, it says it uses Rendezvous, but does it also let you add clusters manually?

Also, how does it split up the work load?  It'd have to give us a large chunk to do and then collect the results like Seti@home or something because if it worked like a traditional cluster you'd need more bandwidth to get good performance.


----------



## Krevinek (Jan 7, 2004)

Captain Code said:
			
		

> I'll give it a try.  1.4GHz G4.
> 
> XGrid looks cool but other than this I wouldn't have any use for it.
> 
> ...



Well, right now I have my GF contributing the use of her iBook currently and I was running a Mandlebrot test using my G4 8600, my Lombard, and her iBook, all done over broadband. We were averaging about 1.2Ghz for 60% of the time the Mandlebrot task was running, not too shabby for 3 older machines.

The task I am designing will split the workload into about 100 tasks, then do 25, then 5, then compute those 5 on a single task on the cluster. All that is being sent back and forth is text, and a plug-in people would have to install for this to work. Which means this is a sort of task that is actually pretty parallel already, and can VERY easily be split across the internet using Xgrid.

I think this would be a good test and give Apple some interesting feedback in the end... I am also looking at ways to do a minor re-write of OpenShiva to allow it to utilize Xgrid to do MPEG-4 conversions (at least for home use for myself, but think about what we could do with a few internet-friendly Xgrid plug-ins and a massive amateur cluster?)


----------



## Captain Code (Jan 7, 2004)

It sounds like it'd work pretty well then.  I have a 9500 on 10.1.5 as well that is just a mail server.  Will XGrid run on it?  I haven't tried yet, but if I could set it to be absolutly lowest priority then I'd install it on there as well.  

I managed to crash XGrid BLAST the first time I ran it.  Tried starting a local service and it said permission denied.  I tried connecting to a localhost service cause I thought maybe it was already running from running XGrid before and it crashed.


----------



## Krevinek (Jan 7, 2004)

XGrid needs 10.2.8 or later... the agent (the app that actually turns your system into a node) doesn't need any apps to run... you configure and start it from your System Preferences. I am going to re-tweak the controller node (my 8600 G4) to use a seperate password which will be given to those who participate in the project so you can have access to the cluster created and mess around with it while I am writing and testing the factorial plug-in.

I use my 8600 as a mail server as well, and it is configured to always accept tasks... I haven't noticed anything unusual since doing this with my mail services.


----------



## mindbend (Jan 7, 2004)

1. Very cool idea. Good luck. Sorry, no machines I can release for this project.

2. When you're done with this, use your talent to write some plug-ins to let me use After Effects network rendering easier, mpeg-2 encoding via Compressor, Maya rendering, etc.!!!! 

3. Does anyone else remember reading that IBM press release from maybe half a year ago? It talked about how they were going to bank everything on a similar technology to Xgrid and develop a massive infrastructure of parallel computing. Their business model was then going to switch to providing "client" machines that tapped into this system. Basically Xgrid on steroids. Anyone have a link for that?

4. Can't wait to see where all of this leads. Great concept with so many possibilities.


----------



## Arden (Jan 7, 2004)

Alrighty then, tell me when it's ready and give me a download link and I'll stick it in.


----------



## powermac (Jan 8, 2004)

I will give it a try!!


----------



## octane (Jan 8, 2004)

Hang on am minute!

How do we know this guy is for real?

For all we know, he could be passively enlisting you guys into a digital bank job!

He could be pooling your computer resources to crack some 40bit encryption on some transatlantic wire transfers.

Sorry about that. I've just been watching the X-Files box set back to back since christmas day. I'm feeling a little paranoid...


----------



## Captain Code (Jan 8, 2004)

Ah, so how'd you get 10.2.8 on your 8500?  I'd like to get my 9500 up to speed as much as I can but it kept KPing trying to boot off the 10.2 disk.


----------



## Captain Code (Jan 8, 2004)

Captain Code said:
			
		

> Ah, so how'd you get 10.2.8 on your 8500?  I'd like to get my 9500 up to speed as much as I can but it kept KPing trying to boot off the 10.2 disk.




Maybe the G4 400 upgrade is enough to get it to work?


----------



## macmikey (Jan 8, 2004)

I'd like to join this, but my prefs pane gives up an err=1 window when it opens and again when I click start agent. I reinstalled it, but no go. App runs fine and does local demos no problem.

Also, trying to join from app gives me a authentication failed message.

Any one else seeing this?     

PB G4 1.25 Al Book, 512MB ram

mike


----------



## cybergoober (Jan 8, 2004)

I'm in. For now at least. We'll see if the network admins shut me down, as they shut down SETI@home

Oh, yeah.

G5 1.6 GHz on a fat-@ss pipe.
XGrid set to "only when idle", though. Sorry, need my cycles when I'm actually using my machine.


----------



## Krevinek (Jan 8, 2004)

Captain Code said:
			
		

> Maybe the G4 400 upgrade is enough to get it to work?



Yup, Ryan of XPostFacto couldn't get 10.2 or later running on non-G3/4 upgraded machines.

For those worried about the project, the plug-in and binary I will be distributing for the project will also have the source for it, for those who can read a little C... mostly the app will be code to multiply two big strings together, nothing more.


----------



## Captain Code (Jan 8, 2004)

DOH, that's what I thought.  I'll look at the code and confirm that it's OK for people.


----------



## Krevinek (Jan 8, 2004)

Okay, I have a design for the command-line tool that will be running on people's machines, and will write it up over the next 2 hours or so. I will post up a tarball for people who wish to peruse the code as well as people to help audit the code. I may or may not make an altivec-enabled version of the code (8 digits per loop versus 1... tasty) by then, which would allow those with a G4/G5 system to help out with to crunch many more pieces. I also will be scouring the plug-in API so see if I can submit jobs in rounds (seems very possible based off the Mandlebrot task), which would help split more of the task (multiplying very large numbers, even if it is only 2, using the method I am working with, takes a lot of processing power).

So far 5 machines have connected to the grid for a period of time:
My control node, up 24/7
My laptop, up when I am
My gf's iBook, up when she is at home
A 1.8Ghz G5 system, from... someone.
An 800Mhz iBook, from... someone.

I will configure the grid so that I can allow others to connect to it and tinker with the demos here shortly, probably the same time I submit the code for auditing.


----------



## Captain Code (Jan 8, 2004)

XGrid seems like it's the basis for the distributed build in XCode.  Seems like they just took what they started with XCode and made it capable of using plugins for custom applications.


----------



## Captain Code (Jan 8, 2004)

I'm all set up in System Prefs but it doesn't seem to be doing anything because the CPU usage is all going to Seti@home.

I also tried to connect via XGrid the program and it said authentication failed.  Is it offline now or something?


----------



## Krevinek (Jan 9, 2004)

No, it simply uses a different password to access the controller at the moment... I will change it over to the same password as the agents use for awhile, then change it to something else when it comes time to run the attempt, as it would be very bad to have the attempt running along-side someone tinkering with Mandlebrot. 

Anyways, it seems interesting that Xgrid runs as 'nobody' as well as every process it starts. There is no real access to the machine that the tasks run on, other than the usual read access. This is very good from the standpoint of security, because it means that it is pretty improbable that Xgrid can be used to gain elevated access on a remote machine, especially with SSL and passwords used to shield Xgrid connections from the world of snoopers.


----------



## Arden (Jan 9, 2004)

Okay, I've got it linked in and everything.  What about that screen saver?


----------



## Studio Zero (Jan 9, 2004)

I am definitely in.  I will add a 1.25 G4 powerbook to the mix when I'm not using it.


----------



## Studio Zero (Jan 9, 2004)

I joined the cluster.  Do I have to access the controller to see the cool little speedometer thing?


----------



## Krevinek (Jan 13, 2004)

Hmm, sorry about the late reply everyone... I have been having trouble getting my tool to produce proper multiplications, and since I am writing this code from memory (from a class 6 years ago), it is slow going.

Working or not-working, I will have the current code for the tool posted tonight.


----------



## Krevinek (Jan 13, 2004)

Attached is the current code... as I promised... the plug-in is nowhere near complete, the UI is just designed and the components of the UI attached to the code... actions still need to be attached and written properly, as well as refined behavior for the job itself. The tool currently just inputs 2 numbers and multiplies them. It can take numbers of ANY number of digits (up to 2 billion digits anyways), and multiply them accurately.


----------



## Captain Code (Jan 14, 2004)

Was the XGrid plugin supposed to compile?
XCode can't find the definition for _commandField:

    NSString *commandString = [_commandField stringValue];
in the PluginJobViewController.m

I guess it's probably just an outlet that's not in the header file or something..


----------



## Krevinek (Jan 14, 2004)

Nope, the plugin is nowhere near ready yet, but I threw it in because I said I would. I have an updated version of the tool working on my machine which can add numbers and multiply them accurately, just need to add subtraction and that SHOULD be enough for the program... but division/remainder might be needed as well.

The plugin is currently being bastardized from the Shell plugin example, so not everything is truly plugged in yet.

Right now I am just focusing on getting the tool ready first, because the plugin needs to know how to call the tool before it is of any use beyond the UI design work I have done so far.

I did a little checking into how large the result would be using ballpark estimation: ~2-3MB. That is 2-3 million digits... definitely not a small feat if this code works as advertised.


----------



## Captain Code (Jan 15, 2004)

OK, I wasn't sure if you posted one that was semi working or not.

So does an XGrid node just download the program automatically and work on a set of data?  I should really look into this more and look at the code more cause I don't have a good idea of how this works exactly.


----------



## Krevinek (Jan 15, 2004)

The plugin with the tool would get installed onto each person's machine (of their free will)... the plugin when run from my machine will then only use nodes that have the tool installed with the plugin. (At least that is how it supposed to work)


----------



## Arden (Jan 16, 2004)

How do we know when it's working?  I left my computer running a few times since downloading Xgrid, you know, "just in case" but I have no idea if it's actually doing anything or just sitting there spinning its hard drive.

I will have to boot back into OS X to get it to do something, of course...


----------



## Krevinek (Jan 16, 2004)

Right now, it isn't... I am still working on the code that will make the project work.


----------



## Arden (Feb 8, 2004)

Got an update on this?  I downloaded the plugin you provided, but I'm not sure what to do with it.


----------



## Krevinek (Feb 8, 2004)

Actually, nothing... I apologize, but college courses and final projects have gotten in my way, as well as work. I probably won't have an update for this for another month as I need to focus on other things... the grid will be shutdown for now, so people aren't wasting CPU cycles on an idle grid.


----------

