What are you most chuffed about getting fixed/working/added?

 To be honest most of my contribution to this release has been in the manager role, most of the code changes were from David and 3rd party contributors.

I guess the thing I'm most jazzed about in the release of this new client that I pushed for would be the CPU feature detection stuff the community has been wanting for a long time. Along with that the client makes an attempt to detect which video card the machine is using.

It has been a 'chicken and the egg' problem, it wasn't something any of the projects were specifically looking for and therefore kept being dropped from the list. Of course the major problem with that is we didn't have any firm numbers on what percentage of the base of machines could handle the more advanced instruction sets.

Now that the information is part of the host record, I'm hoping the stat sites can build some fancy graphics and charts which might encourage one or more of the projects to create a client with specialized instruction sets.

We'll need to improve the server-side scheduler to handle scheduling specialized clients, but in the end I think it'll benefit everyone. Undoubtedly this is going to cause some angst amongst those who are only in it for the credits, as the credit granting gap between a stock client and an optimized client is going to shrink further.

How is boinc going to get around the problem on Windows VISTA where you are not suppose to (and often cannot) write to the 'program files' directory?

We'll be breaking apart the data from the executables using the SHGetFolderPath API. Although we'll need to put a possible override registry value for those who want to store the data on a different drive.

Right now I've been brainstorming on the various upgrade scenario's. ( XP to Vista and  32-bit to 64-bit )

It is a messy problem.

To submit questions for next week just click on the comments link below and submit your question.

Thanks in advance.

----- Rom

Is it possible to include a (empty) "skins" folder in the next release?

Sure.

Seems that SETI@Home is a victim of its own success; or maybe a run-away train. Has there been any discussion on methods to allocate resources between projects? For example, implementing a per user work unit limit based on the users contributions to the project; contributions such as money, time, other projects in which user participates.

The philisophy of the BOINC the framework is to let the participants decide how they want to volunteer their resources.  To that end, it is up to each project to sell their project to the community.  Donating computation time to a project is already an expensive proposition, it costs time, electrictity, and internet bandwidth.  To be honest I cannot see a project being successfull if they started selling workunits to process.  At that point I think a project would loose it's fun and cool factors and would find it's big iron contributors defecting to other projects.  Eventually that project would only be left wth the die hard fans wth money to burn and those who were not affected by the cap.

At that point the project is pretty much on life support, and waiting for the people who are funding the project to pull the plug.

What's the development progress in getting the boinc code (I assume 5.9.x now) up to scratch to use wxWidgets 2.8?

Charlie started the port to 2.8, there are a few things on my plate to tackle. Text gets clipped in the wizard for instance. We'll have it up to spec when we start the 5.9-5.10 release push.

Can auto update (I see it in the code for 5.9.x) be turned on in 5.8.8+ or has it been left out?

I'm not sure what the client will do at this point if it receives the auto update tag, it should just be ignored at this point. That feature has another month of work or more before it is ready for use.

What do you think are the major improvement in 5.8.8+ over 5.4.9/11?

There have been so many under the cover improments, I think the big ones are the improved CPU scheduler, the improved work fetech policy, and CPU throttling.  On the Mac we have the sand boxed environment which is a huge improvement.

What are you most chuffed about getting fixed/working/added?

What do you mean by chuffed?

To submit questions for next week just click on the comments link below and submit your question.

Thanks in advance.

----- Rom

Have you seen this GPU programming kit:
http://www.nvidia.com/object/IO_37226.html
What do you think are the difficulties of GPU programming with this and would some work still need to be done by the CPU like on F@H?

I suspect every project would have to be a hybrid and use both the CPU and GPU. At the end of the day the OS needs to know what is going on, otherwise it would just overwrite whatever work you assigned to the GPU with something else.

To be honest, I haven't studied the S@H or R@H source closely enough to know how easy or hard it would be to port using the various toolkits released by the video card manufacturers.

Hopefully we'll start to see more traction in this area with the release of 5.8, since it has some basic video card detection code in it.

----- Rom

BOINC 5.8 has now ready for public release.

Some of the highlights for this release include:

• Simple GUI (Complements of World Community Grid)
• Basic CPU identification
• CPU throttling
• New CPU scheduler
• New work fetch policy
• Improved memory management

We have one outstanding issue with the Linux client at this time. As soon as we have resolved it we'll release it as well.

We would like to thank everyone for their hard work in making this a solid release.

----- Rom

JM7 posted a nugget of information I missed about how DCF is used in this S@H post.

Actually, ROM missed a little. A score less than 1 means that your computer finishes work faster than expected for its benchmarks. A score that is higher indicates that it finishes slower than expected for its benchmarks. It is calculated as a safety for the CPU scheduler (finish on time) and work fetch (not too much), therefore under estimates are corrected very quickly (single step) and over estimates are corrected much more cautiously.

Thanks John.

----- Rom

Is the boinc core client and manager going to support IPv6 in the near future?

All of the communication between the core client and project servers is done through a library called libCurl. It has an awesome feature set and it wouldn't surprise me if they already supported it. A quick pass over their comparison chart says they do. At this point I'm not sure there is anything more we have to do.

Does anybody have some IPv6 gear to test things out on?

Will a future BOINC have an interface tab or an options extension or the like to set any of the 'override' parameters?

I'm not sure what you mean by override parameters. If you are referring to the global preferences then yes, the manager will include the ability to override the global preferences. That feature will first make it's debut with the BSG with a small subset of the overall features, probably within a release after that I'll add the rest of the global preferences to an enhanced preferences dialog which will be available through the advanced interface.

Currently the simple preferences dialog looks like this:

To be fair though, I just got done butchering everything on Friday to take care of a usability issue and WCG hasn't had a chance to give me an updated bitmap and that is why you can see the magenta border. The general layout is there though, it should look pretty intuitive on how it should work.

Everybody should feel free to provide any first impression feedback, we are all interested in what you all have to say.

any update on new BOINC client interface? can anyone sign up for beta testing?

Well for the last several weeks I've been saying we will hit beta this week. So without further ado, we'll hit beta this week. Kevin and I will probably chat tomorrow and decide what to do. My new target date for a beta release is Wednesday.

Like with all of our beta releases they are available for those who really want to try things out, just be advised that beta releases have bugs and things may not work. In the worst case scenarios' their could even be data corruption.

When BOINC is updated, it ignores already installed folder; user have to manually choose correct folder - every new BOINC version, every machine running over and over. Any good reason for that?

Nope, none. It is on my plate to fix. I was hoping to have more time in this release to do a couple of things like storing setup information/version upgrade notification, I still might after we get the beta process underway, but right now I'm head down on the Simple GUI until things have stabilized.

So when can those of US who run Windows XP x64 see a Native 64bit Boinc and app?

I suppose when I can get my hands on a 64-bit machine.

I generally buy my own hardware, I have expensive tastes and really don't like low-budget computer hardware or base configuration models. Down-side to that is I don't upgrade often, my current workstation I've had for several years and probably has another couple of years left on it.  Although I have been looking at a few of the dual-processor/dual-core/hyperthread-enabled workstations from Dell. Who knows, I might pick one up next year.

If there is enough demand for a 64-bit build, and for whatever reason Crunch3r and crew are having problems releasing builds, I'm sure David would hook me up with a 64-bit machine.

considering it's clock-changing weekend: does boinc take into account the fact that the clocks change when recording/calculating processing time?

For the most part BOINC uses Epoch time internally, I suspect BOINC will be superseded by something else before we run into time keeping issues.

why doesn't boinc use actual CPU time directly?

Any place BOINC can use CPU time to account for the amount of CPU time an application has used, it does.  Some operating systems don't provide a very good way to get at hat information, and in those cases wall clock time is used.

about crashes etc., when something fails/crashes in windows, the user is asked to send a "report" to a Microsoft server somewhere.
Are these reports actually collected from MS for debugging purposes?

Short answer, no, the crashes are uploaded to a Microsoft server, but Microsoft only investigates their own application crashes. Microsoft does offer access to the crash reports to the 'owners' of the software so they can download the crash dump files and try to figure out what is going on. You actually shouldn't be seeing the 'Error Reporting' tool which I'll refer to as Doctor Watson.

BOINC is supposed to be completely autonomous, meaning it just runs in the background and if an application crashes it silently handles it and any diagnostic data that we can get at is analyzed in the background and then uploaded to the project server in a condensed form. I participated in debugging both S@H and R@H applications using this technology and have started to collect and publish little nuggets of information about common crashes. You can find it here:

http://boinc.berkeley.edu/app_debug_win.php

I'll continue to add to the list as I find them, or am called in to help isolate bugs in another application. Most of the examples are R@H crash dumps, I should have started the document during the S@H beta cycle, but I didn't think about it then.

1) the most annoying one is the upload+report & download of new work process with a short cache.
I have a tiny cache (something like 0.0001 days) because i have a premanent Inet connection. When a task is very near finishing, due to my small cache a new workunit is downloaded, and then the near-finishing WU is uploaded and reported. The problem here is that 2 requests are being made, one for new work, then one soon after to report finished work, it would be more sensible to wait for the unit to finish, upload, then report and get new work in one operation, rather than hammer the servers as "return results immediately" does
Will the new CPU scheduler avoid this problem?

John really is the best person to ask about CPU scheduling issues, I'm just a consumer of his and David's work, same as you.

That said, I do not believe the new CPU scheduler will avoid the problem, one of the goals is to keep the CPU busy, if you finish your result and have to wait for the client to download another one, the CPU isn't busy.

If I was in your shoes, after the new scheduler is released, I would set my cache size to 1 and let the client re-normalize on that. The days of having a very small cache to keep from missing a deadline should be coming to a close.

4) not a bug, but a question, i've got some changes to some of the web code, and i want to checkin my changes to the CVS/SVN system, but obviously i don't have the permissions to do so. how do i go about getting my changes merged?

Send them to David and/or Rytis and let them look over the changes.

Carl was unable to trap all the exceptions within Visual Studio (unlike the Linux environment which was more helpful) which is why I suggested having a call-back process so that Boinc could get the science app to help with 'difficult' exceptions. So you'd still have a black box, just not a cubic one

Yeah, I've been working with AutoDock@Home a little bit trying to help them get setup in there Fortran environment. It appears that the Intel Fortran compiler uses a different form of exceptions than Windows knows about. I found some interoperability documentation between C/C++ and Fortran and suggested some changes. When they let me know how things went we might be able to provide some extra information for those using Fortran in the BOINC environment.

To submit questions for next week just click on the comments link below and submit your question.

Thanks in advance.

----- Rom

References:

http://curl.oc1.mirrors.redwire.net/
http://curl.oc1.mirrors.redwire.net/docs/comparison-table.html
http://en.wikipedia.org/wiki/Unix_time
http://boinc.berkeley.edu/app_debug_win.php
http://boinc.berkeley.edu/sched.php

Can we get more (unlimited - well, within reason!) preferences than home, school and work? Three profiles isn't enough for me and I'm only running a small number of computers. I know these can be overridden (although the project preferences for Rosetta (i.e. runtimes) cannot)I'd find it really useful if these profiles could be added to as required, and please can you make them renamable?!?

I believe the account manager folks are working on some features which will allow greater configuration flexibility. The BOINC client is capable of dealing with a greater number of zones, there just hasn't been an easy way of configuring them on a project's web site. Rytis is now at the helm of the project web site and forum features. I'm looking forward at seeing what he is going to cook up.

Also, any update on BOINC on the consoles?

Well there is a lot of buzz, but nobody has signed on the dotted lines yet. David and Eric are going to a Sony R&D center next week to meet some engineers for the PS3. I haven't heard anything new about the XBOX 360, the XNA Game Studio from Microsoft is a bust for BOINC, it assumes all of the game code is going to be managed code on the 360. So that leaves us with the need of the same development kit as the professional game studios use.

Again moor of a request i am attached to a lot of projects and when I need to take a box out of service(without throwing away wu) I have to click "no new tasks" over 30 times. A bit tedious especially over VNC. A global (per host)no new tasks button would be of great use to me.

Is the global update ever returning? Although I can see where it can be abused.

Right now many things are on hold until after we can get the BSG out the door. Tentatively I have some time allocated to re-work the Advanced UI and playing around with Vista has inspired me on how to handle the multi-selection cases in a list view control. We shall see though.

'Retry Communications' is about as close as your going to get for an update all type function. It basically resets the countdown timer for any pending action.

With regards to the whole '-return_results_immediately' thing, from a project perspective it is altogether evil. I'll write up another post about that separately.

1) What are the typical things which cause the work unit to fail?
(Environmental - antivirus, graphics drivers, excessive overclocking, PC crashes, playing games for hours, video encoding, etc.
Human factors - Misunderstanding boinc messages, for example incorrect URL - they detach and attach, then get upset that x months of work is 'down the pan'. Ditto installation of berkeley version over bbc version, easy to fix but they don't know how)

You have nailed the majority of cases. I mean we could go off into the really obscure cases like cosmic rays and the like, but you covered all the things in the majority case.

In the future we won't be allowing a directory name change for any software package that we build for others, so that should take care of any potential future BBC issues. Now before you all think I'm making up the whole cosmic ray thing here is an article from ZDNet about eBay suffering one to two crashes a month due to a defect in their ECC memory which left them prone to cosmic rays.

2) Is there anything which can be done to avoid these, either by the science app or by Boinc itself?
(Uploading partial results as the WU runs. Exception handlers, both at science app and callbacks at boinc? Restart from checkpoint/backup if error code 0,-107...,etc etc received? Going into hibernation if PC is very busy, out of memory, etc)

This is one of those really cool but really though questions. Each environment handles things a bit differently. About the best advice I can give is for each project to really understand how the programming language they are using interacts with the operating system they are using.

CPDN is advancing the trickle model to the point where they could resend out a workunit that has timed out and take the previous users trickles and reuse them as the starting point of the new work unit.

One thing I would like to point out is that BOINC itself cannot do anything about a science application failure except fail the workunit and move on to the next one. To BOINC each of the science applications are a little black box and the only way BOINC knows anything about what is going on inside is through a little 8k chunk of shared memory broken up into 8 channels. Simple commands are passed around in these channels like show graphics, hide graphics, and here is the amount of CPU time I've used.

Now exceptions, and error tracking in general, use pointers in the local address space for the science application. For BOINC to be able to track exceptions in a science application would mean that BOINC would have to act like a debugger while the science application is running which would cause a 20-30% performance decrease for all science applications, and would more than likely negate any optimizations available to an application.

We did add a little something to the BOINC API library which we internally refer to the 'BOINC runtime debugger'. This little chunk of code is compiled into the science application and informs the OS that if any unhandled exceptions happen, it needs to execute a chunk of code. Using stackwalker as a template we expanded the functionality and improved the data returned to the project using a Microsoft library on Windows to dump out as much information about the exception as possible. This code isn't ever executed or used unless an unhandled exception happens within an application, so no performance decrease is experienced.

I'm going to need to write a whole different article on this topic.

3) What support does Boinc have / plan to have which relate to this category of work unit specifically?
(e.g.) some ideas, many of which may be impractical -
* Separation of graphics from the work unit so that a temporary problem with the graphics drivers doesn't cause the WU to fail

Separation of the graphics code from the worker code will probably start at the beginning of next year. It is going to be a requirement for supporting Vista and other OS's as they increase in their defense in depth models.

* Automatic backups
* Backups which are per-workunit rather than for all workunits which happen to be running

There are other tools that can be used for backups. Frankly, trying to tackle that role is complicated and really outside the design scope for BOINC.

* Callbacks from Boinc into science app to allow the science app to handle boinc exceptions it wouldn't normally be able to trap

What kind of exceptions do you think the science applications need to handle?

* Handling of the situation where the PC is very busy, out of memory or other resources, about to crash, TCP/IP stack blocked...)

We are adding more smarts into the CPU scheduler to handle the memory/paging cases.

Crashing is a random event, the only way you could know something is about to crash would be to already know what the bug is.

We added some code awhile back to test the various communication mechanisms when BOINC is first launched, that should have taken care of the TCP/IP blocks. If you know of any cases we haven't covered with recent builds let me know.

how's the progress with allowing AMS/BMS/BAM (whatever it's called these days) to control the state of projects and WUs
such as setting NNW, or suspending a project/task?

I believe this code is in for the 5.8.x release.

Farm Managers ?
Farm Manager ability came with Account Managers, I cannot find any programs on the BOINC website to install a Farm Manager on my computer, what is it? is it working? or has it been abandoned?

A farm manager is an idea that James Drews had, I believe, that is geared towards managing hundreds of machines. Basically you setup a web server which acts as a private AMS, the BOINC client includes it's IP address, port number, and GUI RPC password (I think) when it first connects to the farm manager. After that if you want to do something specific to a machine the farm manager can issue a GUI RPC just like the BOINC Manager. I'm not sure if anybody besides James has done anything about creating a farm manager package.

BOINCView is probably the best bet unless you come by several hundred machines.

Auto update of 'BOINC' ?

Funny you should ask this, WCG was asking about this very same thing. We'll probably start looking into something like this for the 5.10 release.

We were always concerned if we had put something like that in place it might be exploited by an attack vector we never even thought of. At least with a human at the other end of the equation the amount of damage would be limited.

Now with WCG as a contributor we can get the IBM security department to look things over and let us know if something is really wrong. IBM has looked over the BOINC source once already so we are confident we have our i's dotted and our t's crossed but with auto-deployment of code without user intervention you can never be too careful.

I am new to BOINC and I'm loving it, but I was wondering: are any plans for BOINC to use the powerful new age GPU's and PhysX processors that are perfect for floating point computations?

FluffyChicken Wrote:

I can answer the last one,
ATI(AMD) have asked BOINC if they would like help, though it would be the projects that would need the help if the GPU is capable. NVIDIA would probably need to jump in if your(we) are going to get it running on that, or somebody like Microsoft developes an easy to use API (Accelerator in research ?)
As for PhysX, we (some members in the forum) contacted them from Rosetta@home and had no real rosponse.
Rosetta@Home are in talks with Microsoft for the XBOX360 though, apparently.

I would just like to add that with the next release BOINC currently detects your video card and processor capabilities and reports them to the project. If/when a project commits to using a graphics card or physics accelerator we could go through with the rest of the work items to turn them into a resource that can be scheduled for use.

We added in the detection code so we could try and get the stats sites to break down video card usage and processor capabilities, maybe spur on the projects to develop specific customized applications to harness the untapped capabilities of the machines.

It is much easier to go to a project and sell them with hard numbers than to say we think this could help you by 'x' amount.

To submit questions for next week just click on the comments link below and submit your question.

Thanks in advance.

----- Rom

Previous Articles:

BOINC Q&A -- 13/10/06
BOINC Q&A -- 10/06/06
BOINC Q&A -- 09/30/06
BOINC Q&A -- 09/22/06

References:

http://news.zdnet.com/2100-9595_22-525403.html
http://www.codeproject.com/threads/StackWalker.asp

Advanced Memory Management, what is the idea/aim behind that?

Well that is a good question, the advanced memory management is more about setting boundary conditions on how much BOINC and related processes are allowed to use.

We still get a few reports of BOINC causing systems to become unresponsive or sluggish. Most of the investigations we have done revealed a machine that was paging a lot during the times BOINC was running. Paging is the process the OS uses to free up less frequently used memory to make room for active tasks by writing those pages of memory to disk. Each page of memory is roughly 4KB in size on a x86 processor.

So lets say you are running a machine with 512MB's of memory. Windows XP uses roughly 128MB of that on boot-up and will allow parts of itself to be paged out to disk. The last round of virus scanners I looked at want around 100MB of memory, the little system tray icons in the lower right part of your screen generally take about 5MB a piece, with the notable exception of the various IM clients which have bloated out to 20-60MB a piece. Any additional programs running on your machine such as a web browser or email client can take anywhere from 20MB up to 100MB.

When the OS comes under memory pressure it starts looking for chunks of memory that haven't been touched in awhile and writes them out to disk and then loads something into that chunk of memory that is more relevant.

So let us say that you are attached to R@H and you walk away from your computer for an hour or so, during that time R@H has used over 256MBs of memory continuously for at least 30 minutes and the OS has had to page a lot of stuff to make room for it, including itself. You start menu has to be reread from disk or whichever application you happen to be using before you left. All of that paging takes a few moments and makes your computer feel really really slow.

With the introduction of this feature we hope we can finally close one of the last remaining loopholes to user responsiveness.

Right now we have the following two settings planned:

1. Percentage of memory use while user is active.
2. Percentage of memory use while user is idle.

What should happen is that BOINC will detect how much memory is installed on the machine, and every 10 seconds or so looks at how much memory a science application is using. If a science application exceeds the total allotment BOINC will shut it down and look for another application to schedule.

I'm really looking forward to this feature since my 2GB machine uses about 1.2GB of memory without BOINC even running and I have four processors to feed. Up until the middle of last year I only had 1GB in my machine and if I had BOINC running it was pretty painful when BOINC rescheduled all the science applications on the machine while I was working.

Scheduler Improvents (already implemented?) how do these help ?

As far as I know John Mcleod has finished the work on the new scheduler and work-fetch policy. The new system should reduce the number of wasted cycles lost between the last checkpoint for an application and when it needed to quite due to a reschedule to honor resource shares.

John is really the wizard in this area.

How are any other improvement going to improve us? and the projects?

I believe the two major work items over the next year will probably be the inclusion of the projects to be able to use torrents in their file download process and the ability for projects to be able to send out optimized science applications for each processor type and possibly GPU enabled applications.

Is there anybody working on boinczilla? Bug reports are raising and nobody sort it out :/

My bad, I'll see what I can do about that this weekend.

Why not run the benchmark at higher priority, so each system produces a constant value, rather than the haphazard, particular as occurring only every 5 days?

The idea behind running the benchmarks at the same priority level as the science applications is to get a rough idea how how many cycles the science applications will get. If you run the benchmarks at a normal thread priority it won't be that much more consistent, and if you run them at the highest thread priority a user mode application can have you'll get numbers that are not very realistic for a science application running as an idle process.

The systems are benchmarked every 5 days or so to handle changes to the environment, such as a more resource intensive virus scanner or any content indexing systems that might have been installed.

When are we going to see the first alpha/beta with the BSG?

Hopefully next week.

With regard to the idea of switching tasks at a checkpoint, what happens (as in, are there any checks etc) when an application gets "stuck" and doesn't make any progress? This also applies to a similar situation with current apps, where they get stuck and the clint tries and tries to get it done by the aproching deadline, but obviously never will. This pushes the client into NNW and EDF. Will BOINC abandon the unit if no progress is made, or the deadline is met?

To be honest, I don't know. I'll have to bug John and David about that.

Is there any possibilty of releasing 5.6.4 or 5.6.5 as alternate versions?

I don't intend to put them on the download page. But if you feel comfortable with the quality of the client that you feel you can recommend people to use it, then go ahead and give them the link. I think we were far enough along in the testing process to know it isn't going to cause any major problems and might have only a few small bugs left before it was ready to be released.

The reason for not adding it to the download page is then people would receive a message in the message long requesting they upgrade to it. If all goes according to plan we'll be able to release 5.8 in a few weeks, and it would be a bad experience to bug people about upgrading twice in one month.

I suspect that if somebody was experiencing a bug that is fixed in 5.6 they would be happy to start using it now and not be so annoyed when they see the upgrade notice for 5.8.

Is there any chance of a purge function being implemented?

I haven't heard any talk of one. I'll bring it up with David, it sounds like something a project might want.

Hot topic: Why is the hourly benchmark value between Linux and Windows different, or it's claimed. When done with stock BOINC 5.4.9 e.g. on Windows it kicks out 8.1 per hour, when same done under Linux, it kicks out 5.0. The WU's are processed at equal speed i.e. a job on Wondows taking 2 CPU hours would take near equal time on Linux.

It has been my experience that the Microsoft compiler has been better at optimization than the GCC compiler. I'm sure I'll get flamed by the OSS crowd but most of the projects are experiencing the same result.

I should point out that the optimizers have been able to equal things out by a lot of trial and error by turning off and on the various optimization switches for GCC.

If the optimizers want to submit a patch that contains different non-CPU specific optimizations I'm sure we could use them.

To submit questions for next week just click on the comments link below and submit your question.

Thanks in advance.

----- Rom

Can you explain more on Average CPU efficiency and Result Duration Correction Factor? There seems to be some confusion about this, and little definite knowledge. For instance, some say a lower RDCF is better, others say an RDCF closer to 1.0 is best. Which is the truth?

CPU efficiency is the difference between how much CPU time a process received relative to the amount of wall clock time that has passed. It is the answer to the question of "In the last ten minutes or so, how much CPU did BOINC based science applications receive?" The thing to remember here is that the OS is constantly doing things in the background and each of those things eats a little bit of the CPU.

Duration Correction Factor is a per project value that measures the difference between the the expected time to process a result based on the benchmark verses what it actually took. A score of 1.0 means that the benchmark and the application processing time are in sync. The lower the score the greater the variance between what the benchmarks predict verse what it actually took to complete the result.

BOINC tries very hard not to ask for more work than it can actually process in a given period of time, so it tries to keep track of the machine overhead by the CPU efficiency score and Duration Correction Factor. Another thing to keep in mind is that memory speed plays a big part in the Duration Correction Factor. When you see similar processing times for a result for a 3.0Ghz processor and a 2.0Ghz processor it normally means that the 3.0Ghz processor is running with memory that cannot keep up with the processor. Or that both processors are bottlenecked with the memory speed.

We haven't come up with a good solution for measuring the memory bandwidth problem yet. However, we are working on it.

BOINC version release notes do not seam as complete as they were before or am I looking in the incorrect places?

You can checkout the latest and greatest changes to BOINC at this web address:
http://setiathome.berkeley.edu/cgi-bin/cvsweb.cgi/boinc/

The file you'll want to look at is 'checkin_notes' which contains the latest changes made to the client and sever packages.

You can see the check-in history for a specific branch by changing the tag specified near the bottom of the web page. The 5.6 branch tag is 'boinc_core_release_5_6'. If you want to see the changes for 5.4 you would use 'boinc_core_release_5_4' and on it goes.

Any plans on releasing the full minutes of what went on (when your back), I read up on the 1st one but was a bit disappointed with the info on show it only gave a brief overview of what went on.

You can find the workshop proceedings here:
http://boinc.berkeley.edu/ws_06.php

How was the vacation?

I had a blast. I met a bunch of great people. I'm looking forward to going again next year.

wxWidgets 2.7 has been released. Is this going to be used in 5.6 or is it too late?

Too late for this release.

I've seen and myself tried to compile 5.4.x using Microsoft's free Visual Studio Express 2005 editions (with all the bits and bobs needed, wxWidgets, SDK that needed..) Errors show up and does not compile. Is this fixed in 5.6, given this would probably be the major environment used by people trying develop BOINC under windows (since it's free).

The BOINC DLL relies upon the ATL libraries which are not included in the express editions of the MS Development tools. I'm not sure if this is going to change in the future or not. I suspect that if we can incorporate a torrent library that doesn't use COM/DCOM on Windows then I'll invest more time into removing the need for ATL/COM/DCOM so that the DLL can be built with VS Express.

On a side note, I do not believe the express editions of the Visual Studio toolset contain the optimizing compilers or linkers. You might have to upgrade for those, or use the GCC toolset's.

Would it be possible for you (since afaik you compile the final Windows releases) to put up instruction on how you compile BOINC. This may help a lot of people who just wish to dabble.

I'll see what I can do.

How is the progress on low-latency-computing? Which projects expressed their interest in this feature?

I believe this feature was put in for a hospital who wished to be able to process MRI images faster than their current method. I'm not sure this feature will be used by a public project in the near future.

To submit questions for next week just click on the comments link below and submit your question.

Thanks in advance.

----- Rom

This years workshop was very informative. It was really exciting to see the project leaders for some of the large projects as well as meet the project leaders for projects like PrimeGrid and Chess960@Home who started the projects as hobbies and who pay for them out of donations and out of their own pocket.

Here are a list of projects that I remember seeing right off the top of my head, sorry if I missed anybody:

• BOINCStats
• GridRepublic 
• S@H
• CPDN
• PrimeGrid
• Chess960@Home
• LHC@Home
• Africa@Home
• malariacontrol.net
• QMC@Home
• SIMAP 
• R@H
• WCG
• Condor

On the first day, David gave an overall BOINC update of everything thast has been accomplished within the last year, since the first pan-galactic workshop and various projects described what they had been up too. People also had a chance to list out what topics of interest they had for break out sessions on the second day.

We wrapped up after the first day and went out to eat dinner at a fairly nice place to eat. I had to leave early since the motel I was staying at closed their gates at 11pm.

The next day we spent the morning in breakout sessions which included some of the following topics:

• Credits
• Grid Integration
• Security
• AMS Issues
• Project Issues
• Science Application Feature requests

I attended the sessions on security and project issues.

I can't wait to see the summary role up for all the break-out sessions.

After lunch, we assembled and brain stormed about how to attract new members and new projects.

The last session of the day was for David and I to listen to everybodies wishlist of features.  Some of which included the server-side code for CPU feature scheduling, torrent downloads, some more work refining our sample application, and others.

All-in-all it was a great event. Now we just need to come up with a plan to implement it all.

Catch you all later. I'm still on vacation and will return to my normal schedule after the 28th.

If you all want to see additional pictures from the conference you can go here.

----- Rom

I would just like to know more info about the kind of technical challenges you will face whilst attempting to port BOINC over to them?

At this time I really don't know. Until I can get some documentation on the supported API sets it'll be hard to give any answer. The only thing I'm pretty sure about is that the management interface will have to be written from scratch for each console.

How is the simple GUI coming on any ETA on this?

As far as I know it is almost feature complete. The guys at WCG are probably going to kickstart the beta process with their WCG beta testers since it looks as though we'll still be putting the finishing touches on 5.6.  I believe we'll start to see clients with the BOINC Simple GUI (BSG) in a few weeks.

Does it work well from what you have seen?

Kevin Reed has been doing an awesome job polishing up the interface. I like what I see and this will give us a way to add more to the Advanced GUI without having to worry about confusing people.

How is the sand-boxing of BOINC on Windows coming along?

I haven't started that work item yet. I probably won't be able to start until we have completed our agreements with WCG.

When will 5.6 be released and what are the main improvements?

The best answer I can give about when it will be released is when it is ready. Improvements include a new CPU scheduler and work-fetch policy, CPU feature detection, and video card detection. This release will be the first time the BM will use GTK2 on Linux and sandboxing science applications on the Mac.

Will we ever get column sorting in BOINC Manager?

Someday.

Talk between BOINC projects and BOINC and vise versa seems minimal (and project to project). No-one seems to know what each other is doing, could there be an improvement of talk between them, like modifications to the servers, message boards, stats and developments in the pipeline.

I believe that is what the BOINC Workshops are about. We just finished the second one last week. It was productive and the projects gave us a bunch of work to do. I believe David is going to publish the results of the workshop within a week or two. I won't know timeframe until after I return from vacation.

Where does BOINC need help?

Here is a list of things we have identified that we would like to have. Contact David before starting anything though in case somebody else has already started the project. That list is likely to grow after the notes of the second workshop have been processed.

Can you explain more on Average CPU efficiency and Result Duration Correction Factor? There seems to be some confusion about this, and little definite knowledge. For instance, some say a lower RDCF is better, others say an RDCF closer to 1.0 is best. Which is the truth?

I'll have to get back to you on that. I need to look over the code again.

BOINC version release notes do not seam as complete as they were before or am I looking in the incorrect places?

We are moving toward the firefox style of release notes. For a detailed list of changes you'll need to lookup the checkin_notes for a specific release. I'll follow-up in another post on how to do that.

Any plans on releasing the full minutes of what went on (when your back), I read up on the 1st one but was a bit disappointed with the info on show it only gave a brief overview of what went on.

I left the note taking to David and a few others, we'll just have to wait and see what is published. I wouldn't have been able to type fast enough to keep up with all the discussion and I still needed to be able to answer questions.

Sorry for being late with this weeks Q@A but I'm still on vacation. :)

To submit questions for next week just click on the comments link below and submit your question.

Thanks in advance.

----- Rom

Well, since Martin was the only one to post any questions before I go to bed I guess the top ten will be his questions for the week.

When do we get to crunch on the new ALFA WUs?
When next for a reobservation run?
How well do the s@h-enhanced recrunched WUs compare to the s@h-classic results?

Ouch, these questions are best left to the S@H group.  I had meant for this to be about the BOINC software itself since I really can't speak for any of the projects.

I suppose I could attempt to clear up any confusion in what my roll is.

First off let me start out by saying that David A. manages multiple projects, of which S@H and BOINC are but two of them. I was hired to be a BOINC developer.

At the time I was hired bug investigations and fixes for SETIBOINC was divided up amongst the existing team which also had to manage SETI Classic. SETI Classic kept people pretty busy most of the time as people were running from one fire to the next. Under the existing model SETIBOINC was going to take awhile to finish up, so after discussing things over with David we decided that the best course of action would be to have me to head up the SETIBOINC migration until the migration was complete.  SETIBOINC completed its migration in December 2005 and so my regular participation in the day to day happenings with S@H came to an end.

My role these days mostly involve helping David design and implement new features that will help drive further adoption by both new projects and new participants. I also manage the build, test, and release processes for BOINC.

These days I know just about as much as you all do about what S@H is up too.  When David and I talk, in our daily phone calls, it is all about what is going on with BOINC.

What happened to utilising GPUs for processing WUs? (Are single precision floats good enough?)

Starting with BOINC 5.6 it will detect CPU features such as SSE and 3DNow as well as video cards. All the demand for video card detection has come from the community at large, so I don't know of any project that is actively developing a client.

Whether single precision floats are good enough, that is up to the project.

I believe the biggest problem is actually the memory bus bandwidth between the systems main memory and the video card.  Until PCI-Express was released the video card bus was design to push data to the video card as quickly as possible but didn't do very well transferring data from the video card to system memory. It is my understanding that PCI-Express can handle the loads that DC apps will generate.

I suspect that once BOINC 5.6 is released we'll be able to start publishing video card and CPU feature data to the stat sites and let them start generating some useful graphs for projects to look at.  Once the projects know what is out there they might be able to justify the expense of specialized clients.

What of continued funding for Boinc and for s@h?

I can only speak about BOINC here, but I believe BOINC has secured funding for a couple of years.

Well, off to bed now.

----- Rom

Here is another little misconception:
CPU Capability detection coming to a BOINC client near you soon

BOINC does not use the processor's CPUID instruction to determine what instruction sets are supported.

Using CPUID is a good idea if you are an OS, but it is a bad idea if you are an application. There are two parts to the supported instruction set problem, one is the CPU and the other is the OS.  If your OS doesn't support the desired instruction set you are just asking for trouble.

Here is an example of what I mean. Let us take Windows 95 Gold without any patches, and a modern single core processor.

Back when Windows 95 was released MMX and 3DNow was just taking off and SSE wasn't mainstream. MMX still used the standard 80-bit x87 floating point registers and so the OS really didn't have to do anything new to support it. A thread context therefore only had to worry about the general purpose registers, floating point stack registers, and debug registers and everything for the thread stayed consistent when the OS changes execution to another thread.

Now enter modern processors, with the introduction of SSE new registers were added to the CPU. Registers XMM0-XMM7. If the OS doesn't know about those registers it cannot save the registers before moving on to another application. In the worst case scenario you could have data from your favorite DVD interfering with a BOINC Science application since they'll both be overwriting one another's register values. E@H processing TV signals and your DVD player displaying E@H data as video artifacts.

It appears Intel and AMD created something known as enhanced protected mode which exposes the additional SSE registers otherwise they stay hidden from applications and the OS if the OS doesn't initialize itself as enhanced protected mode aware. So if you are attempting to run an SSE application on a CPU/OS combination that doesn't support it you should get an illegal instruction error or privileged instruction error.

Apparently AMD decided to increase the number of registers available on the AMD64 line from 8 to 16, but I don't currently know if this is only for 64-bit OS's or for both 32-bit and 64-bit OS's. If there isn't a special safe guard the OS has to follow, things could end up like the DVD player scenario I mentioned above.

So BOINC queries the OS for what instruction sets are supported, if the OS can detect it, it should support it.

On Windows, the function is called IsProcessorFeaturePresent().

On Linux, we currently read /proc/cpuinfo.

On the Mac we'll probably be using sysctl().

----- Rom

References:
http://en.wikipedia.org/wiki/IA-32

Somebody pointed out a thread to me on E@H:
http://einstein.phys.uwm.edu/forum_thread.php?id=4480

I have to say that I'm a little shocked at some of the themes in attitudes of some of the participants I've seen.

First let me clear up some misunderstandings about what validators and assimilators for a BOINC server cluster are supposed to do.  Validators only check to make sure there is agreement between the machines who have crunched the same workunit.  If all of the machines agree on what the numbers are then the results are considered valid and flagged for assimilation.  Assimilators just copy the result data from the BOINC database/file system to the projects internal database for analysis.  After assimilation a result finally has meaning in the context of the projects goal, prior to that it is a collection of numbers and BOINC doesn't have a clue if they are correct or not. 

Projects are free to add additional logic to their validators and assimilators to try and weed out incorrect results, but to some degree it is still just a guess.  If they already know what the correct answer is then they would not have needed to send out the work to begin with.

For projects that are searching for something, their results can be broken down to into two camps, something that needs further investigation and background noise.  What separates something that needs further investigation and something that is background noise?  There is some value or a set of values in the result files that exceed one or more thresholds.  Some thresholds may have a cap on them in which case an interesting value or set of values falls into.  We can then refer to the lower and upper bound of a threshold as a threshold window.  Those thresholds are typically calibrated against the default client a project sends out.  Tests are run against the default client using special workunits that contain various samples of data that expose what the application is looking for so the scientists can make sure the client is working like it is supposed to.

So now the crux of the problem, changing instruction sets for an application can and will change the level of precision of the data returned back to the project.

Optimized SSE/SSE2/SSE3/3DNow applications change how the mathematical operations are performed vs. and un-optimized application.  Now whether that adversely affects the project totally depends on how the project handles data types internally.  If a project doesn’t release the source code or test workunits for their application then somebody optimizing the application with a disassembler or hex editor is making an assumption about how calculations are being performed and what they can do to optimize it.  If they are wrong then something might be flagged as noise when it should have been flagged as needing to be investigated.  What if something is missed because the thresholds are geared for a different range of values then what the optimized application is producing?

SSE/SSE2/SSE3/3DNow instruction sets use 128-bit registers while the original x87 FPU uses 80-bit registers.  Now most programming languages store floating point numbers as either 32-bit single precision floats or 64-bit double precision floats.  Quite a bit of the performance improvement that these new instruction sets provide comes from packing multiple numbers into a register and then performing mathematical operations on them in a matrix style fashion.  So you could fit 4 single precision floats, or two double precision floats into a single 128-bit register.  Depending on the instruction the result may be bounded to 32-bits, 64-bits, or 128-bits.  That means in the worse case scenarios any optimized application is rounding any computation either higher or lower than the original application.

You might be thinking, why don't projects just enlarge the threshold window so that those small rounding errors can get through.  Some of them have, but others still need to investigate how using different instructions affect the system overall.  A few of the science applications perform calculations on the result of previous calculations over and over again.  How large would the threshold window have to be if the calculations on previous calculations happened 1,000,000 or 10,000,000 times?

Here is an example of two different Intel SSE CPU instructions (one for working on packed data, and the other one using the whole register) on the same processor producing different results:
http://softwareforums.intel.com/ISN/Community/en-US/forums/thread/5484332.aspx

Note, that was using the Intel IPP library.  That is how easy rounding problems can be introduced when optimizing.

For those who are quick to say by using optimized applications I'm doing more science because I can process workunits faster, my response is:
Only if the projects backend databases and tools are equipped to deal with the differences, otherwise something might be missed.  If you processed the one but sent back numbers outside the target threshold windows have you really helped the project? 

Another common thing I've seen is; I've run the standard application and the optimized application across x number of workunits I've been assigned and they produced the same result files so the optimized application must be good in all scenarios, my response is:
What that really means is that no rounding issues occurred with the workunits you had access too.  Without the test workunits a project uses internally you really don't know if you covered all your bases.

The good news in all of this is the projects are listening and are working with the optimizers to incorporate the needed changes into the projects default application.  Please be patient during the transition though, it is going to take a bit of time to double check everything and make sure it is all in working order.

In case you are curious, I do not use any optimized clients on any of my machines.  To me the science applications are big black boxes, I don't know enough about what they do under the hood to smartly make changes for the better.  I'll wait for optimization changes to be released by the projects which means that their backend systems can account for any changes to the data.

At the end of the day most of the projects are probably not concerned with the problems of verifying data that has been flagged as interesting, it is concern about missing something interesting that was flagged as background noise.

----- Rom

References:
http://en.wikipedia.org/wiki/IA-32
http://en.wikipedia.org/wiki/Streaming_SIMD_Extensions
http://en.wikipedia.org/wiki/SSE2
http://en.wikipedia.org/wiki/SSE3
http://en.wikipedia.org/wiki/SSE4
http://en.wikipedia.org/wiki/3dnow
http://docs.sun.com/source/806-3568/ncg_goldberg.html

[08/15/2006] Adding a few more reference articles, the banking industry is still battling with rounding errors in its software.
http://www.regdeveloper.co.uk/2006/08/12/floating_point_approximation/
http://cch.loria.fr/documentation/IEEE754/

Starting in the next version of the BOINC client we'll be able to detect CPU capabilities.

It is important to note that the capability detection is actually done by the operating system and BOINC just queries the operating system for the supported instruction sets.  I bring this up because not all operating systems fully support all additional instruction sets supported by the processor.  We are being conservative here to avoid illegal instruction exceptions or privileged instruction exceptions.

For Windows the following instruction sets or capabilities can be detected:

• fpu
• tsc
• pae
• nx
• sse
• sse2
• sse3
• 3dnow
• mmx

On Linux we read the data out of /proc/cpuinfo.

I still need to write the code for the Mac OS. 

The processor information will be passed to both the science applications and the scheduling server.

I am at something of an impasse with the 1% bug.

In order to gain ground I need to be able to see where the program is stuck on the destination machine.  There are three ways to do this:

1.      Have the community report which workunit stalled on there machine and attempt to reproduce it.

2.      Hook up a debugger on the target machine and have the person at the keyboard create a dump file of the process.

3.      Introduce a trigger into the executable so that on a certain action it causes it to dump its own backtraces.

Option one proves difficult just in managing the sheer number of workunits to look at.  Roughly 550 workunits a day are being aborted or have exceeded their allotted CPU time.  R@H hasn’t been able to reproduce the problem in the lab with the workunits they have looked at and are continuing to look at.

Option two doesn’t scale very well, namely of all the people who are hitting this problem only small fraction of them know how to create a dump file with a debugger and only a small fraction of them are willing to spend the time to compress and break the 200MB to 350MB file into smaller pieces to email them to me so I can look at them.  Then of course there is only one of me and I still have all my other BOINC work to do, like fixing bugs in the 5.3.x clients so we can ship 5.4.0!

Option three didn’t hit me till Monday night.  As part of the feature work we did for CPDN we introduced a way for the core client to notify the science application that it was being aborted so it could clean up after itself.  Well I completely forgot that the 5.2.x clients don’t send the abort command to the client when I burned the midnight oil to deliver the backtrace functionality for R@H 4.94.  At 4am I had the functionality working for Windows and checked it in.

Fast forward to today.  I went looking through the results on Ralph@Home and discovered that the backtraces were not being logged like I thought they should have been.  After further investigation I realized that the 5.2.x clients were sending the quit command instead of the abort command.  Talk about killing morale.  I have posted in the Ralph@Home forums that people should upgrade and I’ve been seeing results come back with 5.3.28 which is good.  I’m just not sure when I’ll have enough information about the bug.

We are pretty close to having 5.4 ready for public release.  I believe in a week or less.  But a big problem remains, typically it takes a few months for a new stable client to reach a high enough level of adoption that patterns emerge that can be tracked.

After some discussions with David Baker we are going to drop the maximum amount of time allotted for a workunit to run on a machine.  That’ll keep a good chunk of the wasted CPU cycles down.  I am also selling the idea of releasing the PDB file with the Rosetta application for the public project.  Now granted, it is a 30MB file.  Without it none of the diagnostic stuff built into the BOINC API for tracking down bugs will work.  Isn’t a 30MB insurance policy for an abort or crash worth it if the project can get something useful out of it which will lead to bug fixes?

----- Rom