Tuesday, April 18, 2006

We are getting close to releasing a new version of the BOINC client software.  Not only does the new version contain a bunch of fixes for the client itself, but it also includes new version of the DLLs required by the BOINC Windows Runtime debugger.

It wasn’t always referred to as the BOINC Windows Runtime Debugger; we used to just refer to it as the stackwalker technology.  Over the last couple of weeks I’ve made significant changes to the infrastructure on the Windows platform that I feel we can call it something different.

Here is a list of additions that I think will help out:

  • Module list dump including location, file versions, and the type of symbols detected.
  • Register dump for each thread.
  • Stack dumps for each thread.
  • Use of Microsoft’s Symbol Store technology.

Upgrading to the latest version of Microsoft’s debugger technology vastly increased the accuracy of the stack traces on Windows when the correct symbol files can be found.  Another huge benefit is the use of the symbol store technology.

Symbol stores are basically websites that contain compressed versions of the symbol files.  Now for those who have not been following my previous posts the symbol files are pretty big. R@H for instance had a symbol file that was 30MB uncompressed.  Now when a crash occurs the debugger will attempt to go to the symbol stores that are defined in the search path and download the correct symbol file for the application.  This is a huge win for a project since the symbol files (PDB) for the application do not need to be included for each workunit.  Those who are on dial-up connections will have less to download.

Hopefully we can get the rest of the code written so that projects can maintain there own symbol stores.  There are many huge wins with getting this technology enabled within BOINC.

Here is a sample of what the new debugger engine produces:

BOINC Windows Runtime Debugger Version 5.5.0

Dump Timestamp    : 04/16/06 23:41:39
Debugger Engine   : 4.0.5.0
Symbol Search Path:
C:\BOINCSRC\Main\boinc_samples\win_build\Release;
C:\BOINCSRC\Main\boinc_samples\win_build\Release;
srv*c:\windows\symbols*http://msdl.microsoft.com/download/symbols;
srv*C:\DOCUME~1\romw\LOCALS~1\Temp\symbols*http://boinc.berkeley.edu/symstore


ModLoad: 00400000 00060000 C:\BOINCSRC\Main\boinc_samples\win_build\Release\uppercase_5.10_windows_intelx86.exe (PDB Symbols Loaded)
ModLoad: 7c800000 000c0000 C:\WINDOWS\system32\ntdll.dll (5.2.3790.1830) (PDB Symbols Loaded)
    File Version   : 5.2.3790.1830 (srv03_sp1_rtm.050324-1447)
    Company Name   : Microsoft Corporation
    Product Name   : Microsoft® Windows® Operating System
    Product Version: 5.2.3790.1830

ModLoad: 77e40000 00102000 C:\WINDOWS\system32\kernel32.dll (5.2.3790.1830) (PDB Symbols Loaded)
    File Version   : 5.2.3790.1830 (srv03_sp1_rtm.050324-1447)
    Company Name   : Microsoft Corporation
    Product Name   : Microsoft® Windows® Operating System
    Product Version: 5.2.3790.1830

ModLoad: 5e8d0000 000ce000 C:\WINDOWS\system32\OPENGL32.dll (5.2.3790.1830) (PDB Symbols Loaded)
    File Version   : 5.2.3790.1830 (srv03_sp1_rtm.050324-1447)
    Company Name   : Microsoft Corporation
    Product Name   : Microsoft® Windows® Operating System
    Product Version: 5.2.3790.1830

ModLoad: 77ba0000 0005a000 C:\WINDOWS\system32\msvcrt.dll (7.0.3790.1830) (PDB Symbols Loaded)
    File Version   : 7.0.3790.1830 (srv03_sp1_rtm.050324-1447)
    Company Name   : Microsoft Corporation
    Product Name   : Microsoft® Windows® Operating System
    Product Version: 7.0.3790.1830

ModLoad: 77f50000 0009c000 C:\WINDOWS\system32\ADVAPI32.dll (5.2.3790.1830) (PDB Symbols Loaded)
    File Version   : 5.2.3790.1830 (srv03_sp1_rtm.050324-1447)
    Company Name   : Microsoft Corporation
    Product Name   : Microsoft® Windows® Operating System
    Product Version: 5.2.3790.1830

ModLoad: 77c50000 0009f000 C:\WINDOWS\system32\RPCRT4.dll (5.2.3790.1830) (PDB Symbols Loaded)
    File Version   : 5.2.3790.1830 (srv03_sp1_rtm.050324-1447)
    Company Name   : Microsoft Corporation
    Product Name   : Microsoft® Windows® Operating System
    Product Version: 5.2.3790.1830

ModLoad: 77c00000 00048000 C:\WINDOWS\system32\GDI32.dll (5.2.3790.2606) (PDB Symbols Loaded)
    File Version   : 5.2.3790.2606 (srv03_sp1_gdr.051230-1233)
    Company Name   : Microsoft Corporation
    Product Name   : Microsoft® Windows® Operating System
    Product Version: 5.2.3790.2606

ModLoad: 77380000 00092000 C:\WINDOWS\system32\USER32.dll (5.2.3790.1830) (PDB Symbols Loaded)
    File Version   : 5.2.3790.1830 (srv03_sp1_rtm.050324-1447)
    Company Name   : Microsoft Corporation
    Product Name   : Microsoft® Windows® Operating System
    Product Version: 5.2.3790.1830

ModLoad: 68720000 00020000 C:\WINDOWS\system32\GLU32.dll (5.2.3790.1830) (PDB Symbols Loaded)
    File Version   : 5.2.3790.1830 (srv03_sp1_rtm.050324-1447)
    Company Name   : Microsoft Corporation
    Product Name   : Microsoft® Windows® Operating System
    Product Version: 5.2.3790.1830

ModLoad: 73860000 0004c000 C:\WINDOWS\system32\DDRAW.dll (5.3.3790.1830) (PDB Symbols Loaded)
    File Version   : 5.3.3790.1830 (srv03_sp1_rtm.050324-1447)
    Company Name   : Microsoft Corporation
    Product Name   : Microsoft(R) Windows(R) Operating System
    Product Version: 5.3.3790.1830

ModLoad: 73b30000 00006000 C:\WINDOWS\system32\DCIMAN32.dll (5.2.3790.0) (PDB Symbols Loaded)
    File Version   : 5.2.3790.0 (srv03_rtm.030324-2048)
    Company Name   : Microsoft Corporation
    Product Name   : Microsoft® Windows® Operating System
    Product Version: 5.2.3790.0

ModLoad: 76aa0000 0002d000 C:\WINDOWS\system32\WINMM.dll (5.2.3790.1830) (PDB Symbols Loaded)
    File Version   : 5.2.3790.1830 (srv03_sp1_rtm.050324-1447)
    Company Name   : Microsoft Corporation
    Product Name   : Microsoft® Windows® Operating System
    Product Version: 5.2.3790.1830

ModLoad: 71b70000 00036000 C:\WINDOWS\system32\uxtheme.dll (6.0.3790.1830) (PDB Symbols Loaded)
    File Version   : 6.00.3790.1830 (srv03_sp1_rtm.050324-1447)
    Company Name   : Microsoft Corporation
    Product Name   : Microsoft® Windows® Operating System
    Product Version: 6.00.3790.1830

ModLoad: 4b8d0000 00051000 C:\WINDOWS\system32\MSCTF.dll (5.2.3790.1830) (PDB Symbols Loaded)
    File Version   : 5.2.3790.1830 (srv03_sp1_rtm.050324-1447)
    Company Name   : Microsoft Corporation
    Product Name   : Microsoft® Windows® Operating System
    Product Version: 5.2.3790.1830

ModLoad: 69500000 004f3000 C:\WINDOWS\system32\nvoglnt.dll (6.14.10.7801) (-exported- Symbols Loaded)
    File Version   : 6.14.10.7801
    Company Name   : NVIDIA Corporation
    Product Name   : NVIDIA Compatible OpenGL ICD
    Product Version: 6.14.10.7801

ModLoad: 02e50000 00006000 C:\WINDOWS\system32\ctagent.dll (1.0.0.11) (-exported- Symbols Loaded)
    File Version   : 1, 0, 0, 11
    Company Name   : Creative Technology Ltd
    Product Name   :   ctagent
    Product Version: 1, 0, 0, 11

ModLoad: 60970000 0000a000 C:\WINDOWS\system32\mslbui.dll (5.2.3790.1830) (PDB Symbols Loaded)
    File Version   : 5.2.3790.1830 (srv03_sp1_rtm.050324-1447)
    Company Name   : Microsoft Corporation
    Product Name   : Microsoft® Windows® Operating System
    Product Version: 5.2.3790.1830

ModLoad: 03030000 00118000 C:\BOINCSRC\Main\boinc_samples\win_build\Release\dbghelp.dll (6.5.3.7) (PDB Symbols Loaded)
    File Version   : 6.5.0003.7 (vbl_core_fbrel(jshay).050527-1915)
    Company Name   : Microsoft Corporation
    Product Name   : Debugging Tools for Windows(R)
    Product Version: 6.5.0003.7

ModLoad: 03250000 00046000 C:\BOINCSRC\Main\boinc_samples\win_build\Release\symsrv.dll (6.5.3.8) (PDB Symbols Loaded)
    File Version   : 6.5.0003.8 (vbl_core_fbrel(jshay).050527-1915)
    Company Name   : Microsoft Corporation
    Product Name   : Debugging Tools for Windows(R)
    Product Version: 6.5.0003.8

ModLoad: 032a0000 00031000 C:\BOINCSRC\Main\boinc_samples\win_build\Release\srcsrv.dll (6.5.3.7) (PDB Symbols Loaded)
    File Version   : 6.5.0003.7 (vbl_core_fbrel(jshay).050527-1915)
    Company Name   : Microsoft Corporation
    Product Name   : Debugging Tools for Windows(R)
    Product Version: 6.5.0003.7

ModLoad: 77b90000 00008000 C:\WINDOWS\system32\version.dll (5.2.3790.1830) (PDB Symbols Loaded)
    File Version   : 5.2.3790.1830 (srv03_sp1_rtm.050324-1447)
    Company Name   : Microsoft Corporation
    Product Name   : Microsoft® Windows® Operating System
    Product Version: 5.2.3790.1830

 

*** UNHANDLED EXCEPTION ****
Reason: Breakpoint Encountered (0x80000003) at address 0x7C822583

*** Dump of the Worker(offending) thread: ***
eax=00000000 ebx=00000000 ecx=77e4245b edx=7c82ed54 esi=77e424a8 edi=00454f20
eip=7c822583 esp=00a1fd64 ebp=00a1ffb4
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000246

ChildEBP RetAddr  Args to Child
00a1fd60 0040203b 00000000 00000000 00000000 00000001 ntdll!_DbgBreakPoint@0+0x0 FPO: [0,0,0]
00a1ffb4 004239ce 77e66063 00000000 00000000 00000000 uppercase_5.10_windows_intelx86!worker+0x0 (c:\boincsrc\main\boinc_samples\uppercase\upper_case.c:174)
00a1ffb8 77e66063 00000000 00000000 00000000 00000000 uppercase_5.10_windows_intelx86!foobar+0x0 (c:\boincsrc\main\boinc\api\graphics_impl.c:75) FPO: [1,0,0]
00a1ffec 00000000 004239c0 00000000 00000000 00000000 kernel32!_BaseThreadStart@8+0x0 (c:\boincsrc\main\boinc\api\graphics_impl.c:75)

*** Dump of the Timer thread: ***
eax=0002625a ebx=00000000 ecx=00000000 edx=00b1feb0 esi=00000001 edi=00000000
eip=7c82ed54 esp=00b1ff0c ebp=00b1ffb8
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000246

ChildEBP RetAddr  Args to Child
00b1ff08 7c822114 76aba0d3 00000002 00b1ff70 00000001 ntdll!_KiFastSystemCallRet@0+0x0 FPO: [0,0,0]
00b1ff0c 76aba0d3 00000002 00b1ff70 00000001 00000001 ntdll!_NtWaitForMultipleObjects@20+0x0 FPO: [5,0,0]
00b1ffb8 77e66063 00000000 00000000 00000000 00000000 WINMM!_timeThread@4+0x0
00b1ffec 00000000 76aba099 00000000 00000000 49474542 kernel32!_BaseThreadStart@8+0x0

*** Dump of the Graphics thread: ***
eax=00000000 ebx=7738e3f7 ecx=00000000 edx=00000000 esi=0012fc00 edi=7739ca9d
eip=7c82ed54 esp=0012fbb4 ebp=0012fbd8
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000246

ChildEBP RetAddr  Args to Child
0012fbb0 7739c78d 77392f3a 0012fc00 00000000 00000000 ntdll!_KiFastSystemCallRet@0+0x0 FPO: [0,0,0]
0012fbd8 00424c3f 0012fc00 00000000 00000000 00000000 USER32!_NtUserGetMessage@16+0x0
0012fcb0 00423ca3 00000001 00000001 00000001 00000001 uppercase_5.10_windows_intelx86!win_graphics_event_loop+0x14 (c:\boincsrc\main\boinc\api\windows_opengl.c:571) FPO: [0,46,0]
0012fcd0 004220eb 00401078 0045c3b0 0040233c 00401078 uppercase_5.10_windows_intelx86!boinc_init_graphics_impl+0x30 (c:\boincsrc\main\boinc\api\graphics_impl.c:84) FPO: [2,7,0]
0012fcdc 0040233c 00401078 00454f00 004483a4 00000002 uppercase_5.10_windows_intelx86!boinc_init_graphics+0x4b (c:\boincsrc\main\boinc\api\graphics_api.c:45) FPO: [1,0,0]
0012fcf4 004023b1 00000002 0012fd0c 00142550 0012fd0c uppercase_5.10_windows_intelx86!main+0xa (c:\boincsrc\main\boinc_samples\uppercase\upper_case.c:233) FPO: [2,0,0]
0012fe98 004035b4 00400000 00000000 001425a7 00000001 uppercase_5.10_windows_intelx86!WinMain+0x0 (c:\boincsrc\main\boinc_samples\uppercase\upper_case.c:110) FPO: [4,100,0]
0012ffc0 77e523cd 00000000 00000000 7ffd9000 8707adb0 uppercase_5.10_windows_intelx86!WinMainCRTStartup+0x1d (f:\vs70builds\3077\vc\crtbld\crt\src\crt0.c:251)
0012fff0 00000000 00403430 00000000 78746341 00000020 kernel32!_BaseProcessStart@4+0x0 (f:\vs70builds\3077\vc\crtbld\crt\src\crt0.c:251)

Exiting...

 

Friday, April 7, 2006

Howdy Folks,

 

Well error rates are still dropping on R@H.  It generally takes a few weeks for the old version of an application to filter out of the system.

 

Here was the pass percentage breakout from yesterday:

 

Version

OS

Total Results

Pass Rate

Fail Rate

483

Darwin

6733

90.24

9.76

483

Windows

99095

95.74

4.26

482

Darwin

213

96.71

3.29

482

Linux

9387

96.48

3.52

482

Windows

6000

84.68

15.32

 

As you can see there was a 10% drop in failure rate for windows.

 

Here is what the current error type breakout looks like:

 

App Version

OS

Exit Status

Error Count

483

Darwin

-197 (0xffffff3b) ERR_ABORTED_VIA_GUI

4

483

Darwin

-186 (0xffffff46) ERR_RESULT_DOWNLOAD

14

483

Darwin

-185 (0xffffff47) ERR_RESULT_START

290

483

Darwin

1 Unknown error number

11

483

Darwin

2 Unknown error number

77

483

Darwin

4 Unknown error number

223

483

Darwin

5 Unknown error number

17

483

Darwin

131 (0x83) Unknown error number

21

483

Windows

-2147483645 (0x80000003) Unknown error number

29

483

Windows

-2147483641 (0x80000007) Unknown error number

1

483

Windows

-1073741819 (0xc0000005) Unknown error number

672

483

Windows

-1073741818 (0xc0000006) Unknown error number

5

483

Windows

-1073741811 (0xc000000d) Unknown error number

935

483

Windows

-1073741795 (0xc000001d) Unknown error number

5

483

Windows

-1073741794 (0xc000001e) Unknown error number

1

483

Windows

-1073741783 (0xc0000029) Unknown error number

1

483

Windows

-1073741675 (0xc0000095) Unknown error number

1

483

Windows

-1073741674 (0xc0000096) Unknown error number

1

483

Windows

-1073741515 (0xc0000135) Unknown error number

143

483

Windows

-1073741502 (0xc0000142) Unknown error number

285

483

Windows

-1073740972 (0xc0000354) Unknown error number

3

483

Windows

-1073740791 (0xc0000409) Unknown error number

16

483

Windows

-529697949 (0xe06d7363) Unknown error number

102

483

Windows

-197 (0xffffff3b) ERR_ABORTED_VIA_GUI

154

483

Windows

-187 (0xffffff45) ERR_RESULT_UPLOAD

4

483

Windows

-186 (0xffffff46) ERR_RESULT_DOWNLOAD

598

483

Windows

-185 (0xffffff47) ERR_RESULT_START

210

483

Windows

-177 (0xffffff4f) ERR_RSC_LIMIT_EXCEEDED

1

483

Windows

-164 (0xffffff5c) ERR_NESTED_UNHANDLED_EXCEPTION_DETECTED

39

483

Windows

-1 (0xffffffff) Unknown error number

74

483

Windows

0

6

483

Windows

1 Unknown error number

806

483

Windows

3 Unknown error number

15

483

Windows

128 (0x80) Unknown error number

79

483

Windows

1073741845 (0x40000015) Unknown error number

1

483

Windows

1073807364 (0x40010004) Unknown error number

32

482

Darwin

-197 (0xffffff3b) ERR_ABORTED_VIA_GUI

2

482

Darwin

-185 (0xffffff47) ERR_RESULT_START

3

482

Darwin

1 Unknown error number

1

482

Darwin

131 (0x83) Unknown error number

1

482

Linux

-197 (0xffffff3b) ERR_ABORTED_VIA_GUI

16

482

Linux

-186 (0xffffff46) ERR_RESULT_DOWNLOAD

46

482

Linux

-185 (0xffffff47) ERR_RESULT_START

1

482

Linux

1 Unknown error number

22

482

Linux

2 Unknown error number

25

482

Linux

4 Unknown error number

1

482

Linux

7 Unknown error number

1

482

Linux

11 (0xb) Unknown error number

29

482

Linux

13 (0xd) Unknown error number

33

482

Linux

26 (0x1a) Unknown error number

1

482

Linux

131 (0x83) Unknown error number

154

482

Linux

139 (0x8b) Unknown error number

1

482

Windows

-1073741819 (0xc0000005) Unknown error number

98

482

Windows

-1073741811 (0xc000000d) Unknown error number

30

482

Windows

-1073741502 (0xc0000142) Unknown error number

7

482

Windows

-529697949 (0xe06d7363) Unknown error number

7

482

Windows

-197 (0xffffff3b) ERR_ABORTED_VIA_GUI

561

482

Windows

-187 (0xffffff45) ERR_RESULT_UPLOAD

9

482

Windows

-186 (0xffffff46) ERR_RESULT_DOWNLOAD

26

482

Windows

-185 (0xffffff47) ERR_RESULT_START

4

482

Windows

-177 (0xffffff4f) ERR_RSC_LIMIT_EXCEEDED

4

482

Windows

-164 (0xffffff5c) ERR_NESTED_UNHANDLED_EXCEPTION_DETECTED

115

482

Windows

1 Unknown error number

57

481

Linux

-186 (0xffffff46) ERR_RESULT_DOWNLOAD

1

481

Linux

1 Unknown error number

2

481

Linux

11 (0xb) Unknown error number

2

481

Linux

131 (0x83) Unknown error number

4

479

Linux

1 Unknown error number

1

 

This weekend I’m going to try and get a register dump of each thread added to the diagnostic output.  Along with that I would like to get the function pointers and function parameters added to the diagnostic output.

 

I did manage to shrink the PDB file size for R@H down to 7MB which still seems to be a little steep for mass consumption.  So maybe with the function pointers and parameters we can continue to bring down the error rates.

 

----- Rom