CUDA Releases: Component Versions and Sizes

While I was working on something else, I stumbled across a PR in Trilinos that referenced a problem in how Kokkos Kernels handles cuSPARSE.

The basic problem is that the cuSPARSE version does not necessarily match the CUDA release version. This means that the cuSPARSE API sometimes changes on a patch release of CUDA. Furthermore, the cuSPARSE release that comes with each CUDA release is not documented anywhere, as far as I can tell. In Kokkos Kernels, we were using the CUDA release version to detect whether certain cuSPARSE APIs were available, which is wrong, and we were also setting the wrong release threshold.

I made two PRs to fix the problem for cusparseSpMV and cusparseSpMM functions.

For future reference, here are the cuSPARSE versions that correspond to each CUDA release:

CUDA 12

CUDA ReleasecuSPARSEcuBLAScuSOLVERcuFFT
12.4.112.3.1.17012.4.5.811.6.1.911.2.1.3
12.3.212.2.0.10312.3.4.111.5.4.10111.0.12.1
12.3.112.2.0.10312.3.4.111.5.4.10111.0.12.1
12.3.012.1.3.15312.3.2.911.5.3.5211.0.11.19
12.2.212.1.2.14112.2.5.611.5.2.14111.0.8.103
12.2.112.1.2.12912.2.4.511.5.1.12911.0.8.91
12.2.012.1.1.5312.2.1.1611.5.0.5311.0.8.15
12.1.112.1.0.10612.1.3.111.4.5.10711.0.2.54
12.1.012.0.2.5512.1.0.2611.4.4.5511.0.2.4
12.0.112.0.1.14012.0.2.22411.4.3.111.0.1.95
12.0.012.0.0.7612.0.1.18911.4.2.5711.0.0.21

CUDA 11

CUDA ReleasecuSPARSEcuBLAScuSOLVERcuFFT
11.0.111.0.0.19111.0.0.19110.4.0.19110.1.3.191
11.0.211.1.0.21811.1.0.22910.5.0.21810.2.0.218
11.0.311.1.1.24511.2.0.25210.6.0.24510.2.1.245
11.1.011.2.0.27511.2.1.7411.0.0.7410.3.0.74
11.1.111.3.0.1011.3.0.10611.0.1.10510.3.0.105
11.2.011.3.1.6811.3.1.6811.0.2.6810.4.0.72
11.2.111.4.0.13511.4.1.102611.1.0.13510.4.0.135
11.2.211.4.1.115211.4.1.104311.1.0.15210.4.1.152
11.2.011.3.1.6811.3.1.6811.0.2.6810.4.0.72
11.2.111.4.0.13511.4.1.102611.1.0.13510.4.0.135
11.3.011.5.0.5811.4.2.1006411.1.1.5810.4.2.58
11.3.111.6.0.10911.5.1.10911.1.2.10910.4.2.109
11.4.011.6.0.4311.5.2.4311.2.0.4310.5.0.43
11.4.111.6.0.10011.5.4.811.2.0.10010.5.1.100
11.4.211.6.0.12011.6.1.5111.2.0.12010.5.2.100
11.4.311.6.0.12011.6.5.211.2.0.12010.5.2.100
11.4.411.6.0.12011.6.5.211.2.0.12010.5.2.100
11.5.011.7.0.3111.7.3.111.2.1.4810.6.0.54
11.5.111.7.0.10711.7.4.611.3.2.10710.6.0.107
11.5.211.7.0.10711.7.4.611.3.2.10710.6.0.107
11.6.011.7.1.5511.8.1.7411.3.2.5510.7.0.55
11.6.111.7.2.11211.8.1.7411.3.3.11210.7.1.112
11.6.211.7.2.12411.9.2.11011.3.4.12410.7.2.124
11.7.011.7.3.5011.10.1.2511.3.5.5010.7.2.50
11.7.111.7.4.9111.10.3.6611.4.0.110.7.2.91
11.8.011.7.5.8611.11.3.611.4.1.4810.9.0.58

CUDA 10

In the CUDA 10 releases, the versions are programmatically accessible through a library call rather than being defined in the headers, so I couldn't just grep them out of the installation.

Release Component Sizes

Since I had to download every single CUDA release to extract this information, I was curious about how the releases have evolved over time.

Stacked plot of CUDA component sizes

CUDA 12

CUDA ReleaseSize (K)cuSPARSE SizecuBLAS Sizenvcc SizecuFFT SizecuRAND SizecuSOLVER Sizenpp SizeNsight ComputeNsight SystemscuPTI SizeCUDA GDB Sizecudart Sizenvrtc Sizensight Sizedriver Size
12.4.17588644582172 (7.67%)1396132 (18.39%)22144 (.29%)921144 (12.13%)190708 (2.51%)352052 (4.63%)497568 (6.55%)1395692 (10.41%)790260 (18.39%)115456 (1.52%)57108 (.75%)7696 (.10%)151244 (1.99%)115900 (1.52%)299672 (3.94%)
12.3.27482692554300 (7.40%)1540360 (20.58%)21276 (.28%)564224 (7.54%)190192 (2.54%)343268 (4.58%)504648 (6.74%)1535344 (10.49%)785304 (20.51%)111948 (1.49%)79564 (1.06%)7624 (.10%)145820 (1.94%)115904 (1.54%)317220 (4.23%)
12.3.17482712554304 (7.40%)1540360 (20.58%)21276 (.28%)564228 (7.54%)190192 (2.54%)343268 (4.58%)504648 (6.74%)1535352 (10.49%)785304 (20.51%)111948 (1.49%)79564 (1.06%)7624 (.10%)145820 (1.94%)115904 (1.54%)317224 (4.23%)
12.3.07458604548472 (7.35%)1537560 (20.61%)21280 (.28%)564228 (7.56%)190192 (2.54%)343272 (4.60%)489568 (6.56%)1534960 (10.52%)785308 (20.57%)111932 (1.50%)79552 (1.06%)7624 (.10%)145820 (1.95%)115908 (1.55%)317220 (4.25%)
12.2.27394064551892 (7.46%)1536844 (20.78%)20784 (.28%)562828 (7.61%)191372 (2.58%)342780 (4.63%)489096 (6.61%)1503072 (10.36%)766144 (20.32%)110996 (1.50%)79356 (1.07%)7456 (.10%)134212 (1.81%)115896 (1.56%)332952 (4.50%)
12.2.17372660551892 (7.48%)1516656 (20.57%)20784 (.28%)562828 (7.63%)191372 (2.59%)342932 (4.65%)489096 (6.63%)1503036 (10.39%)766116 (20.38%)110968 (1.50%)79376 (1.07%)7456 (.10%)134136 (1.81%)115900 (1.57%)332580 (4.51%)
12.2.07444556556224 (7.47%)1584144 (21.27%)20784 (.27%)562832 (7.56%)191376 (2.57%)341372 (4.58%)482728 (6.48%)1502952 (10.29%)766116 (20.18%)110724 (1.48%)79372 (1.06%)7284 (.09%)134992 (1.81%)115824 (1.55%)338980 (4.55%)
12.1.17326380558904 (7.62%)1584144 (21.62%)15680 (.21%)581476 (7.93%)191048 (2.60%)336516 (4.59%)485968 (6.63%)1431064 (9.93%)727968 (19.53%)110344 (1.50%)79404 (1.08%)7064 (.09%)141692 (1.93%)115828 (1.58%)336084 (4.58%)
12.1.07808004539064 (6.90%)1604572 (20.55%)15676 (.20%)581472 (7.44%)191048 (2.44%)819524 (10.49%)485968 (6.22%)1430572 (9.32%)727968 (18.32%)110336 (1.41%)79384 (1.01%)7064 (.09%)141692 (1.81%)115828 (1.48%)336084 (4.30%)
12.0.17707864539280 (6.99%)1604612 (20.81%)14908 (.19%)466724 (6.05%)191052 (2.47%)819584 (10.63%)485940 (6.30%)1408372 (9.16%)706688 (18.27%)107428 (1.39%)79408 (1.03%)6972 (.09%)140244 (1.81%)115828 (1.50%)404788 (5.25%)
12.0.07592228450292 (5.93%)1581884 (20.83%)14908 (.19%)466720 (6.14%)191020 (2.51%)819440 (10.79%)485740 (6.39%)1407216 (9.30%)706684 (18.53%)107396 (1.41%)79376 (1.04%)6972 (.09%)140208 (1.84%)115828 (1.52%)403668 (5.31%)

CUDA 11

CUDA ReleaseSize (K)cuSPARSE SizecuBLAS Sizenvcc SizecuFFT SizecuRAND SizecuSOLVER Sizenpp SizeNsight ComputeNsight SystemscuPTI SizeCUDA GDB Sizecudart Sizenvrtc Sizensight Sizedriver Size
11.0.15473984282904 (5.16%)506536 (9.25%)236 (0%)480532 (8.77%)140688 (2.57%)913816 (16.69%)296768 (5.42%)688368 (16.01%)876460 (12.57%)61680 (1.12%)52144 (.95%)19716 (.36%)27140 (.49%)115796 (2.11%)138812 (2.53%)
11.0.25414976321716 (5.94%)593940 (10.96%)240 (0%)487528 (9.00%)140688 (2.59%)982344 (18.14%)299232 (5.52%)689688 (11.20%)606528 (12.73%)62468 (1.15%)52380 (.96%)19744 (.36%)27148 (.50%)115800 (2.13%)137728 (2.54%)
11.0.35468600320552 (5.86%)608352 (11.12%)240 (0%)487396 (8.91%)140696 (2.57%)1021264 (18.67%)299124 (5.46%)689836 (11.09%)606520 (12.61%)62520 (1.14%)52368 (.95%)19744 (.36%)27148 (.49%)115804 (2.11%)139756 (2.55%)
11.1.05950748462280 (7.76%)745224 (12.52%)4992 (.08%)654456 (10.99%)148380 (2.49%)1273416 (21.39%)400244 (6.72%)645872 (10.25%)610260 (10.85%)65584 (1.10%)49072 (.82%)20276 (.34%)37388 (.62%)115800 (1.94%)164864 (2.77%)
11.1.16022572477860 (7.93%)796912 (13.23%)5004 (.08%)654448 (10.86%)148372 (2.46%)1276128 (21.18%)400200 (6.64%)646288 (10.13%)610244 (10.73%)65712 (1.09%)49108 (.81%)20276 (.33%)37400 (.62%)115804 (1.92%)166012 (2.75%)
11.2.05373772367400 (6.83%)704464 (13.10%)5344 (.09%)530992 (9.88%)148632 (2.76%)956780 (17.80%)339212 (6.31%)657048 (11.22%)603380 (12.22%)71228 (1.32%)49104 (.91%)20092 (.37%)86976 (1.61%)115800 (2.15%)167828 (3.12%)
11.2.15350528472788 (8.83%)707684 (13.22%)5348 (.09%)532240 (9.94%)164396 (3.07%)789948 (14.76%)351252 (6.56%)656588 (11.27%)603380 (12.27%)71380 (1.33%)49092 (.91%)20096 (.37%)86988 (1.62%)115800 (2.16%)173916 (3.25%)
11.2.25350552472744 (8.83%)707692 (13.22%)5352 (.10%)532232 (9.94%)164396 (3.07%)789924 (14.76%)351252 (6.56%)656604 (11.27%)603380 (12.27%)71412 (1.33%)49104 (.91%)20092 (.37%)86996 (1.62%)115804 (2.16%)173916 (3.25%)
11.2.05373772367400 (6.83%)704464 (13.10%)5344 (.09%)530992 (9.88%)148632 (2.76%)956780 (17.80%)339212 (6.31%)657048 (11.22%)603380 (12.22%)71228 (1.32%)49104 (.91%)20092 (.37%)86976 (1.61%)115800 (2.15%)167828 (3.12%)
11.2.15350528472788 (8.83%)707684 (13.22%)5348 (.09%)532240 (9.94%)164396 (3.07%)789948 (14.76%)351252 (6.56%)656588 (11.27%)603380 (12.27%)71380 (1.33%)49092 (.91%)20096 (.37%)86988 (1.62%)115800 (2.16%)173916 (3.25%)
11.3.05343924467992 (8.75%)707076 (13.23%)5548 (.10%)533512 (9.98%)164600 (3.08%)614128 (11.49%)361148 (6.75%)666900 (12.40%)663088 (12.47%)74220 (1.38%)49040 (.91%)6116 (.11%)88832 (1.66%)115820 (2.16%)253768 (4.74%)
11.3.15616324484072 (8.61%)930724 (16.57%)5552 (.09%)533516 (9.49%)164608 (2.93%)641976 (11.43%)361152 (6.43%)667192 (11.80%)663084 (11.87%)78440 (1.39%)49048 (.87%)6116 (.10%)88852 (1.58%)115812 (2.06%)253768 (4.51%)
11.4.06355576482848 (7.59%)951192 (14.96%)5720 (.08%)1111808 (17.49%)164744 (2.59%)663456 (10.43%)371800 (5.84%)700748 (10.89%)692204 (11.02%)83444 (1.31%)76672 (1.20%)6268 (.09%)89788 (1.41%)115820 (1.82%)265420 (4.17%)
11.4.16347244482544 (7.60%)999180 (15.74%)5728 (.09%)1111820 (17.51%)164748 (2.59%)669904 (10.55%)383696 (6.04%)700788 (10.90%)692200 (11.04%)83444 (1.31%)76716 (1.20%)6272 (.09%)50436 (.79%)115828 (1.82%)266824 (4.20%)
11.4.26446744482544 (7.48%)1074584 (16.66%)5732 (.08%)1111812 (17.24%)164744 (2.55%)669904 (10.39%)375700 (5.82%)701776 (11.21%)723104 (10.88%)83444 (1.29%)76680 (1.18%)6272 (.09%)50436 (.78%)115820 (1.79%)266820 (4.13%)
11.4.36445656482540 (7.48%)1074892 (16.67%)5736 (.08%)1111808 (17.24%)164744 (2.55%)669904 (10.39%)375700 (5.82%)701776 (11.21%)723100 (10.88%)83444 (1.29%)76676 (1.18%)6280 (.09%)50444 (.78%)115824 (1.79%)265408 (4.11%)
11.4.46445708482544 (7.48%)1074892 (16.67%)5736 (.08%)1111812 (17.24%)164748 (2.55%)669896 (10.39%)375700 (5.82%)701784 (11.21%)723108 (10.88%)83444 (1.29%)76676 (1.18%)6280 (.09%)50440 (.78%)115820 (1.79%)265412 (4.11%)
11.5.05959068481936 (8.08%)1108280 (18.59%)5860 (.09%)452552 (7.59%)144268 (2.42%)590112 (9.90%)394936 (6.62%)818388 (12.42%)740408 (13.73%)83248 (1.39%)76700 (1.28%)6420 (.10%)126300 (2.11%)115828 (1.94%)275476 (4.62%)
11.5.16060672457072 (7.54%)1144676 (18.88%)5860 (.09%)451424 (7.44%)173720 (2.86%)638772 (10.53%)394932 (6.51%)826820 (12.21%)740404 (13.64%)87660 (1.44%)76716 (1.26%)6420 (.10%)126348 (2.08%)115828 (1.91%)275476 (4.54%)
11.5.26060640457072 (7.54%)1144672 (18.88%)5860 (.09%)451424 (7.44%)173716 (2.86%)638768 (10.53%)394928 (6.51%)826816 (12.21%)740404 (13.64%)87660 (1.44%)76716 (1.26%)6420 (.10%)126348 (2.08%)115828 (1.91%)275472 (4.54%)
11.6.06160404492708 (7.99%)1144824 (18.58%)6324 (.10%)451416 (7.32%)173664 (2.81%)695876 (11.29%)398216 (6.46%)835516 (9.63%)593560 (13.56%)88844 (1.44%)76852 (1.24%)6488 (.10%)390672 (6.34%)115828 (1.88%)320644 (5.20%)
11.6.16214020538324 (8.66%)1144828 (18.42%)6324 (.10%)451420 (7.26%)173664 (2.79%)694916 (11.18%)406204 (6.53%)835964 (9.55%)593568 (13.45%)88860 (1.42%)76952 (1.23%)6488 (.10%)390676 (6.28%)115824 (1.86%)321120 (5.16%)
11.6.25995104538328 (8.97%)1174688 (19.59%)6328 (.10%)451416 (7.52%)173664 (2.89%)708628 (11.82%)406216 (6.77%)835976 (9.90%)593544 (13.94%)88860 (1.48%)76960 (1.28%)6488 (.10%)127100 (2.12%)115824 (1.93%)321120 (5.35%)
11.7.06063180500564 (8.25%)1170468 (19.30%)6388 (.10%)450500 (7.43%)173740 (2.86%)738160 (12.17%)405324 (6.68%)864656 (10.08%)611760 (14.26%)91980 (1.51%)77016 (1.27%)6544 (.10%)128084 (2.11%)115828 (1.91%)350344 (5.77%)
11.7.16096648510704 (8.37%)1130248 (18.53%)6392 (.10%)450500 (7.38%)173740 (2.84%)642528 (10.53%)438192 (7.18%)859920 (10.03%)611776 (14.10%)92008 (1.50%)77032 (1.26%)6544 (.10%)128088 (2.10%)115828 (1.89%)355652 (5.83%)
11.8.07672492589240 (7.67%)1716472 (22.37%)14656 (.19%)862456 (11.24%)200124 (2.60%)821540 (10.70%)503032 (6.55%)939560 (9.12%)700068 (12.24%)103548 (1.34%)77088 (1.00%)6880 (.08%)138016 (1.79%)115828 (1.50%)395384 (5.15%)

CHDA 10

CUDA ReleaseSize (K)cuSPARSEcuBLASnvcccuFFTcuRANDcuSOLVERnppNsight ComputeNsight SystemscuPTICUDA GDBcudartnvrtcnsightdriver
10.0331355657696 (1.74%)69140 (2.08%)192 (0%)101312 (3.05%)59384 (1.79%)135996 (4.10%)127476 (3.84%)463024 (0%)0 (13.97%)6064 (.18%)9260 (.27%)488 (.01%)24364 (.73%)187804 (5.66%)0 (0%)
10.1377973288212 (2.33%)112672 (2.98%)192 (0%)114536 (3.03%)58412 (1.54%)178280 (4.71%)133128 (3.52%)478952 (6.11%)230992 (12.67%)5552 (.14%)9260 (.24%)496 (.01%)25896 (.68%)185344 (4.90%)0 (0%)
10.13948192119212 (3.01%)92892 (2.35%)192 (0%)135444 (3.43%)58412 (1.47%)178196 (4.51%)132240 (3.34%)472972 (7.68%)303228 (11.97%)5512 (.13%)9260 (.23%)496 (.01%)25916 (.65%)185348 (4.69%)0 (0%)
10.13900648115200 (2.95%)90292 (2.31%)196 (0%)135412 (3.47%)58412 (1.49%)173012 (4.43%)133356 (3.41%)615852 (8.06%)314420 (15.78%)5568 (.14%)9260 (.23%)496 (.01%)25924 (.66%)185396 (4.75%)0 (0%)
10.23895664125308 (3.21%)95452 (2.45%)228 (0%)150160 (3.85%)58676 (1.50%)186048 (4.77%)142932 (3.66%)486324 (8.46%)329748 (12.48%)5628 (.14%)9256 (.23%)500 (.01%)26272 (.67%)183124 (4.70%)0 (0%)

Symbols

This section counts the external dynamic symbols present in the libraries for a particular component.

Line chart of dynamic symbols present in libraries in each CUDA release

CUDA ReleasecuSPARSEcuBLAScuSOLVERcuFFTcuRANDcudartcuptinpp
12.4.112783237342639990147367641097
12.3.212723225348639390144670641097
12.3.112723225348639390144670641097
12.3.012723111348639390144670640067
12.2.212723111348640290140371740067
12.2.112723111348640290140371740067
12.2.012723111351940290140371739893
12.1.112723111352540290138565339691
12.1.012543111352540290138565339691
12.0.112543111352540290137265239787
12.0.012543111352540290137265239787
CUDA ReleasecuSPARSEcuBLAScuSOLVERcuFFTcuRANDcudartcuptinpp
11.0.113141731342640890132841437831
11.0.213321731344440890132841438084
11.0.313321752344440890132841438084
11.1.013321752345639990137546038084
11.1.113561752345639990137546038084
11.2.013562088350139990147046138299
11.2.113712088350139990147046138313
11.2.213712088350139990147046138313
11.2.013562088350139990147046138299
11.2.113712088350139990147046138313
11.3.013922088352539990153851738313
11.3.114192094348939990153851738313
11.4.014192094352539990157146838233
11.4.114192094352539990157146838233
11.4.214192106352539990157146838233
11.4.314192106352539990157146838233
11.4.414192106352539990157146838233
11.5.014222106352539990157148738329
11.5.114492106352539990157148738329
11.5.214492106352539990157148738329
11.6.014492106352539990158348938335
11.6.114492106352539990158348938748
11.6.214492154352539990158348938748
11.7.014522154352539990159953838748
11.7.114522154352539990159953839407
11.8.014522424352540290161555039407
CUDA ReleasecuSPARSEcuBLAScuSOLVERcuFFTcuRANDcudartcuptinpp
10.08084776736128289956229
10.1832489717612828911212528
10.1857487741612828911412538
10.1546487736612828911412637
10.2546487830612829511612637

Methodology

The code to download the CUDA releases and extract the data are available here cwpearson/cusparse-versions. I used the Linux runfiles - these self-extracting CUDA installers seem to run a built-in binary, and since I'm using the x86 versions of CUDA, this needs to be run on an x86 machine.

How the sizes are measured varies a bit between CUDA 11/12 and CUDA 10. For CUDA 11/12, the --extract flag to the runfile creates a single nice, neat directory for each component. That directory includes all the binaries, headers, libraries, and whatever else Nvidia ships for that component. For these releases, the measured size is the size of this complete directory. For CUDA 10, the --extract flag only creates 3 additional runfiles - one for the driver, one for the samples, and one for the toolkit. I found interacting with those files difficult, so I just completed the CUDA 10 install, and the sizes of the components is the size of only the binary or the libraries, not including the headers and whatever else. I still used the --extract flag just to get the driver on its own to determine the driver size for CUDA 10.

Component symbols are the number of defined dynamic symbols in the libraries for a component (nm -D --defined-only).