The RFC 4180 standard states that the delimiter for CSV must be comma. Excel, however, ignores this requirement and uses semicolon in those countries where the decimal delimiter is comma. So, what in USA would be
Fred,12.4,276,"Hello World!"
in Italy, Spain, and France, the exported file contains
Fred;12,4;276;"Hello World!"
Here we have two problems: the first is that such a CSV file cannot be imported in any application that is compliant to RFC 4180 standard, the second one, is that you are forced to edit the CSV file and add "sep=;" before the first record to ensure that anybody that receive your file in a country where comma delimiters are used, be able to import it in Excel. The latter problem has a collateral effect: that instruction is Excel-specific, not part of RFC 4180 standard, and not used by any application that is compliant to RFC 4180 standard.
Usually Microsoft propose two solutions to this problem. The first one is to clear the check box 'Use system settings' in advanced options and to set 'Decimal separator' to a point (dot), and 'Thousands separator' to a comma, save the CSV file, and set the check box 'Use system settings'. Doing this every time is certainly not the best. Furthermore, your decimal values are using the dot, which is not necessarily what you want.
The second solution is the worst, that is, changing in the control panel, in regional settings, the List separator to comma. This affects the behavior of all programs on your machine and will have unpredictable effects.
The best approach would be to ask Excel to support the standard, so that the file would be
Fred,"12,4",276,"Hello World!"