I tried a number of ways to work around this. It comes down to the excel ODBC driver converting everything to text 'to be safe'. The most complex sample I tried was using a custom query that looked like:
INSERT INTO [Sheet1$] ([Numbers]) VALUES ({fn CONVERT (?, SQL_INTEGER})
I also tried passing in the parameter as an integer or double with the same result. Info on CONVERT function: http://msdn.microsoft.com/en-us/library/ms713608(v=vs.85).aspx
This article: http://spreadsheets.about.com/od/Text-Functions/ss/2011-05-26-excel-convert-text-to-numbers.htm specifies how to work around this issue inside the spreadsheet. It basically indicates to create another column that uses the VALUE function to cast the inserted values and then sum that column. I think this is the method Kenneth is suggesting.
I recommend using a more robust database system then Excel when updating or inserting values is required. Ultimately it comes down to Excel not being a real database. Excel tends to not work well when multiple instances try to update or if it is updated while the spreadsheet is open.