If i use all runes of Avarice (include 9x seal and 3x Quintnessence)

It's will be increase 5.3 gold / sec, just wondering if the amount will be rounded down to
5 gold / 10 sec

bump

they won't round it down. It will be just that, 5.3 gold/10 seconds. :P

You'd generate the decimal amount, and eventually it would add up.

The gold is not actually generated every 10 seconds. Nor 5 seconds despite being commonly referred to as "gp5". Nor is it actually generated PER SECOND either. If you stack a bunch of Gold Generation items (and runes, in your case), and WATCH your gold, you will see that it "ticks" in fractions of a second.

The 0.3 will make a difference. It will not be rounded. It will be very hard to notice with the naked eye, but 5.3 will definitely generate slightly more than 5.0. You can easily test this by messing around with your rune setup and just AFK sitting in a game and checking your gold amount at a fixed time (in a custom game, of course).