Good Download Speed Bad Upload Speed

I have had problems with my internet for a long time now. My download speed is terrible, but my upload speed seems to always be good. With Ookla's speedtest I get the following results: Ping: 1ms Download Speed: 3.82 Mbps Upload Speed 41.61 Mbps. That is over 35 Mbps difference! The download speeds has sometimes been good, with speeds up to 30Mbps, but that is rearly, and it is just for max 1 hour. My pc is placed far away from the router with walls and a floor between it. My cellphone also get the same results, with a much much higher upload speed than download speed (LG G2)
I resently changed my WLAN card from TP-link TL-WDN4800 to Asus PCE-AC68 since it stopped working.
I have run AVG antivirus and MBAM trying to find viruses etc, but I havn't found anything.
There are many devices connected to the wifi. 1-4 PCs, 1-4 Cellphones, and some other minor stuff.
Homebuild PC
MoBo: Asus Maximus VI Hero
PSU: Corsair AX760i
WLAN card: Both TP-link TL-WDN4800 and Asus PCE-AC68
Windows 7 Ultimate 64-bit with all the latest drivers
Router: Asus RT-AC68U, and I had the same issues with a cisco router
We pay for 120Mbps network, but i dont remember the IPS
Does anyone have any idea why my download speed is so much slower than my upload speed?
All help is appreciated