You answered your own Question. The more users connect to your WIFI the slower your speed will become, another point is TV and ADSL have separate configuration so disconnecting one will not speedup the other.