If you trust your GF and you have a very good relationship then you don't have to worry about a thing. Just show your GF how important she is to you and never mind that co-worker of yours coz he'll get tired of his fantasies someday. Your being conscious of him might start a fight between you and your GF.

"Do not do unto others what you don't want others to do unto you(Wag kang gay-an, kung mang gagay-an ka, wag gay-on)"