Could one be accused of favoritism just because one spends more time with some than others?

Yes.

James 2:1- My brothers and sisters, believers in our glorious Lord Jesus Christ must not show favoritism. 

James states clearly that all forms of favoritism are wrong and then proceeds in verse 2 onwards to give merely an example:

2 Suppose a man comes into your meeting wearing a gold ring and fine clothes, and a poor man in filthy old clothes also comes in. 3 If you show special attention to the man wearing fine clothes and say, “Here’s a good seat for you,” but say to the poor man, “You stand there” or “Sit on the floor by my feet,” 4 have you not discriminated among yourselves and become judges with evil thoughts?