Why did Count Orlok wait years?

Not really a spoiler but tagged anyway. Why wouldn’t he seek Ellen out physically sooner?