Abstract. We present a method for user-based detection and ranking of important video key-frames. Instead of content-based analysis that detects object, shot, and scene changes, we analyze aggregate user interactions (e.g., pause, seek/scrub) within a web video. Moreover, we validated the proposed method in a controlled lab experiment with lecture videos that content-based approaches cannot structure meaningfully. In particular, we modeled the collective information seeking behavior as a time series of user interest. We assumed that replay of a video segment stands for increased user interest in that segment and skip stands for less interesting parts. We found that only the replay time series matches significantly well the semantics of the lecture video. In practice, user-based detection of interesting video frames might improve navigation within information-rich but visually unstructured videos (e.g., lectures) on the web and has also the potential to improve video search results with personalized video thumbnails.