That doesn't work for me. I read non-fiction almost exclusively (Can't resist Harry Bosh crime novels by Michael Connelly, though) and I can't read a book without annotating it. You can tell one of my books because of all the underlining, margin notes and cross-references which I invariably pencil in. I'm so bad about it I had to get one of those 3 in 1 pens (black ink, red ink and pencil lead) just to use when marking up a book.
You should keep a book diary.
You would be surprised by the number of good nonfiction books are in the history and biography sections of the library.