Maybe I'm not explaining myself properly, or it's another one of those issues with text where meaning is not conveyed as it is originally intended.
When a writer tells a reader to "use your imagination," in most cases in my experience, they were covering up a plot hole either after the fact, or something that came up that they missed and just didn't feel like fixing.
Here's a terrible but mostly viable example: Someone is imprisoned for a terrible crime, and they've remained so for quite a while. A major plot point is some mysterious individual letting them go free, they used a key to open the door, and then disappeared before the criminal could see who it was that freed them. Now, this "mysterious individual" is never identified. The author never reveals who it was, and while it's not really terribly important as to who freed the evil guy, it would still be nice to know. Yes, the evil guy now being free is more important than whoever freed him, but that still should be relevant knowledge made known. The individual who freed evil guy is never identified, and when asked, author just says "use your imagination."
Maybe not the best example, but it almost always something like this that causes me to get angry about the "use your imagination" advice.
Also, I mentioned most of my anger for the "use your imagination" advice, is geared for the most part, towards stories that have open-endings. To me it just stinks of laziness where the author didn't know how to end it, or couldn't decide. Case in point, my Cowboy Bebop example and the later interview Shinichiro Watanabe gave about the ending.