I saw Orson Welles' "Citizen Kane" only once, many years ago. But I noticed that a number of images didn't seem realistic. For example, some of the buildings in New York street scenes of the early 1900s looked stiffly unnatural. Unfortunately, I could not find examples of those on the Internet, but have others that support my contention. Chesley Bonestell and maybe other matte artists working on the movie created sharp-edge paintings, perhaps according to Welles' wishes or possibly because that was their preferred style. The extensive Wikipedia entry linked in this paragraph notes that Welles strived to achieve "deep focus," where both near and distant objects appeared sharply defined. I'm not sure why he wanted this, but it might help explain why some of the matte shots (and the film had many) didn't work well because the paintings had to be crisply done, increasing their potential artificiality. Then again, maybe Welles didn't really want the movie to look completely realistic. Film and "Citizen Kane" buffs, please clarify this in Comments.
Much matte art was done using large brushes, especially for background objects that would be slightly out of focus anyway. Where sharp edged features such as door frames, bannisters and other manufactured objects were close to the camera's focal plane, then matte artists had to use distinct edges to blend their painting with those objects in the movie set. Otherwise, looser painting seemed to work better, especially in the years when film stock did not permit high definition for any of what was being shown.
Another point regarding matte painting is that such paintings served best for scenes lasting less than ten seconds; better yet, less than five seconds. Otherwise, matte artists feared that viewers would discover the artifice.
Here are some examples of scenes using matte painting.
Kane's Xanadu, in the distance, might not quite be in sharp focus (the image I found seems a little blurred all over). But it doesn't seem "real" to me.
The matte part is the convention floor and seating. Looks like a painting.
Another false-looking scene. The nearest few rows probably had live actors, and there might have been some on the stage, but I can't be sure, given this static image.
The Gothic windows seem a little too sharply defined.
The reflection of the distant figure on the floor is part of the painting.
The "Star Wars" movies used a variety of special effects including matte painting. Most of the mattes were hard to detect, but a few jarred me, especially those of the Cloud City in "The Empire Strikes Back". The matte artist was Ralph McQuarrie, who did the concept art for the original Star Wars movie but had never done matte painting previously. For some reason, his paintings of Cloud City on the planet Bespin looked phoney, yet were kept in the movie.
Everything is in too sharp focus and there seems to be a lack of reflected light. Also, the mattes were on view for a substantial time, which allowed their unreality to sink in. There might be other problems, but I find it hard to put a finger on them; perhaps some readers can help here too. But the main thing for me was that this special effects lapse (along with a few more minor ones) lessened my enjoyment of the movie when I first viewed it.