in ep3, Zemo being Zemo, he talks about Bucky's notebook and points out the list, Sam looks curious but seeing how uncomfortable Bucky was, he tries to change the subject to music and Steve and he never mentions it, on screen, again.
then Sarah calls Sam in the iconic undercover mission scene. Bucky looks as confused as everyone else in the room, it doesn't seem to click for him who Sarah is, or the bank issue or Cass. he looks as lost as Zemo and Selby were.
but in ep 4 and 5 we see both of them mention these things like they have full knowledge of them.
in ep4 when Sarah calls Sam to inform him Karli have called to threaten her and the kids, Bucky asks what's happening and Sam says "Karli called Sarah" he doesn't say my sister, because Sarah is on bucky's map now and he knows who she exactly is.
in ep5 when bucky goes to Sam's house to give him the suit, he says "so, this is the boat?" because he heard about it, how it's not working and that they might sell it.
then again in ep5, sam asks "are you still having those nightmares" he doesn't ask if he generally has any, he knows that they are the ones relating to the list that was in the notebook.
there was definitely a deadtime in Sharon's house before the party, or they didn't join from the beginning of the party. because these two sat together and discussed the boat, the bank, Sarah and the kids, the list, the notebook, the nightmares and Steve. just imagine them in Madripoor, on a balcony in Sharon's house, a drink in their hands and they just talk about everything.
so in conclusion: we were robbed.