Just found the PR that introduced the csv_headers option.
It was not intended to be used to change the header labels but only to reorder columns.

I may open a PR to change that behaviour with something like this:

If the key of the array passed to csv_headers is a string, then use that key as the header's label.

$contextBuilder = (new CsvEncoderContextBuilder())->withHeaders([
    'Prénom' => 'firstName',
    'Nom' => 'lastName',
    'Nested field label' => 'nested.field',
]);

We should be able to keep BC like this. What do you think?

I think your example would work fine if you added #[SerializedName]/#[SerializedPath] attributes into your entity class:

class YourEntity
{
  #[SerializedName('Prénom')]
  public string $firstName;

  #[SerializedName('Nom')]
  public string $lastName;
}

Of course it may be a design decision to not want to do that, especially until #53858 is implemented.

Anyway, I don't see any bug here, in your serializer context you've defined

1. Only serialize firstName & lastName properties from your entity
2. Treat first column in csv as Prénom and 2nd column as Nom

1 & 2 do not know anything about each other, no mapping is supposed to be happening apart from name converter.

I did not know about the SerializedName and yes, it works as expected.

However, since i want to set the label only when serializing into CSV, I do not want to use attributes on entities (looks like I cannot specify that the names should be used only for CSV).
And I cannot find how to pass the labels in context builder.