allow to config max segmentation tokens length for RAG document using environment variable

[rainchen]

Description

allow to config max segmentation tokens length for RAG document using environment variable

Fixes # (issue)

Fix #2438 #3290
also #4071

Type of Change

Please delete options that are not relevant.

• This change requires a documentation update, included: Dify Document
• Improvement, including but not limited to code refactoring, performance optimization, and UI/UX improvement

How Has This Been Tested?

manually tested

Suggested Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods
• optional I have made corresponding changes to the documentation
• optional I have added tests that prove my fix is effective or that my feature works
• optional New and existing unit tests pass locally with my changes

[JohnJyong]

pls add the default value for the new variable in the file : config.py , thanks @rainchen

[rainchen]

pls add the default value for the new variable in the file : config.py , thanks @rainchen

@JohnJyong refactored, please review again

[JohnJyong]

LGTM

[rennokki]

@rainchen AMAZING PR! I was wondering if something like this is possible because I was having some texts that were > 1k and they got broke down.

[daigo38]

I have set the docker-compose.yaml environment variable on self-host and restarted the container, but the changes are not applied.
I am building with the “langgenius/dify-api:0.6.8” image, is this feature already enabled?