Fund: 🐛 biome breaks characters in stdin mode

Environment information

CLI:
  Version:                      1.7.3
  Color support:                true

Platform:
  CPU Architecture:             aarch64
  OS:                           macos

Environment:
  BIOME_LOG_DIR:                unset
  NO_COLOR:                     unset
  TERM:                         "screen-256color"
  JS_RUNTIME_VERSION:           "v20.12.2"
  JS_RUNTIME_NAME:              "node"
  NODE_PACKAGE_MANAGER:         "yarn/4.1.0"

Biome Configuration:
  Status:                       Loaded successfully
  Formatter disabled:           false
  Linter disabled:              false
  Organize imports disabled:    false
  VCS disabled:                 false

Workspace:
  Open Documents:               0

What happened?

When using the stdin mode in combination with “--colors off", some characters are displayed with broken characters.

I'm not a rustacean but have investigated the code.

The broken characters are Combining character which is represented by multiple code points.

fn main() {
    println!("{}", "⚠️".escape_unicode()); // \u{26a0}\u{fe0f}
    println!("{}", "â".escape_unicode()); // \u{61}\u{302}
    println!("{}", "ｶﾞ".escape_unicode()); // \u{ff76}\u{ff9e}
    println!("{}", "👨🏻‍🦱".escape_unicode()); // \u{1f468}\u{1f3fb}\u{200d}\u{1f9b1}
}

--colors off or not using terminal will result in ColorChoice::Never.

biome/crates/biome_console/src/lib.rs

Lines 94 to 114 in fec262f

    
           fn compute_color(colors: ColorMode) -> (ColorChoice, ColorChoice) { 
        
               match colors { 
        
                   ColorMode::Enabled => (ColorChoice::Always, ColorChoice::Always), 
        
                   ColorMode::Disabled => (ColorChoice::Never, ColorChoice::Never), 
        
                   ColorMode::Auto => { 
        
                       let stdout = if io::stdout().is_terminal() { 
        
                           ColorChoice::Auto 
        
                       } else { 
        
                           ColorChoice::Never 
        
                       }; 
        
                       let stderr = if io::stderr().is_terminal() { 
        
                           ColorChoice::Auto 
        
                       } else { 
        
                           ColorChoice::Never 
        
                       }; 
        
                       (stdout, stderr) 
        
                   } 
        
               } 
        
           }

If you run biome format in stdin mode, the entire code is output here.

biome/crates/biome_cli/src/execute/std_in.rs

Lines 66 to 68 in fec262f

    
           console.append(markup! { 
        
               {output} 
        
           });

Finally, this write_str is called.

biome/crates/biome_console/src/write/termcolor.rs

Line 146 in fec262f

fn write_str(&mut self, content: &str) -> fmt::Result {

In the write_str method, convert characters in the case of Windows or ColorChoice::Never.
By grapheme.chars().nth(0), all combining characters output only the first code point.

biome/crates/biome_console/src/write/termcolor.rs

Lines 167 to 173 in fec262f

    
           if cfg!(windows) || !self.writer.supports_color() { 
        
               let is_ascii = grapheme_is_ascii(grapheme); 
        
               if !is_ascii { 
        
                   let replacement = unicode_to_ascii(grapheme.chars().nth(0).unwrap()); 
        
                   replacement.encode_utf8(&mut buffer);

codesandbox

https://codesandbox.io/p/devbox/serene-swirles-6z4jfd?embed=1&file=%2Fsrc%2Findex.js%3A5%2C1&workspaceId=6833e942-7d98-4ed2-8f40-9a265aa14215

related issues

#2604
#455

Expected result

The output of the code should not be converted anything.

Code of Conduct

I agree to follow Biome's Code of Conduct

Biome/biome

🐛 biome breaks characters in stdin mode

Environment information

What happened?

codesandbox

related issues

Expected result

Code of Conduct

How does funding with Polar work?

Backer

Contributor

Maintainer

	fn compute_color(colors: ColorMode) -> (ColorChoice, ColorChoice) {
	match colors {
	ColorMode::Enabled => (ColorChoice::Always, ColorChoice::Always),
	ColorMode::Disabled => (ColorChoice::Never, ColorChoice::Never),
	ColorMode::Auto => {
	let stdout = if io::stdout().is_terminal() {
	ColorChoice::Auto
	} else {
	ColorChoice::Never
	};

	let stderr = if io::stderr().is_terminal() {
	ColorChoice::Auto
	} else {
	ColorChoice::Never
	};

	(stdout, stderr)
	}
	}
	}

	if cfg!(windows) \|\| !self.writer.supports_color() {
	let is_ascii = grapheme_is_ascii(grapheme);

	if !is_ascii {
	let replacement = unicode_to_ascii(grapheme.chars().nth(0).unwrap());

	replacement.encode_utf8(&mut buffer);

Biome/biome

🐛 biome breaks characters in stdin mode

Environment information

What happened?

codesandbox

related issues

Expected result

Code of Conduct

How does funding with Polar work?

Backer

Why does "Fund on completion" require GitHub login?

When is the invoice due for "Fund on completion"?

What happens if the issue is never completed?

Do I get any extra benefits by funding?

Do I get progress updates?

Contributor

Do I get a reward?

Is rewards guaranteed?

Maintainer

How can I get funding like this for my open source initiatives?