[Bug]: ConsoleNinja augments files that are not being executed

[somewhatabstract]

What happened?

In my project, checksync, I have a folder of examples (named __examples__) that are not executed code but are instead used by integration tests. Unfortunately, Console Ninja is still modifying them on load and causing the tests to fail when VSCode is running.

Version

v0.0.215

Steps to reproduce (or sample repo)

1. Make sure VSCode isn't running and open a terminal window
2. Clone the somewhatabstract/checksync repo
3. Run yarn (you need yarn v1 installed - using npx yarn@1 should work if it isn't installed)
4. Without VSCode running
5. Run yarn test ./src/__tests__/integration.test.ts
6. Tests should pass
7. Open the project in VSCode with Console Ninja installed
8. In the same terminal as used in step 5, re-run the tests
9. Note that they fail
10. Pause Console Ninja
11. Run the tests again
12. They should now pass

You can verify that the console.log calls are getting modified in the example code by adding a console.log in the ./src/checksum.ts file to do console.log(saltedContent); and you will see that when Console Ninja is enabled, it the content includes an oo_oo type call on each console.log in the example code.

It appears that Console Ninja hooks all file reads and modifies them, even if they are not executed code. At the very least, there should be a way to tell it not to process certain files or to only include certain files (like include src only); ideally it should be able to work this out from the existing configurations (like babel config?), but I recognize that's hard.

Log output

19:05:48.169 info buildHook-62161 installing build hook for jest at /users/jeffyates/git/checksync, node v16.20.2
19:05:48.173 info buildHook-62161 allowed tools: vite, jest, webpack, next.js, cypress, http-server, serve, live-server, nuxt, remix, qwik, hydrogen, serverless, astro, node
19:05:48.173 info buildHook-62161 running tools: jest
19:05:48.174 info buildHook-62161 installing fs interceptor
19:05:48.321 info host client connected: buildHook { address: '::', family: 'IPv6', port: 62450 }
19:05:48.323 info buildHook-62161 allowed tools: vite, jest, webpack, next.js, cypress, http-server, serve, live-server, nuxt, remix, qwik, hydrogen, serverless, astro, node
19:05:48.323 info buildHook-62161 running tools: jest
19:05:48.532 info buildHook-62161 file processed (sync): 0 log points, error handler: false
19:05:48.787 info buildHook-62161 file processed (stat): 0 log points, error handler: false
19:05:49.027 info buildHook-62161 file processed (stat): 1 log points, error handler: false
19:05:49.160 info buildHook-62161 file processed (stat): 1 log points, error handler: false
19:05:49.162 info buildHook-62161 file processed (stat): 1 log points, error handler: false
19:05:49.286 info buildHook-62161 file processed (stat): 0 log points, error handler: false
19:05:49.408 info buildHook-62161 file processed (stat): 1 log points, error handler: false
19:05:49.530 info buildHook-62161 file processed (stat): 1 log points, error handler: false
19:05:49.653 info buildHook-62161 file processed (stat): 1 log points, error handler: false
19:05:49.913 info buildHook-62161 file processed (stat): 1 log points, error handler: false
19:05:50.035 info buildHook-62161 file processed (stat): 1 log points, error handler: false
19:05:50.036 info buildHook-62161 file processed (stat): 1 log points, error handler: false
19:05:50.157 info buildHook-62161 file processed (stat): 1 log points, error handler: false
19:05:50.158 info buildHook-62161 file processed (stat): 1 log points, error handler: false
19:05:50.158 info buildHook-62161 file processed (stat): 1 log points, error handler: false
19:05:50.282 info buildHook-62161 file processed (stat): 1 log points, error handler: false
19:05:50.402 info buildHook-62161 file processed (stat): 1 log points, error handler: false
19:05:50.403 info buildHook-62161 file processed (stat): 1 log points, error handler: false
19:05:50.524 info buildHook-62161 file processed (stat): 1 log points, error handler: false
19:05:50.645 info buildHook-62161 file processed (stat): 1 log points, error handler: false
19:05:50.766 info buildHook-62161 file processed (stat): 1 log points, error handler: false
19:05:54.489 info host client disconnected: buildHook 4100
19:05:54.489 info host client disconnected: buildHook 4100, has clients false

[somewhatabstract]

Looks like this is an outcome of not having a feature like #94. It's definitely a bug for me though since it breaks working tests in a less than transparent way - while #94 would provide a "fix", it's only one way this could be addressed.

[smcenlly]

For now, we recommend creating a VS Code workspace setting to disable jest for Console Ninja for your project:

Unfortunately this means that Console Ninja won't run when you're running your jest tests, but it will fix your problem.

---

Are you wanting to use Console Ninja when running your jest tests? Or is this just an unwanted / unexpected side-effect?

[somewhatabstract]

For now, we recommend creating a VS Code workspace setting to disable jest for Console Ninja for your project:

Unfortunately this means that Console Ninja won't run when you're running your `jest` tests, but it will fix your problem.

Are you wanting to use Console Ninja when running your jest tests? Or is this just an unwanted / unexpected side-effect?

In our current setup, I find it most useful when in testing scenarios like jest or storybook, so disabling jest, while addressing the issue, would also remove the utility of ConsoleNinja for me. I would rather see a fix for the underlying issue somehow.

[somewhatabstract]

Things that could be useful:

1. An ignore list setting (globs to ignore)
2. An ignore file like .consoleninjaignore
3. A file annotation I could add to a file like eslint suppressions (i.e. /* disable-console-ninja */)

[somewhatabstract]

The other issue here is that it was not obvious what the issue was. If there's a way to increase the signal that console ninja touched a file, that would be amazing. Like an annotation in the File Explorer, or some sort of verbose output, maybe?

[smcenlly]

Thanks for getting back to me. Your comments make sense.

In our current setup, I find it most useful when in testing scenarios like jest or storybook, so disabling jest, while addressing the issue, would also remove the utility of ConsoleNinja for me. I would rather see a fix for the underlying issue somehow.

Thanks - wasn't sure of your use case and wanted to confirm.

Things that could be useful:

An ignore list setting (globs to ignore)
An ignore file like .consoleninjaignore
A file annotation I could add to a file like eslint suppressions (i.e. /* disable-console-ninja */)

We had discussed something similar. Thinking about the future of what may be required for a project, we may have other project-level configuration that will need to be configured. Instead of an ignore file, we're thinking something like console-ninja.config (.json, .js, .cjs, .mjs, .etc). It may also be useful to support the configuration in package.json.

The other issue here is that it was not obvious what the issue was. If there's a way to increase the signal that console ninja touched a file, that would be amazing. Like an annotation in the File Explorer, or some sort of verbose output, maybe?

Good suggestion. Not entirely sure how we limit the noise, perhaps only emitting once per file. Having said that, I'm not sure how much it would help in this case (unless you've had the problem before and know to expect it).

---

We'll have a think about what we want to do to address your issue and we will get back to you.

[somewhatabstract]

LMK if I can help at all :)

[somewhatabstract]

The other issue here is that it was not obvious what the issue was. If there's a way to increase the signal that console ninja touched a file, that would be amazing. Like an annotation in the File Explorer, or some sort of verbose output, maybe?

Good suggestion. Not entirely sure how we limit the noise, perhaps only emitting once per file. Having said that, I'm not sure how much it would help in this case (unless you've had the problem before and know to expect it).

In this specific case, seeing an indication that the __examples__ files had been processed would've been a great signal, even not knowing immediately what had processed them. I don't know how that could've been surfaced (console.log in my jest output? VSCode indicator? 🤷), but it would've saved me quite a lot of confusion over the last few weeks as I tried to track this down.

I first saw it when I migrated the project to TypeScript and so I thought it was related to that for the longest time.

[ArtemGovorov]

Another workaround for now (before project-level config support gets added), is to use one of the following settings for the project to limit the files that Console Ninja instruments:

[somewhatabstract]

@ArtemGovorov Oh! That's a great suggestion. It's usually only open files I care about - so that would work well and I can set that as a workspace setting so others working with VSCode and ConsoleNinja will just get the right behavior.

Although, what happens if two VSCode projects are open; one only looking at opened files and one looking at all files? I noticed that currently, ConsoleNinja affects even things running from outside VSCode; so which instance of VSCode "wins" in that scenario? The most recent one opened?

[smcenlly]

Although, what happens if two VSCode projects are open; one only looking at opened files and one looking at all files? I noticed that currently, ConsoleNinja affects even things running from outside VSCode; so which instance of VSCode "wins" in that scenario? The most recent one opened?

This shouldn't cause a problem. Unless I misunderstand what you're asking, VS Code won't allow you to open the same project more than once, and Console Ninja starts for your project in each VS Code instance.

If you're seeing different behavior or having problems with multiple Console Ninja's interfering with each other, please let us know.

[somewhatabstract]

If you're seeing different behavior or having problems with multiple Console Ninja's interfering with each other, please let us know.

Ah, I get you. So it applies to the specific workspace. So when I ran an external console and saw an issue, it was because ConsoleNinja is monitoring that specific workspace, not all files anywhere. Got it. Thanks for clarifying that; makes sense. :)