Title
#general
c

Chaim Sternbuch

03/13/2022, 6:17 PM
Hi Team, I am testing osquery cli on a japanese windows machine and it seems not to recognize japanese characters correctly. The query i run is SELECT path FROM file WHERE path like 'C:\Users\admin.TEST\Desktop\TestFolder%%'; , and in that folder i have a japanese file with the name ありがとうございます and it is not found, and i have a file name Jap1typeこんにちは.txt and the result that is returned {"path":"C:\Users\admin.TEST\Desktop\TestFolder\Jap1type縺薙s縺ォ縺。縺ッ.txt"}, (on an english machine the returned value is Jap1typeήΒΥήΓΥήΒτήΒκήΒψ.txt) , it is the same output in a command prompt and when redirected to a file. Anyone have an idea if osquery supports japanese (unicode) characters correctly and if yes, how can i get the expected result? any advice or direction is appreciated
Mike Myers

Mike Myers

03/14/2022, 8:05 PM
Hi Chaim. It looks like a path name is not being preserved in Unicode/UTF8, but that's just a guess. Historically, I think osquery has not really been internationalized.
c

Chaim Sternbuch

03/24/2022, 9:02 AM
Hi Mike, thanks for the response. If i use Japanese characters in the path itself it is responding correctly, just not in the file name.
9:04 AM
osquery not being internationalized might be the cause. thanks you
Mike Myers

Mike Myers

03/28/2022, 11:38 PM
I tested by creating the same filenames, and confirmed that they appear incorrectly when shown in osqueryi output, but also in a PowerShell window and in a CMD window, even though they appear "correctly" in Windows Explorer. It appears the problem is not in how osquery retrieves or stores or displays the names, but in literally the font of the console or PowerShell window. It is defaulted to the
Consolas
font, which doesn't have the glyphs for Japanese (or Chinese, or Korean) characters. Switching the shell to
MS Gothic
, the results appear correctly for me.
11:48 PM
But, there might be problems with some tables not listing Unicode filenames. As below, where I see
osquery> select COUNT(path) AS NumberOfFiles FROM file WHERE path like 'C:\Users\mikem\Desktop\%%';
+---------------+
| NumberOfFiles |
+---------------+
| 7             |
+---------------+
osquery> select COUNT(path) AS NumberOfFiles FROM hash WHERE path like 'C:\Users\mikem\Desktop\%%';
+---------------+
| NumberOfFiles |
+---------------+
| 3             |
+---------------+