[open] Unicode (special characters) errors on Windows

sviter · June 2013

I installed OpenSesame with standard installer and just try to run it, but it show Errors occurred massage. Log file is follow:

Traceback (most recent call last):
  File "opensesame", line 91, in <module>
  File "dist\libqtopensesame\qtopensesame.py", line 227, in resume_init
  File "dist\libqtopensesame\items\experiment.py", line 54, in __init__
  File "dist\libopensesame\experiment.py", line 124, in __init__
  File "dist\libopensesame\item.py", line 78, in __init__
  File "dist\libopensesame\experiment.py", line 261, in from_string
  File "dist\libopensesame\experiment.py", line 206, in parse_definition
  File "dist\libopensesame\plugins.py", line 91, in is_plugin
  File "dist\libopensesame\plugins.py", line 210, in plugin_folder
  File "dist\libopensesame\plugins.py", line 76, in plugin_folders
  File "ntpath.pyo", line 108, in join
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 26: ordinal not in range(128)

How can I fix it?

sebastiaan · June 2013

This looks like a problem with special characters in a pathname somewhere. Could you describe exactly how OpenSesame is installed, i.e. in which folder? Do you have a username with special characters in it?

sviter · June 2013

Yes, it was username problem. Thank you.

sebastiaan · June 2013

I listed this as an issue, and it should be fixed together with some other Unicode-related problems for the next maintenance release. In the meanwhile, please expose OpenSesame to as many weird characters as possible and inform us about any other problems!

sebastiaan · June 2013

Fyi, you could try the latest pre-release package (currently 0.27.3~pre1), which contains a number of Unicode-related bug-fixes:

http://osdoc.cogsci.nl/getting-started/getting-the-latest-development-snapshot/

Among other things, this should fix the username-issue that you encountered. (If not, please let me know.)

Cheers,
Sebastiaan

Koba · June 2013

Thank you always for your help.
I met with the same problem of 0.27.2 under Windows XP conditions. I changed the username and installed, but it didnot work. In my case, error is follow,

Traceback (most recent call last):  
 File "opensesame", line 91, in <module>

 File "dist\libqtopensesame\qtopensesame.py", line 227, in resume_init

 File "dist\libqtopensesame\items\experiment.py", line 54, in __init__

 File "dist\libopensesame\experiment.py", line 115, in __init__

 File "tempfile.pyo", line 327, in mkdtemp

 File "ntpath.pyo", line 108, in join

UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 12: ordinal not in range(128)

How can I do to fix?
Unfortunately I coud not access 0.27.3~pre1-win32-1.zip(http://files.cogsci.nl/software/opensesame/pre-releases/opensesame_0.27.3~pre1-win32-1.zip).

Yours sincerely,
Koba

sviter · June 2013

Simple changing of username isn't work. I created new user to solve this problem.
By the way I could not access 0.27.3~pre1-win32-1.zip too

sebastiaan · June 2013

Unfortunately I coud not access 0.27.3~pre1-win32-1.zip

It should be accessible now:

http://files.cogsci.nl/software/opensesame/pre-releases/

@sviter

Simple changing of username isn't work. I created new user to solve this problem.

Right, that's because Windows doesn't rename the profile folder when the username is changed. And the special characters in the profile folder cause the problem, not the username per se.

@Koba

In my case, error is follow (...) UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 12: ordinal not in range(128)

This seems to be another issue, but also related to special characters. It probably still persists with 0.27.3~pre1 (but please try and let me know). Does the problem also persist if you create a new Windows account without special characters in it (see above)?

Koba · June 2013

Thank you, sebastiaan and sviter.

1 0.27.3~pre1 did not work under special characters' account, but it works under newly created Windows account without special characters.

2 I could install 0.27.2 through the newly created account and it works. However, 0.27.2 didnot work through the other account.

3 I tried in another Windows XP, which is Japanese characters' account and properly working under Opensesame0.26. In this situation; A) I installed 0.27, before I uninstalled 0.26. In this case, 0.27 could run. If I first uninstalled 0.26 and installed 0.27., I failed to install 0.27. However, once 0.27 has been failed to install, I needed to run the Windows' restoration of the system in order to restore the deleated 0.26.

Sorry for poor explanation.

sviter · June 2013

@sebastiaan By the way, built-in OpenSesame function exp.get_file is also sensitive to nonstandard characters.

sebastiaan · June 2013

Hi guys,

Thanks for the feedback! Some Unicode bugs are clearly still in there, but I'm confident we're getting close to the point that OpenSesame will run smoothly on all locales.

One request though: Could you please also post the stacktrace (in case of a runtime error) or contents of opensesame.exe.log (in case of a failure to start) when reporting an error? Otherwise I cannot tell where the error comes from.

Cheers,
Sebastiaan

sviter · June 2013

@sebastiaan Sorry, I didn't post stacktrace because it's not a big problem. Just rename files.

# word_path = exp.get_file('Неконгруэнтные.txt')

Traceback (most recent call last):
  File "dist\libopensesame\inline_script.py", line 123, in prepare
  File "<string>", line 1, in <module>
  File "dist\libopensesame\experiment.py", line 389, in get_file
  File "ntpath.pyo", line 108, in join
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1: ordinal not in range(128)
Traceback (most recent call last):
  File "dist\libqtopensesame\qtopensesame.py", line 1393, in run_experiment
  File "dist\libopensesame\experiment.py", line 280, in run
  File "dist\libopensesame\sequence.py", line 107, in prepare
  File "dist\libopensesame\inline_script.py", line 125, in prepare
inline_error: Error: Inline script error
In: inline_script (prepare phase)
  File "ntpath.pyo", line 108, in join

Python traceback:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1: ordinal not in range(128)

sebastiaan · June 2013

Hi guys,

I have done some more work to improve support on systems with non-Latin characters. The latest pre-release (0.27.3~pre2) should address many of the issues described above. But, as always, let me know if your experience is otherwise. When reporting problems, please include opensesame.exe.log and/ or the contents of the debug window.

http://files.cogsci.nl/software/opensesame/pre-releases/

I have not been able to figure out what the problem was that @Koba reported, so that may or may not still occur.

Cheers!
Sebastiaan

PS. I have taken the liberty of changing the topic name to reflect that the topic is about Unicode-related problems.

Koba · July 2013

Hi sebastiann,

I tried 0.27.3~pre2 on Windows XP . Unfortunately, same as pre1, 0.27.3~pre2 did not work under special characters' account, but it works under the newly created Windows account without special characters. Here is opensesame.exe.log. "line"s have changed a little.

Traceback (most recent call last):
  File "opensesame", line 100, in <module>
  File "dist\libqtopensesame\qtopensesame.py", line 227, in resume_init
  File "dist\libqtopensesame\items\experiment.py", line 53, in __init__
  File "dist\libopensesame\experiment.py", line 115, in __init__
  File "tempfile.pyo", line 327, in mkdtemp
  File "ntpath.pyo", line 108, in join
UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 12: ordinal not in range(128)

Thank you.

sebastiaan · July 2013

Ok, I think I found the bug responsible. It's a general Python bug: http://bugs.python.org/issue1681974

I just uploaded 0.27.3~pre3, which should work around this issue. However, since I don't experience the bug myself, I cannot test it properly. Please let me know if this resolves the issue for you, in which case I'll release 0.27.3.

http://files.cogsci.nl/software/opensesame/pre-releases/

Cheers!

Koba · July 2013

Yes!
0.27.3~pre3 is working correctly on systems with Japanese characters.
Thank you so much.

jjholliday · July 2013

I have a similar issue with Korean characters, but it doesn't have to do with my username or reading in files (so I'm not sure if this is the right thread for this...)

If I have Korean words in a loop (as a variable called 'word') and then I want to, say, display the word in a sketchpad, I should be able to just use "[word]" to get it to show up. From my experience Opensesame is a bit finicky when it comes to getting it to do this.

When I make the sketchpad, I click on the "Ab" button to input text. If I don't do anything special and just enter the font family as "mono" and then input "[word]", all I get is empty boxes instead of text. No surprise there I suppose.

But if when entering the text, if I then click on the font family button, and then under writing systems click on "Korean", then I get a list of Korean fonts. I choose 바탕 (batang), and then input my text. The script looks like this:

draw textline 0.0 0.0 "The word is [word]." center=1 color=white font_family="바탕" font_size=18 font_italic=no font_bold=no show_if="always"

This never works, however, and the error I get is:

An unexpected error occurred, which was not caught by OpenSesame. This should not happen! Message:
The resource '바탕.ttf' could not be found in libopensesame.experiment.resource()

Meanwhile, I was able to get the batang.ttf file from a friend. If I do the exact same thing, but then edit the script to say font_family="batang", it works as long as the batang.ttf file is in the same directory as the experiment.

Here's the strange part, which makes me thing there is some bug at work.

If after I've successfully displayed Korean text using font_family="batang", and then on the same sketchpad if I try to input "[word]" using font_family="mono" (which should display empty boxes), it will work. It's like having Korean text on the sketchpad somehow "cures" it, and now it will display whatever is my loop regardless of the font family. But then if I delete the original line that has the font_family="batang" bit, and now all that's left is the second line that had font_family="mono", it goes back to not working, and just displaying empty boxes.

Has this issue been documented before? As it stands now it can be worked around, but it requires the user to have .ttf files for any Korean (and possible Asian) fonts one wants to use.

jjholliday · July 2013

And just for reference, I'm running this on Windows 7 under legacy.

sebastiaan · July 2013

Thanks for your feedback! The fact that you have to select a custom font to display particular alphabets is known. It's not really a bug, although it is definitely inconvenient. It is a result from the fact that the font rendering system doesn't automatically select a different font when it encounters a character that is not part of the specified font.

I choose 바탕 (batang (...) This never works, however

This sounds like there is a problem with locating fonts that have non-latin filenames (issue).

If after I've successfully displayed Korean text using font_family="batang", and then on the same sketchpad if I try to input "[word]" using font_family="mono" (which should display empty boxes), it will work. It's like having Korean text on the sketchpad somehow "cures" it, and now it will display whatever is my loop regardless of the font family.

That is very odd indeed. Are you sure that there is no trivial explanation for this? For example, one thing that comes to mind is that you may have placed the Batang font in the filepool under the name mono.ttf, or something along those lines?

jjholliday · July 2013

Are you sure that there is no trivial explanation for this? For example, one thing that > comes to mind is that you may have placed the Batang font in the filepool under the > name mono.ttf, or something along those lines?

I actually never even put batang.ttf in the file pool. I just leave it in the same directory and it gets detected automatically.

Howdy, Stranger!

Categories

[open] Unicode (special characters) errors on Windows

Comments

Howdy, Stranger!

Quick Links

Categories

[open] Unicode (special characters) errors on Windows

Comments