Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Supported by

UnicodeEncodeError after using ü in loop variable

edited February 2016 in OpenSesame

Hi

I stumbled upon a problem with character encoding, when I want to concatenate strings to following way inside a loop-sequence block:
sendMessage(slideChangeEvent('Word_' + var.word))

It results in the following error message:

Error while executing inline script

item-stack: experiment[run].word_loop[run].word_sequence[run].sendMessageWord[run]
exception type: UnicodeEncodeError
exception message: 'ascii' codec can't encode character u'\xfc' in position 1: ordinal not in range(128)
item: sendMessageWord
time: Mon Feb 01 15:24:05 2016
phase: run

Traceback:
File "dist\libopensesame\inline_script.py", line 102, in run
File "dist\libopensesame\python_workspace.py", line 159, in _exec
File "", line 2, in
UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 1: ordinal not in range(128)

These are the possible values of var.word:

  • glücklich
  • wütend
  • traurig
  • erstaunt
  • angeekelt
    Of course the problem only arises with "glücklich" and "wütend".

How can i solve this problem?
Thanks for your suggestions.

Cheers,
Stefan

Comments

  • edited 10:59AM

    Hi Stefan,

    I suspect that this bit:

    'Word_' + var.word
    

    ... works fine, and returns a unicode object (e.g. u'Word_glücklich'). You can try this by just doing the concatenation on it's own, not embedded in a larger command.

    So the question is: What is slideChangeEvent or sendMessage doing that triggers a UnicodeEncodeError? Without seeing the full code I cannot tell!

    Cheers,
    Sebastiaan

    Buy Me A Coffee

  • edited February 2016

    Hi Sebastiaan

    You're right of course, a simple print('word' + var.word) is working fine.

    sendMessage()takes the string provided by slideChangeEvent() to send a message through a TCP socket to the external API of iMotions where we capture the people's facial responses together with markers sent from OpenSesame.

    The relevant code is the following:

    import socket
    
    ### Some global settings/variables used
    lnbr = '\r\n'
    IP = "127.0.0.1"
    EAPI_PORT = 8089        # iMotions external API
    CTRL_PORT = 8087        # iMotions Remote Control
    # iMotions parameters
    studyName = 'GamblingGame'  
    subjectName = str(var.subject_nr)
    
    # setup sockets and connect
    sockExtAPI = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sockExtAPI.connect((IP, EAPI_PORT))
    sockCTRL = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
    sockCTRL.connect((IP, CTRL_PORT))
    
    # send external API message (mouseEvent or slideChangeEvent)    
    def sendMessage(message):
        sockExtAPI.send(message)
        #response = sockCTRL.recv(4092)
        log.write('ExtAPI message sent: ' + message)
    
    def slideChangeEvent(slideID):
        # discrete header
        # version 2
        header = 'M;2;'
        # field 5: slideID
        # field 7: marker type N
        # (marks the start of the next segment, automatically closing any currently
        # open segment.)
        event = ';;' + slideID + ';;N;I'
        return header + event + lnbr
    

    Edit: nevermind my previous edit

  • edited 10:59AM

    I suspect the bad guy is here:

    sockExtAPI.send(message)
    

    In your case, message is a unicode object, which will be automatically converted to a str object. Kind of like this:

    _message = str(message)
    sockExtAPI.send(_message)
    

    And this goes wrong, because the default is assumed is ascii, which doesn't contain special characters. To make this unicode-safe, you need to explicitly say which character encoding you want to use. For example:

    _message = message.encode('utf-8')
    sockExtAPI.send(_message)
    

    This will send the message as a utf-8 encoded bytestring. Whether the receiver will take kindly to that is, of course, another matter.

    Cheers,
    Sebastiaan

    Buy Me A Coffee

  • edited 10:59AM

    sockExtAPI.send(message.encode('utf-8')) did the trick at least inside OpenSesame. However, write.log(message.encode('uft-8'))gives this funny mixture of symbols, same in the exported data from iMotions.

    Hm, looks like we will need to encode the stimuli differently.
    But thank you very much for your patience. I'm far from being experienced in python... :-)

    Cheers,
    Stefan

  • edited 10:59AM

    However, write.log(message.encode('uft-8'))gives this funny mixture of symbols, same in the exported data from iMotions.

    The (very common) mistake that you're making is thinking that the problem lies in how the file is written, whereas it lies in how you're reading it. The OpenSesame log file is fine, but it's utf-8 encoded. If it looks funny, this is because the text editor/ spreadsheet has used the wrong encoding to read it, and you'll have to explicitly tell it to use utf-8.

    See also:

    (I should write a blog about character encoding one of these days. It's really one of the main issues that people struggle with.)

    Buy Me A Coffee

  • edited April 5

    On Windows, many editors assume the default ANSI encoding (CP1252 on US Windows) instead of UTF-8 if there is no byte order mark (BOM) character at the start of the file. Files store bytes, which means all unicode have to be encoded into bytes before they can be stored in a file. read_csv takes an encoding option to deal with files in different formats. So, you have to specify an encoding, such as utf-8.

     df.to_csv('D:\panda.csv',sep='\t',encoding='utf-8')

    If you don't specify an encoding, then the encoding used by df.tocsv defaults to ascii in Python2, or utf-8 in Python3.

    Also, you can encode a problematic series first then decode it back to utf-8.

    df['column-name'] = df['column-name'].map(lambda x: x.encode('unicode-escape').decode('utf-8'))

    This will also rectify the problem.

  • Thanks so much for your contribution @carlhyde !

    Did you like my answer? Feel free to Buy Me A Coffee :)

Sign In or Register to comment.

agen judi bola , sportbook, casino, togel, number game, singapore, tangkas, basket, slot, poker, dominoqq, agen bola. Semua permainan bisa dimainkan hanya dengan 1 ID. minimal deposit 50.000 ,- bonus cashback hingga 10% , diskon togel hingga 66% bisa bermain di android dan IOS kapanpun dan dimana pun. poker , bandarq , aduq, domino qq , dominobet. Semua permainan bisa dimainkan hanya dengan 1 ID. minimal deposit 10.000 ,- bonus turnover 0.5% dan bonus referral 20%. Bonus - bonus yang dihadirkan bisa terbilang cukup tinggi dan memuaskan, anda hanya perlu memasang pada situs yang memberikan bursa pasaran terbaik yaitu http://45.77.173.118/ Bola168. Situs penyedia segala jenis permainan poker online kini semakin banyak ditemukan di Internet, salah satunya TahunQQ merupakan situs Agen Judi Domino66 Dan BandarQ Terpercaya yang mampu memberikan banyak provit bagi bettornya. Permainan Yang Di Sediakan Dewi365 Juga sangat banyak Dan menarik dan Peluang untuk memenangkan Taruhan Judi online ini juga sangat mudah . Mainkan Segera Taruhan Sportbook anda bersama Agen Judi Bola Bersama Dewi365 Kemenangan Anda Berapa pun akan Terbayarkan. Tersedia 9 macam permainan seru yang bisa kamu mainkan hanya di dalam 1 ID saja. Permainan seru yang tersedia seperti Poker, Domino QQ Dan juga BandarQ Online. Semuanya tersedia lengkap hanya di ABGQQ. Situs ABGQQ sangat mudah dimenangkan, kamu juga akan mendapatkan mega bonus dan setiap pemain berhak mendapatkan cashback mingguan. ABGQQ juga telah diakui sebagai Bandar Domino Online yang menjamin sistem FAIR PLAY disetiap permainan yang bisa dimainkan dengan deposit minimal hanya Rp.25.000. DEWI365 adalah Bandar Judi Bola Terpercaya & resmi dan terpercaya di indonesia. Situs judi bola ini menyediakan fasilitas bagi anda untuk dapat bermain memainkan permainan judi bola. Didalam situs ini memiliki berbagai permainan taruhan bola terlengkap seperti Sbobet, yang membuat DEWI365 menjadi situs judi bola terbaik dan terpercaya di Indonesia. Tentunya sebagai situs yang bertugas sebagai Bandar Poker Online pastinya akan berusaha untuk menjaga semua informasi dan keamanan yang terdapat di POKERQQ13. Kotakqq adalah situs Judi Poker Online Terpercayayang menyediakan 9 jenis permainan sakong online, dominoqq, domino99, bandarq, bandar ceme, aduq, poker online, bandar poker, balak66, perang baccarat, dan capsa susun. Dengan minimal deposit withdraw 15.000 Anda sudah bisa memainkan semua permaina pkv games di situs kami. Jackpot besar,Win rate tinggi, Fair play, PKV Games