Category: V2V

Are saccades (rapid eye movements) synchronized to speech during lipreading?

Now I am reading the book “Visual Thinking for Design” by Colin Ware (ISBN 978-0-12-370896-0).  The book starts by describing the way the brain processes visual information.  Essentially, the brain processes it chunk by chunk, where the chunks are separated by saccades (rapid eye movements).

Since my V2V project requires finding a way to translate auditory information into visual information for processing by the brain, I’m looking also for the corresponding information about the way a brain processes auditory information.

One question, which arose in my mind, as I am reading the aforementioned book is as follows.

Assume a hearing (or hard of hearing) lipreader, who follows a speech by listening and uses lipreading as an auxiliary aid to filter out environmental noises and other speakers.  Given that the lipreader’s eyes perform saccades as usual, are the saccades synchronized to times at which the speaker produces vowels rather than consonants?

And would the saccades still by synchronized to vowel production periods also for a deaf lipreader?

Ramon Breakthrough 2012

קרן רמון הכריזה על תחרות בין הצעות לפרויקטים פורצי דרך, שישפיעו תוך 5 שנים על חייהם של לפחות 5 מיליון ישראלים.

עולים בדעתי ארבעה רעיונות כאלה, שחלקם דורשים כיפוף חוקים כדי להכנס למסגרת התחרות.

  • תצוגה חזותית של אותות דיבור – ישפיע על חייהם של כ-7000 ישראלים חרשים, כיפוף הכללים הדרוש הוא לקבל פרויקט שישפיע על לפחות מיליון חרשים מכל העולם.
  • חפירת מנהרות עמוק באדמה וחיים בהן, כדי לפנות מקום לחיות בר וצמחיה על פני כדור הארץ. יאפשר לישראל לקלוט עוד מיליון יהודים ויותר בלי שתהיה צפיפות בלתי נסבלת. הפרויקט ידרוש יותר מ-5 שנים כדי לגרום לשינוי משמעותי.
  • שינוי חברתי כך שיהיה מעמד חברתי גבוה יותר למי שיש לו יותר מיומנויות שעוזרות לשרוד ולהתאושש מאסון טבע או מעשה ידי אדם, במקום למי שמפגין יותר צריכה ראוותנית. קורסי הכשרה לכל הציבור ימומנו על ידי חברות הביטוח שמוכרות ביטוחים נגד האסונות למיניהם. ישפיע על כלל האוכלוסיה.
  • בתי מחסה לחתולי רחוב – לא רעיון שלי. ישפיע על כל מי שיש חתולי רחוב באזור מגוריו וזה כמעט כולנו.

הפרס הוא קורס באוניברסיטת הסינגולריטי.

אני מתלבט. האם להגיש הצעה אחת? האם להגיש את כל ההצעות שהן רעיונות שלי? האם בכלל להגיש הצעה? האם לא תהיה בעיית נגישות בקורס או בביצוע הפרויקט הזוכה לאחר מכן?

The Ramon Foundation announced a contest among proposals for breakthrough projects, which will affect the lives of at least million Israelis within 5 years.
I have in mind four such ideas, part of which require some rules to be bent in order to be admissible to the competition.

  • Visual representation of speech signals – will affect the lives of about 7000 Israeli Deaf persons, the needed rule bending will be to accept a project which will affect the lives of at least million Deaf persons from all over the world.
  • Digging tunnels deep in Earth and living in them, to free room for wild animals and vegetations on Earth surface. Will enable Israel to absorb another million Jews and more without incurring unbearable congestion. The project will need more than 5 years to cause a meaningful change.
  • Social change so that high social status will be granted to whomever has more survival and disaster recovery after natural disaster or human-made disaster, instead to whomever demonstrating conspicuous consumption. Training courses for the entire populace will be financed by insurance companies which sell insurances against the various disasters. Will affect the entire population.
  • Hostels for feral cats – not my idea. Will affect anyone whose neighborhood has feral cats and this is almost all of us.

The prize is a course in the Singularity University.

I am agonizing over it. To submit one proposal? To submit proposals for all my ideas? To submit a proposal at all? Won’t there be an accessibility problem in the course or in subsequent execution of the winning project?

Looking for research into visual information processing by humans

There is a family of cognitive tasks, whose successful execution requires continuous information processing by the visual center in the human brain. Examples of such tasks are:

  • Lipreading.
  • Combat airplane pilot following the airplane’s status displays during dogfights.
  • Following arena status displays during land battles by army unit commander (Israelis would recognize this as those plasma displays, which got bad rap during 2nd Lebanon War due to commanders staying in safe place while their soldiers risk their lives).
  • Ticker and notices to stockholders in stock exchanges, which need to be followed by stock traders.

Existing research in visual recognition, of which I am aware, deals with shape recognition, when the person being tested needs to perform a shape recognition operation not more frequently than once each several seconds.

What I need to find is research dealing with optimization of the way visual information is presented, so that recognition operations and subsequent information processing can happen several times each second.

So I am looking for help in finding such research. To be able to help me, you probably need to be researcher in the area of cognitive psychology and be aware of existing research worldwide.

If you know about such research, please let me know!