Fortunately we have additional information that can help us with recognizing call signs. The radar, which every Air Traffic Tower has, tells us what planes are in the vicinity, and as an ATC could only be talking to one of these planes we therefore know that if a call sign was said it must be one of those that is on the radar. We developed two methods to use this information so as to improve callsign recognition.
The first modified the speech recognition system directly to boost the probability of recognizing the callsigns that we knew from the radar. Thanks to advances in efficient transducer composition[1] these modifications can be done so as to allow continuous updating of the model as new information from the radar comes in. This means that the model is tuned in real-time so it can adapt to changes in the real world. We published this technique in a paper at ICASSP 2021[2].
The second method post-processes the output of the speech recognition system to boost the probability of recognizing specific callsigns. This is simpler as it just involves rescoring the system output. Rescoring means giving certain outputs more weight, and thereby increasing the chance that the higher weighted terms are output as the model predictions. The rescoring was implemented as composition[3].
Both methods worked well, increasing call sign accuracy by up to 30\%. We believe there is still further room for improvement and plan on working further on this topic.
[1] Filters for Efficient Composition of Weighted Finite-State Transducers
[2] A comparison of methods for oov-word recognition on a new public dataset
]]>Visiting this site may result into the processing of data concerning identified or identifiable persons.
This page provides information on the processing operations performed on the personal data of the users visiting the ATCO2 project web site.
The information provided here only applies to the ATCO2 web site and does not concern any other online web sites, pages or services that can be accessed via the link https://atco2.org but relate to resources outside ATCO2 domain.
The information is also based on the Article 29 Working Party Guidelines on Transparency under Regulation 2016/679.
The data controller is the web site owner and ATCO2 project partner: Idiap Research Institute. Idiap’s legal adviser can be reached at the following address: (legal at idiap.ch).
Idiap’s mandate is to promote the cultural, social and economic development of Switzerland by the training of the new generation of top-tier scientists and professionals and to develop research excellence for the Society and its Economy. Idiap processes personal data in particular in line with its three academic core tasks which are the Education, Research and Innovation, and its supporting services.
Processing is carried out under the Swiss Federal Data Protection Act dated 19 June 1992 (FDA/R.S 235.1) as well as the Regulation (EU) 2016/679 of the European parliament and of the Council of 27 April 2016 on the protection of natural persons with regards to the processing of personal data and on the free movement of such data (hereinafter “General Data Protection Regulation”, or “GDPR”).
In case you need any information or support concerning the processing of your personal data, please contact us at the following address: (legal at idiap.ch)
The information systems and software procedures relied upon to operate this website acquire personal data as part of their standard functioning; the transmission of such data is an inherent feature of Internet communication protocols.
This data category includes IP addresses and/or the domain names of the computers and devices used by any user connecting with this website, the URI/URL (Uniform Resource Identifier/Locator) addresses of the requested resources, the time of such requests, the method used for submitting a given request to the server, the returned file size, a numerical code relating to server response status (successfully performed, error, etc.), and other parameters related to the user's operating system and computer environment.
These data are necessary to use web-based services and are also processed in order to
Browsing data are kept for no longer than 26 months and for statistical purposes only (except where judicial authorities need such data for establishing the commission of criminal offenses).
Such information is not collected in order to relate it to identified data subjects, however it might allow user identification per se after being processed and matched with data held by third parties.
Sending messages to the e-mail and postal addresses mentioned on this website or filling in the contact form made available on the ATCO2 website, or sending private messages to ATCO2 social media pages and profiles (where such option is available) is done on the basis of a freely chosen, explicit, and voluntary option.
This implies the acquisition of the sender’s e-mail address, which is necessary in order to reply to any request, as well as of such additional personal data as is contained in the message(s) and/or in the form. Specific information notices will be displayed on the pages made available to users for providing certain services, sending messages and requests.
There are three types of cookies used in the ATCO2 website: 1) cookies for the basic functioning of the website, 2) cookies for Google Analytics and 3) cookies collected by Twitter.
1) Basic Cookies
Cookies are used for improving the user browsing experience, to remember information about your browser and your preferences.
Find hereafter cookies for the basic functioning of the website:
2) Cookies for Google Analytics
The ATCO2 website uses Universal Analytics plugin developed by Google. This plugin uses these 3 cookies:
These cookies are used only for statistical purposes within the ATCO2 project (e.g number of visitors, page view). IP addresses are fully anonymized. More information can be found here.
3) cookies collected by Twitter
Third-party cookies are collected by Twitter since the ATCO2 website contains a Twitter widget.
Twitter uses the following types of cookies:
Authentication and security: To log you into Twitter.
Preferences: To remember information about your browser and your preferences.
Analytics and research: To help us improve and understand how people use our services, including Twitter buttons and widgets, and Twitter Ads.
Personalized content: To customize our services with more relevant content, like tailored trends, stories, ads, and suggestions for people to follow.
Advertising: To help us deliver ads, measure their performance, and make them more relevant to you based on criteria like your activity on Twitter and visits to our ad partners' websites.
Personalization across devices: By better understanding how devices are related, we can use information from one device to help personalize the Twitter experience on another device. More information can be found here.
The processing operations related to the web-based services that are made available via this website are carried out at Idiap offices, exclusively by duly authorised staff in charge of said processing, or else by persons tasked with such maintenance activities as may be necessary from time to time. Data are hosted at Idiap virtual servers located in Idiap’s premises.
As concerns data voluntarily provided by users, they will be processed only for the specific and legitimate purposes indicated in this Privacy Policy, which are summarised below.
Type of data | Purpose(s) | Legal basis |
Data processed automatically by the web site (Anonymized IP address, language, captcha, cookiebar, | Correct functioning of the ATCO2 web site. | Legitimate interest |
Data voluntarily communicated by users (name, e-mail address, data contained in the on-line forms) | Replying/fulfilling the user’s requests | User’s explicit informed consent |
Personal data sent by e-mail or via on-line forms will only be processed to reply to senders and to comply with their requests.
Any personal data that are provided by users requesting information, answers to questions and materials such as brochures, DVD, newsletters, etc. is only used to provide the service and/or discharge the tasks requested and is disclosed to third parties only if this is necessary for the said purposes.
Only data strictly necessary to achieve the above-mentioned purposes will be processed.
The legal basis for the processing will be the users’ informed consent.
Collected data, browsing data and tracking devices data are processed on the basis of the legitimate interest of the controller as provided by section 6(1)(f) of the GDPR, in the present case this interest is constituted by the proper functioning of the website, the realization of user statistics and on the basis of the consent of the users as provided by section 6(1)(a) of the GDPR.
Third party Twitter data processing:
What information does Twitter collect through Twitter for Websites and how is that information used?
When you view Twitter content or Twitter products integrated into other websites using Twitter for Websites, Twitter may receive information including the web page you visited, your IP address, browser type, operating system, and cookie information. This information helps Twitter to improve their products and services, including personalized suggestions and personalized ads. Learn more about the information Twitter receives and how they use it in their privacy policy and cookies use.
To protect privacy, Twitter never associates this web browsing history with users’ names, email addresses, phone numbers, or Twitter handles, and they delete, obfuscate, or aggregate it after no longer than 30 days, as explained in their Privacy Policy.
Personal data are processed with electronic and manual means, for no longer than is necessary to achieve the purposes for which they have been collected as indicated below; the logic involved in this will be strictly consistent with the above-mentioned purposes. Specific adequate technical and organizational security measures are implemented to prevent the data from being lost, used unlawfully and/or inappropriately, and accessed without authorization.
Personal data resulting from web-based services may be communicated to other ATCO2 partners, only for the purposes specified in this Privacy Policy. The complete list of partners with contact details is available here.
Personal data may be communicated to competent authorities (e.g. police, Courts) upon request, in compliance with applicable law, e.g. for the prevention of crimes or to defend a right in legal proceedings.
Data will be processed by duly authorized personnel at Idiap or by other ATCO2 partners.
Data may be processed by technical personnel, whether internal or external to Idiap, only to the extent that it is strictly needed to solve technical (e.g. IT) problems and/or to provide technical assistance.
No personal data will be disseminated, unless foreseen for project dissemination and communication activities and with the explicit consent of the data subjects.
Subject to the specifications made with regard to browsing data, the provision of personal data to the ATCO2 website (e.g. via e-mail and/or on-line forms) is free and fully voluntary.
If users fail to provide the personal data needed to reply to users’ requests, it may be impossible to reply to their requests.
Personal data will be stored only for the period strictly necessary to guarantee the basic functioning of the website and to reply to users’ requests and, in any case, no longer than 26 months (duration of the ATCO2 project).
No personal data will be transferred by Idiap to countries outside Switzerland and the European Union. Nevertheless third parties like Twitter and Google have their own regulations on this topic.
Find hereafter the personal data that might be transferred to third countries.
Twitter:
Through the Twitter widget integrated in the ATCO2 website, Twitter may receive information including the web page you visited, your IP address, browser type, operating system, and cookie information. Learn more about the information Twitter receives and how they use it in their privacy policy and cookies use.
Google:
Through Google Analytics, Google may receive information including the web page you visited, browser type, operating system, and cookie information. IP addresses are fully anonymised. More information can be found here.
The ATCO2 web site users, as data subjects, have the right to obtain at any time from the controller, without constraint, at reasonable intervals and without excessive delay or expense:
Users have the right to object in all cases, on legitimate grounds, to the processing of their personal data.
Users have the right to data portability, i.e. to receive their personal data in a structured, commonly used and machine-readable format and to transmit them to another controller.
The controller shall provide a copy of the personal data undergoing processing. For any further copies requested by the data subject, the controller may charge a reasonable fee based on administrative costs. Where the data subject makes the request by electronic means, and unless otherwise requested by the data subject, the information shall be provided in a commonly used electronic form. The right to obtain such copy shall not adversely affect the rights and freedoms of others.
Requests should be e-mailed to (webmaster at idiap.ch)
In all those cases in which processing is based on consent, users have the right to withdraw their consent at any time. The withdrawal of consent does not affect the lawfulness of processing based on consent before its withdrawal. Users can exercise this right by contacting (legal at idiap.ch)
No decisions on users are made solely on the basis of the automated processing of their personal data, including profiling.
This Privacy Policy may be updated as necessary, for instance following updates in applicable laws. The updates will be applicable as soon as they are published on the website. Users are kindly invited to check it periodically. The present Privacy Policy may change with the Law and the doctrine of the Federal Data protection Officer.
Swiss Law and in particular the Swiss Federal Act on Data Protection (FADP) is applicable. The place of execution and of jurisdiction is at the registered office of Idiap.
Website https://www.atco2.org/ is the property of the Idiap Research Institute (Idiap).
Idiap takes great care when presenting information on its websites. Idiap cannot be held responsible for third-party content, even if ATCO2 website link or refer to them.
Idiap reserves the right to modify or erase content published on its websites without prior notice and at any time.
For any questions regarding data protection or the use of cookies, please contact (info at idiap.ch)
Google Analytics and users data
On most of its websites, Idiap uses Google Analytics to collect non-personal visitor data in order to generate metrics that will help improve its content and better organize information. Idiap uses the anonymized version of Google Analytics which transmits only a subset of the visitor’s IP address to Google.
Like all analytics software, Google Analytics uses “cookies”, which are text files inserted in the user’s computer, in order to help websites analyze their traffic. By visiting and using ATCO2 website, you agree to allow Idiap webmaster as well as Google collect your anonymous data.
]]>The ATCO2 project is aimed at the development of an automatic tool for air-traffic data transcription. The - automatic - means, that a speech communication between pilots and ATC will be transcribed using a machine into a textual form (including tagging of important parts like commands, call-signs, values, names, etc.) Next, a web interface (transcription platform) will be developed and fully available for enthusiasts to verify the automatic transcripts. So the output of the transcription platform is data -- lots of data available for the community, research and commerce. Having such data, one can build its own tool to process the speech data and get some useful information out of it.
To make this happen, we need several components:
1] Device for capturing and recording the voice data. ATCO2 project aims to keep the equipment as lightweight as possible to make sure that the receiver installation is easy for the users and at the same time hardware cost is low.
2] Automatic speech recognizers which pre-transcribe and tag the recorded voice data.
3] Transcription platform, where enthusiasts are involved. They help to verify and correct the pre-transcribed data. This process can be done in two ways:
These transcripts will be used as a feedback loop for the automatic speech recognizers to improve themselves by the way.
However, to train a speech recognizer to automatically transcribe the ATC communication reasonably well, we need training and test data. The closer the training data is to the target domain (ATC communication), the better. The training and test data must be carefully transcribed by humans. We can use of course other transcribed data. We even have it. But this data is mostly out-of-domain (youtube videos, read books, telephone conversations, ...), too far from the ATC communications.
We started to record ATC data ad-hoc. The data is processed by voice activity detection (to keep only speech segments) and speaker segmentation (tries to guess which segments are spoken by the same person). These audio segments (~5 minutes long) are provided to a dozen of our annotators. They are the first enthusiasts and volunteers who came on board (thanks to our project partner, the OpenSky Network). These volunteers get the audio segments and listen to it and transcribe it. It is a difficult work, as they need to search for other sources to get the metadata (call signs, local names of runways, etc) so that the final transcription of controller-pilot communication is correct.
The process is shown in the figures below. The pre-transcriptions of the speech (in given figure):
are converted into these transcripts:
To make sure that the process of verification is uniform across different volunteers, the transcribers need to go through a 4 pages long annotation manual so that the speech is transcribed consistently.
The web interface you see is a transcription portal developed by for SpokenData.com service which we (ReplayWell - as the partner of the project) bring into the project. As you can see, the volunteers need to not only transcribe the speech (text in the middle of the picture), but also assign the identities of particular speech segments to the speakers (bottom right list - call signs, tower, etc). They have a waveform on the left side and they also have to check if the automatic segmentation is correct.
ReplayWell’s main goal is to redesign the SpokenData.com for ATCO2 project, to serve the volunteers and provide them all the important information, so they will have all the information on one web-page. And this is the second role of the first dozen of volunteers - provide us with the feedback. We need to understand the process of their transcription, what other data they need and make their life easier. At least, the transcriber’s life would be easier :)
At the time of publishing this blog post, volunteers already transcribed over 3 hours of audio resulting in one hour of speech. This is the perfect starting point to evaluate the automatic speech recognizers for ATCO2 project.
Thank you guys for doing that job! Very appreciated!
Igor@ReplayWell
]]>
Air-Traffic Control (ATC) Radio and Automatic Dependent Surveillance-Broadcast data (ADS-B) provides complementary information. Listening to ATC radio gives insights into upcoming actions of the aircraft based on the information exchange between an Air-Traffic Controller (ATCo) and the pilot. The actual position of the aircraft at any point in time is provided by the radar data. ADS-B data. ADS-B contains an information such as latitude, longitude, altitude and velocity value of a plane at a current point in time. The ADS-B data is collected by Open-Sky server with a resolution of a few seconds and gives therefore a fine granular view on the trajectory of the plain. The matching of the radar data with the speech results therefore in an increase of information. Figure 1 shows the process of the matching, which can be broken down into 3 steps:
1) Matching the callsign in the automatically generated transcript (from voice command issued by ATCo) with the callsign in the radar data:
The voice is transcribed, either by human annotators or by an automatic speech recognition (see corresponding blogs such as here). Inside these transcripts, the callsigns are automatically detected. This detection can be refined manually by a human annotator with the tool displayed in Figure 2. The callsigns are then transformed into the ICAO format, which consists of 3 character ICAO designator followed by the fight ID consisting of letters and numbers. After the conversion, the matching procedure is followed between the callsigns of the radar data.
Figure 1: Matching process.
2) Matching the voice recording time with the timestamp extracted from the radar data:
To get the coordinates of the plane at the time, a voice command given by the ATCo, timestamp extracted from the voice command (i.e. from the the transcript) needs to be matched with the timestamp provided by the radar data which corresponds to the same callsign.
3) Getting the coordinates and combining the data:
When callsign and timestamp are matched between radar and voice command (transcript), the latitude and longitude of plane at the time can be extracted. The combined data can then be plotted. In the example from Figure 1, it is clearly visible that the plane turns left after the turn-left command is spoken.
Figure 2: Callsign detection refinement.
This alignment of the input data is the starting point for several downstream tasks, therefore it is crucial to minimize the misalignments at this stage of the data processing.
If you have questions or comments, fell free to contact us through the contact form.
]]>